Compare commits

..

20 commits

Author SHA1 Message Date
b4ddbf62d0
Videos: Fix audio tracks language.
Video will only return the default language. The rest of the audio
tracks are deleted since they will not be used.
2024-10-31 20:11:49 -03:00
5bac8499ac
External Proxies: Rotate between proxies with balance enabled
Closes #17
2024-10-31 20:11:49 -03:00
e8b37e1760
Config: Also reload env variables 2024-10-31 20:11:49 -03:00
174d468c9c
Use POST requests for /videoplayback requests 2024-10-31 20:11:49 -03:00
8b5edb7c9f
Config: Reload configuration on modification
It detects changes on the config.yml automtically if invidious is
running on linux. If not, the configuration can be reloaded using
`kill -s HUP $(pidof invidious)` or any other tool that sends a SIGHUP
signal to the invidious process.

Closes #16
2024-10-31 20:11:49 -03:00
6e0ab02cd6
Tokens: Option to disable user tokens. 2024-10-31 20:11:49 -03:00
15ced487e8
Tokens: Server side generated tokens.
#18
2024-10-31 20:11:49 -03:00
c04d0e26ae
PubSub: Use external domain for pubsub feeds 2024-10-31 20:11:48 -03:00
0883d9c8fe
External Proxies: Proxyfi HLS Playlists 2024-10-31 20:11:48 -03:00
acc22f1741
Videos: Completly disable annotations due to archive.org being down
Closes #15
2024-10-31 20:11:48 -03:00
d9a00ca397
Tokens: Refresh po_token and visitor_data every 5 seconds
Closes #11
2024-10-31 20:11:48 -03:00
99efc4c10f
External Proxies: Proxyfi HD720 2024-10-31 20:11:48 -03:00
f4d76bcad0
Videos: Increase video cache to 4 hours 2024-10-31 20:11:48 -03:00
c6d96c7276
Feat: Experimental support for potoken inside redis
Using https://git.nadeko.net/Fijxu/youtube-po-token-generator
2024-10-31 20:11:48 -03:00
d189e2ee6e
External Proxies: Use list of external videoplayback proxies 2024-10-31 20:11:48 -03:00
Samantaz Fox
08d2471ebd
Videos: Fix missing host parameter on playback URLs when local=true 2024-10-31 20:11:47 -03:00
c8c20e4592
CI: Experimental branches for testing builds 2024-10-31 20:11:47 -03:00
832fa5e601
Feat: User supplied po_token and visitor_data 2024-10-31 20:11:47 -03:00
af9e8665c8
Small try. 2024-10-31 20:11:47 -03:00
857afcf239
Feeds: Get rid of feed_needs_update() since it appears to be unused 2024-10-31 20:11:47 -03:00
25 changed files with 177 additions and 264 deletions

View file

@ -37,15 +37,13 @@ jobs:
type=sha,format=short,prefix={{date 'YYYY.MM.DD'}}-,enable=${{ github.ref == format('refs/heads/{0}', 'master') }}
type=raw,value=latest,enable=${{ github.ref == format('refs/heads/{0}', 'master') }}
- uses: https://code.forgejo.org/docker/build-push-action@v6
- uses: https://code.forgejo.org/docker/build-push-action@v5
name: Build images
with:
context: .
file: docker/Dockerfile
tags: ${{ steps.meta.outputs.tags }}
platforms: linux/amd64
# cache-from: type=gha
# cache-to: type=gha,mode=max
push: true
build-args: |
"release=1"

View file

@ -38,11 +38,10 @@ jobs:
matrix:
stable: [true]
crystal:
- 1.9.2
- 1.10.1
- 1.11.2
- 1.12.1
- 1.13.2
- 1.14.0
include:
- crystal: nightly
stable: false
@ -52,11 +51,6 @@ jobs:
with:
submodules: true
- name: Install required APT packages
run: |
sudo apt install -y libsqlite3-dev
shell: bash
- name: Install Crystal
uses: crystal-lang/install-crystal@v1.8.0
with:

View file

@ -5,12 +5,6 @@
### Full list of pull requests merged since the last release (newest first)
* Add "Filipino (auto-generated)" to the list of caption languages ([#4995], by @SamantazFox)
* Makefile: Add MT option to enable the 'preview_mt' flag ([#4993], by @SamantazFox)
* SigHelper: Reconnect to signature helper ([#4991], thanks @Fijxu)
* Fix player menus hiding onHover ready ([#4750], thanks @giacomocerquone)
* Use connection pools when requesting images from YouTube ([#4326], thanks @syeopite)
* Add support for using Invidious through a HTTP Proxy ([#4270], thanks @syeopite)
* Search: Fix 'youtu.be' URLs in sanitizer ([#4894], by @SamantazFox)
* Ameba: Disable Style/RedundantNext rule ([#4888], thanks @syeopite)
* Playlists: Fix 'invalid byte sequence' error when subscribing ([#4887], thanks @DmitrySandalov)
@ -28,10 +22,7 @@
[#4122]: https://github.com/iv-org/invidious/pull/4122
[#4193]: https://github.com/iv-org/invidious/pull/4193
[#4270]: https://github.com/iv-org/invidious/pull/4270
[#4326]: https://github.com/iv-org/invidious/pull/4326
[#4652]: https://github.com/iv-org/invidious/pull/4652
[#4750]: https://github.com/iv-org/invidious/pull/4750
[#4850]: https://github.com/iv-org/invidious/pull/4850
[#4862]: https://github.com/iv-org/invidious/pull/4862
[#4863]: https://github.com/iv-org/invidious/pull/4863
@ -42,9 +33,6 @@
[#4928]: https://github.com/iv-org/invidious/pull/4928
[#4930]: https://github.com/iv-org/invidious/pull/4930
[#4942]: https://github.com/iv-org/invidious/pull/4942
[#4991]: https://github.com/iv-org/invidious/pull/4991
[#4993]: https://github.com/iv-org/invidious/pull/4993
[#4995]: https://github.com/iv-org/invidious/pull/4995
## v2.20240825.2 (2024-08-26)

View file

@ -7,11 +7,6 @@ STATIC := 0
NO_DBG_SYMBOLS := 0
# Enable multi-threading.
# Warning: Experimental feature!!
# invidious is not stable when MT is enabled.
MT := 0
FLAGS ?=
@ -24,10 +19,6 @@ ifeq ($(STATIC), 1)
FLAGS += --static
endif
ifeq ($(MT), 1)
FLAGS += -Dpreview_mt
endif
ifeq ($(NO_DBG_SYMBOLS), 1)
FLAGS += --no-debug

View file

@ -68,7 +68,6 @@
.video-js.player-style-youtube .vjs-menu-button-popup .vjs-menu {
margin-bottom: 2em;
padding-top: 2em
}
.video-js.player-style-youtube .vjs-progress-control .vjs-progress-holder, .video-js.player-style-youtube .vjs-progress-control {height: 5px;

View file

@ -173,17 +173,6 @@ https_only: false
##
#force_resolve:
##
## Configuration for using a HTTP proxy
##
## If unset, then no HTTP proxy will be used.
##
http_proxy:
user:
password:
host:
port:
##
## Use Innertube's transcripts API instead of timedtext for closed captions

View file

@ -1,4 +1,4 @@
FROM crystallang/crystal:1.14.0-alpine AS builder
FROM crystallang/crystal:1.12.1-alpine AS builder
RUN apk add --no-cache sqlite-static yaml-static

View file

@ -287,7 +287,6 @@
"Esperanto": "Esperanto",
"Estonian": "Estonian",
"Filipino": "Filipino",
"Filipino (auto-generated)": "Filipino (auto-generated)",
"Finnish": "Finnish",
"French": "French",
"French (auto-generated)": "French (auto-generated)",

2
mocks

@ -1 +1 @@
Subproject commit b55d58dea94f7144ff0205857dfa70ec14eaa872
Subproject commit 11ec372f72747c09d48ffef04843f72be67d5b54

View file

@ -10,7 +10,7 @@ shards:
backtracer:
git: https://github.com/sija/backtracer.cr.git
version: 1.2.2
version: 1.2.1
db:
git: https://github.com/crystal-lang/crystal-db.git
@ -23,9 +23,6 @@ shards:
inotify:
git: https://github.com/petoem/inotify.cr.git
version: 1.0.3
http_proxy:
git: https://github.com/mamantoha/http_proxy.git
version: 0.10.3
kemal:
git: https://github.com/kemalcr/kemal.git
@ -57,7 +54,7 @@ shards:
spectator:
git: https://github.com/icy-arctic-fox/spectator.git
version: 0.10.6
version: 0.10.4
sqlite3:
git: https://github.com/crystal-lang/crystal-sqlite3.git

View file

@ -33,9 +33,6 @@ dependencies:
inotify:
github: petoem/inotify.cr
version: 1.0.3
http_proxy:
github: mamantoha/http_proxy
version: ~> 0.10.3
development_dependencies:
spectator:

View file

@ -17,8 +17,8 @@ Spectator.describe "parse_video_info" do
# Basic video infos
expect(info["title"].as_s).to eq("I Gave My 100,000,000th Subscriber An Island")
expect(info["views"].as_i).to eq(220_226_287)
expect(info["likes"].as_i).to eq(6_870_691)
expect(info["views"].as_i).to eq(126_573_823)
expect(info["likes"].as_i).to eq(5_157_654)
# For some reason the video length from VideoDetails and the
# one from microformat differs by 1s...
@ -48,12 +48,12 @@ Spectator.describe "parse_video_info" do
expect(info["relatedVideos"].as_a.size).to eq(20)
expect(info["relatedVideos"][0]["id"]).to eq("krsBRQbOPQ4")
expect(info["relatedVideos"][0]["title"]).to eq("$1 vs $250,000,000 Private Island!")
expect(info["relatedVideos"][0]["id"]).to eq("Hwybp38GnZw")
expect(info["relatedVideos"][0]["title"]).to eq("I Built Willy Wonka's Chocolate Factory!")
expect(info["relatedVideos"][0]["author"]).to eq("MrBeast")
expect(info["relatedVideos"][0]["ucid"]).to eq("UCX6OQ3DkcsbYNE6H8uQQuVA")
expect(info["relatedVideos"][0]["view_count"]).to eq("230617484")
expect(info["relatedVideos"][0]["short_view_count"]).to eq("230M")
expect(info["relatedVideos"][0]["view_count"]).to eq("179877630")
expect(info["relatedVideos"][0]["short_view_count"]).to eq("179M")
expect(info["relatedVideos"][0]["author_verified"]).to eq("true")
# Description
@ -76,11 +76,11 @@ Spectator.describe "parse_video_info" do
expect(info["ucid"].as_s).to eq("UCX6OQ3DkcsbYNE6H8uQQuVA")
expect(info["authorThumbnail"].as_s).to eq(
"https://yt3.ggpht.com/fxGKYucJAVme-Yz4fsdCroCFCrANWqw0ql4GYuvx8Uq4l_euNJHgE-w9MTkLQA805vWCi-kE0g=s48-c-k-c0x00ffffff-no-rj"
"https://yt3.ggpht.com/ytc/AL5GRJVuqw82ERvHzsmBxL7avr1dpBtsVIXcEzBPZaloFg=s48-c-k-c0x00ffffff-no-rj"
)
expect(info["authorVerified"].as_bool).to be_true
expect(info["subCountText"].as_s).to eq("320M")
expect(info["subCountText"].as_s).to eq("143M")
end
it "parses a regular video with no descrition/comments" do
@ -99,8 +99,8 @@ Spectator.describe "parse_video_info" do
# Basic video infos
expect(info["title"].as_s).to eq("Chris Rea - Auberge")
expect(info["views"].as_i).to eq(14_324_584)
expect(info["likes"].as_i).to eq(35_870)
expect(info["views"].as_i).to eq(10_943_126)
expect(info["likes"].as_i).to eq(0)
expect(info["lengthSeconds"].as_i).to eq(283_i64)
expect(info["published"].as_s).to eq("2012-05-21T00:00:00Z")
@ -132,14 +132,14 @@ Spectator.describe "parse_video_info" do
# Related videos
expect(info["relatedVideos"].as_a.size).to eq(20)
expect(info["relatedVideos"].as_a.size).to eq(19)
expect(info["relatedVideos"][0]["id"]).to eq("gUUdQfnshJ4")
expect(info["relatedVideos"][0]["title"]).to eq("Chris Rea - The Road To Hell 1989 Full Version")
expect(info["relatedVideos"][0]["author"]).to eq("NEA ZIXNH")
expect(info["relatedVideos"][0]["ucid"]).to eq("UCYMEOGcvav3gCgImK2J07CQ")
expect(info["relatedVideos"][0]["view_count"]).to eq("53298661")
expect(info["relatedVideos"][0]["short_view_count"]).to eq("53M")
expect(info["relatedVideos"][0]["id"]).to eq("Ww3KeZ2_Yv4")
expect(info["relatedVideos"][0]["title"]).to eq("Chris Rea")
expect(info["relatedVideos"][0]["author"]).to eq("PanMusic")
expect(info["relatedVideos"][0]["ucid"]).to eq("UCsKAPSuh1iNbLWUga_igPyA")
expect(info["relatedVideos"][0]["view_count"]).to eq("31581")
expect(info["relatedVideos"][0]["short_view_count"]).to eq("31K")
expect(info["relatedVideos"][0]["author_verified"]).to eq("false")
# Description
@ -156,13 +156,11 @@ Spectator.describe "parse_video_info" do
# Author infos
expect(info["author"].as_s).to eq("ChrisReaVideos")
expect(info["author"].as_s).to eq("ChrisReaOfficial")
expect(info["ucid"].as_s).to eq("UC_5q6nWPbD30-y6oiWF_oNA")
expect(info["authorThumbnail"].as_s).to eq(
"https://yt3.ggpht.com/ytc/AIdro_n71nsegpKfjeRKwn1JJmK5IVMh_7j5m_h3_1KnUUg=s48-c-k-c0x00ffffff-no-rj"
)
expect(info["authorThumbnail"].as_s).to be_empty
expect(info["authorVerified"].as_bool).to be_false
expect(info["subCountText"].as_s).to eq("3.11K")
expect(info["subCountText"].as_s).to eq("-")
end
end

View file

@ -23,7 +23,6 @@ require "kilt"
require "./ext/kemal_content_for.cr"
require "./ext/kemal_static_file_handler.cr"
require "http_proxy"
require "athena-negotiation"
require "openssl/hmac"
require "option_parser"
@ -115,10 +114,6 @@ SOFTWARE = {
YT_POOL = YoutubeConnectionPool.new(YT_URL, capacity: CONFIG.pool_size)
# Image request pool
GGPHT_POOL = YoutubeConnectionPool.new(URI.parse("https://yt3.ggpht.com"), capacity: CONFIG.pool_size)
# CLI
Kemal.config.extra_options do |parser|
parser.banner = "Usage: invidious [arguments]"

View file

@ -23,31 +23,14 @@ def produce_channel_content_continuation(ucid, content_type, page = 1, auto_gene
else 15 # Fallback to "videos"
end
sort_type_numerical =
case content_type
when "videos" then 3
when "livestreams" then 5
else 3 # Fallback to "videos"
sort_by_numerical =
case sort_by
when "newest" then 1_i64
when "popular" then 2_i64
when "oldest" then 4_i64
else 1_i64 # Fallback to "newest"
end
if content_type == "livestreams"
sort_by_numerical =
case sort_by
when "newest" then 12_i64
when "popular" then 14_i64
when "oldest" then 13_i64
else 12_i64 # Fallback to "newest"
end
else
sort_by_numerical =
case sort_by
when "newest" then 1_i64
when "popular" then 2_i64
when "oldest" then 4_i64
else 1_i64 # Fallback to "newest"
end
end
object_inner_1 = {
"110:embedded" => {
"3:embedded" => {
@ -58,7 +41,7 @@ def produce_channel_content_continuation(ucid, content_type, page = 1, auto_gene
"2:embedded" => {
"1:string" => "00000000-0000-0000-0000-000000000000",
},
"#{sort_type_numerical}:varint" => sort_by_numerical,
"3:varint" => sort_by_numerical,
},
},
},

View file

@ -57,15 +57,6 @@ struct ConfigPreferences
end
end
struct HTTPProxyConfig
include YAML::Serializable
property user : String
property password : String
property host : String
property port : Int32
end
class Config
include YAML::Serializable
@ -166,8 +157,6 @@ class Config
property host_binding : String = "0.0.0.0"
# Pool size for HTTP requests to youtube.com and ytimg.com (each domain has a separate pool of `pool_size`)
property pool_size : Int32 = 100
# HTTP Proxy configuration
property http_proxy : HTTPProxyConfig? = nil
# Use Innertube's transcripts API instead of timedtext for closed captions
property use_innertube_for_captions : Bool = false

View file

@ -18,40 +18,6 @@ end
class HTTP::Client
property family : Socket::Family = Socket::Family::UNSPEC
# Override stdlib to automatically initialize proxy if configured
#
# Accurate as of crystal 1.12.1
def initialize(@host : String, port = nil, tls : TLSContext = nil)
check_host_only(@host)
{% if flag?(:without_openssl) %}
if tls
raise "HTTP::Client TLS is disabled because `-D without_openssl` was passed at compile time"
end
@tls = nil
{% else %}
@tls = case tls
when true
OpenSSL::SSL::Context::Client.new
when OpenSSL::SSL::Context::Client
tls
when false, nil
nil
end
{% end %}
@port = (port || (@tls ? 443 : 80)).to_i
self.proxy = make_configured_http_proxy_client() if CONFIG.http_proxy
end
def initialize(@io : IO, @host = "", @port = 80)
@reconnect = false
self.proxy = make_configured_http_proxy_client() if CONFIG.http_proxy
end
private def io
io = @io
return io if io

View file

@ -6,14 +6,9 @@ module Tokens
def refresh_tokens
@@po_token = REDIS_DB.get("invidious:po_token")
@@visitor_data = REDIS_DB.get("invidious:visitor_data")
if !@@po_token.nil? && !@@visitor_data.nil?
LOGGER.debug("RefreshTokens: Successfully updated tokens")
else
LOGGER.warn("RefreshTokens: Tokens are empty!")
end
LOGGER.trace("RefreshTokens: Tokens are:")
LOGGER.trace("RefreshTokens: po_token: #{@@po_token}")
LOGGER.trace("RefreshTokens: visitor_data: #{@@visitor_data}")
LOGGER.debug("RefreshTokens: Tokens are:")
LOGGER.debug("RefreshTokens: po_token: #{@@po_token}")
LOGGER.debug("RefreshTokens: visitor_data: #{@@visitor_data}")
end
def get_tokens

View file

@ -175,6 +175,7 @@ module Invidious::SigHelper
@queue = {} of TransactionID => Transaction
@conn : Connection
@uri_or_path : String
def initialize(@uri_or_path)
@ -200,7 +201,7 @@ module Invidious::SigHelper
@conn = Connection.new(@uri_or_path)
LOGGER.info("SigHelper: Reconnected to SigHelper!")
rescue ex
LOGGER.debug("SigHelper: Reconnection to helper unsuccessful with error '#{ex.message}'. Retrying")
LOGGER.debug("SigHelper: Reconnection to helper unsuccessful with error '#{ex.message}' retrying")
sleep 500.milliseconds
next
end

View file

@ -4,28 +4,51 @@ module Invidious::HttpServer
module Utils
extend self
@@proxy_alive : String = ""
@@proxy_list : Array(String) = [] of String
@@current_proxy : String = ""
@@count : Int64 = Time.utc.to_unix
def check_external_proxy
CONFIG.external_videoplayback_proxy.each do |proxy|
begin
response = HTTP::Client.get("#{proxy[:url]}/health")
if response.status_code == 200
@@proxy_alive = proxy[:url]
LOGGER.debug("CheckExternalProxy: Proxy set to: '#{proxy[:url]}'")
break
if @@proxy_list.includes?(proxy[:url])
next
end
if proxy[:balance]
@@proxy_list << proxy[:url]
LOGGER.debug("CheckExternalProxy: Adding proxy '#{proxy[:url]}' to the list of proxies")
end
break if proxy[:balance] == false && !@@proxy_list.empty?
@@proxy_list << proxy[:url]
end
rescue
if @@proxy_list.includes?(proxy[:url])
LOGGER.debug("CheckExternalProxy: Proxy '#{proxy[:url]}' is not available, removing it from the list of proxies")
@@proxy_list.delete(proxy[:url])
end
LOGGER.debug("CheckExternalProxy: Proxy '#{proxy[:url]}' is not available")
end
end
if @@proxy_alive.empty?
LOGGER.warn("CheckExternalProxy: No proxies alive! Using own server proxy")
end
LOGGER.trace("CheckExternalProxy: List of proxies:")
LOGGER.trace("#{@@proxy_list.inspect}")
end
# TODO: If the function is called many times, it will return a random
# proxy from the list. That is not how it should be.
# It should return the same proxy, in multiple function calls
def select_proxy
if (@@count - (Time.utc.to_unix - 30)) <= 0
return if @@proxy_list.size <= 0
@@current_proxy = @@proxy_list[Random.rand(@@proxy_list.size)]
LOGGER.debug("Current proxy is: '#{@@current_proxy}'")
@@count = Time.utc.to_unix
end
end
def get_external_proxy
return @@proxy_alive
return @@current_proxy
end
def proxy_video_url(raw_url : String, *, region : String? = nil, absolute : Bool = false)
@ -38,8 +61,8 @@ module Invidious::HttpServer
url.query_params = params
if absolute
if !@@proxy_alive.empty?
return "#{@@proxy_alive}#{url.request_target}"
if !(proxy = get_external_proxy()).empty?
return "#{proxy}#{url.request_target}"
else
return "#{HOST_URL}#{url.request_target}"
end

View file

@ -5,8 +5,9 @@ class Invidious::Jobs::CheckExternalProxy < Invidious::Jobs::BaseJob
def begin
loop do
HttpServer::Utils.check_external_proxy
LOGGER.info("CheckExternalProxy: Done, sleeping for 10 seconds")
sleep 10.seconds
HttpServer::Utils.select_proxy
LOGGER.info("CheckExternalProxy: Done, sleeping for 15 seconds")
sleep 15.seconds
Fiber.yield
end
end

View file

@ -11,9 +11,29 @@ module Invidious::Routes::Images
end
end
# We're encapsulating this into a proc in order to easily reuse this
# portion of the code for each request block below.
request_proc = ->(response : HTTP::Client::Response) {
env.response.status_code = response.status_code
response.headers.each do |key, value|
if !RESPONSE_HEADERS_BLACKLIST.includes?(key.downcase)
env.response.headers[key] = value
end
end
env.response.headers["Access-Control-Allow-Origin"] = "*"
if response.status_code >= 300
env.response.headers.delete("Transfer-Encoding")
return
end
proxy_file(response, env)
}
begin
GGPHT_POOL.client &.get(url, headers) do |resp|
return self.proxy_image(env, resp)
HTTP::Client.get("https://yt3.ggpht.com#{url}") do |resp|
return request_proc.call(resp)
end
rescue ex
end
@ -41,10 +61,27 @@ module Invidious::Routes::Images
end
end
request_proc = ->(response : HTTP::Client::Response) {
env.response.status_code = response.status_code
response.headers.each do |key, value|
if !RESPONSE_HEADERS_BLACKLIST.includes?(key.downcase)
env.response.headers[key] = value
end
end
env.response.headers["Connection"] = "close"
env.response.headers["Access-Control-Allow-Origin"] = "*"
if response.status_code >= 300
return env.response.headers.delete("Transfer-Encoding")
end
proxy_file(response, env)
}
begin
get_ytimg_pool(authority).client &.get(url, headers) do |resp|
env.response.headers["Connection"] = "close"
return self.proxy_image(env, resp)
HTTP::Client.get("https://#{authority}.ytimg.com#{url}") do |resp|
return request_proc.call(resp)
end
rescue ex
end
@ -64,9 +101,26 @@ module Invidious::Routes::Images
end
end
request_proc = ->(response : HTTP::Client::Response) {
env.response.status_code = response.status_code
response.headers.each do |key, value|
if !RESPONSE_HEADERS_BLACKLIST.includes?(key.downcase)
env.response.headers[key] = value
end
end
env.response.headers["Access-Control-Allow-Origin"] = "*"
if response.status_code >= 300 && response.status_code != 404
return env.response.headers.delete("Transfer-Encoding")
end
proxy_file(response, env)
}
begin
get_ytimg_pool("i9").client &.get(url, headers) do |resp|
return self.proxy_image(env, resp)
HTTP::Client.get("https://i9.ytimg.com#{url}") do |resp|
return request_proc.call(resp)
end
rescue ex
end
@ -111,7 +165,8 @@ module Invidious::Routes::Images
if name == "maxres.jpg"
build_thumbnails(id).each do |thumb|
thumbnail_resource_path = "/vi/#{id}/#{thumb[:url]}.jpg"
if get_ytimg_pool("i9").client &.head(thumbnail_resource_path, headers).status_code == 200
# This can likely be optimized into a (small) pool sometime in the future.
if HTTP::Client.head("https://i.ytimg.com#{thumbnail_resource_path}").status_code == 200
name = thumb[:url] + ".jpg"
break
end
@ -126,28 +181,29 @@ module Invidious::Routes::Images
end
end
request_proc = ->(response : HTTP::Client::Response) {
env.response.status_code = response.status_code
response.headers.each do |key, value|
if !RESPONSE_HEADERS_BLACKLIST.includes?(key.downcase)
env.response.headers[key] = value
end
end
env.response.headers["Access-Control-Allow-Origin"] = "*"
if response.status_code >= 300 && response.status_code != 404
return env.response.headers.delete("Transfer-Encoding")
end
proxy_file(response, env)
}
begin
get_ytimg_pool("i").client &.get(url, headers) do |resp|
return self.proxy_image(env, resp)
# This can likely be optimized into a (small) pool sometime in the future.
HTTP::Client.get("https://i.ytimg.com#{url}") do |resp|
return request_proc.call(resp)
end
rescue ex
end
end
private def self.proxy_image(env, response)
env.response.status_code = response.status_code
response.headers.each do |key, value|
if !RESPONSE_HEADERS_BLACKLIST.includes?(key.downcase)
env.response.headers[key] = value
end
end
env.response.headers["Access-Control-Allow-Origin"] = "*"
if response.status_code >= 300
return env.response.headers.delete("Transfer-Encoding")
end
return proxy_file(response, env)
end
end

View file

@ -152,7 +152,6 @@ module Invidious::Routes::Watch
end
end
# Removes non default audio tracks
audio_streams.reject! do |z|
z if z.dig?("audioTrack", "audioIsDefault") == false
end
@ -219,12 +218,6 @@ module Invidious::Routes::Watch
captions: video.captions
)
begin
video_url = fmt_stream[0]["url"].to_s
rescue
video_url = nil
end
templated "watch"
end

View file

@ -123,7 +123,6 @@ module Invidious::Videos
"Esperanto",
"Estonian",
"Filipino",
"Filipino (auto-generated)",
"Finnish",
"French",
"French (auto-generated)",

View file

@ -13,13 +13,11 @@
<meta property="og:image" content="<%= HOST_URL %>/vi/<%= video.id %>/maxres.jpg">
<meta property="og:description" content="<%= HTML.escape(video.short_description) %>">
<meta property="og:type" content="video.other">
<!-- This shouldn't be empty, ever. -->
<meta property="og:video" content="<%= video_url %>">
<meta property="og:video:url" content="<%= video_url %>">
<meta property="og:video:secure_url" content="<%= video_url %>">
<meta property="og:video:type" content="video/mp4">
<meta property="og:video:width" content="640">
<meta property="og:video:height" content="360">
<meta property="og:video:url" content="<%= HOST_URL %>/embed/<%= video.id %>">
<meta property="og:video:secure_url" content="<%= HOST_URL %>/embed/<%= video.id %>">
<meta property="og:video:type" content="text/html">
<meta property="og:video:width" content="1280">
<meta property="og:video:height" content="720">
<meta name="twitter:card" content="player">
<meta name="twitter:url" content="<%= HOST_URL %>/watch?v=<%= video.id %>">
<meta name="twitter:title" content="<%= title %>">

View file

@ -1,6 +1,17 @@
# Mapping of subdomain => YoutubeConnectionPool
# This is needed as we may need to access arbitrary subdomains of ytimg
private YTIMG_POOLS = {} of String => YoutubeConnectionPool
def add_yt_headers(request)
request.headers.delete("User-Agent") if request.headers["User-Agent"] == "Crystal"
request.headers["User-Agent"] ||= "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
request.headers["Accept-Charset"] ||= "ISO-8859-1,utf-8;q=0.7,*;q=0.7"
request.headers["Accept"] ||= "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
request.headers["Accept-Language"] ||= "en-US,en;q=0.9"
# Preserve original cookies and add new YT consent cookie for EU servers
request.headers["Cookie"] = "#{request.headers["cookie"]?}; CONSENT=PENDING+#{Random.rand(100..999)}"
if !CONFIG.cookies.empty?
request.headers["Cookie"] = "#{(CONFIG.cookies.map { |c| "#{c.name}=#{c.value}" }).join("; ")}; #{request.headers["cookie"]?}"
end
end
struct YoutubeConnectionPool
property! url : URI
@ -15,16 +26,12 @@ struct YoutubeConnectionPool
def client(&)
conn = pool.checkout
# Proxy needs to be reinstated every time we get a client from the pool
conn.proxy = make_configured_http_proxy_client() if CONFIG.http_proxy
begin
response = yield conn
rescue ex
conn.close
conn = HTTP::Client.new(url)
conn.proxy = make_configured_http_proxy_client() if CONFIG.http_proxy
conn.family = CONFIG.force_resolve
conn.family = Socket::Family::INET if conn.family == Socket::Family::UNSPEC
conn.before_request { |r| add_yt_headers(r) } if url.host == "www.youtube.com"
@ -47,21 +54,6 @@ struct YoutubeConnectionPool
end
end
def add_yt_headers(request)
request.headers.delete("User-Agent") if request.headers["User-Agent"] == "Crystal"
request.headers["User-Agent"] ||= "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
request.headers["Accept-Charset"] ||= "ISO-8859-1,utf-8;q=0.7,*;q=0.7"
request.headers["Accept"] ||= "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
request.headers["Accept-Language"] ||= "en-us,en;q=0.5"
# Preserve original cookies and add new YT consent cookie for EU servers
request.headers["Cookie"] = "#{request.headers["cookie"]?}; CONSENT=PENDING+#{Random.rand(100..999)}"
if !CONFIG.cookies.empty?
request.headers["Cookie"] = "#{(CONFIG.cookies.map { |c| "#{c.name}=#{c.value}" }).join("; ")}; #{request.headers["cookie"]?}"
end
end
def make_client(url : URI, region = nil, force_resolve : Bool = false)
client = HTTP::Client.new(url)
@ -85,31 +77,3 @@ def make_client(url : URI, region = nil, force_resolve : Bool = false, &)
client.close
end
end
def make_configured_http_proxy_client
# This method is only called when configuration for an HTTP proxy are set
config_proxy = CONFIG.http_proxy.not_nil!
return HTTP::Proxy::Client.new(
config_proxy.host,
config_proxy.port,
username: config_proxy.user,
password: config_proxy.password,
)
end
# Fetches a HTTP pool for the specified subdomain of ytimg.com
#
# Creates a new one when the specified pool for the subdomain does not exist
def get_ytimg_pool(subdomain)
if pool = YTIMG_POOLS[subdomain]?
return pool
else
LOGGER.info("ytimg_pool: Creating a new HTTP pool for \"https://#{subdomain}.ytimg.com\"")
pool = YoutubeConnectionPool.new(URI.parse("https://#{subdomain}.ytimg.com"), capacity: CONFIG.pool_size)
YTIMG_POOLS[subdomain] = pool
return pool
end
end