Compare commits

..

20 commits

Author SHA1 Message Date
b4ddbf62d0
Videos: Fix audio tracks language.
Video will only return the default language. The rest of the audio
tracks are deleted since they will not be used.
2024-10-31 20:11:49 -03:00
5bac8499ac
External Proxies: Rotate between proxies with balance enabled
Closes #17
2024-10-31 20:11:49 -03:00
e8b37e1760
Config: Also reload env variables 2024-10-31 20:11:49 -03:00
174d468c9c
Use POST requests for /videoplayback requests 2024-10-31 20:11:49 -03:00
8b5edb7c9f
Config: Reload configuration on modification
It detects changes on the config.yml automtically if invidious is
running on linux. If not, the configuration can be reloaded using
`kill -s HUP $(pidof invidious)` or any other tool that sends a SIGHUP
signal to the invidious process.

Closes #16
2024-10-31 20:11:49 -03:00
6e0ab02cd6
Tokens: Option to disable user tokens. 2024-10-31 20:11:49 -03:00
15ced487e8
Tokens: Server side generated tokens.
#18
2024-10-31 20:11:49 -03:00
c04d0e26ae
PubSub: Use external domain for pubsub feeds 2024-10-31 20:11:48 -03:00
0883d9c8fe
External Proxies: Proxyfi HLS Playlists 2024-10-31 20:11:48 -03:00
acc22f1741
Videos: Completly disable annotations due to archive.org being down
Closes #15
2024-10-31 20:11:48 -03:00
d9a00ca397
Tokens: Refresh po_token and visitor_data every 5 seconds
Closes #11
2024-10-31 20:11:48 -03:00
99efc4c10f
External Proxies: Proxyfi HD720 2024-10-31 20:11:48 -03:00
f4d76bcad0
Videos: Increase video cache to 4 hours 2024-10-31 20:11:48 -03:00
c6d96c7276
Feat: Experimental support for potoken inside redis
Using https://git.nadeko.net/Fijxu/youtube-po-token-generator
2024-10-31 20:11:48 -03:00
d189e2ee6e
External Proxies: Use list of external videoplayback proxies 2024-10-31 20:11:48 -03:00
Samantaz Fox
08d2471ebd
Videos: Fix missing host parameter on playback URLs when local=true 2024-10-31 20:11:47 -03:00
c8c20e4592
CI: Experimental branches for testing builds 2024-10-31 20:11:47 -03:00
832fa5e601
Feat: User supplied po_token and visitor_data 2024-10-31 20:11:47 -03:00
af9e8665c8
Small try. 2024-10-31 20:11:47 -03:00
857afcf239
Feeds: Get rid of feed_needs_update() since it appears to be unused 2024-10-31 20:11:47 -03:00
25 changed files with 177 additions and 264 deletions

View file

@ -37,15 +37,13 @@ jobs:
type=sha,format=short,prefix={{date 'YYYY.MM.DD'}}-,enable=${{ github.ref == format('refs/heads/{0}', 'master') }} type=sha,format=short,prefix={{date 'YYYY.MM.DD'}}-,enable=${{ github.ref == format('refs/heads/{0}', 'master') }}
type=raw,value=latest,enable=${{ github.ref == format('refs/heads/{0}', 'master') }} type=raw,value=latest,enable=${{ github.ref == format('refs/heads/{0}', 'master') }}
- uses: https://code.forgejo.org/docker/build-push-action@v6 - uses: https://code.forgejo.org/docker/build-push-action@v5
name: Build images name: Build images
with: with:
context: . context: .
file: docker/Dockerfile file: docker/Dockerfile
tags: ${{ steps.meta.outputs.tags }} tags: ${{ steps.meta.outputs.tags }}
platforms: linux/amd64 platforms: linux/amd64
# cache-from: type=gha
# cache-to: type=gha,mode=max
push: true push: true
build-args: | build-args: |
"release=1" "release=1"

View file

@ -38,11 +38,10 @@ jobs:
matrix: matrix:
stable: [true] stable: [true]
crystal: crystal:
- 1.9.2
- 1.10.1 - 1.10.1
- 1.11.2 - 1.11.2
- 1.12.1 - 1.12.1
- 1.13.2
- 1.14.0
include: include:
- crystal: nightly - crystal: nightly
stable: false stable: false
@ -52,11 +51,6 @@ jobs:
with: with:
submodules: true submodules: true
- name: Install required APT packages
run: |
sudo apt install -y libsqlite3-dev
shell: bash
- name: Install Crystal - name: Install Crystal
uses: crystal-lang/install-crystal@v1.8.0 uses: crystal-lang/install-crystal@v1.8.0
with: with:

View file

@ -5,12 +5,6 @@
### Full list of pull requests merged since the last release (newest first) ### Full list of pull requests merged since the last release (newest first)
* Add "Filipino (auto-generated)" to the list of caption languages ([#4995], by @SamantazFox)
* Makefile: Add MT option to enable the 'preview_mt' flag ([#4993], by @SamantazFox)
* SigHelper: Reconnect to signature helper ([#4991], thanks @Fijxu)
* Fix player menus hiding onHover ready ([#4750], thanks @giacomocerquone)
* Use connection pools when requesting images from YouTube ([#4326], thanks @syeopite)
* Add support for using Invidious through a HTTP Proxy ([#4270], thanks @syeopite)
* Search: Fix 'youtu.be' URLs in sanitizer ([#4894], by @SamantazFox) * Search: Fix 'youtu.be' URLs in sanitizer ([#4894], by @SamantazFox)
* Ameba: Disable Style/RedundantNext rule ([#4888], thanks @syeopite) * Ameba: Disable Style/RedundantNext rule ([#4888], thanks @syeopite)
* Playlists: Fix 'invalid byte sequence' error when subscribing ([#4887], thanks @DmitrySandalov) * Playlists: Fix 'invalid byte sequence' error when subscribing ([#4887], thanks @DmitrySandalov)
@ -28,10 +22,7 @@
[#4122]: https://github.com/iv-org/invidious/pull/4122 [#4122]: https://github.com/iv-org/invidious/pull/4122
[#4193]: https://github.com/iv-org/invidious/pull/4193 [#4193]: https://github.com/iv-org/invidious/pull/4193
[#4270]: https://github.com/iv-org/invidious/pull/4270
[#4326]: https://github.com/iv-org/invidious/pull/4326
[#4652]: https://github.com/iv-org/invidious/pull/4652 [#4652]: https://github.com/iv-org/invidious/pull/4652
[#4750]: https://github.com/iv-org/invidious/pull/4750
[#4850]: https://github.com/iv-org/invidious/pull/4850 [#4850]: https://github.com/iv-org/invidious/pull/4850
[#4862]: https://github.com/iv-org/invidious/pull/4862 [#4862]: https://github.com/iv-org/invidious/pull/4862
[#4863]: https://github.com/iv-org/invidious/pull/4863 [#4863]: https://github.com/iv-org/invidious/pull/4863
@ -42,9 +33,6 @@
[#4928]: https://github.com/iv-org/invidious/pull/4928 [#4928]: https://github.com/iv-org/invidious/pull/4928
[#4930]: https://github.com/iv-org/invidious/pull/4930 [#4930]: https://github.com/iv-org/invidious/pull/4930
[#4942]: https://github.com/iv-org/invidious/pull/4942 [#4942]: https://github.com/iv-org/invidious/pull/4942
[#4991]: https://github.com/iv-org/invidious/pull/4991
[#4993]: https://github.com/iv-org/invidious/pull/4993
[#4995]: https://github.com/iv-org/invidious/pull/4995
## v2.20240825.2 (2024-08-26) ## v2.20240825.2 (2024-08-26)

View file

@ -7,11 +7,6 @@ STATIC := 0
NO_DBG_SYMBOLS := 0 NO_DBG_SYMBOLS := 0
# Enable multi-threading.
# Warning: Experimental feature!!
# invidious is not stable when MT is enabled.
MT := 0
FLAGS ?= FLAGS ?=
@ -24,10 +19,6 @@ ifeq ($(STATIC), 1)
FLAGS += --static FLAGS += --static
endif endif
ifeq ($(MT), 1)
FLAGS += -Dpreview_mt
endif
ifeq ($(NO_DBG_SYMBOLS), 1) ifeq ($(NO_DBG_SYMBOLS), 1)
FLAGS += --no-debug FLAGS += --no-debug

View file

@ -68,7 +68,6 @@
.video-js.player-style-youtube .vjs-menu-button-popup .vjs-menu { .video-js.player-style-youtube .vjs-menu-button-popup .vjs-menu {
margin-bottom: 2em; margin-bottom: 2em;
padding-top: 2em
} }
.video-js.player-style-youtube .vjs-progress-control .vjs-progress-holder, .video-js.player-style-youtube .vjs-progress-control {height: 5px; .video-js.player-style-youtube .vjs-progress-control .vjs-progress-holder, .video-js.player-style-youtube .vjs-progress-control {height: 5px;

View file

@ -173,17 +173,6 @@ https_only: false
## ##
#force_resolve: #force_resolve:
##
## Configuration for using a HTTP proxy
##
## If unset, then no HTTP proxy will be used.
##
http_proxy:
user:
password:
host:
port:
## ##
## Use Innertube's transcripts API instead of timedtext for closed captions ## Use Innertube's transcripts API instead of timedtext for closed captions

View file

@ -1,4 +1,4 @@
FROM crystallang/crystal:1.14.0-alpine AS builder FROM crystallang/crystal:1.12.1-alpine AS builder
RUN apk add --no-cache sqlite-static yaml-static RUN apk add --no-cache sqlite-static yaml-static

View file

@ -287,7 +287,6 @@
"Esperanto": "Esperanto", "Esperanto": "Esperanto",
"Estonian": "Estonian", "Estonian": "Estonian",
"Filipino": "Filipino", "Filipino": "Filipino",
"Filipino (auto-generated)": "Filipino (auto-generated)",
"Finnish": "Finnish", "Finnish": "Finnish",
"French": "French", "French": "French",
"French (auto-generated)": "French (auto-generated)", "French (auto-generated)": "French (auto-generated)",

2
mocks

@ -1 +1 @@
Subproject commit b55d58dea94f7144ff0205857dfa70ec14eaa872 Subproject commit 11ec372f72747c09d48ffef04843f72be67d5b54

View file

@ -10,7 +10,7 @@ shards:
backtracer: backtracer:
git: https://github.com/sija/backtracer.cr.git git: https://github.com/sija/backtracer.cr.git
version: 1.2.2 version: 1.2.1
db: db:
git: https://github.com/crystal-lang/crystal-db.git git: https://github.com/crystal-lang/crystal-db.git
@ -23,9 +23,6 @@ shards:
inotify: inotify:
git: https://github.com/petoem/inotify.cr.git git: https://github.com/petoem/inotify.cr.git
version: 1.0.3 version: 1.0.3
http_proxy:
git: https://github.com/mamantoha/http_proxy.git
version: 0.10.3
kemal: kemal:
git: https://github.com/kemalcr/kemal.git git: https://github.com/kemalcr/kemal.git
@ -57,7 +54,7 @@ shards:
spectator: spectator:
git: https://github.com/icy-arctic-fox/spectator.git git: https://github.com/icy-arctic-fox/spectator.git
version: 0.10.6 version: 0.10.4
sqlite3: sqlite3:
git: https://github.com/crystal-lang/crystal-sqlite3.git git: https://github.com/crystal-lang/crystal-sqlite3.git

View file

@ -33,9 +33,6 @@ dependencies:
inotify: inotify:
github: petoem/inotify.cr github: petoem/inotify.cr
version: 1.0.3 version: 1.0.3
http_proxy:
github: mamantoha/http_proxy
version: ~> 0.10.3
development_dependencies: development_dependencies:
spectator: spectator:

View file

@ -17,8 +17,8 @@ Spectator.describe "parse_video_info" do
# Basic video infos # Basic video infos
expect(info["title"].as_s).to eq("I Gave My 100,000,000th Subscriber An Island") expect(info["title"].as_s).to eq("I Gave My 100,000,000th Subscriber An Island")
expect(info["views"].as_i).to eq(220_226_287) expect(info["views"].as_i).to eq(126_573_823)
expect(info["likes"].as_i).to eq(6_870_691) expect(info["likes"].as_i).to eq(5_157_654)
# For some reason the video length from VideoDetails and the # For some reason the video length from VideoDetails and the
# one from microformat differs by 1s... # one from microformat differs by 1s...
@ -48,12 +48,12 @@ Spectator.describe "parse_video_info" do
expect(info["relatedVideos"].as_a.size).to eq(20) expect(info["relatedVideos"].as_a.size).to eq(20)
expect(info["relatedVideos"][0]["id"]).to eq("krsBRQbOPQ4") expect(info["relatedVideos"][0]["id"]).to eq("Hwybp38GnZw")
expect(info["relatedVideos"][0]["title"]).to eq("$1 vs $250,000,000 Private Island!") expect(info["relatedVideos"][0]["title"]).to eq("I Built Willy Wonka's Chocolate Factory!")
expect(info["relatedVideos"][0]["author"]).to eq("MrBeast") expect(info["relatedVideos"][0]["author"]).to eq("MrBeast")
expect(info["relatedVideos"][0]["ucid"]).to eq("UCX6OQ3DkcsbYNE6H8uQQuVA") expect(info["relatedVideos"][0]["ucid"]).to eq("UCX6OQ3DkcsbYNE6H8uQQuVA")
expect(info["relatedVideos"][0]["view_count"]).to eq("230617484") expect(info["relatedVideos"][0]["view_count"]).to eq("179877630")
expect(info["relatedVideos"][0]["short_view_count"]).to eq("230M") expect(info["relatedVideos"][0]["short_view_count"]).to eq("179M")
expect(info["relatedVideos"][0]["author_verified"]).to eq("true") expect(info["relatedVideos"][0]["author_verified"]).to eq("true")
# Description # Description
@ -76,11 +76,11 @@ Spectator.describe "parse_video_info" do
expect(info["ucid"].as_s).to eq("UCX6OQ3DkcsbYNE6H8uQQuVA") expect(info["ucid"].as_s).to eq("UCX6OQ3DkcsbYNE6H8uQQuVA")
expect(info["authorThumbnail"].as_s).to eq( expect(info["authorThumbnail"].as_s).to eq(
"https://yt3.ggpht.com/fxGKYucJAVme-Yz4fsdCroCFCrANWqw0ql4GYuvx8Uq4l_euNJHgE-w9MTkLQA805vWCi-kE0g=s48-c-k-c0x00ffffff-no-rj" "https://yt3.ggpht.com/ytc/AL5GRJVuqw82ERvHzsmBxL7avr1dpBtsVIXcEzBPZaloFg=s48-c-k-c0x00ffffff-no-rj"
) )
expect(info["authorVerified"].as_bool).to be_true expect(info["authorVerified"].as_bool).to be_true
expect(info["subCountText"].as_s).to eq("320M") expect(info["subCountText"].as_s).to eq("143M")
end end
it "parses a regular video with no descrition/comments" do it "parses a regular video with no descrition/comments" do
@ -99,8 +99,8 @@ Spectator.describe "parse_video_info" do
# Basic video infos # Basic video infos
expect(info["title"].as_s).to eq("Chris Rea - Auberge") expect(info["title"].as_s).to eq("Chris Rea - Auberge")
expect(info["views"].as_i).to eq(14_324_584) expect(info["views"].as_i).to eq(10_943_126)
expect(info["likes"].as_i).to eq(35_870) expect(info["likes"].as_i).to eq(0)
expect(info["lengthSeconds"].as_i).to eq(283_i64) expect(info["lengthSeconds"].as_i).to eq(283_i64)
expect(info["published"].as_s).to eq("2012-05-21T00:00:00Z") expect(info["published"].as_s).to eq("2012-05-21T00:00:00Z")
@ -132,14 +132,14 @@ Spectator.describe "parse_video_info" do
# Related videos # Related videos
expect(info["relatedVideos"].as_a.size).to eq(20) expect(info["relatedVideos"].as_a.size).to eq(19)
expect(info["relatedVideos"][0]["id"]).to eq("gUUdQfnshJ4") expect(info["relatedVideos"][0]["id"]).to eq("Ww3KeZ2_Yv4")
expect(info["relatedVideos"][0]["title"]).to eq("Chris Rea - The Road To Hell 1989 Full Version") expect(info["relatedVideos"][0]["title"]).to eq("Chris Rea")
expect(info["relatedVideos"][0]["author"]).to eq("NEA ZIXNH") expect(info["relatedVideos"][0]["author"]).to eq("PanMusic")
expect(info["relatedVideos"][0]["ucid"]).to eq("UCYMEOGcvav3gCgImK2J07CQ") expect(info["relatedVideos"][0]["ucid"]).to eq("UCsKAPSuh1iNbLWUga_igPyA")
expect(info["relatedVideos"][0]["view_count"]).to eq("53298661") expect(info["relatedVideos"][0]["view_count"]).to eq("31581")
expect(info["relatedVideos"][0]["short_view_count"]).to eq("53M") expect(info["relatedVideos"][0]["short_view_count"]).to eq("31K")
expect(info["relatedVideos"][0]["author_verified"]).to eq("false") expect(info["relatedVideos"][0]["author_verified"]).to eq("false")
# Description # Description
@ -156,13 +156,11 @@ Spectator.describe "parse_video_info" do
# Author infos # Author infos
expect(info["author"].as_s).to eq("ChrisReaVideos") expect(info["author"].as_s).to eq("ChrisReaOfficial")
expect(info["ucid"].as_s).to eq("UC_5q6nWPbD30-y6oiWF_oNA") expect(info["ucid"].as_s).to eq("UC_5q6nWPbD30-y6oiWF_oNA")
expect(info["authorThumbnail"].as_s).to eq( expect(info["authorThumbnail"].as_s).to be_empty
"https://yt3.ggpht.com/ytc/AIdro_n71nsegpKfjeRKwn1JJmK5IVMh_7j5m_h3_1KnUUg=s48-c-k-c0x00ffffff-no-rj"
)
expect(info["authorVerified"].as_bool).to be_false expect(info["authorVerified"].as_bool).to be_false
expect(info["subCountText"].as_s).to eq("3.11K") expect(info["subCountText"].as_s).to eq("-")
end end
end end

View file

@ -23,7 +23,6 @@ require "kilt"
require "./ext/kemal_content_for.cr" require "./ext/kemal_content_for.cr"
require "./ext/kemal_static_file_handler.cr" require "./ext/kemal_static_file_handler.cr"
require "http_proxy"
require "athena-negotiation" require "athena-negotiation"
require "openssl/hmac" require "openssl/hmac"
require "option_parser" require "option_parser"
@ -115,10 +114,6 @@ SOFTWARE = {
YT_POOL = YoutubeConnectionPool.new(YT_URL, capacity: CONFIG.pool_size) YT_POOL = YoutubeConnectionPool.new(YT_URL, capacity: CONFIG.pool_size)
# Image request pool
GGPHT_POOL = YoutubeConnectionPool.new(URI.parse("https://yt3.ggpht.com"), capacity: CONFIG.pool_size)
# CLI # CLI
Kemal.config.extra_options do |parser| Kemal.config.extra_options do |parser|
parser.banner = "Usage: invidious [arguments]" parser.banner = "Usage: invidious [arguments]"

View file

@ -23,22 +23,6 @@ def produce_channel_content_continuation(ucid, content_type, page = 1, auto_gene
else 15 # Fallback to "videos" else 15 # Fallback to "videos"
end end
sort_type_numerical =
case content_type
when "videos" then 3
when "livestreams" then 5
else 3 # Fallback to "videos"
end
if content_type == "livestreams"
sort_by_numerical =
case sort_by
when "newest" then 12_i64
when "popular" then 14_i64
when "oldest" then 13_i64
else 12_i64 # Fallback to "newest"
end
else
sort_by_numerical = sort_by_numerical =
case sort_by case sort_by
when "newest" then 1_i64 when "newest" then 1_i64
@ -46,7 +30,6 @@ def produce_channel_content_continuation(ucid, content_type, page = 1, auto_gene
when "oldest" then 4_i64 when "oldest" then 4_i64
else 1_i64 # Fallback to "newest" else 1_i64 # Fallback to "newest"
end end
end
object_inner_1 = { object_inner_1 = {
"110:embedded" => { "110:embedded" => {
@ -58,7 +41,7 @@ def produce_channel_content_continuation(ucid, content_type, page = 1, auto_gene
"2:embedded" => { "2:embedded" => {
"1:string" => "00000000-0000-0000-0000-000000000000", "1:string" => "00000000-0000-0000-0000-000000000000",
}, },
"#{sort_type_numerical}:varint" => sort_by_numerical, "3:varint" => sort_by_numerical,
}, },
}, },
}, },

View file

@ -57,15 +57,6 @@ struct ConfigPreferences
end end
end end
struct HTTPProxyConfig
include YAML::Serializable
property user : String
property password : String
property host : String
property port : Int32
end
class Config class Config
include YAML::Serializable include YAML::Serializable
@ -166,8 +157,6 @@ class Config
property host_binding : String = "0.0.0.0" property host_binding : String = "0.0.0.0"
# Pool size for HTTP requests to youtube.com and ytimg.com (each domain has a separate pool of `pool_size`) # Pool size for HTTP requests to youtube.com and ytimg.com (each domain has a separate pool of `pool_size`)
property pool_size : Int32 = 100 property pool_size : Int32 = 100
# HTTP Proxy configuration
property http_proxy : HTTPProxyConfig? = nil
# Use Innertube's transcripts API instead of timedtext for closed captions # Use Innertube's transcripts API instead of timedtext for closed captions
property use_innertube_for_captions : Bool = false property use_innertube_for_captions : Bool = false

View file

@ -18,40 +18,6 @@ end
class HTTP::Client class HTTP::Client
property family : Socket::Family = Socket::Family::UNSPEC property family : Socket::Family = Socket::Family::UNSPEC
# Override stdlib to automatically initialize proxy if configured
#
# Accurate as of crystal 1.12.1
def initialize(@host : String, port = nil, tls : TLSContext = nil)
check_host_only(@host)
{% if flag?(:without_openssl) %}
if tls
raise "HTTP::Client TLS is disabled because `-D without_openssl` was passed at compile time"
end
@tls = nil
{% else %}
@tls = case tls
when true
OpenSSL::SSL::Context::Client.new
when OpenSSL::SSL::Context::Client
tls
when false, nil
nil
end
{% end %}
@port = (port || (@tls ? 443 : 80)).to_i
self.proxy = make_configured_http_proxy_client() if CONFIG.http_proxy
end
def initialize(@io : IO, @host = "", @port = 80)
@reconnect = false
self.proxy = make_configured_http_proxy_client() if CONFIG.http_proxy
end
private def io private def io
io = @io io = @io
return io if io return io if io

View file

@ -6,14 +6,9 @@ module Tokens
def refresh_tokens def refresh_tokens
@@po_token = REDIS_DB.get("invidious:po_token") @@po_token = REDIS_DB.get("invidious:po_token")
@@visitor_data = REDIS_DB.get("invidious:visitor_data") @@visitor_data = REDIS_DB.get("invidious:visitor_data")
if !@@po_token.nil? && !@@visitor_data.nil? LOGGER.debug("RefreshTokens: Tokens are:")
LOGGER.debug("RefreshTokens: Successfully updated tokens") LOGGER.debug("RefreshTokens: po_token: #{@@po_token}")
else LOGGER.debug("RefreshTokens: visitor_data: #{@@visitor_data}")
LOGGER.warn("RefreshTokens: Tokens are empty!")
end
LOGGER.trace("RefreshTokens: Tokens are:")
LOGGER.trace("RefreshTokens: po_token: #{@@po_token}")
LOGGER.trace("RefreshTokens: visitor_data: #{@@visitor_data}")
end end
def get_tokens def get_tokens

View file

@ -175,6 +175,7 @@ module Invidious::SigHelper
@queue = {} of TransactionID => Transaction @queue = {} of TransactionID => Transaction
@conn : Connection @conn : Connection
@uri_or_path : String @uri_or_path : String
def initialize(@uri_or_path) def initialize(@uri_or_path)
@ -200,7 +201,7 @@ module Invidious::SigHelper
@conn = Connection.new(@uri_or_path) @conn = Connection.new(@uri_or_path)
LOGGER.info("SigHelper: Reconnected to SigHelper!") LOGGER.info("SigHelper: Reconnected to SigHelper!")
rescue ex rescue ex
LOGGER.debug("SigHelper: Reconnection to helper unsuccessful with error '#{ex.message}'. Retrying") LOGGER.debug("SigHelper: Reconnection to helper unsuccessful with error '#{ex.message}' retrying")
sleep 500.milliseconds sleep 500.milliseconds
next next
end end

View file

@ -4,28 +4,51 @@ module Invidious::HttpServer
module Utils module Utils
extend self extend self
@@proxy_alive : String = "" @@proxy_list : Array(String) = [] of String
@@current_proxy : String = ""
@@count : Int64 = Time.utc.to_unix
def check_external_proxy def check_external_proxy
CONFIG.external_videoplayback_proxy.each do |proxy| CONFIG.external_videoplayback_proxy.each do |proxy|
begin begin
response = HTTP::Client.get("#{proxy[:url]}/health") response = HTTP::Client.get("#{proxy[:url]}/health")
if response.status_code == 200 if response.status_code == 200
@@proxy_alive = proxy[:url] if @@proxy_list.includes?(proxy[:url])
LOGGER.debug("CheckExternalProxy: Proxy set to: '#{proxy[:url]}'") next
break end
if proxy[:balance]
@@proxy_list << proxy[:url]
LOGGER.debug("CheckExternalProxy: Adding proxy '#{proxy[:url]}' to the list of proxies")
end
break if proxy[:balance] == false && !@@proxy_list.empty?
@@proxy_list << proxy[:url]
end end
rescue rescue
if @@proxy_list.includes?(proxy[:url])
LOGGER.debug("CheckExternalProxy: Proxy '#{proxy[:url]}' is not available, removing it from the list of proxies")
@@proxy_list.delete(proxy[:url])
end
LOGGER.debug("CheckExternalProxy: Proxy '#{proxy[:url]}' is not available") LOGGER.debug("CheckExternalProxy: Proxy '#{proxy[:url]}' is not available")
end end
end end
if @@proxy_alive.empty? LOGGER.trace("CheckExternalProxy: List of proxies:")
LOGGER.warn("CheckExternalProxy: No proxies alive! Using own server proxy") LOGGER.trace("#{@@proxy_list.inspect}")
end
# TODO: If the function is called many times, it will return a random
# proxy from the list. That is not how it should be.
# It should return the same proxy, in multiple function calls
def select_proxy
if (@@count - (Time.utc.to_unix - 30)) <= 0
return if @@proxy_list.size <= 0
@@current_proxy = @@proxy_list[Random.rand(@@proxy_list.size)]
LOGGER.debug("Current proxy is: '#{@@current_proxy}'")
@@count = Time.utc.to_unix
end end
end end
def get_external_proxy def get_external_proxy
return @@proxy_alive return @@current_proxy
end end
def proxy_video_url(raw_url : String, *, region : String? = nil, absolute : Bool = false) def proxy_video_url(raw_url : String, *, region : String? = nil, absolute : Bool = false)
@ -38,8 +61,8 @@ module Invidious::HttpServer
url.query_params = params url.query_params = params
if absolute if absolute
if !@@proxy_alive.empty? if !(proxy = get_external_proxy()).empty?
return "#{@@proxy_alive}#{url.request_target}" return "#{proxy}#{url.request_target}"
else else
return "#{HOST_URL}#{url.request_target}" return "#{HOST_URL}#{url.request_target}"
end end

View file

@ -5,8 +5,9 @@ class Invidious::Jobs::CheckExternalProxy < Invidious::Jobs::BaseJob
def begin def begin
loop do loop do
HttpServer::Utils.check_external_proxy HttpServer::Utils.check_external_proxy
LOGGER.info("CheckExternalProxy: Done, sleeping for 10 seconds") HttpServer::Utils.select_proxy
sleep 10.seconds LOGGER.info("CheckExternalProxy: Done, sleeping for 15 seconds")
sleep 15.seconds
Fiber.yield Fiber.yield
end end
end end

View file

@ -11,9 +11,29 @@ module Invidious::Routes::Images
end end
end end
# We're encapsulating this into a proc in order to easily reuse this
# portion of the code for each request block below.
request_proc = ->(response : HTTP::Client::Response) {
env.response.status_code = response.status_code
response.headers.each do |key, value|
if !RESPONSE_HEADERS_BLACKLIST.includes?(key.downcase)
env.response.headers[key] = value
end
end
env.response.headers["Access-Control-Allow-Origin"] = "*"
if response.status_code >= 300
env.response.headers.delete("Transfer-Encoding")
return
end
proxy_file(response, env)
}
begin begin
GGPHT_POOL.client &.get(url, headers) do |resp| HTTP::Client.get("https://yt3.ggpht.com#{url}") do |resp|
return self.proxy_image(env, resp) return request_proc.call(resp)
end end
rescue ex rescue ex
end end
@ -41,10 +61,27 @@ module Invidious::Routes::Images
end end
end end
begin request_proc = ->(response : HTTP::Client::Response) {
get_ytimg_pool(authority).client &.get(url, headers) do |resp| env.response.status_code = response.status_code
response.headers.each do |key, value|
if !RESPONSE_HEADERS_BLACKLIST.includes?(key.downcase)
env.response.headers[key] = value
end
end
env.response.headers["Connection"] = "close" env.response.headers["Connection"] = "close"
return self.proxy_image(env, resp) env.response.headers["Access-Control-Allow-Origin"] = "*"
if response.status_code >= 300
return env.response.headers.delete("Transfer-Encoding")
end
proxy_file(response, env)
}
begin
HTTP::Client.get("https://#{authority}.ytimg.com#{url}") do |resp|
return request_proc.call(resp)
end end
rescue ex rescue ex
end end
@ -64,9 +101,26 @@ module Invidious::Routes::Images
end end
end end
request_proc = ->(response : HTTP::Client::Response) {
env.response.status_code = response.status_code
response.headers.each do |key, value|
if !RESPONSE_HEADERS_BLACKLIST.includes?(key.downcase)
env.response.headers[key] = value
end
end
env.response.headers["Access-Control-Allow-Origin"] = "*"
if response.status_code >= 300 && response.status_code != 404
return env.response.headers.delete("Transfer-Encoding")
end
proxy_file(response, env)
}
begin begin
get_ytimg_pool("i9").client &.get(url, headers) do |resp| HTTP::Client.get("https://i9.ytimg.com#{url}") do |resp|
return self.proxy_image(env, resp) return request_proc.call(resp)
end end
rescue ex rescue ex
end end
@ -111,7 +165,8 @@ module Invidious::Routes::Images
if name == "maxres.jpg" if name == "maxres.jpg"
build_thumbnails(id).each do |thumb| build_thumbnails(id).each do |thumb|
thumbnail_resource_path = "/vi/#{id}/#{thumb[:url]}.jpg" thumbnail_resource_path = "/vi/#{id}/#{thumb[:url]}.jpg"
if get_ytimg_pool("i9").client &.head(thumbnail_resource_path, headers).status_code == 200 # This can likely be optimized into a (small) pool sometime in the future.
if HTTP::Client.head("https://i.ytimg.com#{thumbnail_resource_path}").status_code == 200
name = thumb[:url] + ".jpg" name = thumb[:url] + ".jpg"
break break
end end
@ -126,15 +181,7 @@ module Invidious::Routes::Images
end end
end end
begin request_proc = ->(response : HTTP::Client::Response) {
get_ytimg_pool("i").client &.get(url, headers) do |resp|
return self.proxy_image(env, resp)
end
rescue ex
end
end
private def self.proxy_image(env, response)
env.response.status_code = response.status_code env.response.status_code = response.status_code
response.headers.each do |key, value| response.headers.each do |key, value|
if !RESPONSE_HEADERS_BLACKLIST.includes?(key.downcase) if !RESPONSE_HEADERS_BLACKLIST.includes?(key.downcase)
@ -144,10 +191,19 @@ module Invidious::Routes::Images
env.response.headers["Access-Control-Allow-Origin"] = "*" env.response.headers["Access-Control-Allow-Origin"] = "*"
if response.status_code >= 300 if response.status_code >= 300 && response.status_code != 404
return env.response.headers.delete("Transfer-Encoding") return env.response.headers.delete("Transfer-Encoding")
end end
return proxy_file(response, env) proxy_file(response, env)
}
begin
# This can likely be optimized into a (small) pool sometime in the future.
HTTP::Client.get("https://i.ytimg.com#{url}") do |resp|
return request_proc.call(resp)
end
rescue ex
end
end end
end end

View file

@ -152,7 +152,6 @@ module Invidious::Routes::Watch
end end
end end
# Removes non default audio tracks
audio_streams.reject! do |z| audio_streams.reject! do |z|
z if z.dig?("audioTrack", "audioIsDefault") == false z if z.dig?("audioTrack", "audioIsDefault") == false
end end
@ -219,12 +218,6 @@ module Invidious::Routes::Watch
captions: video.captions captions: video.captions
) )
begin
video_url = fmt_stream[0]["url"].to_s
rescue
video_url = nil
end
templated "watch" templated "watch"
end end

View file

@ -123,7 +123,6 @@ module Invidious::Videos
"Esperanto", "Esperanto",
"Estonian", "Estonian",
"Filipino", "Filipino",
"Filipino (auto-generated)",
"Finnish", "Finnish",
"French", "French",
"French (auto-generated)", "French (auto-generated)",

View file

@ -13,13 +13,11 @@
<meta property="og:image" content="<%= HOST_URL %>/vi/<%= video.id %>/maxres.jpg"> <meta property="og:image" content="<%= HOST_URL %>/vi/<%= video.id %>/maxres.jpg">
<meta property="og:description" content="<%= HTML.escape(video.short_description) %>"> <meta property="og:description" content="<%= HTML.escape(video.short_description) %>">
<meta property="og:type" content="video.other"> <meta property="og:type" content="video.other">
<!-- This shouldn't be empty, ever. --> <meta property="og:video:url" content="<%= HOST_URL %>/embed/<%= video.id %>">
<meta property="og:video" content="<%= video_url %>"> <meta property="og:video:secure_url" content="<%= HOST_URL %>/embed/<%= video.id %>">
<meta property="og:video:url" content="<%= video_url %>"> <meta property="og:video:type" content="text/html">
<meta property="og:video:secure_url" content="<%= video_url %>"> <meta property="og:video:width" content="1280">
<meta property="og:video:type" content="video/mp4"> <meta property="og:video:height" content="720">
<meta property="og:video:width" content="640">
<meta property="og:video:height" content="360">
<meta name="twitter:card" content="player"> <meta name="twitter:card" content="player">
<meta name="twitter:url" content="<%= HOST_URL %>/watch?v=<%= video.id %>"> <meta name="twitter:url" content="<%= HOST_URL %>/watch?v=<%= video.id %>">
<meta name="twitter:title" content="<%= title %>"> <meta name="twitter:title" content="<%= title %>">

View file

@ -1,6 +1,17 @@
# Mapping of subdomain => YoutubeConnectionPool def add_yt_headers(request)
# This is needed as we may need to access arbitrary subdomains of ytimg request.headers.delete("User-Agent") if request.headers["User-Agent"] == "Crystal"
private YTIMG_POOLS = {} of String => YoutubeConnectionPool request.headers["User-Agent"] ||= "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
request.headers["Accept-Charset"] ||= "ISO-8859-1,utf-8;q=0.7,*;q=0.7"
request.headers["Accept"] ||= "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
request.headers["Accept-Language"] ||= "en-US,en;q=0.9"
# Preserve original cookies and add new YT consent cookie for EU servers
request.headers["Cookie"] = "#{request.headers["cookie"]?}; CONSENT=PENDING+#{Random.rand(100..999)}"
if !CONFIG.cookies.empty?
request.headers["Cookie"] = "#{(CONFIG.cookies.map { |c| "#{c.name}=#{c.value}" }).join("; ")}; #{request.headers["cookie"]?}"
end
end
struct YoutubeConnectionPool struct YoutubeConnectionPool
property! url : URI property! url : URI
@ -15,16 +26,12 @@ struct YoutubeConnectionPool
def client(&) def client(&)
conn = pool.checkout conn = pool.checkout
# Proxy needs to be reinstated every time we get a client from the pool
conn.proxy = make_configured_http_proxy_client() if CONFIG.http_proxy
begin begin
response = yield conn response = yield conn
rescue ex rescue ex
conn.close conn.close
conn = HTTP::Client.new(url) conn = HTTP::Client.new(url)
conn.proxy = make_configured_http_proxy_client() if CONFIG.http_proxy
conn.family = CONFIG.force_resolve conn.family = CONFIG.force_resolve
conn.family = Socket::Family::INET if conn.family == Socket::Family::UNSPEC conn.family = Socket::Family::INET if conn.family == Socket::Family::UNSPEC
conn.before_request { |r| add_yt_headers(r) } if url.host == "www.youtube.com" conn.before_request { |r| add_yt_headers(r) } if url.host == "www.youtube.com"
@ -47,21 +54,6 @@ struct YoutubeConnectionPool
end end
end end
def add_yt_headers(request)
request.headers.delete("User-Agent") if request.headers["User-Agent"] == "Crystal"
request.headers["User-Agent"] ||= "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
request.headers["Accept-Charset"] ||= "ISO-8859-1,utf-8;q=0.7,*;q=0.7"
request.headers["Accept"] ||= "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
request.headers["Accept-Language"] ||= "en-us,en;q=0.5"
# Preserve original cookies and add new YT consent cookie for EU servers
request.headers["Cookie"] = "#{request.headers["cookie"]?}; CONSENT=PENDING+#{Random.rand(100..999)}"
if !CONFIG.cookies.empty?
request.headers["Cookie"] = "#{(CONFIG.cookies.map { |c| "#{c.name}=#{c.value}" }).join("; ")}; #{request.headers["cookie"]?}"
end
end
def make_client(url : URI, region = nil, force_resolve : Bool = false) def make_client(url : URI, region = nil, force_resolve : Bool = false)
client = HTTP::Client.new(url) client = HTTP::Client.new(url)
@ -85,31 +77,3 @@ def make_client(url : URI, region = nil, force_resolve : Bool = false, &)
client.close client.close
end end
end end
def make_configured_http_proxy_client
# This method is only called when configuration for an HTTP proxy are set
config_proxy = CONFIG.http_proxy.not_nil!
return HTTP::Proxy::Client.new(
config_proxy.host,
config_proxy.port,
username: config_proxy.user,
password: config_proxy.password,
)
end
# Fetches a HTTP pool for the specified subdomain of ytimg.com
#
# Creates a new one when the specified pool for the subdomain does not exist
def get_ytimg_pool(subdomain)
if pool = YTIMG_POOLS[subdomain]?
return pool
else
LOGGER.info("ytimg_pool: Creating a new HTTP pool for \"https://#{subdomain}.ytimg.com\"")
pool = YoutubeConnectionPool.new(URI.parse("https://#{subdomain}.ytimg.com"), capacity: CONFIG.pool_size)
YTIMG_POOLS[subdomain] = pool
return pool
end
end