Fijxu/invidious
4
4
Fork 2
Code Issues 21 Pull requests Projects Releases Packages 1 Wiki Activity Actions

[Bug] LOST SUBSCRIPTIONS! #41

New issue
Open
opened 2024-11-13 01:06:23 -03:00 by xCbl6YaK4PEX6d9g6m26YMdHJtQgJ · 2 comments
xCbl6YaK4PEX6d9g6m26YMdHJtQgJ commented 2024-11-13 01:06:23 -03:00
Copy link

this is a total nightmare!

due to broken feed issue (#21) I am doing these JSON flash outs (where I save JSON data just to reimport it back to refresh the feed) where I always rewrite/replace old JSON data file with new JSON data file.

by pure luck, I found an older JSON data file, saved in another location. this older JSON is aprox. 1 month older than my most up-to-date JSON file saved 1h ago

  1. the old JSON has 150 subscriptions
  2. the new JSON has 147 subscriptions

image

and here is the issue: I never ever unsubscribed from any channel, so the new JSON should have like 170+ subscriptions! if I run a simple comparison between those two JSONs I get 18 LOST SUBSCRIPTIONS!

image

in reality, I miss way more than that!

checking those 18 on invidious instance (where newest/latest JSON was applied) I really see that those channels are unsubscribed! moreover, crosschecking it with YT I totally lost them somehow, and it is not my imagination as I can check like https://webapps.stackexchange.com/a/126965/186471 that to some of them I was subscribed for years (before I even knew there is invidious and applied for data takeout from YT)

from now on, I will need to keep every saved JSON file and excel-shit out of it every lost subscription :(

not sure why or how it happened, nor how to reproduce it, but I have a vague idea...

assuming there is backward compatibility in place between master releases of invidious, I see two possible scenarios:

  1. when exporting via "Export Invidious data as JSON" the stuff is not collected correctly (less likely)
  2. when importing via "Import Invidious JSON data" the file is uploaded only partially (most likely) because either file is too big (my new JSON has 104kB - which is absolutely not big at all for current internet standards) or processing window is too small so it timeouts*.

* in past (5 months ago) when I picked up invidious I had this great idea to save myriad of stuff into playlists and keep up-to-date JSON file in case the instance dies, so I will just respawn on another instance and redeploy all my baggages in few seconds. sadly, the idea was doomed as upload of JSON file often failed completely resulting in 503, 501 or 500 errors, or JSON file got uploaded only partially with half subscriptions and half playlists missing. this needed to be repeated several times until it went through and all subscriptions were accounted for, but playlists by that time were duplicated several times (like 6-7 incomplete copies of each individual playlist with various video count) resulting in total failure, so the playlists needed to be dropped. Nowadays, I just download everything on usb stick with yt-dlp.exe and JSON file is dedicated only to subscriptions and watched history.

therefore, extending the time window for upload or revamping the current upload process/logic is needed for sure.

this is a total nightmare! due to broken feed issue (https://git.nadeko.net/Fijxu/invidious/issues/21) I am doing these JSON flash outs (where I save JSON data just to reimport it back to refresh the feed) where I always rewrite/replace old JSON data file with new JSON data file. by pure luck, I found an older JSON data file, saved in another location. this older JSON is aprox. 1 month older than my most up-to-date JSON file saved 1h ago 1. the old JSON has 150 subscriptions 2. the new JSON has 147 subscriptions ![image](/attachments/2803653f-e45a-4134-b974-9b9dafa5ba18) and here is the issue: I never ever unsubscribed from any channel, so the new JSON should have like 170+ subscriptions! if I run a simple comparison between those two JSONs I get 18 LOST SUBSCRIPTIONS! ![image](/attachments/75a7459d-37a6-4cf8-a3b4-434090af3175) in reality, I miss way more than that! checking those 18 on invidious instance (where newest/latest JSON was applied) I really see that those channels are unsubscribed! moreover, crosschecking it with YT I totally lost them somehow, and it is not my imagination as I can check like https://webapps.stackexchange.com/a/126965/186471 that to some of them I was subscribed for years (before I even knew there is invidious and applied for data takeout from YT) from now on, I will need to keep every saved JSON file and excel-shit out of it every lost subscription :( _______________ not sure why or how it happened, nor how to reproduce it, but I have a vague idea... assuming there is backward compatibility in place between master releases of invidious, I see two possible scenarios: 1. when exporting via "Export Invidious data as JSON" the stuff is not collected correctly (less likely) 2. when importing via "Import Invidious JSON data" the file is uploaded only partially (most likely) because either file is too big (my new JSON has 104kB - which is absolutely not big at all for current internet standards) or processing window is too small so it timeouts*. \* in past (5 months ago) when I picked up invidious I had this great idea to save myriad of stuff into playlists and keep up-to-date JSON file in case the instance dies, so I will just respawn on another instance and redeploy all my baggages in few seconds. sadly, the idea was doomed as upload of JSON file often failed completely resulting in 503, 501 or 500 errors, or JSON file got uploaded only partially with half subscriptions and half playlists missing. this needed to be repeated several times until it went through and all subscriptions were accounted for, but playlists by that time were duplicated several times (like 6-7 incomplete copies of each individual playlist with various video count) resulting in total failure, so the playlists needed to be dropped. Nowadays, I just download everything on usb stick with yt-dlp.exe and JSON file is dedicated only to subscriptions and watched history. therefore, extending the time window for upload or revamping the current upload process/logic is needed for sure.
image.png
101 KiB
image.png
169 KiB
Fijxu commented 2024-11-13 16:01:26 -03:00
Owner
Copy link

extending the time window for upload or revamping the current upload process/logic is needed for sure.

Yeah that is definitely needed because Invidious does all the process in the request and not on the background. That process of importing playlists and subscriptions takes time because I THINK it needs to fetch the information of every video of the playlist, and that takes time.

when exporting via "Export Invidious data as JSON" the stuff is not collected correctly (less likely)
when importing via "Import Invidious JSON data" the file is uploaded only partially (most likely) because either file is too big (my new JSON has 104kB - which is absolutely not big at all for current internet standards) or processing window is too small so it timeouts*.
  1. Is unlikely, the file is actually uploaded but invidious needs to process it, which it takes a long time and if the timeout from the server is too low, it's going to return an error code to the user but the server is still processing the imported data.
    I'm not sure about 1. tho

assuming there is backward compatibility in place between master releases of invidious

Yeah I barely touched anything related to subscriptions.

Can I ask your Invidious username to take a look at the database?

>extending the time window for upload or revamping the current upload process/logic is needed for sure. Yeah that is definitely needed because Invidious does all the process in the request and not on the background. That process of importing playlists and subscriptions takes time because I THINK it needs to fetch the information of every video of the playlist, and that takes time. > when exporting via "Export Invidious data as JSON" the stuff is not collected correctly (less likely) when importing via "Import Invidious JSON data" the file is uploaded only partially (most likely) because either file is too big (my new JSON has 104kB - which is absolutely not big at all for current internet standards) or processing window is too small so it timeouts*. 2. Is unlikely, the file is actually uploaded but invidious needs to process it, which it takes a long time and if the timeout from the server is too low, it's going to return an error code to the user but the server is still processing the imported data. I'm not sure about 1. tho > assuming there is backward compatibility in place between master releases of invidious Yeah I barely touched anything related to subscriptions. --- Can I ask your Invidious username to take a look at the database?
xCbl6YaK4PEX6d9g6m26YMdHJtQgJ commented 2024-11-13 18:52:38 -03:00
Author
Copy link

Can I ask your Invidious username to take a look at the database?

sure! here: xCbl6YaK4PEX6d9g6m26YMdHJtQgJ

can you perhaps add some alternative import option (to sideload the JSON file) which would be more timeout friendly, e.g. less prone to errors and failing?

or maybe separate those 3 types into 3 individual JSONs ?

  1. subscriptions
  2. watched history
  3. playlists

or maybe upload it raw in server and do "delayed" processing, something like: after the upload, the changes will be reflected in 5-10 minutes (based on some queue) instead of immediate processing (like it is right now)

atm here are my numbers:

  1. subscriptions: 161
  2. watched history: 7135
  3. playlists: 0
> Can I ask your Invidious username to take a look at the database? sure! here: xCbl6YaK4PEX6d9g6m26YMdHJtQgJ can you perhaps add some alternative import option (to sideload the JSON file) which would be more timeout friendly, e.g. less prone to errors and failing? or maybe separate those 3 types into 3 individual JSONs ? 1. subscriptions 2. watched history 3. playlists or maybe upload it raw in server and do "delayed" processing, something like: after the upload, the changes will be reflected in 5-10 minutes (based on some queue) instead of immediate processing (like it is right now) atm here are my numbers: 1. subscriptions: 161 2. watched history: 7135 3. playlists: 0
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
history-testing
testingbuh
testing3
testing2
testing
xdxdxd
experimental
test
external-proxy
debug
test123
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Fijxu/invidious#41
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?