Kaydol

Flood göndermek, insanların floodlarını okumak ve diğer insanlarla bağlantı kurmak için sosyal Floodlar ve Flood Yanıtları Motorumuza kaydolun.

Oturum aç

Flood göndermek, insanların floodlarını okumak ve diğer insanlarla bağlantı kurmak için sosyal Floodlar ve Flood Yanıtları Motorumuza giriş yapın.

Şifremi hatırlamıyorum

Şifreni mi unuttun? Lütfen e-mail adresinizi giriniz. Bir bağlantı alacaksınız ve e-posta yoluyla yeni bir şifre oluşturacaksınız.

3 ve kadim dostu 1 olan sj'yi rakamla giriniz. ( 31 )

Üzgünüz, Flood yazma yetkiniz yok, Flood girmek için giriş yapmalısınız.

Lütfen bu Floodun neden bildirilmesi gerektiğini düşündüğünüzü kısaca açıklayın.

Lütfen bu cevabın neden bildirilmesi gerektiğini kısaca açıklayın.

Please briefly explain why you feel this user should be reported.

After hoarding over 50k YouTube videos, here is the youtube-dl command I settled on.

*EDIT: If you are reading this, I’ve made a few small changes. You can find the actual scripts I use here: https://github.com/velodo/youtube-dl_script. While my serup works great for me, if you’re looking for a more robust solution, please check out TheFrenchGhosty’s scripts here: https://github.com/TheFrenchGhosty/TheFrenchGhostys-YouTube-DL-Archivist-Scripts, with the associated reddit thread here: https://redd.it/h7q4nz.*

After seeing all of the posts recently regarding youtube-dl, I figured I would chime in on the options I use. There are a few things I want to implement as some point, see the bottom of this post for those. Also, if anyone sees anything that can be done better, please let me know as I am always looking for ways to improve everything I do! Also, this post isn’t intended to be a guide on how to use youtube-dl, this is more for the arguments I use and why I use them. If you need help getting youtube-dl running, setting up a batch script, etc. there are plenty of guides for that sort of thing elsewhere.

**The command (DONT COPY PASTE THIS ONE):**

youtube-dl –download-archive “archive.log” -i –add-metadata –all-subs –embed-subs
–embed-thumbnail –match-filter “playlist_title != ‘Liked videos’ & playlist_title !=
‘Favorites'” -f “(bestvideo[vcodec^=av01][height>=1080][fps>30]/bestvideo[vcodec=vp9.2]
[height>=1080][fps>30]/bestvideo[vcodec=vp9][height>=1080][fps>30]/bestvideo[vcodec^=av01]
[height>=1080]/bestvideo[vcodec=vp9.2][height>=1080]/bestvideo[vcodec=vp9]
[height>=1080]/bestvideo[height>=1080]/bestvideo[vcodec^=av01][height>=720]
[fps>30]/bestvideo[vcodec=vp9.2][height>=720][fps>30]/bestvideo[vcodec=vp9][height>=720]
[fps>30]/bestvideo[vcodec^=av01][height>=720]/bestvideo[vcodec=vp9.2]
[height>=720]/bestvideo[vcodec=vp9][height>=720]/bestvideo[height>=720]/bestvideo)+
(bestaudio[acodec=opus]/bestaudio)/best” –merge-output-format mkv -o “%cd%/%%
(playlist_uploader)s/%%(playlist)s/%%(playlist_index)s – %%(title)s – %%(id)s.%%(ext)s”
“[URL HERE TO CHANNELS PLAYLISTS]”

**The command again (copy paste friendly):**

youtube-dl –download-archive “archive.log” -i –add-metadata –all-subs –embed-subs –embed-thumbnail –match-filter “playlist_title != ‘Liked videos’ & playlist_title != ‘Favorites'” -f “(bestvideo[vcodec^=av01][height>=1080][fps>30]/bestvideo[vcodec=vp9.2][height>=1080][fps>30]/bestvideo[vcodec=vp9][height>=1080][fps>30]/bestvideo[vcodec^=av01][height>=1080]/bestvideo[vcodec=vp9.2][height>=1080]/bestvideo[vcodec=vp9][height>=1080]/bestvideo[height>=1080]/bestvideo[vcodec^=av01][height>=720][fps>30]/bestvideo[vcodec=vp9.2][height>=720][fps>30]/bestvideo[vcodec=vp9][height>=720][fps>30]/bestvideo[vcodec^=av01][height>=720]/bestvideo[vcodec=vp9.2][height>=720]/bestvideo[vcodec=vp9][height>=720]/bestvideo[height>=720]/bestvideo)+(bestaudio[acodec=opus]/bestaudio)/best” –merge-output-format mkv -o “%cd%/%%(playlist_uploader)s/%%(playlist)s/%%(playlist_index)s – %%(title)s – %%(id)s.%%(ext)s” “[URL HERE TO CHANNELS PLAYLISTS]”

I know it looks long and scary, let me break it down a little bit:

–download-archive “archive.log”

This keeps track of all the videos you have downloaded so they can be skipped over the next time it’s ran or the next time it finds that video.

-i

Ignore any errors that occur while downloading. Occasionally they will happen and this just ensures things keep moving along as intended. Don’t worry, the next time it is ran any videos that didn’t fully download will most likely be picked right back up where it left off!

–add-metadata –all-subs –embed-subs –embed-thumbnail

These just embed metadata into the video once it’s done downloading. You never know when this will come in handy, and having it all right in the video’s container is nice. Just a little note, at the time of writing this post, ffmpeg can’t embed images into a mkv, but the image is still downloaded and stored in the same location and with the same name as the video.

–match-filter “playlist_title != ‘Liked videos’ & playlist_title != ‘Favorites'”

This will filter videos out that you don’t want to download. Here is just a basic example of filtering out playlists with the title “Liked Videos” and “Favorites”. I find this especially useful for filtering out playlists that contain a bunch of videos from other playlists. For example, if I’m downloading videos from a gaming channel and they have a playlist for “Gmod” and one for “Minecraft PC”, but they also have one called “PC Games” that contains the contents of both the Gmod and the Minecraft playlists, I sometimes will want to keep those separate, so I will filter out the “PC Games” playlist. If there are videos in that playlist you still want, you can always add another youtube-dl command to your script with that playlist specifically. Depending on the channel, this can get rather annoying to manage, but its a good way to keep things better organized.

-f “(bestvideo[vcodec^=av01][height>=1080][fps>30]/bestvideo[vcodec=vp9.2][height>=1080]
[fps>30]/bestvideo[vcodec=vp9][height>=1080][fps>30]/bestvideo[vcodec^=av01]
[height>=1080]/bestvideo[vcodec=vp9.2][height>=1080]/bestvideo[vcodec=vp9]
[height>=1080]/bestvideo[height>=1080]/bestvideo[vcodec^=av01][height>=720]
[fps>30]/bestvideo[vcodec=vp9.2][height>=720][fps>30]/bestvideo[vcodec=vp9][height>=720]
[fps>30]/bestvideo[vcodec^=av01][height>=720]/bestvideo[vcodec=vp9.2][height>=720]/
bestvideo[vcodec=vp9][height>=720]/bestvideo[height>=720]/bestvideo)+
(bestaudio[acodec=opus]/bestaudio)/best”

Ok. This is where things get a little tricky. I first want to start off by saying that this isn’t totally necessary as all the -f command does is allows you to set preferences on what video and audio streams you want to download. If you want a basic rundown on how this works, the youtube-dl readme explains it better than I ever could. For my case here, I want to download video streams in certain codecs, which have a hierarchy of [av1>vp9.2>vp9>whatever is available]. It will keep going down the list until one is found that meets my criteria. You can also see that I prefer videos in 1080 with more than 30 fps, then 1080 30 fps, and that repeats for 720. I also prefer to get audio in opus if it’s available. Just a side note for anyone wondering what vp9.2 is, it is the vp9 codec with HDR.

Why bother with all of that nonsense when youtube-dl will automatically pick the best streams for you? Well, the way youtube-dl picks the best stream is based solely on bitrate. This means that for video it will usually chose the avc1 codec, which is pretty old at this point, and while it still looks good, I’ve found that the other codecs offer a smaller file size and similar or better quality. You may find otherwise and want to do things differently, but for me, this is how I do it as it saves hard drive space and I find the quality good. Also, as you will notice, I don’t have any resolutions higher than 1080 on there. The way I have it, it should catch those higher res streams, but as of now, I don’t archive many youtubers’ videos that upload in higher res so I haven’t found the need, but some day I’m sure I will change it. I already know your asking yourself, “If this will catch the higher resolution streams, why don’t you just leave the 720 options in there and remove the 1080?”. Well, it’s because I’ve noticed that youtube has started to transcode many videos to the newer av1 codec, but so far most videos that I’ve seen only go up to 720 for the av1 codec. This means that if that stream is available, but there isn’t a 1080p av1 stream, then it will always download those videos in 720p even if a higher res stream is available.

–merge-output-format mkv -o “%cd%/%%(playlist_uploader)s/%%(playlist)s/%%(playlist_index)s –
%%(title)s – %%(id)s.%%(ext)s”

This just tells youtube-dl where I want the file and that I want it in an mkv container. It’s pretty self explanatory, but I basically want a folder structure of “[CHANNEL NAME]/[PLAYLIST NAME]/[PLAYLIST INDEX] – [VIDEO TITLE] – [YOUTUBE VIDEO ID].[EXTENSION]”. Feel free to customize this however you see fit. Please note that I used double % in some of these due to my script being a batch file ran on a windows VM.

**Things I want to do:**

– Create a docker container that runs the script. While the Windows VM is working perfect for this, it’s the only thing the VM does now (used to be used for much more, but that has all been offloaded). It should be pretty easy since I leave the executables and the script all in a network share as is, so all it would need is the dependencies (which I think is only python if I’m not mistaken) and to set up a cron job.

– Simplify the script a bit by using the -a argument. This would allow me to set up a file with the links I want to download. This would allow me to group a bunch of commands that all have the same arguments into 1 command.

– Write a script that will move videos that were downloaded before they were put into playlists into their respective playlists once the uploader adds them. Right now what I do is download all of the uploader’s playlists, then download all of their videos (using the same archive file so it doesn’t re-download any). This means if the uploader is slow to add the video to a playlist, it will just be downloaded to a “No Playlist” folder. The other way I could do this would be to find a way to deduplicate all of the videos in the “No Playlist” folder and just use separate archive files for the playlist and non playlist videos, which might download some videos twice, but then later deduplicated.

**Final Thoughts:**

Youtube-dl is a wonderful and powerful tool, and with all of the crap going down on YouTube, you can never be too sure what videos you love might be taken down. Just what I’ve managed to download has already helped me and some of my friends out. It definitely is worth your time to automate downloading videos from channels you enjoy, and with a little know-how and experimentation, it doesn’t take much time or effort to get something to a point where you can set it and forget it.

Anyway, that was certainly longer than I thought it would be, but I really hope it helps some of you guys out. I’ve gained so much knowledge from this subreddit and it would mean a lot if I gave back and helped one of you out in return. Happy Hoarding!

*Just a quick edit:* Be sure to check out the comments for some excellent ideas and more information on some things! As always, take this information and adapt it to your use case. Maybe my configuration will work perfectly for you, but more than likely you will have to tweak it a bit to get it just right for you. If you have any questions, please ask!

*Another quick edit:* Some of the comments have brought up the fact that us as viewers of YouTube content, and even youtube-dl itself don’t have any way to watch or download the original quality of the material as YouTube will automatically transcode videos when they are uploaded. This can be a problem for people who are trying to preserve things in the best quality they possibly can. If you are one of these people, you might want to try looking elsewhere for better quality releases of the content. The one example that immediately comes to mind for me is content from Rooster Teeth. The quality when downloaded directly from their website seems to be better quality than what you can pull from YouTube. For me personally, I will download some movies, and TV Shows and also most music and images in the best possible quality I can find, but when it comes to YouTube content, I just don’t care as much and find the convenience of ripping directly from YouTube hard to beat. I also think the content tends to look great, especially for the file sizes, but this is obviously all up to you to decide.

Benzer Yazılar

Yorum eklemek için giriş yapmalısınız.

28 Yorumları

  1. But what about if the channel doesn’t have their videos in playlists? You’re going to miss out on a large portion of the videos from channels.

  2. Thank you for this, quick question, the thumbnail for me is a separate file. Is this supposed to be the case or am I doing something wrong? Thanks again

  3. >Also, as you will notice, I don’t have any resolutions higher than 1080 on there. The way I have it, it should catch those higher res streams, but as of now, I don’t archive many youtubers’ videos that upload in higher res so I haven’t found the need, but some day I’m sure I will change it.

    I’m running this command and it is downloading 4k videos, about 4gb for a 20min video. So these arguments will catch the highest res video but without a preferred codecs hierarchy above 1080p? If I remove “bestvideo” will it only download the 1080p and 720p?

  4. Hi, anybody can help me out why i get the error on a windows machine:

    WARNING: The url doesn’t specify the protocol, trying with http

    All i did was replace the URL with this value for example:

    check google but nothing similar to my problem. OP said to just replace the URL which is exactly what I did. Do i have to put escape characters inside the URL?

  5. > Just a little note, at the time of writing this post, ffmpeg can’t embed images into a mkv, but the image is still downloaded and stored in the same location and with the same name as the video.

    I got bored of waiting for it to work so I decided to do it manually myself as a post processing process
    https://github.com/Matt07211/dotfiles/blob/master/scripts/.scripts/youtube-dl/ytdl.sh#L26

    >–match-filter “playlist_title != ‘Liked videos’ & playlist_title != ‘Favorites'”

    Yeah I recently justed added this.

    Also another handy tip is when ripping playlists use “`youtude.come/channelname/playlists?view=1“`

    I was wondering why I wasn’t pulling in all playlists on a Channel that I knew had more playlists, adding ?view=1 onto the end of the playlist url allowed YouTube-dl to see and grab them all. The channel I was referring to that this problem was prominent in was https://youtube.com/user/goodgameabctv/playlists

    Where’s if you viewed https://youtube.com/user/goodgameabctv/playlists?view=1 you can see all the playlists

  6. I think I did something wrong… Could someone help?

    * Downloaded youtube-dl.exe (windows)
    * Copy & pasted the script, changing out the last URL for my full playlist URL
    * Ran it…

    Tons of text, and I don’t see any particular error message.

    Folder name is **%(playlist_uploader)s**

    and the files are all **%(playlist_index)s – %(title)s – %(id)s.ar.vtt**

    But they’re all only a few kb in size. So I’m doing something wrong… Help? 🙁

  7. I know this is a long shot, but does anyone in here know how to change the file date to the date that the file was originally updated to Youtube. You can put it in the filename, but from there i don’t know how to have it be the date when it was uploaded, vs the last time it was modified on Youtube.

  8. What a great script, thank you for sharing! But I think everyone should carefully read about original, masterfile-level preservation, which is what YouTube-dl does not do (dev response below):

    ​

    While YTDL is a great resource for archiving at-risk content, it is NOT a solution for true, archival preservation of master-file/original file preservation.

    I’m just going to jump in as some are having questions regarding getting the *actual original file/format*

    [https://www.reddit.com/r/DataHoarder/comments/az7czy/re_youtubedl_archiving_projects_complete_list_of/](https://www.reddit.com/r/DataHoarder/comments/az7czy/re_youtubedl_archiving_projects_complete_list_of/)

    Per an exchange I had a few months ago with one of YouTube-dl’s devs:

    *”To my knowledge, YouTube keeps the original resolution for best format and keep the same aspect ratio for the lower qualities, and in order to reduce the size of the video, they modify the bitrate and file format when possible(av1…) among other things. About the original file uploaded to youtube, it should be accessible only to the uploader of the video.”*

    **In short, this means: YouTube-dl does not tap into the original formatted videos uploaded by the user. Only the original uploader has access to the file that was produced via the camera. Which is what true preservation would mean — the original source material.**

    I gave an example to the dev: When you’re looking to archive anything in highest quality, you go to the source material. If, for example, you were looking to preserve a major motion-picture film, say, Casablanca, the YouTube-dl equivalent is to get a high-quality Blu-Ray rip. It does not grab the original files uploaded on the server.

    So if you were looking to preserve Casablanca, you’d go to the original camera negatives and work off of that. Or to the master print. And work from that. You would not go to a Blu-Ray copy of Casablanca to try and further ‘preserve the integrity’, which is what YouTube-dl does – it pulls a copy of data streams in the highest quality available from the server (post-re-encoding on their side), then remuxes to chosen output format. But it doesn’t pull directly down the original video files uploaded by the given YouTube user.

    **YouTube-dl is great for people taking matters into their own hands for ‘archiving’ video. It is not, however, archiving/downloading videos in original formats, eg; copies of the original videos uploaded from the user’s camera.**

    Source: YouTube-dl dev, Amine Remita.

  9. Hi, nice post, thanks for sharing!

    I made a light Docker container with YouTube-dl https://github.com/qdm12/youtube-dl-docker you can probably copy paste your command in there or in a docker-compose `command` field. Let me know if you would want me to implement something you would need!

  10. –no-mtime is also useful if you want to know when you downloaded a video. Otherwise it doesn’t seem to set the modified time randomly.

  11. > Simplify the script a bit by using the -a argument. This would allow me to set up a file with the links I want to download. This would allow me to group a bunch of commands that all have the same arguments into 1 command.

    This is pretty easy, in my youtube-dl config file, I have this:

    `-a youtube-dl-channels.txt`

    and that file simply contains channel URLS:

    # Binging with Babish
    https://www.youtube.com/user/bgfilms

    # Hot Ones

  12. As a Windows user, command lines scare me. So I made* this shortcut. Put youtube-dl.exe, ffmpeg.exe, and shortcut.bat in the same folder.

    Copy the URL of a video or playlist, and then double click shortcut.bat

    shortcut.bat :

    hornylolifucker (@CodeSection == @Batch) TheNapea

    MaybeNotTheChosenOne off
    setlocal

    set “getclip=cscript /nologo /e:JScript “%~f0″”

    %getclip% > link.txt
    start cmd /c youtube-dl.exe -a link.txt

    goto :EOF

    _FriendlyPanicAttack // begin JScript hybrid chimera
    WSH.Echo(WSH.CreateObject(‘htmlfile’).parentWindow.clipboardData.getData(‘text’));

    *I have no idea what any of this gibberish means, I copied the script from somewhere. I don’t know which lines are actually needed. haha

    (My internet sucks and can’t even stream 144p, downloading is less painful than endless buffering.)

  13. I just woke up and all I can say is wow.. I really wasn’t expecting such a strong response overnight like this! I want to give a big thank you to the people who gave me the platinum and silver awards! I also want to thank those of you who have answered other’s questions, and those who shared their own ideas, scripts, etc. I’ve already got a few ideas from you guys that I may end up using.

  14. > Why bother with all of that nonsense when youtube-dl will automatically pick the best streams for you? Well, the way youtube-dl picks the best stream is based solely on bitrate. This means that for video it will usually chose the avc1 codec, which is pretty old at this point, and while it still looks good, I’ve found that the other codecs offer a smaller file size and similar or better quality.

    If you did not already, you could reach out to the youtube-dl developers and suggest they change their default behavior.

  15. Great write up! I’m at 12k videos and mine is very similar.

    One big difference is that I don’t like dealing with playlists, there are duplicates and many youtubers seldom refresh their playlists so you might miss out on new videos. So I focus on whole channels instead and shaping with –match-title/–reject-title if needed. For example, excluding vlogs or only including certain videos, like series.

    I use one big .bat file that’s being scheduled to run every night and since the configs are predefined, it’s easy to keep adding entries of channels or playlists. I can show it if someone’s interested.

    Other differences from OP is that I use –merge-output-format mp4, instead of mkv because in MPC-HC seeking in MKV files is always just a tiny bit longer in my tests. Otherwise I’d much more prefer MKVs as that’s a much better containers.

    I also store the –download-archive files separately for each youtube channel instead of all of them in one file (my goal is to eventually have a system that can detect videos that I have but have been removed from Youtube).

    I don’t use the followings: –all-subs –embed-subs –embed-thumbnail as when I played with it in the past, it never found any subtitles and I can’t embed thumbnails *reliably*. YMMV. I just have the thumbnails and json files separately (–write-all-thumbnails –write-info-json). Sidenote, I also used to use aria2c as external downloader but it downloads sound files veeery slowly so I gave up on that and youtube-dl came a long way since then so it’s fast enough (for people that might still use aria2c).

  16. How would you rerun over a
    url list with an existing download-archive and upgrade all available to best codec/quality and not just the default highest bitrate?
    Any straightforward advice on how to skip downloaded videos that wouldn’t be improved?

    I don’t archive gaming streams so the fps part is probably not too relevant.

    Looks like I need some kind of external scripting that compares local codec with online available ones first, and then moves the that out of the download archive?

    Ideally one might add a feature to the archive flag itself, where it also stores the video’s basic properties, so you can do rule based overrides.

  17. I think you should definitely add `–write-info-json`. That way you have all the metadata (including description, annotations, view/like count, etc) in a nice and parseable format without any possibility of information loss (as you have when embedding in the video file).

    Also you can then easily filter videos with [jq](https://stedolan.github.io/jq/), for example to list all videos longer than 10 minutes with at least 1 million views:

    `jq ‘select(.duration >= 600 and .view_count > 1e6) | .fulltitle’ *.info.json`

    Edit: the json contains a lot of useful stuff like which format was downloaded which you need do [this](https://www.reddit.com/r/DataHoarder/comments/c6fh4x/after_hoarding_over_50k_youtube_videos_here_is/es8v1jy/)

  18. Did you know that you can put all of these options in a config file? That way you don’t need to have the unending command-line from hell.

    Bonus: The config file allows for comments, so you can document the options.

    I’ve got a similarly complex youtube-dl config, but my invocation is just `youtube-dl [URL]`

  19. Maybe I am doing something wrong, but I just tried it and it is giving this error at every video.

    ERROR: %c requires int or char

    None of them ended up downloading.

  20. That’s some nice scripting you’ve got there OP. I’ve also noticed some of the issues with the thumbnail embedding . I’m also downloading a copy of the thumbnail alongside the video. The point of my script is to create an offline copy of specific youtube channels and then pass them into Plex. This isn’t for “archiving” as much as it’s for my convenience:

    1. The watched/unwatched is managed really well by Plex
    2. No ads
    3. Saves bandwidth (these are being synced to my phone as well, helps with offline viewing)

    You’ll note that I’m opting for 720p mp4 with the best m4a audio. Again, this is to maximize cross platform direct play. You’ll notice that the naming scheme is [Uploader] S01E___ – [Title] [Date] [Youtube-ID].mkv. This works well with the Plex Extended Media Scanner. A few additional options that I’ve passed into youtube-dl is –batchfile=[sometxtfile] and –playlist-reverse. What I have is a list of youtube channels inside of a txt file. This script will open that list, then load their entire set of videos, flip them into chronological order, and then start downloading. This is important because I want to have the episode numbers in order, whereas the default youtube playlist is in reverse chronological. The reason why I’m using a txt file with the script is that I can pass multiple channels through youtube-dl and automate it as a cron job nightly.

    So one channel that I use this for is America’s Got Talent. Watching it on TV, even with DVR skipped commercials, is frustrating. There’s a lot of garbage on the TV which is cut out on the Youtube uploads. If you watch it directly on Youtube, you watch 2 ads per video and deal with a crappy interface. So this script works well for me.

    I have a few other youtube-dl commands similar to this one which are for educational content and random pop stuff. All of this is running through a shell file, passed into cron. By the time I wake up, I’ve got the content where ever I want it.


    **Youtube-DL For Plex (Mobile Optimized)**

    #———————————————-
    #Download entire TV-on-Youtube channel
    #———————————————-

    Echo “Downloading complete TV-on-Youtube channels”

    cd ~/Movies/Youtube/TV-on-Youtube

    /usr/local/bin/youtube-dl –playlist-reverse –batch-file=tv-on-youtube_allvideos.txt
    -i -o “%(uploader)s/%(uploader)s – S01E%(playlist_index)s – %(title)s [%(upload_date)s] [%(id)s].%(ext)s”
    -f 136+bestaudio[ext=m4a] –merge-output-format mkv
    –download-archive tv-on-youtube_downloaded.txt
    –embed-thumbnail –embed-subs –write-sub –sub-lang en
    –ignore-errors –no-continue –no-overwrites –no-post-overwrites
    –write-thumbnail
    –add-metadata

  21. thank you for your time! there are several playlists I would like to preserve and your post will come in handy, Im sure of it!