When I started archiving twitch streams, my data usage went up dramatically. As I had to both download and upload each stream, I was easily pushing 60 gigabytes a day. And that was just streaming. But that’s actually going to be another post, as my setup has gotten significantly more complex now that I’ve moved to a VPS.
Anyway, I was essentially using almost two terabytes of data a month, split among stream archiving, general usage, and media ingesting. And I’m in the evil clutches of Comcast/Xfinity and have to use less than 1 terabyte a month or get billed something crazy like $10/50GB or $50/mo for unlimited data. Seeing as they already charge way too much, something had to be done to get under 1 terabyte a month.

Even though stream archiving was a major data hog (my solution to that will be covered in another post [citation needed] ), my method of gathering TV shows and movies was also using data and in need of a serious overhaul anyway.
In this post, I will be thoroughly covering my automated media setup, as well as sharing any scripts I may use.

I used to manually search, download, and rename files into my local filesystem, and point Plex to that. But I don’t have the redundancy of multiple hard drives in RAID which was worrying me, especially with 5+ year old hard drives. So I moved to Google Drive, with an Enterprise account. This gives me unlimited storage for an affordable price. But of course, I would have to upload to Drive first, which would double my usage.
So now I also rent a VPS from Linode, which has been a great experience so far. I’ve even dealt with their customer service a bit to get my VPS migrated to another data center, and they were very helpful. Another bonus is that, while they have a (generous) data cap, they only count upload usage against the data cap, which is great. They also have affordable bulk storage volumes which I additionally use.

Before you start installing stuff (assuming you’re planning on using this as a guide (don’t)), make sure whatever hardware you’re using is properly secured and hardened against common attacks. Close ports, set up a firewall, force key-only SSH, install AV, etc. Securing your stuff is really really important as, for example, my Linode gets hundreds of SSH attempts every day. I’d recommend changing your SSH port to reduce the noise on your logs, however, not everyone agrees, as there are caveats.

Once you have your Fort Knox, you’re good to go.

1 . Software

I have a basic setup of Sonarr, Radarr, Jackett, and Deluge running on the VPS. Plex and NetDrive run locally on my computer to serve content.

1.1 Sonarr/Radarr

Both of these are similar (Radarr is a fork of Sonarr), and both have similar internals but serve a different purpose. Sonarr deals with TV shows, and Radarr deals with movies.
These two packages include a front-end web interface, which allows me to add content I want downloaded, see what I have downloaded, and request missing content to be searched for.

1.2 Deluge

Deluge is a BitTorrent client that downloads everything I request from Sonarr/Radarr. While it’s not the client I normally use, I think it’s the only client that supports headless Linux installs. It also has a lot of critical CLI commands that I use in scripts.
Deluge for Linux includes both the web interface package, as well as the daemon, which does basically everything.

1.3 Jackett

Jackett is responsible for searching multiple torrent websites for content I request and returning it back to Sonarr/Radarr.
Jackett is fully supported on Linux

1.4 Rclone

Rclone is an excellent program for syncing data to/from cloud providers. It is responsible for transferring finished downloads off the VPS’s small drive to my Google Drive. Also, I have my Google Drive ‘mounted’ on the VPS in order for Sonarr/Radarr to be able to know what content is already stored.
Rclone is fully supported on Linux

2 . Speaking the Same Language

Getting separate programs to talk to each other can be quite difficult. Thankfully, Sonarr/Radarr (hereinafter Sonarr) makes it easy to connect to Jackett and Deluge, and if everything is running on the same server it is pretty simple.
In this section, I’ll be going over how to get everything connected to Sonarr, which manages what Deluge and Jackett need to be doing.

2.1.1 Adding Trackers - Jackett

Sonarr needs the results of many torrent sites (indexer) when it is looking for a specific media, say, an episode of a show. By connecting to Jackett indexers, Sonarr can compare the different torrent files that contain the media and choose the best one based on your restrictions. For example, imagine one indexer with a 720p torrent with 10 seeders, and another indexer with a 1080p torrent with 2 seeders. If your quality profile specifies >=720p, and the indexer settings are all >=5 seeders, Sonarr will choose the 720p torrent and send it to Deluge.
So now that you understand how Sonarr and Jackett work together, let’s add some indexers.
In your Jackett dashboard: Click + Add Indexer and choose one. For general ones, you may not need to configure anything. For sites that offer media in languages other than your own, it may be beneficial to specify categories to restrict the search to. But for many sites, all you have to do is add the site and Jackett will take care of everything else.

2.1.2 Adding Trackers - Sonarr

In Sonarr, go to Settings > Indexers and click the big plus button to start adding an indexer. Choose Torznab Custom and fill in the name of the indexer. For the URL field, click the Copy Torznab Feed button next to the indexer in Jackett. Paste that, then copy the API key in the top right of Jackett and paste that into the API Key field. The Categories field can be configured in either Jackett or Sonarr or both. If doing both, make sure the categories are both the same. Set minimum seeders (I recommend 5+, as some torrent sites fuzz their numbers when there’s little activity) and Save. Sonarr will test the connection to Jackett and the indexer, and will finish saving if everything works.
That’s all that needs to be done on the Jackett front!

2.2 Download Client - Sonarr

Adding Deluge to Sonarr is simple. Go to Settings > Download Clients and click the plus button again. Choose Deluge and fill in the details. If deluge is on the same hardware as Sonarr, the host is localhost, and the port is the deluge-web port (8112 default). The password is the password for the web login (if you have one (you should)).
Once everything is filled out, press save and Sonarr will test the connection. If successful, congratulations! You’re all set and can begin downloading media automatically. If Sonarr fails to connect to deluge, make sure the web UI is running (deluge-web) and can connect to the daemon (deluged). Also, it may be necessary to open the port. However, if running on the same hardware, it isn’t necessary to port forward the ports. Opening the port allows services on the hardware to see other services that have a port, while port forwarding opens the port to the entire internet.

By now, you should be able to: add a show/movie, search for a file, get results from multiple indexers, send a result to deluge, and catch the finished deluge download in Sonarr. In the next sections, I’ll go over what I think are good Sonarr settings, optional remote uploading, and how to further automate some processes.

3 . Tweaking some Settings

3.1 Sonarr

Here are my settings for Sonarr. Note that most of these are highly personal: they fit my media setup and how and where I personally want things to go. If there are settings I think should be set no matter what, I’ll explicitly say so. Also, have show advanced checked for additional settings.

Media Management

Rename episodes: Enabled - Definitely enable so your files aren’t named [YIFY] Over.The.Hedge.2006.720p.8bit [HVEC] and worse. Enabling this setting will show the Format fields, which can be named according to your preferences.
Replace Illegal Characters: Enabled - Don’t want little bobby tables breaking your OS… or something.
Import Extra Files: Enabled - I want subtitle files that aren’t embedded: srt, ass, sub.
Download Propers: Enabled
Analyse video files: Enabled
Rescan Series Folders after Refresh: Always
Change File Date: None

Profiles

Profiles are of course dependent not only on the minimum quality you can handle, but also on the type of media you’re going to download. Old shows and movies? Probably going to want 480p minimum, maybe lower to be safe. Newer content will almost always be available in 1080p+. Here’s my profile I use, which tries to get up to 1080p, and will accept anything 480p+ until it gets to 1080p.

Upgrades Allowed: Enabled
Upgrade Until: Bluray-1080p
Qualities: Everything from WEB 480p to Bluray-1080p Remux is checked, except for Raw-HD

Quality

Here’s for really fine-grained control over what qualifies each quality. I’d recommend changing everything to have a maximum size, instead of unlimited. Be generous, like 10GB max for 1080p. I have the maximums in place because Sonarr will often choose a 1080p file that is extremely high quality, like 15k+ kbps bitrate which ends up with a 20+ GB 1.5 hour movie. While it looks great, it’s (in my opinion) too large of a file for almost imperceptible differences to a 7.5k-10k kbps file.

Indexers

It should already be configured, except for the options underneath the indexers. I leave everything default except for the RSS Sync Interval, I have it at 60 minutes so I don’t get IP blocked by indexers. This is a super conservative number, but also I don’t need to check for new releases every 10 minutes (which you could most likely get away with).

Download Clients

Deluge should be configured. In the options, I’d recommend enabling everything for safety, even though later in this article I’ll be automatically removing downloads through a script and cron.

Connect

I don’t use this. This section is for notifying other services about the status of Sonarr. Think webhooks.

Metadata

I also don’t use this section, as I use Plex. This section generates metadata for files for specific media aggregators, like Kodi and can be useful.

Tags

I don’t use tags

General

Here you can change the port number (I leave mine default on 8989) and other security settings. I highly recommend setting a form of authentication for Sonarr if you are planning on having your Sonarr instance available on the web.

UI

These settings are region-specific and personal.

3.2 Radarr

As Radarr is a fork of Sonarr, there will be a lot of overlap in the settings. Really, the only significant difference between the two’s settings is the media management.

Media Management

Rename Movies: Enabled - Definitely don’t want the torrent-supplied filenames.
Replace Illegal Characters: Enabled
Standard Movie/Folder Format: I have mine very simple: {Movie Title} ({Release Year})
Automatically Rename Folders: Enabled
Analyse video files: Enabled
Again, change these settings as you want your media to be organized.

Everything else

Everything else is about the same as Sonarr or has already been set up (indexers and clients).

4 . Automation

This is the fun part. In this section, I’ll cover the scripts I use to automatically do tasks. One of the most useful ones I use is the Deluge watcher.

4.1 Deluge Watcher

The deluge daemon has CLI commands that can be run to query the status of Deluge. It also has the ability to modify torrents, such as adding and removing them. Instead of just posting my script, I’m going to go through the individual commands and explain them.\

deluge-console -c /home/vpn/.config/deluge info -s Downloading | gawk '/ID/,/Error/' | grep -C 5 "Seeds: 0" | grep -B 4 "Active: [1-9] days" | grep "ID" | sed 's/^....//' > downloadIds.txt

This is a meaty single-liner that probably shouldn’t exist.
The logic behind this command is simple: find torrents that are inactive and unlikely to download anything. Let’s take a look at what deluge-console info -s Downloading gives us, and narrow it down from there.

$ deluge-console -c /home/vpn/.config/deluge info -s Downloading
> Name: <The torrent's name>
> ID: a196f656c05e241f34159c712cb21edd34c7fa27
> State: Downloading Down Speed: 0.0 KiB/s Up Speed: 0.0 KiB/s
> Seeds: 0 (-1) Peers: 0 (-1) Availability: 0.00
> Size: 0.0 KiB/0.0 KiB Ratio: -1.000
> Seed time: 0 days 00:00:00 Active: 9 days 02:47:03
> Tracker status: indexer.com: Error: Operation canceled
> Progress: 0.00% [~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~]

Deluge uses a torrent’s ID to identify it. That’s what we use to do stuff to it, such as removing it from the queue. So now that we know the output, I’ll go through each piped command that narrows it down.

gawk '/ID/,/Error/'

Find the word ID, and select that line and following lines up to the word Error. If a torrent doesn’t have an error, it won’t be matched. Pipe the results.

grep -C 5 "Seeds: 0"

Search for the word “Seeds: 0”, and give 5 lines of Context around the matched word’s line. This returns the lines from ID: to Tracker status:. This selects only torrents with status error and seeds = 0, because sometimes a torrent with error status will still be able to complete a torrent. Pipe the results.

grep -B 4 "Active: [1-9] days"

Search for the word Active: [1-9] days. The [1-9] is a regex that will match any digit from 1-9, inclusive. This means that, now, torrents with error status, no seeders, and over 1 day in the queue will be matched. That’s all I want, and it has served me perfectly. Pipe the results.

grep "ID"

Take the ID from each torrent that fits the criteria above, and pipe them to:

sed 's/^....//' > downloadIds.txt

This is the most arcane-looking command. Breaking it down shows how simple it is:\

s/^....//
s          indicate this is a **s**earch function
 /         indicates that the following is the search context
  ^        select from the start of the line
   ....    match any character. Since it's repeated 4 times, it will match 4 characters
       /   indicates the end of the search context, and what follows is the replace context
        /  indicates the end of the replace context. Since it's empty, the selected text will be replaced with nothing.

I mostly just wanted to do the staggered breakdown like codegolfers do for this section :)

Anyway, in the end, the IDs of Deluge torrents that are unlikely to download anything are added to a file named downloadIds.txt.
Maybe downloadIds.txt is empty, as no torrents with errors? Check that with this:

if [ ! -s downloadIds.txt ]
then
  echo "downloadIds.txt is empty, nothing to do"
  rm downloadIds.txt 2> /dev/null
  exit
fi

If you find that rm doesn’t work, toss a -f flag to show just how much you mean it.

Sonarr can operate on Deluge ID’s, but they have to be uppercase for some reason:

tr a-z A-Z < downloadIds.txt > downloadIdsTEMP.txt
rm downloadIds.txt
mv downloadIdsTEMP.txt downloadIds.txt

tr, short for translate, takes (<) the downloadIds.txt in as input, turns a-z characters uppercase A-Z, then outputs to a new file downloadIdsTEMP.txt, because the input is read line-by-line, and you don’t want to cross the input/output streams!

Next, get the queue that Sonarr has with a cURL request (have your IP and API key at the ready)

curl -s -X GET -G "$IP/api/queue/" -d apikey=$APIKEY >> json.txt

-s for a silent request (no console output), -X specifies a request type, and is tied to -G for get. -d sends additional data with the request, and Sonarr requires the API key for authentication. The whole JSON request then gets piped into json.txt*.

Next, we use the Deluge IDs to match them with their respective Sonarr IDs.

while read downloadId; do
 awk -v id="$downloadId" '$0 ~ id { getline; getline; print }' json.txt | sed 's/^..........//' >> sonarrIds.txt
done < downloadIds.txt

I’m gonna be honest, I forget what this exactly does, except I remember being annoyed by setting variables inside awk. The -v flag sets a variable, which is used inside the awk “function”. The function searches for the given ID, then prints two lines after it for sed to deal with.
The sed command is the same as the one from before, but this time with more periods to match more unneeded characters in front of the sonarr ID.

Lastly, we take the Sonarr IDs we care about and tell Sonarr to delete them from its queue. Sonarr will also delete it from Deluge, making it pretty convenient for us, right? Easy? Right?

while read ID; do
 curl -s -X DELETE -G "$IP/api/queue/$ID" -d blacklist=true -d apikey=$APIKEY
 sleep 1 # just to be safe
done < sonarrIDs.txt

Similar to the previous cURL, but instead of a GET request we’re asking for Sonarr to DELETE something.

rm downloadIds.txt 2> /dev/null
rm sonarrIds.txt 2> /dev/null
rm json.txt 2> /dev/null

I lied about the lastly part earlier. But now this script is done for real. Congrats! If you made it this far, I’m sorry!

4.2 Watchdogs

First and foremost, I highly recommend not following anything in this section (or the rest of this page, I guess). I use screen for all my services, which is definitely not a good way to do it. Look up how to run processes with something like systemctl, which is installed by default on most/all Linux systems. It’s much more robust and reliable.

I have a watchdog.sh script that periodically (through cron) checks whether all my processes are running in screen. Here is one block for rclone:

GREP="/bin/grep"
SCREEN="/usr/bin/screen"
if ! $SCREEN -ls | $GREP -q "mount" ; then
 $SCREEN -dmS mount bash -c 'rclone mount drive: /mnt/drive --vfs-cache-mode off -v --retries 5 --retries-sleep=5s'
 echo "[$(date +%c)]: Rclone mount restarted" >> .watchdog.log
fi

The grep and screen commands have to be fully referenced if run from cron, as it doesn’t run from bash and have the PATH everyone’s used to.

5 Rclone Uploading

Finally. The last piece of the puzzle is (optionally!) uploading the content. If you are doing all of this locally, you may not even need to do this. But I send all of my media to Google Drive and have a rclone mount to handle it.

5.1 Rclone mount

The previous code in 4.2 Watchdogs is actually my rclone mount configuration. With that, rclone will make a virtual drive located at /mnt/drive which will mirror my Google Drive. When adding content to Sonarr, simply make sure the path is in the /mnt/drive area. When Sonarr finishes a download, the file will be moved to the rclone mount. Rclone will then handle uploading it to Drive, and that’s it! This is probably the easiest step. And best, because it’s the last one.

6 . Conclusion

So that’s it. Now, all I have to do when I want a show or movie is go to a very nice looking web interface and add it. Then, minutes later (thanks to a VPS having blazing fast speeds), the content is uploaded and ready to be streamed. My content aggregator of choice is Plex, and all I have to do is point Plex to where the media is located and everything else is taken care of.
Sure, it’s a really actually huge big deal to get all the backend stuff set up, but it’s mostly just set-and-forget at this point. All of the info here is the result of months of research and trial-and-error and wiping the VPS and restarting because I broke something. A lot of the extra stuff in here can be excluded, but I really wouldn’t skip out since I have figured out a decent solution for it already. And the extra little things make a big difference, especially the auto-remove of dead torrents.
This… article? I guess? It is the result of about a year of work and refinement. It’s been rewritten twice because I changed how I was managing my media, and I’m really happy I finally have it all written up. While my solutions aren’t perfect, it’s all learning. Take pieces of code from here and make yours better with your knowledge, because that’s what I did. And after about a year of fiddling with this, I’ve gone from a “Yeah, I can make my way around Linux” to “Yeah I’ve got a solid grasp on the basics of Linux, but it just keeps going - like seriously, how iptables?”
If you want to tell me this is dumb: email me at [email protected]