Admins eHow SysAdmin Tips & Tricks

openssl genrsa -out client.key 4096
openssl req -sha256 -out client.csr -key client.key -new -subj "/C=GB/ST=London/L=London/O=Global Security/OU=IT Department/CN=client"
openssl x509 -sha256 -req -days 365 -CA ca.crt -CAkey ca.key -in client.csr -set_serial 01 -out client.crt

In this post I am going to show you how to setup a system to download YouTube videos on a raspberry pi on a certain time of the day !
I know this is a weird case of usage, but if your internet speed is low and cant watch YouTube videos directly or your daily internet traffic is limited, it may be useful. it can download your favorite YouTube videos for you when you are sleep !
This is actually not a simple system and I am not going through all of the details because the post will become very long and I am lazy :p. I will provide the information which you can not find anywhere else, other steps can be found on other websites.
So here is our design :

Youtube -> Chrome extension -> API (PHP file) on raspberry pi -> a file (/etc/youtube) containing YouTube links
Cronjob 1 -> Process /etc/youtube -> Get download links -> Aria2 (paused mode)
Cronjob 2 -> Start Aria2

Lets start with chrome extension, it is a very simple extension and consists of 2 files.
You can download the extension source by this link : youtube-chrome-ext download
Unzip this file and open sample.js
On line 15 you will see this :

client.get("http://192.168.101.1/ydl.php?url=" + info.linkUrl, function(response) {});

Change 192.168.101.1 to your own raspberry pi IP address.
Now open chrome extensions page chrome://extensions/ and enable developer mode. “Load unpacked extension” button will appear, click on it and browse to extension folder and select it. it will install the extension inside chrome.
Now if you click on any link inside chrome, you would see a new option called “YouTube Downloader”, clicking on it will send the link to our raspberry pi API which we will implement in next step.

Now lets create our PHP API file, needless to say you need to have a web server and PHP installed on your raspberry pi.
Create a file named ydl.php in /var/www/html folder with the following source :

<?php
header('Access-Control-Allow-Origin: *');
$url=$_GET["url"]."\n";
$file = '/etc/youtube';
$current = file_get_contents($file);
$current .= $url;
file_put_contents($file, $current);
?>

As you can see this is a very simple API. it appends the YouTube links which are sent by our chrome extension to a file named /etc/youtube.
As this file does not exist at the first time, lets create it and give it proper permissions. run following commands on raspberry pi :

touch /etc/youtube
chmod 666 /etc/youtube

Now it is time to test our API, open YouTube website, right click on several videos and choose “YouTube Downloader” then check the contents of /etc/youtube on raspberry pi, the links should be there.

Next step is to create the scripts which process /etc/youtube file and send the download links to Aria2.
Create the following files with their respective sources :
/usr/bin/process_youtube :

#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
        /usr/bin/a2youtube.py $line
done < /etc/youtube

rm /etc/youtube.old
mv /etc/youtube /etc/youtube.old
touch /etc/youtube
chmod 666 /etc/youtube

/usr/bin/a2youtube.py :

#!/usr/bin/python
import xmlrpclib,sys,commands
out=commands.getoutput("/usr/local/bin/youtube-dl -f 'best' -g -e --get-id "+sys.argv[1])
s = xmlrpclib.ServerProxy('http://localhost:6800/rpc')
if (len(out.splitlines()[0].strip())<10):
        fn=out.splitlines()[1]
else:
        fn=out.splitlines()[0].strip()

s.aria2.addUri("token:XXXX",[out.splitlines()[2]],dict(out=fn+".mp4",pause="true"))

set proper permissions for both files :

chmod 755 /usr/bin/process_youtube
chmod 755 /usr/bin/a2youtube.py

As you can see we will be using python for second script. so you need to have python installed as well.
There is also another program which is responsible to get the download link for us named youtube-dl.
You should install the latest version from this link : https://rg3.github.io/youtube-dl/

The reason that I chose to use Aria2 is that it is a VERY good and flexible download manager, better than anything else that you can find on Windows or Mac hands down so I highly recommend it. You need to install Aria2 as well : https://aria2.github.io/
here is my aria2 config file :

dir=/media
file-allocation=falloc
continue=true
log-level=notice
check-certificate=false
max-connection-per-server=16
split=16
summary-interval=120
daemon=true
enable-rpc=true
enable-dht=true
max-concurrent-downloads=2
http-auth-challenge=true
log=/var/log/aria2/aria2.log
disable-ipv6=true
disk-cache=25M
timeout=600
retry-wait=30
max-tries=50
save-session=/home/pi/session.gz
input-file=/home/pi/session.gz
seed-time=0
min-split-size=1M
rpc-secret=XXXX
rpc-listen-port=6800
rpc-listen-all=true

Pay attention to last 3 lines of config specially rpc-secret. it is a token that other programs will use to communicate with aria2 daemon. so change XXXX to a password of your choosing. also notice the “token:XXXX” in the /usr/bin/a2youtube.py file. change XXXX to the password that you set in aria2 config file.
You can (should) also install a web user interface for Aria2 from this link : https://github.com/ziahamza/webui-aria2
The webui will act as GUI for aria2 in your web browser so you can see what it is doing and control it as u wish.

If you pay attention to the python code you would see that it adds the links in paused mode :

s.aria2.addUri("token:XXXX",[out.splitlines()[2]],dict(out=fn+".mp4",pause="true"))

The reason is that if we start to download immediately, youtube-dl may fail to get other links from YouTube website because your download bandwidth is full (thats the point).
so we need 2 more scripts to start/stop Aria2 :
/usr/bin/a2stop.py :

#!/usr/bin/python

import xmlrpclib
s = xmlrpclib.ServerProxy('http://localhost:6800/rpc')
s.aria2.pauseAll("token:XXXX")

/usr/bin/a2start.py :

#!/usr/bin/python

import xmlrpclib
s = xmlrpclib.ServerProxy('http://localhost:6800/rpc')
s.aria2.unpauseAll("token:XXXX")

dont forget to set proper permissions :

chmod 755 /usr/bin/a2start.py
chmod 755 /usr/bin/a2stop.py

and change XXXX to the secret that you set in aria2 config file.
now you can create the cronjobs :

2 3 * * * /usr/bin/process_youtube
5 3 * * * /usr/bin/a2start.py
55 8 * * * /usr/bin/a2stop.py

It will add YouTube links to aria2 @ 3:02 AM
Starts all downloads in aria2 @ 3:05 AM
Pauses all downloads in aria2 @ 8:55 AM

Good luck on implementing this system, it is not easy. But you will learn a lot if you try and you are persistent.

https://github.com/cxcv/iops