Making it easy to ssh into a remote server

Logging into a remote server is a drag. Needing to remember the password (or get it from 1Password); needing to remember the IP address of the remote server. Ugh.

It’d be so much easier if I could just

ssh username@servername

and get into the server.

And it turns out, you can. You just need to do two simple things.

Simple thing the first: Update the hosts file on your local computer to map the IP address to a memorable name.

The hosts file is located at /etc/hosts (at least on *nix based systems).

Go to the hosts file in your favorite editor … my current favorite editor for simple stuff like this is vim.

Once there, add the IP address you don’t want to have to remember, and then a name that you will remember. For example:

67.176.220.115    easytoremembername

One thing to keep in mind, you’ll already have some entries in this file. Don’t mess with them. Leave them there. Seriously … it’ll be better for everyone if you do.

Simple thing the second: Generate a public-private key and share the public key with the remote server

From the terminal run the command ssh-keyken -t rsa. This will generate a public and private key. You will be asked for a location to save the keys to. The default (on MacOS) is /Users/username/.ssh/id_rsa. I tend to accept the default (no reason not to) and leave the passphrase blank (this means you won’t have to enter a password which is what we’re looking for in the first place!)

Next, we copy the public key to the host(s) you want to access using the command

ssh-copy-id <username>@<hostname>

for example:

ssh-copy-id pi@rpicamera

The first time you do this you will get a message asking you if you’re sure you want to do this. Type in yes and you’re good to go.

One thing to note, doing this updates the file known_hosts. If, for some reason, the server you are ssh-ing to needs to be rebuilt (i.e. you have to keep destroying your Digital Ocean Ubuntu server because you can’t get the static files to be served properly for your Django project) then you need to go to the known_hosts file and remove the entry for that known host.

When you do that you’ll be asked about the identity of the server (again). Just say yes and you’re good to go.

If you forget that step then when you try to ssh into the server you get a nasty looking error message saying that the server identities don’t match and you can’t proceed.

Automating the Hummingbird Video Upload to YouTube or How I finally got Cron to do what I needed it to do but in the ugliest way possible

Several weeks ago in Cronjob Redux I wrote that I had finally gotten Cron to automate the entire process of compiling the h264 files into an mp4 and uploading it to YouTube.

I hadn’t. And it took the better part of the last 2 weeks to figure out what the heck was going on.

Part of what I wrote before was correct. I wasn’t able to read the client_secrets.json file and that was leading to an error.

I was not correct on the creation of the create_mp4.sh though.

The reason I got it to run automatically that night was because I had, in my testing, created the create_mp4.sh and when cron ran my run_script.sh it was able to use what was already there.

The next night when it ran, the create_mp4.sh was already there, but the h264 files that were referenced in it weren’t. This lead to no video being uploaded and me being confused.

The issue was that cron was unable to run the part of the script that generates the script to create the mp4 file.

I’m close to having a fix for that, but for now I did the most inelegant thing possible. I broke up the script in cron so it looks like this:

00 06 * * * /home/pi/Documents/python_projects/cleanup.sh
10 19 * * * /home/pi/Documents/python_projects/create_script_01.sh
11 19 * * * /home/pi/Documents/python_projects/create_script_02.sh >> $HOME/Documents/python_projects/create_mp4.sh 2>&1
12 19 * * * /home/pi/Documents/python_projects/create_script_03.sh
13 19 * * * /home/pi/Documents/python_projects/run_script.sh

At 6am every morning the cleanup.sh runs and removes the h264 files, the mp4 file and the create_mp4.sh script

At 7:10pm the ‘header’ for the create_mp4.sh runs. At 7:11pm the ‘body’ for create_mp4.sh runs. At 7:12pm the ‘footer’ for create_mp4.sh runs.

Finally at 7:13pm the run_script.sh compiles the h264 files into an mp4 and uploads it to YouTube.

Last night while I was at a School Board meeting the whole process ran on it’s own. I was super pumped when I checked my YouTube channel and saw that the May 1 hummingbird video was there and I didn’t have to do anything.

Cronjob Redux

After days of trying to figure this out, I finally got the video to upload via a cronjob.

There were 2 issues.

Issue the first

Finally found the issue. Original script from YouTube developers guidehad this:

CLIENT_SECRETS_FILE = "client_secrets.json"

And then a couple of lines later, this:

% os.path.abspath(os.path.join(os.path.dirname(__file__), CLIENT_SECRETS_FILE))

When crontab would run the script it would run from a path that wasn’t where the CLIENT_SECRETS_FILE file was and so a message would be displayed:

WARNING: Please configure OAuth 2.0
To make this sample run you will need to populate the client_secrets.json file
found at:

  %s

with information from the Developers Console
https://console.developers.google.com/

For more information about the client_secrets.json file format, please visit:
https://developers.google.com/api-client-library/python/guide/aaa_client_secrets

What I needed to do was to update the CLIENT_SECRETS_FILE to be the whole path so that it could always find the file.

A simple change:

CLIENT_SECRETS_FILE  = os.path.abspath(os.path.join(os.path.dirname(__file__), CLIENT_SECRETS_FILE))

Issue the second

When the create_mp4.sh script would run it was reading all of the h264 files from the directory where they lived BUT they were attempting to output the mp4 file to / which it didn’t have permission to write to.

This was failing silently (I’m still not sure how I could have caught the error). Since there was no mp4 file to upload that script was failing (though it was true that the location of the CLIENT_SECRETS_FILE was an issue).

What I needed to do was change the create_mp4.sh file so that when the MP4Box command output the mp4 file to the proper directory. The script went from this:

(echo '#!/bin/sh'; echo -n "MP4Box"; array=($(ls ~/Documents/python_projects/*.h264)); for index in ${!array[@]}; do if [ "$index" -eq 0 ]; then echo -n " -add ${array[index]}"; else echo -n " -cat ${array[index]}"; fi; done; echo -n " hummingbird.mp4") > create_mp4.sh

To this:

(echo '#!/bin/sh'; echo -n "MP4Box"; array=($(ls ~/Documents/python_projects/*.h264)); for index in ${!array[@]}; do if [ "$index" -eq 0 ]; then echo -n " -add ${array[index]}"; else echo -n " -cat ${array[index]}"; fi; done; echo -n " /home/pi/Documents/python_projects/hummingbird.mp4") > /home/pi/Documents/python_projects/create_mp4.sh

The last bit /home/pi/Documents/python_projects/create_mp4.sh may not be neccesary but I’m not taking any chances.

The video posted tonight is the first one that was completely automatic!

Now … if I could just figure out how to automatically fill up my hummingbird feeder.

Cronjob … Finally

I’ve mentioned before that I have been working on getting the hummingbird video upload automated.

Each time I thought I had it, and each time I was wrong.

For some reason I could run it from the command line without issue, but when the cronjob would try and run it … nothing.

Turns out, it was running, it just wasn’t doing anything. And that was my fault.

The file I had setup in cronjob was called run_scrip.sh

At first I was confused because the script was suppose to be writing out to a log file all of it’s activities. But it didn’t appear to.

Then I noticed that the log.txt file it was writing was in the main \` directory. That should have been my first clue.

I kept trying to get the script to run, but suddenly, in a blaze of glory, realized that it was running, it just wasn’t doing anything.

And it wasn’t doing anything for the same reason that the log file was being written to the \` directory.

All of the paths were relative instead of absolute, so when the script ran the command ./create_mp4.sh it looks for that script in the home directory, didn’t find it, and moved on.

The fix was simple enough, just add absolute paths and we’re golden.

That means my run_script.sh goes from this:

# Create the script that will be run
./create_script.sh
echo "Create Shell Script: $(date)" >> log.txt

# make the script that was just created executable
chmod +x /home/pi/Documents/python_projects/create_mp4.sh

# Create the script to create the mp4 file
/home/pi/Documents/python_projects/create_mp4.sh
echo "Create MP4 Shell Script: $(date)" >> /home/pi/Documents/python_projects/log.txt

# upload video to YouTube.com
/home/pi/Documents/python_projects/upload.sh
echo "Uploaded Video to YouTube.com: $(date)" >> /home/pi/Documents/python_projects/log.txt

# Next we remove the video files locally
rm /home/pi/Documents/python_projects/*.h264
echo "removed h264 files: $(date)" >> /home/pi/Documents/python_projects/log.txt

rm /home/pi/Documents/python_projects/*.mp4
echo "removed mp4 file: $(date)" >> /home/pi/Documents/python_projects/log.txt

To this:

# change to the directory with all of the files
cd /home/pi/Documents/python_projects/

# Create the script that will be run
/home/pi/Documents/python_projects/create_script.sh
echo "Create Shell Script: $(date)" >> /home/pi/Documents/python_projects/log.txt

# make the script that was just created executable
chmod +x /home/pi/Documents/python_projects/create_mp4.sh

# Create the script to create the mp4 file
/home/pi/Documents/python_projects/create_mp4.sh
echo "Create MP4 Shell Script: $(date)" >> /home/pi/Documents/python_projects/log.txt

# upload video to YouTube.com
/home/pi/Documents/python_projects/upload.sh
echo "Uploaded Video to YouTube.com: $(date)" >> /home/pi/Documents/python_projects/log.txt

# Next we remove the video files locally
rm /home/pi/Documents/python_projects/*.h264
echo "removed h264 files: $(date)" >> /home/pi/Documents/python_projects/log.txt

rm /home/pi/Documents/python_projects/*.mp4
echo "removed mp4 file: $(date)" >> /home/pi/Documents/python_projects/log.txt

I made this change and then started getting an error about not being able to access a json file necessary for the upload to YouTube. Sigh.

Then while searching for what directory the cronjob was running from I found this very simple idea. The response was, why not just change it to the directory you want. 🤦‍♂️

I added the cd to the top of the file:

# change to the directory with all of the files
cd /home/pi/Documents/python_projects/

Anyway, now it works. Finally!

Tomorrow will be the first time (unless of course something else goes wrong) that The entire process will be automated. Super pumped!