Setting up Jupyter Notebook on my Linode

A Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text.

Uses include:

  1. data cleaning and transformation
  2. numerical simulation
  3. statistical modeling
  4. data visualization
  5. machine learning
  6. and other stuff

I’ve been interested in how to set up a Jupyter Notebook on my Linode server for a while, but kept running into a roadblock (either mental or technical I’m not really sure).

Then I came across this ‘sweet’ solution to get them set up athttp://blog.lerner.co.il/five-minute-guide-setting-jupyter-notebook-server/

My main issue was what I needed to to do keep the Jupyter Notebook running once I disconnected from command line. The solution above gave me what I needed to solve that problem

nohup jupyter notebook

nohup allows you to disconnect from the terminal but keeps the command running in the background (which is exactly what I wanted).

The next thing I wanted to do was to have the jupyter notebook server run from a directory that wasn’t my home directory.

To do this was way easier than I thought. You just run nohup jupyter notebook from the directory you want to run it from.

The last thing to do was to make sure that the notebook would start up with a server reboot. For that I wrote a shell script

# change to correct directory
cd /home/ryan/jupyter

nohup jupyter notebook &> /home/ryan/output.log

The last command is a slight modification of the line from above. I really wanted the output to get directed to a file that wasn’t in the directory that the Jupyter notebook would be running from. Not any reason (that I know of anyway) … I just didn’t like the nohup.out file in the working directory.

Anyway, I now have a running Jupyter Notebook at http://python.ryancheley.com:88881

  1. I’d like to update this to be running from a port other than 8888 AND I’d like to have it on SSL, but one thing at a time!

Cronjob Redux

After days of trying to figure this out, I finally got the video to upload via a cronjob.

There were 2 issues.

Issue the first

Finally found the issue. Original script from YouTube developers guidehad this:

CLIENT_SECRETS_FILE = "client_secrets.json"

And then a couple of lines later, this:

% os.path.abspath(os.path.join(os.path.dirname(__file__), CLIENT_SECRETS_FILE))

When crontab would run the script it would run from a path that wasn’t where the CLIENT_SECRETS_FILE file was and so a message would be displayed:

WARNING: Please configure OAuth 2.0
To make this sample run you will need to populate the client_secrets.json file
found at:

  %s

with information from the Developers Console
https://console.developers.google.com/

For more information about the client_secrets.json file format, please visit:
https://developers.google.com/api-client-library/python/guide/aaa_client_secrets

What I needed to do was to update the CLIENT_SECRETS_FILE to be the whole path so that it could always find the file.

A simple change:

CLIENT_SECRETS_FILE  = os.path.abspath(os.path.join(os.path.dirname(__file__), CLIENT_SECRETS_FILE))

Issue the second

When the create_mp4.sh script would run it was reading all of the h264 files from the directory where they lived BUT they were attempting to output the mp4 file to / which it didn’t have permission to write to.

This was failing silently (I’m still not sure how I could have caught the error). Since there was no mp4 file to upload that script was failing (though it was true that the location of the CLIENT_SECRETS_FILE was an issue).

What I needed to do was change the create_mp4.sh file so that when the MP4Box command output the mp4 file to the proper directory. The script went from this:

(echo '#!/bin/sh'; echo -n "MP4Box"; array=($(ls ~/Documents/python_projects/*.h264)); for index in ${!array[@]}; do if [ "$index" -eq 0 ]; then echo -n " -add ${array[index]}"; else echo -n " -cat ${array[index]}"; fi; done; echo -n " hummingbird.mp4") > create_mp4.sh

To this:

(echo '#!/bin/sh'; echo -n "MP4Box"; array=($(ls ~/Documents/python_projects/*.h264)); for index in ${!array[@]}; do if [ "$index" -eq 0 ]; then echo -n " -add ${array[index]}"; else echo -n " -cat ${array[index]}"; fi; done; echo -n " /home/pi/Documents/python_projects/hummingbird.mp4") > /home/pi/Documents/python_projects/create_mp4.sh

The last bit /home/pi/Documents/python_projects/create_mp4.sh may not be neccesary but I’m not taking any chances.

The video posted tonight is the first one that was completely automatic!

Now … if I could just figure out how to automatically fill up my hummingbird feeder.

Cronjob … Finally

I’ve mentioned before that I have been working on getting the hummingbird video upload automated.

Each time I thought I had it, and each time I was wrong.

For some reason I could run it from the command line without issue, but when the cronjob would try and run it … nothing.

Turns out, it was running, it just wasn’t doing anything. And that was my fault.

The file I had setup in cronjob was called run_scrip.sh

At first I was confused because the script was suppose to be writing out to a log file all of it’s activities. But it didn’t appear to.

Then I noticed that the log.txt file it was writing was in the main \` directory. That should have been my first clue.

I kept trying to get the script to run, but suddenly, in a blaze of glory, realized that it was running, it just wasn’t doing anything.

And it wasn’t doing anything for the same reason that the log file was being written to the \` directory.

All of the paths were relative instead of absolute, so when the script ran the command ./create_mp4.sh it looks for that script in the home directory, didn’t find it, and moved on.

The fix was simple enough, just add absolute paths and we’re golden.

That means my run_script.sh goes from this:

# Create the script that will be run
./create_script.sh
echo "Create Shell Script: $(date)" >> log.txt

# make the script that was just created executable
chmod +x /home/pi/Documents/python_projects/create_mp4.sh

# Create the script to create the mp4 file
/home/pi/Documents/python_projects/create_mp4.sh
echo "Create MP4 Shell Script: $(date)" >> /home/pi/Documents/python_projects/log.txt

# upload video to YouTube.com
/home/pi/Documents/python_projects/upload.sh
echo "Uploaded Video to YouTube.com: $(date)" >> /home/pi/Documents/python_projects/log.txt

# Next we remove the video files locally
rm /home/pi/Documents/python_projects/*.h264
echo "removed h264 files: $(date)" >> /home/pi/Documents/python_projects/log.txt

rm /home/pi/Documents/python_projects/*.mp4
echo "removed mp4 file: $(date)" >> /home/pi/Documents/python_projects/log.txt

To this:

# change to the directory with all of the files
cd /home/pi/Documents/python_projects/

# Create the script that will be run
/home/pi/Documents/python_projects/create_script.sh
echo "Create Shell Script: $(date)" >> /home/pi/Documents/python_projects/log.txt

# make the script that was just created executable
chmod +x /home/pi/Documents/python_projects/create_mp4.sh

# Create the script to create the mp4 file
/home/pi/Documents/python_projects/create_mp4.sh
echo "Create MP4 Shell Script: $(date)" >> /home/pi/Documents/python_projects/log.txt

# upload video to YouTube.com
/home/pi/Documents/python_projects/upload.sh
echo "Uploaded Video to YouTube.com: $(date)" >> /home/pi/Documents/python_projects/log.txt

# Next we remove the video files locally
rm /home/pi/Documents/python_projects/*.h264
echo "removed h264 files: $(date)" >> /home/pi/Documents/python_projects/log.txt

rm /home/pi/Documents/python_projects/*.mp4
echo "removed mp4 file: $(date)" >> /home/pi/Documents/python_projects/log.txt

I made this change and then started getting an error about not being able to access a json file necessary for the upload to YouTube. Sigh.

Then while searching for what directory the cronjob was running from I found this very simple idea. The response was, why not just change it to the directory you want. 🤦‍♂️

I added the cd to the top of the file:

# change to the directory with all of the files
cd /home/pi/Documents/python_projects/

Anyway, now it works. Finally!

Tomorrow will be the first time (unless of course something else goes wrong) that The entire process will be automated. Super pumped!

Hummingbird Video Capture

I previously wrote about how I placed my Raspberry Pi above my hummingbird feeder and added a camera to it to capture video.

Well, the day has finally come where I’ve been able to put my video of it up on YouTube! It’s totally silly, but it was satisfying getting it out there for everyone to watch and see.

Hummingbird Video Capture: Addendum

The code used to generate the the mp4 file haven’t changed (really). I did do a couple of things to make it a little easier though.

I have 2 scripts that generate the file and then copy it from the pi to my MacBook Pro and the clean up:

Script 1 is called create_script.sh and looks like this:

(echo '#!/bin/sh'; echo -n "MP4Box"; array=($(ls *.h264)); for index in ${!array[@]}; do if [ "$index" -eq 0 ]; then echo -n " -add ${array[index]}"; else echo -n " -cat ${array[index]}"; fi; done; echo -n " hummingbird.mp4") > create_mp4.sh | chmod +x create_mp4.sh

This creates a script called create_mp4.sh and makes it executable.

This script is called by another script called run_script.sh and looks like this:

./create_script.sh
./create_mp4.sh

scp hummingbird.mp4 ryan@192.168.1.209:/Users/ryan/Desktop/

# Next we remove the video files locally

rm *.h264
rm *.mp4

It runs the create_script.sh which creates create_mpr.sh and then runs it.

Then I use the scp command to copy the mp4 file that was just created over to my Mac Book Pro.

As a last bit of housekeeping I clean up the video files.

I’ve added this run_script.sh to a cron job that is scheduled to run every night at midnight.

We’ll see how well it runs tomorrow night!

Daylight Savings Time

Dr Drang has posted on Daylight Savings in the past, but in a recent post he critiqued (rightly so) the data presentation by a journalist at the Washington Post on Daylight Savings, and that got me thinking.

In the post he generated a chart showing both the total number of daylight hours and the sunrise / sunset times in Chicago. However, initially he didn’t post the code on how he generated it. The next day, in a follow up post, he did and that really got my thinking.

I wonder what the chart would look like for cities up and down the west coast (say from San Diego, CA to Seattle WA)?

Drang’s post had all of the code necessary to generate the graph, but for the data munging, he indicated:

If I were going to do this sort of thing on a regular basis, I’d write a script to handle this editing, but for a one-off I just did it “by hand.”

Doing it by hand wasn’t going to work for me if I was going to do several cities and so I needed to write a parser for the source of the data (The US Naval Observatory).

The entire script is on my GitHub sunrisesunset repo. I won’t go into the nitty gritty details, but I will call out a couple of things that I discovered during the development process.

Writing a parser is hard. Like really hard. Each time I thought I had it, I didn’t. I was finally able to get the parser to work o cities with 01, 29,30, or 31 in their longitude / latitude combinations.

I generated the same graph as Dr. Drang for the following cities:

  • Phoenix, AZ
  • Eugene, OR
  • Portland
  • Salem, OR
  • Seaside, OR
  • Eureka, CA
  • Indio, CA
  • Long Beach, CA
  • Monterey, CA
  • San Diego, CA
  • San Francisco, CA
  • San Luis Obispo, CA
  • Ventura, CA
  • Ferndale, WA
  • Olympia, WA
  • Seattle, WA

Why did I pick a city in Arizona? They don’t do Daylight Savings and I wanted to have a comparison of what it’s like for them!

The charts in latitude order (from south to north) are below:

San Diego

Phoenix

Indio

Long Beach

Ventura

San Luis Obispo

Monterey

San Francisco

Eureka

Eugene

Salem

Portland

Seaside

Olympia

Seattle

Ferndale

While these images do show the different impact of Daylight Savings, I think the images are more compelling when shown as a GIF:

We see just how different the impacts of DST are on each city depending on their latitude.

One of Dr. Drang’s main points in support of DST is:

If, by the way, you think the solution is to stay on DST throughout the year, I can only tell you that we tried that back in the 70s and it didn’t turn out well. Sunrise here in Chicago was after 8:00 am, which put school children out on the street at bus stops before dawn in the dead of winter. It was the same on the East Coast. Nobody liked that.

I think that comment says more about our school system and less about the need for DST.

For this whole argument I’m way more on the side of CGP Grey who does a great job of explaining what Day Lights Time is.

I think we may want to start looking at a Universal Planetary time (say UTC) and base all activities on that regardless of where you are in the world. The only reason 5am seems early (to some people) is because we’ve collectively decided that 5am (depending on the time of the year) is either WAY before sunrise or just a bit before sunrise, but really it’s just a number.

If we used UTC in California (where I’m at) 5am would we 12pm. Normally 12pm would be lunch time, but that’s only a convention that we have constructed. It could just as easily be the crack of dawn as it could be lunch time.

Do I think a conversion like this will ever happen? No. I just really hope that at some point in the distant future when aliens finally come and visit us, we aren’t late (or them early) because we have such a wacky time system here.

Setting up ITFDB with a voice

In a previous post I wrote about my Raspberry Pi experiment to have the SenseHat display a scrolling message 10 minutes before game time.

One of the things I have wanted to do since then is have Vin Scully’s voice come from a speaker and say those five magical words, It's time for Dodger Baseball!

I found a clip of Vin on Youtube saying that (and a little more). I wasn’t sure how to get the audio from that YouTube clip though.

After a bit of googling1 I found a command line tool called youtube-dl. The tool allowed me to download the video as an mp4 with one simple command:

youtube-dl https://www.youtube.com/watch?v=4KwFuGtGU6c

Once the mp4 was downloaded I needed to extract the audio from the mp4 file. Fortunately, ffmpeg is a tool for just this type of exercise!

I modified this answer from StackOverflow to meet my needs

ffmpeg -i dodger_baseball.mp4 -ss 00:00:10 -t 00:00:9.0 -q:a 0 -vn -acodec copy dodger_baseball.aac

This got me an aac file, but I was going to need an mp3 to use in my Python script.

Next, I used a modified version of this suggestion to create write my own command

ffmpeg -i dodger_baseball.aac -c:a libmp3lame -ac 2 -b:a 190k dodger_baseball.mp3

I could have probably combined these two steps, but … meh.

OK. Now I have the famous Vin Scully saying the best five words on the planet.

All that’s left to do is update the python script to play it. Using guidance from here I updated my itfdb.py file from this:

if month_diff == 0 and day_diff == 0 and hour_diff == 0 and 0 >= minute_diff >= -10:
   message = '#ITFDB!!! The Dodgers will be playing {} at {}'.format(game.game_opponent, game.game_time)
   sense.show_message(message, scroll_speed=0.05)

To this:

if month_diff == 0 and day_diff == 0 and hour_diff == 0 and 0 >= minute_diff >= -10:
   message = '#ITFDB!!! The Dodgers will be playing {} at {}'.format(game.game_opponent, game.game_time)
   sense.show_message(message, scroll_speed=0.05)
   os.system("omxplayer -b /home/pi/Documents/python_projects/itfdb/dodger_baseball.mp3")

However, what that does is play Vin’s silky smooth voice every minute for 10 minutes before game time. Music to my ears but my daughter was not a fan, and even my wife who LOVES Vin asked me to change it.

One final tweak, and now it only plays at 5 minutes before game time and 1 minute before game time:

if month_diff == 0 and day_diff == 0 and hour_diff == 0 and 0 >= minute_diff >= -10:
   message = '#ITFDB!!! The Dodgers will be playing {} at {}'.format(game.game_opponent, game.game_time)
   sense.show_message(message, scroll_speed=0.05)

if month_diff == 0 and day_diff == 0 and hour_diff == 0 and (minute_diff == -1 or minute_diff == -5):
   os.system("omxplayer -b /home/pi/Documents/python_projects/itfdb/dodger_baseball.mp3")

Now, for the rest of the season, even though Vin isn’t calling the games, I’ll get to hear his voice letting me know, “It’s Time for Dodger Baseball!!!”

  1. Actually, it was an embarrassing amount

Fixing the Python 3 Problem on my Raspberry Pi

In my last post I indicated that I may need to

reinstalling everything on the Pi and starting from scratch

While speaking about my issues with pip3 and python3. Turns out that the fix was easier than I though. I checked to see what where pip3 and python3 where being executed from by running the which command.

The which pip3 returned /usr/local/bin/pip3 while which python3 returned /usr/local/bin/python3. This is exactly what was causing my problem.

To verify what version of python was running, I checked python3 --version and it returned 3.6.0.

To fix it I just ran these commands to unlink the new, broken versions:

sudo unlink /usr/local/bin/pip3

And

sudo unlink /usr/local/bin/python3

I found this answer on StackOverflow and tweaked it slightly for my needs.

Now, when I run python --version I get 3.4.2 instead of 3.6.0

Unfortunately I didn’t think to run the --version flag on pip before and after the change, and I’m hesitant to do it now as it’s back to working.

ITFDB!!!

My wife and I love baseball season. Specifically we love the Dodgers and we can’t wait for Spring Training to begin. In fact, today pitchers and catchers report!

I’ve wanted to do something with the Raspberry Pi Sense Hat that I got (since I got it) but I’ve struggled to find anything useful. And then I remembered baseball season and I thought, ‘Hey, what if I wrote something to have the Sense Hat say “#ITFDB” starting 10 minutes before a Dodgers game started?’

And so I did!

The script itself is relatively straight forward. It reads a csv file and checks to see if the current time in California is within 10 minutes of start time of the game. If it is, then it will send a show_message command to the Sense Hat.

I also wrote a cron job to run the script every minute so that I get a beautiful scrolling bit of text every minute before the Dodgers start!

The code can be found on my GitHub page in the itfdb repository. There are 3 files:

  1. Program.py which does the actual running of the script
  2. data_types.py which defines a class used in Program.py
  3. schedule.csv which is the schedule of the games for 2018 as a csv file.

I ran into a couple of issues along the way. First, my development environment on my Mac Book Pro was Python 3.6.4 while the Production Environment on the Raspberry Pi was 3.4. This made it so that the code about time ran locally but not on the server 🤦‍♂️.

It took some playing with the code, but I was finally able to go from this (which worked on 3.6 but not on 3.4):

now = utc_now.astimezone(pytz.timezone("America/Los_Angeles"))
game_date_time = game_date_time.astimezone(pytz.timezone("America/Los_Angeles"))

To this which worked on both:

local_tz = pytz.timezone('America/Los_Angeles')
now = utc_now.astimezone(local_tz)
game_date_time = local_tz.localize(game_date_time)

For both, the game_date_time variable setting was done in a for loop.

Another issue I ran into was being able to display the message for the sense hat on my Mac Book Pro. I wasn’t ever able to because of a package that is missing (RTIMU ) and is apparently only available on Raspbian (the OS on the Pi).

Finally, in my attempts to get the code I wrote locally to work on the Pi I decided to install Python 3.6.0 on the Pi (while 3.4 was installed) and seemed to do nothing but break pip. It looks like I’ll be learning how to uninstall Python 3.4 OR reinstalling everything on the Pi and starting from scratch. Oh well … at least it’s just a Pi and not a real server.

Although, I’m pretty sure I hosed my Linode server a while back and basically did the same thing so maybe it’s just what I do with servers when I’m learning.

One final thing. While sitting in the living room watching DC Legends of Tomorrow the Sense Hat started to display the message. Turns out, I was accounting for the minute, hour, and day but NOT the month. The Dodgers play the Cubs on September 12 at 9:35 (according to the schedule.csv file anyway) and so the conditions to display were met.

I added another condition to make sure it was the right month and now we’re good to go!

Super pumped for this season with the Dodgers!

Using MP4Box to concatenate many .h264 files into one MP4 file: revisited

In my last post I wrote out the steps that I was going to use to turn a ton of .h264 files into one mp4 file with the use of MP4Box.

Before outlining my steps I said, “The method below works but I’m sure that there is a better way to do it.”

Shortly after posting that I decided to find that better way. Turns out, it wasn’t really that much more work it was much harder than originally thought.

The command below is a single line and it will create a text file (com.txt) and then execute it as a bash script:

(echo '#!/bin/sh'; for i in *.h264; do if [ "$i" -eq 1 ]; then echo -n " -add $i"; else echo -n " -cat $i"; fi; done; echo -n " hummingbird.mp4") > /Desktop/com.txt | chmod +x /Desktop/com.txt | ~/Desktop/com.txt

(echo '#!/bin/sh'; echo -n "MP4Box"; array=($(ls *.h264)); for index in ${!array[@]}; do if [ "$index" -eq 1 ]; then echo -n " -add ${array[index]}"; else echo -n " -cat ${array[index]}"; fi; done; echo -n " hummingbird.mp4") > com.txt | chmod +x com.txt

Next you execute the script with

./com.txt

OK, but what is it doing? The parentheses surround a set of echo commands that output to com.txt. I’m using a for loop with an if statement. The reason I can’t do a straight for loop is because the first h264 file used in MP4Box needs to have the -add flag while all of the others need the -cat flag.

Once the file is output to the com.txt file (on the Desktop) I pipe it to the chmod +x command to change it’s mode to make it executable.

Finally, I pipe that to a command to run the file ~/Desktop/com.txt

I was pretty stoked when I figured it out and was able to get it to run.

The next step will be to use it for the hundreds of h264 files that will be output from my hummingbird camera that I just installed today.

I’ll have a post on that in the next couple of days.

Using MP4Box to concatenate many .h264 files into one MP4 file

The general form of the concatenate command for MP4Box is:

MP4Box -add <filename>.ext -cat <filename>.ext output.ext1

When you have more than a couple of output files, you’re going to want to automate that -cat part as much as possible because let’s face it, writing out that statement even more than a couple of times will get really old really fast.

The method below works but I’m sure that there is a better way to do it.

  1. echo out the command you want to run. In this case:

(echo -n "MP4Box "; for i in *.h264; do echo -n " -cat $i"; done; echo -n " hummingbird.mp4") >> com.txt

  1. Edit the file com.txt created in (1) so that you can change the first -cat to -add

vim com.txt

  1. While still in vim editing the com.txt file add the #!/bin/sh to the first line. When finished, exit vim2
  2. Change the mode of the file so it can run

chmod +x com.txt

  1. Run the file:

./com.txt

Why am I doing all of this? I have a Raspberry Pi with a Camera attachment and a motion sensor. I’d like to watch the hummingbirds that come to my hummingbird feeder with it for a day or two and get some sweet video. We’ll see how it goes.

  1. The -add will add the <filename> to the output file while the -cat will add any other files to the output file (all while not overwriting the output file so that the files all get streamed together).
  2. I’m sure there’s an xkcd comic about this, but I just can’t find it!