Setting up Jupyter Notebook on my Linode

A Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text.

Uses include:

  1. data cleaning and transformation
  2. numerical simulation
  3. statistical modeling
  4. data visualization
  5. machine learning
  6. and other stuff

I’ve been interested in how to set up a Jupyter Notebook on my Linode server for a while, but kept running into a roadblock (either mental or technical I’m not really sure).

Then I came across this ‘sweet’ solution to get them set up athttp://blog.lerner.co.il/five-minute-guide-setting-jupyter-notebook-server/

My main issue was what I needed to to do keep the Jupyter Notebook running once I disconnected from command line. The solution above gave me what I needed to solve that problem

nohup jupyter notebook

nohup allows you to disconnect from the terminal but keeps the command running in the background (which is exactly what I wanted).

The next thing I wanted to do was to have the jupyter notebook server run from a directory that wasn’t my home directory.

To do this was way easier than I thought. You just run nohup jupyter notebook from the directory you want to run it from.

The last thing to do was to make sure that the notebook would start up with a server reboot. For that I wrote a shell script

# change to correct directory
cd /home/ryan/jupyter

nohup jupyter notebook &> /home/ryan/output.log

The last command is a slight modification of the line from above. I really wanted the output to get directed to a file that wasn’t in the directory that the Jupyter notebook would be running from. Not any reason (that I know of anyway) … I just didn’t like the nohup.out file in the working directory.

Anyway, I now have a running Jupyter Notebook at http://python.ryancheley.com:88881

  1. I’d like to update this to be running from a port other than 8888 AND I’d like to have it on SSL, but one thing at a time!

A Summary of Dale Carnegie’s “How to Win Friends and Influence People”

This is mostly for me to write down my notes and thoughts about the book “How to Win Friends and Influence People.”

I’ve noted below the summary from the end of each section below (so I don’t forget what they were).

The first three sections seemed to speak to my modern sensibilities the most (keep in mind this book was published in 1936 … the version I read was revised in 1981).

I have the summaries below, for reference, but I wanted to have my own take on each.

Fundamental Techniques in Handling People

This seems to be a long way of saying the “Use the Golden Rule” over and over again. The three points are:

  1. Don’t criticize, condemn or complain
  2. Give honest and sincere appreciation
  3. Arouse in the other person an eager want

Six ways to make people like you

The ‘rules’ presented here are also useful for making small talk at parties (or other gatherings). I find that talking about myself with a total stranger is about the hardest thing I can do. I try to engage with people at parties and have what I hope are interesting questions to ask should I need to. Stuff I tend to avoid:

  • What do you do for a living?
  • Where do you work?
  • Sports
  • Politics

Stuff I try to focus on:

  • How do you know the host / acquaintance we may have in common
  • What’s the most interesting problem you’ve solved or are working to solve in the last week
  • Have you been on a vacation recently? What was your favorite part about it? (With this one I don’t let people off the hook with, ‘being away from work’ … I try to find something that they really found enjoyable and interesting

These talking points are usually a pretty good starting point. Sometimes when I’m introduced to a person and the person introduces them as their job, i.e. This is Sally Jones, she’s a Doctor at the local Hospital, I’ll use that to parlay away from something work focused (what kind of doctor are you) to something more person focused, why did you want to become a doctor? Where did you go to Medical School? Did you know you always wanted to be a doctor? I try to focus on getting to know them better and have them talk about themselves.

The tips from the book support my intuition when meeting new people. They are:

  1. Become genuinely interested in other people
  2. Smile
  3. Remember that a person’s name is to that person the sweetest and most important sound in any language
  4. Be a good listener. Encourage to talk about themselves
  5. Talk in terms of the other person’s interest
  6. Make the other person feel important – and do it sincerely

How to Win People to your way of thinking

This section provided the most useful and helpful information (for me anyway!). It really leads to how to have better influence (than winning friends).

One of the problems I’ve suffered from throughout my life is the need to be right about a thing. This section has concrete tips and examples of how to not be the smartest person in the room, but working on being the most influential person in the room.

My favorite is the first one, which I’ll paraphrase to be “The only way to win an argument is to avoid it!” I’d never thought about trying to avoid arguments, only how to win them once I was in them. The idea reminds me a bit of War Games. At the end, Joshua, the super computer that is trying to figure out how to win a Nuclear War with the USSR, concedes that the only way to win is to not play at all. Just like an argument.

The other piece that really struck me was get the other person to say ‘Yes’. This is kind of sales-y and could be smarmy if used with a subtext of insincerity, but I think that the examples given in the book, and using it in the context of trying to win friends AND influence people it can go a long way.

The tips from this section of the book are:

  1. The only way to get the best of an argument is to avoid it
  2. Show respect for the other person’s opinions. Never say “You’re wrong”
  3. If you are wrong, admit it quickly and emphatically
  4. Begin in a friendly way
  5. Get the other person saying “yes, yes” immediately
  6. Let the other person do a great deal of the talking
  7. Let the other person feel that the idea is his or hers
  8. Try honestly to see things from the other persons perspective
  9. BE sympathetic with the other persons ideas and desires
  10. Appeal to the nobler motives
  11. Dramatize your ideas
  12. throw down a challenge

Be a Leader: How to change people without giving offense or arousing resentment

This section has the best points, but the stories were very contrived. Again, this goes to how to win influence more than winning friends. Some of the items are a bit too 1930s for my taste (numbers 2, 3, and 6 in particular seem overly outdated). But overall, they are good ideas to work towards.

The tips are:

  1. Begin with praise and honest appreciation
  2. Call attention to the person’s mistake indirectly
  3. Talk about your own mistakes before criticizing the other person
  4. Ask questions instead of giving direct orders
  5. Let the other person save face
  6. Praise the slightest improvement and praise every improvement. Be “hearty in your approbation and lavish in your praise”
  7. Give the other person a fine reputation to live up to
  8. Use encouragement. make the fault seem easy to correct
  9. Make the other person gabby about doing the thing you suggest

Overall I’m really glad that I read this book and glad that my CHIME mentor Tim Gibbs recommended it to me.

I’ve been actively working to include these ideas into my work and home life and have found some surprising benefits. It’s also helping to make me a little less stressed out.

If you’re looking for a bit of help in trying to be a better influencer in your organization, or your personal life, this book is well worth the read.

Updating my meeting Workflow for Drafts 5

Drafts is a productivity app created by Greg Pierce (@AgileTortoise).

I’ve loved and used Drafts 4 every day for the last several years. I loved it so much I even contributed to the Tip Jar Greg had in the app. Seriously, it’s an amazing app. If you haven’t downloaded it already you totally should.

Recently, Greg released Drafts 5. With this new version comes a new Business Model as well. Instead of a single pay (and hope people ‘tip’ you) he’s converted to a subscription model.

I signed up for the free week and didn’t have a real opportunity to use it before my free week was converted into a pay week but I’ve no regrets. I like what Greg does and want him to keep updating his app so that I can get the benefits of it once I have a real chance to dive in.

Part of the reason I wasn’t able to really use the new version is the way that I primarily use Drafts. I have a WorkFlow that takes a meeting on my work calendar and allows me to take notes about that meetings.

It’s one of the most useful productivity tools I have during my morning standup meetings with my team, and it’s useful for the other (sometimes endless) meetings that I go to.

With the release of Drafts 5 I was not longer able to use both Drafts 5 AND my workflow, so I needed to update my workflow.

With Drafts 4 it was just one of the built in Apps. Because Drafts 5 limits some of the functionality unless you have the PRO version I don’t think that Workflow will be updated to include Drafts 5 like it did Drafts 4.

Once I realized that AND since I’m paying for the app I figured I’d need to update my Workflow instead of waiting and hoping that Workflow would be updated to include Drafts 5.

In order to make the update I had to look for URL Scheme for Drafts 5 … but I couldn’t really find one. I assumed that Drafts 5 URL Scheme would be the same as Drafts 4 (I was right) and made various attempts at getting a copy of the Workflow to work with Drafts 5.

This is the section of the workflow that needs to be updated:

Since Drafts 5 isn’t included in the Built in Apps I was going to need to pass a URL and open the app.

This would require 3 separate steps in Workflow

  1. Convert Text into URL Encoded string
  2. Prepend the URL Scheme for creating a new draft to the URL Encoded String
  3. Open the URL

This basically means that 1 step is now replaced with 3 … but hey, that’s the price of progress must be paid!

Both the Drafts 4 and Drafts 5 versions of these workflows are available.

If you enjoy them, hit me up in the comments or let me know on Twitter @ryancheley!

OmniFocus 3!

The OmniGroup posted on their blog the other day that they have a ship date for OmniFocus 3 … May 30.

To say that I’m excited is a bit of an understatement. I’ve been a loyal OmniFocus user for about 5 years now, and though I have declared OmniFocus bankruptcy once, I still believe it’s a super useful App and I could not imagine life without it.

OmniFocus3 will bring with it tags which is something I’ve really wanted and struggled without in using OmniFocus2. Sometimes things don’t just fall into pure GTD contexts and tags appear to be a recognition of that.

The initial release on May 30 is iOS only. The Mac Version is set to be released in 2018 but no firm date has been announced.

Only 25 days until upgrade day 😊

Making it easy to ssh into a remote server

Logging into a remote server is a drag. Needing to remember the password (or get it from 1Password); needing to remember the IP address of the remote server. Ugh.

It’d be so much easier if I could just

ssh username@servername

and get into the server.

And it turns out, you can. You just need to do two simple things.

Simple thing the first: Update the hosts file on your local computer to map the IP address to a memorable name.

The hosts file is located at /etc/hosts (at least on *nix based systems).

Go to the hosts file in your favorite editor … my current favorite editor for simple stuff like this is vim.

Once there, add the IP address you don’t want to have to remember, and then a name that you will remember. For example:

67.176.220.115    easytoremembername

One thing to keep in mind, you’ll already have some entries in this file. Don’t mess with them. Leave them there. Seriously … it’ll be better for everyone if you do.

Simple thing the second: Generate a public-private key and share the public key with the remote server

From the terminal run the command ssh-keyken -t rsa. This will generate a public and private key. You will be asked for a location to save the keys to. The default (on MacOS) is /Users/username/.ssh/id_rsa. I tend to accept the default (no reason not to) and leave the passphrase blank (this means you won’t have to enter a password which is what we’re looking for in the first place!)

Next, we copy the public key to the host(s) you want to access using the command

ssh-copy-id <username>@<hostname>

for example:

ssh-copy-id pi@rpicamera

The first time you do this you will get a message asking you if you’re sure you want to do this. Type in yes and you’re good to go.

One thing to note, doing this updates the file known_hosts. If, for some reason, the server you are ssh-ing to needs to be rebuilt (i.e. you have to keep destroying your Digital Ocean Ubuntu server because you can’t get the static files to be served properly for your Django project) then you need to go to the known_hosts file and remove the entry for that known host.

When you do that you’ll be asked about the identity of the server (again). Just say yes and you’re good to go.

If you forget that step then when you try to ssh into the server you get a nasty looking error message saying that the server identities don’t match and you can’t proceed.

Automating the Hummingbird Video Upload to YouTube or How I finally got Cron to do what I needed it to do but in the ugliest way possible

Several weeks ago in Cronjob Redux I wrote that I had finally gotten Cron to automate the entire process of compiling the h264 files into an mp4 and uploading it to YouTube.

I hadn’t. And it took the better part of the last 2 weeks to figure out what the heck was going on.

Part of what I wrote before was correct. I wasn’t able to read the client_secrets.json file and that was leading to an error.

I was not correct on the creation of the create_mp4.sh though.

The reason I got it to run automatically that night was because I had, in my testing, created the create_mp4.sh and when cron ran my run_script.sh it was able to use what was already there.

The next night when it ran, the create_mp4.sh was already there, but the h264 files that were referenced in it weren’t. This lead to no video being uploaded and me being confused.

The issue was that cron was unable to run the part of the script that generates the script to create the mp4 file.

I’m close to having a fix for that, but for now I did the most inelegant thing possible. I broke up the script in cron so it looks like this:

00 06 * * * /home/pi/Documents/python_projects/cleanup.sh
10 19 * * * /home/pi/Documents/python_projects/create_script_01.sh
11 19 * * * /home/pi/Documents/python_projects/create_script_02.sh >> $HOME/Documents/python_projects/create_mp4.sh 2>&1
12 19 * * * /home/pi/Documents/python_projects/create_script_03.sh
13 19 * * * /home/pi/Documents/python_projects/run_script.sh

At 6am every morning the cleanup.sh runs and removes the h264 files, the mp4 file and the create_mp4.sh script

At 7:10pm the ‘header’ for the create_mp4.sh runs. At 7:11pm the ‘body’ for create_mp4.sh runs. At 7:12pm the ‘footer’ for create_mp4.sh runs.

Finally at 7:13pm the run_script.sh compiles the h264 files into an mp4 and uploads it to YouTube.

Last night while I was at a School Board meeting the whole process ran on it’s own. I was super pumped when I checked my YouTube channel and saw that the May 1 hummingbird video was there and I didn’t have to do anything.

Whoops! Or how I broke my website by installing Nginx with Apache

I’ve been working on a project to create a Django based website. Over the weekend (Saturday I think) I tried to get it up and running on my Linode server. However, after a couple of failed attempts I decided to use the free hosting coupon1 I had for DigitalOcean to see if that allowed me to reply more easily deploy … the short answer … meh.

What I didn’t realize over the weekend is that while I had been trying to deploy my Django site, I had installed Nginx on my Linode server that was also running apache2. This lead to them both trying to listen on port 80 but because Nginx was the last thing I had kicked off, it was winning.

While I was working on my Django site I should have realized that something was up when I tried to connect to the blog for the site (still a WordPress site on my Linode server) and it returned a ‘Can not connect to the server message’. I didn’t pay much attention because I figured (incorrectly) that I had done something specific to that subdomain, and not that I had made all of the sites on my Linode server inaccessible.

Last night at about 9 I thought, “Well, it should’t take long for me to figure out the issue with the new blog. ”

By 10:15 I tried everything the internet had told me to try and I was still unable to get apache2 to reload.

I googled a bunch of stuff, but nothing was helping.

When I tried to get the status on apache2 I would get this:

● apache2.service - LSB: Apache2 web server
   Loaded: loaded (/etc/init.d/apache2; bad; vendor preset: enabled)
  Drop-In: /lib/systemd/system/apache2.service.d
           └─apache2-systemd.conf
   Active: inactive (dead) since Tue 2018-05-01 05:01:03 PDT; 5s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 7718 ExecStop=/etc/init.d/apache2 stop (code=exited, status=0/SUCCESS)
  Process: 7703 ExecStart=/etc/init.d/apache2 start (code=exited, status=0/SUCCESS)

May 01 05:01:03 milo apache2[7703]: (98)Address already in use: AH00072: make_sock: could not bind to address [::]:80
May 01 05:01:03 milo apache2[7703]: (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:80
May 01 05:01:03 milo apache2[7703]: no listening sockets available, shutting down
May 01 05:01:03 milo apache2[7703]: AH00015: Unable to open logs
May 01 05:01:03 milo apache2[7703]: Action 'start' failed.
May 01 05:01:03 milo apache2[7703]: The Apache error log may have more information.
May 01 05:01:03 milo apache2[7703]:  *
May 01 05:01:03 milo apache2[7718]:  * Stopping Apache httpd web server apache2
May 01 05:01:03 milo apache2[7718]:  *
May 01 05:01:03 milo systemd[1]: Started LSB: Apache2 web server.

This morning I started to google each line of the status message and finally got to this:

no listening sockets available, shutting down

Googling for that lead me to trying this:

sudo netstat -ltnp | grep ':80'

Which output this:

tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      3324/nginx -g daemo
tcp6       0      0 :::80                   :::*                    LISTEN      3324/nginx -g daemo

And finally, I saw the issue. Over the weekend while I was futzing around I had apparently installed Nginx and let it listen on port 80 AND kept it running.

Once I killed the Nginx process with this:

sudo kill -9 3324

I was able to restart apache2 with no problems.

Thank goodness.

I find that when I mess something up like this it’s important to ask myself what I learned from the experience.

In that vein …

What did I learn from this experience?

  1. Can’t run apache2 and Nginx on the same server and have them listen on the same port. Seems obvious, but you know having to actually deal with it really seals the deal
  2. The output messages are super helpful … google each part of them and don’t give up
  3. A good night’s sleep can make all the difference
  4. Rolling your own web server is less expensive than having it be Turnkey (a la SquareSpace, or some other hosted solution) but you end up being your own Sys Admin and that’s actually pretty easy when things are going well, and a freaking nightmare when they’re not
  1. Thanks to the Talk Python to Me Course for Entrepreneurs