Categories
Django

Deploying a Django Site to Digital Ocean – A Series

Previous Efforts

When I first heard of Django I thought it looks like a really interesting, and Pythonic way, to get a website up and running. I spent a whole weekend putting together a site locally and then, using Digital Ocean, decided to push my idea up onto a live site.

One problem that I ran into, which EVERY new Django Developer will run into was static files. I couldn’t get static files to work. No matter what I did, they were just … missing. I proceeded to spend the next few weekends trying to figure out why, but alas, I was not very good (or patient) with reading documentation and gave up.

Fast forward a few years, and while taking the 100 Days of Code on the Web Python course from Talk Python to Me I was able to follow along on a part of the course that pushed up a Django App to Heroku.

I wrote about that effort here. Needless to say, I was pretty pumped. But, I was wondering, is there a way I can actually get a Django site to work on a non-Heroku (PaaS) type infrastructure.

Inspiration

While going through my Twitter timeline I cam across a retweet from TestDrive.io of Matt Segal. He has an amazing walk through of deploying a Django site on the hard level (i.e. using Windows). It’s a mix of Blog posts and YouTube Videos and I highly recommend it. There is some NSFW language, BUT if you can get past that (and I can) it’s a great resource.

This series is meant to be a written record of what I did to implement these recommendations and suggestions, and then to push myself a bit further to expand the complexity of the app.

Articles

A list of the Articles will go here. For now, here’s a rough outline of the planned posts:

The ‘Enhancements’ will be multiple follow up posts (hopefully) as I catalog improvements make to the site. My currently planned enhancements are:

Categories
Django

Django form filters

I’ve been working on a Django Project for a while and one of the apps I have tracks candidates. These candidates have dates of a specific type.

The models look like this:

Candidate

class Candidate(models.Model):
    first_name = models.CharField(max_length=128)
    last_name = models.CharField(max_length=128)
    resume = models.FileField(storage=PrivateMediaStorage(), blank=True, null=True)
    cover_leter = models.FileField(storage=PrivateMediaStorage(), blank=True, null=True)
    email_address = models.EmailField(blank=True, null=True)
    linkedin = models.URLField(blank=True, null=True)
    github = models.URLField(blank=True, null=True)
    rejected = models.BooleanField()
    position = models.ForeignKey(
        "positions.Position",
        on_delete=models.CASCADE,
    )
    hired = models.BooleanField(default=False)

CandidateDate

class CandidateDate(models.Model):
    candidate = models.ForeignKey(
        "Candidate",
        on_delete=models.CASCADE,
    )
    date_type = models.ForeignKey(
        "CandidateDateType",
        on_delete=models.CASCADE,
    )
    candidate_date = models.DateField(blank=True, null=True)
    candidate_date_note = models.TextField(blank=True, null=True)
    meeting_link = models.URLField(blank=True, null=True)

    class Meta:
        ordering = ["candidate", "-candidate_date"]
        unique_together = (
            "candidate",
            "date_type",
        )

CandidateDateType

class CandidateDateType(models.Model):
    date_type = models.CharField(max_length=24)
    description = models.CharField(max_length=255, null=True, blank=True)

You’ll see from the CandidateDate model that the fields candidate and date_type are unique. One problem that I’ve been running into is how to help make that an easier thing to see in the form where the dates are entered.

The Django built in validation will display an error message if a user were to try and select a candidate and date_type that already existed, but it felt like this could be done better.

I did a fair amount of Googling and had a couple of different bright ideas, but ultimately it came down to a pretty simple implementation of the exclude keyword in the ORM

The initial Form looked like this:

class CandidateDateForm(ModelForm):
   class Meta:
        model = CandidateDate
        fields = [
            "candidate",
            "date_type",
            "candidate_date",
            "meeting_link",
            "candidate_date_note",
        ]
        widgets = {
            "candidate": HiddenInput,
        }

I updated it to include a __init__ method which overrode the options in the drop down.

def __init__(self, *args, **kwargs):
    super(CandidateDateForm, self).__init__(*args, **kwargs)
    try:
        candidate = kwargs["initial"]["candidate"]
        candidate_date_set = CandidateDate.objects.filter(candidate=candidate).values_list("date_type", flat=True)
        qs = CandidateDateType.objects.exclude(id__in=candidate_date_set)
        self.fields["date_type"].queryset = qs
    except KeyError:
        pass

Now, with this method the drop down will only show items which can be selected, not all CandidateDateType options.

Seems like a better user experience AND I got to learn a bit about the Django ORM

Categories
Django

Using different .env files

In a Django project I’m working on I use a library called Django-environ which

allows you to utilize 12factor inspired environment variables to configure your Django application.

It’s a pretty sweet library as well. You create a .env file to store your variable that you don’t want in a public repo for your settings.py.

The big issue I have is that my .env file for my local development isn’t what I want on my production server (obviously … never set DEBIG=True in production!)

I had tried to use a different .env file using an assortment of methods, but to no avail. And the documentation wasn’t much of a help for using Multiple env file

It is possible to have multiple env files and select one using environment variables.

Now ENV_PATH=other-env ./manage.py runserver uses other-env while ./manage.py runserver uses .env.

But there’s no example about how to actually set that up 🤦🏻‍♂️1.

In fact, this bit in the documentation reminded me of thisvideo on YouTube.

Instead of trying to figure out the use of multiple .env files I instead used a just recipe in my justfile to get the job done.

# checks the deployment for prod settings; will return error if the check doesn't pass
check:
    cp core/.env core/.env_staging
    cp core/.env_prod core/.env
    -python manage.py check --deploy
    cp core/.env_staging core/.env

OK. What does this recipe do?

First, we copy the development .env file to a .env_staging file to keep the original development settings ‘somewhere’

 cp core/.env core/.env_staging

Next, we copy the .env_prod to the .env so that we can use it when we run -python manage.py check --deploy.

cp core/.env_prod core/.env
-python manage.py check --deploy

Why do we use the -? That allows the justfile to keep going if it runs into an error. Since we’re updating our main .env file I want to make sure it gets restored to its original state … just in case!

Finally, we copy the original contents of the .env file from the .env_staging back to the .env to restore it to its development settings.

Now, I can simply run

just check

And I’ll know if I have passed the 12 factor checking for my Django project or somehow introduced something that makes the check not pass.

I’d like to figure out how to set up multiple .env files, create an example and contribute to the docs … but honestly I have no freaking clue how to do it. If I am able to figure it out, you can bet I’m going to write up a PR for the docs!

  1. I’d like to figure out how to set up multiple .env files, create an example and contribute to the docs … but honestly I have no freaking clue how to do it. If I am able to figure it out, you can bet I’m going to write up a PR for the docs!
Categories
Django

Logging in a Django App

Per the Django Documentation you can set up

A list of all the people who get code error notifications. When DEBUG=False and AdminEmailHandler is configured in LOGGING (done by default), Django emails these people the details of exceptions raised in the request/response cycle.

In order to set this up you need to include in your settings.py file something like:

ADMINS = [
	('John', 'john@example.com'), 
	('Mary', 'mary@example.com')
]

The difficulties I always ran into were:

  1. How to set up the AdminEmailHandler
  2. How to set up a way to actually email from the Django Server

Again, per the Django Documentation:

Django provides one log handler in addition to those provided by the Python logging module

Reading through the documentation didn’t really help me all that much. The docs show the following example:

'handlers': {
    'mail_admins': {
        'level': 'ERROR',
        'class': 'django.utils.log.AdminEmailHandler',
        'include_html': True,
    }
},

That’s great, but there’s not a direct link (that I could find) to the example of how to configure the logging in that section. It is instead at the VERY bottom of the documentation page in the Contents section in the Configured logging > Examples section … and you really need to know that you have to look for it!

The important thing to do is to include the above in the appropriate LOGGING setting, like this:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
	    'mail_admins': {
	        'level': 'ERROR',
	        'class': 'django.utils.log.AdminEmailHandler',
	        'include_html': True,
	    }
       },
    },
}

Sending an email with Logging information

We’ve got the logging and it will be sent via email, but there’s no way for the email to get sent out yet!

In order to accomplish this I use SendGrid. No real reason other than that’s what I’ve used in the past.

There are great tutorials online for how to get SendGrid integrated with Django, so I won’t rehash that here. I’ll just drop my the settings I used in my settings.py

SENDGRID_API_KEY = env("SENDGRID_API_KEY")

EMAIL_HOST = "smtp.sendgrid.net"
EMAIL_HOST_USER = "apikey"
EMAIL_HOST_PASSWORD = SENDGRID_API_KEY
EMAIL_PORT = 587
EMAIL_USE_TLS = True

One final thing I needed to do was to update the email address that was being used to send the email. By default it uses root@localhost which isn’t ideal.

You can override this by setting

SERVER_EMAIL = myemail@mydomain.tld

With those three settings, everything should just work.

Categories
Django Python

My First Django Project

I’ve been writing code for about 15 years (on and off) and Python for about 4 or 5 years. With Python it’s mostly small scripts and such. I’ve never considered myself a ‘real programmer’ (Python or otherwise).

About a year ago, I decided to change that (for Python at the very least) when I set out to do 100 Days Of Web in Python from Talk Python To Me. Part of that course were two sections taught by Bob regarding Django. I had tried learn Flask before and found it … overwhelming to say the least.

Sure, you could get a ‘hello world’ app in 5 lines of code, but then what? If you wanted to do just about anything it required ‘something’ else.

I had tried Django before, but wasn’t able to get over the ‘hump’ of deploying. Watching the Django section in the course made it just click for me. Finally, a tool to help me make AND deploy something! But what?

The Django App I wanted to create

A small project I had done previously was to write a short script for my Raspberry Pi to tell me when LA Dodger (Baseball) games were on (it also has beloved Dodger Announcer Vin Scully say his catch phrase, “It’s time for Dodger baseball!!!”).

I love the Dodgers. But I also love baseball. I love baseball so much I have on my bucket list a trip to visit all 30 MLB stadia. Given my love of baseball, and my new found fondness of Django, I thought I could write something to keep track of visited stadia. I mean, how hard could it really be?

What does it do?

My Django Site uses the MLB API to search for games and allows a user to indicate a game seen in person. This allows them to track which stadia you’ve been to. My site is composed of 4 apps:

  • Users
  • Content
  • API
  • Stadium Tracker

The API is written using Django Rest Framework (DRF) and is super simple to implement. It’s also really easy to changes to your models if you need to.

The Users app was inspired by Will S Vincent ( a member of the Django Software Foundation, author, and podcaster). He (and others) recommend creating a custom user model to more easily extend the User model later on. Almost all of what’s in my Users App is directly taken from his recommendations.

The Content App was created to allow me to update the home page, and about page (and any other content based page) using the database instead of updating html in a template.

The last App, and the reason for the site itself, is the Stadium Tracker! I created a search tool that allows a user to find a game on a specific day between two teams. Once found, the user can add that game to ‘Games Seen’. This will then update the list of games seen for that user AND mark the location of the game as a stadium visited. The best part is that because the game is from the MLB API I can do some interesting things:

  1. I can get the actual stadium from visited which allows the user to indicate historic (i.e. retired) stadia
  2. I can get details of the game (final score, hits, runs, errors, stories from MLB, etc) and display them on a details page.

That’s great and all, but what does it look like?

The Search Tool

Stadia Listing

National League West

American League West

What’s next?

I had created a roadmap at one point and was able to get through some (but not all) of those items. Items left to do:

  • Get Test coverage to at least 80% across the app (currently sits at 70%)
  • Allow users to be based on social networks (right now I’m looking at Twitter, and Instagram) probably with the Django Allauth Package
  • Add ability to for minor league team search and stadium tracking (this is already part of the MLB API, I just never implemented it)
  • Allow user to search for range of dates for teams
  • Update the theme … it’s the default MUI CSS which is nice, but I’d rather it was something a little bit different
  • Convert Swagger implementation from django-rest-swagger to drf-yasg

Final Thoughts

Writing this app did several things for me.

First, it removed some of the tutorial paralysis that I felt. Until I wrote this I didn’t think I was a web programmer (and I still don’t really), and therefore had no business writing a web app.

Second, it taught me how to use git more effectively. This directly lead to me contributing to Django itself (in a very small way via updates to documentation). It also allowed me to feel comfortable enough to write my first post on this very blog.

Finally, it introduced me to the wonderful ecosystem around Django. There is so much to learn, but the great thing is that EVERYONE is learning something. There isn’t anyone that knows it all which makes it easier to ask questions! And helps me in feeling more confident to answer questions when asked.

The site is deployed on Heroku and can be seen here. The code for the site can be seen here.

This article was also posted on the PyBit.es Blog

Categories
Django PyCharm Python

Mischief Managed

A few weeks back I decided to try and update my Python version with Homebrew. I had already been through an issue where the an update like this was going to cause an issue, but I also knew what the fix was.

With this knowledge in hand I happily performed the update. To my surprise, 2 things happened:

  1. The update seemed to have me go from Python 3.7.6 to 3.7.3
  2. When trying to reestablish my Virtual Environment two packages wouldn’t installed: psycopg2 and django-heroku

Now, the update/backdate isn’t the end of the world. Quite honestly, next weekend I’m going to just ditch homebrew and go with the standard download from Python.org because I’m hoping that this non-sense won’t be an issue anymore

The second issue was a bit more irritating though. I spent several hours trying to figure out what the problem was, only to find out, there wasn’t one really.

The ‘fix’ to the issue was to

  1. Open PyCharm
  2. Go to Setting
  3. Go to ‘Project Interpreter’
  4. Click the ‘+’ to add a package
  5. Look for the package that wouldn’t install
  6. Click ‘Install Package’
  7. Viola … mischief managed

The next time this happens I’m just buying a new computer

Categories
Contributing Django

My first commit to an Open Source Project: Django

Last September the annual Django Con was held in San Diego. I really wanted to go, but because of other projects and conferences for my job, I wasn’t able to make it.

The next best thing to to watch the videos from DjangoCon on YouTube. I watched a couple of the videos, but one that really caught my attention was by Carlton Gibson titled “Your Web Framework Needs You: An Update by Carlton Gibson”.

I took what Carlton said to heart and thought, I really should be able to do something to help.

I went to the Django Issues site and searched for an Easy Pickings issue that involved documentation and found issue 31006 “Document how to escape a date/time format character for the |date and |time filters.”

I read the steps on what I needed to do to submit a pull request, but since it was my first time ever participating like this … I was a bit lost.

Luckily there isn’t anything that you can break, so I was able to wonder around for a bit and get my bearings.

I forked the GitHub repo and I cloned it locally.

I then spent an embarasingly long time trying to figure out where the change was going to need to be made, and exactly what needed to change.

Finally, with my changes made, I pushed my code changes to GitHub and waited.

Within a few hours Mariusz Felisiak replied back and asked about a suggestion he had made (but which I missed). I dug back into the documentation, found what he was referring to, and made (what I thought) was his suggested change.

Another push and a bit more waiting.

Mariusz Felisiak replied back with some input about the change I pushed up, and I realized I had missed the mark on what he was suggesting.

OK. Third time’s a charm, right?

Turns out, in this case it was. I pushed up one last time and this time, my changes were merged into the master and just like that, I am now a contributor to Django (albeit a very, very, very minor contributor).

Overall, this was a great experience, both with respect to learning about contributing to an open source project, as well as learning about GitHub.

I’m hoping that with the holidays upon us I’ll be able to find the time to pick up one or two (maybe even three) Easy Pickings issues from the Django issue tracker.

Categories
Django

Updating the models for my Django Rest Framework API

I’ve been working on a Django project which would allow users to track games that they’ve seen and, therefore, see what stadia they have visited.

This is all being done at a site i set up called StadiaTracker.com. Initially when constructing my model I kept it relatively simple. I had one model that had two fields. The two fields tied the User from my CustomUser Model to a Game ID that I retrieve from an API that MLB provides.

I thought this simple approach would be the best approach. In addition to having a Django App I set up a Django Rest Framework (DRF) API. My initial plan was to have a DRF backend with a Vue (or React) front end. (I still want to do that, but I really wanted to try and finish a project before proceeding down that path).

After some development and testing I quickly realized that the page loads for the app were suffering because of the number of API calls to MLB that were being made.

I created a new model to tie the user id (still from the CustomUser model I’d created) to the game id, but in addition I’d get and store the following information:

  • Home Team Name
  • Home Team Score
  • Home Team Hits
  • Home Team Errors
  • Away Team Name
  • Away Team Score
  • Away Team Hits
  • Away Team Errors
  • Game Recap Headline
  • Game Recap Summary
  • Game Date / Time

By storing all of this my views could render more quickly because they didn’t have to go to the MLB API to get the information.

Of course, once I did this I realized that the work I had done on the DRF API would also need to be updated.

Initially I kept putting off the refactoring that was going to have to be done. Finally, I just sat down and did it. And you know what, within 10 minutes I was done.

I only had to change 3 files:

  • serializers.py
  • urls.py
  • views.py

For the searializers.py and views.py all I had to do was add the new model and then copy and paste what I had done for the previous model.

For the urls.py it was just a simple matter of updating the the DRF path and detail path to use the new views I had just created.

It was so amazingly simple I could barely believe it. This thing I had put off for a couple of weeks because I was afraid it was going to be really hard, just wasn’t.

Categories
Django Heroku

My first project after completing the 100 Days of Web in Python

As I mentioned in my last post, after completing the 100 Days of Web in Python I was moving forward with a Django app I wrote.

I pushed up my first version to Heroku on August 24. At that point it would allow users to add a game that they had seen, but when it disaplyed the games it would show a number (the game’s ID) instead of anything useful.

A few nights ago (Aug 28) I committed a version which allows the user to see which game they add, i.e. there are actual human readable details versus just a number!

The page can be found here. It feels really good to have it up in a place where people can actually see it. That being said I discovered a a couple of things on the publish that I’d like to fix.

I have a method that returns details about the game. One problem is that if any of the elements return None then the front page returns a Server 500 error … this is not good.

It took a bit of googling to see what the issue was. The way I found the answer was to see an idea to turn Debug to True on my ‘prod’ server and see the output. That helped me identify the issue.

To ‘fix’ it in the short term I just deleted all of the data for the games seen in the database.

I’m glad that it happened because it taught me some stuff that I knew I needed to do, but maybe didn’t pay enough attention to … like writing unit tests.

Based on that experience I wrote out a roadmap of sorts for the updates I want to get into the app:

  • Tests for all classes and methods
  • Ability to add minor league games
  • Create a Stadium Listing View
  • More robust search tool that allows a single team to be selected
  • Logged in user view for only their games
  • Create a List View of games logged per stadium
  • Create a List View of attendees (i.e. users) at games logged
  • Add more user features:
    • Ability to add a picture
    • Ability to add Twitter handle
    • Ability to add Instagram handle
    • Ability to add game notes
  • Create a Heroku Pipeline to ensure that pushes to PROD are done through a UAT site
  • Create a blog (as a pelican standalone sub domain)

It’s a lot of things but I’ve already done some things that I wanted to:

  • Added SSL
  • Set up to go to actual domain instead of Heroku subdomain

I’ll write up how I did the set up for the site so I can do it again. It’s not well documented when your registrar is Hover and you’ve got your site on Heroku. Man … it was an tough.