Posts

Usefulness of Caching and Databases

Posted February 13, 2015

While working on my website, SFGinfo, I've come across different interesting dilemmas for the first time. One in particular was the cause of inefficient operation for the Roster page. A goal I had from the onset was to have thumbnail images of each player in the list. While this goal was simple enough to implement, its first iteration wasn't very pretty. While thumbnails exist for all currently activated players, most who are still in the minor leagues do not have image files available. This presented a problem. As shown below, I needed to dynamically figure out which players had a photo and which did not, all while keeping the process relatively efficient.

I had the idea of making an HTTP request to each player thumb's purported location. While this worked and allowed me to know which minor league players needed my custom placeholder image, this caused a massive slowdown on page load for each user every time the page was rendered. A friend gave me a suggestion that I hadn't considered: using a database to cache the available URLs, as well as which players had them available and which did not.

Since I had never used a database in a real application before, I spent some time researching the NoSQL datastore provided by Google Cloud¹. By using this database, I was able to set up a cache of sorts. Now instead of a 5-8 second process caused by doing a batch of HTTP requests every time the page is rendered, it only needed to happen once. Now the Roster page loads almost immediately for all users.

^{1. Google Cloud Datastore}