Category Archives: Software Development

Using Python to scrape website data loaded through JavaScript

As more sites are leaning on JavaScript to load dynamic data, web scraping is hitting new hurdles. Purely using the urllib2 library from Python will return the site without the dynamic data available to gather if a JavaScript library is used to populate site information.

Hitting this wall, I decided to take a deep dive in scraping data from a local paper who I freelance for’s rosters to create code replacement files to use in Photo Mechanic in this upcoming season. This makes captioning faster and easier, making photos available quicker to the paper.

Continue reading Using Python to scrape website data loaded through JavaScript

Gateway to Innovation

I was able to attend the Gateway to Innovation conference in St. Louis for the first time this year, thanks in large part to Joe Parker being in London last week and giving me his ticket.

The conference celebrated its 10th year of bringing technology-based innovation speakers and vendors to St. Louis. While the conference focuses largely on the business side of innovative technologies, there was a handful of talks on new tech. and how businesses are implementing it. (My favorite was a talk on Microsoft’s development of an Internet of Things platform.)

Continue reading Gateway to Innovation

Building a digital newspaper machine

Awhile back, a friend of mine, Nate Papes, ran across Scott Walker, director of enterprise and investigation at Alabama Media Group, who had retrofitted a newspaper machine to display newspaper front pages digitally. (Update March 13, 2016: This link is dead; it appears Walker removed the post.) We talked about creating our own, but never followed through.

A few years later, Nate found a New York Times newspaper machine in a Joplin, Mo., junkyard and began to work on his own machine. I offered to write the software to make the box run since Walker’s was written for a Macintosh and we both were toying with Raspberry Pi computers. In the end, Nate kept his box as original to the Times box as possible and I wrote the software to run as JavaScript in the browser and pull the pages down.

Continue reading Building a digital newspaper machine