As more sites are leaning on JavaScript to load dynamic data, web scraping is hitting new hurdles. Purely using the urllib2 library from Python will return the site without the dynamic data available to gather if a JavaScript library is used to populate site information.
Hitting this wall, I decided to take a deep dive in scraping data from a local paper who I freelance for’s rosters to create code replacement files to use in Photo Mechanic in this upcoming season. This makes captioning faster and easier, making photos available quicker to the paper.
Continue reading Using Python to scrape website data loaded through JavaScript