Less Painful AJAX / Javascript Web Scraping

Published on 2/11/2012

---

If you read my previous post, you’ll see that scraping ajax pages can be a pain. So I wrote a little Java program to make it easier. It takes a list of URLs to scrape, and will render them in a browser, and save the (normal and ajax) rendered HTML and screenshots to a folder.

Here’s the how-to video:

You need Firefox 3+ installed, as well as Java 1.6. This is a beta project, and no warranty is implied. You can get the file here:

http://dl.dropbox.com/u/1015920/VancouverData/ajaxscraper.zip

Mad props to the Selenium team

... views