Today I started an i2x project that required me to do a bit of web scraping. I had to make my way through an SSL login and then navigate through a few pages worth, picking up cookies and stuff along the way. I knew this was going to be tricky to script as it wasn't quite obvious what needed to passed around where.
To put it mildly, boy am I lucky I found Fiddler. Fiddler is debugging proxy that sits between your browser and the web and records all the traffic that passes through it. It then allows you to review it and see every last bit of information that's exchanged, so scripting the interaction becomes easier. Or in my case, possible.
While I've used debugging proxies before, Fiddler is by the far the easiest to install, use and seems quite stable. Amazingly, it appears to be a Microsoft product. This has to be among the most useful tools Microsoft has put out since, well, since forever.
Here's how tricky it is to use: download it. Kick off IE. Kick off Fiddler (under your All Programs, among other places). Poof, your done. Your traffic will start getting tracked. You can hit F-12 in Fiddler to toggle whether or not it should pick up traffic.
And Fiddler does more than just allow you track traffic - you can set debug break points, mangle requests/responses and do a whole bunch of other tricks.
How I went this long without having Fiddler in my toolkit is beyond me. It's good stuff.
I should also mention, I'm doing the web scraping in Scheme and found: this article, this recipe and this package all handy in getting me started.
This comment has been removed by the author.
ReplyDeletePerhaps something like Web-Harvest or Selenium would work well for what you're trying to do?
ReplyDeleteThanks Mark, that's handy stuff.
ReplyDeleteIn this case, the app I'm developing is stand alone - so I'm leaning towards not making in Java.
I'm just not excited about the prospect of developing a Swing app.
But those are definitely handy ones to have around.