Snoopy
Snoopy is a PHP class that simulates a web browser. It automates the task of retrieving web page content and posting forms
- supports basic user/pass authentication
- supports setting user_agent, referer, cookies and header content (also suports 'raw' headers, so any imaginable http-header can be done)
- supports browser redirects, and controlled depth of redirects
- expands fetched links to fully qualified URLs (default)
This PHP class has been around some time now, and supports anything I could think of to build a good data-fetcher for a PHP-based NewsAggregator. With the possibility to set abritrary http-headers, even the contitional-GET stuff is possible.
Scott points to HTTP Client Class from Leo West, which has a slightly different feature set (supporting WebDAV, and no follow-redirects), althouh I guess after looking at the code to Snoopy, the basic principle is very simiöar, and the supporting code differs.