September 03, 2002
Utility Dilution

The HTTP "referer[sic]" header is pretty cool. I can use it to see where people have linked to lemonodor. I've found some good sites this way.

Here are a few random referers from my logs. If you follow these links, you'll find a link to lemonodor somewhere.

Unfortunately, the referer header seems to be becoming less useful by the day. There is a trend for RSS readers and aggregator applications to use the application's homepage as the referer url.

For example, here are some referers from my logs that will take you to pages that have nothing to do with lemonodor:

It's all just fodder for grep -v. And while it's not a big deal, it is annoying. The user-agent header is right there for the very purpose of expressing information about the program fetching the url. I would imagine it makes it harder for people to use log file analyzers and things and get meaningful results, when suddenly it looks like several hundred hits are coming from a single url--a url that has nothing to do with your site

I think this behavior was pioneered by the Radio aggregator. When Dave Winer added the feature, though, I thought it was pretty cool because Radio actually sent a referer header containing the url of the radio weblog of the person running the aggregator. It wasn't strictly by the RFCs or anything, but it did give me information about people that were reading this site.

But just sticking your reader's homepage url in there... well, it's only slightly more interesting to me than the person that spams my logs with a referer to

Posted by jjwiseman at September 03, 2002 12:02 AM

Only marginally related, but: it would be nice if the user-agent actually did express information about the program fetching the url.

I get a lot of rss requests from gnome-vfs, apparently. I have no idea which of the vast panoply of gnome applications that is, and if it decides to go wild (e.g. to repeat the download once a minute) there's not much I can do other than block the ip address altogether. Which is excessively harsh for one request per minute, really. It's not causing me any real harm

Posted by: Daniel Barlow on September 11, 2002 08:21 AM
Post a comment

Email Address:


Unless you answer this question, your comment will be classified as spam and will not be posted.
(I'll give you a hint: the answer is “lisp”.)


Remember info?