Andy Brudtkuhl laments a common problem one encounters as a feed aggregator: It isn’t always simple to eliminate duplicates when comparing RSS items. This is even true when you’re comparing two items that are actually the same thing, but one has been republished.

He has discovered that when you are trying to compare the items, you can’t rely on the link alone to determine uniqueness. Essentially: the link in an RSS item is not a guaranteed unique ID (GUID).

In Andy’s case, he’s seeing items that originated from the same site, but are either linked to the original site (http://www.converstations.com/… /discovery-along.html), or have the FeedBurner link to track click-throughs: (http://feeds.feedburner.com/… /discovery-along.html).

This is one of those tough problems with RSS that the optional GUID element was supposed to solve. If the GUID element is present in the item – as is properly formed – you should be able to rely on that regardless of link variation.

Unfortunately, the RSS GUIDs are rarely used correctly among publishers, making it fairly unreliable. Mark Pilgrim discussed this back in 2004:

1. Older RSS versions don’t have it, and even in the latest version, it’s still optional. So very few feeds actually have it.
2. The RSS spec doesn’t give clear guidance on how to make a unique identifier, or how unique it really needs to be, or why you would bother. So many publishers generate useless IDs.
3. It’s difficult to compare them, because the data type of the <guid> element isn’t stable. If a certain attribute is present and contains a certain value, then the element must be treated as a string. But in other cases, the element must be treated as a URL. As we’ll see in a minute, these data types have different rules for equality, so comparing GUIDs is more difficult than it sounds.

Pilgrim and the folks working on Atom came to the conclusion that a unique id should be on every item – atom:id - and that it must be formed according to a spec that ensures canonical uniqueness. If you have an Atom feed to work with, this is a reliable mechanism.

For better or worse, however, the adoption of Atom has been slow and the market still overwhelmingly uses RSS. For aggregators like Andy (or any feed reader) who have a mixed bag of feeds, they are forced to kludge their way to uniqueness.

Update:  James Holderness reminded me of his excellent discussion on this issue from last year:  RSS Duplicate Detection.