Some stories are picked up by multiple outlets... and nothing can be done if they put the story in their own words etc,... (in fact there are times when I want all I can get on a given item; any original input in this case is worthwhile)....

but often sources are just picking up 'canned' stories/pr etc.  and posting verbatim the release... as shown in the screen capture below (this one came to my attention as I had 'caught  the first item as an example of 'premature fading'..while the  second came on the ticker much, much later from a different source)... I don't want to cut either source (they focus on different material for the most part) and don't necessarily want every similar story dropped... but I really don't need to see both when the title and the synopsis are the same ... 

This is a feature that I would like to see even in RSS readers ... remove duplicate articles... or dont show me something I have already read.  But as your title says ... its a tought nut ... or at least ... needs to be down intelligently.

A feature that clusters similar topics would solve this - it would not remove the duplicates but at least they would all be in one place with a master item on the ticker

Hmmm... interesting it  would meaning giving up variety over the short-term in order to have dups out of the way rather than popping up across time.... I'm not sure that I would want this (or certainly not all the time) If you do consider this could you perhaps make clustering/clustering a user selectable option that could easily be turned on and off depending on  exact needs at the time (for example, general alert seeking vs targeted following of a breaking hot news item)?

(yes I know there's that control issue of mine again)

haha

The Particls architecture actually already supports sub-items and item updates - the trick is to build an algorithm that determined similarities in posts that didn't cluster everything about dogs - not sure if it would work - it's for the smart programmers we have to figure out

But, like everything, u would certainly be able to turn it off hehe

Would be nice to see a solution to this problem, but, still beta, so, hopefully by release they have something for it.

Would a scan of the feed items, finding duplicate permalinks in the feeds, and then groups items with the same permalink be enough?

In the case where the permalink is exactly the same then it will work - but that would not stop 10 items from 10 different blogs reporting the same thing each coming through on the ticker