Is collaborative filtering (the generic term for the techniques by which websites decide what to recommend to you based on the behavior of others) getting better or worse? Looking at a recent email from Amazon, I was first impressed by the diversity of themes they correctly identified as interesting to me....

...Then I looked a little deeper and saw that each choice was a pretty straightforward extrapolation from something I'd purchased before, rather than a surprising recombination of my tastes and those of other customers.  (Ok, except for the C++ Primer--no clue.) Curious, dear reader, what your experience has been lately: are iTunes, Amazon, CNET, et al bringing interesting surprises to your attention?

[edit to add: Before I could get the foregoing posted, Netflix answered the question. On Oct. 2, it announced the creation of the "Netflix Prize," an award of $1 million to the first person who can improve Netflix's recommendation process by 10% or more. The company's real ante is that it'll make 100 million of their customers' movie ratings available for analysis--a boon to students of collaborative filtering everywhere.

Netflix's current system is highly regarded by filtering experts, but improvement has stalled and the company is turning to the biggest possible network for help. Said CEO Reed Hastings to the New York Times: "If we knew how to [keep improving the system ourselves]," CEO Reed Hastings told the New York Times, "we'd have done it already. We're pretty darn good at this now. We've been doing it a long time."

In other words, the insider pros have had their crack. Now it's everybody's turn.]