When I signed up for the Netflix Prize, I had to pick a team name. I like the show Seinfeld, and Newman is my favorite character in the show, due to his sheer absurdity and hilariousness. So I chose "Newman!" as my team name.
Later I formed a joint team with another contestant, and we called this team "Newman and Kramer !", named after the two most absurd characters in the show. Kramer helped me with my RBM implementation and then quickly went MIA (but not presumed dead, where are you Eric ?).
Then Greg McAlphin (ADifferentName/OfADifferentKind) and I formed a new team named "Newman and George !".
Then Bill Bame (clueless) joined us and we formed a new team named "Newman, George, and Peterman !".
So at this point, we were "polluting" the leaderboard in four teams:
*Newman!
*Newman and Kramer !
*Newman and George !
*Newman, George, and Peterman !
Then Chris Hefele (chef-ele) joined us, and we were going to be "Newman, George, Peterman, and Bania !". Now obviously we had a scalability problem: the team name was growing too long. And at this point, we fully expected other people to join later, so what were we going to be in the end ? "Newman, George, Peterman, Bania, Mr Pitt, Uncle Leo, and Soup Nazi !" ?
We decided to call our new 4-member team "Vandelay Industries".
Then Jeff Howbert (howbert) joined. Shortly afterwards, the contest entered the last 30 days, and a slew of others on the leaderboard joined Vandelay Industries. The team expanded faster than Newman's waistline (the character's, not mine).
Members of Vandelay Industries created a few sub-teams to explore different blends. Later, in preparation to merge with the Grand Prize Team, Opera Solutions, and Feeds2 to form The Ensemble, we got rid of all the sub-teams including the Newman-and-whoever teams.
Except "Newman and Kramer !". Kramer is MIA but out of respect for him and his help, I left "Newman and Kramer !" on the leaderboard.
That's my journey from Newman! to The Ensemble. Here's David Lin's story of Dinosaur Planets and The Ensemble.
We Are The Borg
We are the borg. Your algorithm will be assimilated. Resistance is irrelevant.
We're also known as: xlvector, OfADifferentKind, & Newman(me).
Update: to clean up the leaderboard, we have voluntarily removed "We are the Borg" and other sub-teams of "Vandelay Industries !". The pre-Vandelay Industries team "Newman, George, and Peterman !" is gone too.
We're also known as: xlvector, OfADifferentKind, & Newman(me).
Update: to clean up the leaderboard, we have voluntarily removed "We are the Borg" and other sub-teams of "Vandelay Industries !". The pre-Vandelay Industries team "Newman, George, and Peterman !" is gone too.
The End Is Near
The end Netflix Prize is near: BPC has reached 10%.
Now, how do I apply what I've learning during the participation of this contest ? Maybe I should get into trying to predict the stock market or commodity prices, or something. People are already doing that on the stock market, it's called algorithmic trading.
Of course recommender systems and data-mining can be applied to all kinds of products and services, not just movies. For example truck dealers can try to find people who are most likely to buy trucks and send them truck advertisement; online matchmaking services can match people based on user data and history of matches.
Of course, for most products and service there are obvious "indicators" that we all know. For example, males in construction industry are probably far more likely to buy trucks than people in general. But still, there might be some subtle indicators and trends that can only be discovered by a computer algorithm, which might improve the accuracy of recommendations by 10%, as measured by RMSE. And as BellKor's research shows, even a few percentages of RMSE improvement can translate into huge increase in the quality of the recommendations you get. You'll actually like the products recommended to you, or the people.
How Can It Work For Matchmaking ?
Well, now I think about it, it probably won't work for matchmaking services. Why ? Because for matchmaking services, it will take a long time to truly know the quality of the recommendations.
Sure, you can find out about bad ones pretty soon: people go on one date and can't stand each other. But how do you know which recommendations are really good, and which ones only look good now but will end up in messy divorce 10 year later ? I think for matchmaking services, you'll have to wait for a generation to tell which recommendations are truly good. But by that time, society and people in general would have chance a lot, so whatever worked 20 years ago probably won't work so well today.
So for matchmaking services, the value of recommender systems is probably to filter out potential incompatible dates.
Now, how do I apply what I've learning during the participation of this contest ? Maybe I should get into trying to predict the stock market or commodity prices, or something. People are already doing that on the stock market, it's called algorithmic trading.
Of course recommender systems and data-mining can be applied to all kinds of products and services, not just movies. For example truck dealers can try to find people who are most likely to buy trucks and send them truck advertisement; online matchmaking services can match people based on user data and history of matches.
Of course, for most products and service there are obvious "indicators" that we all know. For example, males in construction industry are probably far more likely to buy trucks than people in general. But still, there might be some subtle indicators and trends that can only be discovered by a computer algorithm, which might improve the accuracy of recommendations by 10%, as measured by RMSE. And as BellKor's research shows, even a few percentages of RMSE improvement can translate into huge increase in the quality of the recommendations you get. You'll actually like the products recommended to you, or the people.
How Can It Work For Matchmaking ?
Well, now I think about it, it probably won't work for matchmaking services. Why ? Because for matchmaking services, it will take a long time to truly know the quality of the recommendations.
Sure, you can find out about bad ones pretty soon: people go on one date and can't stand each other. But how do you know which recommendations are really good, and which ones only look good now but will end up in messy divorce 10 year later ? I think for matchmaking services, you'll have to wait for a generation to tell which recommendations are truly good. But by that time, society and people in general would have chance a lot, so whatever worked 20 years ago probably won't work so well today.
So for matchmaking services, the value of recommender systems is probably to filter out potential incompatible dates.
Subscribe to:
Posts (Atom)