Tuesday, February 7, 2012

How Google Test New Search Algorithms?

I read one interesting interview with Amit Singhal - the guy behind Google search algorithms. This following is highlighted what he explained about how Google test new search algorithms. 
"We have the entire web in a sandbox that only our engineers can see, and our engineers can take their new algorithm and see it change millions of queries. If it works, we send it to testers, whom we pay, but we don't tell them what they are testing.
If the tweaks are still deemed useful they are unleashed into the wild - but only to some users.
Then we take a tiny slice, one per cent of our users, and expose them to this change. We measure things such as where on the page they click, when they click higher - that's good for us.
That one per cent are not told but it's just an experimental algorithm, and the changes are potentially beneficial, so not hurting the user's experience.
Concurrently we have approximately 100 ideas floating around that people are testing - we test thousands in a year. Last year we ran around 20,000 experiments. Clearly they don't all make it out there but we run the process very scientifically.
Once the tests are done, a report is built by an independent statistician. "We look at that with a group of senior people who come together every week, and we decide if it's good for users, the web ecosystem and for our systems."
One Percent of Google users - Is it means Page Layout Algorithms is still experimental algorithms? Regarding to this blog post, Google said "This algorithmic change noticeably affects less than 1% of searches globally." How much different between number of Google users and number of searches?

Most interesting part is the last sentence : "We decide if it's good for users, the web ecosystem and for our systems." I always hear Google emphasizes on "Users" word when they are talking about search quality. 

What do you think about this?