Archive for Google

Google Human Raters – How Processes and 7 Biases influence results

Blogs, Googleon November 23rd, 2012No Comments

Google Search results are not entirely based on algorithms. Human Raters play a crucial role in evaluating algorithm changes.  They have to first understand the query type – navigational, informational & transactional. Then they have to evaluate the pages in the search results for relevance and the value it offers to the user. Based on relevance, Human Raters (Quality Raters) have to rate the pages as Vital, Useful, Relevant, Not Relevant, or Off-Topic. Vital rating is mostly seen in navigational searches like if I search for “bestbuy cameras”, my intent is to browse the cameras available in bestbuy. Here are the top results and the most likely evaluation that a Human Rater might perform.

Best Buy Camera Query Evaluation by Human Raters


Best Buy Camera Query Example

Before an algorithm change is made live, the search quality team follows this process:

1) Search Result, Not Satisfactory: Search Engineers notices that the search result in a particular query space is not satisfactory.

2) Idea from Engineer to Improve Results:  Engineers provide ideas to improve search results

3) Implement Idea in a Sandbox: The ideas are then implemented in a Sandbox or a test environment.

4) Sent the before and After Implementation: The search results before and after implementing the idea is shown side by side to Human Raters. Please note that Human raters will have no idea whether the results are before or after the algorithm change.

5) Send fraction of the Traffic to the new Results: A fraction of live traffic is sent to results with the algorithm change to evaluate how users are responding to the new results and whether the results are answering user’s questions and providing a better user experience.

6) Analyst Evaluate Results: The Analyst evaluates the real time response to algorithm changes and rating by Human Raters and a comprehensive report is sent to a Search Committee.

7) Result Accepted or Rejected: A detailed discussion follows and the algorithm changes are accepted or rejected.

On rejecting the change, the results for the query space remains the same and engineers will have to come up with another idea to tweak the algorithm.

Although the process seems foolproof, many factors can influence the quality of search results.

1) Design Bias: When a human rater evaluates a website, the design of the page plays a major role in evaluating whether the website is current or not. There are several WordPress or blogger themes that might look primitive.  First impression matters and it is highly likely that web design can create false notion of quality.

2) Domain Bias: Another major factor in evaluating the quality of the page is the domain of the site. A webpage hosted in .biz or .info Domain is not likely to give the same impression as a .org or a .com page.

3) Knowledge Bias: Human raters are selected after a rigorous recruitment process and they are allowed to do rating for one year, after which they have to wait for 3 months to reapply.  But if your job is to evaluate query space for relevance on a regular basis, it is likely that you will learn more about SEO and Internet marketing. With knowledge comes biases, mostly against certain thought leaders and if the query space is related to the rater’s educational background or SEO, the biases will come into play during the evaluation process.

4) Amateur Bias: Like with knowledge, amateur raters will not be able to differentiate between an expert site with original content and a site that is good at copying and deriving content from experts. Also, raters without the knowledge about an industry cannot spot spam unless the spamming site is using obvious spamming techniques like keyword stuffing.

5) Intent Bias: Understanding relevance is completely different from understanding intent. For short keywords, it becomes extremely difficult to understand intent unless you have expertise in that industry. Human raters cannot provide value in query spaces that are technical. 

6) Relevance Bias: Content Scrappers might rank in the first page for certain keywords than original content creators, who might be ranked in the fourth page. Human raters are likely to rate the results for the 1st page and might look into pages 2 and 3. They might miss the original page that should have been ranked in the first page. Since Human raters are asked to evaluate the relevance for a particular keyword, they are unlikely to look for original content and based on content, scrapper site might rank better than the original site.

7) Cultural Bias: Although raters are recruited from all around the world, biases against cultures are ingrained in the psyche. How often we associate quality with German Engineering, outsourcing with India and cheap goods with China.  Humans tend to generalize cultural traits, and it can affect how raters evaluate a page.

Although the Human Rater training material shows that a thorough understanding of relevance is required before evaluation, the knowledge or lack of it about a query space can help or hamper how the search results are evaluated.

In the following video, Matt Cutts explains how Human Raters are used in Web Search

To see how algorithm changes are implemented, watch the following video.

Google URL Shortner

Blogs, Googleon June 20th, 2012No Comments

Google URL shortener ( is just like You can access information about referrers, countries of visitors that have clicked your URL, Browser type and Platforms used by your visitor with the URL. But like in links, the information would be public and can be accessed by other users having a account.


Google URL Shortener

Get your Google Shortner

What is Google PageRank?

Blogs, Googleon May 10th, 2012No Comments

Google Page Rank are logarithmic value given to each web page based on its importance. The importance is determined by the number of citation(links) that a page receives both internally(from the same website) and externally(from Authoritative and New Websites). Earlier page rank was defined simplistically

Google Page Rank Simplistic Model

As you can see, a web page with a page rank 100 divides its value equally with two linked pages But this led to a situation where if pages are linked in such a way that it goes into an infinite loop(like below) then all the pages will have a page rank of 100. 

Google Page Rank Infinite Loop Problem

To avoid this, Larry Page introduced a concept called 'Damping factor', which dissipates 15% Page Rank value with each link. The page rank is based on random surfer model, which states that even for random visitors browsing through pages, with each click the probability to click another link is at 0.85 (losing 15% of page rank value with each click) read more

Google introduces Event Goals

Analytics, Blogs, Googleon April 6th, 2011No Comments

On April 6th 2011, Google announced the new Event Goals feature. The new feature is still in Beta and would be available only to a handful of Google Analytics users. 

How Event Goals can help you?

Although we were able to track downloads in zip format or pdf by segmenting the interactions that our users have with the website, the new Events Goal will enable webmasters to track ajax and javascript events(I guess Google was listening to our previous post on Problem with Google Analytics).

As mentioned in "Problem with Google Analytics' post, Google has yet to address the problem of tracking events in Goal Funnels. But there are some welcome additions with the new feature. One good thing that we have noticed with the Event Goal is the introduction of Event Value. Earlier you have to enter constant value  like:

Google Analytics Goal Value

But the new Events Goal will give you an additional option to use Event Value as Goal Value.

Google Analytics Event Value

Google Team has also introduced filtering Goals on Event Category, Action, Label and Value.

Now you don't have to create multiple Goals to match your Sales value. You can use the actual event value instead. 

Read more about the new Feature from the Google Analytics Team

Content Protected Using Blog Protector By: PcDrome.