How Quality Raters Make Google Search Results Better

Recently, Google’s Public Liaison for Search, Danny Sullivan, explains the role of search quality raters and how their evaluations are integrated into algorithm updates. A detailed explainer is published on how changes to search rankings are assessed internally before being rolled out to users.

The world’s most used search engine, Google, enjoys more than 70% of market share and nearly 85% of mobile traffic. The company’s rise to the top was a result of years of fine-tuning search rankings. At the core, its focus is on providing high-quality results that translate to satisfied users. Each year, the search engine giant rolls out hundreds of algorithm changes that range from small to huge. In the midst of all the tech-driven changes, there are actual human beings, known as Search Quality Raters that evaluate search results and rate the quality of top-ranking pages. There are more than 10,000 of these individuals scattered across the world who play a key role in helping Google fine-tune its search with their valuable feedback.

One of the major ways for incorporating feedback – from quality raters and regular people in algorithm updates – is by “Research”. As Google’s ultimate goal is to make information universally accessible and helpful, they have a research team that communicates to people around the world to understand how Search can be more useful. On Google’s official blog post, Sullivan says that: ‘We invite them to give feedback on different iterations of our projects and we do field research to understand how people in different communities access information online.’

The Responsibilities of Search Quality Raters

Though many signals are picked up by Google’s algorithms automatically, when it comes to signals like relevance and trustworthiness, human judgment is highly required. And for that, Google has a swarm of people referred to as “search quality raters”, who evaluate how people are likely to experience search results.

Google’s third-party search quality raters are spread around the globe. To evaluate search results more effectively, they are highly trained using the company’s publicly available rater guidelines – that describe how its algorithms intend to surface content. The 160 pages guidelines are all about how search is designed to return relevant results from the most reliable sources available.

Search Quality Raters do their job by evaluating two versions of search results: one from the current version of Google and the other from an improved version (set of results Google considers to need improvement). Based on the guidelines, the raters may conduct research and then provide ratings for each page. Raters are also asked to do reputational research on the sources to evaluate certain things like expertise, authoritativeness, and trustworthiness. Their feedback helps the company learn what changes they need to implement to make Google an even more effective search engine.

The Impact of Search Quality Raters on Rankings

Despite the important part they play to give users better search results, search quality raters do not directly impact rankings. Their feedback is just a means to ensure that Google returns the best quality search results possible.

According to Mr. Sullivan, once search quality raters have done their research, they provide a quality rating for each page. It’s important to note that the rating does not directly impact how a page or site ranks in search”. Specifically, pages are not assigned ratings as a way to determine how well to rank them as that might be an impossible task and a poor signal for Google to use. With billions of pages that are constantly changing, there’s no way humans could evaluate every page on a recurring basis.

Instead, search ratings are considered a data point – when taken in aggregate – it helps Google measure how well their systems are working to deliver great content that’s associated with how people around the world evaluate information.

In 2019, Google did more than 383,605 search quality tests and 62,937 side-by-side experiments with its search quality raters to measure its quality. It helps Google to make more than 3,600 improvements to its search algorithms. 

In addition to using the services of search quality raters, Google also conducts live experiments by having a small proportion of random people use new features. It then looks at the metrics to see how impactful the features were. In 2019, it conducted over 17,000 live traffic experiments.

Thanks to such tests and improvements, as over the past two decades, these research and evaluation processes have proven to be very effective. It helps Google to dominate search and continue to be a favorite among users.