A Detailed Explanation Of How Google Search Works
Google Search is a fully automated search engine that uses web crawlers, or software, that browses the internet regularly to discover pages that can be added to our database. Most sites in our search results aren't manually added to our index. Instead, they are found and added automatically as our web crawlers search the internet. This document provides how Google Search is used concerning your website. Knowing this basic information will help you resolve problems with crawling, get your site indexed, and understand how to optimize how your site will appear in Google Search.
Some Notes To Make Before We Begin
Before we dive deep into the intricacies of how search works, it is essential to know that Google does not accept payments to crawl a website more often or make it rank higher. If someone claims otherwise, they're lying.
Google cannot assure you that it will crawl, index, or serve your site, regardless of whether your site is in line with the guidelines and policies of Google for owners of websites.
Google Search Has Three Levels:
Google Search works in three stages. Not every page can make it through each step.
Crawling
Google downloads images, text, and videos from web pages it finds on the internet using automated programs known as crawlers.
Indexing
Google analyses the text, images, and video files displayed on the webpage and then stores all the information inside the Google index, which is a vast database.
Serving Results From Searches
When a user types in a query through Google, Google returns information pertinent to the question.
Crawling
The first step is to find the pages that are on the internet. There isn't a single index of all the pages on the internet, so Google has to constantly search for new and up-to-date pages to include them in its list of known websites. This process is known as "URL discovery." Certain pages are recognizable because Google has visited them before. Other pages are found by Google links from a well-known page to a brand new page, for instance, hub pages like an article page or hyperlinks to a blog post. Another type of page is found when you provide the list of websites (a sitemap) to Google to search.
When Google detects a webpage's URL, it can go to (or "crawl") the page to see what's there. Google uses a vast collection of computers to search billions of pages across the internet. The program responsible for fetching is Googlebot (a bot, robot, or spider). The Googlebot utilizes an algorithmic method to determine which websites to visit, how often, and the number of pages it will get from each site. Google's crawlers are also designed to not crawl the site too quickly to prevent overloading. This relies on results to the site (for instance, HTTP 500 errors refer to "slow down") and the settings within the Search Console.
But, Googlebot isn't able to crawl the entire pages that it discovers. Certain pages might not be allowed to be crawled by the website owner, while other pages might not be accessible if you sign in to the site, and some pages could be duplicates of visited pages. For instance, many sites can be accessed via the web (www.example.com) and other than the www (example.com) version of the domain's name, even though the content is identical for both versions.
When crawling, Google renders the page and runs the JavaScript it discovers with the current version of Chrome, much like how your browser renders pages that you browse. It is essential to generate since websites often depend on JavaScript for bringing content onto the page. Without rendering, Google may not be able to see the content.
Crawling will depend on the ability of Google's crawlers to access the website. The most frequent issues related to Googlebot accessing sites include:
- Problems with the server that handles the website.
- Network Issues.
- The robots.txt directives block Googlebot from accessing the page.
Indexing
After a page has been crawled, Google will attempt to determine what the content of the page is. This is known as indexing. It involves analyzing and processing the textual content and important attributes and tags of the content, like elements and alt attributes, images, videos, and much more.
In indexing, Google determines if a page is an actual duplicate of a web page or is canonical. The canonical page is the one that appears in the results of a search. To choose the canonical page, you first have to group the pages we've discovered on the internet that contain similar content. Then, we pick one more representative of the entire group. The other pages within the group are variants that can be used in different ways, for instance, when a user is browsing using a mobile device or searching for a specific webpage from the cluster.
Google also collects signals regarding the canonical website and its content, which can be used at the next stage when we display the site in search results. Specific signals are the language used by that page's language, the nation the content is located in, and the accessibility of the site, among other factors.
The information gathered regarding the canonical page and its cluster might be kept within the Google index, a massive database hosted on thousands of computers. The indexing process isn't 100% guaranteed. Every page Google processes will be included in the index.
Indexing is also dependent on the contents of the page as well as the metadata associated with it. Common indexing issues are:
- The quality of the content on this page is poor.
- Robots' meta directives disallow indexing.
- The style of the website could hinder indexing.
Search results for Serving
When a user types in an inquiry, our systems scan the index for matching pages and then return the results, which we believe to be the most high-quality and relevant for the user. Relevance is determined by various variables, which can include details such as the location of the user, their language, and the device (desktop or mobile). For instance, a search in the search engine for "bicycle repair shops" would give different results to users in Paris compared to those from Hong Kong.
The Search Console may inform you that a webpage has been indexed, but you won't find it in the search results. This could be due to:
- The content pages have no relevance to the users.
- The content is of poor quality.
- Robots' meta directives prevent serving.