Googlebot is Google’s web crawler, an automated program that discovers and accesses webpages across the internet. It follows links, reads website resources and revisits known URLs to find new or updated content.
The process generally works in three stages:
- Crawling – Googlebot discovers URLs through links, XML sitemaps and previously known pages, then requests accessible content
- Indexing – Google analyses the crawled page, including its text, images, videos, links, canonical information and other elements, and decides whether to store it in the Google index
- Serving Search Results – When someone searches, Google’s ranking systems evaluate relevant indexed pages and determine which results to display
Google primarily uses Googlebot Smartphone for Search crawling because its transition to mobile-first indexing is complete.
However, being crawled does not guarantee that a page will be indexed, and being indexed does not guarantee that it will rank highly.
Last Updated: 27.08.2026
How Does Google Decide Which Search Results To Display?
Google does not simply look for pages containing the exact words someone types into the search box.
It uses automated ranking systems to examine pages in its index and determine which results are likely to be the most relevant and useful for each search.
Google says these systems consider many different factors and signals across hundreds of billions of web pages and other content.
Rankings can therefore change depending on the query, the content available and the context of the person searching.
How Google’s Ranking Systems Work?
Google uses several ranking systems rather than one single algorithm. Some are part of its core ranking systems, while others are designed for particular types of searches.
Examples include systems that help Google understand:
- Meaning And Intent – Google attempts to determine what the searcher actually wants rather than relying entirely on exact keyword matching
- Content Relevance – The systems analyse whether a page contains information that is relevant to the query
- Links Between Pages – Google uses link analysis systems, including PageRank, to understand relationships between pages
- Freshness – Queries where users are likely to expect recent information may receive fresher results
- Original Content – Google has systems designed to surface original content rather than pages that simply reproduce existing information
- Location And Language – Search results can differ according to the user’s location and language
- Device – The type of device being used can influence which results and Search features are most appropriate
Google’s Search ranking systems documentation explains that individual pages are normally assessed using a combination of page-level signals, although site-wide signals can also contribute.
Do Websites Need To Satisfy A Fixed Number Of Ranking Factors?
There is no useful checklist where satisfying a certain number of factors automatically earns a first-page ranking.
Google has previously been associated with claims about “200 ranking factors”, but its current documentation is better understood as describing many signals and multiple ranking systems rather than a fixed SEO checklist.
A webpage should therefore focus on satisfying the searcher’s needs rather than attempting to mechanically optimise for a particular number of supposed ranking factors.
Are Google Ads Connected To Organic Rankings?
Google Ads and Google’s organic Search rankings operate separately.
Paying to advertise through Google does not make Googlebot crawl a website more frequently and does not improve the website’s organic position.
Google explicitly states that it does not accept payment to crawl a website more frequently or rank it higher in its organic results.
This means businesses should treat paid search and SEO as separate marketing channels, even though advertisements and organic results can appear on the same search results page.
2. How Does Googlebot Crawl And Index Web Content?
For a webpage to have an opportunity to appear in Google Search, Google normally needs to discover and process it first.
Google describes Search as having three main stages:
- Crawling – Google discovers URLs and downloads content using automated crawlers
- Indexing – Google analyses the page and may store information about it in its index
- Serving Results – Google’s systems select relevant indexed content when somebody performs a search
Not every URL successfully moves through all three stages. A page being discovered or crawled does not guarantee that Google will index it.
How Does Google Discover New URLs?
There is no central database containing every webpage on the internet. Google therefore constantly discovers new and updated URLs.
It can find pages through sources including:
- Internal Links – Links from pages Google already knows about can lead its crawler to new pages
- External Links – Links from other websites can help Google discover previously unknown URLs
- XML Sitemaps – Website owners can provide lists of important URLs through their sitemap
- Previously Known URLs – Google can return to pages that it has crawled before to look for changes
Clear website architecture is important because pages that cannot be reached through crawlable links can be more difficult for search engines to discover.
What Happens When Google Crawls A Page?
Once Google discovers a URL, Googlebot may request the page from the website’s server.
Googlebot determines which sites to crawl, how frequently to visit them and how many URLs to request. Google also attempts to avoid putting unnecessary pressure on web servers.
During crawling, Google can download text, Images and videos. It can also render webpages and process JavaScript using technology based on a recent version of Chrome.
However, Googlebot does not necessarily crawl every URL it discovers.
Crawling may be affected by:
- txt Restrictions
- Server Problems
- Network Errors
- Pages Requiring A Login
- Repeated HTTP Errors
- Website Capacity
If a server repeatedly responds slowly or produces errors such as HTTP 500 responses, Google’s systems can reduce crawling automatically.
What Happens During Indexing?
After crawling a page, Google attempts to understand its content.
It can analyse elements including:
- Main Page Content
- Title Elements
- Images
- Videos
- Alt Attributes
- Links
- Canonical Information
- Language
- Page Quality
Google may also identify groups of similar or duplicate URLs and choose one as the canonical version.
Importantly, indexing is not guaranteed. A page can be successfully crawled without being added to Google’s index.
Do XML Sitemaps Guarantee Indexing?
No. An XML sitemap helps Google discover URLs but does not guarantee that those URLs will be crawled, indexed or ranked.
Sitemaps are particularly useful for large websites, new websites and sites where some important pages are difficult to discover through normal links.
Website owners can also use the URL Inspection tool in Google Search Console to investigate individual URLs and request indexing.
Can Any Website Use The Google Indexing API?
The Google Indexing API is not a general tool for submitting ordinary website pages or blog posts.
Google currently limits the Indexing API to pages containing particular types of structured data, primarily:
- JobPosting
- BroadcastEvent Embedded In VideoObject
For standard web pages, Google recommends normal discovery methods such as crawlable links, XML sitemaps and Search Console.
3. How Can You Control Googlebot Crawling And Indexing?
Crawling and indexing are different processes, so website owners need to choose the correct method depending on what they are trying to achieve.
Blocking Googlebot from crawling a URL does not necessarily mean that the URL can never appear in Search.
How Does Robots.txt Control Crawling?
A robots.txt file gives crawlers instructions about which URLs they are allowed to access.
It is mainly useful for managing crawler traffic and preventing crawlers from spending resources on areas of a site that do not need to be crawled.
For example, a website might restrict crawling of certain:
- Filtered URLs
- Search Result Pages
- Duplicate URL Patterns
- Unimportant Resources
However, robots.txt is not designed to guarantee removal from Google Search.
Google explains in its robots.txt guidance that a blocked URL can potentially still appear in Search when Google discovers that URL from another source.
When Should You Use Noindex?
If a publicly accessible webpage should not appear in Google Search, a noindex directive is generally more appropriate.
It can be implemented through:
- Robots Meta Tag
- HTTP Response Header
Google needs to crawl the URL before it can see the noindex directive.
For this reason, website owners should normally avoid simultaneously blocking a URL through robots.txt and expecting Google to discover a noindex directive on that same page. If crawling is blocked, Google may be unable to see the instruction.
What Does Nofollow Actually Do?
A link containing rel=”nofollow” gives Google information about the relationship between the linking page and the destination.
It should not be treated as an absolute method of preventing a URL from being crawled.
Google may still discover the destination through:
- Another Internal Link
- An External Website
- An XML Sitemap
- Previously Discovered URLs
If preventing access to a page is important, proper access restrictions should be used instead.
Can Password Protection Stop Googlebot?
Yes. Content requiring authentication normally cannot be accessed by Googlebot.
Password protection is therefore more appropriate than robots.txt when information genuinely needs to remain private.
Private customer portals, account dashboards and confidential documents should rely on proper access controls rather than crawler instructions.
How Does The Search Console Removals Tool Work?
Google Search Console includes a Removals tool that website owners can use when they need to quickly hide certain content from Google Search.
However, this is primarily a temporary removal mechanism.
For long-term removal, the underlying URL should normally be:
- Deleted
- Password Protected
- Given A Noindex Directive
- Otherwise Made Permanently Unavailable
The appropriate method depends on whether the page should remain available to normal visitors.
Can You Still Change Googlebot’s Crawl Rate?
The old Search Console Crawl Rate Limiter is no longer available.
Google retired the tool in January 2024 because its crawling systems had improved and could automatically respond to website and server conditions.
Website owners should therefore concentrate on maintaining reliable hosting, appropriate HTTP responses and sensible crawl controls rather than attempting to manually set Googlebot’s crawl speed.
4. How Can You Improve Your Website’s Crawlability?
Improving crawlability means making it easier for Googlebot to discover, access and understand important pages without wasting crawling resources on unnecessary URLs.
One of the most important changes to understand is that Google Search now relies primarily on the smartphone version of website content.
Google completed the wider transition to mobile-first indexing in 2023 and moved the remaining Search crawling to Googlebot Smartphone after 5 July 2024.
Make Important Content Accessible To Googlebot Smartphone
Google uses the mobile version of a site’s content for indexing and ranking.
Important information should therefore be accessible when Googlebot Smartphone visits the page.
Website owners should check that important:
- Text
- Images
- Videos
- Links
- Structured Data
- Metadata
are available to the mobile crawler.
Googlebot Desktop may still appear in server logs for certain specialised Google services, but normal Search crawling is primarily performed using Googlebot Smartphone.
Use Crawlable Internal Links
Internal links help Google discover pages and understand relationships between different sections of a website.
Important pages should not exist as isolated URLs that users and crawlers can reach only by entering the address manually.
A clear structure could connect:
Homepage → Category → Subcategory → Article
Useful internal linking can also help Google understand which pages are important within a website.
However, adding large numbers of unnecessary links simply to attract Googlebot is unlikely to improve SEO. Internal links should primarily help users navigate relevant content.
Maintain An Accurate XML Sitemap
An XML sitemap can provide Google with a structured list of URLs that the website owner considers important.
Where appropriate, the sitemap can also include accurate modification dates.
Keep the sitemap clean by avoiding unnecessary inclusion of:
- Broken URLs
- Redirected URLs
- Duplicate URLs
- Noindex Pages
- Unimportant Parameter URLs
A sitemap should complement good internal linking rather than replace it.
Avoid Accidentally Blocking Important Pages
Small technical configuration errors can prevent important pages from being crawled or indexed.
Regularly check:
- txt Rules
- Robots Meta Directives
- Canonical Tags
- HTTP Status Codes
- Authentication Requirements
- JavaScript Rendering
- Internal Links
A single incorrect noindex directive can prevent an otherwise useful page from appearing in Google Search.
Keep Important Mobile Content Accessible
A mobile page does not have to look exactly the same as its desktop version.
Layouts can change to accommodate smaller screens, menus can collapse and design elements can adapt.
The important issue is whether Googlebot Smartphone can access the primary content required to understand the page.
Google’s mobile-first indexing best practices recommend ensuring that important mobile content, structured data and other SEO elements are accessible to the smartphone crawler.
Improve Server Reliability And Response Times
Google adjusts its crawling according to how a website’s server responds.
Persistent server errors can encourage Googlebot to reduce its crawling activity.
Website owners should monitor issues such as:
- HTTP 5xx Errors
- Server Timeouts
- DNS Problems
- Very Slow Responses
- Periods Of Downtime
Fast and reliable hosting benefits visitors as well as search engine crawling.
Use URL Inspection To Diagnose Problems
Google Search Console’s URL Inspection tool can help website owners understand how Google sees individual pages.
It can provide information about:
- Indexing Status
- Last Crawl
- Canonical Selection
- Crawl Availability
- Mobile Crawling
- Page Discovery
It can also be used to request another crawl after important changes.
A request does not guarantee immediate crawling or indexing, but it can be useful after fixing a technical problem or publishing an important update.
5. How Can You Verify Whether A Crawler Is Really Googlebot?
Seeing “Googlebot” in a server request does not automatically prove that the visitor is Google’s genuine crawler.
User-agent strings can be copied or impersonated.
For websites where crawler verification matters, particularly those dealing with large volumes of automated traffic, it can be useful to distinguish genuine Google requests from bots pretending to be Googlebot.
Why Is The User-Agent Alone Not Enough?
Googlebot normally identifies itself using a recognisable HTTP user-agent.
However, another crawler can send a request containing the same wording.
This means website owners should not automatically whitelist an unknown visitor simply because its user-agent contains “Googlebot”.
How Can Genuine Googlebot Traffic Be Verified?
Google provides methods for verifying whether requests originate from its crawlers.
Website administrators can use technical verification methods and Google’s published information about crawler IP ranges rather than relying entirely on the visible user-agent.
This can be particularly useful when investigating:
- Unusual Server Traffic
- High Crawling Activity
- Security Logs
- Suspected Fake Bots
- Crawler Blocking Rules
Care should be taken before blocking genuine Googlebot because preventing it from accessing important content can affect crawling and indexing.
What Is Google-Extended?
Google-Extended is a product token that publishers can use in robots.txt to manage whether Google can use crawled website content for certain Gemini-related purposes.
It is different from the standard Googlebot control used for Google Search.
Google has stated that using Google-Extended does not determine whether content can appear or rank in Google Search.
That distinction matters as Google operates multiple crawlers, fetchers and product-specific systems.
Googlebot Vs Google’s Other Crawlers
Not every request from Google is necessarily the standard Googlebot used for Search.
Google operates different crawlers and user-agent tokens for different products and services.
For SEO purposes, website owners should understand which crawler they are controlling before creating broad robots.txt restrictions.
Blocking everything associated with Google without understanding the crawler’s purpose could unintentionally affect services that the website actually wants to use.
Conclusion
Googlebot plays an important role in helping Google discover and access web content, but crawling alone does not guarantee visibility in Search.
Google first needs to discover and crawl a URL, then decide whether to index it. Only after that can its ranking systems determine whether the page is sufficiently relevant and useful to appear for a particular search.
Website owners should therefore concentrate on:
- Making Important Pages Crawlable – Ensure Googlebot Smartphone can access the content
- Creating Clear Internal Links – Help users and crawlers discover related pages
- Maintaining XML Sitemaps – Keep important canonical URLs easy to discover
- Using Robots.txt Correctly – Control crawling rather than treating it as an indexing tool
- Using Noindex Correctly – Prevent unwanted pages from appearing in Search
- Monitoring Technical Errors – Resolve server, rendering and indexing problems
- Checking Search Console – Investigate how Google discovers and processes important URLs
The technical details surrounding Googlebot have changed significantly over time, particularly with the completion of mobile-first indexing and the retirement of manual crawl-rate controls. Keeping crawling and indexing practices aligned with Google’s current guidance gives websites a much stronger technical foundation for Search.
Frequently Asked Questions
What Is Googlebot?
Googlebot is the general name for Google’s web crawlers used by Google Search. Its two main variants are Googlebot Smartphone and Googlebot Desktop, with Smartphone handling most Search crawling.
Does Googlebot Automatically Index Every Page It Crawls?
No. Google can crawl a webpage and still decide not to add it to its index. Crawling, indexing and ranking are separate stages.
Can Robots.txt Stop A Page Appearing In Google?
Not reliably. Robots.txt primarily controls crawling, and a blocked URL may still be discovered from other sources. Use an appropriate noindex directive or access restriction when exclusion from Search is required.
Can I Still Change Googlebot’s Crawl Rate In Search Console?
No. Google retired its Search Console Crawl Rate Limiter in January 2024 and now manages Googlebot’s crawl rate mainly through automated systems.
Does Nofollow Stop Google From Crawling A Link?
Not necessarily. Nofollow provides Google with information about a link relationship, but the destination URL may still be discovered through other links, sitemaps or previous crawling.
Does Google Still Use Mobile-First Indexing?
Yes. Google primarily uses the mobile version of website content for indexing and ranking, with Googlebot Smartphone handling the majority of Search crawling.
Can Every Website Use Google’s Indexing API?
No. Google’s Indexing API is restricted to specific content types such as pages containing JobPosting or qualifying BroadcastEvent structured data. Standard webpages should rely on crawlable links, sitemaps and Search Console.
