Tips on how to Set it Up Correctly and Test it After

Have you ever ever puzzled how web sites management which areas are off-limits to search engines like google and yahoo? The reply lies in a tiny but influential file referred to as ‘robots.txt’. You should utilize it to speak with the search bots that crawl your web site, however it’s essential to have a deeper understanding of their language to have the ability to use it correctly.

On this article, we are going to delve into the small print of what a robots.txt file is, methods to configure it, and methods to examine if the file is working correctly. What’s extra, we are going to present normal tips for the contents of a robots.txt file.

Let’s leap in!

What’s a robots.txt file?

A robots.txt file is a textual content doc situated within the root listing of a web site, containing data particularly supposed for search engine crawlers. It instructs them on which URLs, together with pages, information, folders, and many others., must be crawled and which of them mustn’t. Whereas the presence of this file shouldn’t be necessary for a web site’s operation of the web site, its appropriate setup is essential for efficient website positioning.

The choice to make use of robots.txt was made again in 1994 as a part of the Robotic Exclusion Normal. In accordance with Google Search Central, the first goal of this file is to not conceal internet pages from search outcomes, however as an alternative to restrict the variety of requests made by robots to websites and to cut back server load.

Typically talking, the content material of the robots.txt file must be considered as a advice for search crawlers that defines the principles for web site crawling. To entry the content material of a website’s robots.txt file, merely kind “/robots.txt” after the area identify within the browser.

How does robots.txt work?

Initially, it’s essential to notice that search engines like google and yahoo must crawl and index particular search outcomes displayed on SERPs. To perform this job, internet crawlers systematically browse the net, amassing knowledge from every webpage they encounter. The time period “spidering” is often used to explain this crawling exercise.

When crawlers attain a web site, they examine the robots.txt file, which incorporates directions on methods to crawl and index pages on the web site. If there is no such thing as a robots.txt file, or it doesn’t embrace any directives that forbid user-agent exercise, search bots will proceed to crawl different data on the location.

Why do you want robots.txt?

The first perform of the robots.txt file is to stop the scanning of pages and useful resource information, which permits the crawl funds to be allotted extra effectively. Within the overwhelming majority of instances, the robots.txt file hides data that gives no worth to web site guests and search bots. What’s extra, the robots.txt file is commonly used to enhance how effectively internet crawling assets are utilized.

Observe: Utilizing the “robots.txt disallow” directive doesn’t assure {that a} explicit webpage can be excluded from SERPs. Google reserves the best to think about varied exterior components, comparable to incoming hyperlinks, when figuring out the relevance of a webpage and its inclusion in search outcomes. To explicitly stop a web page from being listed, it is strongly recommended to make use of the “noindex” robots meta tag or the X-Robots-Tag HTTP header. Password safety can be used to stop indexing.

Optimize Crawl Price range

Crawl funds refers back to the variety of internet pages {that a} search robotic devotes to crawling a particular web site. To make use of the crawl funds extra effectively, search robots must be directed solely to an important content material on web sites and blocked from accessing unhelpful data.

Optimizing the crawl funds helps search engines like google and yahoo allocate their restricted assets effectively, leading to quicker indexing of recent content material and improved visibility in search outcomes. It’s essential to bear in mind, nonetheless, that surpassing your website’s allotted crawl capability can lead to unindexed pages in your web site, and unindexed pages can’t seem anyplace on the SERP. So contemplate your crawl funds when you have a big web site or a major share of unindexed pages.

To observe and analyze the rankings of webpages listed by Google, you should utilize the Google Rank Monitoring instrument. This instrument offers 100% correct key phrase rankings in Google and priceless insights on search quantity, SERP snippets, site visitors forecast, visibility, and far more.

Let’s contemplate a situation the place your web site has a major quantity of content material comparable to PDFs, movies, and pictures that maintain much less significance in comparison with the web site’s main content material. In such instances, you possibly can tailor your method to exclude these assets from search engine indexing, thereby optimizing the crawl funds.

For example, you should utilize the “Disallow” directive adopted by a particular file extension, comparable to “Disallow:/*.pdf,” to stop search engines like google and yahoo from crawling and indexing any PDF assets in your website. This offers an efficient technique to conceal such assets and guarantee they don’t seem to be included in search engine outcomes.

One other frequent good thing about utilizing robots.txt is its skill to deal with content-crawling points in your server, if any. For example, when you have infinite calendar scripts that will trigger issues when regularly accessed by robots, you possibly can disallow the crawling of that script by means of the robots.txt file.

You might also wonder if it’s higher to make use of robots.txt to dam affiliate hyperlinks to be able to handle your web site’s crawl funds or to make the most of the noindex tag to stop search engines like google and yahoo from indexing these hyperlinks. The reply is easy: Google is fairly good at figuring out and disregarding affiliate hyperlinks by itself. However through the use of robots.txt to disallow them, you keep management and doubtlessly preserve the crawl funds extra successfully.

Instance of robots.txt content material

Having a template with up-to-date directives may help you in making a correctly formatted robots.txt file precisely, specifying the required robots and limiting entry to related information.

Person-agent: [bot name]

Disallow: /[path to file or folder]/

Disallow: /[path to file or folder]/

Disallow: /[path to file or folder]/

Sitemap: [Sitemap URL]

Now, let’s discover a number of examples of what a robots.txt file would possibly appear like.

1. Permitting all internet crawlers entry to all content material.

Right here’s a primary instance of a robotx.txt file that grants entry to all web sites to all internet crawlers:

WizzAir robots.txt

On this instance, the “Person-agent” directive makes use of an asterisk (*) to use the directions to all internet crawlers. The “Disallow” directive is left empty, indicating that no content material is blocked. This permits all internet crawlers unrestricted entry to all elements of the web site.

2. Blocking a particular internet crawler from a particular internet web page.

The next instance specifies the entry permissions for the “Bingbot” user-agent, which is the net crawler utilized by Microsoft’s search engine, Bing. It features a listing of web site directories which are closed for scanning, in addition to a number of directories and pages which are allowed to be accessed on the web site.

Airbnb robots.txt

3. Blocking all internet crawlers from all content material.

Person-agent: *
Disallow: /

On this instance, the “Person-agent” directive nonetheless applies to all internet crawlers. Nonetheless, the “Disallow” directive makes use of a ahead slash (/) as its worth, indicating that every one content material on the web site must be blocked from entry by any internet crawler. This successfully tells all robots to not crawl any pages on the location.

Please word that blocking all internet crawlers from accessing a web site’s content material utilizing the robots.txt file is an excessive measure and isn’t really helpful normally. Web sites usually use the robots.txt file to manage entry to particular elements of their website, comparable to blocking sure directories or information, quite than blocking all content material.

Tips on how to discover robots.txt

In the case of finding the robots.txt file on a web site, there are a few strategies you should utilize:

  1. Test the area + “/robots.txt”. 

The most typical technique to discover the robots.txt file is by appending “/robots.txt” to the area identify of the web site you need to study. For instance, if the web site’s area is “,” you’ll enter “” into your internet browser’s handle bar. It will take you on to the robots.txt file if it exists on the web site.

  1. Analyze your web site utilizing automated instruments like SE Rating’s Website Audit.

One other technique to determine the presence of a robots.txt file is by using a web site audit instrument. This instrument, for instance, checks your website and offers data on whether or not you will have a robots.txt file and which pages it blocks. Overview the blocked pages to find out if they need to be blocked or if entry was by chance restricted.

To start out the audit, merely provoke the method and look forward to it to finish (you’ll obtain a notification in your inbox). Then, go to the Challenge Report, choose the Crawling block, and examine for the Robots.txt file not discovered downside.

Robots.txt file not found problem


Rating your web site in 2 minutes.

Enter any web site URL to get an in depth report on tech points and prompt options.

How search engines like google and yahoo discover your robots.txt file

Engines like google have particular mechanisms to find and entry the robots.txt file in your web site. Right here’s how they usually discover it:

1. Crawling a web site: Search engine crawlers constantly traverse the net, visiting web sites and following hyperlinks to find internet pages.

2. Requesting robots.txt: When a search engine crawler accesses a web site, it seems for the presence of a robots.txt file by including “/robots.txt” to the web site’s area. 

Observe: After efficiently importing and testing your robots.txt file, Google’s crawlers will robotically detect it and start utilizing its directions. There isn’t a want so that you can take any additional motion. Nonetheless, when you have made modifications to your robots.txt file and need to promptly replace Google’s cached model, you’ll must be taught methods to submit an up to date robots.txt file

3. Retrieving robots.txt: If a robots.txt file exists on the requested location, the crawler will obtain and parse the file to find out the crawling directives. 

4. Following directions: After acquiring the robots.txt file, the search engine crawler follows the directions outlined inside it. 

Robots.txt vs meta robots vs x-robots

Whereas the robots.txt file, robots meta tag, and X-Robots-Tag serve comparable functions by way of instructing search engine bots, they differ of their software and effectiveness.

In the case of hiding the location content material from search outcomes, relying solely on the robots.txt file might not be sufficient. As talked about above, the robots.txt file is primarily used to speak with internet crawlers and inform them about which areas of a web site they’re allowed to entry. Nonetheless, it doesn’t assure that the content material won’t be listed by search engines like google and yahoo. To stop indexing, site owners ought to make use of extra strategies.

One efficient approach is utilizing the robots meta tag, which is positioned throughout the <head> part of a web page’s HTML code. By together with a meta tag with the “noindex” directive, site owners explicitly sign search engine bots that the web page’s content material shouldn’t be listed. This methodology offers extra exact management over particular person pages and their indexing standing in comparison with the broad directives of the robots.txt file.

Right here’s an instance code snippet for stopping search engine indexing on the web page degree:

<meta identify=“robots” content material=“noindex”>

By together with this meta tag throughout the <head> part, web site homeowners can successfully talk to go looking engine bots that the content material of this explicit web page shouldn’t be listed.

You may as well make the most of the X-Robots-Tag within the website’s configuration file to additional restrict web page indexing. By specifying the URL of a web page within the X-Robots-Tag, site owners can immediately talk to go looking engine bots that the web page’s content material shouldn’t be listed. This methodology provides a further layer of management and suppleness in managing indexing at a granular degree.

To be taught extra about this matter, be certain that to learn our full information on the robots meta tag and X-Robots-Tag.

Pages and information which are normally closed off through robots.txt

1. Admin dashboard and system information.

Inside and repair information that web site directors or webmaster work together with.

2. Auxiliary pages that solely seem after particular consumer actions.

These can embrace messages that shoppers obtain after efficiently finishing an order, consumer varieties, authorization or password restoration pages.

3. Search pages.

Pages displayed after a web site customer enters a question into the location’s search field are normally closed off from search engine crawlers.  

4. Filter pages.

Outcomes which are displayed with an utilized filter (measurement, colour, producer, and many others.) are separate pages and could be checked out as duplicate content material. website positioning consultants usually stop them from being crawled until they drive site visitors for model key phrases or different goal queries. Aggregator websites could also be an exception.

5. Recordsdata of a sure format.

Recordsdata like pictures, movies, .PDF paperwork, JS information. With the assistance of robots.txt, you possibly can prohibit the scanning of particular person or extension-specific information.

Robots.txt syntax

Understanding the syntax and construction of the robots.txt file is important for site owners to manage the visibility of their internet pages on search engines like google and yahoo. Normally, the robots.txt file incorporates a algorithm that decide which information on a site or subdomain could be accessed by crawlers. These guidelines can both block or enable entry to particular file paths. By default, if not explicitly acknowledged within the robots.txt file, all information are assumed to be allowed for crawling.

The robots.txt file consists of teams, every containing a number of guidelines or directives. These guidelines are listed one per line. Every group begins with a Person-agent line that specifies the audience for the principles.

A gaggle offers the next data:

  • The consumer agent to which the principles apply.
  • The directories or information that the consumer agent is allowed to entry.
  • The directories or information that the consumer agent shouldn’t be allowed to entry.

When processing the robots.txt file, crawlers observe a top-to-bottom method. A consumer agent can solely match one rule set. If there are a number of teams focusing on the identical consumer agent, these teams are merged right into a single group earlier than being processed.

Right here’s an instance of a primary robots.txt file with two guidelines:

Person-agent: Googlebot
Disallow: /nogooglebot/

Person-agent: *
Permit: /


If you need extra exact management over internet crawler conduct, you possibly can use common expressions alongside versatile directions

A generally used image in common expressions for robots.txt information is the asterisk (*), which acts as a wildcard, representing any variation in worth. For instance, to permit entry to all URLs below a particular listing, you should utilize the sample “/instance/” in your robots.txt file. This might match URLs like “/instance/page1.html”, “/instance/subdirectory/page2.html”, and so forth, permitting the net robots to crawl these URLs.

One other image that can be utilized in common expressions for robots.txt information is the greenback signal ($), which signifies the top of the URL path.. For example, when you have a URL sample “/weblog/$”, it might solely match URLs that finish with “/weblog/”, comparable to “/weblog/” or “/class/weblog/”. It might not match URLs like “/weblog/article” or “/weblog/web page/2”.

Now, let’s have a look at totally different components of robots.txt syntax in additional element.

The Person-Agent Directive

The user-agent directive is necessary and defines the search robotic to which the principles apply. Every rule group begins with this directive if there are a number of bots.

Google has a number of bots liable for several types of content material.

  • Googlebot: crawls web sites for desktop and cellular units
  • Googlebot Picture: shows website pictures within the “Pictures” part
  • Googlebot Video: scans and shows movies
  • Googlebot Information: selects helpful and high-quality articles for the “Information” part
  • Google-InspectionTool: a URL testing instrument that mimics Googlebot by crawling each web page it’s allowed entry to
  • Google StoreBot: scans varied internet web page sorts, comparable to product particulars, cart, and checkout pages
  • Adsense: ranks a website as an advert platform by way of advert relevance

The entire listing of Google robots (consumer brokers) is accessible within the official Assist documentation.

Different search engines like google and yahoo even have their related robots, comparable to Bingbot for Bing, Slurp for Yahoo!, Baiduspider for Baidu, and lots of extra. There are over 500 varied search engine bots.


  • Person-agent: * applies to all present robots.
  • Person-agent: Googlebot applies to Google’s robotic.
  • Person-agent: Bingbot applies to Bing’s robotic.
  • Person-agent: Slurp applies to Yahoo!’s Robotic.

The Disallow Directive

Disallow is a key command that instructs search engine bots to not scan a web page, file or folder. The names of the information and folders that you simply need to prohibit entry to are indicated after the “/” image.

Instance 1. Specifying totally different parameters after Disallow.

Disallow: /hyperlink to web page disallows entry to a particular URL.

Disallow: /folder identify/ closes entry to the folder.

Disallow: /picture/ closes entry to the picture.

Disallow: /. The absence of any directions after the “/” image signifies that the location is totally closed off from scanning, which could be helpful throughout web site improvement.

Instance 2. Disabling the scanning of all .PDF information on the location.

Person-agent: Googlebot

Disallow: /*.pdf

The Permit Directive

Within the robots.txt file, the Permit directive features reverse to Disallow by granting entry to web site content material. These instructions are sometimes used collectively, particularly when you have to open entry to particular data like a photograph in a hidden media file listing.

Instance. Utilizing Permit to scan one picture in a closed album.

Specify the Permit directive with the picture URL and, in one other line, the Disallow directive together with the folder identify the place the file is situated. The order of strains is essential, as crawlers course of teams from high to backside.

Disallow: /album/

Permit: /album/picture1.jpg

The “robots.txt Permit All” directive is usually used when there are not any particular restrictions or disallowances for search engines like google and yahoo. Nonetheless, it’s essential to notice that the “Permit: /” directive shouldn’t be a essential part of the robots.txt file. In truth, some site owners select to not embrace it in any respect, relying solely on the default conduct of search engine crawlers.

The Sitemap Directive

The sitemap directive in robots.txt signifies the trail to the sitemap. This directive could be omitted if the sitemap has an ordinary identify, is situated within the root listing, and is accessible by means of the hyperlink “website identify”/sitemap.xml, much like the robots.txt file.


Sitemap: https://web

Whereas the robots.txt file is primarily used to manage the scanning of your web site, the sitemap helps search engines like google and yahoo perceive the group and hierarchy of your content material. By together with a hyperlink to your sitemap within the robots.txt file, you present search engine crawlers with a straightforward technique to find and analyze the sitemap, resulting in extra environment friendly crawling and indexing of your web site. So together with a reference to your sitemap within the robots.txt file shouldn’t be necessary, however extremely really helpful. 

Tips on how to сreate a robots.txt file

A well-crafted robots.txt file serves as the muse of technical website positioning. 

Because the file has a .txt extension, any textual content editor that helps UTF-8 encoding will suffice. The best choices are Notepad (Home windows) or TextEdit (Mac).

Most CMS platforms additionally present options for making a robots.txt file. For example, WordPress creates a digital robots.txt file by default, which could be considered on-line by appending “/robots.txt” to the web site’s area identify. Nonetheless, to change this file, you have to create your individual model. This may be carried out both by means of a plugin (e.g., Yoast or All in One website positioning Pack) or manually.

Magento and Wix, as CMS platforms, additionally robotically generate the robots.txt file, nevertheless it incorporates solely primary directions for internet crawlers. That is why it’s really helpful to make customized robots.txt directions inside these programs to precisely optimize the crawling funds. 

You may as well use instruments like SE Rating’s Robots.txt Generator to generate a customized robots.txt file primarily based on the desired data. You’ve gotten the choice to create a robots.txt file from scratch or to decide on one of many prompt choices. 

If you happen to create a robots.txt file from scratch, you possibly can personalize the file within the following methods:

  • By configuring directives for crawling permissions. 
  • By specifying particular pages and information by means of the trail parameter. 
  • By figuring out which bots ought to adhere to those directives. 

Alternatively, pre-existing robots.txt templates, together with extensively used normal and CMS directives, could be chosen. It is usually attainable to incorporate a sitemap throughout the file. This instrument saves time by offering a ready-made robots.txt file for obtain.

Doc title and measurement

The robots.txt file must be named precisely as talked about, with out using capital letters. In accordance with Google tips, the file measurement mustn’t exceed 500 KiB. Exceeding this restrict could end in partial processing, no crawling of the web site in any respect, or, conversely, full scanning of the web site’s content material.

The place to position the file

The robots.txt file should be situated on the root listing of the web site host and could be accessed through FTP. Earlier than making any modifications, it is strongly recommended to obtain the unique robots.txt file in its authentic kind.

Tips on how to examine your robots.txt file

Errors within the robots.txt file can result in the exclusion of essential pages from the search index and even render the whole website virtually invisible to search engines like google and yahoo.

You may simply examine your Robots.txt file with SE Rating’s free Robots.txt Tester. Merely enter as much as 100 URLs to check and confirm if they’re allowed for scanning. 

Alternatively, you should utilize the testing instrument in Google Search Console. Observe that the robots.txt file examine choice is lacking within the new Google Search Console interface and must be accessed immediately.

robots.txt Tester

Frequent robots.txt points

When managing your web site’s robots.txt file, a number of points can impression how search engine crawlers work together together with your website. Some commonest ones embrace:

  • Format mismatch: If the file shouldn’t be created within the .txt format, internet crawlers won’t be able to detect and analyze it.
  • Incorrect placement: Your robots.txt file must be situated within the root listing. Whether it is situated, for example, in a subfolder, search bots could fail to search out and entry it.
  • Disallow with out worth: A Disallow directive with none content material implies that bots have permission to go to any pages in your web site.
  • Clean strains within the robots.txt file: Guarantee there are not any clean strains between directives. In any other case, internet crawlers may need problem parsing the file. The one case the place are clean hyperlink is allowed is earlier than indicating a brand new Person-agent.
  • Blocking a web page in robots.txt and including a “noindex” directive: This creates conflicting indicators. Engines like google could not perceive the intent or ignore the “noindex” instruction altogether. It’s greatest to make use of both robots.txt to dam crawling or “noindex” to stop indexing, however not each concurrently.

Further instruments/studies to examine for points

There are various methods to examine your web site for possiblerobots.txt file-related points. Let’s overview probably the most extensively used ones.

1. Google Search Console.

Inside the Pages part of GSC, you’ll find priceless details about your robots.txt file.

To examine in case your web site’s robots.txt file is obstructing Googlebot from crawling a web page, observe these steps:

  • Entry the Pages part and navigate to the Not Listed class.
GSC Pages report
  • Search for the error labeled Blocked by robots.txt and choose it.
blocked by robots.txt
  • Clicking on this part will present you a listing of pages at present blocked by your web site’s robots.txt file. Be certain these are the supposed blocked pages.
URL examples blocked by robots.txt

You also needs to examine when you have the next problem on this part: Listed, although blocked by robots.txt.

Indexed though blocked by robots.txt

You may as well examine if particular person URLs are listed by pasting them into the search field in Google Search Console’s URL Inspection instrument. They may help determine pages that seem in SERPs regardless of being blocked with a Disallow directive in your robots.txt file. It could actually additionally enable you to detect potential indexing points brought on by conflicting directives or misconfigured robots.txt guidelines.

Right here’s an entire Google Search Console information on detecting and addressing indexing-related issues.

2. SE Rating’s Web site Audit

SE Rating’s Web site Audit instrument (and others prefer it) offers a complete overview of your robots.txt file, together with details about pages which are blocked by the file. It could actually additionally enable you to examine indexing and XML sitemap-related points.

To achieve priceless insights about your robots.txt file, begin by exploring the Challenge Report generated by the instrument. Amongst over 120 metrics analyzed, you’ll discover the Blocked by robots.txt parameter below the Crawling part. Clicking on it’ll show a listing of webpages blocked from crawling, together with problem descriptions and fast repair ideas.

This instrument additionally makes it simple to determine whether or not you will have added a hyperlink to the sitemap file within the robots.txt file. Merely examine the XML sitemap not present in robots.txt file standing below the identical part.

XML sitemap not found in robots.txt file status

When navigating to the Crawled Pages tab on the left-hand menu, you possibly can analyze the tech parameters of every web page individually. By making use of filters, you possibly can ​​deal with fixing crucial points on an important pages. For instance, making use of the filter Blocked by robots.txt > Sure will present all pages blocked by the file.

Crawled Pages report

website positioning greatest practices

To make sure optimum efficiency and correct indexing of your web site’s content material by internet crawlers, you will need to observe website positioning greatest practices, together with:

  1. Guarantee appropriate case utilization in robots.txt: Net crawlers interpret folder and part names with case sensitivity, so utilizing applicable case utilization is essential to keep away from confusion and guarantee correct crawling and indexing.
  2. Start every directive on a brand new line, with just one parameter per line.
  3. Keep away from utilizing areas, citation marks, or semicolons when writing directives.
  4. Use the Disallow directive to dam all information inside a particular folder or listing from crawling. This method is extra environment friendly than itemizing every file individually.
  5. Make use of common expressions for extra versatile directions when creating the robots.txt file. The asterisk (*) signifies any variation in worth, whereas the greenback signal ($) acts as a restriction and signifies the top of the URL path.
  6. Create a separate robots.txt file for every area. This establishes crawl tips for various websites individually.
  7. All the time check a robots.txt file to ensure that essential URLs should not blocked by it.


To recap, listed below are some essential takeaways concerning robots.txt information:

  • The robots.txt file serves as a tenet for robots, informing them which pages ought to and shouldn’t be crawled.
  • The robots.txt file can’t stop indexing immediately, however it will possibly affect a robotic’s choice to crawl or ignore sure paperwork or information.
  • Hiding unhelpful web site content material with the disallow directive saves the crawl funds. That is true for each multi-page and small web sites.
  • It’s essential to observe syntax guidelines to ensure that search bots to learn your robots.txt file.


Leave a Reply

Your email address will not be published. Required fields are marked *