Published: 2022-09-23
Views: 372
Author: Writer
Published in: SEO & SEM

A robots.txt file informs search engine crawlers which URLs they can access on your website. This file is used to prevent overloading your site by requests. It is not an algorithm for keeping a page from Google.

You can block indexing and password-protect a page to keep it out of Google.

What's a robots.txt File used for?

A robots.txt is used to manage crawler traffic to your website and, usually, to keep a file off Google depending on the type of file

Make a robots.txt document

To create robots.txt files, you can use any text editor. You can use TextEdit to create robots.txt file.

Avoid using a word processor. Word processors can save files in proprietary formats and add unanticipated characters such as curly quotations, which can be problematic for crawlers.

If you are prompted to do so, ensure that the file is saved with UTF-8 encryption.

Format and location rules

The file must be called robots.txt

Only one robots.txt can be added to a site.

The robots.txt must be located at the root of the website host to which it is applied. For instance, to control crawling on all URLs below https://www.example.com/, the robots.txt file must be located at https://www.example.com/robots.txt

It cannot be placed in a subdirectory (for example, at https://example.com/pages/robots.txt). Contact your web hosting provider if you have any questions about accessing your website root or require permissions.

You can block access to your website root using meta tags.

A robots.txt file can apply to subdomains (for example, https://website.example.com/robots.txt) or on non-standard ports (for example, http://example.com:8181/robots.txt).

Robots.txt files must be UTF-8 encoded text files (which include ASCII). Google could ignore characters not within the UTF-8 range and render robots.txt rules inapplicable.

Why is Robots.txt so important?

A robots.txt file is not required by most websites.

Google can index most pages on your website.

They won't index pages that aren’t important or duplicates of pages.

There are three main reasons you would want to use a robots.txt.

Block non-public pages:

There are some pages that you don’t want to be indexed. You might have a staging page. Or a login page. These pages must exist.

However, you don't want random visitors landing on them. You can use robots.txt, in this case, to stop search engine crawlers from finding these pages.

Maximize your Crawl Budget Googlebot can use more of its crawl budget to block pages that aren't important by using robots.txt.

Meta directives can be used to prevent pages from being indexed. Meta directives are not compatible with multimedia resources like images and PDFs. Robots.txt is the solution.

Google and other search engines crawl the web constantly in search for new data to use as sources for their search results. Search engine crawlers (also known as search engine bots) can use the robots.txt file to tell them which pages they should request to view from your online shop. Shopify stores all have a default robots.txt that is optimized for Search Engine Optimization (SEO).

A robot searches for "/robots.txt", which is the URL file. It removes the path component (everything after the first single slash) and places "/robots.txt".

For example, for "http://www.example.com/shop/index.html, it will remove the "/shop/index.html", and replace it with "/robots.txt", and will end up with "http://www.example.com/robots.txt".

For the URL to work, you must place it on your web server as an owner of a website. This is usually the same location where your main "index.html welcome page" is located. Depending on the software you use to manage your web server, where exactly it is and how to place it there will determine what file it is.

How can I locate the robots.txt file from a website?

To locate highlighted syntax warnings or logic errors, open the tester tool for your website. Scroll through robots.txt to find them. Below the editor, you will see the number of syntax errors and warnings.

In the box at the bottom, type the URL for a page you want to link to.

In the dropdown menu to the right, select the user agent you want to emulate.

To test access, click the TEST link.

To find out if your URL is blocked by Google web crawlers, check if the TEST button reads ACCEPTED.

Retest the file and edit the page. Please note that any changes to the page will not be saved to your site. 

Your changes will be copied to your robots.txt folder on your website. This tool doesn't make any changes to your site's actual file. It only checks against the copy that is hosted within the tool.