View all Products Close Menu Menu
Creating a robots.txt file for your website
Updated Friday February 5th 2021 / Category Web Hosting

What is the robots.txt file used for?

Sometimes you might want to stop search engines and/or crawlers from accessing/indexing your website. You can use a robots.txt file to stop this from happening.

How to create a robots.txt file

Creating a robots.txt file is easy, here are the basic steps:

  1. Login to cPanel.
  2. Click on File Manager under Files.
  3. Navigate to the folder your website is stored in (this is normally the public_html folder).
  4. Click on the +File button to the top left of the page.
    cPanel File Manager Create File
  5. In the New File Name field, set the filename as robots.txt
  6. Click Create New File.

Basic Usage

Here are some examples of code/rules you can add to your robots.txt file, to achieve different things.

Blocking Google from indexing anything under /blog (e.g. https://my-domain-name.com/blog) on your website:

User-agent: Googlebot
Disallow: /blog/

Allow all bots to access any part of your website:

User-agent: *
Allow: /

Let the bot know where your sitemap is:

Sitemap: https://example.com/sitemap.xml

 

Can’t find the answers you're looking for? Check out these other methods of support!