Why not to change Robots.txt file?

Robots.txt is a file that contains the directives for search engines. It tells the search engine spiders to not crawl certain pages or sections of a website. Most major search engines (including Google, Bing, and Yahoo) recognize and honor Robots.txt requests. You can change your robots.txt file as per your need. However, StoreHippo does not support editing of the robots.txt file. 

Robots.txt creates dead ends

Before you can compete for visibility in the search results, search engines need to discover, crawl and index your pages. If you’ve blocked certain URLs via robots.txt, search engines can no longer crawl through those pages to discover others. That might mean that key pages don’t get discovered.

Robots.txt denies links their value

One of the basic rules of SEO is that links from other pages can influence your performance. If a URL is blocked, not only search engines won't crawl it, but they also might not distribute any ‘link value’ pointing to that URL to, or through that URL to other pages on the site.

Does your website need a robots.txt file?

No. When Googlebot visits a website, it first ask for permission to crawl by attempting to retrieve the robots.txt file. A website without a robots.txt file, robots meta tags or X-Robots-Tag HTTP headers will generally be crawled and indexed normally.