Robots.txt specifications

Shifter automatically creates robots.txt on the root of the artefacts. This file contains some codes to tell engines crawlers.

  • Tell search engines crawlers that this site disallows access to /wp-admin/ (Disallow: /wp-admin/ ).

  • Tell the path to the sitemap file. (supported XML Sitemap generators are WordPress default, Yoast SEO or XML Sitemaps plugins). Refer to our support docs (Yoast SEO, XML Sitemaps) or plugins’ official sites.

  • Tell search engines crawlers to disallow access to all contents if the custom domain name is not applied to the website.

  • Tell search engines crawlers to disallow access to all contents if a subdirectory is enabled (Disallow: /).

Example robots.txt for Shifter default artiafcts

User-agent: * 
Disallow: /wp-admin/
Sitemap: https://{RANDOM-WORDS}.on.getshifter.io/wp-sitemap.xml
Disallow: /

Example robots.txt for custom domain name attached artiafcts

User-agent: * 
Disallow: /wp-admin/
Sitemap: https://{YOUR-CUSTOM-DOMAIN-NAME}/wp-sitemap.xml

Take the Next Step

Get started on your website today with Shifter. Try out our free plan and take the first step towards building the perfect website for you and your visitors.

Copyright © 2023 DigitalCube

A product by

Check out our other products