About Shifter’s robots.txt

By default, Shifter creates a robots.txt file in the root directory of your deployment.

The robots.txt file contains the following instructions for search engines:

  • Notify search engines not to access /wp-admin/ ( Disallow: /wp-admin/)
  • XML Sitemap paths (XML Sitemap generation tools confirmed to work include the standard WordPress tool and Yoast SEO, WP Sitemap Configuration, and XML Sitemap & Google News)
  • (If a custom domain has not been applied) Notify search engines not to access any content, including subdirectories ( Disallow: /)
    This line will be removed once a custom domain is applied.

robots.txt for sites on the Shifter default domain

User-agent: *
 Disallow: /wp-admin/
 Sitemap: https://{RANDOM-WORDS}.on.getshifter.io/wp-sitemap.xml
 Disallow: /

robots.txt for sites with a custom domain attached

User-agent: *
 Disallow: /wp-admin/
 Sitemap: https://{YOUR-CUSTOM-DOMAIN-NAME}/wp-sitemap.xml