Robots.txt for WordPress: Best Example and How to Edit It

By Digitallyz Team · Updated October 2026 · 9 min read

A single line in one tiny text file can stop Google from crawling your whole website. That is why the robots txt for WordPress deserves five minutes of your attention, even if you never plan to touch code.

The good news: most WordPress sites need a very simple file. This guide shows you what the default file does, a safe example you can copy, how to edit it with or without a plugin, and the mistakes that quietly hurt rankings.

What is robots.txt in WordPress?

Robots.txt is a plain text file that sits at the root of your domain (for example, yourdomain.com/robots.txt). It tells search engine crawlers which URLs they may and may not request.

Its real job is managing crawl traffic. According to Google’s introduction to robots.txt, the file is mainly used to avoid overloading your site with requests. It is not a way to keep a page out of Google.

Crawling vs. indexing: the distinction that matters

Blocking a URL in robots.txt stops Google from crawling it, but the URL can still be indexed if other pages link to it. It may then appear in results without a description.

If you want a page out of search results, use a noindex robots meta tag and leave the page crawlable, so Google can actually see that tag. Blocking and noindexing the same page cancels out the noindex.

The four directives Google supports

• User-agent: which crawler the rules apply to (* means all).
• Disallow: a path the crawler should not request.
• Allow: an exception inside a disallowed path.
• Sitemap: the full URL of your XML sitemap.

Google ignores other fields such as Crawl-delay. It supports * as a wildcard and $ to mark the end of a URL. When rules conflict, the most specific (longest) matching rule wins, as explained in Google’s robots.txt specification.

What does the default WordPress robots.txt look like?

Fresh WordPress installs do not ship with a physical robots.txt file. Instead, WordPress generates a “virtual” one on the fly when a crawler requests it. On a standard install, it looks like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

WordPress 5.5 and later also add a Sitemap: line pointing to the built-in wp-sitemap.xml when your site is public. SEO plugins often replace that line with their own sitemap.

Why admin-ajax.php is allowed

Many themes and plugins load content through admin-ajax.php. Allowing it lets Google render those pages the way visitors see them, even though the rest of the admin area is blocked.

What “Discourage search engines” does now

The checkbox under Settings → Reading no longer adds Disallow: / to robots.txt. Since WordPress 5.3 it adds a noindex robots meta tag instead, as documented in the WordPress developer reference for do_robots(). If a live site suddenly drops out of Google, this checkbox is one of the first things to check.

The best robots txt for WordPress (copy-ready example)

For most small-business sites, blogs and service websites, this file is all you need:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yourdomain.com/sitemap.xml

Replace the sitemap URL with your real one. AIOSEO and Yoast typically use /sitemap.xml or /sitemap_index.xml; WordPress core uses /wp-sitemap.xml. Open the URL in your browser to confirm it loads before you add it.

Optional rules for specific situations

Add extra rules only when you have a clear reason. The table below covers the most common ones.

Rules you can delete from old tutorials

Older guides often told you to block /wp-includes/, /wp-content/plugins/ or /wp-content/themes/. These folders hold the CSS and JavaScript Google needs to render your pages. Blocking them can make your site look broken to Google, so remove those lines if you find them.

How to edit robots.txt in WordPress (3 methods)

Pick the method that matches your setup. If you already use an SEO plugin, start there; it is the safest option.

Method 1: All in One SEO (AIOSEO)

1. In your dashboard, go to All in One SEO → Tools.
2. Open the Robots.txt Editor tab.
3. Turn on Enable Custom Robots.txt.
4. Add or edit rules, then click Save Changes.

AIOSEO shows a preview of the final file, which helps you spot mistakes before they go live. See AIOSEO’s robots.txt editing guide for screenshots.

Method 2: Yoast SEO

1. Go to Yoast SEO → Tools → File editor.
2. Click Create robots.txt file if one does not exist yet.
3. Edit the rules and save.

If you do not see the file editor, your host may have disabled file editing in WordPress. Use Method 3 instead.

Method 3: Upload a file via FTP or your hosting file manager

1. Create a plain text file named exactly robots.txt (lowercase).
2. Paste in your rules and save it with UTF-8 encoding.
3. Upload it to the root folder of your site (usually public_html), the same folder that holds wp-config.php.

A physical file overrides WordPress’s virtual file and most plugin editors. If your plugin changes do not show up, an old physical file on the server is the usual reason.

What you should never block in robots.txt

Most robots.txt problems we see are not missing rules. They are rules that block too much. Avoid these:

• Disallow: / on a live site. This blocks every crawler from every page. It often gets copied over from a staging site during launch.
• CSS, JavaScript and image folders. Google renders pages like a browser. Blocked resources can hurt how it understands your layout and content.
• Pages you want deindexed. Use noindex and keep them crawlable until they drop out.
• Your sitemap or important category pages. These help Google discover your content.
• Private or sensitive files. Robots.txt is public. Anyone can read it, and listing a secret folder just advertises it. Protect those areas with a login.

If you are working through a wider WordPress cleanup, our guide on how to improve WordPress SEO covers the other settings worth checking at the same time.

Should you block AI crawlers in your WordPress robots.txt?

AI companies run their own crawlers, and many respect robots.txt. Whether you block them is a business decision, not a technical requirement.

The trade-off

Blocking AI crawlers may keep your content out of some AI training datasets. It can also reduce the chance that AI assistants and answer engines reference your business. For most service businesses that want to be found, visibility usually matters more.

Know which token does what

Some companies use separate crawlers for training and for search. For example, OpenAI documents GPTBot for model training and OAI-SearchBot for ChatGPT search features. Google uses a separate Google-Extended token to control use in Gemini and Vertex AI, and Google states it does not affect inclusion or ranking in Google Search.

If you decide to opt out of training only, a rule looks like this:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Check each company’s official crawler documentation before adding rules, because token names change. If AI visibility is part of your growth plan, read our guide to AI for small business for the bigger picture.

How to test your robots.txt file

Always test after every change. A typo can block more than you intended.

Step-by-step checklist

1. Open it in a browser. Visit yourdomain.com/robots.txt. You should see plain text, not an error page.
2. Check the robots.txt report in Google Search Console. Under Settings → robots.txt, Google shows the version it last fetched, when, and any errors or warnings.
3. Inspect key URLs. Run your homepage and main service pages through the URL Inspection tool and confirm crawling is allowed.
4. Confirm your sitemap loads. Click the sitemap URL listed in the file and make sure it returns a valid sitemap.
5. Clear caches. If you use a caching plugin or CDN, purge it so crawlers see the new version.

Google generally caches robots.txt for up to 24 hours, so changes may not take effect instantly. If something urgent is blocked, you can ask Search Console to recrawl the file from the robots.txt report.

Running a local business? Pair a clean crawl setup with the steps in our local SEO for small business guide to turn better crawling into more calls and visits.

Frequently asked questions

Does WordPress automatically create a robots.txt file?

Yes. WordPress generates a virtual robots.txt that blocks /wp-admin/ but allows admin-ajax.php, and recent versions add a sitemap line. There is no physical file on your server unless you or a plugin create one.

Where is the robots.txt file in WordPress?

It is always served from the root of your domain, at yourdomain.com/robots.txt. If a physical file exists, it lives in your site’s root folder, usually public_html. Otherwise WordPress or your SEO plugin generates it dynamically.

Can robots.txt remove a page from Google?

No. Robots.txt controls crawling, not indexing. A blocked page can still appear in results if other sites link to it. To remove a page, add a noindex tag, keep it crawlable, or delete it and return a 404 or 410.

Is robots.txt important for SEO?

It matters most when it is wrong. A correct, minimal file helps crawlers find your sitemap and skip low-value URLs. A bad rule, such as Disallow: /, can block your entire site, so check it after every launch, migration or plugin change.