Discourage Search Engines From Indexing This Site: A Setup Guide
By Toy Wen Da, Co-Founder · · 7 min read · scored 93/100
Checking the box to discourage search engines from indexing this site tells WordPress to add a "noindex" rule to your robots.txt, asking Google, Bing, and other crawlers to skip your pages. It's a request, not a lock: well-behaved bots honor it, but it won't password-protect anything or block direct links.
What This Checkbox Actually Does
When you check "discourage search engines from indexing this site" and save your WordPress Reading settings, two things typically happen behind the scenes.
First, WordPress rewrites your site's robots.txt file, the simple text file crawlers check before they crawl anything, to include:
User-agent: *
Disallow: /
That tells every compliant crawler not to request any page on your domain. Second, many WordPress installs also add a noindex, follow meta tag to your page headers, which is a more direct instruction: even if a bot does see the page, don't list it in search results.
Why "Discourage," Not "Block"
WordPress uses the word "discourage" deliberately. Google's own developer documentation explains that robots.txt directives are instructions, not access controls, compliant crawlers like Googlebot respect them, but the file can't physically prevent access the way a password or firewall rule can (source: Google Search Central). If a bot ignores robots.txt, or if other sites link directly to your pages, your content can still surface in results.
Noindex vs. Robots.txt: They Don't Stack Well
This is the part people get wrong most often. If a page is disallowed in robots.txt, search engines never crawl it, which means they never see a noindex tag on that page either. Using both at once on the same URL can actually backfire: the crawler stops at robots.txt and never reads the noindex instruction, so a page that's already indexed can stay indexed longer than if you'd used noindex alone. Pick one method per URL, not both.
When You Should Actually Use This Setting
Not every site should flip this switch, and not every reason for wanting to is a good one. Here's how the common cases break down.
| Situation | Recommended Approach | Why |
|---|---|---|
| Site under active development on a live URL | Enable the setting, or better, use a password-protected staging subdomain | Keeps half-built pages out of search while you work |
| Private or family blog | Enable the setting plus consider a plugin that restricts viewing | Checkbox alone won't stop direct visitors |
| Internal company intranet or dashboard | Password protection at the server level, not just this checkbox | Robots.txt is a request; it doesn't block unauthorized people |
| Site that's fully launched and wants traffic | Leave it unchecked | This setting will actively suppress your organic visibility |
| Recycled domain with old backlinks | Enable temporarily, audit old links, then disable at launch | Buys time to clean up before the public sees anything |
The One Mistake That Tanks Traffic
The single most common WordPress support request related to this setting isn't "how do I turn it on", it's "why did my traffic disappear." Sites get migrated from staging to production, and the discourage-indexing checkbox comes along for the ride, left checked by accident. Every page on the live site effectively tells Google not to index it, and rankings built over months can quietly evaporate. Always double-check this setting immediately after any site migration or redesign launch.
How to Turn It On (or Off) Correctly
- Log in to your WordPress admin dashboard.
- Go to Settings → Reading.
- Find "Search Engine Visibility."
- Check the box labeled "Discourage search engines from indexing this site."
- Click Save Changes.
Reversing it is the same process in reverse, uncheck the box and save. But unchecking it doesn't instantly restore your rankings. Search engines need to recrawl your site, notice the robots.txt has changed, and re-add your pages to their index. That process can take anywhere from a few days to several weeks depending on how often your site gets crawled.
If Your Pages Are Already Indexed
Checking the box stops future crawling, but it doesn't erase pages search engines already have. If Google indexed your content before you enabled this setting, removal takes an extra step:
- Add a
noindexmeta tag directly to the affected pages (don't combine this with a robots.txt block on the same URLs, for the reason above). - Make sure the page is still publicly accessible, a password-protected or private page can't be crawled, which means a bot can never read the noindex tag telling it to drop the page.
- Use Google Search Console's URL removal tool to speed up deindexing, rather than waiting for a natural recrawl.
- Confirm there's no conflicting
Disallowrule in robots.txt blocking the crawler from reaching the noindex tag in the first place.
This mirrors how major platforms handle the same problem, HubSpot's own documentation on preventing content from appearing in search results walks through the identical sequence: robots.txt for content that hasn't been indexed yet, noindex tags for content that has. The logic holds regardless of which CMS you're running.
Why a Checkbox Isn't Real Security
If your goal is actually to prevent people from finding or viewing sensitive content, not just keeping it off Google, this setting is the wrong tool. It only talks to search engines that choose to listen. For true privacy, you want server-level password protection on the directory, which blocks both bots and humans before WordPress even loads. If you're unsure which approach fits your situation, understanding how search engine indexing actually works will make it clearer why a request-based setting can't substitute for access control.
Checking Whether It Worked
After saving the setting, visit yourdomain.com/robots.txt directly in a browser. You should see Disallow: / under the user-agent line. If you don't see that, your changes didn't save, or a caching layer is serving an old version of the file, clear your cache and check again.
It's also worth remembering that "search engines" isn't just one company. Bing publishes its own crawling rules separately from Google's (see Bing Webmaster Guidelines), and while most major crawlers respect the standard robots.txt format, behavior can vary slightly between them. If you're only ever thinking about Google, it's worth a quick read on the different types of search engines that might be sending (or withholding) traffic to your site.
FAQs
Does checking this box remove my site from Google immediately?
No. It stops new crawling going forward, but pages already in Google's index stay there until Google recrawls your site and notices the change, typically days to weeks, faster if you use Search Console's removal tool.
Will this setting stop people from visiting my site directly?
No. It only asks search engines not to list your pages in results. Anyone with a direct link, bookmark, or your site's exact URL can still visit normally.
Can I use this and a noindex tag at the same time on the same page?
It's not recommended. A robots.txt disallow stops crawlers from ever reaching the page, so they never see the noindex tag, the two instructions can work against each other instead of reinforcing each other.
I just launched my site and traffic dropped to zero. Could this setting be the cause?
Yes, this is one of the most common causes. Check Settings → Reading immediately, if the box got left checked after a migration from staging, uncheck it and save.
Do all search engines respect this request?
No. It relies on voluntary compliance. Major engines like Google and Bing generally honor robots.txt, but it's explicitly described as a directive, not an enforced block, so some crawlers or bad actors may disregard it.
See Exactly What's Happening on Your Site
Before you flip the discourage search engines from indexing this site setting on or off, it helps to know what's actually indexed right now and what isn't, guessing is how migrations go wrong. Talk to the team and pull a sample SEO report to see your site's current indexing status, crawl issues, and visibility at a glance before you make the change.
Resources / References
Sources
About the author
SearchBlueprint researched, wrote and scored this article with the content pipeline our customers run on their own sites. The score in the byline is what our content engine gave this page before it published. Nothing here is reviewed by hand, which is the point: it is the product's output, published where anyone can check it.