X-Robots-Tag: How to Check, Set, and Fix It (2026)

An X-Robots-Tag is an HTTP response header that tells search engines whether to index or follow a page or file. Unlike a meta tag, it lives in the server's response, not the HTML, so it works on PDFs, images, and any file type. To fix one, edit your server config or CMS header settings and confirm the change by checking the live response.

Right after you confirm what the header is, the fastest next move is to look at what your own site is actually sending. The free meta tag checker pulls the live response headers and meta tags for any URL you give it, so you can see in one pass whether an X-Robots-Tag is present, what directive it carries, and whether it conflicts with a meta robots tag on the same page.

What an X-Robots-Tag Actually Does

The tag carries one or more directives, separated by commas, inside the HTTP response. Search engines read these directives the same way they read a meta robots tag, but because the instruction sits in the header, it applies to any file the server serves, not just HTML pages.

That distinction matters for a specific reason: you cannot put a meta tag inside a PDF or a JPEG. If you need to keep a downloadable file or an image out of search results, the X-Robots-Tag is your only option.

The Directives You'll Use Most

Directive	What it does	Typical use case
noindex	Keeps the page or file out of search results	Internal reports, staging pages, duplicate PDFs
nofollow	Tells crawlers not to follow links on that URL	Pages with outbound links you don't want to pass authority
noarchive	Stops Google from showing a cached copy	Pages with time-sensitive pricing or content
nosnippet	Blocks a text or image preview in results	Content you don't want excerpted in the SERP
unavailable_after	Removes the page from results after a set date	Limited-time offers, expired listings

How to Set It on Your Own Site

The exact steps depend on your server, but the pattern is the same everywhere: add the header at the server level, then verify it on the live URL.

On Apache (.htaccess)

1. Open your site's .htaccess file.
2. Target the file type you want to control, for example PDFs:

   <Files ~ "\.pdf$">
   Header set X-Robots-Tag "noindex, nofollow"
   </Files>
   

3. Save and upload the file to your server's root directory.
4. Reload the affected URL and confirm the header shows up in the response.

On Nginx

1. Open your server block configuration.
2. Add a location rule for the file type, for example images:

   location ~* \.(png|jpe?g|gif)$ {
     add_header X-Robots-Tag "noindex";
   }
   

3. Reload Nginx so the config takes effect.

On WordPress or a Managed CMS

Most CMS platforms handle meta robots tags through an SEO plugin's page-level settings, which is simpler for ordinary pages and posts. Reach for the X-Robots-Tag specifically when you need to block PDFs, downloads, or other non-HTML files that a plugin's page settings can't touch. If your CMS doesn't expose header controls directly, you'll usually need to edit the server config through your host's file manager or ask your developer to add the rule.

Before and after any change, confirm your robots.txt file isn't blocking the same URL. If crawlers can't reach a page, they never see the X-Robots-Tag directive sitting inside it. Run the URL through a robots.txt tester to rule that out, and if you need to rebuild the file from scratch, a robots.txt generator will produce a valid one without the syntax mistakes that cause accidental site-wide blocks.

Robots.txt, Meta Robots, or X-Robots-Tag?

These three tools sound similar and get confused constantly. They solve different problems.

Tool	Where it lives	Controls	Best for
Robots.txt	Root-level text file	What crawlers can access	Blocking crawl of entire folders or sections
Meta robots tag	Inside the HTML <head>	Indexing of one HTML page	Simple, page-by-page control on standard pages
X-Robots-Tag	HTTP response header	Indexing of any file type	PDFs, images, large batches of URLs by pattern

Robots.txt and meta/X-Robots directives work at different layers: robots.txt governs crawling, while the other two govern indexing. A page can be crawlable but still excluded from results with a noindex directive, and the reverse, a page that's blocked from crawling entirely, means Google never even reads the indexing instruction sitting inside it.

When Your Own Tools Are Enough, and When They Aren't

Checking a single URL with a free header checker is enough when you've just made a change and want to confirm it took effect, or when someone reports one specific page showing up in search that shouldn't be there. That's a one-off lookup, and the free meta tag checker handles it in seconds.

It stops being enough once the problem spans your whole site: a server migration that wiped your header rules, a CDN config pushed to every file type, or a WordPress plugin update that silently changed default behavior. In those cases you're not checking one URL, you're trying to catch which of hundreds or thousands of pages picked up an unwanted noindex. That calls for a recurring technical audit rather than a manual spot-check, since a one-time look won't catch a regression that happens next month.

Common Mistakes

• Setting noindex in both robots.txt and the X-Robots-Tag. If robots.txt blocks the URL, Google never reads the header, so the indexing directive sits unused.
• Applying the header to an entire file type when only some files need it. A blanket PDF rule can accidentally deindex documents you wanted found.
• Forgetting to check after a server migration. New hosting environments often strip custom header rules without warning.
• Confusing nofollow on the page with rel="nofollow" on a link. They affect different things, one is page-wide, the other is link-specific.
• Assuming a plugin setting covers non-HTML files. Most SEO plugins only touch meta tags on pages and posts, not PDFs or images served directly.

FAQs

Does the X-Robots-Tag work the same as a meta robots tag?

They accept the same directives, but the X-Robots-Tag is an HTTP header set at the server level, while the meta robots tag is HTML markup inside a page's <head>. The header approach also works on non-HTML files, which meta tags cannot touch.

Can I use the X-Robots-Tag and robots.txt together?

Yes, but only if robots.txt still allows crawling of the URL. If the page is disallowed in robots.txt, crawlers never reach it to read the X-Robots-Tag directive.

How do I check if a page has an X-Robots-Tag?

View the HTTP response headers for that URL. Browser developer tools, a header-inspection browser extension, or a dedicated checker tool will all show you the raw response, including any X-Robots-Tag directive present.

Will adding noindex remove a page from Google immediately?

No. Google has to recrawl the URL and process the directive before the page drops out of results, which can take anywhere from a few days to a few weeks depending on how often the page is crawled.

What happens if I set conflicting directives on the same page?

Search engines generally apply the more restrictive directive when conflicts arise, but relying on that behavior is risky. Clean up conflicting rules rather than assuming the stricter one always wins.

Get a Full Read on Your Site's Indexing Signals

A one-off check tells you about a single URL; it won't tell you whether the same misconfiguration is quietly repeating across your site. If you want a full picture of how your pages are being indexed, crawled, and served, X-Robots-Tag included, talk to the team and get a sample report built from your own site's data.

Sources

• All You Need To Know About the X- Robots-Tags - DWS
• How to add x-robots-tag for certain url pattern
• X-Robots-Tag 2026 - Build Websites With AI
• x-robots-tags noindex even though robots already set to ...
• How to Add X-Robots-Tag HTTP Header using .htaccess file in ...