Crawling, noindex and canonicals: choose the right control
A blocked crawl, an excluded page and a different canonical are different findings. Before changing any setting, decide whether the page should be public, searchable and represented by this exact URL.
Start with the intended outcome
Investigate when a public page you want discovered is blocked unexpectedly, carries a stale staging noindex instruction, or points to an unrelated canonical. Leave deliberate exclusions alone: customer accounts, unfinished campaigns and internal tools may correctly stay outside search.
Write down three separate decisions: who may access the content, whether it should appear in search, and which address should represent equivalent versions. Search settings do not replace authentication. Never remove a login requirement or open private pages just to improve a diagnostic result.
Four controls that solve different problems
- Authentication: protects content from unauthorised visitors. Use this for genuinely private information.
- robots.txt: requests that a compliant crawler avoid specified paths. Google warns that a blocked URL can still appear in search if discovered elsewhere.
- noindex: tells supporting search engines not to index a page. Google must be able to fetch the page to read its meta tag or HTTP header.
- Canonical annotation: identifies a preferred representative for duplicate or very similar pages. It is a signal, not a guaranteed selection or a privacy control.
Do not put a noindex rule in robots.txt: Google does not support that implementation. Do not combine a crawl block with page-level noindex and assume Google can read both.
Trace the response before editing it
- Record the submitted URL, redirects, final URL and response status. A successful HTTP response can still contain a login or error screen.
- Read the applicable robots.txt group for the crawler concerned. A rule for one provider does not establish access for every bot.
- Inspect both HTML robots metadata and the
X-Robots-Tagresponse header. Check whether the restriction comes from hosting, a plugin, a template or the page itself. - Read the canonical address and open that destination. Compare content, redirects, internal links and sitemap URLs before deciding that a different canonical is wrong.
For a fictional public workshop page that is intentionally searchable, this self-referencing annotation expresses its preferred address. Replace the reserved example domain; do not paste it across unrelated pages.
<link rel="canonical"
href="https://workshop.example/services/bicycle-repairs/">Google describes redirects and canonical annotations as strong canonicalisation signals, with sitemap inclusion a weaker signal. Avoid using noindex merely to choose between duplicate URLs.
Make the smallest justified platform change
WordPress: review the site-wide Search Engine Visibility setting and any page-level SEO controls. Confirm whether the site is production or staging before changing either. Themes, plugins and the hosting layer can each contribute directives; inspect the output rather than assuming one checkbox owns everything.
Shopify: review the platform’s existing rules before touching robots.txt.liquid. Shopify treats robots customisation as an advanced change. Keep a copy of any custom template and preserve intentional exclusions. Consult the relevant theme or app documentation if a page receives an unexpected noindex or canonical.
Custom HTML: change the specific head template, response-header rule or robots group responsible. Do not globally remove every robots directive. Keep the configuration change separate from an unrelated redesign so it can be reviewed and reverted independently.
Separate technical verification from indexing
After publishing, request the page again without a privileged browser session. Verify the status, final URL, robots rules, metadata, headers and canonical destination. Check a deliberately excluded page as a regression case: it should retain its intended restriction.
Use Search Console’s URL Inspection, where authorised, to investigate Google’s indexed state and selected canonical. A passing live technical check is not confirmation that Google has indexed the page. Recrawling and canonical selection are separate processes; do not promise a fixed recovery date.
If unrelated pages become indexable or point to the wrong canonical, restore the previous rule or template, clear relevant caches and inspect the affected routes again. Restoring access controls takes priority over maintaining an attractive audit result.
What an audit can and cannot establish
Common mistakes include redirecting every retired page to the homepage, declaring unrelated pages equivalent, and interpreting an unavailable robots file as a deliberate block. Diagnose the evidence and page purpose before choosing a fix.
SingCompare checks returned HTML, headers and bounded discovery resources. It reports declared crawler rules; its request is not a verified request from Googlebot or another named crawler. It does not establish Google’s index state, selected canonical, CDN treatment of every bot or access to private content. An unknown check remains unknown until better evidence is available.
Sources and review date
Official documentation reviewed on 17 November 2026. Platform controls and search policies can change. Examples are fictional; implementation and prioritisation advice is SingCompare’s own.