Technical SEO makes it possible for search engines to discover, retrieve and understand the pages you want people to find. Begin with accessible URLs, correct status codes and consistent indexing signals, then connect those pages through useful navigation. A fast page with a broken canonical or an accidental noindex still has a fundamental visibility problem.
This guide is a foundation for developers and site owners, not a promise of rankings. Search visibility also depends on content and relevance. The objective here is narrower and testable: the intended page should return the intended content with signals that agree about its identity and purpose.
Separate discovery, crawling, indexing and ranking
Discovery means a search engine learns that a URL exists, for example through a link or sitemap. Crawling means retrieving that URL. Rendering can execute JavaScript and produce the content used for further processing. Indexing involves analyzing content and deciding how to represent it in the index. Ranking is the later selection and ordering of results for a query.
These are distinct stages. A URL listed in a sitemap is not necessarily crawled immediately, and a successfully crawled page is not guaranteed to be indexed. An indexed page is not guaranteed to appear for the phrase you have in mind. Diagnose the stage where the evidence stops before changing unrelated settings.
Start with a representative set of templates: homepage, category, article or product, pagination, language variant, redirect and missing page. Inspect individual examples and the rules that generate them. A single correctly configured homepage cannot establish that thousands of parameterized URLs behave correctly.
For each example, record the expected status, indexability, canonical URL and source of internal links. Compare that expectation with the HTTP response and Search Console's information. This small inventory becomes much more useful than a long list of warnings with no decision about what each URL is supposed to do.
Use robots.txt to control crawling, not access or indexing
A robots.txt file gives supported crawlers instructions about which resources they may request. It does not authenticate visitors or protect confidential content. A blocked URL can still be known from links, and blocking retrieval prevents Google from seeing instructions inside the page.
This illustrative root robots.txt blocks an internal search path and declares a sitemap:
User-agent: *
Disallow: /internal-search/
Sitemap: https://example.com/sitemap.xmlThe example is not a universal recommendation to block every search URL. Decide whether those pages need crawling, exclusion from indexing or a different URL strategy. If a page must be removed with noindex, Google must be able to retrieve that instruction. Blocking it first can prevent the very check you need.
Do not block CSS, JavaScript or image resources required to render important public content. A rule aimed at an old assets directory can affect current templates after a deployment. Check the exact hostname, protocol and path you are auditing, and verify that a broad wildcard does not catch more than intended.
For confidential material, use proper access controls. For a public page that should remain usable but not appear in search, use an indexing instruction. Google supports the following robots meta tag in a crawlable page's head:
<meta name="robots" content="noindex">An X-Robots-Tag response header can express indexing directives for resources such as PDFs as well as HTML. Check headers and markup together. Conflicting instructions can make the effective outcome more restrictive than one visible tag suggests. The default permission to index does not require adding an index tag everywhere.
Make status codes describe the actual result
The server response should distinguish useful content, a moved page, a missing page and a temporary service failure. Serving the same successful application shell for every path can hide those distinctions until JavaScript runs. That creates extra work for both debugging and search processing.
Use this table as a decision guide, not a replacement for your application's HTTP behavior:
| Situation | Typical response | What to verify |
|---|---|---|
| Useful available page | 200 |
The body contains the expected content |
| Permanently moved equivalent | 301 or 308 |
The destination is the intended replacement |
| Temporary move | 302 or 307 |
Temporary behavior is intentional |
| Missing or permanently removed content | 404 or 410 |
No irrelevant success page replaces it |
| Temporary server problem | Appropriate 5xx |
Monitoring identifies and resolves the outage |
A 200 response containing only an error message may be treated as a soft 404. Changing the message without changing the response does not repair the underlying mismatch. Likewise, redirecting every deleted product or article to the homepage does not create a meaningful replacement for each one.
Prefer a direct redirect to the final equivalent URL. Chains add requests, complicate testing and create more opportunities for inconsistent rules. Check redirects with query strings, old trailing-slash patterns and HTTP-to-HTTPS transitions. Preserve a historical URL when it still serves the right content; a new visual design alone does not require a new address.
Investigate sustained server errors before spending time on minor metadata improvements. The server response time guide helps distinguish origin delays and delivery problems that affect both visitors and reliable retrieval.
Align canonical URLs with links and redirects
Canonicalization identifies the preferred representative among duplicate or very similar pages. A canonical annotation is a signal, not an order that overrides all other evidence. Google can select a different canonical when content and signals point elsewhere.
This HTML declares the intended canonical for an illustrative guide:
<link rel="canonical" href="https://example.com/guides/caching/">Use an absolute URL and keep the choice consistent across the page's markup, internal links, redirects and sitemap. If the sitemap promotes one address while the template canonical points to another and navigation links to a third, the problem is the conflicting architecture, not the lack of another tag.
Self-canonical annotations can make a preferred page's identity explicit. Duplicate tracking variants can point to the representative page when the content is equivalent. Do not canonicalize a distinct article to a category merely because the category is more commercially important. Canonicals do not merge unrelated intent or rescue empty content.
Translations also require care. A fully localized page normally has its own canonical identity, with language alternatives expressed through hreflang. Pointing every translated page's canonical to English undermines the intended separation. Regional duplicates in the same language may need a more specific policy, but that decision should follow the actual content and targeting.
In Search Console, compare the user-declared canonical with the Google-selected canonical for the inspected URL. A disagreement is a starting point for investigation. Look for redirects, duplicate content, inconsistent links and old crawl information before repeatedly requesting indexing.
Build sitemaps from the URLs you actually want indexed
An XML sitemap is a discovery aid and a canonicalization signal. It is not an indexing command. Include the preferred, accessible URLs that belong in search, and exclude obvious redirects, error pages and pages intentionally marked noindex.
Generate the sitemap from the same content inventory that publishes the site. A separate manual list can drift: deleted pages remain while new sections never appear. Inspect a sample after deployment and check that the URLs have the expected hostname, HTTPS scheme, language prefix and slash convention.
Use lastmod only when it reflects a meaningful page update. Automatically replacing every date on every build makes the value less useful. A footer year change is not the same as revising the substantive information on every page. Keep publication and modification logic tied to real content changes.
Submit the sitemap in Search Console and examine processing errors and discovered URL counts in context. A successfully processed sitemap does not establish that all its entries are indexed. Use page-level inspection to investigate important exceptions, and compare similar templates before treating every exclusion as a defect.
Connect important pages through useful internal links
A crawler and a reader both need a route to useful pages. Use ordinary links with descriptive text and real destination URLs. A clickable card driven only by a JavaScript event handler may work for a person while providing a weak discovery path. The underlying anchor is the reliable starting point.
Organize categories around the reader's task, and add contextual links when one page explains a prerequisite or next step. The technical SEO collection is an example of a thematic route between related guides. Navigation should reveal the site's structure, not merely expose every URL in a long undifferentiated block.
Crawl depth is a diagnostic, not a universal numeric rule. A valuable page buried behind many unnecessary navigation steps deserves attention, but there is no single click count that guarantees indexing or rankings. Look for orphaned pages, broken pagination and categories that can only be reached through internal search.
For large catalogs, examine filters and sorting controls. A small set of deliberate landing pages can serve a search need; unbounded parameter combinations can create many near-duplicate URLs. Decide which combinations deserve stable indexable pages, and keep linking, canonicals and crawling controls consistent with that choice.
Do not hide missing navigation by adding every URL to a sitemap. The sitemap can expose the address, but it does not repair the user journey or the context provided by links. Review important pages from the homepage and from the category where a visitor would naturally look for them.
Describe language alternatives without merging them
hreflang helps Google associate equivalent language or regional versions. It is not a translation mechanism and does not replace meaningful localized content. Each alternative should be accessible at its own stable URL and should link back to the other members of the same group.
This simplified English-Spanish pair also declares a fallback destination:
<link rel="alternate" hreflang="en" href="https://example.com/guides/caching/">
<link rel="alternate" hreflang="es" href="https://example.com/es/guides/caching/">
<link rel="alternate" hreflang="x-default" href="https://example.com/guides/caching/">Place the same complete set on both equivalent pages, including each page's own language entry. The x-default value is an optional fallback for users whose language or region does not match a specified alternative. Use valid language and, when needed, region codes; a region by itself is not a language.
Do not connect unrelated pages merely because they occupy the same navigation position. An English category and a Spanish homepage are not equivalent translations. Keep each language's canonical consistent with the destination named in its alternate entry, and verify that the destinations do not redirect somewhere unexpected.
Provide visible language links as well. Forced redirects based solely on inferred location or language can make alternate versions harder to access and test. A visitor should be able to choose a version, and a crawler should be able to retrieve it directly.
Add structured data that agrees with visible content
Structured data describes entities and page content in a machine-readable form. Choose a suitable supported type and satisfy its required properties with real information. A valid JSON document is only the syntax layer; accuracy and eligibility still matter.
For example, article markup should describe the article actually displayed. Product markup should not invent availability or ratings. Do not add review stars, people or organization details merely because a template expects a field. Omit unsupported information and fix the template instead.
Use the Rich Results Test for Google's supported rich-result features and investigate errors in the context of the chosen type. Passing the test does not guarantee a rich result or prove that the page is indexed. Keep structured data synchronized when the visible content changes, particularly where prices, availability or dates are generated separately.
Check mobile content and performance separately
Google primarily uses the mobile version's content for indexing. Ensure important text, headings, links, images and structured data remain available on mobile. A compact layout can reorganize content, but removing the essential explanation or requiring an interaction to fetch it creates a different search document.
Also inspect the actual experience: readable text, usable controls, stable layouts and responsive interactions. Good Core Web Vitals support the user experience, but they do not replace accessibility, relevance or indexability. The guide to Core Web Vitals and rankings separates performance signals from ranking promises.
A fast static response and a slow interactive page can coexist. Equally, a good performance score can accompany a page whose main content never appears in Google's rendered output. If JavaScript supplies the important content, follow the rendering and JavaScript SEO workflow rather than inferring indexability from speed alone.
Turn the audit into a repeatable release check
For a priority URL, save the headers and response body before inspecting the rendered page. This command retrieves the example page with a normal GET request:
curl -sS -D response-headers.txt -o response.html https://example.com/guides/caching/The files show the status and initial HTML, not the final JavaScript-rendered DOM. Compare them with a browser inspection and, where available, Search Console's URL Inspection results. Distinguish the indexed report, which reflects Google's stored information, from a live test of current accessibility and rendering.
Check the canonical, robots directives, important text and links at each relevant stage. Then test a missing URL, a redirect and a language alternative from the same template family. This catches rule-level problems that a single successful page can conceal.
Prioritize access and response failures first, then contradictory indexing signals, missing content and broken discovery paths. Use the performance checklist for the performance work that follows. Record the expected behavior, owner and retest result for each fix, and repeat the template checks after releases that change routing, rendering or metadata.
