Technical SEO: How to Audit Your Website, With a Checklist

Summary: Technical SEO is the work that lets search engines find, read, understand and trust your website: crawling, indexing, site structure, speed, mobile usability, security and structured data. A technical SEO audit checks each of those in turn, usually with Google Search Console and a crawling tool, and produces a prioritised list of fixes. For most UK business sites, the problems that matter are a small number of basics: pages blocked or missing from the index, slow pages, duplicate URLs, broken links and redirect chains.

Good content on a site Google cannot crawl properly is money wasted. Technical SEO is the part of search optimisation that most business owners never see and most web designers never do, which is why our own study of WordPress sites found technical errors on almost every site we audited before starting work. This guide explains what technical SEO covers, then walks through a technical SEO audit step by step, with the tool to use and the fix to make at each stage. A checklist at the end pulls it together.

What is technical SEO?

Technical SEO is the optimisation of a website’s structure and infrastructure so that search engines can crawl it, index it and rank it efficiently. It is distinct from on-page SEO, which is about the content and HTML of individual pages, and from off-page SEO, which is about links and mentions from other sites. Technical SEO is the foundation the other two stand on.

It covers crawlability, indexability, site architecture, page speed and Core Web Vitals, mobile usability, HTTPS, duplicate content and canonicalisation, structured data, and increasingly, access for AI crawlers. Most of it is invisible to visitors. All of it is visible to Googlebot.

Why it matters more than it looks

Search engines cannot rank what they cannot reach, and they rank reluctantly what loads slowly, breaks on a phone or appears in several duplicate versions. Technical problems rarely cause a dramatic collapse. They cause a steady underperformance that is easy to miss because nothing looks broken.

In our study of technical SEO errors in WordPress, drawn from audits we ran before starting work on client sites, 95% of sites had technical errors that could affect organic ranking, 84% had oversized image files, 78% had broken internal links and 23% had at least one orphan page. These are sites that were live and, in most cases, being actively marketed. The errors were there because nobody had looked.

How to do a technical SEO audit

Work through the steps in order. The first three decide whether Google can see the site at all; the rest decide how well it performs once seen. You need two things to start: Google Search Console verified for the site, and a crawling tool. Screaming Frog SEO Spider is the standard and its free version crawls up to 500 URLs, which covers most small business sites. Sitebulb, Ahrefs Site Audit and Semrush Site Audit do the same job with more reporting.

Step 1: Crawl the site and see what Google sees

Run a full crawl. The crawler follows every link from the homepage and reports every URL it finds, with its status code, title, headings, canonical tag, indexability and more. This is your inventory. Compare the number of URLs the crawler finds with the number Search Console reports as indexed. A big gap in either direction is the first thing to investigate: pages the crawler cannot reach, or pages Google has indexed that you did not know existed.

Step 2: Check crawlability

Crawlability is the ability of search engine bots to access and move through your site. A page that cannot be crawled cannot be indexed, and a site that is hard to crawl gets crawled less often, so new content and changes are picked up slowly.

What to check:

• Robots.txt is not blocking pages or resources Google needs (see the robots.txt section below).
• Every important page is reachable by following links from the homepage, ideally within three clicks.
• No orphan pages: pages that exist but nothing links to. The crawler will not find them, and neither will Google unless they are in the sitemap.
• Server responses are healthy: no 5xx errors, no pages timing out.
• JavaScript is not hiding content or links. If the main content only appears after scripts run, test it with the URL Inspection tool in Search Console to confirm Google renders it.

Where to look: Search Console’s Crawl stats report (under Settings) shows how often Google crawls, what response codes it gets and whether the server is struggling. The crawler’s own report shows blocked URLs and orphan pages when you upload the sitemap alongside the crawl.

Crawlability and indexability are not the same thing

A page can be crawlable but not indexable: Google can reach it but a noindex tag or canonical to another page tells it not to store it. A page can also be indexable but not crawlable: nothing blocks indexing, but no link or sitemap leads Google to it. An audit checks both. Our guide to indexing covers the indexing side in depth.

Step 3: Check indexing

Open Search Console, Indexing, Pages. It lists every URL Google knows about and sorts them into indexed and not indexed, with a reason for each exclusion. Work through the reasons:

• “Excluded by noindex tag” on a page that should rank: usually a setting left over from development or a plugin option ticked by mistake.
• “Discovered, currently not indexed” or “Crawled, currently not indexed”: Google found the page and chose not to index it. On a small site this usually means the page is thin, duplicated or very similar to another. Improve it or remove it.
• “Duplicate without user-selected canonical” or “Google chose different canonical than user”: duplicate content problems, dealt with in step 5.
• “Not found (404)” and “Soft 404”: dealt with in step 6.

Then check the XML sitemap. It should list every page you want indexed and nothing else: no redirected URLs, no noindexed pages, no 404s. WordPress generates one at /wp-sitemap.xml by default; Rank Math and Yoast replace it with their own. Submit it in Search Console under Indexing, Sitemaps, and check it reports as successful. Our guide to sitemaps explains the options.

Step 4: Review site architecture and internal links

Site architecture is how pages are organised and linked. A clear hierarchy (homepage, service or category pages, individual pages) helps Google understand which pages matter most and helps visitors find what they came for.

• Important pages within three clicks of the homepage.
• Descriptive URLs that reflect the hierarchy: /services/boiler-repair/ rather than /p=123.
• Breadcrumbs on deeper pages, marked up with structured data.
• Internal links between related pages using descriptive anchor text, not “click here”.
• The pages that matter most commercially receiving the most internal links.

The crawler’s “inlinks” count per URL shows this quickly. A key service page with two internal links and a three-year-old blog post with forty is a structure problem.

Step 5: Find and resolve duplicate content

Duplicate content is the same or very similar content reachable at more than one URL. It is rarely penalised but it splits ranking signals between versions and wastes crawl budget. Common causes: http and https both live; www and non-www both live; trailing slash and no trailing slash; URL parameters for sorting and tracking; printer-friendly or AMP versions; category and tag archives that reproduce post content; ecommerce product URLs reachable through several category paths.

Fixes: choose one version of each and redirect the rest with a 301, or set a canonical tag pointing at the preferred version. Every page should carry a self-referencing canonical. Our guide to canonical tags covers when to use each approach.

Step 6: Fix broken links, 404s and redirect chains

The crawler lists every internal link that returns a 404, every redirect and every redirect chain (a URL that redirects to another URL that redirects again). Fix broken internal links at source by pointing them at the live page. Where a page has been removed for good, redirect the old URL to the closest equivalent, or let it return a proper 404 if there is no equivalent; a 404 for a page that no longer exists is correct behaviour, not an error. Our article on whether 404s are bad for SEO covers how Google treats them. Flatten redirect chains so every old URL redirects once, directly to its final destination.

Step 7: Measure speed and Core Web Vitals

Google measures page experience with three Core Web Vitals: Largest Contentful Paint (how quickly the main content appears), Interaction to Next Paint (how quickly the page responds when someone taps or clicks) and Cumulative Layout Shift (how much the layout jumps around while loading). Search Console’s Core Web Vitals report groups your pages as good, needs improvement or poor based on real visitor data, and PageSpeed Insights diagnoses an individual page.

The fixes that make the most difference on business sites are unglamorous: compress and resize images, serve them in WebP or AVIF, lazy-load anything below the fold, set explicit dimensions on images and embeds so the layout does not shift, remove plugins and scripts that are not earning their place, use a caching plugin and a CDN, and choose hosting that is fast for UK visitors. Speed is a ranking factor, but a modest one. Its larger effect is on whether visitors stay.

Step 8: Test mobile usability

Google indexes the mobile version of your site, so what Googlebot sees is the phone layout. Check on a real phone and in Chrome’s device emulation: text readable without zooming, tap targets not crowded together, no content cut off, no interstitials covering the page on arrival, and the same content on mobile as on desktop. A responsive design handles most of this, but check the pages that have been edited since launch; mobile problems creep in one page at a time.

Step 9: Confirm HTTPS and security

The whole site should load over https, with http redirecting to it, and with no mixed content (images or scripts still loading over http on an https page). Check that the certificate is valid and renews automatically. Browsers mark sites without https as not secure, and Google has preferred https pages for years. On WordPress, also check that the login page is protected and that the site is not exposing its version number and plugin list; neither is a ranking issue, but a hacked site quickly becomes one.

Step 10: Check structured data

Structured data is code, usually JSON-LD, that labels what a page is about in a vocabulary from schema.org: an Organisation, a LocalBusiness, a Product, an FAQ, a breadcrumb. It helps search engines understand the page and makes it eligible for rich results such as star ratings and FAQ dropdowns, and it is one of the clearer signals AI systems use when deciding what a business is. Test pages with Google’s Rich Results Test and check the Enhancements section of Search Console for errors. Our guide to schema and structured data covers what to mark up.

Step 11: Check access for AI crawlers

In 2026 the crawlers that matter are no longer only Googlebot and Bingbot. OpenAI’s GPTBot, Anthropic’s ClaudeBot, PerplexityBot and Google-Extended (which controls use of your content in Google’s AI products) all identify themselves and all respect robots.txt. Blocking them is a legitimate choice for some publishers, but for a business that wants to be recommended by AI assistants it is a mistake, and it is frequently done by accident through a security plugin or firewall rule. Check robots.txt and your firewall’s bot settings, and check the server logs for AI user agents being refused. Our guide to generative engine optimisation covers what to do once they can get in.

Robots.txt: what it does and how to get it right

Robots.txt is a plain text file at the root of your domain (yoursite.co.uk/robots.txt) that tells crawlers which parts of the site they may and may not request. It is a request that well-behaved crawlers follow, not a security measure; anything sensitive needs proper access control. Its main SEO use is to keep crawlers away from areas that waste their time, such as admin pages, internal search results and parameter-driven duplicates, so they spend their crawl budget on the pages you want indexed.

Common robots.txt problems

• Not valid: missing User-agent lines or syntax errors, so crawlers ignore it or misread it.
• Not accessible: the file returns an error, so some crawlers treat the whole site as off limits.
• Overblocking: blocking CSS, JavaScript or image folders, so Google cannot render pages properly and ranks them accordingly.
• Using it to hide pages from the index. A page blocked in robots.txt can still be indexed if other sites link to it; use a noindex tag for that instead, and make sure the page is not blocked so Google can see the tag.

A sensible robots.txt for WordPress

WordPress generates a basic file automatically if none exists. A reasonable manual version looks like this:

User-agent: *

Disallow: /wp-admin/

Allow: /wp-admin/admin-ajax.php

Sitemap: https://yoursite.co.uk/sitemap_index.xml

Keep /wp-admin/ blocked but leave admin-ajax.php accessible, because themes and plugins use it on the front end. Point the Sitemap line at whatever your SEO plugin generates. Do not block /wp-includes/ or /wp-content/; they hold the CSS, scripts and images Google needs to render the page. Rank Math and Yoast both let you edit the file from the dashboard.

Useful patterns

Test changes with the robots.txt report in Search Console (under Settings) before relying on them, and never leave a staging “Disallow: /” in place when a site goes live. It is the most expensive single line in SEO.

Technical SEO audit checklist

Tools for a technical SEO audit

How often to audit

A full audit at the start of any SEO work and after any significant change to the site: a redesign, a migration, a new plugin that touches URLs or templates. Between those, a monthly check of Search Console’s Pages and Core Web Vitals reports catches most problems early, and a scheduled crawl from one of the paid tools will flag new broken links and redirects as they appear. For a new website, run the audit before launch; our guide to SEO for a new website lists what to check.

Technical SEO on WordPress

Most of the sites we audit run on WordPress, and most of their technical problems come from the same places: plugins that generate duplicate URLs, themes that load far more code than they use, image uploads that were never compressed, and settings left over from development. WordPress is perfectly capable of being technically excellent; it just does not arrive that way. Our WordPress SEO service exists for exactly that gap.

How we approach technical SEO

Every campaign we run starts with a technical audit, because content and links built on a site with crawl or indexing problems underperform from day one. We fix the issues that affect ranking first, report what we found and what we changed, and re-crawl to confirm. Our SEO packages include this as standard, and if you would like to know what your site’s technical health looks like, we offer a free SEO audit.

Frequently asked questions

See also, our complete guide to SEO for UK businesses.