Crawlability and Indexing Checklist

1 Verify Index Coverage in Google Search Console

Open the Pages report in Google Search Console. Review total indexed vs not indexed count. Click each exclusion reason to see which pages are affected. Important pages showing "Crawled but not indexed" typically need content improvement, better internal links, or removal of low-value signals.

Set a weekly calendar reminder to check this report. A sudden spike in "Not Indexed" pages is often the first sign of a site-wide issue like a misconfigured robots.txt or a server error.

2 Fix Duplicate Website Versions

Your site should resolve to one canonical version. Test all four combinations: http://, https://, www, and non-www. Each should 301 redirect to your chosen version. Search engines see each unique URL as a separate site if not redirected, splitting your authority across four versions.

# Apache: Force HTTPS + WWW
RewriteEngine On
RewriteCond %{HTTPS} off [OR]
RewriteCond %{HTTP_HOST} !^www\. [NC]
RewriteRule ^(.*)$ https://www.yourdomain.com/$1 [R=301,L]

# Nginx: Force HTTPS + WWW
server {
    listen 80;
    server_name yourdomain.com www.yourdomain.com;
    return 301 https://www.yourdomain.com$request_uri;
}

3 Optimize Robots.txt for All Crawlers

Your robots.txt is the first file every crawler reads. A single mistake here can block your entire site. Allow Googlebot, Bingbot, GPTBot, and Claude-Web while blocking low-value paths. Use the URL Inspection tool in GSC to test whether specific pages are actually blocked.

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /login/
Disallow: /cart/
Disallow: /checkout/
Disallow: /wp-admin/
Disallow: /tmp/

User-agent: GPTBot
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://www.yourdomain.com/sitemap.xml

Test your robots.txt using Google's robots.txt tester in GSC or the Robots Exclusion Checker tool. Verify that important pages are not accidentally blocked.

4 Submit XML Sitemap to Google Search Console

An XML sitemap is your direct channel to tell search engines which pages exist and when they were last updated. Only include canonical, indexable URLs. Exclude paginated pages, parameter-based URLs, and thin content. Submit the sitemap URL in GSC under the Sitemaps section.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.yourdomain.com/</loc>
    <lastmod>2026-07-15</lastmod>
    <changefreq>weekly</changefreq>
    <priority>1.0</priority>
  </url>
  <url>
    <loc>https://www.yourdomain.com/technical-seo-checklist</loc>
    <lastmod>2026-07-29</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.9</priority>
  </url>
</urlset>

Large sites should split sitemaps into a sitemap index file with individual sitemaps of up to 50,000 URLs each. Use Yoast SEO, Rank Math, or Screaming Frog to auto-generate sitemaps.

5 Fix Redirect Chains and Loops

Every hop in a redirect chain adds latency and wastes crawl budget. A page that requires 3+ redirects to reach the final URL is wasting 60-70% of its crawl equity. Use Screaming Frog or Semrush to detect chains and flatten every redirect to point directly at the final destination.

# Screaming Frog: Export redirect chains
# Crawl your site, go to Response Codes tab
# Filter by Status Code = 301, 302, 303, 307, 308
# Sort by "Redirect URL" to see chains

# Use a redirect checker tool
curl -I https://www.yourdomain.com/old-page | grep "location:"

A redirect loop (A → B → C → A) will cause the crawler to give up and treat the URL as broken. Test all major redirect paths manually.

6 Fix Broken Internal Links

Broken internal links waste crawl budget and frustrate users. Every 404 on your site represents a failed crawl opportunity. Run a full crawl with Screaming Frog, export all 4xx errors, and fix each one by restoring the page, adding a 301 redirect, or updating the link.

# Bulk check URLs with curl
for url in $(cat urls.txt); do
  status=$(curl -o /dev/null -s -w "%{http_code}" "$url")
  echo "$url - $status"
done

# Find broken links with wget
wget --spider --force-html -r -l2 https://www.yourdomain.com 2>&1 | grep "404"

7 Fix Server Errors (5xx)

5xx errors completely block crawling. Googlebot will retry a few times, but persistent errors lead to deindexing. Check GSC for server error spikes. Common causes: overloaded servers, exhausted PHP memory, database connection failures, or faulty CDN configurations.

8 Find and Fix Orphan Pages

Orphan pages have zero internal links pointing to them. They are invisible to crawlers because spiders navigate by following links. Cross-reference your sitemap with a full crawl to identify orphans, then add contextual internal links from relevant parent pages.

# Orphan detection process:
# 1. Export all URLs from your sitemap
# 2. Run a full site crawl
# 3. Cross-reference: URLs in sitemap but not in crawl = orphans
# 4. For each orphan: find a relevant page and add a link

# Use Screaming Frog's "Extraction" feature
# Export all inlinks per page, filter for pages with 0 inlinks

9 Audit Canonical Tags

Canonical tags tell search engines which version of a page is the authoritative one. Common mistakes: canonicals pointing to different domains, multiple canonicals on one page, canonical pointing to a 404, or missing self-referencing canonicals on paginated pages.

<!-- Self-referencing canonical -->
<link rel="canonical" href="https://www.yourdomain.com/current-page" />

<!-- Cross-domain canonical (rare, use with caution) -->
<link rel="canonical" href="https://www.yourdomain.com/original-article" />

10 Check Pagination Implementation

Paginated series confuse crawlers without proper signals. Use rel="next" and rel="prev" to show relationships between pages. Apply a self-referencing canonical on each page. For SEO, consider merging paginated content into a single page or using "View All" pages.

<!-- Page 1 of a series -->
<link rel="next" href="https://www.yourdomain.com/series/page/2/" />
<link rel="canonical" href="https://www.yourdomain.com/series/" />

<!-- Page 2 of a series -->
<link rel="prev" href="https://www.yourdomain.com/series/" />
<link rel="next" href="https://www.yourdomain.com/series/page/3/" />
<link rel="canonical" href="https://www.yourdomain.com/series/" />

11 Implement IndexNow

IndexNow notifies search engines the instant content changes, bypassing the normal recrawl schedule. Supported by Bing, Yandex, Naver, and Seznam (Google uses a similar but separate system). Every content management system should ping IndexNow on publish or update.

# IndexNow API call
curl -X POST "https://api.indexnow.org/indexnow" \
  -H "Content-Type: application/json" \
  -d '{
    "host": "www.yourdomain.com",
    "key": "a1b2c3d4e5f6",
    "keyLocation": "https://www.yourdomain.com/a1b2c3d4e5f6.txt",
    "urlList": [
      "https://www.yourdomain.com/new-page",
      "https://www.yourdomain.com/updated-page"
    ]
  }'

# Generate your IndexNow key
openssl rand -hex 16 > a1b2c3d4e5f6.txt
# Place the key file at the root of your domain

12 Review Crawl Stats and Crawl Budget

Google allocates limited crawl resources to each site. If your site has thousands of low-value URLs, Google may not crawl your best content frequently. Block thin pages, consolidate duplicate content, and monitor crawl stats in GSC to see how Google spends its budget.

Check the Crawl Stats report in GSC Settings. Look for: total crawl requests, average response time, and host status. A sudden drop in crawl requests may indicate a server issue or robots.txt change.

13 Audit JavaScript Rendering for SEO

Google renders JavaScript, but with limits. Critical content (headings, text, links) must be in the initial HTML response, not injected via JS. Use the URL Inspection tool to see the rendered HTML. If Google sees a blank page, your JS SEO is broken.

<!-- Server-side render critical content -->
<h1>This heading is visible without JavaScript</h1>
<div id="app">
  <!-- Static fallback content -->
  <p>Core content that renders without JS.</p>
</div>

<!-- Use dynamic rendering as a safety net -->
# Nginx: Serve static cached version to bots
if ($http_user_agent ~* "Googlebot|Bingbot|GPTBot") {
  rewrite ^(.*)$ /__static$1 last;
}

14 Check HTTP Status Codes on All Pages

Every page on your site should return the correct HTTP status code. 200 for live pages, 301/308 for moved pages, 410 for permanently deleted content, and 404 only when the page truly does not exist. Pages returning 200 with thin or no content confuse crawlers.

15 Verify Googlebot Access and Rendering

Use the URL Inspection tool in Google Search Console to test how Googlebot sees each page. Check the "Crawled" and "Indexing" tabs. If Googlebot cannot access resources (CSS, JS, images), the rendered page may be incomplete or broken.

# Test with Googlebot user agent
curl -A "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) \
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z \
Mobile Safari/537.36 (compatible; Googlebot/2.1; \
+http://www.google.com/bot.html)" \
https://www.yourdomain.com

16 Enable Compression (Brotli/Gzip)

Compression reduces transfer size by 60-80%, speeding up crawl time and improving server response. Brotli offers better compression than Gzip and is supported by all modern browsers and crawlers.

# Nginx: Enable Brotli
brotli on;
brotli_types text/html text/css application/javascript image/svg+xml;
brotli_comp_level 6;

# Apache: Enable Brotli
AddOutputFilterByType BROTLI_COMPRESS text/html text/css application/javascript

17 Review Log Files for Crawl Patterns

Server log files reveal how Googlebot actually crawls your site, not just what GSC reports. GSC shows what Google found. Logs show what Google attempted, how often, and at what speed. This is your most accurate source of crawl behavior data.

Log file analysis answers questions GSC cannot: Which bots visit most frequently? Which URL patterns get the most crawl attention? Is Googlebot hitting rate limits? Are there abnormal crawl patterns that indicate a technical problem?

# Quick log analysis with GoAccess
goaccess /var/log/nginx/access.log \
  --log-format=COMBINED \
  --agent-list \
  --ignore-crawlers \
  -o report.html

# Filter for Googlebot only
grep "Googlebot" /var/log/nginx/access.log | \
  awk '{print $7}' | sort | uniq -c | sort -rn | head -20

# Check crawl frequency per URL
awk '$1 ~ /Googlebot/ {count[$7]++}
  END {for (url in count) print count[url], url}' \
  /var/log/nginx/access.log | sort -rn | head -20

# Detect crawl rate limiting (429 responses)
grep " 429 " /var/log/nginx/access.log | \
  grep "Googlebot" | awk '{print $7}' | sort | uniq -c

What to look for: Sudden drops in crawl volume (may indicate a robots.txt block or server issue), excessive crawl on low-value pages (crawl budget waste), 429 rate-limiting responses (server too slow), and uncrawled important pages (internal linking gap). Cross-reference log patterns with GSC crawl stats monthly.

Tools for log analysis: GoAccess (free, terminal-based), AWStats (free, web UI), Logstash + Elasticsearch (enterprise), and Screaming Frog Log File Analyzer (SEO-specific, paid). For most sites, GoAccess run weekly on the last 7 days of logs provides sufficient insight.

18 Implement hreflang for Multilingual Sites

If your site serves content in multiple languages, hreflang tags tell search engines which language version to show in each region. Incorrect hreflang implementation causes wrong pages to appear in wrong regions or no pages to appear at all.

<link rel="alternate" hreflang="en" href="https://www.yourdomain.com/page" />
<link rel="alternate" hreflang="es" href="https://www.yourdomain.com/es/pagina" />
<link rel="alternate" hreflang="fr" href="https://www.yourdomain.com/fr/page" />
<link rel="alternate" hreflang="x-default" href="https://www.yourdomain.com/page" />

<!-- Or use sitemap-based hreflang -->
<url>
  <loc>https://www.yourdomain.com/page</loc>
  <xhtml:link rel="alternate" hreflang="es"
    href="https://www.yourdomain.com/es/pagina"/>
  <xhtml:link rel="alternate" hreflang="fr"
    href="https://www.yourdomain.com/fr/page"/>
</url>

19 Test with Text-Only Browser View

Use a text-only browser or disable JavaScript in Chrome DevTools to see what crawlers experience. If critical navigation or content disappears, your site relies too heavily on JavaScript for crawling. AI agents often parse raw HTML without executing JS, making this test essential for GEO readiness.

20 Set Up Weekly GSC Monitoring Alerts

GSC is your early warning system for crawl issues. Set up email alerts for: index coverage drops, new 404 spikes, server error increases, manual actions, and Core Web Vitals regressions. A 5-minute weekly check prevents small issues from becoming ranking disasters.