How I Built an Automated Technical SEO Monitoring System for a 2,500 Page Website - and What It Found in 48 Hours

I’ve spent eight years doing SEO. I have never been a developer but i was a computer science engineering student back in the day. I can read code, I can break code, and until recently I could not write a working script without three hours and a lot of swearing. More about my SEO experience → pradhumnyakhanayat.com

Two months ago I got tired of finding out about technical problems from a traffic graph. So I built a monitoring system that tells me instead.

It runs on free infrastructure. It checks 2,520 pages. It cost me nothing but time, and in its first 48 hours it found around 60 genuine faults on the site I manage — including 22 pages that were quietly telling Google to index somebody else.

This is the full account: the architecture, the decisions, the mistakes, the code-level gotchas nobody warns you about, and the things I’d still like to solve. If you manage a large site on a CMS where pages get cloned, you’ll recognise the pain in the next section immediately.

What the SEO monitoring system checks

Check	What it detects	Why it matters
Google URL Inspection	Indexing changes	Finds pages Google has stopped indexing
HTTP status	404/5xx errors	Detects broken URLs
Canonical	Wrong/missing canonical	Finds pages telling Google to index another URL
Robots meta	noindex	Detects accidental exclusion
X-Robots-Tag	HTTP-level noindex	Catches non-HTML indexing directives
Sitemap	URL changes/errors	Identifies sitemap health issues
Title/H1	Metadata mismatch	Detects likely CMS clones
Content hash	Major content changes	Detects unexpected page changes
Response time	Slow pages	Finds sudden performance degradation
robots.txt	Blocking changes	Detects catastrophic crawl restrictions

Who Is This Technical SEO Monitoring System For?

This guide is for you if:

• You manage a website with 1,000+ URLs.

• Your CMS allows teams to clone existing pages.

• You rely on Google Search Console but usually discover problems after traffic drops.

• You want technical SEO alerts without paying for an enterprise crawler.

• You're an SEO who can read HTML but isn't a developer.

• You want to automate repetitive technical SEO checks with tools like GitHub Actions.

Why I Built an Automated Technical SEO Monitoring System?

I work on the search visibility of upes- a university website of roughly 3,000 indexable pages. Course pages, faculty profiles, admissions pages, a large blog, event pages, sustainability reporting. A big, busy, multi-team site.

Two problems kept costing me traffic, and both share a single characteristic: by the time I noticed, the damage was already several weeks old.

1: The problem with discovering SEO issues after traffic drops

The normal workflow for a technical SEO is reactive by design. You open Search Console, you see a dip, you investigate, you find the cause, you fix it, you wait for recovery.

That’s fine when the dip is small. It’s brutal during admissions season, when a course page dropping out of the index for three weeks is a measurable number of lost applications.

The specific things I wanted to know about as they happened, not later:

• A page that was indexed and is now excluded

• 404s and 500s on pages still listed in my sitemap

• A noindex tag appearing somewhere it shouldn’t

• A sitemap that Google has stopped reading

• A page that suddenly stopped earning impressions

Search Console reports all of this. But Search Console is a place you have to remember to visit, its data lags two to three days, and its “Page indexing” report has no API. So checking it thoroughly is manual work, and manual work doesn’t happen on the days you’re busy.

I've seen this failure mode from the other direction too: a host connectivity problem can trigger a dramatic indexing and traffic collapse. I documented that incident separately in my [Google Search Console host problems case study].

Duplicate Content Created by CMS Template Cloning

This is the one that really hurt, and it’s the reason I think this article is worth writing.

Our site runs on Drupal. When a team member needs a new course page or faculty profile, the fastest route is to clone an existing page and edit it. Everybody does this. It’s sensible. It preserves the template, the components, the layout.

The problem is what gets left behind. The visible content gets rewritten. The <head> often doesn’t.

So the new page goes live with the old page’s title tag and — critically — the old page’s rel=canonical. A canonical pointing at another URL is a strong signal to Google that the other URL is the preferred version. Google can choose to follow or ignore that signal, but on a CMS-cloned page it can contribute to the wrong URL being selected for indexing.

The page looks perfect to every human who checks it. It renders, it’s on-brand, the content is unique and good. And it is completely invisible in search, because its own metadata is voting against it.

You cannot catch this by browsing. You cannot catch it by spot-checking. It’s silent, it’s sporadic, and on a site where cloning is the normal way to create a page, it accumulates.

That’s the problem I actually wanted to solve.

The Architecture: Three Decisions That Shaped Everything

Decision 1: Where It Runs — And Why Not My Laptop

My work laptop is company-managed and intercepts HTTPS traffic for security scanning. This breaks Python’s certificate validation against Google’s APIs. Every script fails with an SSL error.

I could have exported the corporate root certificate and pointed Python at it via REQUESTS_CA_BUNDLE — that is the correct fix, and I’ll come back to it. But a monitoring system that only runs when my laptop is open and awake isn’t monitoring. It’s a script I sometimes remember to run.

I chose GitHub Actions. It’s a scheduler that runs code in the cloud, free for private repositories, with encrypted secret storage built in. No server to maintain, no laptop dependency, and it sidesteps the certificate problem entirely because nothing runs locally.

If you’re weighing this up: a small cloud VM with cron works equally well and gives you more control. GitHub Actions wins on zero setup and zero cost at this scale.

Decision 2: Where the Memory Lives — The Non-Obvious Problem

Here’s something I didn’t anticipate, and it’s the first thing that will bite you.

The whole system works by comparing “now” to “last time.” “This page is indexed” is not information. “This page was indexed and now isn’t” is an alert. That comparison requires memory.

But GitHub Actions wipes the machine after every single run. There is no disk that persists. So where does the previous state live?

My answer: commit it back into the repository. After every run, the system writes its state as JSON and CSV files and commits them.

This sounds odd for about ten seconds and then becomes obviously right:

• It’s free, and needs no database or external service

• Every run produces a git commit, so git log is a complete timestamped history of every change Google has ever reported about the site

• Anyone can open the files in Excel and understand them without special tooling

That last point matters more than it sounds. And the git history solves a requirement I hadn’t even got to yet: before-and-after snapshots of page content. It comes free with the storage choice.

Decision 3: Two Sources of Truth, Not One

The system asks two completely different questions, and understanding why is the key architectural insight.

Google’s opinion (via the Search Console API): authoritative, but 2–3 days behind and quota-limited. Google is the referee — if Google says a page isn’t indexed, it isn’t indexed.

My own server’s behaviour (via direct HTTP requests): instant, unlimited, and it sees the cause rather than the consequence.

That distinction is the whole design. A broken canonical today becomes a de-indexed page in Google next week. Checking Google alone means always finding out after the fact. Checking your own HTML means catching the cause while it’s still cheap.

So I built both, in phases.

Phase 1: The Indexing and Crawl Health Monitor

This layer asks Google what it currently thinks of my pages, records the answers, and reports the transitions.

What It Actually Does

1.Reads every URL from the sitemap

2.Ranks those URLs by real traffic (Search Console clicks + GA4 organic sessions)

3.Splits them into tiers, because of a quota limit I’ll explain in a moment

4.Calls the URL Inspection API for each URL in scope

5.Compares each result against the previous run

6.Reports only what changed

The Google Search Console URL inspection API Quota Limit That Shapes Everything

The Google Search Console URL Inspection API allows 2,000 queries per day and 600 per minute, per property. That means a 2,520-page website cannot inspect every URL every day using URL Inspection alone. That’s the single most important constraint in this build, and it’s not negotiable.

My site has around 2,520 monitored URLs. So “check everything daily” is just out of reach. This forces a design decision, and the design decision turns out to be the right SEO answer anyway:

Tier	What’s in it	Checked
Tier A (~400 URLs)	Course pages, admissions pages, top pages by real traffic	Every run
Tier B (~2,120 URLs)	Everything else	Rotating, full cycle every ~2 runs

Importance is scored from actual data — clicks × 10, plus GA4 organic sessions × 5, plus impressions × 0.1- not from guesswork.

But there’s a catch that took me a while to spot. A brand-new course page has no traffic history, so it scores zero and lands in Tier B. That’s exactly backwards: new pages are precisely where the cloning problem shows up. So I added a pattern override — any URL containing /btech, /mba, /admission, /school-of- and so on is forced into Tier A from the day it appears in the sitemap, regardless of score.

Which caused a different bug. More on that in the mistakes section.

How Can You Detect De-indexed Pages Without Inspecting Every URL?

URL Inspection is quota-limited. The Search Analytics API isn’t — one query returns every page that got any search activity.

So I use it as a broad, cheap de-indexing detector with pure SEO reasoning behind it: a page that reliably earned impressions for 28 days and has earned zero in the last 7 has almost certainly stopped ranking.

It’s a hint, not proof. Seasonality, a deliberate retirement, or a genuine ranking loss produce the same signal. But it covers every URL every day for almost no cost, and it catches things the Tier B rotation hasn’t reached yet.

On the very first run, this check flagged two pages. Both turned out to be real.

What Phase 1 Cannot Do

Being straight about limits, because the docs aren’t:

• There is no bulk index-coverage API. The “Page indexing” report you see in the Search Console interface has no API equivalent. You can only ask about URLs you already know from your sitemap. A page Google discovered that isn’t in your sitemap is invisible to this system.

• Search Console data lags 2–3 days. Nothing built on it is truly real-time.

• The Sitemaps API’s “indexed” count is no longer populated reliably and usually reads zero. Ignore it; use submitted, errors, warnings and lastDownloaded.

Phase 2: Checking My Own Site Directly

This is the layer that earned its keep, and if you build only one thing, build this one.

Every run, it requests every page on the site and checks:

Check	Why it matters
HTTP status	404s and 5xx on pages still in the sitemap
Redirects	A sitemap URL that has started redirecting — sitemaps should list final URLs
Canonical missing or malformed	Broken tags from data-entry mistakes
Canonical pointing to a different page	The single most damaging tag to get wrong
noindex	In the meta tag and the X-Robots-Tag HTTP header
Title vs H1 disagreement	The fingerprint of an unfinished page clone
Content fingerprint + word count	Catches a page that’s been gutted
Response time	A page that suddenly takes 8 seconds is a symptom
robots.txt	Hashed every run — Disallow: / is the most damaging one-line change anyone can make to a website

The Rule Change That Made It Work

• Phase 1 reports only changes, because Google’s opinions fluctuate.
  

• Phase 2 needed a different rule, and getting this wrong wasted a day. Defects in your own HTML don’t fluctuate. They just sit there. A canonical that’s been wrong for six months is still wrong, and nobody has ever told you.

So the Phase 2 checks for broken canonicals, noindex, and title/H1 mismatches fire on first sight, not only on change. Everything still de-duplicates after that — an already-reported problem doesn’t come back every two hours.

Canonicals become particularly dangerous during migrations too. An outdated canonical can continue pointing to the old domain even after redirects are in place, creating conflicting signals for search engines. I’ve documented the technical SEO checks, redirects, sitemaps and canonical issues to watch during a domain migration in my canonical tags during a domain migration guide.

Why Content Hashing Alone Would Have Missed the Real Problem

I originally assumed duplicate-content detection meant comparing page content. Hash each page, compare the hashes, flag the matches.

Then I looked at the worst real example on my site: a blog post whose body was entirely unique and well-written, but whose <title> and rel=canonical both belonged to a completely different article. It had been losing about 81 impressions a day.

Content hashing would never have caught it. The bodies are completely different. Only a <head> check catches it.

That reframed the whole phase. The clone-detection signal isn’t similar content — it’s mismatched metadata. And the strongest signal is two independent checks agreeing: a title that disagrees with the H1 and a canonical pointing somewhere else. Either alone is weak. Together they’re near-conclusive.

How to Crawl Your Own Website Without Overloading the Server

Worth saying plainly: 2,520 requests fired simultaneously at your own server looks exactly like an attack to a web application firewall, and can genuinely slow the site for real visitors.

The fetcher caps requests per second across all workers, uses a modest number of parallel workers, and sends an honest User-Agent naming the tool and my contact address — so whoever reads the server logs at 3am knows what it is and who to ask. Retries happen on connection errors and 5xx, never on a clean 404.

If your ops team ever queries the traffic, that User-Agent string is your answer, and lowering the rate limit is your fix.

Where the Alerts Actually Arrive (I Don’t Have Slack)

Most write-ups on this topic assume a Slack webhook. My organisation doesn’t use Slack, and company Microsoft 365 accounts usually block SMTP authentication, which rules out sending email directly. A third-party mail API would work but means another account and another key to manage.

I used GitHub Issues. It needs no setup at all, and it’s better than a chat message in three ways:

1. The notification email contains the findings. Not a link — the actual text: “Server error: HTTP 500 on /school-of-business/mba-marketing”.

2. It’s a worklist. Each alert is something to fix and close, with a permanent record of what was found when.

3. Phone notifications, free, via the GitHub mobile app.

The authentication token is provided automatically inside GitHub Actions. There is nothing to create.

The Design Detail That Prevents Alert Fatigue

This matters more than the detection logic, and it’s where most home-built monitoring dies.

Phase 2 runs up to twelve times a day. Twelve alerts a day would be worse than no alerts, because you’d mute the channel and then miss the real one.

So:

• Only CRITICAL and HIGH raise an alert. MEDIUM and INFO go to a CSV log only.

• One issue per day. Later runs the same day add a comment to it rather than opening another.

• Rules report transitions, so in steady state most runs find nothing and send nothing.

• Long lists truncate with a pointer to the full log, so the first full crawl doesn’t produce an unreadable wall of text.

Green vs Red Now Mean Different Things

An early version marked any run that found something as failed, so a run with findings looked identical to a run where the monitor itself crashed. I hit that confusion three times before fixing it.

Now:

• Green run + new issue → your site has a problem

• Red run → the monitor has a problem

• Green, no issue → nothing changed. This is the normal state.

Scheduling: The Constraint Nobody Mentions

GitHub gives private repositories 2,000 free Actions minutes per month, and billing rounds every run up to the next whole minute.

A full 2,520-URL sweep costs about 4–5 billed minutes. Hourly full sweeps would be roughly 3,500 minutes a month — over the allowance. That constraint forced a tiered schedule, which again happens to be the correct SEO answer:

Job	Frequency	Approx monthly minutes
Tier A page sweep (~400 pages)	Every 2 hours	~720
Full sweep (2,520 pages)	3× daily	~450
Phase 1 indexing check	Daily, 08:00 IST	~480
Total		~1,650 of 2,000

• Cron is UTC only and best-effort. My 03:30 UTC job actually started at 03:42. If you need results by a certain time, schedule earlier than that time, not at it.

• Scheduled workflows are auto-disabled after 60 days of repository inactivity. This can’t affect a system that commits its own results daily — but it will silently kill a workflow that doesn’t write anything.

What My SEO Monitoring System Found in the First 48 Hours

This is the part I didn’t expect to be so blunt.

The Good News First

All 2,520 pages returned HTTP 200. No 404s, no server errors, no unreachable pages. Server health was genuinely clean, and knowing that with certainty is worth something on its own.

The Faculty Profile Pages with incorrect canonicals

Around 22 faculty profile pages had the title and canonical of a different professor. The pattern was identical every time: clone the previous person’s page, change the visible name, leave the head untouched.

Every one of those faculty members was invisible in Google search. Their page was telling Google to index a colleague instead.

Blog pages forming canonical clusters

This is where a site-wide view beats spot-checking, and it’s the single most useful view the system produces.

One page canonicalised elsewhere might be a deliberate consolidation. But when the system grouped pages by canonical target, the picture changed completely:

• Four blog posts all canonicalised to one unrelated article

• Three more all pointing at a single different article

• Several two-page clusters

Four pages pointing at one target is never deliberate. It’s a template being cloned repeatedly without the canonical being updated — and every one of those pages was switched off in Google. Several were targeting high-volume commercial queries.

12 Different Technical Canonical Bugs

Around 12 pages had canonical tags that were broken as URLs, in six distinct varieties:

• The domain appearing twice — an absolute URL pasted into a field expecting a relative path

• The same mistake without the leading slash, which corrupts the domain and points at a host that doesn’t exist

• A double slash in the path

• The section name entered twice (/research/research/...)

• Capitalisation-only differences, which Google treats as separate pages

• The wrong protocol and a missing www, pointing outside the property entirely

Google usually spots an invalid canonical and ignores it, so these are less damaging. But each one tells you the CMS field is being filled in wrongly, which means it’s happening on pages you haven’t looked at.

The Root Cause Behind Almost All of It

Every Priority 1 and 2 item existed for one reason: the canonical is a manually-entered field, and a manually-entered canonical survives a page clone.

On a site where cloning is the normal way to create a page, a manual canonical field is a trap. The most valuable output of this entire build wasn’t a list of 60 fixes. It was one question to ask the CMS team: can the canonical field auto-generate from the page URL? Change that one setting and the whole category stops recurring.

13 Problems I Hit While Building the System

If you build this, you’ll meet most of these. In rough order of how much time each one cost me.

1. The API Is Slower Than You Think, and It Will Time Out

The URL Inspection API takes about eight seconds per call. Not 0.2 seconds. Eight.

Do the arithmetic: 1,800 URLs × 8 seconds = four hours. My workflow had a 45-minute timeout, so my first full scheduled run was killed a third of the way through. I have the cancelled run in my history to prove it.

The fix is concurrency. Google’s limit is 600 calls per minute and I was doing seven. Running 16 requests in parallel turned a four-hour job into fifteen minutes, and still only reaches about 120 requests per minute — well inside the ceiling. Almost all of that eight seconds is spent waiting for Google, so doing many at once costs nothing.

Measure your API latency before you design your schedule. Don’t assume.

2. Your Service Account Is Probably on the Wrong Property

My first diagnostic run came back clean on the credentials and completely broken on everything else. The reason: the service account had Full access to a different website in the same organisation, and no access at all to the one I wanted.

Everything downstream failed with “You do not own this site”, which reads like an API problem and isn’t.

Check sites().list() first and print exactly which properties the account can see. It’s the cheapest diagnostic there is.

3. “Restricted” Permission Silently Breaks URL Inspection

Search Console has Owner, Full, Restricted and Associate permission levels. A Restricted user commonly cannot use the URL Inspection API even though other endpoints work fine.

The resulting error doesn’t make the cause obvious. Add the service account as a Full user. Also note that only an Owner can add users, so if you’re a Full user yourself, you’ll need to ask someone.

4. The Search Console API Is Separate From the GA4 API

Enabling the Google Analytics Data API does not enable the Search Console API. They’re separate products in Google Cloud and each needs enabling individually. Obvious in hindsight, invisible at the time.

5. HTTP 403 Means Two Completely Different Things

Google returns 403 both for “you’re going too fast” (retry and it’ll work) and “you don’t have permission” (retrying is pointless).

My first version retried everything five times with exponential backoff. So a permission error took 30 seconds of pointless waiting before reporting the same failure.

Inspect the response body, not just the status code. A 403 mentioning rateLimitExceeded deserves a retry. One mentioning “do not own this site” or “insufficient permission” should fail immediately with a message naming the real cause.

6. Search Analytics Silently Truncates at 25,000 Rows

The Search Analytics API returns a maximum of 25,000 rows per request. If you don’t paginate with startRow, you get the first page and no error — just quietly missing data.

A page-level query on a 2,520-page site fits. A query-plus-page breakdown blows past it immediately. Always loop until a page returns fewer rows than the limit.

7. Heuristics Will Cry Wolf, and That’s the Real Risk

My “title and H1 describe different topics” check is a heuristic, and its first live run produced eight HIGH-severity findings. Seven were false alarms.

The worst was a contact page: title “UPES University - Contact Us”, H1 “Contact Us”. Obviously fine. But my code split the title at the dash and compared only “UPES University” against “Contact Us”, found nothing in common, and raised an alert.

Three separate bugs behind it:

• Truncating titles at dashes instead of removing the brand name wherever it appears

• No plural handling — “Download Brochures” didn’t match “Downloads”

• Judging headings too short to judge — an H1 of “Downloads” reduces to one word, and a one-word-versus-many comparison is noise, not evidence

The fix that mattered most was a rule: both sides must have at least two meaningful words before the check will make a judgement. Losing a little sensitivity there is the right trade, because a genuine clone almost always also has a wrong canonical, and that check is exact rather than heuristic.

I then turned all eight false alarms into permanent test cases. If a future change starts flagging “Contact Us” again, the test suite fails before it ever runs against the live site.

If you take one thing from this article, take this: a monitoring system that cries wolf gets muted, and a muted monitor is worth exactly nothing. Spend more effort on suppressing false positives than on adding checks.

8. A Broad Pattern Match Will Swallow Your Priority List

I forced anything matching /school-of- into Tier A, reasoning that school and program pages are the money pages.

Faculty profile URLs on my site look like /faculty/school-of-business/name. They contain /school-of-. So half the site got forced into Tier A, and Tier A came out at 1,250 URLs against a configured limit of 400 — pushing the actual course pages out of the daily check.

Test your priority patterns against a real sample of your URLs before trusting them. And add an exclusion list, checked first.

9. Your CMS Sitemap May Tell You Nothing Useful

My sitemap has no lastmod dates at all, and every entry carries changefreq=weekly with priority 0.9 or 1.0. Those fields are boilerplate.

This matters: it means the sitemap can’t tell you which pages changed recently, so change detection has to work by hashing page content instead. Check what your sitemap actually contains before designing anything that relies on it. If you can get real lastmod values out of your CMS, that’s a genuinely valuable ask.

10. A Failed Run Can Silently Skip Part of Your Site

Subtle and nasty. My rotation pointer advanced even when a run died partway through, which meant the URLs it never reached were marked as “done” and skipped until the next full cycle.

Only advance a rotation pointer after the work actually completes. An interrupted run should retry, not skip.

11. Windows Hides the Folder That Makes Everything Work

Practical and annoying. GitHub Actions workflows live in a folder called .github. Windows File Explorer hides folders beginning with a dot by default, so when I dragged my files up to GitHub, that folder silently didn’t come with them — and nothing ran.

Turn on View → Hidden items first. Then verify the folder actually appears in your repository before wondering why the schedule isn’t firing.

12. Your Fetcher Doesn’t Run JavaScript

Mine requests raw HTML. Anything your CMS renders client-side is invisible to it.

I got lucky — titles, canonicals and robots tags are all server-rendered on my site, and the missing H1s I found turned out to be genuinely missing rather than JS-injected. Verify this for your own site before trusting a “missing element” finding. View source on a page you know is fine and check the element is actually there.

13. Certificate Interception on Corporate Laptops

If your company laptop intercepts HTTPS, Python will reject Google’s certificates and every script fails locally.

The correct fix is to export your organisation’s root CA certificate and point Python at it:

REQUESTS_CA_BUNDLE = C:\certs\company-root-ca.pem

SSL_CERT_FILE = C:\certs\company-root-ca.pem

Never “fix” this by disabling certificate verification. That turns off the check that stops someone impersonating Google, in a script holding your service-account key. Running in the cloud avoids the problem entirely, which is part of why I chose to.

What I Learned From Automating Technical SEO Monitoring

The First Full Crawl Is an Audit, Not an Alert

My first complete run produced 317 findings. That number panicked me for about a minute.

It broke down as roughly 60 things genuinely worth fixing, around 90 template-level title/H1 patterns that were deliberate design choices rather than faults, and around 120 pages missing an H1 — a real but low-priority site-wide template issue.

Nothing had just broken. Several checks report problems the first time they see them, and some of those problems had been sitting there for months.

Read your first full run as a backlog audit. From the second run onward it only tells you what changed. That transition — from “here’s everything wrong” to “here’s what broke since yesterday” — is the moment it stops being an audit tool and becomes an early-warning system.

Root causes matter more than individual errors

40 scattered canonical findings became 5 fixable root causes once I grouped them. Grouping also tells you who fixes them: one CMS configuration change, one content-team briefing, one sitemap regeneration.

A list of 300 tickets gets ignored. A list of five root causes gets done.

Monitoring should prioritize business-critical URLs

After the canonical fixes deploy, Google has to re-crawl before anything recovers. Expect two to six weeks before impressions return on the affected pages. Watch Search Console rather than refreshing your rank tracker.

What I’d Improve in the next Phase

This is where I’d genuinely like other people’s input, because I don’t think I’ve solved these well.

Getting Below Google’s Crawl Delay

Right now the fastest signal is my own HTTP check. But there’s a faster one available: connecting directly to the CMS.

Drupal API monitoring

Drupal’s JSON:API can be polled for recently created or updated nodes (?sort=-changed). That would surface a cloned page within minutes of it being saved, with the raw field content, before Google ever crawls it. That’s the difference between catching a duplicate in minutes and catching it in a day.

I don’t have that access, so I couldn’t build it. If you’re building this and you do control your CMS, start there- it’s strictly better than crawling. If you’ve built CMS-webhook-driven SEO monitoring, I’d love to hear how you structured it.

Duplicate Detection Beyond Metadata

My current clone detection is metadata-based, which works because metadata is what actually gets left behind. But it won’t catch a genuinely near-duplicate page where somebody did update the head and only lightly edited the body.

For that you need real text similarity. The candidate approaches:

• Shingling + MinHash / SimHash — cheap, scales to thousands of pages, catches near-duplicates well. My instinct is that this is the right answer.

• TF-IDF + cosine similarity — more accurate on medium-sized corpora, but comparing every page against every other page is quadratic and gets expensive fast.

• Sentence embeddings — catches semantic duplication that word-overlap methods miss, but needs a model and a vector store, which is a big jump in complexity.

I’m collecting content hashes already, so whichever I pick starts with history rather than from zero. My question for anyone who’s done this at scale: is MinHash good enough, or does semantic similarity earn its complexity?

Grouping Alerts by Root Cause Automatically

Right now the clone clusters are printed in the run log, and I do the root-cause grouping by reading it. The alert itself is still a flat list.

It should cluster automatically: “22 faculty pages share this fault, here’s the one fix.” Very doable, not yet done.

What Else Belongs in Phase 2

The obvious candidates, roughly in the order I’d add them:

• Redirect chain depth — a sitemap URL that redirects twice

• hreflang reciprocity, if you’re multilingual

• Structured data validation — schema breaking silently is a classic

• Core Web Vitals via the CrUX API — real-user data, no crawling needed

• Internal link changes — a page losing all its internal links is usually accidental

The Question I Keep Coming Back To

Everything above makes detection better. But my actual bottleneck isn’t detection any more- it’s the fix cycle. The system finds a broken canonical in minutes; getting it corrected in the CMS takes days of coordination across teams.

So the highest-leverage improvement might not be a better crawler at all. It might be prevention: CMS validation that refuses to save a page whose canonical points at a different page. Detection is a workaround for a system that lets the mistake happen.

Has anyone actually solved the prevention side? That’s the conversation I’d most like to have.

Technical SEO monitoring framework

Here's The 4-Layer Technical SEO Monitoring Framework

1. Layer 1 - Google signalsSearch Console / indexing

2. Layer 2 - Website signalsHTTP / HTML / canonical / robots.txt / metadata

3. Layer 3 - Historical signalsHashes / state / Git

4. Layer 4 - Action signalsSeverity / alerts / GitHub Issues

Frequently Asked Questions

How do I detect de-indexed pages automatically?

Two complementary methods. Call the Search Console URL Inspection API on a schedule and compare each page’s coverageState and verdict against the previous run — a page moving from PASS to anything else has lost its indexing. Then, as a broader and cheaper backstop, use the Search Analytics API to find pages that earned impressions consistently over 28 days and zero in the last 7. The first is authoritative but quota-limited; the second covers your whole site for almost nothing.

What is the Google Search Console URL Inspection API quota?

2,000 queries per day and 600 per minute, per property. Design around it from the start: if your site has more URLs than the daily budget, you need to tier by importance and rotate the remainder. Also budget for latency- each call takes roughly 8 seconds, so run them concurrently or a large site will take hours.

How do I find duplicate content created by CMS template cloning?

Don’t start with content comparison — start with metadata comparison. When a page is cloned, the body usually gets rewritten but the <head> doesn’t. So check whether rel=canonical points at a different URL, whether the <title> and <h1> describe different topics, and whether several pages share the same canonical target. That last one is the strongest signal: four pages pointing at one target is never deliberate. Content hashing misses this entirely because the bodies genuinely differ.

Can I run technical SEO monitoring for free?

Yes, at this scale. GitHub Actions gives private repositories 2,000 free minutes a month, which covers a daily indexing check plus a site sweep every couple of hours on a 2,520-page site. The Search Console and Analytics APIs are free. The costs are your time to build it and your attention to keep the false-positive rate low.

Why is my page not indexed even though it loads perfectly?

The most common cause on a CMS-driven site is a rel=canonical tag pointing at a different URL — often left behind when the page was cloned from another. Google reads that as an instruction not to index this URL. Check for it with view-source and Ctrl+F for “canonical”. Other candidates: a stray noindex in the meta robots tag or the X-Robots-Tag HTTP header, a robots.txt disallow, or the page being genuinely too thin to earn indexing.

Where should SEO alerts be delivered if my company doesn’t use Slack?

GitHub Issues, if your monitoring already runs on GitHub. The notification email contains the actual findings rather than a link, it doubles as a worklist you close as you fix things, and the mobile app gives you free push notifications. No SMTP server, no third-party mail service, no API keys. Microsoft Teams and Google Chat both accept incoming webhooks if you’d rather have a shared channel.

How often should I check my site for technical SEO issues?

Match the frequency to how fast the signal changes and how much the page matters. Google-sourced data lags 2–3 days, so checking it more than daily is pointless. Direct HTTP checks have no lag, so your important pages are worth checking every couple of hours; the long tail every few hours is plenty. Frequency without good de-duplication just produces noise.

Want to build this yourself?

Start with just three checks: canonical, noindex and HTTP status. Don't build the entire system on day one. Get one useful alert working, then add indexing data, state comparison and prioritisation.