Technical Search Optimization

Technical SEO is now the eligibility layer
for two different indexes.

Google states the requirement plainly: to appear in its AI features a page must be indexed and eligible to be shown with a snippet. Microsoft Copilot answers from Bing, which is a separate index with its own submission surface. A page can be perfectly eligible in one and invisible in the other, and most technical audits only ever check one.

Built for home services contractors, franchise systems and private equity portfolios whose sites were built by someone else, years ago, and have been quietly losing visibility since. One location or two hundred, the crawl either works or it does not.

2
Separate indexes a page has to be eligible in, with two separate consoles reporting on them
0
Special files, markup or optimizations Google requires for its generative AI features
14
Google Search Central documents this page is built on, each linked at the point of use
What Has To Be True Before Anything Ranks
Four gates
1
Crawlable Not blocked by robots.txt or a login
2
Indexable No noindex, canonical resolving as intended
3
Snippet-eligible Google's stated requirement for AI features
4
Present in Bing Copilot's index, submitted separately
What Changed

Technical SEO didn't get harder. It got a second index to satisfy.

The work itself is recognizable — crawlability, indexing, speed, structure. What changed is that failing it now removes a site from more surfaces than it used to, and that one of those surfaces is not reading Google's index at all.

Diagram showing one website feeding two separate search indexes — Google's, which serves organic results, AI Overviews and AI Mode, and Bing's, which serves Microsoft Copilot — with the separate submission surface for each marked
Indexed
Plus snippet-eligible — Google's stated precondition for appearing in its AI features
Documented by Google Search Central
RAG
Retrieval-augmented generation — the technique Google names for grounding AI answers in indexed pages
Documented by Google Search Central
Bing
The index behind Microsoft Copilot, with its own webmaster tools and its own submission
Separate surface, separate reporting
None
Additional requirements or special optimizations Google names for AI Overviews and AI Mode
Documented by Google Search Central

Start with the mechanic, in Google's words rather than an agency's. Its generative features use retrieval-augmented generation — also called grounding — which relies on the core Search ranking systems to retrieve relevant, current pages from the index, then reviews the information on those pages to build a response with links back to the sources. Alongside it runs query fan-out, where the system issues several concurrent related queries behind the one that was typed.

Every part of that sentence is downstream of the index. A page that cannot be crawled is not in the index. A page carrying noindex is not in the index. A page whose canonical points somewhere else is represented in the index by that other URL. None of them can be retrieved, so none of them can be cited, however good the content is.

Google is the only one of these systems that documents how it works. Gemini and Google's AI Overviews and AI Mode draw on the same Search index described above, which is why the technical fundamentals move both at once. ChatGPT, Perplexity and Claude publish no equivalent account of how they retrieve. And Microsoft Copilot is a different problem entirely — it is not reading Google's index. It reads Bing's. Everything this page says about those four is observation rather than documentation, and is labeled that way wherever it appears.

That last point is the one most technical audits miss. A site can be immaculate in Search Console — indexed, fast, valid structured data — and be thin or absent in Bing, because Bing crawls separately, indexes separately, and is submitted to separately. The remedy is not exotic. It is remembering that the second index exists and treating sitemap submission and crawl access as things you do twice.

The Bridge

The checklist is the same. What failing it costs is not.

Nothing in the left column below is wrong. It is technical SEO calibrated for a world where the only prize was a position in a list of ten blue links.

Calibrated for the ranked list

Technical SEO as a hygiene exercise

  • Speed as a score. Chase a number in a testing tool, on whichever page the tool happened to load.
  • Crawlability checked once. Robots.txt reviewed at onboarding and never again, on Google only.
  • Schema as a rankings lever. Markup deployed because it is assumed to help position, not because it enables anything specific.
  • Duplicate content as a penalty myth. Framed as punishment rather than as consolidation and wasted crawling.
  • Sitemaps submitted to Google. One console, one index, one report.
  • Errors triaged by count. Fix the biggest number in the crawl report first.
Calibrated for retrieval and two indexes

Technical SEO as an eligibility exercise

  • Speed where the money is. Field data on the pages that actually convert, because page experience is a conversion problem that happens to be a search input.
  • Crawlability as a standing check. Access verified continuously and in both indexes, since a deploy can revoke it silently.
  • Schema for what it actually does. Deployed for rich-result eligibility and entity clarity — never sold as an AI unlock, because Google says it isn't one.
  • Duplication as retrieval confusion. Consolidated so one URL holds the signals and one page is the thing that gets retrieved.
  • Two submissions, two reports. Search Console and Bing Webmaster Tools, because Copilot reads the second one.
  • Errors triaged by consequence. One blocked template that removes a whole section beats four hundred cosmetic warnings.
The shift in one line: technical SEO used to decide how well you ranked. Now it decides whether you are available to be retrieved at all — and it decides that separately in each index.
The Eligibility Gates

Four things have to be true before content quality matters at all.

These are sequential. A site failing the first gate cannot be helped by anything done at the third, which is why the order of technical work matters more than its inventory.

C

Crawlable

The gate that fails silently
Governed byrobots.txt
Breaks onDeploys
Detect inMinutes
What Governs It

robots.txt tells crawlers which URLs they may request. It is the cheapest file on a site to get wrong and the most expensive to leave wrong, because nothing downstream reports a problem — the pages simply stop being fetched.

Where It Breaks
  • A staging Disallow: / promoted to production with the rest of the deploy
  • Blocked CSS or JavaScript that the page needs in order to render as users see it
  • Crawl access granted to Google and never checked for Bing
  • Content behind a login or an interaction, invisible to any crawler
I

Indexable

Where canonical decisions land
Governed bynoindex + canonical
Breaks onMigrations
Detect inDays
What Governs It

A noindex directive removes a page from results once the crawler sees it — which means the page must remain crawlable for the directive to be read at all. Blocking a page in robots.txt to remove it from the index does the opposite of what it looks like it does.

Where It Breaks
  • A template-level noindex surviving a launch, silently removing a whole section
  • Canonical tags pointing at the homepage from every page, collapsing the site to one URL
  • Redirect chains left behind by successive migrations
  • Parameters generating near-infinite URLs and spending the crawl budget on nothing
B

Present In Both

The gate almost nobody checks
Governed byTwo consoles
Breaks onAssumption
Detect inOne check
Why It Is Separate

Microsoft Copilot answers from the Bing index. Bing crawls with its own agent, maintains its own index, and is submitted to through Bing Webmaster Tools rather than Search Console. Google's generative AI performance report reports on Google's AI surfaces and tells you nothing whatsoever about Copilot.

What This Costs In Practice
  • Sites verified in one console and never the other, with no visibility into half the picture
  • Sitemaps submitted once and assumed to have been submitted everywhere
  • Crawl directives written for Googlebot that inadvertently exclude other agents
  • Reporting that shows healthy AI visibility while an entire assistant never names the business
The Diagnostic

The audit starts by asking what is currently invisible.

Most technical audits produce a long list sorted by error count. Ours sorts by consequence, because one blocked template outranks four hundred cosmetic warnings and the two are usually reported side by side as if they were comparable.

What the audit covers

The output is a prioritized set of findings with the evidence tier stated on each — documented by the issuing authority, observed by us, or inferred. Several have been handed to in-house teams who executed them without us.

  • Eligibility census. Every URL classified as crawlable, indexable and snippet-eligible — or not, with the specific directive responsible named.
  • Both indexes. Coverage compared across Search Console and Bing Webmaster Tools, since Copilot answers from the second and most sites have never looked at it.
  • Canonical and duplication map. Which URL each cluster consolidates to, and whether that is the URL you would have chosen.
  • Render check. What a crawler receives versus what a browser displays, on the templates that matter.
  • Field performance. Core Web Vitals on the pages that convert, using real-world data rather than a lab score from the homepage.
  • Structured data validity. Parsed, typed correctly, and agreeing with what the page actually says.
Audit output showing each site template classified as crawlable, indexable and snippet-eligible, with the blocking directive named beside each failure and the Google-versus-Bing coverage gap highlighted
The Execution Sequence

Eligibility first, because everything else depends on it.

Performance work on pages that cannot be indexed is money spent on something no system will ever retrieve. The order is not a preference.

Sequence diagram of a technical SEO engagement showing three dependent stages — restore eligibility, consolidate and accelerate, monitor and defend — with the failure path marked for running them out of order
30

Restore eligibility

Foundation
  • Crawl access audited and corrected in both indexes
  • Template-level noindex and canonical faults cleared
  • Redirect chains collapsed to single hops
  • Sitemaps rebuilt and submitted to both consoles
  • Render check on every template that carries content
  • Baseline captured in Search Console and Bing Webmaster Tools
OutcomeEverything worth retrieving is available to be retrieved.
60

Consolidate and accelerate

Acceleration
  • Duplicate clusters consolidated to the intended canonical
  • Core Web Vitals remediation on converting templates
  • Structured data deployed and validated against the page
  • Internal linking rebuilt so depth and equity follow priority
  • Crawl budget reclaimed from parameters and dead paths
  • Security and scheme migration finished properly
OutcomeOne URL per intent, fast, and legible to both indexes.
90

Monitor and defend

Compounding
  • Standing crawl and index monitoring, both indexes
  • Deploy-time checks so a release cannot revoke access silently
  • Generative AI performance reporting tracked as its own layer
  • Traffic movements worked through Google's debugging sequence
  • Structured data revalidated whenever templates change
  • Findings fed back into the content and linking plan
OutcomeRegressions get caught by a check rather than by a quarter of lost traffic.
The Work Itself

Eight characteristics of a technically sound site — and what each one is actually for.

This is the spine of the engagement. Each item below links to the primary documentation behind it, because the honest version of this list is short and the sold version usually isn't.

01

Page speed and experience

Core Web Vitals are the metrics Google reports for real-world page experience, and its page experience guidance frames them as part of delivering a good experience rather than as a single ranking dial.

The four classic culprits from the original version of this page still hold: bloated markup, unoptimized images, unnecessary redirect hops, and slow server response. We remediate on the pages that convert first, because a slow service page loses the call whether or not it costs a position.

02

Crawlability

Two controls do most of the work and are routinely confused. robots.txt governs what may be requested. The robots meta tag governs what may be indexed once it has been requested.

Using the first to achieve the second is the single most common technical error we inherit — the crawler never fetches the page, so it never reads the instruction telling it to drop the page.

03

Dead links and redirects

Broken internal links waste crawling and strand users. Redirects are the correct tool when a URL genuinely moves, and a liability when they accumulate into chains nobody has audited since the last replatform.

Our rule: a redirect exists to preserve something. If nothing links to the old URL and nothing ranks for it, the redirect is maintenance debt.

04

Duplicate content

Duplication is a consolidation problem before it is anything else — Google picks a canonical and reports on that one. Where it becomes a policy problem is scale: the spam policies define scaled content abuse as generating many pages primarily to manipulate rankings rather than to help people.

Mass-produced near-duplicate pages sit exactly on that line, which is why we consolidate them rather than rewriting them one adjective at a time.

05

Site security

HTTPS is table stakes and has been for years. The technical work that still matters is making sure the migration to it was completed properly: no mixed content, no internal links still pointing at the insecure scheme, and no redirect hop added to every request on the site.

A half-finished HTTPS migration is one of the few technical faults that taxes every single page load at once.

06

Structured data

Deploy it, and deploy it for the right reason. Structured data makes pages eligible for rich results and helps establish what a page is about.

"@type": "Service", "provider": , "areaServed":

Where we stop: Google states that structured data is not required for generative AI search and that no special schema markup exists for it. We do not bill it as an AI unlock.

07

XML sitemaps — plural

A sitemap tells search engines which pages you consider worth crawling. It is a hint, not a command, and it does not rescue a page that is blocked or noindexed.

The part that gets skipped: submitting it in both places. A sitemap filed with Search Console and never with Bing Webmaster Tools is doing half its job, and Copilot reads the half you skipped.

08

Rendering and testing

If content depends on JavaScript to appear, it depends on rendering to be indexed. That is a solvable problem and a routinely undiagnosed one — the page looks complete in a browser and arrives empty at the index.

When traffic moves unexpectedly, Google publishes a structured way to debug it rather than guessing, and we work that sequence before proposing anything.

Where It Bites Hardest

The same faults, different consequences.

Technical failure is universal. What it costs depends entirely on how the business acquires customers.

🏠

Home services

Mass-produced location pages are the standard inheritance, and they are simultaneously a duplication problem and a spam-policy exposure. Consolidating them usually recovers crawl budget that was being spent on pages nobody was ever going to read.
Pressure pointDuplicate location templates and slow mobile service pages. Pairs directly with local SEO and home services.
🔗

Franchise systems

Multisite architectures multiply every technical fault by the number of units. One bad template is not one problem — it is forty. Governance over who can change a template is a technical control, not an administrative one.
Pressure pointTemplate governance across units, and Bing Places accuracy, since franchise-development queries skew toward Copilot. See franchise systems.
🛒

Ecommerce

Faceted navigation generates URL space faster than any crawler will work through it, and parameter handling is where crawl budget is either preserved or squandered. Product duplication is structural rather than accidental.
Pressure pointParameter and facet control, canonical strategy across variants, and render dependence in cart and category templates.
🏢

Mid-market and B2B

Long buying cycles mean the research pages matter as much as the product pages, and those are exactly the pages most often left render-dependent, unlinked, or stranded behind a gate that no crawler can pass.
Pressure pointGated content invisible to retrieval, and JavaScript-dependent resource libraries. Related: private equity and competitor analysis.
Why Allegiant

Every claim here links to the documentation behind it.

Technical SEO is the one discipline where the issuing authority publishes almost everything you need. Google documents the requirements, the directives, the metrics and the debugging sequence. There is very little left to guess about — and yet most agency pages on this subject cite each other rather than the source.

Fourteen Google Search Central documents are linked on this page at the point of use, four of them updated within the last four months. Where we describe something Google has not published — how ChatGPT or Perplexity choose sources, for instance — we say that it is observation.

That is a harder page to write and a much easier agency to verify. It is also the only defensible position when the honest answer to several popular questions is the issuing authority has published nothing about that.

14
Primary-source documents behind the claims on this page
2
Indexes audited, because only one of them serves Copilot
Verified Credentials
G
Verified partner listing
S
Agency directory profile
C
Certified CallRail Agency
Call attribution
I
Inc. Power Partner
2024 and 2025
5
Agency ranking
B
Accredited profile
The Comparison

What separates this from a standard technical audit.

Not a competitor teardown — a description of where the practices differ, so the choice can be made on substance.

Side-by-side comparison of a conventional crawl report sorted by error count against an eligibility census sorted by consequence, showing one blocked template outranking several hundred cosmetic warnings
Practice Standard technical audit AllegiantOMNIVIZ™
Indexes audited Google only Google and Bing, because Copilot reads Bing
Finding priority Sorted by error count Sorted by consequence to eligibility
Speed measurement Lab score on the homepage Field data on pages that convert
Structured data Sold as an AI ranking lever Deployed for rich results, per Google's own wording
Sources cited Vendor blogs and secondary write-ups Issuing-authority documentation, linked in place
Regression control Re-audit next quarter Deploy-time checks so access cannot be revoked silently
Work we decline Rarely itemized Published on this page, with sources
What We Decline To Sell

Five technical line items you can stop paying for.

Each is drawn from Google's own mythbusting guidance for its generative AI features. If a technical proposal on your desk includes one of these, this is the source to read before signing.

AI text files. Google states you do not need to create AI text files such as llms.txt, special markup, or Markdown versions of pages to appear in Search including its generative capabilities, because Search does not use them. Maintaining one for other systems is fine and will neither help nor harm Google visibility.

Content chunking. There is no requirement to break content into small pieces for AI to understand it, and no ideal page length exists. Structure pages for readers.

Schema sold as an AI unlock. Structured data is not required for generative AI search and there is no special markup for it. Deploy it for rich-result eligibility — a real reason, and a different one.

Rewriting pages into an "AI-friendly" voice. Not needed. The systems understand synonyms and general meaning, which is also why generating a separate page for every query variation is wasted budget and a scaled-content risk.

Tools claiming inside knowledge. Google states plainly that no third-party tool has access to its internal ranking or AI systems, and publishes guidance on evaluating third-party SEO advice. That includes our own tooling, which we label observational for exactly this reason.

The pattern beneath all five: there is no separate AI technical channel to buy. There is a site that is eligible to be retrieved, or one that isn't.

Evidence note. Claims attributed to Google are drawn from its published documentation, linked at the point of use and verified live at the review date shown in the byline. Claims describing patterns Allegiant has observed across engagements are identified as observations in the text. Where neither applies, no claim is made. Correction on record: the previous version of this page sourced its page-experience claim to a vendor blog. That citation has been replaced with Google's own page-experience and Core Web Vitals documentation.
Frequently Asked

The questions operators actually ask.

Answered against primary sources where they exist, and answered honestly where they do not.

Every answer below links at least one issuing-authority source
Does technical SEO still matter if AI is answering the question?+

It matters more, because it became the eligibility layer rather than a ranking tweak. Google requires that a page be indexed and eligible to be shown with a snippet in order to appear in its generative features, on top of the ordinary technical requirements. A page that fails those cannot be retrieved, so it cannot be cited, regardless of how good the content is. That is the same foundation described in our AI SEO practice.

We rank fine in Google. Why would we care about Bing?+

Because Microsoft Copilot answers from the Bing index, not Google's. Bing crawls with its own agent, maintains its own index, and is submitted to through its own webmaster tools. Google's generative AI performance report covers AI Overviews and AI Mode and tells you nothing about Copilot at all. A site can be healthy in Search Console and thin in Bing without anyone noticing, because nobody is looking at the second report. The remedy is unglamorous: verify the site there, submit the sitemap there too, and check that crawl directives aren't excluding non-Google agents. It runs alongside the local SEO work for any business with a physical footprint.

Can we just block a page in robots.txt to get it out of Google?+

No, and this is the most common technical error we inherit. robots.txt controls whether a URL may be requested. A noindex directive controls whether it may be indexed — but the crawler has to fetch the page to read that directive. Block the page and the instruction is never seen, so the URL can persist in results with no description. Use noindex and leave the page crawlable. Related cleanup usually runs through website maintenance.

Do we need structured data to show up in AI answers?+

No. Google states that structured data is not required for generative AI search and that there is no special schema markup to add for it. Deploy it anyway, for a different and legitimate reason: structured data makes pages eligible for rich results and helps clarify what a page is about. Any proposal that prices schema as an AI unlock is pricing something the issuing authority says does not exist. Markup we deploy is validated as part of the wider SEO program rather than billed separately.

Our site is fast in PageSpeed Insights but conversions are flat. What now?+

A lab score on one URL is not the same as field performance on the pages people actually use. Core Web Vitals are reported from real-world data, and Google's page experience guidance frames them as one part of a good experience rather than a single dial to turn. We measure on the converting templates — service pages, contact paths, mobile — because that is where speed is a revenue problem before it is a search problem. It usually pairs with conversion rate optimization rather than replacing it.

A previous agency built hundreds of near-identical pages. Are we exposed?+

Potentially. Google's spam policies define scaled content abuse as generating many pages primarily to manipulate rankings rather than to help people, and its AI guidance separately warns against creating separate content for every query variation, noting that a high quantity of pages does not make a site higher quality. Before the policy question, there is a mechanical one: Google picks one canonical per cluster, so the rest are spending crawl budget to be ignored. We consolidate rather than rewriting them one adjective at a time, then rebuild the survivors properly through content writing.

Our content is built in JavaScript. Is that a problem?+

It is a solvable problem and a frequently undiagnosed one. If content depends on JavaScript to appear, it depends on rendering to be indexed — the page looks complete in a browser and can arrive empty at the index. The check is simple and nobody runs it: compare what a crawler receives against what a browser displays, template by template. This is a standing item in any website build or rebuild we take on.

Traffic dropped last month. Where do we start?+

Not with guesses. Google publishes a structured debugging sequence for exactly this, and working it in order separates a technical fault from a seasonal change from a broad ranking movement before anyone proposes a fix. In our experience the technical branch resolves it more often than the market branch, and the specific culprit is usually a deploy that revoked crawl access or introduced a template-level directive — which is why deploy-time checks are part of the ninety-day plan rather than an upsell. Where the cause turns out to be competitive rather than technical, it moves to competitor analysis.

How much crawl budget should we actually worry about?+

For most sites, less than the conversation suggests. Google's own crawl budget guidance is aimed at large sites, and a few hundred pages rarely has a budget problem. Where it becomes real is faceted navigation and parameters generating URL space faster than anything will work through it. The honest test: if crawl stats show most requests landing on URLs you would not want indexed, there is a problem worth solving. If not, the effort belongs in content or structural work instead.

If we fix everything, are results guaranteed?+

No, and any agency saying otherwise is telling you something about itself. Google states that indexing and serving are not guaranteed even when every requirement and policy is met. What technical work does reliably is remove the reasons a page cannot be retrieved — and that is a precondition, not a promise. What we commit to is a sequence, a baseline in both indexes, and reporting that separates measured data from observed data. If you want the fuller picture before deciding, start with an A.R.C. Report or a website audit.

Find out what is currently invisible.

A technical eligibility audit: which URLs can be crawled, indexed and shown with a snippet, what is blocking the ones that can't, and how the picture differs between Google and Bing. Findings are yours whether or not we work together.

What the audit includes
  • Eligibility census across every template, with blocking directives named
  • Coverage compared across Google and Bing, not Google alone
  • Canonical and duplication map for every cluster
  • Render check — what a crawler receives versus what a browser shows
  • Field Core Web Vitals on the pages that convert
  • Findings ordered by consequence, with the evidence tier on each

Explore the wider program: all services, SEO, generative engine optimization, answer engine optimization, LLM optimization and reputation management.

Get Your Technical Eligibility Audit

We will show you exactly which pages can be retrieved, which cannot, and what is standing in the way — in both indexes.

No cost, no commitment. We will follow up by email or phone to walk you through the findings.