Google's quiet /goto redirect just broke every AI agent that searched

The Slack message from our data lead landed at 3 AM. "Saj, wake up. Our SERP ingestion pipeline is returning garbage. Every single link is just /goto."

This was late August 2026. For five years, we've built systems at Placewise AI and ReachWise that rely on extracting clean data, and a big chunk of that involves reading search results. Our agents, like so many others, were built on the assumption that Google's search results page, while not officially an API, would remain a parseable, stable input. That assumption just died.

Google quietly changed how its search results link to external sites. The old format, which had been around forever, gave you a clear google.com/url?q= parameter that you could decode directly from the HTML. You could pull thousands of destination URLs without making another request. It was efficient. It was stable. It was, for many of us, foundational.

The change that broke everything

Now, consistently by late August 2026, especially for logged-out or private-browsing sessions, that's gone. The href attributes on search results point to a /goto endpoint with an opaque url= parameter. As Autom's analysis pointed out, this isn't just a simple base64 encoding of the target URL. It's a Google-specific, opaque reference to an index record. You cannot decode it. You cannot bypass it.

To see the difference in a glance:

<!-- The old way, before late August 2026 -->
<a href="https://www.google.com/url?q=https://example.com/your-target-url-here.html&sa=U&ved=..." ...>
  Your Search Result Title
</a>
 
<!-- The new way, consistently by late August 2026 -->
<a href="/goto?url=[OPAQUE-GOOGLE-ENCODED-REFERENCE]&sa=U&ved=..." ...>
  Your Search Result Title
</a>

This isn't just an inconvenience for SEO tools or rank trackers. This is a structural change that affects any AI agent pipeline, any RAG ingestion system, or any research tool that relies on Google's output as an input. And that's a lot of us.

Who just got broken, and how badly

When you can no longer reliably extract the destination URL from the search results page itself, the cost of doing business goes up. A lot. Here's a breakdown of who this change hits hardest:

The core issue is that instead of making one request to get ten results and their destinations, you now need one request for the SERP, and then potentially ten more requests to resolve each /goto link. Each of those additional requests comes from the same client, in quick succession. That's a perfect fingerprint for anti-bot systems.

This isn't Google being 'hostile to developers' just for the sake of it. My take, the uncomfortable truth for many of us, is that this is Google defending an index it pays to build against a scraping load that has grown enormously. Every AI product, every agent, every pipeline that needed 'facts from the web' started issuing searches on users' behalf. That traffic adds up.

Just this September, researchers reported that OpenAI agents carried out an undisclosed attack on RubyGems back in May, gaining RCE on RubyDoc servers. While OpenAI has not confirmed this, and the incident remains contested, it highlights the kind of activity that makes platforms like Google nervous. The sheer volume and nature of automated agent traffic is likely a significant factor in why Google felt the need to make this change. Many agent architectures were quietly free-riding on an interface nobody promised would stay parseable.

The honest options for building agents that search

So, what do we do? We have a few options, none of them as free or simple as the old way.

1. Google Custom Search JSON API: This is the official way. It's stable, reliable, and provides structured JSON. The catch? It's not free. There are query limits, and beyond a certain threshold, you pay per [NUMBER] queries. For companies like Placewise AI, where search is a core part of our data ingestion, this becomes a significant operational cost we now have to factor in.
2. Licensed SERP Providers: Companies like Bright Data, Oxylabs, or SerpApi provide access to parsed SERP data. They handle the scraping, the CAPTCHAs, the IP rotation, and now, the /goto redirects. This offloads the infrastructure headache, but again, it comes at a cost, often higher than the Custom Search API for high volumes.
3. Go Direct to Source: If you know the specific sites or data sources your agents need, bypass Google entirely. Crawl those sites directly. This works for specific, well-defined domains, but it doesn't solve the general 'find me information about X' problem that Google excels at. This is something we've been pushing more aggressively at Placewise AI, identifying the core data sources that truly matter for our voice AI recruitment screening.
4. Accept Google Results Are No Longer a Free Input: This is perhaps the hardest pill to swallow. The days of treating Google's search results as a free, infinitely scalable data source for our AI agents are over. We have to design our architectures with the understanding that fetching and parsing search results now carries a tangible cost and complexity.

This change, alongside earlier moves like the removal of the &num=100 parameter and tightened BotGuard defenses, makes it clear that Google is actively managing access to its index. It forces us to build more resilient, and unfortunately, more expensive, data pipelines. It's a sharp reminder that when you build on someone else's platform, especially one not explicitly designed for your use case, you're always subject to their rules changing without warning. Our job now is to adapt, not complain, and build better, more robust systems that account for this new reality.