Tech Brief: Fixing a Silent Web-Search Failure

Introduction

Web search had been returning an empty result set for every query while reporting success. The failure looked like a broken tool and turned out to be five faults stacked on one root cause: the machine reaches the internet through a mobile-carrier address behind carrier-grade NAT, and the major search engines classify that address as automated traffic. No amount of engine configuration could fix it, because the engines were not misconfigured – they were refusing to serve the host. The repair had two parts: point the local search instance at the engines that still answered and make it honest about the ones that answered wrongly, then put an authenticated API in front so the address stops mattering. A small health check now proves that results match the query instead of trusting a status code.

The Symptom

The first visible failure was a query that came back with nothing:

web_search("stable diffusion lora training best practices")
-> {"success": true, "data": {"web": []}}

success: true with zero results is the worst kind of failure, because the tool reports that it worked. Two earlier queries in the same session had returned empty as well, and both had been dismissed as unusual queries rather than as a broken backend.

Table of Faults

Table.1.Faults – what was wrong, in the order it was found

Fault One: A Note That Named a Cause

A stored note said the search container breaks when it stops: “connection refused three times means the daemon is down”. That was a real past failure and the wrong diagnosis for this one. The container was running, the HTTP endpoint answered 200, and the JSON API responded in under a second.

The note had recorded a cause instead of a symptom, so it sent me to the wrong layer. It now records the signature – empty results – and the two candidate causes: a stopped daemon, or engine bans upstream.

Fault Two: A Whitelist of Refusals

The self-hosted instance listed seven engines: brave, duckduckgo, google, wikipedia, github, arxiv and stackoverflow. Probing each one individually against the JSON API, rather than asking the aggregate endpoint and hoping, produced Table 2.

Table.2.Engines – per-engine probe, first pass

The whitelist contained almost only engines that were refusing to serve, and the two that worked well were not on it. That alone explains the empty result sets.

Fault Three: Enable Is Not Keep-Only

The obvious repair – rewriting the engine list to the working set – did nothing. After a restart, the blocked engines were still being queried and still failing.

The cause is a configuration semantic that is easy to miss. With SearXNG’s use_default_settings: true, a top-level engines: list adds or enables entries; it never removes the built-in defaults.[1] Disabling requires the nested form:

use_default_settings:
  engines:
    keep_only:
      - duckduckgo
      - arxiv
      - github
      - wikipedia
      - stackexchange

After that change the unresponsive engines stopped being queried at all, which cut query latency and removed five sources of noise from every result set. The full option set is documented in the engine settings reference.[2]

Fault Four: The Engine That Answered the Wrong Question

With the dead engines gone, one general engine became the default, and it was worse than broken – it was silently degraded. Asked for stable diffusion lora training, it returned:

stableprogram.org -- "The S.T.A.B.L.E. Program"
merriam-webster.com/dictionary/stable
cambridge.org/dictionary/english/stable
en.wikipedia.org/wiki/Stable

It had answered for the word “stable” and discarded the rest of the query. Every result looks like a plausible web result; nothing in the response signals that the question was narrowed. This is the failure mode that motivated the health check described below: results that do not mention the query are now reported as degraded, not healthy.

Querying the engine directly with curl, in both the plus and percent-encoded forms, reproduced the same reduced results. So the fault was upstream, not in the search aggregator: the engine was serving a reduced query to a suspected bot. A bug in the aggregator would have been fixable in software. Not repairable, and not worth relying on.

At that point the only healthy engines were domain-specific, so a general web query returned nothing at all:

engine         status         n   secs  note
duckduckgo     UNRESPONSIVE   0   0.24  suspended upstream
wikipedia      UNRESPONSIVE   0   0.01  suspended upstream
arxiv          OK            10   0.85  4/5 query terms present
github         EMPTY          0   0.37  no results

That is the point where the free route was exhausted, and the useful question stopped being “which engine” and became “why is every engine refusing this host”.

Fault Five: The Host’s Identity

Three candidates were on the table: a stale VPN, a broken container network, or an upstream ban. The evidence pointed at the third.

• A VPN client was installed but was not carrying traffic; the default route was the ordinary local interface, not a tunnel.
• The egress address belonged to a mobile carrier, behind carrier-grade NAT, where thousands of subscribers share one public address.
• Every failure had the same shape: refusal or degradation from the major engines, silence from the rest.

That combination explains why the failures were simultaneous and total, and why engine-level tuning was treating a symptom that lives one layer below the engines. Carrier-grade NAT addresses are exactly what bot protection is designed to fence off, and a personal search instance scraping from one looks like a bot because, at the network layer, it is indistinguishable from one.

The Fix

Two levers, in order: make the local instance honest, then stop depending on anonymously scraped search.

Table.3.Fix – what changed and what it bought

The authenticated API was chosen for one reason: the key is the identity, so the network address stops mattering.[3] It also fixed a second latent fault. The extraction backend had been pointed at a different provider with no key configured anywhere on the machine, and had been silently falling back to something else for weeks.

Verification

Claims about a repaired search tool are worth very little without evidence, so verification ran from the credential outward.

Table.4.Checks – verification ladder

The before-and-after difference was not subtle. Queries that returned an empty list returned citations; page extraction that had been silently falling back returned clean text.

The Health Check

The check exists because every fault above shared one property: the system said it was fine. It is deliberately small and has one opinion – a backend is healthy only if its results mention the query.

• It reads the configured backend from the agent’s config rather than being told which one to test.
• For an API backend it pulls the key from the environment or the system keychain and reports credits consumed, so cost stays visible.
• For the self-hosted backend it keeps the per-engine sweep, so the free fallback stays measurable instead of becoming a second silent failure.
• It exits non-zero unless the backend answered the whole query, which makes it usable in a cron job or as a pre-research gate without parsing output.

What Generalizes

Four lessons survived the incident better than the fix itself.

Test the layer below the symptom. Four rounds of engine tuning were wasted because the refusal happened at the network identity layer, not in configuration. Two commands – print the default route, look up the egress address – would have collapsed three of the five faults into one.

A config key that reads as declarative may not be. “Here are my engines” can mean “also add these engines”, and the difference is invisible until something counts which engines actually ran.

A source that answers the wrong question is more dangerous than one that fails. A silently reduced query would have quietly corrupted research, and no status code would have caught it. Any automated retrieval needs a relevance assertion.

An empty result set deserves the same suspicion as an error. Tools that report success alongside zero results are common, and the zero is the signal.

There is a fifth lesson about documentation: a note that records a cause outlives its truth. Notes that record symptoms, plus the test that discriminates between causes, stay useful.

Reference

• Per-engine probing, one engine at a time:

curl -s "http://127.0.0.1:8888/search?q=QUERY&format=json&engines=ENGINE"

• Engine configuration semantics and the full engine option list: the SearXNG settings and engine-settings pages (notes 1 and 2).
• Authenticated search and extraction: the provider’s introduction page (note 3).
• Health check usage: search-health (probes the configured backend), search-health --backend searxng (per-engine sweep), search-health --json (machine-readable).

Notes