[fix] ddg-lite & ddg-extra: don't send empty vqd value

DDG's bot detection is sensitive to the vqd value.  For some search terms (such
as extremely long search terms that are often sent by bots), no vqd value can be
determined.

If SearXNG cannot determine a vqd value, then no request should go out to
DDG (WEB): a request with a wrong vqd value leads to DDG temporarily putting
SearXNG's IP on a block list.

Requests from IPs in this block list run into timeouts.

Not sure, but it seems the block list is a sliding window: to get my IP rid from
the bot list I had to cool down my IP for 1h (send no requests from that IP to
DDG).

Since such issues can't reproduce in a local instance I tested this patch 24h on
my public SearXNG instance: There are still errors (rare), but the reliability
is still 100%.

Related:

- https://github.com/searxng/searxng/pull/2922
- https://github.com/searxng/searxng/pull/2923

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This commit is contained in:
Markus Heiser 2023-10-10 11:08:14 +02:00 committed by Markus Heiser
parent 24561370b7
commit 14323d683f
2 changed files with 53 additions and 19 deletions

View file

@ -48,6 +48,13 @@ search_path_map = {'images': 'i', 'videos': 'v', 'news': 'news'}
def request(query, params):
# request needs a vqd argument
vqd = get_vqd(query)
if not vqd:
# some search terms do not have results and therefore no vqd value
params['url'] = None
return params
eng_region = traits.get_region(params['searxng_locale'], traits.all_locale)
eng_lang = get_ddg_lang(traits, params['searxng_locale'])
@ -57,7 +64,7 @@ def request(query, params):
# 'u': 'bing',
'l': eng_region,
'f': ',,,,,',
'vqd': get_vqd(query),
'vqd': vqd,
}
if params['pageno'] > 1: