Icycoide/searxng
1
0
Fork 1
mirror of https://github.com/searxng/searxng.git synced 2025-07-13 16:29:17 +02:00
Code Issues Projects Releases Packages Wiki Activity
8761 commits 5 branches 0 tags 163 MiB
Commit graph

507 commits

Author SHA1 Message Date
github-actions[bot]
60be0f453e
[data] update searx.data - update_firefox_version.py (#4954)
Some checks failed
Documentation / Release (push) Has been cancelled
Integration / Python 3.10 (push) Has been cancelled
Integration / Python 3.11 (push) Has been cancelled
Integration / Python 3.12 (push) Has been cancelled
Integration / Python 3.13 (push) Has been cancelled
Integration / Python 3.9 (push) Has been cancelled
Integration / Theme (push) Has been cancelled
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
 2025-06-29 08:17:46 +02:00
github-actions[bot]
4fd3217786
[data] update searx.data - update_external_bangs.py (#4955) 
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
 2025-06-29 08:17:06 +02:00
github-actions[bot]
a8cc4ad877
[data] update searx.data - update_wikidata_units.py (#4956) 
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
 2025-06-29 08:16:07 +02:00
github-actions[bot]
c475508a1b
[data] update searx.data - update_ahmia_blacklist.py (#4957) 
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
 2025-06-29 08:15:38 +02:00
github-actions[bot]
2c0bda729a
[data] update searx.data - update_currencies.py (#4958) 
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
 2025-06-29 08:15:05 +02:00
github-actions[bot]
8de4119179
[data] update searx.data - update_engine_traits.py (#4959) 
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
 2025-06-29 08:14:42 +02:00
github-actions[bot]
6df862a4c7
[data] update searx.data - update_engine_descriptions.py (#4960) 
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
 2025-06-29 08:14:18 +02:00
Markus Heiser
2dd4f7b972 [mod] data: implement a simple tracker URL (SQL) database 
On demand, the tracker data is loaded directly into the cache, so that the
maintenance of this data via PRs is no longer necessary.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2025-06-23 22:12:18 +02:00
Bnyro
2001efbbab [data] update searx.data - add tracker_patterns.json 2025-06-23 22:12:18 +02:00
Bnyro
8f7eee2473 [feat] tracker url plugin: use ClearURL tracking param list 2025-06-23 22:12:18 +02:00
Markus Heiser
913dfbde3c [fix] plugins: currency processor bug 
Fix C&P typo from  https://github.com/searxng/searxng/pull/4836

Closes: https://github.com/searxng/searxng/issues/4861
 2025-05-29 07:37:59 +02:00
searxng-bot
7448a18efa [data] update searx.data - update_ahmia_blacklist.py 2025-05-29 07:11:55 +02:00
searxng-bot
23396f5a1d [data] update searx.data - update_firefox_version.py 2025-05-29 07:10:22 +02:00
searxng-bot
dbe6a6f4fa [data] update searx.data - update_wikidata_units.py 2025-05-29 07:08:32 +02:00
searxng-bot
b2727e1be9 [data] update searx.data - update_currencies.py 2025-05-29 07:07:55 +02:00
searxng-bot
9d8592afd7 [data] update searx.data - update_engine_traits.py 2025-05-29 07:07:05 +02:00
searxng-bot
70a387a48f [data] update searx.data - update_engine_descriptions.py 2025-05-29 07:06:38 +02:00
Markus Heiser
848c8d0544
[mod] data: implement a simple currencies (SQL) database (#4836) 
To reduce the memory footprint, this patch no longer loads the JSON data
completely into memory.  Instead, there is an SQL database based on
`ExpireCacheSQLite`.

The class CurrenciesDB is a simple DB application that encapsulates the
DB (queries and initialization) and provides convenient methods like
`name_to_iso4217` and `iso4217_to_name`.

Related:

- https://github.com/searxng/searxng/discussions/1892
- https://github.com/searxng/searxng/pull/3458#issuecomment-2900807671
- https://github.com/searxng/searxng/pull/4650

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2025-05-25 10:40:57 +02:00
Markus Heiser
d29cf64ce4
[mod] searx.data: lazy load of the data objects (databases) (#4834) 
In the previous implementation, all databases were loaded into memory when
importing the searx.data package, regardless of whether they were ever needed.

Regardless of this, it is an antipattern to load entire databases into memory
when importing a package or module; databases should be loaded when needed.

Lazy loading is a first step toward improving memory usage and also improves
performance when setting up the runtime environment.  Building on this,
subsequent PRs will be able to further optimize memory behavior, e.g., by using
a real database application such as the one already available via

    searx.cache.ExpireCache

Related:

- https://github.com/searxng/searxng/discussions/1892
- https://github.com/searxng/searxng/pull/3458
- https://github.com/searxng/searxng/pull/4650

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2025-05-22 16:23:30 +02:00
github-actions[bot]
d76f030cb3
[data] update searx.data - update_wikidata_units.py (#4738) 2025-05-09 07:09:58 +02:00
github-actions[bot]
b3b15ecc72
[data] update searx.data - update_ahmia_blacklist.py (#4739) 
Co-authored-by: inetol <inetol@users.noreply.github.com>
 2025-05-09 07:09:00 +02:00
github-actions[bot]
1319b250af
[data] update searx.data - update_currencies.py (#4740) 
Co-authored-by: inetol <inetol@users.noreply.github.com>
 2025-05-09 07:08:26 +02:00
github-actions[bot]
198928de05
[data] update searx.data - update_engine_traits.py (#4741) 
Co-authored-by: inetol <inetol@users.noreply.github.com>
 2025-05-09 07:07:33 +02:00
github-actions[bot]
11d9c830b8
[data] update searx.data - update_engine_descriptions.py (#4742) 
Co-authored-by: inetol <inetol@users.noreply.github.com>
 2025-05-09 07:06:52 +02:00
return42
41e3a0baa7 [data] update searx.data - update_engine_descriptions.py 2025-05-02 16:45:47 +02:00
return42
c45b970546 [data] update searx.data - update_currencies.py 2025-04-29 09:12:06 +02:00
Markus Heiser
a4251be397 [data] update searx.data - make data.traits 
Related:

- https://github.com/searxng/searxng/pull/4687

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2025-04-29 09:11:33 +02:00
Markus Heiser
c20038e7c3 [fix] engine yahoo: replace fetch_traits by a list of languages 
The Yahoo engine's fetch_traits function has been encountering an error in CI
jobs for several months [1], thus aborting the process for all other engines as
well.

The language selection dialog (which fetch_traits calls) requires an `EuConsent`
cookie. Strangely, the cookie is not needed for searching, which is why the
engine itself still works.

Since Yahoo won't be conquering any new marketplaces in the foreseeable future,
it should be sufficient to hard-implement the list of currently available
languages ​​(`yahoo_languages`).

[1] https://github.com/searxng/searxng/actions/runs/14720458830/job/41313149268

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2025-04-29 08:48:56 +02:00
return42
20e40ded6d [data] update searx.data - update_wikidata_units.py 2025-04-29 07:07:15 +02:00
return42
4de0766b76 [data] update searx.data - update_ahmia_blacklist.py 2025-04-29 07:06:49 +02:00
return42
6de83ca47f [data] update searx.data - update_firefox_version.py 2025-04-29 07:06:26 +02:00
return42
a32bcd54c5 [data] update searx.data - update_external_bangs.py 2025-04-29 07:06:10 +02:00
Markus Heiser
937eb907d3 [data] update searx.data - make data.traits (mullvad leta) 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2025-04-23 14:06:32 +02:00
Markus Heiser
8c8aba8cf5 [fix] engine radio browser: get servers from DNS api.radio-browser.info 
Do a DNS-lookup of 'all.api.radio-browser.info', add reverse lookup and select
randomly a URL from available servers

Closes: https://github.com/searxng/searxng/issues/4576
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2025-04-06 18:59:10 +02:00
Markus Heiser
237267ffbe [data] update searx.data - update_engine_descriptions.py 2025-03-30 09:54:03 +02:00
return42
d36da0a6c3 [data] update searx.data - update_ahmia_blacklist.py 2025-03-29 09:49:27 +01:00
return42
d7ca257fcd [data] update searx.data - update_firefox_version.py 2025-03-29 09:48:39 +01:00
return42
98024d8518 [data] update searx.data - update_wikidata_units.py 2025-03-29 09:48:06 +01:00
return42
96d0854a50 [data] update searx.data - update_currencies.py 2025-03-29 09:47:33 +01:00
Markus Heiser
5986629c6b [fix] make data.traits - partial revert of commit 30a8204 
The entries in the catalog of search languages are build up from the "Engine
Traits" [1] and which entries are included in the catalog is controlled qby two
threshold values [2].

If possible, the values should ensure that no languages or regions disappear
from the catalog of search languages [3].

The threshold values should have been adjusted in commit 30a8204:

- ``min_eng_per_region = 18``
- ``min_eng_per_lang = 22``

Because the threshold values were not adjusted, many entries were missing in the
search language catalog.  This bug has been fixed with this patch: the threshold
values have been adjusted and the catalog of search languages has been completed
again.

[1] https://docs.searxng.org/dev/engines/enginelib.html#module-searx.enginelib.traits
[2] 96a6e3dcb2/searxng_extra/update/update_engine_traits.py (L104-L105)
[3] https://github.com/searxng/searxng/blob/master/searx/sxng_locales.py

Closes: https://github.com/searxng/searxng/issues/4519
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2025-03-22 07:52:36 +01:00
Markus Heiser
1d16b94279 [fix] wikidata: increase wikidata queries timeout 
The big queries for initializing and updating the currencies take longer than
the default of the wikidata engine, which is only 3sec.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2025-03-01 12:21:29 +01:00
return42
30a820437d [data] update searx.data - update_engine_traits.py 2025-03-01 09:43:06 +01:00
return42
01e2b3e325 [data] update searx.data - update_firefox_version.py 2025-03-01 07:47:48 +01:00
return42
ccf485c519 [data] update searx.data - update_wikidata_units.py 2025-03-01 07:47:24 +01:00
return42
64810ee911 [data] update searx.data - update_ahmia_blacklist.py 2025-03-01 07:44:28 +01:00
return42
a4b120c33d [data] update searx.data - update_engine_descriptions.py 2025-03-01 07:43:29 +01:00
return42
73e4128970 [data] update searx.data - update_external_bangs.py 2025-03-01 07:43:00 +01:00
return42
738906358b [data] update searx.data - update_currencies.py 2025-01-29 06:23:04 +01:00
return42
fc8938c968 [data] update searx.data - update_ahmia_blacklist.py 2025-01-29 06:07:28 +01:00
return42
3b9e06fbd2 [data] update searx.data - update_wikidata_units.py 2025-01-29 06:06:50 +01:00