[doc] rearranges Settings & Engines docs for better readability

We have built up detailed documentation of the *settings* and the *engines* over
the past few years.  However, this documentation was still spread over various
chapters and was difficult to navigate in its entirety.

This patch rearranges the Settings & Engines documentation for better
readability.

To review new ordered docs::

   make docs.clean docs.live

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This commit is contained in:
Markus Heiser 2023-06-30 18:07:02 +02:00 committed by Markus Heiser
parent 8e8d8dabe9
commit 5720844fcd
84 changed files with 1715 additions and 1414 deletions

View file

@ -4,7 +4,7 @@
How to contribute
=================
.. contents:: Contents
.. contents::
:depth: 2
:local:
:backlinks: entry

View file

@ -0,0 +1,14 @@
.. _demo offline engine:
===================
Demo Offline Engine
===================
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.demo_offline
:members:

View file

@ -0,0 +1,14 @@
.. _demo online engine:
==================
Demo Online Engine
==================
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.demo_online
:members:

View file

@ -4,6 +4,11 @@
Engine Overview
===============
.. contents::
:depth: 3
:local:
:backlinks: entry
.. _metasearch-engine: https://en.wikipedia.org/wiki/Metasearch_engine
.. sidebar:: Further reading ..
@ -11,10 +16,6 @@ Engine Overview
- :ref:`configured engines`
- :ref:`settings engine`
.. contents::
:depth: 3
:backlinks: entry
SearXNG is a metasearch-engine_, so it uses different search engines to provide
better results.
@ -49,12 +50,12 @@ Engine File
categories list categories, in which the engine is working
paging boolean support multiple pages
time_range_support boolean support search time range
engine_type str - ``online`` :ref:`[ref] <demo online engine>` by
engine_type str - ``online`` :ref:`[ref] <online engines>` by
default, other possibles values are:
- ``offline`` :ref:`[ref] <offline engines>`
- ``online_dictionary``
- ``online_currency``
- ``online_url_search``
- ``online_dictionary`` :ref:`[ref] <online dictionary>`
- ``online_currency`` :ref:`[ref] <online currency>`
- ``online_url_search`` :ref:`[ref] <online url search>`
======================= =========== ========================================================
.. _engine settings:
@ -239,12 +240,18 @@ following parameters can be used to specify a search request:
.. _engine results:
.. _engine media types:
Media Types
===========
Result Types (``template``)
===========================
Each result item of an engine can be of different media-types. Currently the
following media-types are supported. To set another media-type as ``default``,
the parameter ``template`` must be set to the desired type.
following media-types are supported. To set another media-type as
:ref:`template default`, the parameter ``template`` must be set to the desired
type.
.. _template default:
``default``
-----------
.. table:: Parameter of the **default** media type:
:width: 100%
@ -259,6 +266,11 @@ the parameter ``template`` must be set to the desired type.
========================= =====================================================
.. _template images:
``images``
----------
.. table:: Parameter of the **images** media type:
:width: 100%
@ -277,6 +289,11 @@ the parameter ``template`` must be set to the desired type.
========================= =====================================================
.. _template videos:
``videos``
----------
.. table:: Parameter of the **videos** media type:
:width: 100%
@ -292,6 +309,12 @@ the parameter ``template`` must be set to the desired type.
thumbnail string, url to a small-preview image
========================= =====================================================
.. _template torrent:
``torrent``
-----------
.. _magnetlink: https://en.wikipedia.org/wiki/Magnet_URI_scheme
.. table:: Parameter of the **torrent** media type:
@ -315,6 +338,12 @@ the parameter ``template`` must be set to the desired type.
torrentfile string, torrentfile of the result
========================= =====================================================
.. _template map:
``map``
-------
.. table:: Parameter of the **map** media type:
:width: 100%
@ -342,6 +371,12 @@ the parameter ``template`` must be set to the desired type.
address.country country of object
========================= =====================================================
.. _template paper:
``paper``
---------
.. _BibTeX format: https://www.bibtex.com/g/bibtex-format/
.. _BibTeX field types: https://en.wikipedia.org/wiki/BibTeX#Field_types
@ -430,3 +465,4 @@ the parameter ``template`` must be set to the desired type.
* - html_url
- :py:class:`str`
- URL to full article, HTML version

View file

@ -0,0 +1,22 @@
.. _searx.enginelib:
==============
Engine Library
==============
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.enginelib
:members:
.. _searx.enginelib.traits:
Engine traits
=============
.. automodule:: searx.enginelib.traits
:members:

View file

@ -0,0 +1,9 @@
.. _searx.engines loader:
========================
SearXNG's engines loader
========================
.. automodule:: searx.engines
:members:

View file

@ -0,0 +1,97 @@
.. _engine implementations:
======================
Engine Implementations
======================
Framework Components
====================
.. toctree::
:maxdepth: 2
enginelib
engines
engine_overview
Engine Types
============
The :py:obj:`engine_type <searx.enginelib.Engine.engine_type>` of an engine
determines which :ref:`search processor <searx.search.processors>` is used by
the engine.
In this section a list of the enignes that are documented is given, a complete
list of the engines can be found in the source under: :origin:`searx/engines`.
.. _online engines:
Online Engines
--------------
.. sidebar:: info
- :py:obj:`processors.online <searx.search.processors.online>`
.. toctree::
:maxdepth: 1
:glob:
demo/demo_online
xpath
online/*
.. _offline engines:
Offline Engines
---------------
.. sidebar:: info
- :py:obj:`processors.offline <searx.search.processors.offline>`
.. toctree::
:maxdepth: 1
:glob:
offline_concept
demo/demo_offline
offline/*
.. _online url search:
Online URL Search
-----------------
.. sidebar:: info
- :py:obj:`processors.online_url_search <searx.search.processors.online_url_search>`
.. toctree::
:maxdepth: 1
:glob:
online_url_search/*
.. _online currency:
Online Currency
---------------
.. sidebar:: info
- :py:obj:`processors.online_currency <searx.search.processors.online_currency>`
*no engine of this type is documented yet / comming soon*
.. _online dictionary:
Online Dictionary
-----------------
.. sidebar:: info
- :py:obj:`processors.online_dictionary <searx.search.processors.online_dictionary>`
*no engine of this type is documented yet / comming soon*

View file

@ -0,0 +1,23 @@
.. _engine command:
====================
Command Line Engines
====================
.. sidebar:: info
- :origin:`command.py <searx/engines/command.py>`
- :ref:`offline engines`
.. contents::
:depth: 2
:local:
:backlinks: entry
.. sidebar:: info
Initial sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
.. automodule:: searx.engines.command
:members:

View file

@ -0,0 +1,97 @@
.. _nosql engines:
===============
NoSQL databases
===============
.. sidebar:: further read
- `NoSQL databases <https://en.wikipedia.org/wiki/NoSQL>`_
- `redis.io <https://redis.io/>`_
- `MongoDB <https://www.mongodb.com>`_
.. contents::
:depth: 2
:local:
:backlinks: entry
.. sidebar:: info
Initial sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
The following `NoSQL databases`_ are supported:
- :ref:`engine redis_server`
- :ref:`engine mongodb`
All of the engines above are just commented out in the :origin:`settings.yml
<searx/settings.yml>`, as you have to set various options and install
dependencies before using them.
By default, the engines use the ``key-value`` template for displaying results /
see :origin:`simple <searx/templates/simple/result_templates/key-value.html>`
theme. If you are not satisfied with the original result layout, you can use
your own template, set ``result_template`` attribute to ``{template_name}`` and
place the templates at::
searx/templates/{theme_name}/result_templates/{template_name}
Furthermore, if you do not wish to expose these engines on a public instance, you
can still add them and limit the access by setting ``tokens`` as described in
section :ref:`private engines`.
Extra Dependencies
==================
For using :ref:`engine redis_server` or :ref:`engine mongodb` you need to
install additional packages in Python's Virtual Environment of your SearXNG
instance. To switch into the environment (:ref:`searxng-src`) you can use
:ref:`searxng.sh`::
$ sudo utils/searxng.sh instance cmd bash
(searxng-pyenv)$ pip install ...
Configure the engines
=====================
`NoSQL databases`_ are used for storing arbitrary data without first defining
their structure.
.. _engine redis_server:
Redis Server
------------
.. _redis: https://github.com/andymccurdy/redis-py#installation
.. sidebar:: info
- ``pip install`` redis_
- redis.io_
- :origin:`redis_server.py <searx/engines/redis_server.py>`
.. automodule:: searx.engines.redis_server
:members:
.. _engine mongodb:
MongoDB
-------
.. _pymongo: https://github.com/mongodb/mongo-python-driver#installation
.. sidebar:: info
- ``pip install`` pymongo_
- MongoDB_
- :origin:`mongodb.py <searx/engines/mongodb.py>`
.. automodule:: searx.engines.mongodb
:members:

View file

@ -0,0 +1,62 @@
=================
Local Search APIs
=================
.. sidebar:: further read
- `Comparison to alternatives
<https://docs.meilisearch.com/learn/what_is_meilisearch/comparison_to_alternatives.html>`_
.. contents::
:depth: 1
:local:
:backlinks: entry
.. sidebar:: info
Initial sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
Administrators might find themselves wanting to integrate locally running search
engines. The following ones are supported for now:
* `Elasticsearch`_
* `Meilisearch`_
* `Solr`_
Each search engine is powerful, capable of full-text search. All of the engines
above are added to ``settings.yml`` just commented out, as you have to
``base_url`` for all them.
Please note that if you are not using HTTPS to access these engines, you have to
enable HTTP requests by setting ``enable_http`` to ``True``.
Furthermore, if you do not want to expose these engines on a public instance,
you can still add them and limit the access by setting ``tokens`` as described
in section :ref:`private engines`.
.. _engine meilisearch:
MeiliSearch
===========
.. automodule:: searx.engines.meilisearch
:members:
.. _engine elasticsearch:
Elasticsearch
=============
.. automodule:: searx.engines.elasticsearch
:members:
.. _engine solr:
Solr
====
.. automodule:: searx.engines.solr
:members:

View file

@ -0,0 +1,121 @@
.. _sql engines:
===========
SQL Engines
===========
.. sidebar:: further read
- `SQLite <https://www.sqlite.org/index.html>`_
- `PostgreSQL <https://www.postgresql.org>`_
- `MySQL <https://www.mysql.com>`_
.. contents::
:depth: 2
:local:
:backlinks: entry
.. sidebar:: info
Initial sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
With the *SQL engines* you can bind SQL databases into SearXNG. The following
Relational Database Management System (RDBMS) are supported:
- :ref:`engine sqlite`
- :ref:`engine postgresql`
- :ref:`engine mysql_server`
All of the engines above are just commented out in the :origin:`settings.yml
<searx/settings.yml>`, as you have to set the required attributes for the
engines, e.g. ``database:`` ...
.. code:: yaml
- name: ...
engine: {sqlite|postgresql|mysql_server}
database: ...
result_template: {template_name}
query_str: ...
By default, the engines use the ``key-value`` template for displaying results /
see :origin:`simple <searx/templates/simple/result_templates/key-value.html>`
theme. If you are not satisfied with the original result layout, you can use
your own template, set ``result_template`` attribute to ``{template_name}`` and
place the templates at::
searx/templates/{theme_name}/result_templates/{template_name}
If you do not wish to expose these engines on a public instance, you can still
add them and limit the access by setting ``tokens`` as described in section
:ref:`private engines`.
Extra Dependencies
==================
For using :ref:`engine postgresql` or :ref:`engine mysql_server` you need to
install additional packages in Python's Virtual Environment of your SearXNG
instance. To switch into the environment (:ref:`searxng-src`) you can use
:ref:`searxng.sh`::
$ sudo utils/searxng.sh instance cmd bash
(searxng-pyenv)$ pip install ...
Configure the engines
=====================
The configuration of the new database engines are similar. You must put a valid
SQL-SELECT query in ``query_str``. At the moment you can only bind at most one
parameter in your query. By setting the attribute ``limit`` you can define how
many results you want from the SQL server. Basically, it is the same as the
``LIMIT`` keyword in SQL.
Please, do not include ``LIMIT`` or ``OFFSET`` in your SQL query as the engines
rely on these keywords during paging. If you want to configure the number of
returned results use the option ``limit``.
.. _engine sqlite:
SQLite
------
.. sidebar:: info
- :origin:`sqlite.py <searx/engines/sqlite.py>`
.. automodule:: searx.engines.sqlite
:members:
.. _engine postgresql:
PostgreSQL
----------
.. _psycopg2: https://www.psycopg.org/install
.. sidebar:: info
- :origin:`postgresql.py <searx/engines/postgresql.py>`
- ``pip install`` `psycopg2-binary <psycopg2>`_
.. automodule:: searx.engines.postgresql
:members:
.. _engine mysql_server:
MySQL
-----
.. sidebar:: info
- :origin:`mysql_server.py <searx/engines/mysql_server.py>`
- ``pip install`` :pypi:`mysql-connector-python <mysql-connector-python>`
.. automodule:: searx.engines.mysql_server
:members:

View file

@ -1,15 +1,14 @@
.. _offline engines:
===============
Offline Engines
Offline Concept
===============
.. sidebar:: offline engines
- :ref:`demo offline engine`
- :ref:`sql engines`
- :ref:`engine command`
- :origin:`Redis <searx/engines/redis_server.py>`
- :ref:`sql engines`
- :ref:`nosql engines`
- :py:obj:`searx.search.processors.offline`
To extend the functionality of SearXNG, offline engines are going to be
introduced. An offline engine is an engine which does not need Internet
@ -31,7 +30,6 @@ Programming Interface
in advance.
:py:func:`search(query, params) <searx.engines.demo_offline.searc>`
Each offline engine has a function named ``search``. This function is
responsible to perform a search and return the results in a presentable
format. (Where *presentable* means presentable by the selected result
@ -69,10 +67,3 @@ administrators can set token(s) for each of the :ref:`private engines`. If a
query contains a valid token, then SearXNG performs the requested private
search. If not, requests from an offline engines return errors.
Acknowledgement
===============
This development was sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_ .

View file

@ -0,0 +1,13 @@
.. _annas_archive engine:
==============
Anna's Archive
==============
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.annas_archive
:members:

View file

@ -0,0 +1,14 @@
.. _archlinux engine:
==========
Arch Linux
==========
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.archlinux
:members:

View file

@ -0,0 +1,43 @@
.. _bing engines:
============
Bing Engines
============
.. contents::
:depth: 2
:local:
:backlinks: entry
.. _bing web engine:
Bing WEB
========
.. automodule:: searx.engines.bing
:members:
.. _bing images engine:
Bing Images
===========
.. automodule:: searx.engines.bing_images
:members:
.. _bing videos engine:
Bing Videos
===========
.. automodule:: searx.engines.bing_videos
:members:
.. _bing news engine:
Bing News
=========
.. automodule:: searx.engines.bing_news
:members:

View file

@ -0,0 +1,13 @@
.. _dailymotion engine:
===========
Dailymotion
===========
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.dailymotion
:members:

View file

@ -0,0 +1,22 @@
.. _duckduckgo engines:
=================
DukcDukGo Engines
=================
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.duckduckgo
:members:
.. automodule:: searx.engines.duckduckgo_images
:members:
.. automodule:: searx.engines.duckduckgo_definitions
:members:
.. automodule:: searx.engines.duckduckgo_weather
:members:

View file

@ -0,0 +1,76 @@
.. _google engines:
==============
Google Engines
==============
.. contents::
:depth: 2
:local:
:backlinks: entry
.. _google API:
Google API
==========
.. _Query Parameter Definitions:
https://developers.google.com/custom-search/docs/xml_results#WebSearch_Query_Parameter_Definitions
SearXNG's implementation of the Google API is mainly done in
:py:obj:`get_google_info <searx.engines.google.get_google_info>`.
For detailed description of the *REST-full* API see: `Query Parameter
Definitions`_. The linked API documentation can sometimes be helpful during
reverse engineering. However, we cannot use it in the freely accessible WEB
services; not all parameters can be applied and some engines are more *special*
than other (e.g. :ref:`google news engine`).
.. _google web engine:
Google WEB
==========
.. automodule:: searx.engines.google
:members:
.. _google autocomplete:
Google Autocomplete
====================
.. autofunction:: searx.autocomplete.google_complete
.. _google images engine:
Google Images
=============
.. automodule:: searx.engines.google_images
:members:
.. _google videos engine:
Google Videos
=============
.. automodule:: searx.engines.google_videos
:members:
.. _google news engine:
Google News
===========
.. automodule:: searx.engines.google_news
:members:
.. _google scholar engine:
Google Scholar
==============
.. automodule:: searx.engines.google_scholar
:members:

View file

@ -0,0 +1,27 @@
.. _peertube engines:
================
Peertube Engines
================
.. contents::
:depth: 2
:local:
:backlinks: entry
.. _peertube video engine:
Peertube Video
==============
.. automodule:: searx.engines.peertube
:members:
.. _sepiasearch engine:
SepiaSearch
===========
.. automodule:: searx.engines.sepiasearch
:members:

View file

@ -0,0 +1,13 @@
.. _engine recoll:
=============
Recoll Engine
=============
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.recoll
:members:

View file

@ -0,0 +1,13 @@
.. _startpage engines:
=================
Startpage Engines
=================
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.startpage
:members:

View file

@ -0,0 +1,13 @@
.. _torznab engine:
==============
Torznab WebAPI
==============
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.torznab
:members:

View file

@ -0,0 +1,27 @@
.. _wikimedia engines:
=========
Wikimedia
=========
.. contents::
:depth: 2
:local:
:backlinks: entry
.. _wikipedia engine:
Wikipedia
=========
.. automodule:: searx.engines.wikipedia
:members:
.. _wikidata engine:
Wikidata
=========
.. automodule:: searx.engines.wikidata
:members:

View file

@ -0,0 +1,13 @@
.. _yahoo engine:
============
Yahoo Engine
============
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.yahoo
:members:

View file

@ -0,0 +1,14 @@
.. _tineye engine:
======
Tineye
======
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.tineye
:members:

View file

@ -0,0 +1,14 @@
.. _xpath engine:
============
XPath Engine
============
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.xpath
:members:

View file

@ -4,12 +4,10 @@ Developer documentation
.. toctree::
:maxdepth: 2
:caption: Contents
quickstart
contribution_guide
engine_overview
offline_engines
engines/index
search_api
plugins
translation

View file

@ -22,7 +22,7 @@ In this article we will show, how you can make use of Linux Containers (LXC_) in
section :ref:`internet connectivity docker`.
.. contents:: Contents
.. contents::
:depth: 2
:local:
:backlinks: entry

View file

@ -22,7 +22,7 @@ Calling the ``help`` target gives a first overview (``make help``):
.. program-output:: bash -c "cd ..; make --no-print-directory help"
.. contents:: Contents
.. contents::
:depth: 2
:local:
:backlinks: entry
@ -243,14 +243,14 @@ calling ``make clean`` stop all processes using the :ref:`make install` or
We describe the usage of the ``doc.*`` targets in the :ref:`How to contribute /
Documentation <contrib docs>` section. If you want to edit the documentation
read our :ref:`make docs.live` section. If you are working in your own brand,
adjust your :ref:`settings global`.
adjust your :ref:`settings brand`.
.. _make docs.gh-pages:
``make docs.gh-pages``
======================
To deploy on github.io first adjust your :ref:`settings global`. For any
To deploy on github.io first adjust your :ref:`settings brand`. For any
further read :ref:`deploy on github.io`.
.. _make test:

View file

@ -37,7 +37,7 @@ docs.live <make docs.live>` to build HTML while editing.
- DOT_, `Graphviz's dot`_, Graphviz_
.. contents:: Contents
.. contents::
:depth: 3
:local:
:backlinks: entry

View file

@ -9,7 +9,6 @@ developers.
.. toctree::
:maxdepth: 2
:caption: Contents
update
standalone_searx.py