Icycoide/searxng
1
0
Fork 1
mirror of https://github.com/searxng/searxng.git synced 2025-08-28 22:36:54 +02:00
Code Issues Projects Releases Packages Wiki Activity

[refactor] typification of SearXNG (initial) / result items (part 1)

Browse source 
Typification of SearXNG
=======================

This patch introduces the typing of the results.  The why and how is described
in the documentation, please generate the documentation ..

    $ make docs.clean docs.live

and read the following articles in the "Developer documentation":

- result types --> http://0.0.0.0:8000/dev/result_types/index.html

The result types are available from the `searx.result_types` module.  The
following have been implemented so far:

- base result type: `searx.result_type.Result`
  --> http://0.0.0.0:8000/dev/result_types/base_result.html

- answer results
  --> http://0.0.0.0:8000/dev/result_types/answer.html

including the type for translations (inspired by #3925).  For all other
types (which still need to be set up in subsequent PRs), template documentation
has been created for the transition period.

Doc of the fields used in Templates
===================================

The template documentation is the basis for the typing and is the first complete
documentation of the results (needed for engine development).  It is the
"working paper" (the plan) with which further typifications can be implemented
in subsequent PRs.

- https://github.com/searxng/searxng/issues/357

Answer Templates
================

With the new (sub) types for `Answer`, the templates for the answers have also
been revised, `Translation` are now displayed with collapsible entries (inspired
by #3925).

    !en-de dog

Plugins & Answerer
==================

The implementation for `Plugin` and `Answer` has been revised, see
documentation:

- Plugin: http://0.0.0.0:8000/dev/plugins/index.html
- Answerer: http://0.0.0.0:8000/dev/answerers/index.html

With `AnswerStorage` and `AnswerStorage` to manage those items (in follow up
PRs, `ArticleStorage`, `InfoStorage` and .. will be implemented)

Autocomplete
============

The autocompletion had a bug where the results from `Answer` had not been shown
in the past.  To test activate autocompletion and try search terms for which we
have answerers

- statistics: type `min 1 2 3` .. in the completion list you should find an
  entry like `[de] min(1, 2, 3) = 1`

- random: type `random uuid` .. in the completion list, the first item is a
  random UUID

Extended Types
==============

SearXNG extends e.g. the request and response types of flask and httpx, a module
has been set up for type extensions:

- Extended Types
  --> http://0.0.0.0:8000/dev/extended_types.html

Unit-Tests
==========

The unit tests have been completely revised.  In the previous implementation,
the runtime (the global variables such as `searx.settings`) was not initialized
before each test, so the runtime environment with which a test ran was always
determined by the tests that ran before it.  This was also the reason why we
sometimes had to observe non-deterministic errors in the tests in the past:

- https://github.com/searxng/searxng/issues/2988 is one example for the Runtime
  issues, with non-deterministic behavior ..

- https://github.com/searxng/searxng/pull/3650
- https://github.com/searxng/searxng/pull/3654
- https://github.com/searxng/searxng/pull/3642#issuecomment-2226884469
- https://github.com/searxng/searxng/pull/3746#issuecomment-2300965005

Why msgspec.Struct
==================

We have already discussed typing based on e.g. `TypeDict` or `dataclass` in the past:

- https://github.com/searxng/searxng/pull/1562/files
- https://gist.github.com/dalf/972eb05e7a9bee161487132a7de244d2
- https://github.com/searxng/searxng/pull/1412/files
- https://github.com/searxng/searxng/pull/1356

In my opinion, TypeDict is unsuitable because the objects are still dictionaries
and not instances of classes / the `dataclass` are classes but ...

The `msgspec.Struct` combine the advantages of typing, runtime behaviour and
also offer the option of (fast) serializing (incl. type check) the objects.

Currently not possible but conceivable with `msgspec`: Outsourcing the engines
into separate processes, what possibilities this opens up in the future is left
to the imagination!

Internally, we have already defined that it is desirable to decouple the
development of the engines from the development of the SearXNG core / The
serialization of the `Result` objects is a prerequisite for this.

HINT: The threads listed above were the template for this PR, even though the
implementation here is based on msgspec.  They should also be an inspiration for
the following PRs of typification, as the models and implementations can provide
a good direction.

Why just one commit?
====================

I tried to create several (thematically separated) commits, but gave up at some
point ... there are too many things to tackle at once / The comprehensibility of
the commits would not be improved by a thematic separation. On the contrary, we
would have to make multiple changes at the same places and the goal of a change
would be vaguely recognizable in the fog of the commits.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This commit is contained in:
Markus Heiser 2024-12-15 09:59:50 +01:00 • committed by Markus Heiser
parent 9079d0cac0
commit edfbf1e118
143 changed files with 3877 additions and 2118 deletions
Download patch file Download diff file Expand all files Collapse all files

19
docs/admin/plugins.rst
View file

@ -1,12 +1,14 @@
.. _plugins generic:
.. _plugins admin:
===============
Plugins builtin
List of plugins
===============
.. sidebar:: Further reading ..
- :ref:`SearXNG settings <settings plugins>`
- :ref:`dev plugin`
- :ref:`builtin plugins`
Configuration defaults (at built time):
@ -25,15 +27,10 @@ Configuration defaults (at built time):
- DO
- Description
JS & CSS dependencies
{% for plg in plugins %}
{% for plgin in plugins %}
* - {{plgin.name}}
- {{(plgin.default_on and "y") or ""}}
- {{plgin.description}}
{% for dep in (plgin.js_dependencies + plgin.css_dependencies) %}
| ``{{dep}}`` {% endfor %}
* - {{plg.info.name}}
- {{(plg.default_on and "y") or ""}}
- {{plg.info.description}}
{% endfor %}

2
docs/admin/settings/index.rst
View file

@ -22,6 +22,6 @@ Settings
settings_redis
settings_outgoing
settings_categories_as_tabs
settings_plugins

67
docs/admin/settings/settings_plugins.rst Normal file
View file

@ -0,0 +1,67 @@
.. _settings plugins:
=======
Plugins
=======
.. sidebar:: Further reading ..
- :ref:`plugins admin`
- :ref:`dev plugin`
- :ref:`builtin plugins`
The built-in plugins can be activated or deactivated via the settings
(:ref:`settings enabled_plugins`) and external plugins can be integrated into
SearXNG (:ref:`settings external_plugins`).
.. _settings enabled_plugins:
``enabled_plugins:`` (internal)
===============================
In :ref:`plugins admin` you find a complete list of all plugins, the default
configuration looks like:
.. code:: yaml
enabled_plugins:
- 'Basic Calculator'
- 'Hash plugin'
- 'Self Information'
- 'Tracker URL remover'
- 'Unit converter plugin'
- 'Ahmia blacklist'
.. _settings external_plugins:
``plugins:`` (external)
=======================
SearXNG supports *external plugins* / there is no need to install one, SearXNG
runs out of the box. But to demonstrate; in the example below we install the
SearXNG plugins from *The Green Web Foundation* `[ref]
<https://www.thegreenwebfoundation.org/news/searching-the-green-web-with-searx/>`__:
.. code:: bash
$ sudo utils/searxng.sh instance cmd bash -c
(searxng-pyenv)$ pip install git+https://github.com/return42/tgwf-searx-plugins
In the :ref:`settings.yml` activate the ``plugins:`` section and add module
``only_show_green_results`` from ``tgwf-searx-plugins``.
.. code:: yaml
plugins:
- only_show_green_results
# - mypackage.mymodule.MyPlugin
# - mypackage.mymodule.MyOtherPlugin
.. hint::
``only_show_green_results`` is an old plugin that was still implemented in
the old style. There is a legacy treatment for backward compatibility, but
new plugins should be implemented as a :py:obj:`searx.plugins.Plugin` class.

5
docs/conf.py
View file

@ -54,7 +54,7 @@ searx.engines.load_engines(searx.settings['engines'])
jinja_contexts = {
'searx': {
'engines': searx.engines.engines,
'plugins': searx.plugins.plugins,
'plugins': searx.plugins.STORAGE,
'version': {
'node': os.getenv('NODE_MINIMUM_VERSION')
},
@ -129,8 +129,9 @@ extensions = [
'notfound.extension', # https://github.com/readthedocs/sphinx-notfound-page
]
# autodoc_typehints = "description"
autodoc_default_options = {
'member-order': 'groupwise',
'member-order': 'bysource',
}
myst_enable_extensions = [

11
docs/dev/answerers/builtins.rst Normal file
View file

@ -0,0 +1,11 @@
.. _builtin answerers:
==================
Built-in Answerers
==================
.. toctree::
:maxdepth: 1
random
statistics

7
docs/dev/answerers/development.rst Normal file
View file

@ -0,0 +1,7 @@
.. _dev answerers:
====================
Answerer Development
====================
.. automodule:: searx.answerers

9
docs/dev/answerers/index.rst Normal file
View file

@ -0,0 +1,9 @@
=========
Answerers
=========
.. toctree::
:maxdepth: 2
development
builtins

8
docs/dev/answerers/random.rst Normal file
View file

@ -0,0 +1,8 @@
.. _answerer.random:
======
Random
======
.. autoclass:: searx.answerers.random.SXNGAnswerer
:members:

8
docs/dev/answerers/statistics.rst Normal file
View file

@ -0,0 +1,8 @@
.. _answerer.statistics:
==========
Statistics
==========
.. autoclass:: searx.answerers.statistics.SXNGAnswerer
:members:

335
docs/dev/engines/engine_overview.rst
View file

@ -237,335 +237,18 @@ following parameters can be used to specify a search request:
=================== =========== ==========================================================================
.. _engine results:
.. _engine media types:
Making a Response
=================
Result Types (``template``)
===========================
In the ``response`` function of the engine, the HTTP response (``resp``) is
parsed and a list of results is returned.
Each result item of an engine can be of different media-types. Currently the
following media-types are supported. To set another media-type as
:ref:`template default`, the parameter ``template`` must be set to the desired
type.
A engine can append result-items of different media-types and different
result-types to the result list. The list of the result items is render to HTML
by templates. For more details read section:
.. _template default:
- :ref:`simple theme templates`
- :ref:`result types`
``default``
-----------
.. table:: Parameter of the **default** media type:
:width: 100%
========================= =====================================================
result-parameter information
========================= =====================================================
url string, url of the result
title string, title of the result
content string, general result-text
publishedDate :py:class:`datetime.datetime`, time of publish
========================= =====================================================
.. _template images:
``images``
----------
.. list-table:: Parameter of the **images** media type
:header-rows: 2
:width: 100%
* - result-parameter
- Python type
- information
* - template
- :py:class:`str`
- is set to ``images.html``
* - url
- :py:class:`str`
- url to the result site
* - title
- :py:class:`str`
- title of the result
* - content
- :py:class:`str`
- description of the image
* - publishedDate
- :py:class:`datetime <datetime.datetime>`
- time of publish
* - img_src
- :py:class:`str`
- url to the result image
* - thumbnail_src
- :py:class:`str`
- url to a small-preview image
* - resolution
- :py:class:`str`
- the resolution of the image (e.g. ``1920 x 1080`` pixel)
* - img_format
- :py:class:`str`
- the format of the image (e.g. ``png``)
* - filesize
- :py:class:`str`
- size of bytes in :py:obj:`human readable <searx.humanize_bytes>` notation
(e.g. ``MB`` for 1024 \* 1024 Bytes filesize).
.. _template videos:
``videos``
----------
.. table:: Parameter of the **videos** media type:
:width: 100%
========================= =====================================================
result-parameter information
------------------------- -----------------------------------------------------
template is set to ``videos.html``
========================= =====================================================
url string, url of the result
title string, title of the result
content *(not implemented yet)*
publishedDate :py:class:`datetime.datetime`, time of publish
thumbnail string, url to a small-preview image
length :py:class:`datetime.timedelta`, duration of result
views string, view count in humanized number format
========================= =====================================================
.. _template torrent:
``torrent``
-----------
.. _magnetlink: https://en.wikipedia.org/wiki/Magnet_URI_scheme
.. table:: Parameter of the **torrent** media type:
:width: 100%
========================= =====================================================
result-parameter information
------------------------- -----------------------------------------------------
template is set to ``torrent.html``
========================= =====================================================
url string, url of the result
title string, title of the result
content string, general result-text
publishedDate :py:class:`datetime.datetime`,
time of publish *(not implemented yet)*
seed int, number of seeder
leech int, number of leecher
filesize int, size of file in bytes
files int, number of files
magnetlink string, magnetlink_ of the result
torrentfile string, torrentfile of the result
========================= =====================================================
.. _template map:
``map``
-------
.. table:: Parameter of the **map** media type:
:width: 100%
========================= =====================================================
result-parameter information
------------------------- -----------------------------------------------------
template is set to ``map.html``
========================= =====================================================
url string, url of the result
title string, title of the result
content string, general result-text
publishedDate :py:class:`datetime.datetime`, time of publish
latitude latitude of result (in decimal format)
longitude longitude of result (in decimal format)
boundingbox boundingbox of result (array of 4. values
``[lat-min, lat-max, lon-min, lon-max]``)
geojson geojson of result (https://geojson.org/)
osm.type type of osm-object (if OSM-Result)
osm.id id of osm-object (if OSM-Result)
address.name name of object
address.road street name of object
address.house_number house number of object
address.locality city, place of object
address.postcode postcode of object
address.country country of object
========================= =====================================================
.. _template paper:
``paper``
---------
.. _BibTeX format: https://www.bibtex.com/g/bibtex-format/
.. _BibTeX field types: https://en.wikipedia.org/wiki/BibTeX#Field_types
.. list-table:: Parameter of the **paper** media type /
see `BibTeX field types`_ and `BibTeX format`_
:header-rows: 2
:width: 100%
* - result-parameter
- Python type
- information
* - template
- :py:class:`str`
- is set to ``paper.html``
* - title
- :py:class:`str`
- title of the result
* - content
- :py:class:`str`
- abstract
* - comments
- :py:class:`str`
- free text display in italic below the content
* - tags
- :py:class:`List <list>`\ [\ :py:class:`str`\ ]
- free tag list
* - publishedDate
- :py:class:`datetime <datetime.datetime>`
- last publication date
* - type
- :py:class:`str`
- short description of medium type, e.g. *book*, *pdf* or *html* ...
* - authors
- :py:class:`List <list>`\ [\ :py:class:`str`\ ]
- list of authors of the work (authors with a "s")
* - editor
- :py:class:`str`
- list of editors of a book
* - publisher
- :py:class:`str`
- name of the publisher
* - journal
- :py:class:`str`
- name of the journal or magazine the article was
published in
* - volume
- :py:class:`str`
- volume number
* - pages
- :py:class:`str`
- page range where the article is
* - number
- :py:class:`str`
- number of the report or the issue number for a journal article
* - doi
- :py:class:`str`
- DOI number (like ``10.1038/d41586-018-07848-2``)
* - issn
- :py:class:`List <list>`\ [\ :py:class:`str`\ ]
- ISSN number like ``1476-4687``
* - isbn
- :py:class:`List <list>`\ [\ :py:class:`str`\ ]
- ISBN number like ``9780201896831``
* - pdf_url
- :py:class:`str`
- URL to the full article, the PDF version
* - html_url
- :py:class:`str`
- URL to full article, HTML version
.. _template packages:
``packages``
------------
.. list-table:: Parameter of the **packages** media type
:header-rows: 2
:width: 100%
* - result-parameter
- Python type
- information
* - template
- :py:class:`str`
- is set to ``packages.html``
* - title
- :py:class:`str`
- title of the result
* - content
- :py:class:`str`
- abstract
* - package_name
- :py:class:`str`
- the name of the package
* - version
- :py:class:`str`
- the current version of the package
* - maintainer
- :py:class:`str`
- the maintainer or author of the project
* - publishedDate
- :py:class:`datetime <datetime.datetime>`
- date of latest update or release
* - tags
- :py:class:`List <list>`\ [\ :py:class:`str`\ ]
- free tag list
* - popularity
- :py:class:`str`
- the popularity of the package, e.g. rating or download count
* - license_name
- :py:class:`str`
- the name of the license
* - license_url
- :py:class:`str`
- the web location of a license copy
* - homepage
- :py:class:`str`
- the url of the project's homepage
* - source_code_url
- :py:class:`str`
- the location of the project's source code
* - links
- :py:class:`dict`
- additional links in the form of ``{'link_name': 'http://example.com'}``

7
docs/dev/extended_types.rst Normal file
View file

@ -0,0 +1,7 @@
.. _extended_types.:
==============
Extended Types
==============
.. automodule:: searx.extended_types

6
docs/dev/index.rst
View file

@ -8,9 +8,13 @@ Developer documentation
quickstart
rtm_asdf
contribution_guide
extended_types
engines/index
result_types/index
templates
search_api
plugins
plugins/index
answerers/index
translation
lxcdev
makefile

106
docs/dev/plugins.rst
View file

@ -1,106 +0,0 @@
.. _dev plugin:
=======
Plugins
=======
.. sidebar:: Further reading ..
- :ref:`plugins generic`
Plugins can extend or replace functionality of various components of searx.
Example plugin
==============
.. code:: python
name = 'Example plugin'
description = 'This plugin extends the suggestions with the word "example"'
default_on = False # disabled by default
# attach callback to the post search hook
# request: flask request object
# ctx: the whole local context of the post search hook
def post_search(request, search):
search.result_container.suggestions.add('example')
return True
External plugins
================
SearXNG supports *external plugins* / there is no need to install one, SearXNG
runs out of the box. But to demonstrate; in the example below we install the
SearXNG plugins from *The Green Web Foundation* `[ref]
<https://www.thegreenwebfoundation.org/news/searching-the-green-web-with-searx/>`__:
.. code:: bash
$ sudo utils/searxng.sh instance cmd bash -c
(searxng-pyenv)$ pip install git+https://github.com/return42/tgwf-searx-plugins
In the :ref:`settings.yml` activate the ``plugins:`` section and add module
``only_show_green_results`` from ``tgwf-searx-plugins``.
.. code:: yaml
plugins:
...
- only_show_green_results
...
Plugin entry points
===================
Entry points (hooks) define when a plugin runs. Right now only three hooks are
implemented. So feel free to implement a hook if it fits the behaviour of your
plugin. A plugin doesn't need to implement all the hooks.
.. py:function:: pre_search(request, search) -> bool
Runs BEFORE the search request.
`search.result_container` can be changed.
Return a boolean:
* True to continue the search
* False to stop the search
:param flask.request request:
:param searx.search.SearchWithPlugins search:
:return: False to stop the search
:rtype: bool
.. py:function:: post_search(request, search) -> None
Runs AFTER the search request.
:param flask.request request: Flask request.
:param searx.search.SearchWithPlugins search: Context.
.. py:function:: on_result(request, search, result) -> bool
Runs for each result of each engine.
`result` can be changed.
If `result["url"]` is defined, then `result["parsed_url"] = urlparse(result['url'])`
.. warning::
`result["url"]` can be changed, but `result["parsed_url"]` must be updated too.
Return a boolean:
* True to keep the result
* False to remove the result
:param flask.request request:
:param searx.search.SearchWithPlugins search:
:param typing.Dict result: Result, see - :ref:`engine results`
:return: True to keep the result
:rtype: bool

15
docs/dev/plugins/builtins.rst Normal file
View file

@ -0,0 +1,15 @@
.. _builtin plugins:
================
Built-in Plugins
================
.. toctree::
:maxdepth: 1
calculator
hash_plugin
hostnames
self_info
tor_check
unit_converter

8
docs/dev/plugins/calculator.rst Normal file
View file

@ -0,0 +1,8 @@
.. _plugins.calculator:
==========
Calculator
==========
.. automodule:: searx.plugins.calculator
:members:

7
docs/dev/plugins/development.rst Normal file
View file

@ -0,0 +1,7 @@
.. _dev plugin:
==================
Plugin Development
==================
.. automodule:: searx.plugins

8
docs/dev/plugins/hash_plugin.rst Normal file
View file

@ -0,0 +1,8 @@
.. _hash_plugin plugin:
===========
Hash Values
===========
.. autoclass:: searx.plugins.hash_plugin.SXNGPlugin
:members:

9
docs/src/searx.plugins.hostnames.rst → docs/dev/plugins/hostnames.rst
View file

@ -1,9 +1,8 @@
.. _hostnames plugin:
================
Hostnames plugin
================
=========
Hostnames
=========
.. automodule:: searx.plugins.hostnames
:members:
:members:

9
docs/dev/plugins/index.rst Normal file
View file

@ -0,0 +1,9 @@
=======
Plugins
=======
.. toctree::
:maxdepth: 2
development
builtins

8
docs/dev/plugins/self_info.rst Normal file
View file

@ -0,0 +1,8 @@
.. _self_info plugin:
=========
Self-Info
=========
.. autoclass:: searx.plugins.self_info.SXNGPlugin
:members:

9
docs/src/searx.plugins.tor_check.rst → docs/dev/plugins/tor_check.rst
View file

@ -1,9 +1,8 @@
.. _tor check plugin:
================
Tor check plugin
================
=========
Tor check
=========
.. automodule:: searx.plugins.tor_check
:members:
:members:

7
docs/src/searx.plugins.unit_converter.rst → docs/dev/plugins/unit_converter.rst
View file

@ -1,9 +1,8 @@
.. _unit converter plugin:
=====================
Unit converter plugin
=====================
==============
Unit Converter
==============
.. automodule:: searx.plugins.unit_converter
:members:

7
docs/dev/result_types/answer.rst Normal file
View file

@ -0,0 +1,7 @@
.. _result_types.answer:
==============
Answer Results
==============
.. automodule:: searx.result_types.answer

5
docs/dev/result_types/base_result.rst Normal file
View file

@ -0,0 +1,5 @@
======
Result
======
.. automodule:: searx.result_types._base

34
docs/dev/result_types/correction.rst Normal file
View file

@ -0,0 +1,34 @@
.. _result_types.corrections:
==================
Correction Results
==================
.. hint::
There is still no typing for these result items. The templates can be used as
orientation until the final typing is complete.
The corrections area shows the user alternative search terms.
A result of this type is a very simple dictionary with only one key/value pair
.. code:: python
{"correction" : "lorem ipsum .."}
From this simple dict another dict is build up:
.. code:: python
# use RawTextQuery to get the corrections URLs with the same bang
{"url" : "!bang lorem ipsum ..", "title": "lorem ipsum .." }
and used in the template :origin:`corrections.html
<searx/templates/simple/elements/corrections.html>`:
title : :py:class:`str`
Corrected search term.
url : :py:class:`str`
Not really an URL, its the value to insert in a HTML form for a SearXNG query.

95
docs/dev/result_types/index.rst Normal file
View file

@ -0,0 +1,95 @@
.. _result types:
============
Result Types
============
To understand the typification of the results, let's take a brief look at the
structure of SearXNG .. At its core, SearXNG is nothing more than an aggregator
that aggregates the results from various sources, renders them via templates and
displays them to the user.
The **sources** can be:
1. :ref:`engines <engine implementations>`
2. :ref:`plugins <dev plugin>`
3. :ref:`answerers <dev answerers>`
The sources provide the results, which are displayed in different **areas**
depending on the type of result. The areas are:
main results:
It is the main area in which -- as is typical for search engines -- the
results that a search engine has found for the search term are displayed.
answers:
This area displays short answers that could be found for the search term.
info box:
An area in which additional information can be displayed, e.g. excerpts from
wikipedia or other sources such as maps.
suggestions:
Suggestions for alternative search terms can be found in this area. These can
be clicked on and a search is carried out with these search terms.
corrections:
Results in this area are like the suggestion of alternative search terms,
which usually result from spelling corrections
At this point it is important to note that all **sources** can contribute
results to all of the areas mentioned above.
In most cases, however, the :ref:`engines <engine implementations>` will fill
the *main results* and the :ref:`answerers <dev answerers>` will generally
provide the contributions for the *answer* area. Not necessary to mention here
but for a better understanding: the plugins can also filter out or change
results from the main results area (e.g. the URL of the link).
The result items are organized in the :py:obj:`results.ResultContainer` and
after all sources have delivered their results, this container is passed to the
templating to build a HTML output. The output is usually HTML, but it is also
possible to output the result lists as JSON or RSS feed. Thats quite all we need
to know before we dive into typification of result items.
.. hint::
Typification of result items: we are at the very first beginng!
The first thing we have to realize is that there is no typification of the
result items so far, we have to build it up first .. and that is quite a big
task, which we will only be able to accomplish gradually.
The foundation for the typeless results was laid back in 2013 in the very first
commit :commit:`ae9fb1d7d`, and the principle has not changed since then. At
the time, the approach was perfectly adequate, but we have since evolved and the
demands on SearXNG increase with every feature request.
**Motivation:** in the meantime, it has become very difficult to develop new
features that require structural changes and it is especially hard for newcomers
to find their way in this typeless world. As long as the results are only
simple key/value dictionaries, it is not even possible for the IDEs to support
the application developer in his work.
**Planning:** The procedure for subsequent typing will have to be based on the
circumstances ..
.. attention::
As long as there is no type defined for a kind of result the HTML template
specify what the properties of a type are.
In this sense, you will either find a type definition here in the
documentation or, if this does not yet exist, a description of the HTML
template.
.. toctree::
:maxdepth: 2
base_result
main_result
answer
correction
suggestion
infobox

60
docs/dev/result_types/infobox.rst Normal file
View file

@ -0,0 +1,60 @@
.. _result_types.infobox:
===============
Infobox Results
===============
.. hint::
There is still no typing for these result items. The templates can be used as
orientation until the final typing is complete.
The infobox is an area where addtional infos shown to the user.
Fields used in the :origin:`infobox.html
<searx/templates/simple/elements/infobox.html>`:
img_src: :py:class:`str`
URL of a image or thumbnail that is displayed in the infobox.
infobox: :py:class:`str`
Title of the info box.
content: :py:class:`str`
Text of the info box.
The infobox has additional subsections for *attributes*, *urls* and
*relatedTopics*:
attributes: :py:class:`List <list>`\ [\ :py:class:`dict`\ ]
A list of attributes. An *attribute* is a dictionary with keys:
- label :py:class:`str`: (mandatory)
- value :py:class:`str`: (mandatory)
- image :py:class:`List <list>`\ [\ :py:class:`dict`\ ] (optional)
A list of images. An *image* is a dictionary with keys:
- src :py:class:`str`: URL of an image/thumbnail (mandatory)
- alt :py:class:`str`: alternative text for the image (mandatory)
urls: :py:class:`List <list>`\ [\ :py:class:`dict`\ ]
A list of links. An *link* is a dictionary with keys:
- url :py:class:`str`: URL of the link (mandatory)
- title :py:class:`str`: Title of the link (mandatory)
relatedTopics: :py:class:`List <list>`\ [\ :py:class:`dict`\ ]
A list of topics. An *topic* is a dictionary with keys:
- name: :py:class:`str`: (mandatory)
- suggestions: :py:class:`List <list>`\ [\ :py:class:`dict`\ ] (optional)
A list of suggestions. A *suggestion* is simple dictionary with just one
key/value pair:
- suggestion: :py:class:`str`: suggested search term (mandatory)

17
docs/dev/result_types/main_result.rst Normal file
View file

@ -0,0 +1,17 @@
============
Main Results
============
There is still no typing for the items in the :ref:`main result list`. The
templates can be used as orientation until the final typing is complete.
- :ref:`template default`
- :ref:`template images`
- :ref:`template videos`
- :ref:`template torrent`
- :ref:`template map`
- :ref:`template paper`
- :ref:`template packages`
- :ref:`template code`
- :ref:`template files`
- :ref:`template products`

38
docs/dev/result_types/suggestion.rst Normal file
View file

@ -0,0 +1,38 @@
.. _result_types.suggestion:
==================
Suggestion Results
==================
.. hint::
There is still no typing for these result items. The templates can be used as
orientation until the final typing is complete.
The suggestions area shows the user alternative search terms.
A result of this type is a very simple dictionary with only one key/value pair
.. code:: python
{"suggestion" : "lorem ipsum .."}
From this simple dict another dict is build up:
.. code:: python
{"url" : "!bang lorem ipsum ..", "title": "lorem ipsum" }
and used in the template :origin:`suggestions.html
<searx/templates/simple/elements/suggestions.html>`:
.. code:: python
# use RawTextQuery to get the suggestion URLs with the same bang
{"url" : "!bang lorem ipsum ..", "title": "lorem ipsum" }
title : :py:class:`str`
Suggested search term
url : :py:class:`str`
Not really an URL, its the value to insert in a HTML form for a SearXNG query.

1
docs/dev/searxng_extra/update.rst
View file

@ -60,6 +60,7 @@ Scripts to update static data in :origin:`searx/data/`
.. automodule:: searxng_extra.update.update_engine_traits
:members:
.. _update_osm_keys_tags.py:
``update_osm_keys_tags.py``
===========================

577
docs/dev/templates.rst Normal file
View file

@ -0,0 +1,577 @@
.. _simple theme templates:
======================
Simple Theme Templates
======================
The simple template is complex, it consists of many different elements and also
uses macros and include statements. The following is a rough overview that we
would like to give the developerat hand, details must still be taken from the
:origin:`sources <searx/templates/simple/>`.
A :ref:`result item <result types>` can be of different media types. The media
type of a result is defined by the :py:obj:`result_type.Result.template`. To
set another media-type as :ref:`template default`, the field ``template``
in the result item must be set to the desired type.
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
.. _result template macros:
Result template macros
======================
.. _macro result_header:
``result_header``
-----------------
Execpt ``image.html`` and some others this macro is used in nearly all result
types in the :ref:`main result list`.
Fields used in the template :origin:`macro result_header
<searx/templates/simple/macros.html>`:
url : :py:class:`str`
Link URL of the result item.
title : :py:class:`str`
Link title of the result item.
img_src, thumbnail : :py:class:`str`
URL of a image or thumbnail that is displayed in the result item.
.. _macro result_sub_header:
``result_sub_header``
---------------------
Execpt ``image.html`` and some others this macro is used in nearly all result
types in the :ref:`main result list`.
Fields used in the template :origin:`macro result_sub_header
<searx/templates/simple/macros.html>`:
publishedDate : :py:obj:`datetime.datetime`
The date on which the object was published.
length: :py:obj:`time.struct_time`
Playing duration in seconds.
views: :py:class:`str`
View count in humanized number format.
author : :py:class:`str`
Author of the title.
metadata : :py:class:`str`
Miscellaneous metadata.
.. _engine_data:
``engine_data_form``
--------------------
The ``engine_data_form`` macro is used in :origin:`results,html
<searx/templates/simple/results.html>` in a HTML ``<form/>`` element. The
intention of this macro is to pass data of a engine from one :py:obj:`response
<searx.engines.demo_online.response>` to the :py:obj:`searx.search.SearchQuery`
of the next :py:obj:`request <searx.engines.demo_online.request>`.
To pass data, engine's response handler can append result items of typ
``engine_data``. This is by example used to pass a token from the response to
the next request:
.. code:: python
def response(resp):
...
results.append({
'engine_data': token,
'key': 'next_page_token',
})
...
return results
def request(query, params):
page_token = params['engine_data'].get('next_page_token')
.. _main result list:
Main Result List
================
The **media types** of the **main result type** are the template files in
the :origin:`result_templates <searx/templates/simple/result_templates>`.
.. _template default:
``default.html``
----------------
Displays result fields from:
- :ref:`macro result_header` and
- :ref:`macro result_sub_header`
Additional fields used in the :origin:`default.html
<searx/templates/simple/result_templates/default.html>`:
content : :py:class:`str`
General text of the result item.
iframe_src : :py:class:`str`
URL of an embedded ``<iframe>`` / the frame is collapsible.
audio_src : uri,
URL of an embedded ``<audio controls>``.
.. _template images:
``images.html``
---------------
The images are displayed as small thumbnails in the main results list.
title : :py:class:`str`
Title of the image.
thumbnail_src : :py:class:`str`
URL of a preview of the image.
resolution :py:class:`str`
The resolution of the image (e.g. ``1920 x 1080`` pixel)
Image labels
~~~~~~~~~~~~
Clicking on the preview opens a gallery view in which all further metadata for
the image is displayed. Addition fields used in the :origin:`images.html
<searx/templates/simple/result_templates/images.html>`:
img_src : :py:class:`str`
URL of the full size image.
content: :py:class:`str`
Description of the image.
author: :py:class:`str`
Name of the author of the image.
img_format : :py:class:`str`
The format of the image (e.g. ``png``).
source : :py:class:`str`
Source of the image.
filesize: :py:class:`str`
Size of bytes in :py:obj:`human readable <searx.humanize_bytes>` notation
(e.g. ``MB`` for 1024 \* 1024 Bytes filesize).
url : :py:class:`str`
URL of the page from where the images comes from (source).
.. _template videos:
``videos.html``
---------------
Displays result fields from:
- :ref:`macro result_header` and
- :ref:`macro result_sub_header`
Additional fields used in the :origin:`videos.html
<searx/templates/simple/result_templates/videos.html>`:
iframe_src : :py:class:`str`
URL of an embedded ``<iframe>`` / the frame is collapsible.
The videos are displayed as small thumbnails in the main results list, there
is an additional button to collaps/open the embeded video.
content : :py:class:`str`
Description of the code fragment.
.. _template torrent:
``torrent.html``
----------------
.. _magnet link: https://en.wikipedia.org/wiki/Magnet_URI_scheme
.. _torrent file: https://en.wikipedia.org/wiki/Torrent_file
Displays result fields from:
- :ref:`macro result_header` and
- :ref:`macro result_sub_header`
Additional fields used in the :origin:`torrent.html
<searx/templates/simple/result_templates/torrent.html>`:
magnetlink:
URL of the `magnet link`_.
torrentfile
URL of the `torrent file`_.
seed : ``int``
Number of seeders.
leech : ``int``
Number of leecher
filesize : ``int``
Size in Bytes (rendered to human readable unit of measurement).
files : ``int``
Number of files.
.. _template map:
``map.html``
------------
.. _GeoJSON: https://en.wikipedia.org/wiki/GeoJSON
.. _Leaflet: https://github.com/Leaflet/Leaflet
.. _bbox: https://wiki.openstreetmap.org/wiki/Bounding_Box
.. _HTMLElement.dataset: https://developer.mozilla.org/en-US/docs/Web/API/HTMLElement/dataset
.. _Nominatim: https://nominatim.org/release-docs/latest/
.. _Lookup: https://nominatim.org/release-docs/latest/api/Lookup/
.. _place_id is not a persistent id:
https://nominatim.org/release-docs/latest/api/Output/#place_id-is-not-a-persistent-id
.. _perma_id: https://wiki.openstreetmap.org/wiki/Permanent_ID
.. _country code: https://wiki.openstreetmap.org/wiki/Country_code
Displays result fields from:
- :ref:`macro result_header` and
- :ref:`macro result_sub_header`
Additional fields used in the :origin:`map.html
<searx/templates/simple/result_templates/map.html>`:
content : :py:class:`str`
Description of the item.
address_label : :py:class:`str`
Label of the address / default ``_('address')``.
geojson : GeoJSON_
Geometries mapped to HTMLElement.dataset_ (``data-map-geojson``) and used by
Leaflet_.
boundingbox : ``[ min-lon, min-lat, max-lon, max-lat]``
A bbox_ area defined by min longitude , min latitude , max longitude and max
latitude. The bounding box is mapped to HTMLElement.dataset_
(``data-map-boundingbox``) and is used by Leaflet_.
longitude, latitude : :py:class:`str`
Geographical coordinates, mapped to HTMLElement.dataset_ (``data-map-lon``,
``data-map-lat``) and is used by Leaflet_.
address : ``{...}``
A dicticonary with the address data:
.. code:: python
address = {
'name' : str, # name of object
'road' : str, # street name of object
'house_number' : str, # house number of object
'postcode' : str, # postcode of object
'country' : str, # country of object
'country_code' : str,
'locality' : str,
}
country_code : :py:class:`str`
`Country code`_ of the object.
locality : :py:class:`str`
The name of the city, town, township, village, borough, etc. in which this
object is located.
links : ``[link1, link2, ...]``
A list of links with labels:
.. code:: python
links.append({
'label' : str,
'url' : str,
'url_label' : str, # set by some engines but unused (oscar)
})
data : ``[data1, data2, ...]``
A list of additional data, shown in two columns and containing a label and
value.
.. code:: python
data.append({
'label' : str,
'value' : str,
'key' : str, # set by some engines but unused
})
type : :py:class:`str` # set by some engines but unused (oscar)
Tag label from :ref:`OSM_KEYS_TAGS['tags'] <update_osm_keys_tags.py>`.
type_icon : :py:class:`str` # set by some engines but unused (oscar)
Type's icon.
osm : ``{...}``
OSM-type and OSM-ID, can be used to Lookup_ OSM data (Nominatim_). There is
also a discussion about "`place_id is not a persistent id`_" and the
perma_id_.
.. code:: python
osm = {
'type': str,
'id': str,
}
type : :py:class:`str`
Type of osm-object (if OSM-Result).
id :
ID of osm-object (if OSM-Result).
.. hint::
The ``osm`` property is set by engine ``openstreetmap.py``, but it is not
used in the ``map.html`` template yet.
.. _template paper:
``paper.html``
--------------
.. _BibTeX format: https://www.bibtex.com/g/bibtex-format/
.. _BibTeX field types: https://en.wikipedia.org/wiki/BibTeX#Field_types
Displays result fields from:
- :ref:`macro result_header`
Additional fields used in the :origin:`paper.html
<searx/templates/simple/result_templates/paper.html>`:
content : :py:class:`str`
An abstract or excerpt from the document.
comments : :py:class:`str`
Free text display in italic below the content.
tags : :py:class:`List <list>`\ [\ :py:class:`str`\ ]
Free tag list.
type : :py:class:`str`
Short description of medium type, e.g. *book*, *pdf* or *html* ...
authors : :py:class:`List <list>`\ [\ :py:class:`str`\ ]
List of authors of the work (authors with a "s" suffix, the "author" is in the
:ref:`macro result_sub_header`).
editor : :py:class:`str`
Editor of the book/paper.
publisher : :py:class:`str`
Name of the publisher.
journal : :py:class:`str`
Name of the journal or magazine the article was published in.
volume : :py:class:`str`
Volume number.
pages : :py:class:`str`
Page range where the article is.
number : :py:class:`str`
Number of the report or the issue number for a journal article.
doi : :py:class:`str`
DOI number (like ``10.1038/d41586-018-07848-2``).
issn : :py:class:`List <list>`\ [\ :py:class:`str`\ ]
ISSN number like ``1476-4687``
isbn : :py:class:`List <list>`\ [\ :py:class:`str`\ ]
ISBN number like ``9780201896831``
pdf_url : :py:class:`str`
URL to the full article, the PDF version
html_url : :py:class:`str`
URL to full article, HTML version
.. _template packages:
``packages``
------------
Displays result fields from:
- :ref:`macro result_header`
Additional fields used in the :origin:`packages.html
<searx/templates/simple/result_templates/packages.html>`:
package_name : :py:class:`str`
The name of the package.
version : :py:class:`str`
The current version of the package.
maintainer : :py:class:`str`
The maintainer or author of the project.
publishedDate : :py:class:`datetime <datetime.datetime>`
Date of latest update or release.
tags : :py:class:`List <list>`\ [\ :py:class:`str`\ ]
Free tag list.
popularity : :py:class:`str`
The popularity of the package, e.g. rating or download count.
license_name : :py:class:`str`
The name of the license.
license_url : :py:class:`str`
The web location of a license copy.
homepage : :py:class:`str`
The url of the project's homepage.
source_code_url: :py:class:`str`
The location of the project's source code.
links : :py:class:`dict`
Additional links in the form of ``{'link_name': 'http://example.com'}``
.. _template code:
``code.html``
-------------
Displays result fields from:
- :ref:`macro result_header` and
- :ref:`macro result_sub_header`
Additional fields used in the :origin:`code.html
<searx/templates/simple/result_templates/code.html>`:
content : :py:class:`str`
Description of the code fragment.
codelines : ``[line1, line2, ...]``
Lines of the code fragment.
code_language : :py:class:`str`
Name of the code language, the value is passed to
:py:obj:`pygments.lexers.get_lexer_by_name`.
repository : :py:class:`str`
URL of the repository of the code fragment.
.. _template files:
``files.html``
--------------
Displays result fields from:
- :ref:`macro result_header` and
- :ref:`macro result_sub_header`
Additional fields used in the :origin:`code.html
<searx/templates/simple/result_templates/files.html>`:
filename, size, time: :py:class:`str`
Filename, Filesize and Date of the file.
mtype : ``audio`` | ``video`` | :py:class:`str`
Mimetype type of the file.
subtype : :py:class:`str`
Mimetype / subtype of the file.
abstract : :py:class:`str`
Abstract of the file.
author : :py:class:`str`
Name of the author of the file
embedded : :py:class:`str`
URL of an embedded media type (``audio`` or ``video``) / is collapsible.
.. _template products:
``products.html``
-----------------
Displays result fields from:
- :ref:`macro result_header` and
- :ref:`macro result_sub_header`
Additional fields used in the :origin:`products.html
<searx/templates/simple/result_templates/products.html>`:
content : :py:class:`str`
Description of the product.
price : :py:class:`str`
The price must include the currency.
shipping : :py:class:`str`
Shipping details.
source_country : :py:class:`str`
Place from which the shipment is made.
.. _template answer results:
Answer results
==============
See :ref:`result_types.answer`
Suggestion results
==================
See :ref:`result_types.suggestion`
Correction results
==================
See :ref:`result_types.corrections`
Infobox results
===============
See :ref:`result_types.infobox`

6
docs/src/index.rst
View file

@ -2,9 +2,9 @@
Source-Code
===========
This is a partial documentation of our source code. We are not aiming to document
every item from the source code, but we will add documentation when requested.
This is a partial documentation of our source code. We are not aiming to
document every item from the source code, but we will add documentation when
requested.
.. toctree::
:maxdepth: 2