[mod] improve 'Autodetect search language' plugin

- Add documentation to the plugin
- Harmonize FastText language model with SearXNG's language model

Reosurces::

    import fasttext                                    # --> +10 MB
    fasttext.load_model(str(data_dir / 'lid.176.ftz')) # --> +4MB

Suggested-by: @dalf

- To speed up and simplify the deployment use fasttext-wheel instead of fasttext
- Building numpy on the Alpine Linux of docker-images takes ages --> install
  py3-numpy from Alpines package manager (apk)
- Alpine Linux on docker-images (musl libc) do not support fasttext-wheel (gnu
  libc) --> patch Dockerfile and build from fastetxt:

     sed -i s/fasttext-wheel/fasttext/ requirements.txt

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This commit is contained in:
Markus Heiser 2022-12-10 16:20:09 +01:00
parent 9925a20950
commit ed901ab18e
5 changed files with 119 additions and 9 deletions

View file

@ -36,6 +36,7 @@ RUN apk add --no-cache -t build-dependencies \
su-exec \
python3 \
py3-pip \
py3-numpy \
libxml2 \
libxslt \
openssl \
@ -43,6 +44,8 @@ RUN apk add --no-cache -t build-dependencies \
uwsgi \
uwsgi-python3 \
brotli \
&& pip3 install --no-cache setuptools wheel \
&& sed -i s/fasttext-wheel/fasttext/ requirements.txt \
&& pip3 install --no-cache -r requirements.txt \
&& apk del build-dependencies \
&& rm -rf /root/.cache