On demand, the tracker data is loaded directly into the cache, so that the
maintenance of this data via PRs is no longer necessary.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The action does not take into account all cases of how an image is stored, causing errors like the ones below on image pull. I exclude `base` until I find a solution.
*Error: internal error: unable to copy from source ...: initializing source ...: reading manifest ... in ghcr.io/searxng/base: manifest unknown*
Integration / Python 3.10 (push) Has been cancelled
Integration / Python 3.11 (push) Has been cancelled
Integration / Python 3.12 (push) Has been cancelled
Integration / Python 3.13 (push) Has been cancelled
Integration / Python 3.9 (push) Has been cancelled
Integration / Theme (push) Has been cancelled
I'm not too pleased to reverse this, but issues like https://github.com/searxng/searxng/issues/4792 have not been foreseen, and we can't just turn away. It has become apparent over the last weeks that there are still quite a few people with an incompatible CPU or having SearXNG on some random VM provider who can't (or won't) modify the configuration of their machines to expose the features needed for x86_64v2 march.
As I don't want to trash the work with apko and base images, I thought about trying building Alpine again now that we have all the container related workflow refactored.
There will still be the discussion of whether to use musl and its drawbacks, but right now I don't know any other alternatives.
The nice part of this is that both Dockerfiles (mainline and legacy) can now be unified under the same umbrella again.
Closes https://github.com/searxng/searxng/issues/4792
Closes https://github.com/searxng/searxng/issues/4753
Integration / Python 3.10 (push) Has been cancelled
Integration / Python 3.11 (push) Has been cancelled
Integration / Python 3.12 (push) Has been cancelled
Integration / Python 3.13 (push) Has been cancelled
Integration / Theme (push) Has been cancelled
Documentation / Release (push) Has been cancelled
Integration / Python 3.9 (push) Has been cancelled
checker.yml
1. The checker is not yet of sufficient quality to allow the results of the
check to be evaluated / we do not evaluate them ourselves.
2. The checker sends hundreds of requests to the search engines and causes
problems there / we either overload small providers or we train their bot
defenses to use the SearXNG signature.
documentation.yml
Building the documentation and deploying it on GH-docs of a clones (GH forks) is
generally not desirable either --> We have >2k clones, but we only need one
up-to-date documentation and that is the one from the master branch of the
searxng/searxng repo.
If search engines like Google start linking to the documentation in the clones,
SearXNG users may no longer find the original documentation or be lost in the
flood of options.
Related:
- https://github.com/searxng/searxng/issues/4847
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Instead of using Wolfi base images from cgr.dev and making that mess on the Dockerfile, why don't we build the base images ourselves from Wolfi repos with apko? The intention of this is to simplify the main Dockerfile and avoid having to patch the base image every time, it also simplifies some steps like image ownership management and provides extremely fast builds.
If the workflow is executed with the "workflow_dispatch" trigger, the user who executed the workflow becomes the author of the commit on the PR, this is not intended.
It also reverts the body param so that the default text of the action does not appear.
`checker.yml` and `integration.yml` are the only workflows that are currently safe to be executed simultaneously, the others present a risk that the order of completion may not be expected. The ones that are chained from `integration.yml` can be called as many times as `integration.yml` workflows are running at that moment, the same with the trigger "workflow_dispatch".
This can be fatal for workflows like `container.yml` that use a centralized cache to store and load the candidate images in a common tag called "searxng-<arch>".
* For example, a `container.yml` workflow is executed after being chained from `integration.yml` (called "~1"), and seconds later it may be triggered again because another PR merged some breaking changes (called "~2"). While "~1" has already passed the test job successfully and is about to start the release job, "~2" finishes building the container and overwrites the references on the common tag. When "~1" in the release job loads the images using the common tag, it will load the container of "~2" instead of "~1" having skipped the whole test job process.
The example is only set for the container workflow, but the other workflows might occur in a similar way.
Currently, we have 1100~ cache images uploaded to GHCR that weigh more than 300 MB each (most of them are layers from the second phase of the Dockerfile that were uploaded by mistake, read below). To avoid problems, I have set up a new job in a new workflow to be run weekly purging all images older than 1 week, but leaving always the 100 most recent ones.
Only the builder images should be uploaded to cache, the actual behaviour not only slows down the time for building the container, but also wastes lots of space by saving large and useless layers to GHCR that will never be used again.
When making the container rework, I unknowingly deleted the section where an env with the same name as the secret was defined on the job scope, making it look like it was originally defined as an organization env.
Since we can't validate the secrets in a condition directly, it's better to let docker/login-action take care of failing the entire job if the credentials are invalid.
Reported in: https://github.com/searxng/searxng/issues/4777
env.DOCKERHUB_USERNAME shouldn't be an empty string as it's defined and set (I think, I can't see this). Even if wasn't defined, GitHub Org/Repo wide envs/secrets should return an empty string (?)
container.yml will run after integration.yml COMPLETES successfully and in master branch.
Style changes, cleanup and improved integration with CI by leveraging the use of
shared cache between all workflows.
* Podman is now supported to build the container images (Docker also received a refactor, merging both build and buildx)
* Container images are being built by Buildah instead of Docker BuildKit.
* Container images are tested before release.
* Splitting "modern" (amd64 & arm64) and "legacy" (armv7) arches on different Dockerfiles allowing future optimizations.
l10n.yml will run after integration.yml finishes successfully (will defer anything depending on integration.yml until heavy loads like container building are moved to separate workflows) and in master branch.
* After every integration.yml workflow completes successfully, only the `update` job runs.
* Dispatch and Crontab triggers only the `pr` job.
Style changes, cleanup and improved integration with CI by leveraging the use of shared cache between all workflows (not functional until all workflows have been refactored).
Integration / Update translations branch (push) Blocked by required conditions
Integration / Docker (push) Blocked by required conditions
Instead of executing the workflow after integration.yml completes correctly, let's run this workflow parallel to integration.yml restoring the original behaviour.
We actually don't need to keep the token on checkout because `peter-evans/create-pull-request` will read from `github.token`. The obvious `pull-requests` write permission wasn't set in the last fix, so I added it now.
documentation.yml will run after integration.yml COMPLETES successfully (will
defer anything depending on integration.yml until heavy loads like container
building are moved to separate workflows) and in master branch.
Style changes, cleanup and improved integration with CI by leveraging the use of
shared cache between all workflows (not functional until all workflows have been
refactored).
Style changes, cleanup and improved integration with CI by leveraging the use of
shared cache between all workflows (not functional until all workflows have been
refactored).
Style changes, cleanup and improved integration with CI by leveraging the use of
shared cache between all workflows (not functional until all workflows have been
refactored).
- upgrade to ubuntu-24.04 to get NodeJS v20
- remove DEBUG (V=1) environment / in a ViteJS env the DEBUG environment is
reserved for interactive debugging tasks (not for verbose build messages).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Vite is a build tool, it consists of two major parts [1]:
- A dev server that provides rich feature enhancements over native ES modules,
for example extremely fast Hot Module Replacement (HMR).
- A build command that bundles your code with Rollup, pre-configured to output
highly optimized static assets for production.
[1] https://vite.dev/guide/
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Add a leading tag (in square brackets) about the scope/type to commit messages
from automated tasks (commits from CI).
dependantbot::
[upd] pypi: Bump .. from .. to ..
[upd] npm: Bump .. from .. to .. in /searx/static/themes/simple
Weblate translation updates::
[l10n] update translations from Weblate
updates of ./data::
[data] update searx.data ...
build commit of gh-pages::
[doc] build from commit ...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
All the environments defined in ./utils/brand.env are generated on the fly, so
there is no longer a need to define the brand environment in this file and all
the workflows to handle this file.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Implementations of the *traits* of the engines.
Engine's traits are fetched from the origin engine and stored in a JSON file in
the *data folder*. Most often traits are languages and region codes and their
mapping from SearXNG's representation to the representation in the origin search
engine.
To load traits from the persistence::
searx.enginelib.traits.EngineTraitsMap.from_data()
For new traits new properties can be added to the class::
searx.enginelib.traits.EngineTraits
.. hint::
Implementation is downward compatible to the deprecated *supported_languages
method* from the vintage implementation.
The vintage code is tagged as *deprecated* an can be removed when all engines
has been ported to the *traits method*.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
