[enh] initial structure

docs/dev/contribution_guide.rst

@@ -0,0 +1,71 @@
+How to contribute
+-----------------
+
+Prime directives: Privacy, Hackability
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Searx has 2 prime directives, privacy-by-design and hackability. The
+hackability comes in at least 3 levels:
+
+-  support for search engines
+-  plugins for altering search behaviour
+-  hacking searx itself.
+
+Happy hacking. Observe the lack of "world domination" among the
+directives, searx has no intentions for wide mass-adoption, rounded
+corners, etc. The prime directive: "privacy" - deserves a seperate
+chapter, as it's quite uncommon unfortunately, here it goes:
+
+Privacy-by-design
+^^^^^^^^^^^^^^^^^
+
+Searx is a privacy-respecting, hackable meta-search engine. It was born
+out of the need for a privacy-respecing search facility that can be
+expanded easily to maximise both its search and it's privacy protecting
+capabilities.
+
+Consequences of Privacy-by-design are that some widely used features
+work differently or not by default or at all. If some feature reduces
+the privacy perserving aspects of searx, it should by default be
+switched of, if implemented at all. There is enough search engines
+already out there providing such features. = Since privacy-preservation
+is a prime goal, if some feature does reduce the protection of searx and
+is implemented, care should be taken to educate the user about the
+consequences of choosing to enable this. Further features which
+implement widely known features in a manner that protects privacy but
+thus deviate from the users expectations should also be explained to the
+user. Also if you think that something works weird with searx, maybe
+it's because of the tool you use is designed in a way to interfere with
+privacy respect, submiting a bugreport to the vendor of the tool that
+misbehaves might be a good feedback for the vendor to reconsider his
+disrespect towards his customers (e.g. GET vs POST requests in various
+browsers).
+
+Remember the other prime directive of searx is to be hackable, so if the
+above privacy concerns do not fancy you, simply fork it.
+
+Code
+~~~~
+
+Code modifications are accepted in pull requests, don't forget to add
+yourself to the AUTHORS file.
+
+Python code follows all the pep8 standards except maximum line width
+which is 120 char.
+
+Please be sure that the submitted code doesn't break existing tests and
+follows coding conventions.
+
+If new functionality implemented, tests are highly appreciated.
+
+Translation
+~~~~~~~~~~~
+
+Translation currently happens on
+`transifex <https://transifex.com/projects/p/searx>`__. Please do not
+update translation files in the repo.
+
+Documentation
+~~~~~~~~~~~~~
+
+The main place of the documentation is this wiki, updates are welcome.

docs/dev/install/installation.rst

@@ -0,0 +1,277 @@
+Installation
+============
+
+Step by step installation for Debian / Ubuntu with virtualenv.
+
+Source: https://about.okhin.fr/posts/Searx/ with some additions
+
+How to: `Setup searx in a couple of hours with a free SSL
+certificate <https://www.reddit.com/r/privacytoolsIO/comments/366kvn/how_to_setup_your_own_privacy_respecting_search/>`__
+
+Basic installation
+------------------
+
+For Ubuntu, be sure to have enable universe repository.
+
+Install packages :
+
+.. code:: sh
+
+    sudo apt-get install git build-essential libxslt-dev python-dev python-virtualenv python-pybabel zlib1g-dev libffi-dev libssl-dev
+
+Install searx :
+
+.. code:: sh
+
+    cd /usr/local
+    sudo git clone https://github.com/asciimoo/searx.git
+    sudo useradd searx -d /usr/local/searx
+    sudo chown searx:searx -R /usr/local/searx
+
+Install dependencies in a virtualenv :
+
+.. code:: sh
+
+    sudo -u searx -i
+    cd /usr/local/searx
+    virtualenv searx-ve
+    . ./searx-ve/bin/activate
+    pip install -r requirements.txt
+    python setup.py install
+
+Configuration
+-------------
+
+.. code:: sh
+
+    sed -i -e "s/ultrasecretkey/`openssl rand -hex 16`/g" searx/settings.yml
+
+Edit searx/settings.yml if necessary.
+
+Check
+-----
+
+Start searx :
+
+.. code:: sh
+
+    python searx/webapp.py
+
+Go to http://localhost:8888
+
+If everything works fine, disable the debug option in settings.yml :
+
+.. code:: sh
+
+    sed -i -e "s/debug : True/debug : False/g" searx/settings.yml
+
+At this point searx is not demonized ; uwsgi allows this.
+
+You can exit the virtualenv and the searx user bash (enter exit command
+twice).
+
+uwsgi
+-----
+
+Install packages :
+
+.. code:: sh
+
+    sudo apt-get install uwsgi uwsgi-plugin-python
+
+Create the configuration file /etc/uwsgi/apps-available/searx.ini with
+this content :
+
+::
+
+    [uwsgi]
+    # Who will run the code
+    uid = searx
+    gid = searx
+
+    # disable logging for privacy
+    disable-logging = true
+
+    # Number of workers (usually CPU count)
+    workers = 4
+
+    # The right granted on the created socket
+    chmod-socket = 666
+
+    # Plugin to use and interpretor config
+    single-interpreter = true
+    master = true
+    plugin = python
+
+    # Module to import
+    module = searx.webapp
+
+    # Virtualenv and python path
+    virtualenv = /usr/local/searx/searx-ve/
+    pythonpath = /usr/local/searx/
+    chdir = /usr/local/searx/searx/
+
+Activate the uwsgi application and restart :
+
+.. code:: sh
+
+    cd /etc/uwsgi/apps-enabled
+    ln -s ../apps-available/searx.ini
+    /etc/init.d/uwsgi restart
+
+Web server
+----------
+
+with nginx
+^^^^^^^^^^
+
+If nginx is not installed (uwsgi will not work with the package
+nginx-light) :
+
+.. code:: sh
+
+    sudo apt-get install nginx
+
+Hosted at /
+"""""""""""
+
+Create the configuration file /etc/nginx/sites-available/searx with this
+content :
+
+.. code:: nginx
+
+    server {
+        listen 80;
+        server_name searx.example.com;
+        root /usr/local/searx;
+
+        location / {
+                include uwsgi_params;
+                uwsgi_pass unix:/run/uwsgi/app/searx/socket;
+        }
+    }
+
+Restart service :
+
+.. code:: sh
+
+    sudo service nginx restart
+    sudo service uwsgi restart
+
+from subdirectory URL (/searx)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Add this configuration in the server config file
+/etc/nginx/sites-available/default :
+
+.. code:: nginx
+
+    location = /searx { rewrite ^ /searx/; }
+    location /searx {
+            try_files $uri @searx;
+    }
+    location @searx {
+            uwsgi_param SCRIPT_NAME /searx;
+            include uwsgi_params;
+            uwsgi_modifier1 30;
+            uwsgi_pass unix:/run/uwsgi/app/searx/socket;
+    }
+
+Enable base\_url in searx/settings.yml
+
+::
+
+    base_url : http://your.domain.tld/searx/
+
+Restart service :
+
+.. code:: sh
+
+    sudo service nginx restart
+    sudo service uwsgi restart
+
+disable logs
+~~~~~~~~~~~~
+
+for better privacy you can disable nginx logs about searx.
+
+how to proceed : below ``uwsgi_pass`` in
+/etc/nginx/sites-available/default add
+
+::
+
+    access_log /dev/null;
+    error_log /dev/null;
+
+Restart service :
+
+.. code:: sh
+
+    sudo service nginx restart
+
+with apache
+-----------
+
+Add wsgi mod :
+
+.. code:: sh
+
+    sudo apt-get install libapache2-mod-uwsgi
+    sudo a2enmod uwsgi
+
+Add this configuration in the file /etc/apache2/apache2.conf :
+
+.. code:: apache
+
+    <Location />
+        Options FollowSymLinks Indexes
+        SetHandler uwsgi-handler
+        uWSGISocket /run/uwsgi/app/searx/socket
+    </Location>
+
+Note that if your instance of searx is not at the root, you should
+change ``<Location />`` by the location of your instance, like
+``<Location /searx>``.
+
+Restart Apache :
+
+.. code:: sh
+
+    sudo /etc/init.d/apache2 restart
+
+disable logs
+------------
+
+For better privacy you can disable Apache logs.
+
+WARNING : not tested
+
+WARNING : you can only disable logs for the whole (virtual) server not
+for a specific path.
+
+Go back to /etc/apache2/apache2.conf and above ``<Location />`` add :
+
+.. code:: apache
+
+    CustomLog /dev/null combined
+
+Restart Apache :
+
+.. code:: sh
+
+    sudo /etc/init.d/apache2 restart
+
+How to update
+-------------
+
+.. code:: sh
+
+    cd /usr/local/searx
+    sudo -u searx -i
+    . ./searx-ve/bin/activate
+    git stash
+    git pull origin master
+    git stash apply
+    pip install --upgrade -r requirements.txt
+    sudo service uwsgi restart
+

docs/dev/plugins.rst

@@ -0,0 +1,45 @@
+Plugins
+-------
+
+Plugins can extend/replace functionality of various components inside
+searx.
+
+example\_plugin.py
+~~~~~~~~~~~~~~~~~~
+
+.. code:: python
+
+    name = 'Example plugin'
+    description = 'This plugin extends the suggestions with the word "example"'
+    default_on = False  # disable by default
+
+    js_dependencies = tuple()  # optional, list of static js files
+    css_dependencies = tuple()  # optional, list of static css files
+
+
+    # attach callback to the post search hook
+    #  request: flask request object
+    #  ctx: the whole local context of the post search hook
+    def post_search(request, ctx):
+        ctx['search'].suggestions.add('example')
+        return True
+
+Currently implemented plugin entry points (a.k.a hooks)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+-  Pre search hook (``pre_search``)
+-  Post search hook (``post_search``)
+-  Result hook (``on_result``) (is called if a new result is added (see
+   https\_rewrite plugin))
+
+Feel free to add more hooks to the code if it is required by a plugin.
+
+TODO
+~~~~
+
+-  Better documentation
+-  More hooks
+-  search hook (is called while searx is requesting results (for
+   example: things like math-solver), the different hooks are running
+   parallel)
+

docs/dev/search_api.rst

@@ -0,0 +1,24 @@
+Search API
+==========
+
+Search API endpoints: ``/``, ``/search``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Endpoints have equivalent functionality.
+
+Parameters
+^^^^^^^^^^
+
++------------------+----------------------------------------------------------------------------------------------------+-----------------------------+
+| Name             | Description                                                                                        |                             |
++==================+====================================================================================================+=============================+
+| ``q``            | The search query, see :doc:`/user/search_syntax`                                                   | required                    |
++------------------+----------------------------------------------------------------------------------------------------+-----------------------------+
+| ``categories``   | Comma separated list, specifies the active search categories                                       | optional                    |
++------------------+----------------------------------------------------------------------------------------------------+-----------------------------+
+| ``engines``      | Comma separated list, specifies the active search engines                                          | optional                    |
++------------------+----------------------------------------------------------------------------------------------------+-----------------------------+
+| ``pageno``       | Search page number                                                                                 | optional (default: ``1``)   |
++------------------+----------------------------------------------------------------------------------------------------+-----------------------------+
+
+Both ``GET`` and ``POST`` methods are supported.

docs/dev/translation.rst

@@ -0,0 +1,74 @@
+Translation
+===========
+
+run these commands in the root directory of searx
+
+Add new language
+~~~~~~~~~~~~~~~~
+
+``pybabel init -i messages.pot -d searx/translations -l it``
+
+Update .po files
+~~~~~~~~~~~~~~~~
+
+``./utils/update-translations.sh``
+
+You may have errors here. In that case, edit the
+``update-translations.sh`` script to change ``pybabel`` to
+``pybabel-python2``
+
+After this step, you can modify the .po files.
+
+Compile translations
+~~~~~~~~~~~~~~~~~~~~
+
+``pybabel compile -d searx/translations``
+
+Transifex stuff
+~~~~~~~~~~~~~~~
+
+Init Project
+^^^^^^^^^^^^
+
+.. code:: shell
+
+    tx set --auto-local -r searx.messagespo 'searx/translations/<lang>/LC_MESSAGES/messages.po' \
+    --source-lang en --type PO --source-file messages.pot --execute
+
+http://docs.transifex.com/developer/client/set
+
+*TODO: mapping between transifex and searx*
+
+Get translations
+^^^^^^^^^^^^^^^^
+
+.. code:: shell
+
+    tx pull -a
+
+http://docs.transifex.com/developer/client/pull
+
+Upload source File
+^^^^^^^^^^^^^^^^^^
+
+::
+
+    tx push -s
+
+Upload all Translation
+^^^^^^^^^^^^^^^^^^^^^^
+
+::
+
+    tx push -s -t
+
+upload specifc Translation (only for admins)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+    tx push -t -l tr
+
+http://docs.transifex.com/developer/client/push
+
+*TODO: upload empty files? (new translations)*