Finish replacement of Redis with RabbitMQ.

The current batch pipeline relies on a RabbitMQ. The progress value is
not incremented and goes directly from 0 to 100 when the job finishes.
Therefore at any given time the current processing jobs' number is
always 0.

Gaël authored 2 years ago and Alexandre committed 1 year ago

* use a fixed number of process in the test agent
* poll for new tests in workers and not in parent process
* move metrics to engine

Credits @blacksponge

To avoid potential memory exhaustion.

Clickhouse does not have a UNIQUE constraint per column. Therefore the
check for user uniqueness is performed manually.

There is no COMMIT with Clickhouse. However there is no restriction
either on the number of bind parameters. Therefore we can pass all
domains in one pass via bind parameters.

This has been successfully tested with a batch made of 2 millions
domains (with Clickhouse only).

The "id" field is not defined with Clickhouse. Therefore use the
"created_at" field always defined.
Caution: indexes have not been tested

Because of asynchronous update in Clickhouse, sleep an arbitrary time
after each table UPDATE.

Connections to Clickhouse are made through its MySQL interface.
Clickhouse mostly supports standard SQL, however sometimes there are
some deviation (for instance to UPDATE a row, or no native support for
incremental indexes).