For gevent pool, we could use the paramter '-c' to specify the greenlet number. As the following shows.
celery worker -A celerytasks.celery_worker_init -P gevent -c 1000 --loglevel=info
Questions:
What is the appropriate value for it?
What does it really mean for 'greenlet number'?
If I specify 1000 greenlets, what will happen if they are used out?
The -c parameter is a Celery parameter that allows you to specify how many task at the same time will be executed from the worker, here is an extract from the doc:
The number of worker processes/threads can be changed using the --concurrency argument and defaults to the number of CPUs available on the machine.
When running celery and instructing to use gevent this means that instead of using prefork celery will use green thread to process your tasks. In this case you can refer to this page about eventlet on celery docs, that is not the same as gevent but the same concepts apply.
Here an extract that can help you understand:
The prefork pool can take use of multiple processes, but how many is often limited to a few processes per CPU. With Eventlet you can efficiently spawn hundreds, or thousands of green threads.
However note that gevent or eventlet is not always better than prefork, often depends on what type of task you need to execute like explained in this part of the doc:
Celery supports Eventlet as an alternative execution pool implementation. It is in some cases superior to prefork, but you need to ensure your tasks do not perform blocking calls, as this will halt all other operations in the worker until the blocking call returns.
About the dimensioning is hard to answer because also this depends strongly on your type of tasks and average load. In my personal experience is often better to don't go over 1000-3000 for one celery worker.
One last note, from celery docs looks like that gevent is still in experimental state , you may want to use eventlet instead.