[Maas-devel] We might want to upgrade to celery 3.0

[Julian Edwards]

On 04/11/13 21:01, Gavin Panella wrote:
> On 4 November 2013 07:25, Julian Edwards <julian.edwards at canonical.com> wrote:
>> It came out in June last year but seems not to be packaged as the celery
>> in Saucy is still on the same version that was distributed with Quantal.
>>
>> There's a few nice changes:
>>
>> - new api (old one still available)
>> - chaining of tasks now possible, also with callbacks on groups of tasks
>> - uses librabbitmq C lib instead of py-amqp (one fewer Python dependency)
>> - threadless; performs better
>> - Now uses a single "celery" command line which wraps up some
>> interesting management commands.
>> - runs on Python 3
>>
>> Also while I am talking about celery, I noticed that it has fanout
>> queues called "Broadcast" queues.  This will enable us to easily solve
>> the region worker SPoF problem.
>>
>> Sadly there doesn't seem to be anything built-in to detect missing
>> workers, so we'll probably have to DIY that.
> 
> Given that RabbitMQ and Celery pose problems to HA, it may be worth
> considering their replacement with ZeroMQ. The HA problems are not
> insurmountable, but they do represent work that needs doing, and
> debugging, and maintenance, and so forth. Every service that we can
> get rid of from MAAS is one less thing for end users to administer and
> maintain too.
> 
> RabbitMQ is also a large moving part that we don't understand very
> well. Moving to something like ZeroMQ would, I hope, reduce the
> cognitive load on developers and users alike. Celery 3.1 (in
> development) has hints of ZeroMQ compatibility as a transport so this
> might be something that we can do in stages, or just stop after
> eliminating RabbitMQ from the stack.
> 
> I'll collect the ways in which we're currently using RabbitMQ and
> Celery and come up with ways to replace them with ZeroMQ with Celery
> 3.1+ (or another task runner, or something home-grown), and estimate
> the work needed.
> 
> Gavin.
> 


Awesome, thanks.  I saw on the celery IRC channel that they're releasing
3.1 on Thursday.