pythongoogle-app-engine

GAE List tasks that start with a name?

When I add a queue, I give it an unique name, like longprocess-{id}-{timestamp}. ID is the id in the database for that entry to work, and the timestamp ensure I don't have colliding names in the queue.

The issue is that the user can stop/resume the longprocess if he wants, so in the stop request, I'd like to list all tasks that starts with longporcess-1 (for {id} = 1), and stop all of them (expected 1 entry).

I can target a task with :

q = taskqueue.Queue('longprocess')
q.delete_tasks(taskqueue.Task(name='longprocess-{0}'.format(longprocess.id,)))

But of course, this doesn't work because the name is incorrect (missing it's -{timestamp} part).

Is there something like a q.search('longprocess-1-*') that I would loop over and delete ?

Solution

  • No, there is nothing like q.search('longprocess-1-*') and there could not be (nor it's technically impossible but is just not reasonable) due to nature of queues (in principal, otherwise it's just going to be a DB table).

    The advantage (and limitations) of queues is that they use FIFO (firs-in-first out) - not strictly, sometimes with some extensions like "delay" parameters for a task. But anyway tasks scheduler/dispatcher/coordinator does not need to care about deleting tasks from the middle of the queue and is concentrated on work with limited number of tasks in the head of queue. From this specialization we gaining speed, cost effectiveness & reliability of the queues concept.

    It's your job to handle how do you cancel a task. You have at least 2 options:

    1. Store somewhere a task name and use it to delete the task from a queue.
    2. Store somewhere intention (request) of canceling a task. When the task hits the worker you check the flag and if needed just ignore the task.

    You can use combination of this 2 methods for an edge case when a task has been dispatched to a worker but has not been completed yet. But in most cases it does not worth the effort.

    By the way lots of message queuing systems does not have "task deletion" at all. As Russian saying says "A word is not a bird - if it's gone you can not put it back".