mesosmesosphere

Mesos task history after restart


I am using Mesos for container orchestration and get task history from Mesos using /task endpoint.

Mesos is running in a 7 nodes cluster and zookeeper is running in a 3 node cluster. I hope, Mesos uses Zookeeper to store the task History. We lost history sometimes when we restart Mesos. Does it store in memory? I am trying to understand what is happening here.

My questions are,

  1. Where does it store task histories?
  2. How can we configure the task history cleanup policy?
  3. Why do we lose complete task history on restarting Mesos?

Solution

  • To answer your questions:

    1. Task history/state for Mesos is stored in memory, and in the replicated_log (details here). The default is set to use the replicated_log, to store state completely in memory without the replicated_log you would have to specify this in your Mesos flags seen here in the configuration page as --registry=in_memory
    2. Most users typically configure task history cleanup by using these three flags (there are more, but these are most common) --max_completed_frameworks=VALUE, --max_completed_tasks_per_framework=VALUE, and --max_unreachable_tasks_per_framework=VALUE as described in the previous document.
    3. Yes, task history for the /tasks endpoint is lost every time a Mesos Master is restarted. However, the /state endpoint will still contain all task status changes over time.

    **Edited to reflect information about the /tasks endpoint, not the /state endpoint.