I am trying to update my ECS cluster by starting a new task but I keep having this message
STOPPED (VolumeError: Agent could not create task's volume resources).
Anybody has an idea of what does it mean? or how can I debug this ?
For this ECS task my launching method was EC2 which means my container use the real ec2 instance to run. So the problem was that those instances were full of dangling docker containers, docker volumes, docker images. To be certain of this I went on the instances (via ssh) and check memory (by using a build in Linux command)
df
the "df" command showed that my disk( /dev/xvda1 ) was used at 85% there was no way i could a single file on this instance
this i clean it by using those commands
docker ps -q --filter "status=exited" | xargs --no-run-if-empty docker rm ;
docker volume ls -qf dangling=true | xargs -r docker volume rm;
docker rmi $(docker images -f "dangling=true" -q)
and then the rate went from 85% to 15% and everything just went fine from there
if you need more help check this article about the same issue AWS ECS “no space left on device”