I recently ran into my disk limits on a Jenkins server. I decided to solve this permanently by mounting a larger volume and move the Jenkins home directory to that volume. I used this wiki page as a reference for the move: https://wiki.jenkins.io/display/JENKINS/Administering+Jenkins. I am using init.d to start Jenkins and changed the JENKINS_HOME directory in /etc/default/jenkins (the init script refers to this location to get its defaults).
After rebooting Jenkins I could confirm it was running in the new Home directory. My build works initially (scm checkout->docker build->test->push to docker registry). However, when arriving at the push stage, an error is generated saying that the directory /var/lib/jenkins could not be created. This is the old JENKINS_HOME (as the new JENKINS_HOME is set at /mnt/jenkins). The full stacktrace can be seen below.
What did I miss here? Is there another setting relevant in the move? I am using the docker-commons and docker-pipeline plugins. Any help is highly appreciated.
java.io.IOException: Directory '/var/lib/jenkins' could not be created
at org.apache.commons.io.FileUtils.openOutputStream(FileUtils.java:363)
at org.apache.commons.io.FileUtils.writeStringToFile(FileUtils.java:1928)
at org.apache.commons.io.FileUtils.writeStringToFile(FileUtils.java:1951)
at org.apache.commons.io.FileUtils.writeStringToFile(FileUtils.java:1911)
at org.jenkinsci.plugins.docker.commons.credentials.DockerRegistryToken$1.call(DockerRegistryToken.java:113)
at org.jenkinsci.plugins.docker.commons.credentials.DockerRegistryToken$1.call(DockerRegistryToken.java:82)
at hudson.remoting.LocalChannel.call(LocalChannel.java:45)
at org.jenkinsci.plugins.docker.commons.credentials.DockerRegistryToken.newKeyMaterialFactory(DockerRegistryToken.java:82)
at org.jenkinsci.plugins.docker.commons.credentials.DockerRegistryEndpoint.newKeyMaterialFactory(DockerRegistryEndpoint.java:219)
at org.jenkinsci.plugins.docker.workflow.RegistryEndpointStep$Execution.newKeyMaterialFactory(RegistryEndpointStep.java:65)
at org.jenkinsci.plugins.docker.workflow.AbstractEndpointStepExecution.start(AbstractEndpointStepExecution.java:44)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:229)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:153)
at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:19)
at org.jenkinsci.plugins.docker.workflow.Docker.withRegistry(jar:file:/mnt/jenkins/jenkins/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:37)
at ___cps.transform___(Native Method)
at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
at com.cloudbees.groovy.cps.Next.step(Next.java:83)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:35)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:32)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:32)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:331)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:82)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:243)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:231)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE
TLDR; Solution: Move any credential files and dirs back to the original location OR change the jenkins user home dir [usermod -d /new/dir jenkins] (this is what I have done). Make sure to move the files out of the way first as the usermod command cannot be run on an existing directory.
After some fiddling around I found the issue. Maybe this is helpful for others.
The jenkins users' home directory was still set at /var/lib/jenkins. This is the directory where the credentials for dockerhub are stored. During the job, Jenkins tried to access the credentials on the old location because by default it looks in the jenkins user home directory.