I'm using Apache Camel 3.4.10 to upload some files to an SFTP server and, since almost one week, this is not working anymore and below there is the output I get:
2025-05-23 12:02:08,890 WARN [org.apache.camel.component.file.remote.RemoteFileProducer:87] (Camel (tsbulk-sftp-context) thread #1 - file:///export/home/tsbulk_delivery) Writing file failed with: Cannot store file: tsbulk-prod/CH36158/tsbulk_delivery/selection/labsconfig1.xml
2025-05-23 12:02:10,199 WARN [org.apache.camel.component.file.remote.RemoteFileProducer:87] (Camel (tsbulk-sftp-context) thread #1 - file:///export/home/tsbulk_delivery) Writing file failed with: Cannot store file: tsbulk-prod/CH36158/tsbulk_delivery/selection/labsconfig1.xml
2025-05-23 12:02:11,530 WARN [org.apache.camel.component.file.remote.RemoteFileProducer:87] (Camel (tsbulk-sftp-context) thread #1 - file:///export/home/tsbulk_delivery) Writing file failed with: Cannot store file: tsbulk-prod/CH36158/tsbulk_delivery/selection/labsconfig1.xml
2025-05-23 12:02:12,925 WARN [org.apache.camel.component.file.remote.RemoteFileProducer:87] (Camel (tsbulk-sftp-context) thread #1 - file:///export/home/tsbulk_delivery) Writing file failed with: Cannot store file: tsbulk-prod/CH36158/tsbulk_delivery/selection/labsconfig1.xml
2025-05-23 12:02:14,255 WARN [org.apache.camel.component.file.remote.RemoteFileProducer:87] (Camel (tsbulk-sftp-context) thread #1 - file:///export/home/tsbulk_delivery) Writing file failed with: Cannot store file: tsbulk-prod/CH36158/tsbulk_delivery/selection/labsconfig1.xml
2025-05-23 12:02:15,658 WARN [org.apache.camel.component.file.remote.RemoteFileProducer:87] (Camel (tsbulk-sftp-context) thread #1 - file:///export/home/tsbulk_delivery) Writing file failed with: Cannot store file: tsbulk-prod/CH36158/tsbulk_delivery/selection/labsconfig1.xml
2025-05-23 12:02:17,103 WARN [org.apache.camel.component.file.remote.RemoteFileProducer:87] (Camel (tsbulk-sftp-context) thread #1 - file:///export/home/tsbulk_delivery) Writing file failed with: Cannot store file: tsbulk-prod/CH36158/tsbulk_delivery/selection/labsconfig1.xml
2025-05-23 12:02:18,452 WARN [org.apache.camel.component.file.remote.RemoteFileProducer:87] (Camel (tsbulk-sftp-context) thread #1 - file:///export/home/tsbulk_delivery) Writing file failed with: Cannot store file: tsbulk-prod/CH36158/tsbulk_delivery/selection/labsconfig1.xml
2025-05-23 12:02:19,787 WARN [org.apache.camel.component.file.remote.RemoteFileProducer:87] (Camel (tsbulk-sftp-context) thread #1 - file:///export/home/tsbulk_delivery) Writing file failed with: Cannot store file: tsbulk-prod/CH36158/tsbulk_delivery/selection/labsconfig1.xml
2025-05-23 12:02:21,133 WARN [org.apache.camel.component.file.remote.RemoteFileProducer:87] (Camel (tsbulk-sftp-context) thread #1 - file:///export/home/tsbulk_delivery) Writing file failed with: Cannot store file: tsbulk-prod/CH36158/tsbulk_delivery/selection/labsconfig1.xml
2025-05-23 12:02:22,506 WARN [org.apache.camel.component.file.remote.RemoteFileProducer:87] (Camel (tsbulk-sftp-context) thread #1 - file:///export/home/tsbulk_delivery) Writing file failed with: Cannot store file: tsbulk-prod/CH36158/tsbulk_delivery/selection/labsconfig1.xml
2025-05-23 12:02:22,508 ERROR [org.apache.camel.processor.errorhandler.DefaultErrorHandler:205] (Camel (tsbulk-sftp-context) thread #1 - file:///export/home/tsbulk_delivery) Failed delivery for (MessageId: 44C4CB7F36787D0-000000000000001F on ExchangeId: 44C4CB7F36787D0-000000000000001F). Exhausted after delivery attempt: 11 caught: org.apache.camel.component.file.GenericFileOperationFailedException: Cannot store file: tsbulk-prod/CH36158/tsbulk_delivery/selection/labsconfig1.xml
Message History (complete message history is disabled)
---------------------------------------------------------------------------------------------------------------------------------------
RouteId ProcessorId Processor Elapsed (ms)
[selection-files ] [selection-files ] [from[file:///export/home/tsbulk_delivery?antInclude=*%2Fselection%2F*.xml&flat] [ 55166]
...
[selection-files ] [to1 ] [sftp:<SERVER>:6323/tsbulk-prod?preferredAuthentications=publi] [ 0]
Stacktrace
---------------------------------------------------------------------------------------------------------------------------------------
: org.apache.camel.component.file.GenericFileOperationFailedException: Cannot store file: tsbulk-prod/CH36158/tsbulk_delivery/selection/labsconfig1.xml
at org.apache.camel.component.file.remote.SftpOperations.doStoreFile(SftpOperations.java:1061)
at org.apache.camel.component.file.remote.SftpOperations.storeFile(SftpOperations.java:977)
at org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:290)
at org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:173)
at org.apache.camel.component.file.remote.RemoteFileProducer.process(RemoteFileProducer.java:61)
at org.apache.camel.support.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:66)
at org.apache.camel.processor.SendProcessor.lambda$process$2(SendProcessor.java:191)
at org.apache.camel.support.cache.DefaultProducerCache.doInAsyncProducer(DefaultProducerCache.java:327)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:190)
at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$RedeliveryTask.redeliver(RedeliveryErrorHandler.java:895)
at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:193)
at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:64)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:185)
at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:398)
at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:492)
at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:245)
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:206)
at org.apache.camel.support.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:202)
at org.apache.camel.support.ScheduledPollConsumer.run(ScheduledPollConsumer.java:116)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: 3: Permission denied.
at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2830)
at com.jcraft.jsch.ChannelSftp._put(ChannelSftp.java:561)
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:509)
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:464)
at org.apache.camel.component.file.remote.SftpOperations.doStoreFile(SftpOperations.java:1039)
... 25 more
The weird thing is that, if I connect to the SFTP server with WinSCP, I can actually see the files in there being uploaded/updated.
The SFTP endpoint is defined as follows:
protected StringBuilder buildTargetSftpEndpoint() {
final StringBuilder builder = new StringBuilder();
builder.append("sftp:");
builder.append(config.getHost());
builder.append(":");
builder.append(config.getPort());
if(!config.getRootDirectoryName().startsWith("/")) builder.append("/");
builder.append(config.getRootDirectoryName());
builder.append("?preferredAuthentications=publickey");
builder.append("&fastExistsCheck=true");
builder.append("&greedy=true");
builder.append("&throwExceptionOnConnectFailed=true");
builder.append("&initialDelay=0");
builder.append("&stepwise=false");
builder.append("&reconnectDelay=0");
builder.append("&serverAliveCountMax=0");
builder.append("&serverAliveInterval=0");
builder.append("&username=");
builder.append(config.getUsername());
builder.append("&privateKeyFile=");
builder.append(config.getSshPrivateKey());
builder.append("&knownHostsFile=");
builder.append(config.getSshKnownHostsFile());
builder.append("&maximumReconnectAttempts=");
builder.append(maxRetries);
return builder;
}
The file endpoint is defined as follows:
private String buildSourceEndpointUri() {
final StringBuilder builder = new StringBuilder();
builder.append("file:");
if(!getBaseDirectory().startsWith("/")) builder.append("/");
builder.append(getBaseDirectory());
builder.append("?recursive=true");
builder.append("&flatten=false");
builder.append("&startingDirectoryMustExist=true");
builder.append("&noop=true");
builder.append("&idempotent=true");
builder.append("&idempotentKey=${file:name}-${file:modified}");
builder.append("&greedy=true");
builder.append("&initialDelay=0");
builder.append("&antInclude=");
builder.append("*/");
builder.append(selectionSpace.getDir());
builder.append("/*.xml");
return builder.toString();
}
The route is defined as follows:
@PostConstruct
private void setup() {
processor = new FilePathAdjusterProcessor(getBaseDirectory());
}
@Override
public void configure() {
onException(FileNotFoundException.class).continued(true);
errorHandler(defaultErrorHandler().maximumRedeliveries(max(0, config.getMaxRetries())).redeliveryDelay(max(0, config.getRetriesDelayMs())));
configureRoute();
}
@Override
protected StringBuilder buildTargetSftpEndpoint() {
return super.buildTargetSftpEndpoint().append("&disconnect=true");
}
@Override
protected void configureRoute() {
from(buildSourceEndpointUri())
.routeId(routeId)
.process(processor)
.to(buildTargetSftpEndpoint().toString())
.log(logLevel, logMessageTemplate);
}
The processor is
public class FilePathAdjusterProcessor implements Processor {
private final Path baseDirectory;
FilePathAdjusterProcessor(String baseDirectory) {
this.baseDirectory = Paths.get(baseDirectory);
}
@Override
public void process(Exchange exchange) {
Path deliveryFolderName = baseDirectory.getFileName();
Path relativeFilePath = baseDirectory.relativize(exchange.getIn().getBody(File.class).toPath());
Path customerName = relativeFilePath.getName(0);
Path filePathWithoutCustomerName = relativeFilePath.subpath(1, relativeFilePath.getNameCount());
Path extendedPath = customerName.resolve(deliveryFolderName).resolve(filePathWithoutCustomerName);
exchange.getMessage().setHeader(OVERRULE_FILE_NAME, extendedPath);
}
}
The code below checks the same, but using JSch only
private static final int PORT = 6323;
private static final String USERNAME = "tsbulk";
private static final String HOST = "<SERVER>";
private static final String ROOT_DIRECTORY = "/tsbulk-prod";
private static final String PRIVATE_KEY = "<PRIVATE-KEY-PATH>"
public static void main(String[] args) throws JSchException, IOException, SftpException {
Path file = Files.write(createTempFile("foo", "bar"), "test".getBytes(UTF_8));
Path targetPath = Paths.get(ROOT_DIRECTORY, "CH10601", "tsbulk_delivery", "test", "jsch.xml");
JSch jSch = new JSch();
jSch.addIdentity(PRIVATE_KEY);
Session session = jSch.getSession(USERNAME, HOST, PORT);
session.setConfig("StrictHostKeyChecking", "no");
session.setConfig("PreferredAuthentications", "publickey");
session.connect();
ChannelSftp sftp = (ChannelSftp) session.openChannel("sftp");
sftp.connect();
for(int i = 0; i < targetPath.getNameCount() - 1; ++i) {
try {
sftp.cd(targetPath.getName(i).toString());
} catch (SftpException ex) {
sftp.mkdir(targetPath.getName(i).toString());
sftp.cd(targetPath.getName(i).toString());
}
}
sftp.put(file.toString(), targetPath.getFileName().toString());
sftp.rm(targetPath.getFileName().toString());
sftp.disconnect();
session.disconnect();
}
and it works with no issues.
Honestly, I don't know what to look for anymore, especially given the fact that the same exact code is working correctly in the QA environment against the QA SFTP server, but not in PROD anymore and only since a week or so.
In the end, the issue on my side was caused by the SFTP server that didn't release the locks that it creates. As such, given that Camel tries to delete the files first before uploading them, this deletion fails with "Permission denied". In such a scenario, also downloading of the files fails and only by deleting the locks the situation gets again under control.
Given the above, I was able to reproduce the issue also with WinSCP and the RHEL sftp client.