I have a file containing a bunch of folder names in it. After using getfile
to get the data from the file, I'm splitting the data into strings in different queue and each queue contains the name of the folder in the attribute of flowfile.
Now I want to pass those queues to getsftp
processor where I'll use $FolderName
attribute in the Remote Path
property of the processor to fetch data based on it.
Here the problem is that when I'm trying to connect any queue's success or failure to getsftp
processor, it is not allowing me to connect to it. Here I researched and came to know that getsftp
can't be subsequent processor of any other processor and only it can be parent of processors.
Is there any way to make getsftp
a subsequent processor of any other processor? If no then is there any way where I can store the values of queues and then use that values in the Remote Path
property of the processor to fetch data based on it?
No processor in NiFi can read a SFTP $foldername
dynamically from a flow-file attribute and read all the file contents under the folder. You have to do some workaround - I have listed 2 workarounds as 2 separate answers.
I always use the combination of ListSFTP
and FetchSFTP
(which accepts an incoming connection). For your use case (you don't need ListSFTP
), just connect the queue from your screenshot to the FetchSFTP
processor and you should be good to go.
Note: FetchSFTP can only read contents of an SFTP file. So this will only work if you know the SFTP $path
and $filename
variables (i.e., complete name of the SFTP file to be read)
I have explained the usage of ListSFTP
and FetchSFTP
processors as a separate answer.