hadoopfile-uploadhdfspipelinestreamsets

File is not Loaded Into HDFS from Local Using Streamsets (validated Successfully!)


I just have started using streamsets, and i'm trying to load a text file from local to HDFS. Please note: I'm using Cloudera Manager, here is a view of "core-site.xml":

<property>
 <name>hadoop.ssl.server.conf</name>
 <value>ssl-server.xml</value>
 <final>true</final>
</property>
<property>
 <name>hadoop.ssl.client.conf</name>
 <value>ssl-client.xml</value>
 <final>true</final>
</property>
<property>
 <name>hadoop.proxyuser.sdc.hosts</name>
 <value>*</value>
</property>
<property>
 <name>hadoop.proxyuser.sdc.groups</name>
 <value>*</value>
</property>
</configuration>

The local file is a text file stored in "/home/cloudera/Desktop".

Here is a view of the source (Local) configuration in Streamsets:

Local file configuration in Streamsets

Here is a view of Hadoop fs configuration in Streamsets:

Hadoop fs configuration in Streamsets

It was validated successfully!

Pipline Has Been Validated

After I played the pipline, I'm supposed to find the file in HDFS directory that I specified, especially at "/user/cloudera".

But when I run it the file hasn't been loaded.

I'm sure I missed something, and I couldn't find answer for this. Could you please help!

Thanks,


Solution

  • You need to play the pipeline, not only validate it.