unixhadoopsshscpcopying

scp remote file into hadoop without copying it to edge node

Want to copy a file from a remote server to hadoop without copying to edge node.

1.Per below article we can do it in 2 step by first doing scp to local edge node and then perform hdfs fs command to move from edge node to hdfs https://community.cloudera.com/t5/Support-Questions/Import-data-from-remote-server-to-HDFS/td-p/233148

2.Per below article we can do ssh cat, but we have files like .gz which cannot be CAT putting a remote file into hadoop without copying it to local disk

But I am looking for a 3rd option where we can scp instead of ssh CAT and copy to hadoop without copying to Edge node.

Solution

  • Hadoop doesnt have an SCP upload feature.

    If you want to get files in without an edge node or SSH, then that's what WebHDFS or the NFSGateway offer