csvhadoophivehdfsflume

How a HDFS directory by year month and day is created?


Following the question in this link, there is another question about the creating the directory on Hadoop HDFS.

I am new to Hadoop/Flume and I have picked up a project which use Flume to save csv data into HDFS. The setting for the Flume sink will be as follows:

contract-snapshot.sinks.hdfs-sink-contract-snapshot.hdfs.path = /dev/wimp/contract-snapshot/year=%Y/month=%n/day=%e/snapshottime=%k%M

With this Flume setting, the corresponding csv file will be saved into the HDFS, under the folder:

"/wimp/contract-snapshot/year=2020/month=6/day=10/snapshottime=1055/contract-snapshot.1591779548475.csv"

I am trying to setup the whole system locally, I have hadoop installed locally on my windows pc, how can I create a directory of "/wimp/contract-snapshot/year=2020/month=6/day=10/snapshottime=1055/" on the local hdfs?

In the cmd terminal, the code:

hadoop fs -mkdir /wimp/contract-snapshot

can create a folder /wimp/contract-snapshot. However the following code does not work in the cmd terminal

hadoop fs -mkdir /wimp/contract-snapshot/year=2020

How to create hdfs directory by year, month, day?


Solution

  • hadoop fs -mkdir "/wimp/contract-snapshot/year=2020"

    Adding quotation solves the problem.