how can i bring data from static websites to HDFS?

What are the other available framework like spring XD, Flume for that? Which one is the best among; please advise steps to bring data.

Solution

Using NUTCH
Using Kafka flume
Using spring xd
scraper import.io
java program by producer consumer

how to set "api-version" dynamically in fs.azure.account.oauth2.msi.endpoint
NoClassDefFoundError: org/apache/parquet/conf/ParquetConfiguration
java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StorageStatistics
Missing PutHDFS Processor in Apache NiFi 2.0.0
Apache Nifi: PutHDFS Processor issue - PutHDFS Failed to write to HDFS java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configurable
how to check which HDFS datanode ip is returned by namenode to spark?
How to use hadoop with laravel 5.2
What is the principle of "code moving to data" rather than data to code?
java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
How to understand the result of yarn queue status
Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
connect to host localhost port 22: Connection refused
Where is yarn.nodemanager.log-dirs in spark?
How to change date format in hive?
Parquet without Hadoop?
Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z unresolved
Hive Always Fails at Mapreduce
Hql query through using YARN APPLICATION ID
Unable to run hadoop application due to NoClassDefFoundError
Hadoop on Windows - "Error JAVA_HOME is incorrectly set."
Ports are not available: listen tcp 0.0.0.0/50070: bind: An attempt was made to access a socket in a way forbidden by its access permissions
Confusion between Operational and Analytical Big Data and on which category Hadoop operates?
Any command to get active namenode for nameservice in hadoop?
Datanode process not running in Hadoop
Python read file as stream from HDFS
What is the purpose of "uber mode" in hadoop?
Change block size of dfs file
Map Reduce Job Failing with OOM [org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster]
Unable to access Hadoop CLI after enabling Kerberos
How to check if Hadoop daemons are running?