how can i bring data from static websites to HDFS?

What are the other available framework like spring XD, Flume for that? Which one is the best among; please advise steps to bring data.

Solution

Using NUTCH
Using Kafka flume
Using spring xd
scraper import.io
java program by producer consumer

Why do I need to source bash_profile every time
Apache Spark: Get number of records per partition
Unable to exit Hive
can Configuration.set be used in the Mapper?
Loading Files in UDF
Error: `callbackHandler` may not be null when connecting to HDFS using Kerberos in Jakarta EE
how to tune out of memory exception spark
Can't connect from Spark to S3 - AmazonS3Exception Status Code: 400
How to delete and update a record in Hive
What is Google's Dremel? How is it different from Mapreduce?
how to set "api-version" dynamically in fs.azure.account.oauth2.msi.endpoint
NoClassDefFoundError: org/apache/parquet/conf/ParquetConfiguration
java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StorageStatistics
Missing PutHDFS Processor in Apache NiFi 2.0.0
Apache Nifi: PutHDFS Processor issue - PutHDFS Failed to write to HDFS java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configurable
how to check which HDFS datanode ip is returned by namenode to spark?
How to use hadoop with laravel 5.2
java.lang.UnsupportedOperationException: 'posix:permissions'
What is the principle of "code moving to data" rather than data to code?
java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
How to understand the result of yarn queue status
Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
connect to host localhost port 22: Connection refused
Where is yarn.nodemanager.log-dirs in spark?
How to change date format in hive?
Parquet without Hadoop?
Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z unresolved
Hive Always Fails at Mapreduce
Hql query through using YARN APPLICATION ID
Unable to run hadoop application due to NoClassDefFoundError