hadooptwitterflumeflume-ng

How to solve 404 error in flume while streaming data from twitter?


I am trying to stream some data from Twitter API using flume. The code did work initially. but now I am getting 404 error:

INFO twitter4j.TwitterStreamImpl: 404: The URI requested is invalid or the resource requested, such as a user, does not exist.

Unknown URL. See Twitter Streaming API documentation at http://dev.twitter.com/pages/streaming_api 

below is my conf file code.

TwitterAgent.sources= Twitter
TwitterAgent.channels= MemChannel
TwitterAgent.sinks=HDFS
TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
TwitterAgent.sources.Twitter.channels=MemChannel

TwitterAgent.sources.Twitter.consumerKey=<code>
TwitterAgent.sources.Twitter.consumerSecret=    <code>
TwitterAgent.sources.Twitter.accessToken=<code>
TwitterAgent.sources.Twitter.accessTokenSecret= <code>

TwitterAgent.sources.Twitter.keywords= hadoop, bigdata

TwitterAgent.sinks.HDFS.channel=MemChannel
TwitterAgent.sinks.HDFS.type=hdfs
TwitterAgent.sinks.HDFS.hdfs.path=hdfs://localhost:8020/user/flume/tweets
TwitterAgent.sinks.HDFS.hdfs.fileType=DataStream
TwitterAgent.sinks.HDFS.hdfs.writeformat=Text
TwitterAgent.sinks.HDFS.hdfs.batchSize=1000
TwitterAgent.sinks.HDFS.hdfs.rollSize=0
TwitterAgent.sinks.HDFS.hdfs.rollCount=10000
TwitterAgent.sinks.HDFS.hdfs.rollInterval=600
TwitterAgent.channels.MemChannel.type=memory
TwitterAgent.channels.MemChannel.capacity=10000
TwitterAgent.channels.MemChannel.transactionCapacity=100

Solution

  • I just synchronized the time of VMware and my windows system manually and the problem got solved.