I am streaming some Twitter data using Twitter4J and I am able to generate the tweet text. The tweet is consumed by a TwitterConsumer class that is saving the text to Kafka. So far everything works as expected. My problem is that I am trying to get the date and time when together with the location (country code is enough) of the tweet. I understand that the location may be null but in my case, both the date/time and location are always empty or null. I must be doing something wrong.
Sample output;
[Twitter4J Async Dispatcher[0]] INFO uom_bigdataAss1.producer.TwitterProducer - Created on: [Thread-3] INFO uom_bigdataAss1.consumer.TwitterConsumer - Location: No Location [Thread-3] INFO uom_bigdataAss1.consumer.TwitterConsumer - delta plus
- a variant of concern. more transmissible, could even escape vaccines. we will continue to learn more about… [Thread-3] INFO uom_bigdataAss1.consumer.TwitterConsumer - Created at: [Thread-3] INFO uom_bigdataAss1.consumer.TwitterConsumer - User: [Thread-3] INFO uom_bigdataAss1.consumer.TwitterConsumer - User Name: [Thread-3] INFO uom_bigdataAss1.consumer.TwitterConsumer - _________________
This is the code I am using to produce the Twitter streaming side;
public class TwitterProducer {
final Logger logger = LoggerFactory.getLogger(TwitterProducer.class);
Thread twitterConsumerThread = null;
//Queue for tweets
private LinkedBlockingQueue<Status> queue;
//stream of tweets
private TwitterStream twitterStream;
private BlockingQueue<String> msgQueue = new LinkedBlockingQueue<>(100);
public static void main(String[] args) {
new TwitterProducer().run();
}
// Twitter Client
public void setTwitterStream(BlockingQueue<String> msgQueue){
/** Setting up a connection */
ConfigurationBuilder cb = new ConfigurationBuilder();
cb.setDebugEnabled(true)
.setOAuthConsumerKey(TwitterConfig.CONSUMER_KEYS)
.setOAuthConsumerSecret(TwitterConfig.CONSUMER_SECRETS)
.setOAuthAccessToken(TwitterConfig.TOKEN)
.setOAuthAccessTokenSecret(TwitterConfig.SECRET);
this.twitterStream = new TwitterStreamFactory(cb.build()).getInstance();
}
private void run(){
logger.info("Setting up");
// 1. Twitter Stream
setTwitterStream(msgQueue);
this.queue = new LinkedBlockingQueue<Status>();
StatusListener listener = new StatusListener() {
@Override
public void onStatus(Status status) {
try {
queue.offer(status, 5000, TimeUnit.MILLISECONDS);
logger.info("Created on:", status.getCreatedAt().getTime());
Place place = status.getPlace();
if (place != null ) {
logger.info("Created on:", place.getCountry());
}
} catch (InterruptedException e) {
// TODO Auto-generated catch block
logger.error("BlockinQueue Full", e);
}
}
@Override
public void onDeletionNotice(StatusDeletionNotice sdn) {
}
@Override
public void onTrackLimitationNotice(int i) {
}
@Override
public void onScrubGeo(long l, long l1) {
}
@Override
public void onException(Exception e) {
e.printStackTrace();
}
@Override
public void onStallWarning(StallWarning warning) {
}
};
twitterStream.addListener(listener);
FilterQuery query = new FilterQuery();
String keywords[] = {"covid", "vaccine"};
query.track(keywords);
query.language("en");
twitterStream.filter(query);
// Shutdown Hook
Runtime.getRuntime().addShutdownHook(new Thread( () -> {
logger.info("Application is now stopping!");
twitterStream.shutdown();
//logger.info("Closing Producer");
//producer.close();
logger.info("Closing twitterConsumer");
this.twitterConsumerThread.interrupt();
logger.info("Finished closing");
logger.info("\n Application End");
}));
logger.info("Running Twitter Consumer");
TwitterConsumer twitterConsumer = new TwitterConsumer(this.queue);
this.twitterConsumerThread = new Thread(twitterConsumer);
twitterConsumerThread.start();
}
}
Any ideas what could be happening/cause?
Change your log statement like
logger.info("Created on: {}", status.getCreatedAt().getTime());
and
logger.info("Created on: {}", place.getCountry());
you are missing {}-placeholders
with above changes, log will be similar to
Created on: 1624430231000
Created on: India
check http://www.slf4j.org/manual.html#typical_usage
Also, not all status will have country information with them.