javatwitterjava-streamtwitter4j

Twitter4j getting blank or null for getCreatedAt and getCountry


I am streaming some Twitter data using Twitter4J and I am able to generate the tweet text. The tweet is consumed by a TwitterConsumer class that is saving the text to Kafka. So far everything works as expected. My problem is that I am trying to get the date and time when together with the location (country code is enough) of the tweet. I understand that the location may be null but in my case, both the date/time and location are always empty or null. I must be doing something wrong.

Sample output;

[Twitter4J Async Dispatcher[0]] INFO uom_bigdataAss1.producer.TwitterProducer - Created on: [Thread-3] INFO uom_bigdataAss1.consumer.TwitterConsumer - Location: No Location [Thread-3] INFO uom_bigdataAss1.consumer.TwitterConsumer - delta plus

  • a variant of concern. more transmissible, could even escape vaccines. we will continue to learn more about… [Thread-3] INFO uom_bigdataAss1.consumer.TwitterConsumer - Created at: [Thread-3] INFO uom_bigdataAss1.consumer.TwitterConsumer - User: [Thread-3] INFO uom_bigdataAss1.consumer.TwitterConsumer - User Name: [Thread-3] INFO uom_bigdataAss1.consumer.TwitterConsumer - _________________

This is the code I am using to produce the Twitter streaming side;


    public class TwitterProducer {
    
        final Logger logger = LoggerFactory.getLogger(TwitterProducer.class);       
        
        Thread twitterConsumerThread = null;
              
        //Queue for tweets
        private LinkedBlockingQueue<Status> queue;
        //stream of tweets
        private TwitterStream twitterStream;
            
        private BlockingQueue<String> msgQueue = new LinkedBlockingQueue<>(100);
    
        public static void main(String[] args) {
            new TwitterProducer().run();
        }
       
        // Twitter Client
        public void setTwitterStream(BlockingQueue<String> msgQueue){
            /** Setting up a connection   */
            ConfigurationBuilder cb = new ConfigurationBuilder();
            cb.setDebugEnabled(true)
                    .setOAuthConsumerKey(TwitterConfig.CONSUMER_KEYS)
                    .setOAuthConsumerSecret(TwitterConfig.CONSUMER_SECRETS)
                    .setOAuthAccessToken(TwitterConfig.TOKEN)
                    .setOAuthAccessTokenSecret(TwitterConfig.SECRET);
    
            this.twitterStream = new TwitterStreamFactory(cb.build()).getInstance();
            
        }
        
    
        private void run(){
            logger.info("Setting up");
    
            // 1. Twitter Stream
            setTwitterStream(msgQueue);
            
            this.queue = new LinkedBlockingQueue<Status>();
    
            StatusListener listener = new StatusListener() {
                
                @Override
                public void onStatus(Status status) {               
                    try {
                        queue.offer(status, 5000, TimeUnit.MILLISECONDS);
                        logger.info("Created on:", status.getCreatedAt().getTime());
                        Place place = status.getPlace();  
                        if (place != null ) {
                            logger.info("Created on:", place.getCountry());
                        }
                    } catch (InterruptedException e) {
                        // TODO Auto-generated catch block
                        logger.error("BlockinQueue Full", e);
                    }
    
                }
    
                @Override
                public void onDeletionNotice(StatusDeletionNotice sdn) {
                }
    
                @Override
                public void onTrackLimitationNotice(int i) {
                }
    
                @Override
                public void onScrubGeo(long l, long l1) {
                }
    
                @Override
                public void onException(Exception e) {
                    e.printStackTrace();
                }
    
                @Override
                public void onStallWarning(StallWarning warning) {
                }
            };
            
            twitterStream.addListener(listener);
                    
            FilterQuery query = new FilterQuery();
            String keywords[] = {"covid", "vaccine"};
            query.track(keywords);
            query.language("en");        
            twitterStream.filter(query);
    
            // Shutdown Hook
            Runtime.getRuntime().addShutdownHook(new Thread( () -> {
                logger.info("Application is now stopping!");
                twitterStream.shutdown();
                //logger.info("Closing Producer");
                //producer.close();
                logger.info("Closing twitterConsumer");
                this.twitterConsumerThread.interrupt();
                logger.info("Finished closing");
                logger.info("\n Application End");
            }));
            
            logger.info("Running Twitter Consumer");
            
            TwitterConsumer twitterConsumer = new TwitterConsumer(this.queue);
            this.twitterConsumerThread = new Thread(twitterConsumer);
            twitterConsumerThread.start();
            
        }
    
    
    }

    

Any ideas what could be happening/cause?


Solution

  • Change your log statement like

    logger.info("Created on: {}", status.getCreatedAt().getTime()); 
    

    and

    logger.info("Created on: {}", place.getCountry());
    

    you are missing {}-placeholders

    with above changes, log will be similar to

    Created on: 1624430231000
    Created on: India
    

    check http://www.slf4j.org/manual.html#typical_usage

    Also, not all status will have country information with them.