This document discusses how to stream Twitter data using Kafka. It explains that Twitter's Hosebird client allows consuming the Twitter Streaming API in near real-time. The document then provides code to create a Kafka producer that connects to Twitter and sends tweets to a Kafka topic. It also includes code for a Kafka consumer that reads tweets from that topic and prints them. Running the producer streams tweets from Twitter to Kafka, while the consumer prints the tweets to the console in real-time.

1. Streaming Twitter data using Kafka
In this blog we will be discussing about how to stream twitter data using kafka. Before going
through this blog, we hope that you have pre installed kafka and zookeeper services in your system.
Please go through this blog to Install kafka
Please go through this blog to Install zookeeper
Twitter provides Hosebird client (hbc), a robust Java HTTP library for consuming Twitter’s Streaming
API.
Hosebird is the server implementation of the Twitter Streaming API. The Streaming API allows clients to
receive Tweets in near real-time. Various resources allow filtered, sampled or full access to some or all
Tweets. Every Twitter account has access to the Streaming API and any developer can build
applications today. Hosebird also powers the recently announced User Streams feature that streams all
events related to a given user to drive desktop Twitter clients.
We will start kafka and zookeeper services
Start zookeeper server by moving into the bin folder of zookeeper installed directory by using the
command
zkServer.sh start
Start kafka server by moving into the bin folder of kafka installed directory by using the command
./kafka-server-start.sh ../config/server.properties

2. In kafka we use two classes called Producers and Consumers.
The Producer class is used to create new messages for a specific Topic and optional Partition.
Twitter Producer Class
package kafka;
import java.util.*;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;
import com.google.common.collect.Lists;
import com.twitter.hbc.ClientBuilder;
import com.twitter.hbc.core.Client;
import com.twitter.hbc.core.Constants;
import com.twitter.hbc.core.endpoint.StatusesFilterEndpoint;
import com.twitter.hbc.core.processor.StringDelimitedProcessor;
import com.twitter.hbc.httpclient.auth.Authentication;
import com.twitter.hbc.httpclient.auth.OAuth1;
public class TwitterKafkaProducer {
private static final String topic = "hadoop";
public static void run() throws InterruptedException {
Properties properties = new Properties();
properties.put("metadata.broker.list", "localhost:9092");
properties.put("serializer.class", "kafka.serializer.StringEncoder");

3. properties.put("client.id","camus");
ProducerConfig producerConfig = new ProducerConfig(properties);
kafka.javaapi.producer.Producer<String, String> producer = new
kafka.javaapi.producer.Producer<String, String>(
producerConfig);
BlockingQueue<String> queue = new LinkedBlockingQueue<String>(100000);
StatusesFilterEndpoint endpoint = new StatusesFilterEndpoint();
endpoint.trackTerms(Lists.newArrayList("twitterapi",
"#AAPSweep"));
String consumerKey= TwitterSourceConstant.CONSUMER_KEY_KEY;
String consumerSecret=TwitterSourceConstant.CONSUMER_SECRET_KEY;
String accessToken=TwitterSourceConstant.ACCESS_TOKEN_KEY;
String accessTokenSecret=TwitterSourceConstant.ACCESS_TOKEN_SECRET_KEY;
Authentication auth = new OAuth1(consumerKey, consumerSecret,
accessToken,
accessTokenSecret);
Client client = new ClientBuilder().hosts(Constants.STREAM_HOST)
.endpoint(endpoint).authentication(auth)
.processor(new StringDelimitedProcessor(queue)).build();
client.connect();
for (int msgRead = 0; msgRead < 1000; msgRead++) {
KeyedMessage<String, String> message = null;
try {
message = new KeyedMessage<String, String>(topic, queue.take());
} catch (InterruptedException e) {
//e.printStackTrace();
System.out.println("Stream ended");
}
producer.send(message);
}
producer.close();
client.stop();
}
public static void main(String[] args) {
try {
TwitterKafkaProducer.run();
} catch (InterruptedException e) {
System.out.println(e);
}
}
}
Twitter authorization is done through
consumerKey,consumerSecret,accessToken,accessTokenSecret.
So we are passing them through a class called TwitterSourceConstant
public class TwitterSourceConstant {
public static final String CONSUMER_KEY_KEY = "xxxxxxxxxxxxxxxxxxxxxxxx";

4. public static final String CONSUMER_SECRET_KEY = "xxxxxxxxxxxxxxxxxxxxxxxxx";
public static final String ACCESS_TOKEN_KEY = "xxxxxxxxxxxxxxxxxxxxxxxxxxx";
public static final String ACCESS_TOKEN_SECRET_KEY = "xxxxxxxxxxxxxxxxxxxxxx";
In the private static final String topic = "hadoop"; of producer class we will pass our
topic to stream the particular data from twitter.
So first we need to start this producer class to start streaming data from twitter.
Now we will write a consumer class to print the streamed tweets. The consumer class is as follows
package kafka;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import kafka.consumer.Consumer;
import kafka.consumer.ConsumerConfig;
import kafka.consumer.ConsumerIterator;
import kafka.consumer.KafkaStream;
import kafka.javaapi.consumer.ConsumerConnector;
public class KafkaConsumer {
private ConsumerConnector consumerConnector = null;
private final String topic = "twitter-topic1";
public void initialize() {
Properties props = new Properties();
props.put("zookeeper.connect", "localhost:2181");
props.put("group.id", "testgroup");
props.put("zookeeper.session.timeout.ms", "400");
props.put("zookeeper.sync.time.ms", "300");
props.put("auto.commit.interval.ms", "100");
ConsumerConfig conConfig = new ConsumerConfig(props);
consumerConnector = Consumer.createJavaConsumerConnector(conConfig);
}
public void consume() {
//Key = topic name, Value = No. of threads for topic
Map<String, Integer> topicCount = new HashMap<String, Integer>();
topicCount.put(topic, new Integer(1));
//ConsumerConnector creates the message stream for each topic
Map<String, List<KafkaStream<byte[], byte[]>>> consumerStreams =
consumerConnector.createMessageStreams(topicCount);
// Get Kafka stream for topic 'mytopic'
List<KafkaStream<byte[], byte[]>> kStreamList =
consumerStreams.get(topic);
// Iterate stream using ConsumerIterator
for (final KafkaStream<byte[], byte[]> kStreams : kStreamList) {
ConsumerIterator<byte[], byte[]> consumerIte =
kStreams.iterator();
while (consumerIte.hasNext())
System.out.println("Message consumed from topic[" +
topic + "] : " +

5. new
String(consumerIte.next().message()));
}
//Shutdown the consumer connector
if (consumerConnector != null) consumerConnector.shutdown();
}
public static void main(String[] args) throws InterruptedException {
KafkaConsumer kafkaConsumer = new KafkaConsumer();
// Configure Kafka consumer
kafkaConsumer.initialize();
// Start consumption
kafkaConsumer.consume();
}
}
When we run this above consumer class it will print all the tweets collected in that moment.
We have build this project through maven and the pom.xml file is as follows
pom.xml
<?xml version="1.0"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.twitter</groupId>
<artifactId>hbc-example</artifactId>
<version>2.2.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>Hosebird Client Examples</name>
<properties>
<git.dir>${project.basedir}/../.git</git.dir>
<!-- this makes maven-tools not bump us to snapshot versions -->
<stabilized>true</stabilized>
<!-- Fill these in via https://dev.twitter.com/apps -->
<consumer.key>TODO</consumer.key>
<consumer.secret>TODO</consumer.secret>
<access.token>TODO</access.token>
<access.token.secret>TODO</access.token.secret>
</properties>
<dependencies>
<dependency>
<groupId>com.twitter</groupId>
<artifactId>hbc-twitter4j</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.7.2</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.10</artifactId>
<version>0.8.2.1</version>
</dependency>
</dependencies>

6. <build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-deploy-plugin</artifactId>
<version>2.7</version>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.2.1</version>
</plugin>
</plugins>
</build>
</project>
We need to run the producer and consumer programs in eclipse. So first we need to run the producer
to stream the tweets from twitter.
The eclipse console of the Producer is as shown in the screen shot.
Now let's run consumer class of kafka. The console of the consumer with the collected tweets is as
shown in the below screen shot.

7. Here we have collected the tweets related to hadoop topic which we have set in the producer class.
We can also check for the topics on which kafka is running now by using the command
./kafka-topics.sh --zookeeper localhost:2181 –list
We can check the consumer console also parallely to check the tweets which are being collected in
the real time by using the below command:
./kafka-console-consumer.sh --zookeeper localhost:2181 --topic "hadoop" –from-beginning
Below is the screen shot of the consumer console with the tweets.

8. So this is how we collect streaming data from twitter using kafka.
We hope this blog helped you in understanding how to collect streaming data from twitter using
kafka. Keep visiting our site www.acadgild.com for more updates on Big data and other
technologies.