Linked Data &Semantic WebTechnologyDevelopment ofTwitter ApplicationsPart 8. Streaming APIDr. Myungjin Lee
Linked Data & Semantic Web TechnologyStreaming API• Overview– low latency access to Twitters global stream of Tweetdata– A...
Linked Data & Semantic Web TechnologyDifferences between Streaming and REST• REST API• Streaming API3
Linked Data & Semantic Web TechnologyConnecting to a Streaming Endpoint• Authentication• Connecting– form a HTTP request a...
Linked Data & Semantic Web TechnologyConnecting to a Streaming Endpoint• Stalls– If 90 seconds pass with no data received,...
Linked Data & Semantic Web TechnologyConnecting to a Streaming Endpoint• Connection Churn– churn: repeatedly opening and c...
Linked Data & Semantic Web TechnologyConnecting to a Streaming Endpoint• HTTP Error Codes7Status Text Description200 Succe...
Linked Data & Semantic Web TechnologyStreaming API Request Parameters• delimited– be used on all streaming endpoints– deli...
Linked Data & Semantic Web TechnologyStreaming API Request Parameters• stall_warnings– be used on all streaming endpoints–...
Linked Data & Semantic Web TechnologyStreaming API Request Parameters• follow– available on POST statuses/filter– a comma-...
Linked Data & Semantic Web TechnologyStreaming API Request Parameters• track– available on POST statuses/filter– what Twee...
Linked Data & Semantic Web TechnologyStreaming API Request Parameters• locations– available on POST statuses/filter– to sp...
Linked Data & Semantic Web TechnologyStreaming API Request Parameters• with– available on GET users and GET site– to contr...
Linked Data & Semantic Web TechnologyProcessing Streaming Data• Parsing responses– The body of a streaming API response co...
Linked Data & Semantic Web TechnologyProcessing Streaming Data• Transfer-Encoding: chunked– Most streaming connections wil...
Linked Data & Semantic Web TechnologyProcessing Streaming Data• Message ordering– Messages from the Streaming API are not ...
Linked Data & Semantic Web TechnologyPublic Streams• Overview– samples of the public data flowing through Twitter• Endpoin...
Linked Data & Semantic Web TechnologyPublic Stream Messages Types• Blank lines– some messages may be blank lines which ser...
Linked Data & Semantic Web TechnologyPublic Stream Messages Types• Limit notices (limit)– to indicate that a filtered stre...
Linked Data & Semantic Web TechnologyPublic Stream Messages Types• Disconnect messages (disconnect)– to deliver a message ...
Linked Data & Semantic Web TechnologyGET statuses/sample• sample Streaming API– Returns a small random sample of all publi...
Linked Data & Semantic Web TechnologyTwitter4J Classes for sample Streaming• TwitterStreamFactory Class– A factory class f...
Linked Data & Semantic Web TechnologyTwitter4J Classes for sample Streaming• StatusListener Interface– Methods• void onDel...
Linked Data & Semantic Web TechnologyExample of sample Streaming API1. import twitter4j.StallWarning;2. import twitter4j.S...
Linked Data & Semantic Web TechnologyExample of sample Streaming API1. import twitter4j.*;2. public class PrintSampleStrea...
Linked Data & Semantic Web TechnologyGET statuses/firehose• firehose Streaming API– This endpoint requires special permiss...
Linked Data & Semantic Web TechnologyTwitter4J Classes for firehose Streaming• Twitter4J Classes for firehose Streaming– T...
Linked Data & Semantic Web TechnologyPOST statuses/filter• filter Streaming API– Returns public statuses that match one or...
Linked Data & Semantic Web TechnologyTwitter4J Classes for filter Streaming• TwitterStream Interface– Methods• void filter...
Linked Data & Semantic Web TechnologyExample of filter Streaming API1. import twitter4j.*;2. public final class PrintFilte...
Linked Data & Semantic Web TechnologyUser Streams• Overview– a stream of data and events specific to theauthenticated user...
Linked Data & Semantic Web TechnologyUser Stream Messages Types• Friends lists (Friends)– a preamble which is a list of th...
Linked Data & Semantic Web TechnologyUser Stream Messages Types• Events (event)– notifications about non-Tweet events33Des...
Linked Data & Semantic Web TechnologyGET user• user Streaming API– streams messages for a single user, as described inUser...
Linked Data & Semantic Web TechnologyTwitter4J Classes for user Streaming• TwitterStream Interface– Methods• void user()– ...
Linked Data & Semantic Web TechnologyExample of user Streaming API1. import twitter4j.*;2. public class UserStreamListener...
Linked Data & Semantic Web TechnologyExample of user Streaming API39. public void onFavorite(User source, User target, Sta...
Linked Data & Semantic Web TechnologyExample of user Streaming API76. public void onUserListUnsubscription(User subscriber...
Linked Data & Semantic Web TechnologyExample of user Streaming API1. import twitter4j.*;2. public final class PrintUserStr...
Linked Data & Semantic Web TechnologySite Streams• Overview– services to receive real-time updates for a large number of u...
Linked Data & Semantic Web TechnologySite Stream Messages Types• Envelopes (for_user)– an additional wrapper to identify t...
Linked Data & Semantic Web TechnologyTwitter4J Classes for site Streaming• TwitterStream Interface– Methods• StreamControl...
Linked Data & Semantic Web TechnologyExample of site Streaming API1. import twitter4j.*;2. public class SiteStreamsListene...
Linked Data & Semantic Web TechnologyExample of site Streaming API38. public void onUnfollow(long forUser, User source, Us...
Linked Data & Semantic Web TechnologyExample of site Streaming API75. public void onUserListCreation(long forUser, User li...
Linked Data & Semantic Web TechnologyExample of site Streaming API1. import twitter4j.*;2. public class PrintSiteStream {3...
Upcoming SlideShare
Loading in...5
×

Development of Twitter Application #8 - Streaming API

876
-1

Published on

This series of slides describes how to develop a twitter application.
This slide shows how to search tweets using Twitter Search RESTful Open API and how to implement it using Twitter4J.

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
876
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Development of Twitter Application #8 - Streaming API

  1. 1. Linked Data &Semantic WebTechnologyDevelopment ofTwitter ApplicationsPart 8. Streaming APIDr. Myungjin Lee
  2. 2. Linked Data & Semantic Web TechnologyStreaming API• Overview– low latency access to Twitters global stream of Tweetdata– A proper implementation of a streaming client will bepushed messages indicating Tweets and other eventshave occurred.• Streaming Endpoints2Public streamsStreams of the public data flowing through Twitter.Suitable for following specific users or topics, and datamining.User streamsSingle-user streams, containing roughly all of the datacorresponding with a single users view of Twitter.Site streamsThe multi-user version of user streams. Site streams areintended for servers which must connect to Twitter onbehalf of many users.
  3. 3. Linked Data & Semantic Web TechnologyDifferences between Streaming and REST• REST API• Streaming API3
  4. 4. Linked Data & Semantic Web TechnologyConnecting to a Streaming Endpoint• Authentication• Connecting– form a HTTP request and consume the resultingstream– holding the connection open indefinitely, barringserver-side error, excessive client-side lag, networkhiccups, routine server maintenance or duplicatelogins• Disconnections– too many connections with the same credentials– to stop reading data– reading data too slowly– a streaming server is restarted– twitter’s network configuration changes4Auth Type Supported APIs DescriptionBasic Public StreamsRequests include the credentials of avalid Twitter account.OAuthPublic StreamsUser StreamsSite StreamsRequests must be authorized according tothe OAuth specification.
  5. 5. Linked Data & Semantic Web TechnologyConnecting to a Streaming Endpoint• Stalls– If 90 seconds pass with no data received, includingnewlines, disconnect and reconnect immediately.– The Streaming API will send a keep-alive newlineevery 30 seconds to prevent your application fromtiming out the connection.• Reconnecting– reconnecting attempts according to the type of errorexperienced• TCP/IP level network errors– reconnect by 250ms each attempt, up to 16 seconds• HTTP errors– start with a 5 second wait, doubling each attempt, up to320 seconds• HTTP 420 errors– start with a 1 minute wait and double each attempt5
  6. 6. Linked Data & Semantic Web TechnologyConnecting to a Streaming Endpoint• Connection Churn– churn: repeatedly opening and closing a connection– keep your connections as stable and long-lived aspossible• Rate Limiting– Client which do not implement backoff and attempt toreconnect as often as possible will have theirconnections rate limited for a small number ofminutes.– Rate limited clients will receive HTTP 420 responsesfor all connection requests.– If your client is rate limited frequently, it is possiblethat your IP will be blocked from accessing Twitter foran indeterminate period of time.6
  7. 7. Linked Data & Semantic Web TechnologyConnecting to a Streaming Endpoint• HTTP Error Codes7Status Text Description200 Success Self evident.401 UnauthorizedHTTP authentication failed due to either:•Invalid basic auth credentials, or an invalid OAuth request.•Out-of-sync timestamp in your OAuth request (the response body will indicatethis).•Too many incorrect passwords entered or other login rate limiting.403 Forbidden The connecting account is not permitted to access this endpoint.404 Unknown There is nothing at this URL, which means the resource does not exist.406NotAcceptableAt least one request parameter is invalid. For example, the filter endpoint returnsthis status if:•The track keyword is too long or too short.•An invalid bounding box is specified.•Neither the track nor follow parameter are specified.•The follow user ID is not valid.413 Too LongA parameter list is too long. For example, the filter endpoint returns this status if:•More track values are sent than the user is allowed to use.•More bounding boxes are sent than the user is allowed to use.•More follow user IDs are sent than the user is allowed to follow.416RangeUnacceptableFor example, an endpoint returns this status if:•A count parameter is specified but the user does not have access to use the countparameter.•A count parameter is specified which is outside of the maximum/minimumallowable values.420 Rate LimitedThe client has connected too frequently. For example, an endpoint returns this statusif:•A client makes too many login attempts in a short period of time.•Too many copies of an application attempt to authenticate with the samecredentials.503ServiceUnavailableA streaming server is temporarily overloaded. Attempt to make another connection,keeping in mind the connection attempt rate limiting and possible DNS caching inyour client.
  8. 8. Linked Data & Semantic Web TechnologyStreaming API Request Parameters• delimited– be used on all streaming endpoints– delimited=length• statuses should be delimited in the stream by a length,in bytes, a newline, and the status text8
  9. 9. Linked Data & Semantic Web TechnologyStreaming API Request Parameters• stall_warnings– be used on all streaming endpoints– stall_warnings=true• periodic messages to be delivered every 5 minutes if theclient is in danger of being disconnected• filter_level– be used on all streaming endpoints– to set the minimum value of the Tweet attributerequired to be included in the stream– one of none, low, and medium– the default value is none.• language– be used on all streaming endpoints– to return Tweets that have been detected as beingwritten in the specified languages9
  10. 10. Linked Data & Semantic Web TechnologyStreaming API Request Parameters• follow– available on POST statuses/filter– a comma-separated list of user IDs– to indicate the users whose Tweets should bedelivered on the stream– the stream contain:• Tweets created by the user.• Tweets which are retweeted by the user.• Replies to any Tweet created by the user.• Retweets of any Tweet created by the user.• Manual replies, created without pressing a reply button(e.g. “@twitterapi I agree”).– the stream does not contain:• Tweets mentioning the user (e.g. “Hello @twitterapi!”).• Manual Retweets created without pressing a Retweetbutton (e.g. “RT @twitterapi The API is great”).• Tweets by protected users.10
  11. 11. Linked Data & Semantic Web TechnologyStreaming API Request Parameters• track– available on POST statuses/filter– what Tweets will be delivered on the stream• a comma-separated list of phrases that may be one or more termsseparated by spaces• a phrase will match if all of the terms in the phrase are present inthe Tweet, regardless of order and ignoring case• commas as logical ORs and white spaces as logical ANDs• each phrase must be between 1 and 60 bytes• exact matching of phrases is not supported• punctuation and special characters will be considered part of theterm they are adjacent to– to check the text attribute of the Tweet, expanded_url anddisplay_url for links and media, text for hashtags, andscreen_name for user mentions11
  12. 12. Linked Data & Semantic Web TechnologyStreaming API Request Parameters• locations– available on POST statuses/filter– to specify a set of bounding boxes to filter Tweets– a comma-separated list of longitude, latitude pairs– conditions• •If the coordinates field is populated, the values there willbe tested against the bounding box.• •If coordinates is empty but place is populated, the regiondefined in place is checked .• If none of the rules listed above match, the Tweet does notmatch the location query.• count– to attempt to backfill missed messages which occurredduring the disconnect periot– an integer from 1 to 150000 or from -1 to -150000– use of this parameter should be carefully considered, ashigh values increase the chance of a subsequentdisconnect12
  13. 13. Linked Data & Semantic Web TechnologyStreaming API Request Parameters• with– available on GET users and GET site– to controls the types of messages delivered to Userand Site Streams clients– the default is with=user– The value with=followings adds messages fromaccounts the user follows, equivalent to the usershome timeline.• replies– available on GET users and GET site– replies=all• to have mimics of all Tweets returned in a streamingconnection13
  14. 14. Linked Data & Semantic Web TechnologyProcessing Streaming Data• Parsing responses– The body of a streaming API response consists of aseries of newline-delimited messages.• "newline" is considered to be rn (in hex, 0x0D 0x0A)• "message" is a JSON encoded data structure or a blankline• JSON data– The individual messages streamed by this API areJSON encoded.– the attributes of a JSON-encoded object are unordered• Missing counts– In very rare cases, the Streaming API may elect todeliver an incomplete Tweet field.14
  15. 15. Linked Data & Semantic Web TechnologyProcessing Streaming Data• Transfer-Encoding: chunked– Most streaming connections will be encoded usingchunked transfer encoding, as a Transfer-Encoding: chunked HTTP header in theresponse.• Delimited messages– Each message will be preceded by a stringrepresentation of a base-10 integer indicating thelength of the message in bytes.• Falling behind– A way to track whether your client is falling behind isto compare the timestamp of the Tweets you receivewith the current time.– If the difference between the timestamps increasesover time, then the client is not processing Tweets asfast as they are being delivered.15
  16. 16. Linked Data & Semantic Web TechnologyProcessing Streaming Data• Message ordering– Messages from the Streaming API are not delivered insorted order.– Delete messages may be delivered before the originalTweet so implementations should be able to replay alocal cache of unrecognized deletes.• Duplicate messages– Duplicate messages may be delivered soimplementations should be tolerant of receiving aTweet more than once.• Gzip compression– Gzip compression may reduce the bandwidth neededto process a stream to as small as 1/5th the size of anuncompressed stream.16
  17. 17. Linked Data & Semantic Web TechnologyPublic Streams• Overview– samples of the public data flowing through Twitter• Endpoints– GET statuses/sample– GET statuses/firehose– POST statuses/filter• Using the streaming APIs– connect to the endpoinsts– process data returned by the streaming APIs– There are some streaming messages which are notreturned by the REST APIs. See streaming messagetypes for information.• Connections– each account may create only one standing connectionto the public endpoints– connecting to a public stream more than once with thesame account credentials will cause the oldestconnection to be disconnected17
  18. 18. Linked Data & Semantic Web TechnologyPublic Stream Messages Types• Blank lines– some messages may be blank lines which serve as"keep-alive" signals to prevent clients have stalled andclosing the connection.• Status deletion notices (delete)– These messages indicate that a given Tweet has beendeleted.• Location deletion notices (scrub_geo)– to indicate that geolocated data must be stripped froma range of Tweets18
  19. 19. Linked Data & Semantic Web TechnologyPublic Stream Messages Types• Limit notices (limit)– to indicate that a filtered stream has matched moreTweets than its current rate limit allows to bedelivered– Limit notices contain a total count of the number ofundelivered Tweets.• Stall warnings (warning)– When connected to a stream using the stall_warningsparameter, you may receive status notices indicatingthe current health of the connection.19
  20. 20. Linked Data & Semantic Web TechnologyPublic Stream Messages Types• Disconnect messages (disconnect)– to deliver a message indicating why a stream wasclosed– status codes20Code Name Description1 Shutdown The feed was shutdown (possibly a machine restart)2DuplicatestreamThe same endpoint was connected too many times.3 Control request Control streams was used to close a stream (applies to sitestreams).4 Stall The client was reading too slowly and was disconnected by the server.5 Normal The client appeared to have initiated a disconnect.6 Token revoked An oauth token was revoked for a user (applies to site and userstreams).7 Admin logoutThe same credentials were used to connect a new stream and the oldest wasdisconnected.8 Reserved for internal use. Will not be delivered to external clients.9Max messagelimitThe stream connected with a negative count parameter and was disconnectedafter all backfill was delivered.10StreamexceptionAn internal issue disconnected the stream.11 Broker stall An internal issue disconnected the stream.12 Shed loadThe host the stream was connected to became overloaded and streams weredisconnected to balance load. Reconnect as usual.
  21. 21. Linked Data & Semantic Web TechnologyGET statuses/sample• sample Streaming API– Returns a small random sample of all public statuses.The Tweets returned by the default access level are thesame, so if two different clients connect to thisendpoint, they will see the same Tweets.• Resource URL– https://stream.twitter.com/1.1/statuses/sample.json• Parameters• Other Information– Rate Limited? Yes– Authentication: Requires user context– Response Object: Tweets21delimitedoptionalSpecifies whether messages should be length-delimited.stall_warningsoptionalSpecifies whether stall warnings should be delivered.
  22. 22. Linked Data & Semantic Web TechnologyTwitter4J Classes for sample Streaming• TwitterStreamFactory Class– A factory class for TwitterFactory.– An instance of this class is completely thread safe andcan be re-used and used concurrently.– Methods• TwitterStream getInstance()– Returns a instance associated with the configurationbound to this factory.• TwitterStream getInstance(AccessToken accessToken)– Returns a OAuth Authenticated instance.– consumer key and consumer Secret must be provided bytwitter4j.properties, or system properties.• TwitterStream Interface– Methods• void addListener(StatusListener listener)• void sample()– Starts listening on random sample of all public statuses.• void shutdown()– Shuts down internal dispatcher thread shared by allTwitterStream instances.22
  23. 23. Linked Data & Semantic Web TechnologyTwitter4J Classes for sample Streaming• StatusListener Interface– Methods• void onDeletionNotice(StatusDeletionNotice statusDeletionNotice)– Called upon deletionNotice notices.• void onScrubGeo(long userId, long upToStatusId)– Called upon location deletion messages.• void onStallWarning(StallWarning warning)– Called when receiving stall warnings.• void onStatus(Status status)• void onTrackLimitationNotice(int numberOfLimitedStatuses)– This notice will be sent each time a limited stream becomesunlimited.23
  24. 24. Linked Data & Semantic Web TechnologyExample of sample Streaming API1. import twitter4j.StallWarning;2. import twitter4j.Status;3. import twitter4j.StatusDeletionNotice;4. import twitter4j.StatusListener;5. public class StatusListenerImpl implements StatusListener {6. @Override7. public void onStatus(Status status) {8. System.out.println("@" + status.getUser().getScreenName() + " - "9. + status.getText());10. }11. @Override12. public void onDeletionNotice(StatusDeletionNotice statusDeletionNotice) {13. System.out.println("Got a status deletion notice id:"14. + statusDeletionNotice.getStatusId());15. }16. @Override17. public void onTrackLimitationNotice(int numberOfLimitedStatuses) {18. System.out.println("Got track limitation notice:"19. + numberOfLimitedStatuses);20. }21. @Override22. public void onScrubGeo(long userId, long upToStatusId) {23. System.out.println("Got scrub_geo event userId:" + userId24. + " upToStatusId:" + upToStatusId);25. }26. @Override27. public void onStallWarning(StallWarning warning) {28. System.out.println("Got stall warning:" + warning);29. }30. @Override31. public void onException(Exception ex) {32. ex.printStackTrace();33. }34. }24
  25. 25. Linked Data & Semantic Web TechnologyExample of sample Streaming API1. import twitter4j.*;2. public class PrintSampleStream {3. public static void main(String[] args) throws TwitterException {4. TwitterStream twitterStream = new TwitterStreamFactory().getInstance();5. twitterStream.setOAuthConsumer(TwitterAccessToken.consumerKey,6. TwitterAccessToken.consumerSecret);7. twitterStream.setOAuthAccessToken(TwitterAccessToken.loadAccessToken());8. StatusListenerImpl listener = new StatusListenerImpl();9. twitterStream.addListener(listener);10. twitterStream.sample();11. }12. }25
  26. 26. Linked Data & Semantic Web TechnologyGET statuses/firehose• firehose Streaming API– This endpoint requires special permission to access.– Returns all public statuses.– Few applications require this level of access.– Creative use of a combination of other resources andvarious access levels can satisfy nearly everyapplication use case.• Resource URL– https://stream.twitter.com/1.1/statuses/firehose.json• Parameters• Other Information– Rate Limited? Yes– Authentication: Requires user context– Response Object: Tweets26countoptionalThe number of messages to backfill.delimitedoptionalSpecifies whether messages should be length-delimited.stall_warningsoptionalSpecifies whether stall warnings should be delivered.
  27. 27. Linked Data & Semantic Web TechnologyTwitter4J Classes for firehose Streaming• Twitter4J Classes for firehose Streaming– TwitterStream Interface• Methods– void firehose(int count)» Starts listening on all public statuses.• Example of firehose Streaming API1. import twitter4j.*;2. public class PrintFirehoseStream {3. public static void main(String[] args) throws TwitterException {4. TwitterStream twitterStream = new TwitterStreamFactory().getInstance();5. twitterStream.setOAuthConsumer(TwitterAccessToken.consumerKey,6. TwitterAccessToken.consumerSecret);7. twitterStream.setOAuthAccessToken(TwitterAccessToken.loadAccessToken());8. StatusListenerImpl listener = new StatusListenerImpl();9. twitterStream.addListener(listener);10. twitterStream.firehose(0);11. }12. }27
  28. 28. Linked Data & Semantic Web TechnologyPOST statuses/filter• filter Streaming API– Returns public statuses that match one or more filterpredicates.– Both GET and POST requests are supported, but GETrequests with too many parameters may cause the requestto be rejected for excessive URL length.• Resource URL– https://stream.twitter.com/1.1/statuses/filter.json• Parameters• Other Information– Rate Limited? Yes– Authentication: Requires user context– Response Object: Tweets28followA comma separated list of user IDs, indicating the users to return statuses for in thestream.track Keywords to track. Phrases of keywords are specified by a comma-separated list.locations Specifies a set of bounding boxes to track.delimitedoptionalSpecifies whether messages should be length-delimited.stall_warningsoptionalSpecifies whether stall warnings should be delivered.
  29. 29. Linked Data & Semantic Web TechnologyTwitter4J Classes for filter Streaming• TwitterStream Interface– Methods• void filter(FilterQuery query)– Start consuming public statuses that match one or morefilter predicates.• FilterQuery Class– Methods• FilterQuery count(int count)– Sets count• FilterQuery follow(long[] follow)– Sets follow• FilterQuery locations(double[][] locations)– Sets locations• FilterQuery track(java.lang.String[] track)– Sets track29
  30. 30. Linked Data & Semantic Web TechnologyExample of filter Streaming API1. import twitter4j.*;2. public final class PrintFilterStream {3. public static void main(String[] args) throws TwitterException {4. StatusListenerImpl listener = new StatusListenerImpl();5. TwitterStream twitterStream = new TwitterStreamFactory().getInstance();6. twitterStream.setOAuthConsumer(TwitterAccessToken.consumerKey,7. TwitterAccessToken.consumerSecret);8. twitterStream.setOAuthAccessToken(TwitterAccessToken.loadAccessToken());9. twitterStream.addListener(listener);10. FilterQuery fq = new FilterQuery();11. // long[] follow = { 22931893, 16227629, 464020027, 299014098, 40203602,12. // 121858259, 631481692, 173523502, 64972381, 357617589, 77689885 };13. // fq.follow(follow);14. // String[] track = { "psy", "#psy", "@psy_oppa", "gentleman",15. // "#gentleman", "@ygent_official" };16. // fq.track(track);17. // double[][] locations = { { -74, 40 }, { -73, 41 } }; // New York City18. // fq.locations(locations);19. twitterStream.filter(fq);20. }21. }30
  31. 31. Linked Data & Semantic Web TechnologyUser Streams• Overview– a stream of data and events specific to theauthenticated user• Endpoint– GET user• Connections– limited to only a few simultaneous User Streamsconnections per OAuth application, regardless of IP– to handle the HTTP 420 error code that indicates thatthe account has been logging in to often• User streams messages– Types of messages• some streaming messages which are not returned by theREST APIs31
  32. 32. Linked Data & Semantic Web TechnologyUser Stream Messages Types• Friends lists (Friends)– a preamble which is a list of the user’s friends beforestarting regular message delivery• Too many follows (warning)– warning message when your application connects onbehalf of a user who follows more than 10,000accounts32
  33. 33. Linked Data & Semantic Web TechnologyUser Stream Messages Types• Events (event)– notifications about non-Tweet events33Description Event Name Source TargetTargetObjectUser blocks someone block Current user Blocked user NullUser removes a block unblock Current userUnblockeduserNullUser favorites a Tweet favorite Current user Tweet author TweetUsers Tweet is favorited favorite Favoriting user Current user TweetUser unfavorites a Tweet unfavorite Current user Tweet author TweetUsers Tweet is unfavorited unfavoriteUnfavoritinguserCurrent user TweetUser follows someone follow Current user Followed user NullUser is followed follow Following user Current user NullUser unfollows someone unfollow Current user Followed user NullUser creates a list list_created Current user Current user ListUser deletes a list list_destroyed Current user Current user ListUser edits a list list_updated Current user Current user ListUser adds someone to a list list_member_added Current user Added user ListUser is added to a list list_member_added Adding user Current user ListUser removes someonefrom a listlist_member_removed Current user Removed user ListUser is removed from a list list_member_removed Removing user Current user ListUser subscribes to a list list_user_subscribed Current user List owner ListUsers list is subscribed to list_user_subscribedSubscribinguserCurrent user ListUser unsubscribes from alistlist_user_unsubscribed Current user List owner ListUsers list is unsubscribedfromlist_user_unsubscribedUnsubscribinguserCurrent user ListUser updates their profile user_update Current user Current user Null
  34. 34. Linked Data & Semantic Web TechnologyGET user• user Streaming API– streams messages for a single user, as described inUser streams• Resource URL– https://userstream.twitter.com/1.1/user.json• Parameters• Other Information– Rate Limited? Yes– Authentication: Requires user context34delimitedoptionalSpecifies whether messages should be length-delimited.stall_warningsoptionalSpecifies whether stall warnings should be delivered.withoptionalSpecifies whether to return information for just the authenticating user, or includemessages from accounts the user follows.repliesoptionalSpecifies whether to return additional @replies.trackoptionalIncludes additional Tweets matching the specified keywords. Phrases of keywordsare specified by a comma-separated list.locationsoptionalIncludes additional Tweets falling within the specified bounding boxes.
  35. 35. Linked Data & Semantic Web TechnologyTwitter4J Classes for user Streaming• TwitterStream Interface– Methods• void user()– User Streams provides real-time updates of all data needed to update adesktop application display.• void user(java.lang.String[] track)– User Streams provides real-time updates of all data needed to update adesktop application display.• UserStreamListener Interface– Methods• void onBlock(User source, User blockedUser)• void onDeletionNotice(long directMessageId, long userId)• void onDirectMessage(DirectMessage directMessage)• void onFavorite(User source, User target, Status favoritedStatus)• void onFollow(User source, User followedUser)• void onFriendList(long[] friendIds)• void onUnblock(User source, User unblockedUser)• void onUnfavorite(User source, User target,Status unfavoritedStatus)• void onUserListCreation(User listOwner, UserList list)• void onUserListDeletion(User listOwner, UserList list)• void onUserListMemberAddition(User addedMember,User listOwner, UserList list)• void onUserListMemberDeletion(User deletedMember,User listOwner, UserList list)• void onUserListSubscription(User subscriber,User listOwner, UserList list)• void onUserListUnsubscription(User subscriber,User listOwner, UserList list)• void onUserListUpdate(User listOwner, UserList list)• void onUserProfileUpdate(User updatedUser)35
  36. 36. Linked Data & Semantic Web TechnologyExample of user Streaming API1. import twitter4j.*;2. public class UserStreamListenerImpl implements UserStreamListener {3. public void onStatus(Status status) {4. System.out.println("onStatus @" + status.getUser().getScreenName()5. + " - " + status.getText());6. }7. public void onDeletionNotice(StatusDeletionNotice statusDeletionNotice) {8. System.out.println("Got a status deletion notice id:"9. + statusDeletionNotice.getStatusId());10. }11. public void onDeletionNotice(long directMessageId, long userId) {12. System.out.println("Got a direct message deletion notice id:"13. + directMessageId);14. }15. public void onTrackLimitationNotice(int numberOfLimitedStatuses) {16. System.out.println("Got a track limitation notice:"17. + numberOfLimitedStatuses);18. }19. public void onScrubGeo(long userId, long upToStatusId) {20. System.out.println("Got scrub_geo event userId:" + userId21. + " upToStatusId:" + upToStatusId);22. }23. public void onStallWarning(StallWarning warning) {24. System.out.println("Got stall warning:" + warning);25. }26. public void onFriendList(long[] friendIds) {27. System.out.print("onFriendList");28. for (long friendId : friendIds) {29. System.out.print(" " + friendId);30. }31. System.out.println();32. }36
  37. 37. Linked Data & Semantic Web TechnologyExample of user Streaming API39. public void onFavorite(User source, User target, Status favoritedStatus) {40. System.out.println("onFavorite source:@" + source.getScreenName()41. + " target:@" + target.getScreenName() + " @"42. + favoritedStatus.getUser().getScreenName() + " - "43. + favoritedStatus.getText());44. }45. public void onUnfavorite(User source, User target, Status unfavoritedStatus) {46. System.out.println("onUnFavorite source:@" + source.getScreenName()47. + " target:@" + target.getScreenName() + " @"48. + unfavoritedStatus.getUser().getScreenName() + " - "49. + unfavoritedStatus.getText());50. }51. public void onFollow(User source, User followedUser) {52. System.out.println("onFollow source:@" + source.getScreenName()53. + " target:@" + followedUser.getScreenName());54. }55. public void onDirectMessage(DirectMessage directMessage) {56. System.out.println("onDirectMessage text:" + directMessage.getText());57. }58. public void onUserListMemberAddition(User addedMember, User listOwner,59. UserList list) {60. System.out.println("onUserListMemberAddition added member:@"61. + addedMember.getScreenName() + " listOwner:@"62. + listOwner.getScreenName() + " list:" + list.getName());63. }64. public void onUserListMemberDeletion(User deletedMember, User listOwner,65. UserList list) {66. System.out.println("onUserListMemberDeleted deleted member:@"67. + deletedMember.getScreenName() + " listOwner:@"68. + listOwner.getScreenName() + " list:" + list.getName());69. }70. public void onUserListSubscription(User subscriber, User listOwner,71. UserList list) {72. System.out.println("onUserListSubscribed subscriber:@"73. + subscriber.getScreenName() + " listOwner:@"74. + listOwner.getScreenName() + " list:" + list.getName());75. }37
  38. 38. Linked Data & Semantic Web TechnologyExample of user Streaming API76. public void onUserListUnsubscription(User subscriber, User listOwner,77. UserList list) {78. System.out.println("onUserListUnsubscribed subscriber:@"79. + subscriber.getScreenName() + " listOwner:@"80. + listOwner.getScreenName() + " list:" + list.getName());81. }82. public void onUserListCreation(User listOwner, UserList list) {83. System.out.println("onUserListCreated listOwner:@"84. + listOwner.getScreenName() + " list:" + list.getName());85. }86. public void onUserListUpdate(User listOwner, UserList list) {87. System.out.println("onUserListUpdated listOwner:@"88. + listOwner.getScreenName() + " list:" + list.getName());89. }90. public void onUserListDeletion(User listOwner, UserList list) {91. System.out.println("onUserListDestroyed listOwner:@"92. + listOwner.getScreenName() + " list:" + list.getName());93. }94. public void onUserProfileUpdate(User updatedUser) {95. System.out.println("onUserProfileUpdated user:@"96. + updatedUser.getScreenName());97. }98. public void onBlock(User source, User blockedUser) {99. System.out.println("onBlock source:@" + source.getScreenName()100. + " target:@" + blockedUser.getScreenName());101. }102. public void onUnblock(User source, User unblockedUser) {103. System.out.println("onUnblock source:@" + source.getScreenName()104. + " target:@" + unblockedUser.getScreenName());105. }106. public void onException(Exception ex) {107. ex.printStackTrace();108. System.out.println("onException:" + ex.getMessage());109. }110. }38
  39. 39. Linked Data & Semantic Web TechnologyExample of user Streaming API1. import twitter4j.*;2. public final class PrintUserStream {3. public static void main(String[] args) throws TwitterException {4. TwitterStream twitterStream = new TwitterStreamFactory().getInstance();5. twitterStream.setOAuthConsumer(TwitterAccessToken.consumerKey,6. TwitterAccessToken.consumerSecret);7. twitterStream.setOAuthAccessToken(TwitterAccessToken.loadAccessToken());8. UserStreamListenerImpl listener = new UserStreamListenerImpl();9. twitterStream.addListener(listener);10. twitterStream.user();11. }12. }39
  40. 40. Linked Data & Semantic Web TechnologySite Streams• Overview– services to receive real-time updates for a large number of users– for any user who has granted OAuth access to your application– Site Streams is currently in a limited beta. Access is restricted towhitelisted accounts.• Endpoint– GET site• Types of messages– the same types of messages streamed to User Stream• Protected data– to provide both public and private data in a single stream– meticulous about what data they display• Connecting– to only receive messages for users who have approved OAuthaccess to your application– to consume significant (> 1 Mbit/sec) bandwith• Limits– must be respected during the beta period• limit the follow count to 100 users per stream while connecting• open no more than 25 new connections per second40
  41. 41. Linked Data & Semantic Web TechnologySite Stream Messages Types• Envelopes (for_user)– an additional wrapper to identify the target of eachmessage except for blank keep-alive lines• Control messages (control)– a control message which may be used to modify theSite Streams connection without reconnecting for newSite Streams41
  42. 42. Linked Data & Semantic Web TechnologyTwitter4J Classes for site Streaming• TwitterStream Interface– Methods• StreamController site(boolean withFollowings, long[] follow)– Site Streams, a new feature on the Streaming API, is now available for beta testing.• SiteStreamsListener Interface– Methods• void onBlock(long forUser, User source, User blockedUser)• void onDeletionNotice(long forUser, long directMessageId, long userId)• void onDeletionNotice(long forUser,StatusDeletionNotice statusDeletionNotice)• void onDirectMessage(long forUser, DirectMessage directMessage)• void onDisconnectionNotice(java.lang.String line)• void onException(java.lang.Exception ex)• void onFavorite(long forUser, User source, User target, Status favoritedStatus)• void onFollow(long forUser, User source, User followedUser)• void onFriendList(long forUser, long[] friendIds)• void onStatus(long forUser, Status status)• void onUnblock(long forUser, User source, User unblockedUser)• void onUnfavorite(long forUser, User source, User target,Status unfavoritedStatus)• void onUnfollow(long forUser, User source, User unfollowedUser)• void onUserListCreation(long forUser, User listOwner, UserList list)• void onUserListDeletion(long forUser, User listOwner, UserList list)• void onUserListMemberAddition(long forUser, User addedMember,User listOwner, UserList list)• void onUserListMemberDeletion(long forUser, User deletedMember,• User listOwner, UserList list)• void onUserListSubscription(long forUser, User subscriber, User listOwner,UserList list)• void onUserListUnsubscription(long forUser, User subscriber, User listOwner,UserList list)• void onUserListUpdate(long forUser, User listOwner, UserList list)• void onUserProfileUpdate(long forUser, User updatedUser)42
  43. 43. Linked Data & Semantic Web TechnologyExample of site Streaming API1. import twitter4j.*;2. public class SiteStreamsListenerImpl implements SiteStreamsListener {3. public void onStatus(long forUser, Status status) {4. System.out.println("onStatus for_user:" + forUser + " @"5. + status.getUser().getScreenName() + " - " + status.getText());6. }7. public void onDeletionNotice(long forUser,8. StatusDeletionNotice statusDeletionNotice) {9. System.out.println("Got a status deletion notice for_user:" + forUser10. + " id:" + statusDeletionNotice.getStatusId());11. }12. public void onFriendList(long forUser, long[] friendIds) {13. System.out.print("onFriendList for_user:" + forUser);14. for (long friendId : friendIds) {15. System.out.print(" " + friendId);16. }17. System.out.println();18. }19. public void onFavorite(long forUser, User source, User target,20. Status favoritedStatus) {21. System.out.println("onFavorite for_user:" + forUser + " source:@"22. + source.getScreenName() + " target:@" + target.getScreenName()23. + " @" + favoritedStatus.getUser().getScreenName() + " - "24. + favoritedStatus.getText());25. }26. public void onUnfavorite(long forUser, User source, User target,27. Status unfavoritedStatus) {28. System.out.println("onUnFavorite for_user:" + forUser + " source:@"29. + source.getScreenName() + " target:@" + target.getScreenName()30. + " @" + unfavoritedStatus.getUser().getScreenName() + " - "31. + unfavoritedStatus.getText());32. }33. public void onFollow(long forUser, User source, User followedUser) {34. System.out.println("onFollow for_user:" + forUser + " source:@"35. + source.getScreenName() + " target:@"36. + followedUser.getScreenName());37. }43
  44. 44. Linked Data & Semantic Web TechnologyExample of site Streaming API38. public void onUnfollow(long forUser, User source, User followedUser) {39. System.out.println("onUnfollow for_user:" + forUser + " source:@"40. + source.getScreenName() + " target:@"41. + followedUser.getScreenName());42. }43. public void onDirectMessage(long forUser, DirectMessage directMessage) {44. System.out.println("onDirectMessage for_user:" + forUser + " text:"45. + directMessage.getText());46. }47. public void onDeletionNotice(long forUser, long directMessageId, long userId) {48. System.out.println("Got a direct message deletion notice for_user:"49. + forUser + " id:" + directMessageId);50. }51. public void onUserListMemberAddition(long forUser, User addedMember,52. User listOwner, UserList list) {53. System.out.println("onUserListMemberAddition for_user:" + forUser54. + " member:@" + addedMember.getScreenName() + " listOwner:@"55. + listOwner.getScreenName() + " list:" + list.getName());56. }57. public void onUserListMemberDeletion(long forUser, User deletedMember,58. User listOwner, UserList list) {59. System.out.println("onUserListMemberDeletion for_user:" + forUser60. + " member:@" + deletedMember.getScreenName() + " listOwner:@"61. + listOwner.getScreenName() + " list:" + list.getName());62. }63. public void onUserListSubscription(long forUser, User subscriber,64. User listOwner, UserList list) {65. System.out.println("onUserListSubscribed for_user:" + forUser66. + " subscriber:@" + subscriber.getScreenName() + " listOwner:@"67. + listOwner.getScreenName() + " list:" + list.getName());68. }69. public void onUserListUnsubscription(long forUser, User subscriber,70. User listOwner, UserList list) {71. System.out.println("onUserListUnsubscribed for_user:" + forUser72. + " subscriber:@" + subscriber.getScreenName() + " listOwner:@"73. + listOwner.getScreenName() + " list:" + list.getName());74. }44
  45. 45. Linked Data & Semantic Web TechnologyExample of site Streaming API75. public void onUserListCreation(long forUser, User listOwner, UserList list) {76. System.out.println("onUserListCreated for_user:" + forUser77. + " listOwner:@" + listOwner.getScreenName() + " list:"78. + list.getName());79. }80. public void onUserListUpdate(long forUser, User listOwner, UserList list) {81. System.out.println("onUserListUpdated for_user:" + forUser82. + " listOwner:@" + listOwner.getScreenName() + " list:"83. + list.getName());84. }85. public void onUserListDeletion(long forUser, User listOwner, UserList list) {86. System.out.println("onUserListDestroyed for_user:" + forUser87. + " listOwner:@" + listOwner.getScreenName() + " list:"88. + list.getName());89. }90. public void onUserProfileUpdate(long forUser, User updatedUser) {91. System.out.println("onUserProfileUpdated for_user:" + forUser92. + " user:@" + updatedUser.getScreenName());93. }94. public void onBlock(long forUser, User source, User blockedUser) {95. System.out.println("onBlock for_user:" + forUser + " source:@"96. + source.getScreenName() + " target:@"97. + blockedUser.getScreenName());98. }99. public void onUnblock(long forUser, User source, User unblockedUser) {100. System.out.println("onUnblock for_user:" + forUser + " source:@"101. + source.getScreenName() + " target:@"102. + unblockedUser.getScreenName());103. }104. public void onDisconnectionNotice(String line) {105. System.out.println("onDisconnectionNotice:" + line);106. }107. public void onException(Exception ex) {108. ex.printStackTrace();109. System.out.println("onException:" + ex.getMessage());110. }111. }45
  46. 46. Linked Data & Semantic Web TechnologyExample of site Streaming API1. import twitter4j.*;2. public class PrintSiteStream {3. public static void main(String[] args) throws TwitterException {4. TwitterStream twitterStream = new TwitterStreamFactory().getInstance();5. twitterStream.setOAuthConsumer(TwitterAccessToken.consumerKey,6. TwitterAccessToken.consumerSecret);7. twitterStream.setOAuthAccessToken(TwitterAccessToken.loadAccessToken());8. SiteStreamsListenerImpl listener = new SiteStreamsListenerImpl();9. twitterStream.addListener(listener);10. long[] followArray = {52401019, 1476571, 822659, 20606823, 18304822};11. twitterStream.site(true, followArray);12. }13. }46

×