genn.ai v2
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

genn.ai v2

  • 1,048 views
Uploaded on

The realtime processing tool based on Storm and Kafka.

The realtime processing tool based on Storm and Kafka.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,048
On Slideshare
1,037
From Embeds
11
Number of Embeds
2

Actions

Shares
Downloads
2
Comments
0
Likes
5

Embeds 11

https://twitter.com 8
http://www.linkedin.com 3

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. genn.ai (源内) Realtime processing for everyone Takeshi Nakano @tf0054 1
  • 2. Who am I? Takeshi NAKANO  Senior Researcher / Architect.  Co-authered Getting started with Solr in Japanese.  Co-authered Hadoop Hacks in Japanese.  Writing Getting started with Kafka in Japanese. http://genn.ai/ 2
  • 3. Agenda  What is genn.ai?  Let s use genn.ai  Realtime processing  Download & get your key  Usecases  Define a schema and a filter  Make it real! http://genn.ai/ 3
  • 4. A usecase of genn.ai Customer A Web Sites じゃらん Customer B Customer C Marketer Analyst Realtime Analysis Platform Client (Hotels) 4
  • 5. Visual editor for genn.ai
  • 6. Structure of genn.ai MongoDB MetaStore genn.ai UI User account Tuple schema Query design view Query Gungnir Client on finagle Gungnir Cosole Query Gungnir Thrift Server Gungnir on finagle Thrift Server Storm nimbus Topology Kafka User account Tuple schema JSON Tuple Gungnir Rest Server Gungnir Gungnir on finagle Rest Server Rest Server Tuple Kafka Kafka broker Kafka broker broker Tuple Storm Storm supervisor Storm supervisor supervisor MongoD B HDFS Tuple Topic http://genn.ai/ 6
  • 7. Preparing for starting. http://dev.genn.ai/
  • 8. Preparing for starting. http://dev.genn.ai/ > cd gungnir-client > bin/gungnir -u tf0054 -p 2e2232073109 Picked up _JAVA_OPTIONS: -Dfile.encoding=UTF-8 2013/10/14 22:34:58 com.twitter.finagle.Init$ apply 情報: Finagle version 6.5.1 (rev=59..9cde41) built at 20130626-111057 Gungnir server connected... ec2-xxx.amazonaws.com/54.238.99.212:9290 Welcome tf0054 gungnir>
  • 9. Define your schema! CREATE TUPLE userAction ( userId STRING, hotelId STRING ); CREATE TUPLE commitAction ( userId STRING, hotelId STRING, checkin TIMESTAMP('yyyy-MM-dd'), nights INT ); STRING             Java type string TINYINT            Java type byte SMALLINT          Java type short INT                 Java type int BIGINT              Java type long FLOAT              Java type float DOUBLE            Java type double TIMESTAMP        long value LIST MAP STRUCT
  • 10. Define your filter! (1/4) FROM userAction AS ua, commitAction AS ca USING kafka_spout() EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user') ; From EMIT
  • 11. Define your filter! (2/4) FROM userAction AS ua, commitAction AS ca USING kafka_spout() JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image USING mongo_fetch('getusers', 'userInfo') JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo USING mongo_fetch('hotelscom', 'hotel') EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user') ; From JOIN (user) JOIN (hotel) EMIT
  • 12. Define your filter! (3/4) FROM userAction AS ua, commitAction AS ca USING kafka_spout() JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image USING mongo_fetch('getusers', 'userInfo') JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo USING mongo_fetch('hotelscom', 'hotel') INTO stream_0 ; FROM stream_0 EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user') ; From JOIN (user) JOIN (hotel) stream_0 EMIT (user)
  • 13. Define your filter! (4/4) FROM userAction AS ua, commitAction AS ca USING kafka_spout() JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image USING mongo_fetch('getusers', 'userInfo') JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo USING mongo_fetch('hotelscom', 'hotel') INTO stream_0 ; FROM stream_0 EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user') ; FROM stream_0 BEGIN GROUP BY hotelId EACH count() AS count, * EMIT hotelId, hotelname, photo, count USING kafka_emit('${TOPOLOGY_ID}_hotel') END GROUP ; EMIT (user) From JOIN JOIN EACH sum(nights) EACH avg(nights) stream_0 GROUP BY EMIT (hotel)
  • 14. Register your filter! and ... SUBMIT TOPOLOGY; DESC TOPOLOGY; SHOW TOPOLOGIES; STOP TOPOLOGY xxx; gungnir> SUBMIT TOPOLOGY; OK gungnir> DESC TOPOLOGY; { "id":"52373363e4b0a525adadb4f2", "explain":" ... ", "status":"RUNNING", "owner":"tf0054@gmail.com", "createTime":"2013-09-16T16:35:47.432Z", "summary":{ ... } } gungnir> STOP TOPOLOGY 52373363e4b0a525adadb4f2 ; DROP TOPOLOGY xxx; gungnir> DELETE TOPOLOGY 52373363e4b0a525adadb4f2 ;
  • 15. Testing it! TRACK userAction {"userId":"siniida","hotelId":"226979"}; TRACK commitAction {"userId":"siniida","hotelId":"226979","checkin":"2013-10-18","nights":1}; *** We are changing this name because “TRACK” isn’t fit to its function! (advised in GOTO berlin 2013. Thx.) *** The function of this command is to make and post a event to gennai. gungnir> DESC TOPOLOGY; { "id":"52373363e4b0a525adadb4f2", "explain":" ... ", "status":"RUNNING", "owner":"tf0054@gmail.com", ... "hotelId" : "128767" "hotelId" : "356564" "hotelId" : "226979" "hotelId" : "325073" "hotelId" : "357090" "hotelId" : "126942" "hotelId" : "310294" ... USING kafka_emit('${TOPOLOGY_ID}_user') "hotelId" : "367746" "hotelId" : "232896" > bin/kafka-consumer.sh 52373363e4b0a525adadb4f2_user ; "hotelId" : "220231" or "hotelId" : "163175" > bin/kafka-consumer.sh 52373363e4b0a525adadb4f2_hotel; "hotelId" : "367856" "hotelId" : "360863" "hotelId" : "119747"
  • 16. Knowing http tuple receiver gungnir> DESC USER; { "id":"525e67f8e4b0e7cfca98d040", "name":"siniida", "password":"CmGOdN2UU/Xtmk0wRCZKzDy/BN65LAHPCp2+1KMDOOM=", "createTime":"2013-10-16T10:18:32.849Z" } gungnir> > curl -v -H "Content-type: application/json" -X POST -d '{"userId":"tf0054","hotelId":"226979"}' http://dev.genn.ai:9191/gungnir/v1.0/track/525e67f8e4b0e7cfca98d040/userAction
  • 17. Design of genn.ai Built on Storm  Converts  The  .. properly (e.g. building AST) Good Parts (e.g. Supporting parallelism setting)
  • 18. The future of genn.ai genn.ai wants friends!  genn.ai codes will be open-sourced in the future.  We are looking for co-challenging partners for improving genn.ai.  You can use genn.ai on your web sites completely for free (infrastructure etc..) if we can share the dream together. Please feel free to contact us!  http://genn.ai/  tf0054@gmail.com
  • 19. Thanks! http://genn.ai/ 19
  • 20. 20