Your SlideShare is downloading. ×
genn.ai v2
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

genn.ai v2

795

Published on

The realtime processing tool based on Storm and Kafka.

The realtime processing tool based on Storm and Kafka.

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
795
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
3
Comments
0
Likes
5
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. genn.ai (源内) Realtime processing for everyone Takeshi Nakano @tf0054 1
  • 2. Who am I? Takeshi NAKANO  Senior Researcher / Architect.  Co-authered Getting started with Solr in Japanese.  Co-authered Hadoop Hacks in Japanese.  Writing Getting started with Kafka in Japanese. http://genn.ai/ 2
  • 3. Agenda  What is genn.ai?  Let s use genn.ai  Realtime processing  Download & get your key  Usecases  Define a schema and a filter  Make it real! http://genn.ai/ 3
  • 4. A usecase of genn.ai Customer A Web Sites じゃらん Customer B Customer C Marketer Analyst Realtime Analysis Platform Client (Hotels) 4
  • 5. Visual editor for genn.ai
  • 6. Structure of genn.ai MongoDB MetaStore genn.ai UI User account Tuple schema Query design view Query Gungnir Client on finagle Gungnir Cosole Query Gungnir Thrift Server Gungnir on finagle Thrift Server Storm nimbus Topology Kafka User account Tuple schema JSON Tuple Gungnir Rest Server Gungnir Gungnir on finagle Rest Server Rest Server Tuple Kafka Kafka broker Kafka broker broker Tuple Storm Storm supervisor Storm supervisor supervisor MongoD B HDFS Tuple Topic http://genn.ai/ 6
  • 7. Preparing for starting. http://dev.genn.ai/
  • 8. Preparing for starting. http://dev.genn.ai/ > cd gungnir-client > bin/gungnir -u tf0054 -p 2e2232073109 Picked up _JAVA_OPTIONS: -Dfile.encoding=UTF-8 2013/10/14 22:34:58 com.twitter.finagle.Init$ apply 情報: Finagle version 6.5.1 (rev=59..9cde41) built at 20130626-111057 Gungnir server connected... ec2-xxx.amazonaws.com/54.238.99.212:9290 Welcome tf0054 gungnir>
  • 9. Define your schema! CREATE TUPLE userAction ( userId STRING, hotelId STRING ); CREATE TUPLE commitAction ( userId STRING, hotelId STRING, checkin TIMESTAMP('yyyy-MM-dd'), nights INT ); STRING             Java type string TINYINT            Java type byte SMALLINT          Java type short INT                 Java type int BIGINT              Java type long FLOAT              Java type float DOUBLE            Java type double TIMESTAMP        long value LIST MAP STRUCT
  • 10. Define your filter! (1/4) FROM userAction AS ua, commitAction AS ca USING kafka_spout() EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user') ; From EMIT
  • 11. Define your filter! (2/4) FROM userAction AS ua, commitAction AS ca USING kafka_spout() JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image USING mongo_fetch('getusers', 'userInfo') JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo USING mongo_fetch('hotelscom', 'hotel') EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user') ; From JOIN (user) JOIN (hotel) EMIT
  • 12. Define your filter! (3/4) FROM userAction AS ua, commitAction AS ca USING kafka_spout() JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image USING mongo_fetch('getusers', 'userInfo') JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo USING mongo_fetch('hotelscom', 'hotel') INTO stream_0 ; FROM stream_0 EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user') ; From JOIN (user) JOIN (hotel) stream_0 EMIT (user)
  • 13. Define your filter! (4/4) FROM userAction AS ua, commitAction AS ca USING kafka_spout() JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image USING mongo_fetch('getusers', 'userInfo') JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo USING mongo_fetch('hotelscom', 'hotel') INTO stream_0 ; FROM stream_0 EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user') ; FROM stream_0 BEGIN GROUP BY hotelId EACH count() AS count, * EMIT hotelId, hotelname, photo, count USING kafka_emit('${TOPOLOGY_ID}_hotel') END GROUP ; EMIT (user) From JOIN JOIN EACH sum(nights) EACH avg(nights) stream_0 GROUP BY EMIT (hotel)
  • 14. Register your filter! and ... SUBMIT TOPOLOGY; DESC TOPOLOGY; SHOW TOPOLOGIES; STOP TOPOLOGY xxx; gungnir> SUBMIT TOPOLOGY; OK gungnir> DESC TOPOLOGY; { "id":"52373363e4b0a525adadb4f2", "explain":" ... ", "status":"RUNNING", "owner":"tf0054@gmail.com", "createTime":"2013-09-16T16:35:47.432Z", "summary":{ ... } } gungnir> STOP TOPOLOGY 52373363e4b0a525adadb4f2 ; DROP TOPOLOGY xxx; gungnir> DELETE TOPOLOGY 52373363e4b0a525adadb4f2 ;
  • 15. Testing it! TRACK userAction {"userId":"siniida","hotelId":"226979"}; TRACK commitAction {"userId":"siniida","hotelId":"226979","checkin":"2013-10-18","nights":1}; *** We are changing this name because “TRACK” isn’t fit to its function! (advised in GOTO berlin 2013. Thx.) *** The function of this command is to make and post a event to gennai. gungnir> DESC TOPOLOGY; { "id":"52373363e4b0a525adadb4f2", "explain":" ... ", "status":"RUNNING", "owner":"tf0054@gmail.com", ... "hotelId" : "128767" "hotelId" : "356564" "hotelId" : "226979" "hotelId" : "325073" "hotelId" : "357090" "hotelId" : "126942" "hotelId" : "310294" ... USING kafka_emit('${TOPOLOGY_ID}_user') "hotelId" : "367746" "hotelId" : "232896" > bin/kafka-consumer.sh 52373363e4b0a525adadb4f2_user ; "hotelId" : "220231" or "hotelId" : "163175" > bin/kafka-consumer.sh 52373363e4b0a525adadb4f2_hotel; "hotelId" : "367856" "hotelId" : "360863" "hotelId" : "119747"
  • 16. Knowing http tuple receiver gungnir> DESC USER; { "id":"525e67f8e4b0e7cfca98d040", "name":"siniida", "password":"CmGOdN2UU/Xtmk0wRCZKzDy/BN65LAHPCp2+1KMDOOM=", "createTime":"2013-10-16T10:18:32.849Z" } gungnir> > curl -v -H "Content-type: application/json" -X POST -d '{"userId":"tf0054","hotelId":"226979"}' http://dev.genn.ai:9191/gungnir/v1.0/track/525e67f8e4b0e7cfca98d040/userAction
  • 17. Design of genn.ai Built on Storm  Converts  The  .. properly (e.g. building AST) Good Parts (e.g. Supporting parallelism setting)
  • 18. The future of genn.ai genn.ai wants friends!  genn.ai codes will be open-sourced in the future.  We are looking for co-challenging partners for improving genn.ai.  You can use genn.ai on your web sites completely for free (infrastructure etc..) if we can share the dream together. Please feel free to contact us!  http://genn.ai/  tf0054@gmail.com
  • 19. Thanks! http://genn.ai/ 19
  • 20. 20

×