genn.ai (源内)
Realtime processing for everyone
Takeshi Nakano
@tf0054

1
Who am I?
Takeshi NAKANO
 Senior Researcher / Architect.
 Co-authered Getting started with Solr in Japanese.
 Co-authe...
Agenda
 What is genn.ai?

 Let s use genn.ai

 Realtime processing

 Download & get your key

 Usecases

 Define a sc...
A usecase of genn.ai
Customer
A

Web Sites
じゃらん

Customer
B

Customer
C

Marketer
Analyst

Realtime Analysis
Platform

Cli...
Visual editor for genn.ai
Structure of genn.ai
MongoDB
MetaStore

genn.ai UI

User account
Tuple schema

Query design
view
Query

Gungnir
Client
on ...
Preparing for starting.

http://dev.genn.ai/
Preparing for starting.

http://dev.genn.ai/
> cd gungnir-client
> bin/gungnir -u tf0054 -p 2e2232073109
Picked up _JAVA_O...
Define your schema!
CREATE TUPLE userAction (
userId STRING,
hotelId STRING
);
CREATE TUPLE commitAction (
userId STRING,
h...
Define your filter! (1/4)
FROM userAction AS ua, commitAction AS ca USING kafka_spout()
EMIT userId, hotelId, name, image US...
Define your filter! (2/4)
FROM userAction AS ua, commitAction AS ca USING kafka_spout()
JOIN userInfo ON userInfo.uid = user...
Define your filter! (3/4)
FROM userAction AS ua, commitAction AS ca USING kafka_spout()
JOIN userInfo ON userInfo.uid = user...
Define your filter! (4/4)
FROM userAction AS ua, commitAction AS ca USING kafka_spout()
JOIN userInfo ON userInfo.uid = user...
Register your filter! and ...
SUBMIT TOPOLOGY;

DESC TOPOLOGY;

SHOW TOPOLOGIES;

STOP TOPOLOGY xxx;

gungnir> SUBMIT TOPOL...
Testing it!
TRACK userAction {"userId":"siniida","hotelId":"226979"};
TRACK commitAction {"userId":"siniida","hotelId":"22...
Knowing http tuple receiver
gungnir> DESC USER;
{
"id":"525e67f8e4b0e7cfca98d040",
"name":"siniida",
"password":"CmGOdN2UU...
Design of genn.ai
Built on Storm
 Converts
 The
 ..

properly (e.g. building AST)

Good Parts (e.g. Supporting paralle...
The future of genn.ai
genn.ai wants friends!
 genn.ai

codes will be open-sourced in the future.

 We

are looking for ...
Thanks!

http://genn.ai/
19
20
Upcoming SlideShare
Loading in...5
×

genn.ai v2

836

Published on

The realtime processing tool based on Storm and Kafka.

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
836
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
3
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

genn.ai v2

  1. 1. genn.ai (源内) Realtime processing for everyone Takeshi Nakano @tf0054 1
  2. 2. Who am I? Takeshi NAKANO  Senior Researcher / Architect.  Co-authered Getting started with Solr in Japanese.  Co-authered Hadoop Hacks in Japanese.  Writing Getting started with Kafka in Japanese. http://genn.ai/ 2
  3. 3. Agenda  What is genn.ai?  Let s use genn.ai  Realtime processing  Download & get your key  Usecases  Define a schema and a filter  Make it real! http://genn.ai/ 3
  4. 4. A usecase of genn.ai Customer A Web Sites じゃらん Customer B Customer C Marketer Analyst Realtime Analysis Platform Client (Hotels) 4
  5. 5. Visual editor for genn.ai
  6. 6. Structure of genn.ai MongoDB MetaStore genn.ai UI User account Tuple schema Query design view Query Gungnir Client on finagle Gungnir Cosole Query Gungnir Thrift Server Gungnir on finagle Thrift Server Storm nimbus Topology Kafka User account Tuple schema JSON Tuple Gungnir Rest Server Gungnir Gungnir on finagle Rest Server Rest Server Tuple Kafka Kafka broker Kafka broker broker Tuple Storm Storm supervisor Storm supervisor supervisor MongoD B HDFS Tuple Topic http://genn.ai/ 6
  7. 7. Preparing for starting. http://dev.genn.ai/
  8. 8. Preparing for starting. http://dev.genn.ai/ > cd gungnir-client > bin/gungnir -u tf0054 -p 2e2232073109 Picked up _JAVA_OPTIONS: -Dfile.encoding=UTF-8 2013/10/14 22:34:58 com.twitter.finagle.Init$ apply 情報: Finagle version 6.5.1 (rev=59..9cde41) built at 20130626-111057 Gungnir server connected... ec2-xxx.amazonaws.com/54.238.99.212:9290 Welcome tf0054 gungnir>
  9. 9. Define your schema! CREATE TUPLE userAction ( userId STRING, hotelId STRING ); CREATE TUPLE commitAction ( userId STRING, hotelId STRING, checkin TIMESTAMP('yyyy-MM-dd'), nights INT ); STRING             Java type string TINYINT            Java type byte SMALLINT          Java type short INT                 Java type int BIGINT              Java type long FLOAT              Java type float DOUBLE            Java type double TIMESTAMP        long value LIST MAP STRUCT
  10. 10. Define your filter! (1/4) FROM userAction AS ua, commitAction AS ca USING kafka_spout() EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user') ; From EMIT
  11. 11. Define your filter! (2/4) FROM userAction AS ua, commitAction AS ca USING kafka_spout() JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image USING mongo_fetch('getusers', 'userInfo') JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo USING mongo_fetch('hotelscom', 'hotel') EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user') ; From JOIN (user) JOIN (hotel) EMIT
  12. 12. Define your filter! (3/4) FROM userAction AS ua, commitAction AS ca USING kafka_spout() JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image USING mongo_fetch('getusers', 'userInfo') JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo USING mongo_fetch('hotelscom', 'hotel') INTO stream_0 ; FROM stream_0 EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user') ; From JOIN (user) JOIN (hotel) stream_0 EMIT (user)
  13. 13. Define your filter! (4/4) FROM userAction AS ua, commitAction AS ca USING kafka_spout() JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image USING mongo_fetch('getusers', 'userInfo') JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo USING mongo_fetch('hotelscom', 'hotel') INTO stream_0 ; FROM stream_0 EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user') ; FROM stream_0 BEGIN GROUP BY hotelId EACH count() AS count, * EMIT hotelId, hotelname, photo, count USING kafka_emit('${TOPOLOGY_ID}_hotel') END GROUP ; EMIT (user) From JOIN JOIN EACH sum(nights) EACH avg(nights) stream_0 GROUP BY EMIT (hotel)
  14. 14. Register your filter! and ... SUBMIT TOPOLOGY; DESC TOPOLOGY; SHOW TOPOLOGIES; STOP TOPOLOGY xxx; gungnir> SUBMIT TOPOLOGY; OK gungnir> DESC TOPOLOGY; { "id":"52373363e4b0a525adadb4f2", "explain":" ... ", "status":"RUNNING", "owner":"tf0054@gmail.com", "createTime":"2013-09-16T16:35:47.432Z", "summary":{ ... } } gungnir> STOP TOPOLOGY 52373363e4b0a525adadb4f2 ; DROP TOPOLOGY xxx; gungnir> DELETE TOPOLOGY 52373363e4b0a525adadb4f2 ;
  15. 15. Testing it! TRACK userAction {"userId":"siniida","hotelId":"226979"}; TRACK commitAction {"userId":"siniida","hotelId":"226979","checkin":"2013-10-18","nights":1}; *** We are changing this name because “TRACK” isn’t fit to its function! (advised in GOTO berlin 2013. Thx.) *** The function of this command is to make and post a event to gennai. gungnir> DESC TOPOLOGY; { "id":"52373363e4b0a525adadb4f2", "explain":" ... ", "status":"RUNNING", "owner":"tf0054@gmail.com", ... "hotelId" : "128767" "hotelId" : "356564" "hotelId" : "226979" "hotelId" : "325073" "hotelId" : "357090" "hotelId" : "126942" "hotelId" : "310294" ... USING kafka_emit('${TOPOLOGY_ID}_user') "hotelId" : "367746" "hotelId" : "232896" > bin/kafka-consumer.sh 52373363e4b0a525adadb4f2_user ; "hotelId" : "220231" or "hotelId" : "163175" > bin/kafka-consumer.sh 52373363e4b0a525adadb4f2_hotel; "hotelId" : "367856" "hotelId" : "360863" "hotelId" : "119747"
  16. 16. Knowing http tuple receiver gungnir> DESC USER; { "id":"525e67f8e4b0e7cfca98d040", "name":"siniida", "password":"CmGOdN2UU/Xtmk0wRCZKzDy/BN65LAHPCp2+1KMDOOM=", "createTime":"2013-10-16T10:18:32.849Z" } gungnir> > curl -v -H "Content-type: application/json" -X POST -d '{"userId":"tf0054","hotelId":"226979"}' http://dev.genn.ai:9191/gungnir/v1.0/track/525e67f8e4b0e7cfca98d040/userAction
  17. 17. Design of genn.ai Built on Storm  Converts  The  .. properly (e.g. building AST) Good Parts (e.g. Supporting parallelism setting)
  18. 18. The future of genn.ai genn.ai wants friends!  genn.ai codes will be open-sourced in the future.  We are looking for co-challenging partners for improving genn.ai.  You can use genn.ai on your web sites completely for free (infrastructure etc..) if we can share the dream together. Please feel free to contact us!  http://genn.ai/  tf0054@gmail.com
  19. 19. Thanks! http://genn.ai/ 19
  20. 20. 20
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×