BugSenseBig Data for mobilewith Erlang, C, LISP   Dionisis "Dio" Kakoliris      Head of Engineering      dgk@bugsense.com
BugSense Trivia●   Third biggest mobile SDK in the world●   Analyzing data from ~400M devices●   More than ~120k writes/up...
Data map landscape
Data processing landscape
Enter LDB
Distributed concurrent updates
Overview●   Complex Event Processing, In-Memory DB●   Time-series Stream Processor●   Super easy to setup/use - one packag...
Overview (contd)●   LISP-like DSL for custom processing / views●   Ideal for parallelism (functional)●   Lazy loading of f...
...And now, lets go BIG●   We love Erlang!●   Node isolation, replication, supervision●   Request processing and forwardin...
Why Erlang?●   Scaling●   Handles lots of connections efficiently (HTTP)●   Sending/receiving messages from/to nodes is tr...
Architecture
Snippet RDLOCK_HASH_STR(kvm1->hashtab, id, bucket, 1, &hashv1, &hashv2,              &bloom); if (!bloom) {      RDUNLOCK_...
Erlang <--> LDB●    Erlang is the right tool for the job●    Small language that handles a very crucial sector●    Actors ...
Snippets...spawn_ldb_app(NumOfKeyCells) ->       Pid = spawn(ldb_appnode, create, [self()]),       receive                ...
Snippets...create(ParentPid) ->       Port = open_port({spawn_driver, lethe_drv}, [binary]),       ParentPid ! {ok, Port},...
Bonus Stage: Why LISP?●   Very easy lexer, parser, analyzer, interpreter impl.●   SQL-ish queries●   Expressive power, abs...
Yet another snippet...(define (in-all? arrays uid) (reduce (lambda (x acc) (and x acc))         (map (lambda (x) (in? x ui...
Moving Forward●   Not easy hot code swapping●   Auxiliary tools for monitoring, support, configuration●   GUIs for the abo...
And?●   Use Erlang for your projects!●   Its small, easy and above-all it DELIVERS!●   Dont be scared by the complexity of...
Thank you!Stay tuned for fresh stuff!  www.bugsense.com   blog.bugsense.com
Upcoming SlideShare
Loading in …5
×

Big Data for Mobile

702 views

Published on

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
702
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
9
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Big Data for Mobile

  1. 1. BugSenseBig Data for mobilewith Erlang, C, LISP Dionisis "Dio" Kakoliris Head of Engineering dgk@bugsense.com
  2. 2. BugSense Trivia● Third biggest mobile SDK in the world● Analyzing data from ~400M devices● More than ~120k writes/updates per sec● Custom Big Data database (LDB)● Eleven engineers● Cash positive
  3. 3. Data map landscape
  4. 4. Data processing landscape
  5. 5. Enter LDB
  6. 6. Distributed concurrent updates
  7. 7. Overview● Complex Event Processing, In-Memory DB● Time-series Stream Processor● Super easy to setup/use - one package● No big locks (fine-grained locking)● Describe-your-data mentality● C is fast● C is ideal for destructive updates (imperative)● "Let it crash!"
  8. 8. Overview (contd)● LISP-like DSL for custom processing / views● Ideal for parallelism (functional)● Lazy loading of files (asynchronous)● Saves data to disk (asynchronous)● Modular / Extendable architecture● Custom reducers / off-line processing
  9. 9. ...And now, lets go BIG● We love Erlang!● Node isolation, replication, supervision● Request processing and forwarding● Distributed● Ideal for building real-time systems● Fast, robust, reliable● YOU TRUST ERLANG
  10. 10. Why Erlang?● Scaling● Handles lots of connections efficiently (HTTP)● Sending/receiving messages from/to nodes is trivial● Building a replication/take over engine is easy● Mnesia for storage and shared info● C Linkedin Drivers● Being able to connect to remote nodes● All of the above - integrated in one language
  11. 11. Architecture
  12. 12. Snippet RDLOCK_HASH_STR(kvm1->hashtab, id, bucket, 1, &hashv1, &hashv2, &bloom); if (!bloom) { RDUNLOCK_HASH_STR(kvm1->hashtab, bucket); return 0; } FIND_HASH_STR(kvm1->hashtab, bucket, id, val, struct kvm1_val_t *); if (val) { result = val->cntr; RDUNLOCK_HASH_STR(kvm1->hashtab, bucket); return result; } RDUNLOCK_HASH_STR(kvm1->hashtab, bucket);
  13. 13. Erlang <--> LDB● Erlang is the right tool for the job● Small language that handles a very crucial sector● Actors are magnificent!● Mnesia is your "faithful" companion● Constantly evolving / getting better● Erlang processes are your (million) friends!More at: http://highscalability.com/blog/2012/11/26/bigdata-using-erlang-c-and-lisp-to-fight-the-tsunami-of-mobi.html
  14. 14. Snippets...spawn_ldb_app(NumOfKeyCells) -> Pid = spawn(ldb_appnode, create, [self()]), receive {ok, Port} -> io:format("New ldb_app started ~n", []), Pid ! {?START, self(), integer_to_list(1), integer_to_list(NumOfKeyCells)}, receive {ok, []} -> ok end end, {Pid, Port}.
  15. 15. Snippets...create(ParentPid) -> Port = open_port({spawn_driver, lethe_drv}, [binary]), ParentPid ! {ok, Port}, loop(Port).loop(Port) -> receive {?VIEW, Caller, Key, Value} -> port_command(Port, term_to_binary({?VIEW, Key, Value})), receive {ok, []} -> Caller ! {ok, []}; % THIS SHOULDNT BE FOO! {view, Key, Value, Result} -> Caller ! {view, Key, Value, Result}; _ -> ignored end, loop(Port);
  16. 16. Bonus Stage: Why LISP?● Very easy lexer, parser, analyzer, interpreter impl.● SQL-ish queries● Expressive power, abstraction● S-expressions, heterogeneous multi-dim. lists● Immutability => Ideal for parallel computing● Tons of Data => Prefix notation● Tons of Data => Data transformation FTW!
  17. 17. Yet another snippet...(define (in-all? arrays uid) (reduce (lambda (x acc) (and x acc)) (map (lambda (x) (in? x uid)) arrays)))(define (in-all-timespan arrays uid from to) (map (lambda (x) (timebubble (timewarp (current-timespace) x) (in-all? arrays uid))) (range from to)))(define (at-least-once-in-all? arrays uid from to) (reduce (lambda (x acc) (or x acc)) (in-all-timespan arrays uid from to)))(define (always-in-all? arrays uid from to) (reduce (lambda (x acc) (and x acc)) (in-all-timespan arrays uid from to)))(define (never-in-all? arrays uid from to) (not (at-least-once-in-all? arrays uid from to)))
  18. 18. Moving Forward● Not easy hot code swapping● Auxiliary tools for monitoring, support, configuration● GUIs for the above● Cross-platform● Interoperability with other systems● Support role for big and complex systems● Tweaking - LDB has the potential to run everywhere!● ...From your cell-phone to an HPC cluster!
  19. 19. And?● Use Erlang for your projects!● Its small, easy and above-all it DELIVERS!● Dont be scared by the complexity of modern systems● Distributed systems are the present and future● Harness this massive potential with Erlang● START A PROJECT NOW!
  20. 20. Thank you!Stay tuned for fresh stuff! www.bugsense.com blog.bugsense.com

×