VoltDB and Erlang - Tech planet 2012

  • 3,856 views
Uploaded on

VoltDB and Erlang: two very promising beasts, made for the new parallel world, but still lingering in the wings. Not only are they addressing todays challenges but they are using parallel …

VoltDB and Erlang: two very promising beasts, made for the new parallel world, but still lingering in the wings. Not only are they addressing todays challenges but they are using parallel architectures as corner stone of their new and surprising approach to be faster and more productive. What are they good for? Why are we working to team them up?

Erlang promises faster implementation, way better maintenance and 4 times shorter code. VoltDB claims to be two orders of magnitude faster than its competitors. The two share many similarities: both are the result of scientific research and designed from scratch to address the new reality of parallel architectures with full force.

This talk presents the case for Erlang as server language, where it shines, how it looks, and how to get started. It details Erlang's secret sauce: microprocesses, actors, atoms, immutable variables, message passing and pattern matching. (Note: for a longer version of this treatment of Erlang only see: Why Erlang? http://www.slideshare.net/eonblast/why-erlang-gdc-online-2012)

VoltDB's inner workings are explained to understand why it can be so incredibly fast and still better than its NoSQL competitors. The well publicized Node.js benchmark clocking in at 695,000 transactions per second is described and the simple steps to get VoltDB up and running to see the prodigy from up close.

Source examples are presented that show Erlang and VoltDB in action.

The speaker is creator and maintainer of the Erlang VoltDB driver Erlvolt.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • The first part of the talk on YouTube: http://www.youtube.com/watch?v=fw2pgEZySXg
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
3,856
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
61
Comments
1
Likes
4

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. VoltDB and Erlang TechPlanet 2012 H. Diedrich http://www.eonblast.com – twitter @hdiedrich
  • 2. Your HostHenning Diedrich• Founder, CEO Eonblast• Maintainer of Emysql, Erlvolt
  • 3. This Presentation Review With Leasure• Video & Slides will be online at SK planet• Slides will be online at slideshare.net
  • 4. Eonblast
  • 5. VoltDB and Erlang Open Source Tech for Scale1. What makes Them Special?2. How do They Look?3. Are They For Me?
  • 6.  Made for a Concrete Need Made for Distribution Made for Multi-Core Truly Different Approaches Based on Parallelism Improving on Previous Solutions Corporate-Created Open Source Professional Support Known by Those in the Know
  • 7. The free, scaling, SQL DBRedis + SQL + Scale = VoltDB
  • 8. Why VoltDB?Business Perspective• Reduce Costs• Scale Seamlessly• Maintain Strategic Flexibility
  • 9. Why VoltDB?Production Perspective• Keep The Good of SQL• Use the Speed of In-Memory• Take Pain out of Scaling
  • 10. In-MemoryThe Redis of clusters• Today good for 100s of GB of data• Sheds 75% of DBM system activity• Made from scratch for distribution• Full disk persistence
  • 11. SQLThe MySQL of NewSQLs• SQL Has Flaws• Essentially Math, Not Syntax• Youd be missing Queries• VoltDB is even more SQL than SQL
  • 12. CAP • Distributed • Consistent • Highly-Available • Partition-Tolerant … have it all!Brewer on CAP 2012: http://www.infoq.com/articles/cap-twelve-years-later-how-the-rules-have-changed
  • 13. ACID• Atomicity• Consistency• Isolation• Durability… for granted?
  • 14. Double Bookkeeping• Not Every App needs It• Requires ACID Transactions• Neigh Impossible to emulate• Impossible With BASE (Eventual Consistency)
  • 15. The Magic• Compiled-In Queries• Single-Thread Execution• Snowflakes & Clones
  • 16. Partitions State #
  • 17. K Safety 1 State #
  • 18. Snowflake Structures FK FK FK PK PK FK FK FK
  • 19. Snowflake Structures FK FK FK PK FK FK
  • 20. Other Data PK PK PK
  • 21. Partitions State #ABC EFG HIJ KLM NOP
  • 22. Partitions State #ABC EFG HIJ KLM NOPA-Z A-Z A-Z A-Z A-Z
  • 23. Single Thread Execution• One Thread per Partition• One Thread per Transaction• One Thread cant race itself• Main Design Criterion
  • 24. Benchmark• Node.js• Amazon EC2• 64 core node.js clusters + 96 core VoltDB cluster• 695,000 transactions per second (TPS)• 2,780,000 operations per second• 100,000 TPS per 8 core client• 12,500 TPS per node.js core• Stable even under overload• Pretty much linear scaleDetails: http://voltdb.com/company/blog/695k-tps-nodejs-and-voltdb
  • 25. Benchmark TV Contest• Millions of callers• Small set of candidates• Massive peak• One transaction is one vote• Callers are identified by their telephone number• Callers must not be allowed to vote more than once
  • 26. BenchmarkCREATE TABLE contestants ( contestant_number integer NOT NULL , contestant_name varchar(50) NOT NULL , CONSTRAINT PK_contestants PRIMARY KEY ( contestant_number ) );CREATE TABLE votes ( phone_number bigint NOT NULL , state varchar(2) NOT NULL , contestant_number integer NOT NULL);CREATE TABLE area_code_state ( area_code smallint NOT NULL , state varchar(2) NOT NULL , CONSTRAINT PK_area_code_state PRIMARY KEY ( area_code ) );
  • 27. BenchmarkThe Transaction in 4 Operations// Check if the vote is for a valid contestantSELECT contestant_number FROM contestants WHERE contestant_number = ?;// Check if the voter has exceeded their allowed number of votesSELECT num_votes FROM v_votes_by_phone_number WHERE phone_number = ?;// Check an area code to retrieve the corresponding stateSELECT state FROM area_code_state WHERE area_code = ?;// Record a voteINSERT INTO votes (phone_number, state, contestant_number) VALUES (?, ?, ?);
  • 28. Drivers• Java• C, C++• node.js• Python• Ruby• Go• Erlang
  • 29. • VoltDB, Inc. 2009 – commercial developer, support• Open Source – 100% dictatorial by VoltDB, Inc• Made for OLTP – fast cheap writes, high throughput• CA of CAP – 100% consistent & highly available• Simple SQL – real queries• In-memory – 100x faster than MySQL• ACID transactions – double bookkeeping• Distributed – for painless growth• Linear scale – predictable, low cost• Replication, Snapshots – disk persistence, hot backup• More SQL than SQL – clean separation of data
  • 30. Getting Started• Download and Install from voltdb.com• Open Getting Started inside the installation• Run Hello World locally• Read M.I.T. Paper A New Architecture• Watch webinars at voltdb.com• Free Help also from VoltDB staff http://community.voltdb.com/forum• Use EC2 to benchmark your ideas
  • 31. Upcoming Blog Post• Looking at 14 databases• Riak, Cassandra, Membase, H-Base, Voldemort, MySQL, MySQL Cluster, Redis, Redis Cluster, Tokyo Cabinet, Memcached, CouchDB, Couchbase, VoltDB, MongoDB, Postgres• In the light of what games need• Unbiased comparison• Twitter @hdiedrich
  • 32. Erlang may be to Java what Java was to C++ C++ – pointers = JavaJava – deadlocks = Erlang
  • 33. Why Erlang?Business Perspective• Reduce Costs• Reduce Downtimes• Shorten Time To Market
  • 34. Why Erlang?Production Perspective• High Productivity• Low Hardware Requirements• Fewer Errors
  • 35. Who Is using It? “You probably use systemsbased on Erlang/OTP every day without knowing it.“ Mike Williams
  • 36. Erlang Game ServersZynga: FarmVille via membase, Activision Blizzard: Call of Duty, Bigpoint: Battle Star Galactica, Wooga: Magic Land
  • 37. Erlang Poster ChildsKlarna AB• Financial Services for E-Commerce• 30 seconds downtime in 3 yearsDistributed Databases• Membase• Riak• BigCouch
  • 38. Sweet Spots• Stateful Servers with High Throughput• Cluster Distribution Layers• Chats
  • 39. OriginsPLEX• Ericsson makes billions with telecom switches• They used PLEX, an all proprietary software• PLEX delivers, but has bad productivity
  • 40. Origins• The 80s: Ericsson Computer Science Lab Joe Armstrong, Robert Virding, Mike Williams „What aspects of computer languages make it easier to program telecom systems?“
  • 41. Erlang was Built For• Reliability• Maintenance• Distribution• Productivity
  • 42. Concurrency Built In• In the Language, not OS• Easier Testing• Massively Battle-Tested• Closer To Real Problems
  • 43. The Magic• Microprocesses• Pattern Matching *• Immutable Variables* Not your familiar Regex string matching
  • 44. Thinking Erlang• The Actor Model• Thinking Parallel• Thinking Functional• Thinking Processes• Pattern Matching• Let It Crash!
  • 45. The Actor ModelCarl Hewitt 1973• Behavior• State Actor• Parallel• Asynchronous Messages Data Data Code Code• Mailboxes• No Shared State• Self-Contained Machines Object ProcessBenefits• More true to the real world• Better suited for parallel hardware• Better suited for distributed architectures• Scaling garbage collection (sic!)• Less Magic
  • 46. Asynchronous Messages Pid ! Msg Message dispatch is one-way, truly asynchronous. Not function calls but something in their own right. Clean break from the FP paradigm.
  • 47. Thinking ProcessesWhat should be a Process?Processes “Its easy!” Joe Armstrong• Don’t share State• Communicate Asynchronously• Are Very Cheap to create And keep• Monitor Each Other• Provide Contention Handling• Constitute the Error Handling Atom
  • 48. Objects share Threads Multiple objects share threads. Objects can be accessed across threads. Threads - and objects - share state.
  • 49. Actors are Processes State, code and process form a unity: the actor. Like processes, actors do not share state. In fact, like humans. Who mostly work quite well.
  • 50. Objects and ThreadsC C++ C# Java JavaScript Node Lua Python
  • 51. Objects and Threads Idle Threads
  • 52. Objects and Threads Leeroy Jenkins!!! Unwanted surviving objects
  • 53. Objects and Threads 2*B || !2*B ? Prematurely destroyed objects
  • 54. Erlang ActorsErlang Actors: State + Code + Process
  • 55. Erlang Actors One dies.
  • 56. Erlang Actors Its restarted.
  • 57. Errors propagateLoss of state can affect multiple modules.State errors cross thread boundaries.Defensive code multiplies.
  • 58. Objects need LocksWorker 1 Doing X1Resource Locked LockedWorker 2 Waiting Doing X2 ● System design is disrupted by explicit locks. ● Overly cautious locking slows things down. ● Forgotten locks create errors that show under load.
  • 59. Crashed Locks StallWorker 1 Doing X1Resource Locked LockedWorker 2 Waiting Doing X2 ● Locks can need cross-thread error handling. ● Stalling and time outs aggravate load.
  • 60. Processes are Transactional Do X1 for me! Funnel Doing X1 Doing X2 Do X2 for me!● One actor is one process, obviously cannot “race itself”.● Mandating a job type to an actor creates a transactional funnel.● Only one such job will ever be executing at any one time.
  • 61. Thinking Parallel • The Generals’ Problem • Lamport Clocks • No Guarantees“Its not easy.”Robert Virding
  • 62. Generals’ Problem State Two generals need to agree when to attack.
  • 63. Generals’ Problem State One sends a messenger.
  • 64. Generals’ Problem State Acknowledge.
  • 65. Generals’ Problem State ACK the ACK. Etc.
  • 66. Generals’ Problem State The messenger may get lost.
  • 67. Byzantine Generals State The generals, actually, too.
  • 68. Lamport ClocksOrder matters more than time.Source: Lamport http://research.microsoft.com/users/lamport/pubs/time-clocks.pdf
  • 69. Thinking Functional Small Functions+ Immutable Variables→ Don’t assign variables: return results!Complete State in Plain Sight→ Awful for updates in place.→ Awsome for debugging & maintenance.Erlang is not side-effect free at all.
  • 70. Immutable Variables Can’t assign a second time: A = A + 1. A = 1, A = 2.
  • 71. Immutable Variables It has to be: B = A + 1. A = 1, B = 2.● Prevent Coding Errors● Provide Transactional Semantic● Allow For Pattern Matching Syntax● Can be a Nuisance
  • 72. Pattern Matching This can mean two things: A = func().The meaning depends on whether A is already assigned.
  • 73. Pattern MatchingThe common, mixed case: {ok, A} = func(). ok is an assertion AND A is being assigned.
  • 74. Pattern Matching The common, mixed case: {ok, A} = func().“This makes it hard to give Erlang a syntax like e.g. Lua.” Robert Virding
  • 75. Let It Crash!• No Defense Code• On Error, restart Entire Process• Built-In Process Supervision & Restart• Missing Branches, Matches cause Crash→ Shorter, Cleaner Code→ Faster Implementation→ More Robust: handles All Errors
  • 76. Syntax● Small● Easy● Stable● Declarative● Inspired by Prolog and ML● Obvious State, Implicit Thread
  • 77. FibonacciLooks like the math descriptionfib(0) -> 0;fib(1) -> 1;fib(N) when N>1 -> fib(N-1) + fib(N-2).
  • 78. Hello, World!-module(hello).-export([start/0]).start() -> io:format("Hello, World!~n").
  • 79. Hello, World! -module(hello). -export([start/0, loop/0]). start() -> Pid = spawn(hello, loop, []), Pid ! hello. loop() -> receive hello -> io:format("Hello, World!~n"), loop() end.From Edward Garsons Blog at http://egarson.blogspot.de/2008/03/real-erlang-hello-world.html
  • 80. Start-module(hello).-export([start/0, loop/0]).start() -> Pid = spawn(hello, loop, []), Pid ! hello.loop() -> receive hello -> io:format("Hello, World!~n"), loop() end. From Edward Garsons Blog at http://egarson.blogspot.de/2008/03/real-erlang-hello-world.html
  • 81. Output-module(hello).-export([start/0, loop/0]).start() -> Pid = spawn(hello, loop, []), Pid ! hello.loop() -> receive hello -> io:format("Hello, World!~n"), loop() end. From Edward Garsons Blog at http://egarson.blogspot.de/2008/03/real-erlang-hello-world.html
  • 82. Process Spawning-module(hello).-export([start/0, loop/0]).start() -> Pid = spawn(hello, loop, []), Pid ! hello.loop() -> New Process receive hello -> io:format("Hello, World!~n"), loop() end. From Edward Garsons Blog at http://egarson.blogspot.de/2008/03/real-erlang-hello-world.html
  • 83. Blocking Receive-module(hello).-export([start/0, loop/0]).start() -> Pid = spawn(hello, loop, []), Pid ! hello.loop() -> receive hello -> io:format("Hello, World!~n"), loop() end. From Edward Garsons Blog at http://egarson.blogspot.de/2008/03/real-erlang-hello-world.html
  • 84. Message Passing-module(hello).-export([start/0, loop/0]).start() -> Pid = spawn(hello, loop, []), Pid ! hello.loop() -> receive hello -> io:format("Hello, World!~n"), loop() end. From Edward Garsons Blog at http://egarson.blogspot.de/2008/03/real-erlang-hello-world.html
  • 85. Pattern Matching-module(hello).-export([start/0, loop/0]).start() -> Pid = spawn(hello, loop, []), Pid ! hello.loop() -> receive hello -> io:format("Hello, World!~n"), loop() end. From Edward Garsons Blog at http://egarson.blogspot.de/2008/03/real-erlang-hello-world.html
  • 86. Tail Recursion-module(hello).-export([start/0, loop/0]).start() -> Pid = spawn(hello, loop, []), Pid ! hello.loop() -> receive hello -> io:format("Hello, World!~n"), loop() end. From Edward Garsons Blog at http://egarson.blogspot.de/2008/03/real-erlang-hello-world.html
  • 87. ProductivityMotorola Study running 2002 – 2006: “Erlang shows … • 2x higher throughput • 3x better latency • 3 - 7x shorter code … than the equivalent C++ implementation.”
  • 88. Erlang• Ericsson AB 1983 – commercial developer, support• Open Source – 100% dictatorial by Ericsson• Professional Support – by veterans at Erlang Solutions• Made for telecom – complexity & robustness• Functional – improves maintainability• Actors Model – better than OO for parallel worlds• Distributed – seemless use of multi-cores• Parallel – Processes as basic building block• Simple Language – Easy to learn and maintain• Huge Library – OTP is made for server development• Battle-tested – billion $ projects, 1 million LOC• Great Community – bright and friendly devs
  • 89. Learning Erlang• Scour Post-Mortems• Download and Install from erlang.org• Fred’s site Learn You Some Erlang!• Joe’s book Programming Erlang• IRC #erlounge• Erlang Mailing List• Local Erlounge Meetings• Erlang Factories & User Conferences
  • 90. Erlvolt
  • 91. Erlvolt• The Erlang VoltDB Driver• Stable Synchronous Version• Upcoming Asynchronous Version https://github.com/Eonblast/Erlvolt
  • 92. ErlvoltInserterlvolt:callProcedure(Connection, "Insert", ["안녕하세요", "세계", "Korean"]),SelectResponse = erlvolt:callProcedure(Connection, "Select", ["Korean"]),Row = erlvolt:fetchRow(Table, 1),io:format("~n~n~s, ~s!~n",[ erlvolt:getString(Row, Table, "HELLO"), erlvolt:getString(Row, Table, "WORLD") ]);
  • 93. ResourcesErlang VoltDBWeb Downloadhttp://www.erlang.org/ http://voltdb.com/products-services/downloadshttp://learnyousomeerlang.com/http://www.erlang.org/static/getting_started_quickly.html Webinars http://community.voltdb.com/weninarsList Forumhttp://erlang.org/mailman/listinfo/erlang-questions http://community.voltdb.com/forumhttp://groups.google.com/group/erlang-programming Voter ExampleBooks https://github.com/VoltDB/voltdb/tree/master/examples/voterhttp://pragprog.com/book/jaerlang/programming-erlanghttp://shop.oreilly.com/product/9780596518189.do Benchmark Blog Post http://voltdb.com/company/blog/695k-tps-nodejs-and-voltdbReferenceshttp://erldocs.com/ Volt’s Magic Saucehttp://www.erlang.org/doc/ http://nms.csail.mit.edu/~stavros/pubs/hstore.pdfPost Mortemshttp://www.facebook.com/note.php?note_id=14218138919http://www.slideshare.net/wooga/erlang-the-big-switch-in-social-gamesErlvolthttps://github.com/Eonblast/ErlvoltEonblasthttps://www.eonblast.com
  • 94. Upcoming• Erlvolt 2 – asynchronous Erlang VoltDB driver• 14 New Databases for Games – Blog Post TBA on Twitter @hdiedrich
  • 95. Questions• Email: hdiedrich*eonblast.com• Twitter: @hdiedrich• VoltDB: http://community.voltdb.com/forum• Erlang: erlang-questions@erlang.org• Eonblast: http://www.eonblast.com• Game News: @eonblast