rqlite - replicating SQLite via Raft consensu
•
4 likes•2,309 views
rqlite is a distributed, replicated, relational database that uses SQLite as its storage engine. This presentation was give at the San Francisco Go Meetup in April 2016.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to rqlite - replicating SQLite via Raft consensu
Similar to rqlite - replicating SQLite via Raft consensu (20)
Recently uploaded
Recently uploaded (20)
rqlite - replicating SQLite via Raft consensu
- 1. 4/20/2016 rqlite http://127.0.0.1:3999/rqlite.slide#17 1/22 rqlite Replicating SQLite using Raft consensus Philip O'Toole GoSF April 20th 2016
- 2. 4/20/2016 rqlite http://127.0.0.1:3999/rqlite.slide#17 2/22 About me Director of Data Platform Engineering at Percolate. Previously a full-time core developer with In uxDB. Led the team that built Loggly's 2nd generation indexing and search platform. Big fan of Go, databases, and distributed systems.
- 3. 4/20/2016 rqlite http://127.0.0.1:3999/rqlite.slide#17 3/22 About rqlite rqlite is a distributed system, which fully replicates a SQLite database using the Raft consensus protocol. You can nd the source at github.com/otoolep/rqlite(https://github.com/otoolep/rqlite). The current release is v2.2.1.
- 5. 4/20/2016 rqlite http://127.0.0.1:3999/rqlite.slide#17 5/22 Why replicate SQLite? Rock solid relational database, within a single C source le. With replication, you get reliability. Easy installation and deployment, thanks to Go. Lightweight operation.
- 7. 4/20/2016 rqlite http://127.0.0.1:3999/rqlite.slide#17 7/22 Raft is a distributed consensus protocol. Such protocols are used to ensure multiple di erent nodes -- servers -- always agree on a given set of values. It allows us to build a cluster of servers, such that for a quorum of servers within the cluster, each of those servers has the same state. Within rqlite that state is a SQLite database.
- 9. 4/20/2016 rqlite http://127.0.0.1:3999/rqlite.slide#17 9/22 A rqlite node ┌──────────────┐ ┌────┐ Clients Other └──────────────┘ │ Nodes │ │ ───── │ ▲ │ │ │ │ ▼ ▼ ┌─────────────────────────────┐┌───────────────┐ │ HTTP(S) ││ TCP │ └─────────────────────────────┘└───────────────┘ ┌───────────────────────────────────────────────┐ │ Raft(hashicorp/raft) │ └───────────────────────────────────────────────┘ ┌───────────────────────────────────────────────┐ │ matt-n/go-sqlite3 │ └───────────────────────────────────────────────┘ ┌───────────────────────────────────────────────┐ │ sqlite3.c │ └───────────────────────────────────────────────┘ All repositories hosted on Github
- 10. 4/20/2016 rqlite http://127.0.0.1:3999/rqlite.slide#17 10/22 rqlite clusters rqlite clusters are most practical when 3, 5, or 7 nodes in size. Even numbers don't get you anything, and larger than 9 becomes unwieldy. rqlite is not suitable for replicating massive clusters, such as all the SQLite databases on a smartphone network.
- 11. 4/20/2016 rqlite http://127.0.0.1:3999/rqlite.slide#17 11/22 Hashicorp Raft Raft implementation from Hashicorp, available at github.com/hashicorp/raft (https://github.com/hashicorp/raft) Used by Consul, Nomad, and In uxDB. Another well known Raft implementation has been written by CoreOS, available at github.com/coreos/etcd/tree/master/raft(https://github.com/coreos/etcd/tree/master/raft)
- 12. 4/20/2016 rqlite http://127.0.0.1:3999/rqlite.slide#17 12/22 Integrating with Hashicorp Raft Integrating with the Raft consensus module involves implementing ve key functions. Apply(l*raft.Log)interface{} //Applyacommittedentrytothestatemachine Snapshot()(raft.FSMSnapshot,error)//Returnsasnapshotofthestatemachine Restore(rcio.ReadCloser)error //Createstatemachinefromsnapshot Persist(sinkraft.SnapshotSink) //Writesnapshottopersistentstorage Release() //Snapshotrelease.Usuallyano-op A reference use of Hashicorp's Raft implementation is available at github.com/otoolep/hraftd(https://github.com/otoolep/hraftd)
- 15. 4/20/2016 rqlite http://127.0.0.1:3999/rqlite.slide#17 15/22 "Plan to throw one away. You will, anyhow." rqlite v1.0 su ered from many shortcomings. Poor API. Used the generic Go SQL interfaces. Non-idiomatic Go code. go-raft as the consensus module, which was nearing its end-of-life. But v1.0 was still used to build github.com/openmarket/openmarket (https://github.com/openmarket/openmarket), a bitcoin-based marketplace!
- 16. 4/20/2016 rqlite http://127.0.0.1:3999/rqlite.slide#17 16/22 rqlite v2.2.1 Idiomatic Go code. Much better API, thanks to experience with In uxDB. Limited use of 3rd party libraries. Good use of interfaces, which allows better testing. Direct use of the Go SQLite package.
- 17. 4/20/2016 rqlite http://127.0.0.1:3999/rqlite.slide#17 17/22 Better code means better features Read consistency levels: strong, weak, and none. Hot backups. In-memory database support. User-level permissioning. There is now a Python driver available at github.com/zmedico/pyrqlite (https://github.com/zmedico/pyrqlite)
- 18. 4/20/2016 rqlite http://127.0.0.1:3999/rqlite.slide#17 18/22 Next steps rqlite is software, and software is never nished. Proper dependency management. A fully-featured CLI. Leader redirection.
- 19. 4/20/2016 rqlite http://127.0.0.1:3999/rqlite.slide#17 19/22 What rqlite can do With it you've got a lightweight and reliable distributed store for relational data. You could use rqlite as part of a larger system, as a central store for some critical relational data, without having to run a heavier solution like MySQL. rqlite might also be an e ective way to provide a small number of SQLite read-replicas.
- 21. 4/20/2016 rqlite http://127.0.0.1:3999/rqlite.slide#17 21/22 Thank you Philip O'Toole GoSF April 20th 2016 http://www.philipotoole.com/(http://www.philipotoole.com/) @general_order24(http://twitter.com/general_order24)