Downloaded 20 times
![trove/guestagent/strategies
backup/
[…]
replication/
restore/
storage/
trove/guestagent/
[…]
backup/
common/
datastore/
[…]
strategies/
[…]
trove/guestagent/strategies/backup
base.py
base.pyc
couchbase_impl.py
couchbase_impl.pyc
__init__.py
__init__.pyc
mysql_impl.py
mysql_impl.pyc
postgresql_impl.py
grep class guestagent/strategies/backup/mysql_impl.py
class MySQLDump(base.BackupRunner):
class InnoBackupEx(base.BackupRunner):
class InnoBackupExIncremental(InnoBackupEx):
Trove Architecture: Code Modularity
A Deep Dive into Trove - SCALE 13x2/22/2015](https://image.slidesharecdn.com/adeepdiveintotrove-scale13x20150222-150223090827-conversion-gate02/85/A-deep-dive-into-trove-Scale-13x-Linux-Expo-2-22-15-11-320.jpg)
Uploaded byTesora
A deep dive into trove: Scale 13x Linux Expo 2/22/15
Kenneth Rugg, founder and CEO of Tesora, gave a presentation on OpenStack Trove at SCALE 13x in Los Angeles on February 22, 2015. He discussed how Trove provides database as a service capabilities for OpenStack, including self-service provisioning and management of both relational and non-relational databases. Rugg also provided an overview of Trove's architecture, key features including backups and replication, and plans for upcoming releases such as support for database clusters.
More Related Content
Similar to A deep dive into trove: Scale 13x Linux Expo 2/22/15
A deep dive into trove: Scale 13x Linux Expo 2/22/15
- 1. A Deep Diveinto Trove SCALE 13x, Los Angeles, CA February 22, 2015 Kenneth Rugg
- 2. Who Am I? •Ken Rugg, Founder & CEO of Tesora • Tesora: the Trove Company – #1 contributor to Trove project – Tesora DBaaS Platform • packaged, supported version of Trove – Nearly all work is upstream first – Database & distribution certifications • Database Virtualization Engine (DVE) – Open source, transparent sharding for MySQL Trove Contributors (Juno) Diverse and growing community • 128 contributors from 28 companies • 1723 commits, and 231829 lines of code A Deep Dive into Trove - SCALE 13x2/22/2015 2
- 3. Traditional IT • Provisioningby DBA’s • Database management by specialists • Waterfall development • Few large machines / bare metal • Oracle enterprise licenses • Captive audience Cloud • Self-service provisioning • Developers manage their own databases • Agile development • Many small machines / virtualization • Many data management technologies • Competition with AWS Transformation of Cloud Data Management A Deep Dive into Trove - SCALE 13x2/22/2015 3
- 4. What is Trove? •Database as a Service for OpenStack • API’s for both development and operations • Self service database provisioning • Full database lifecycle management • Multi-database support • Both Relational and NoSQL A Deep Dive into Trove - SCALE 13x2/22/2015 4
- 5. What is Trove?RDS for OpenStack …and DynamoDB …and RedShift Amazon AWS OpenStack EC2 SwiftS3 DynamoDB RedShift CinderEBS Nova … Trove RDBMS NoSQL DW RDS … …and API’s for DevOPs A Deep Dive into Trove - SCALE 13x2/22/2015 5
- 6. What is Trove?– The Trove Mission OpenStack Trove mission statement: “To provide scalable and reliable Cloud Database as a Service provisioning functionality for both relational and non-relational database engines, and to continue to improve its fully-featured and extensible open source framework.” A Deep Dive into Trove - SCALE 13x2/22/2015 6
- 7. Tuning • Automatically tune my.cnf •Buffer Pool Size • Log file size • max_connections • Sane defaults • InnoDB only • Disable load data infile • Disable select into outfile • API to programmatically set configuration groups Management • Create database / schema • Create users • Grant permissions to a Schema • Enable root user • Resize flavor • Resize volume • Full and incremental backups • Create replica What is Trove? – More than just provisioning… Security • Security groups • Turn off SSH • Remove anonymous user • Remove non-localhost users • Remove local file access • Mangle root user password • Apply security patches automatically 2/22/2015 7A Deep Dive into Trove - SCALE 13x
- 8. Trove In ProductionToday • Public Cloud – HP Cloud Relational Database • Launched May 2012 – Rackspace Cloud Databases • Launched August 2012 • Private Cloud: eBay – Began mid 2013 – Multiple Databases • MySQL, MongoDB, Redis Cassandra, Couchbase – Multi-region + HA – Working on Clustering Key Use Cases Development & test Web application hosting On-demand analytics Critical Capabilities Self-service provisioning & management Fleet wide configuration Multi-datastore architecture A Deep Dive into Trove - SCALE 13x2/22/2015 8
- 9. What is Trove?– OpenStack Architecture OpenStackTrove A Deep Dive into Trove - SCALE 13x2/22/2015
- 10. Datastore agnostic codein Trove Controller & Dashboard Datastore specific code isolated to Guest Agents What is Trove? Multi-Datastore Architecture Trove Controller Guest Agent Guest Agent Guest Agent Guest Agent Trove Dashboard (Horizon) Guest Agent Guest Agent Guest AgentGuest Agent A Deep Dive into Trove - SCALE 13x2/22/2015 10
- 11. trove/guestagent/strategies backup/ […] replication/ restore/ storage/ trove/guestagent/ […] backup/ common/ datastore/ […] strategies/ […] trove/guestagent/strategies/backup base.py base.pyc couchbase_impl.py couchbase_impl.pyc __init__.py __init__.pyc mysql_impl.py mysql_impl.pyc postgresql_impl.py grep class guestagent/strategies/backup/mysql_impl.py classMySQLDump(base.BackupRunner): class InnoBackupEx(base.BackupRunner): class InnoBackupExIncremental(InnoBackupEx): Trove Architecture: Code Modularity A Deep Dive into Trove - SCALE 13x2/22/2015
- 12.
- 13. Getting started withOpenStack Trove • As a Trove user – OpenStack distribution that includes Trove (such as HP Helion Dev Platform) • http://www8.hp.com/us/en/cloud/hphelion-openstack.html – Tesora DBaaS platform, a Trove packaging tailored for the enterprise • http://tesora.com/download-tesora-dbaas-platform-community-edition • As a Trove developer $ git clone http://github.com/openstack/trove-integration $ cd trove-integration/scripts $ ./redstack install $ ./redstack kick-start mysql • On top of DevStack – Add to localrc: • ENABLED_SERVICES+=,trove,tr-api,tr-tmgr,tr-cond – Swift should also be enabled for Backup and Restore. • ENABLED_SERVICES+=,s-proxy,s-object,s-container,s-account • SWIFT_HASH=<swift-hash-here> A Deep Dive into Trove - SCALE 13x2/22/2015 13
- 14. Provisioning a databaseinstance with Trove $ trove create <instance-name> <flavor-id> --size <volume-size> • Support for flavors • Support for volumes using Cinder • Optional parameters to create – Image per ‘datastore type’ and ‘version’ – Support AZs using --availability_zone – Support for Neutron using --nic A Deep Dive into Trove - SCALE 13x2/22/2015 14
- 15. Managing a databasewith Trove • Resize flavor • Resize volume • Datastore specific extensions: – Create Database / Schema – Create Users – Grant Users Permissions on Databases • Enable a Root User A Deep Dive into Trove - SCALE 13x2/22/2015 15
- 16. $ trove backup-create <backup-name><instance-id> • Optional params: – Description through --description – Incremental backups using –parent • To Restore backup use create: $ trove create <instance-name> <flavor-id> --size <volume-size> --backup <backup-id> • Fully managed • Triggered and tracked via API • Streamed to Swift (OpenStack Object Storage) • Incremental & full backups • Multiple formats per datastore supported via strategies: – XtraBackup (Percona) – mysqldump Backup and Restore A Deep Dive into Trove - SCALE 13x2/22/2015 16
- 17. OpenStack Trove: Completedin Juno • Async MySQL replication (master-slave) • Clusters for MongoDB • Neutron Support • Support for PostgreSQL • Config-groups enhancements – Configuration groups per datastore / version – Config-groups for MongoDB • Backups for Cassandra and Couchbase • Additional Tempest Tests A Deep Dive into Trove - SCALE 13x2/22/2015 17
- 18. New in Juno:Replication $ trove create <instance-name> <flavor-id> --size <volume-size> --replica_of <instance-id> – Support for async MySQL replication (MySQL slave instances) – Manual detach using $ trove update <instance-id> --detach-replica-source A Deep Dive into Trove - SCALE 13x2/22/2015 18
- 19. New in Juno:Clusters $ trove cluster-create <datastore> <ds-version> – Optional parameters to cluster-create • --instance <flavor_id=flavor_id,volume=volume> • Specify multiple times to create multiple instances for your cluster – Initial support in Juno added for MongoDB Clusters • Sets up mongo config server, and mongo query routers • Transparent and driven by configuration options • Support for adding shards to existing cluster for horizontal scale out. A Deep Dive into Trove - SCALE 13x2/22/2015 19
- 20. Planned for Kilo •Building out clusters – Semi Synchronous MySQL clusters (Galera) • Async Replication v2 – GTID based replication – Manual failover support • Associate flavors with datastores • Access datastore logs via API • Removing deprecated oslo-incubator code • Upgrade testing through grenade A Deep Dive into Trove - SCALE 13x2/22/2015 20
- 21. Where can Iget OpenStack Trove? • Get the source – OpenStack Trove Project • https://git.openstack.org/cgit/openstack/trove / http://github.com/openstack/trove.git – Trove python client binding and command line client • https://git.openstack.org/cgit/openstack/python-troveclient / http://github.com/openstack/python- troveclient.git – Trove design specifications for blueprints • https://git.openstack.org/cgit/openstack/trove-specs / http://github.com/openstack/trove-specs – Trove scripts for installation and testing, and elements for building guest images • https://git.openstack.org/cgit/openstack/trove-integration / http://github.com/openstack/trove- integration.git • Installable packages and guest images from Tesora – Community Edition: http://tesora.com/download-tesora-dbaas-platform-community-edition – Enterprise Edition trial: http://tesora.com/download-tesora-dbaas-platform-enterprise-edition A Deep Dive into Trove - SCALE 13x2/22/2015
- 22. More about Trove •Trove Wiki – https://wiki.openstack.org/wiki/Trove • Trove Source – https://git.openstack.org/cgit/openstack/trove • On IRC – #openstack-trove • Trove Day – Tesora.com/troveday – Slideshare.net/Tesoracorp • LinkedIn – OpenStack Trove Group Contact information Contact Tesora info@tesora.com www.tesora.com @tesoracorp Contact Ken krugg@tesora.com www.tesora.com @kenrugg A Deep Dive into Trove - SCALE 13x2/22/2015 22
- 23.
Editor's Notes
- #12 Let’s shift gears for a couple of minutes and talk briefly about the architecture from a different perspective. Trove must be able to address many different kinds of datastores. Some are NoSQL, some are SQL (and I’m playing with a graph database as well). Each of these implements a certain capability in a different way. Let’s talk about backup and restore as an example. Once you get the software, in the trove/guestagent directory, you’ll find a directory called strategies. And under strategies, one called backup, and under backup you’ll see the actual implementation for the specific datastore. Now looking at the MySQL implementation, you’ll see that there are two backup capabilities (full backup) and one incremental backup method. At configuration you can specify which implementation you would like to use. And you could implement a new strategy for your datastore thereby making Trove extensible and modular.
- #15 { "instance": { "status": "BUILD", "updated": "2014-06-05T19:33:46", "name": "test", "links": [ { "href": "https://region-a.geo-1.database.hpcloudsvc.com/v1.0/39745227274204/instances/ce629494-a64d-41ed-b73c-04cb28bb33bb", "rel": "self" }, { "href": "https://region-a.geo-1.database.hpcloudsvc.com/instances/ce629494-a64d-41ed-b73c-04cb28bb33bb", "rel": "bookmark" } ], "created": "2014-06-05T19:33:46", "id": "ce629494-a64d-41ed-b73c-04cb28bb33bb", "flavor": { "id": "1002", "links": [ { "href": "https://region-a.geo-1.database.hpcloudsvc.com/v1.0/39745227274204/flavors/1002", "rel": "self" }, { "href": "https://region-a.geo-1.database.hpcloudsvc.com/flavors/1002", "rel": "bookmark" } ] }, "datastore": { "version": "5.5", "type": "mysql" } } }