ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов

10,057 views
9,962 views

Published on

Конференция "AI&BigData Lab", 12 апреля 2014

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
10,057
On SlideShare
0
From Embeds
0
Number of Embeds
9,208
Actions
Shares
0
Downloads
23
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов

  1. 1. MagnetoDB Key-Value Storage service for OpenStack Mirantis, 2014
  2. 2. Serge.Kovaleff gmail.com linkedin.com facebook.com @ sviridov.ilya@gmail.com Odessa, GeeksLab 4/12/2014@ isviridov at FreeNode
  3. 3. WII FM
  4. 4. WII F
  5. 5. WII FM
  6. 6. What the heck is OpenStack?
  7. 7. What the heck is OpenStack?
  8. 8. to be continued ...
  9. 9. Is it Big? Community 1227 active contributors in IceHouse more than 166 organizations Communication 44 active official IRC channels 2887 e-mails in March
  10. 10. Is it Big?
  11. 11. Is it about data? CERN - 3 PB per custer - 50000 + 35000 core PayPal 80000 VMs (according to Forbes)
  12. 12. OpenStack Big Data OpenStack Sahara - Elastic Hadoop clusters provisioning and management on OpenStack and elastic data processing (on-demand Hadoop job workflow) OpenStack MagentoDB - key-value storage service
  13. 13. What is MagnetoDB? MagnetoDB - OpenSource Amazon DynamoDB API implementation for OpenStack, the key-value database service for storing any amount of data with seamless scalability and predictable performance.
  14. 14. MagnetoDB key features ● Easy-to-integrate REST-like API (AWS SDK, boto clients) ● OpenStack standards following REST API ● Schemaless, non-relational table-based model ● Put/get/query/scan item operations ● Eventual and strong consistency reads ● Local Secondary indexes ● Batch read/write operations ● Designed to handle any amount of data and any level of request traffic ● Seamless throughput and storage scaling ● Fault tolerance
  15. 15. MagnetoDB use cases ● High request traffic level applications ● Time series data ● Queries are distributed across the data ● Any other applications ● Migration of workloads between AWS and OpenStack
  16. 16. Architecture highlights ● Layered architecture ● Horizontally scalable ● Pluggable database backend ● Uses native OpenStack infrastructure ○ OpenStack HEAT for provisioning ○ OpenStack Neutron LoadBalancer ○ Integration with OpenStack Keystone planned
  17. 17. Overall architecture
  18. 18. Integration with OpenStack services
  19. 19. Current status ○ 2.0.2 version released ○ Open source, Apache 2 license ○ Launchpad homepage: https://launchpad.net/magnetodb ○ Code published on github: https://github. com/stackforge/magnetodb ○ Deployable to Vanilla OpenStack ○ Compatible with existing DynamoDB clients
  20. 20. MagnetoDB Pilot Features ● Table CRUD API ○ ListTables ○ DescribeTable ○ CreateTable ○ DeleteTable ● Item CRUD API ○ PutItem ○ UpdateItem ○ DeleteItem ○ GetItem ● Data querying API ○ Query ○ Scan ● Error Handling API
  21. 21. MagentoDB use-case: dynamodb-geo “Geo Library for Amazon DynamoDB allows to easily create and query geospatial data”
  22. 22. Next steps ● General ○ Move the development to OpenStack Infra - done ○ Integration with OpenStack Keystone - done ● MagnetoDB API, following OpenStack tenets - in progress ● DynamoDB API ○ Atomic counters ○ Asynchronous table creation ○ Atomic conditional write operations ○ Batch operations - in progress ● Other features ○ Management API ○ Quota management
  23. 23. Summary ● The 2.0.2 implementation is available on GitHub in public repository ● MagnetoDB is compatible with AWS SDK and boto library ● It is now possible to run applications which use DynamoDB as a storage on top of OpenStack using MagnetoDB
  24. 24. Links and Resources ○ Wiki: https://wiki.openstack.org/wiki/MagnetoDB ○ Launchpad: https://launchpad.net/magnetodb/ ○ Source code: https://github.com/stackforge/magnetodb ○ IRC: #magnetodb @ FreeNode ○ AWS DynamoDB http://aws.amazon.com/dynamodb/ ○ dynamodb-geo https://github.com/awslabs/dynamodb-geo
  25. 25. Q&A Got Questions?

×