Amazon DynamoDB Lessen's Learned by Beginner

5,775 views

Published on

Presented 14-Mar-2014 at Sendai, JAWS-UG.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,775
On SlideShare
0
From Embeds
0
Number of Embeds
41
Actions
Shares
0
Downloads
41
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Amazon DynamoDB Lessen's Learned by Beginner

  1. 1. Amazon DynamoDB Lessen's Learned by Beginner Correct me if anything wrong JAWS-UG Aizu Tokuno
  2. 2. About me ● TOKUNO, Hirokazu ● IT department in factory ● 4th years moved to IT at age of 39 – Studying hard to catch up ● Little know about – VBA, SQL, JavaScript, Java, Perl, Shell... ● Like to not write code – Pentaho ETL, Oracle APEX, Mojolicious
  3. 3. Learning/Challenge in 2013 ● Unicage Development Method ● Perl, Mojolicious ● Jenkins ● Automate Test jUnit, Mockito, shUnit2 ● Twitter Bootstrap, Google Maps API etc. ● Fun to attend workshop/seminar! – efsta! – Open Data Cafe Aizu – JAWS-US Aizu ● Join to CODE for AIZU
  4. 4. Learning/Challenge in 2014 ● Enroll to Linked Open Data Challenge 2013 ● Attend Hack for Town in Aizu ● Meteor(Node.js, MongoDB) on Nitrous.IO ● GitHub Debut ● Amazon DynamoDB Right now!
  5. 5. No experience in Could
  6. 6. Created AWS account just 7 days ago
  7. 7. What's DynamoDB? ● Fully Managed NoSQL Database Service
  8. 8. What's DynamoDB? ● Fully Managed NoSQL Database Service What's the hell are you talking about?
  9. 9. What's DynamoDB? ● Fully Managed NoSQL Database Service – NoSQL – Fully Managed Database Service Split into peaces
  10. 10. Understanding of NoSQL ● Not only SQL ● Fast to read from huge amount of data – Accept inconsistent ● Flexible Data Structure (DynamoDB etc.) – Attributes can be differ for each record ● Not suitable for summary and query
  11. 11. Fully Managed DB Service ● Rent database service as server – Pay for read/write speed, storage, transaction ● Managed by Amazon – Install, Upgrade and Patching – Keep storage space, no limit! – Distribute 3 data centers – Improve performance
  12. 12. Sum up! ● Ask Amazon for boring tasks! ● Extreme performance with SSD! ● Money solve the issue! ● Not to think about storage limitation! ● High reliability by distributing at 3 data center! ● Be careful! It's not traditional database.
  13. 13. Let's try! ● Service - DynamoDB
  14. 14. Create Table ● Create Table
  15. 15. Configure Table ● Primary Key – Only Hash or Hash/Range combination
  16. 16. Primary Key ● Hash Key : Column for partitioning – Pick not converged column like UserID – Do not use picky column like Game ID ● Range Key : – Column for search ● Push into 2 columns to be unique
  17. 17. Example of Primary Keys ● Present forum in 3 tables Table Primary Key Attributes Hash Range Group JAWS-UG { LastPostBy="Kurihara" LastPostDateTime= "2012-01- 03T00:40:57.165Z" } Thread JAWS-UG Session { Message = "How about unconference?" Tags = [ "JAWS-UG", "Aizu" ] LastPostDateTime = "2012-01- 03T00:40:57.165Z" } Reply JAWS- UG#Session 2011-12- 11T00:40:57. 165Z { Message = "Extreeeeeeme!" PostedBy = "Kurihara" }
  18. 18. Attributes ● Contents other than Hash, Range ● Each entry may have own attributes ● Data Type – String – Number – Binary – String Set (Array) – Number Set (Array) – Binary Set (Array) Convert Date to string
  19. 19. Create Index ● May skip since optional
  20. 20. Index ● Upto 5 indexes can be attached for each table – Hash-Range table is created under the table – Cost for read/write and storage ● Able to create at table creation. No edit, add. ● Local secondary index – Create Hash-Range table with shared Hash Key as table, Range Key is not ● Global secondary index – Create Hash-Range table with non shared Hash and Range
  21. 21. Index ● Sounds working like this... Hash Range Attribute1 Attribute2 A 1 abc def B 2 ghi jkl Hash Attribute2 Attribute1 Range A def abc 1 B jkl ghi 2 Primary Key Primary Key Index
  22. 22. Throughput ● Read Capacity Unit – Able to read one item per second for 4KB ● Write Capacity Unit – Able to write one item per second for 1KB
  23. 23. Alarm ● Send notification if exceed specified threathold, 75~95%
  24. 24. Final confirmation ● As usual ¥60/month!
  25. 25. Complete! ● 3 tables for forum
  26. 26. Disadvantages ● Unable to modify Index – Need to re-build if need to modify – Requires cost to migrate data ● (Possibly) No system to sync with doc like, ERD ● Suitable application and not suitable – Need to scan all records to query by othet than Primary Key, Index – Requires RDB for consistent transaction or use Java Transaction Library – Complex with RDB as recommended
  27. 27. Access from programs! ● Official SDKs ● Unofficial SDK for Perl etc. Android Browser iOS Java .NET Node.js Python PHP Ruby
  28. 28. SDK API ● Table – CreateTable – DeleteTable – DescribeTable – ListTables – UpdateTable ● Batch processing – BatchGetItem – BatchWriteItem ● Item – PutItem – GetItem – UpdateItem – DeleteItem ● Search – Query : Primary or Index – Scan : all record
  29. 29. Let's play with DynamoDB!
  30. 30. NoSQL is for Big Data
  31. 31. Big Data for Twitter
  32. 32. Make clear what's interesting!
  33. 33. What is to be investigated...
  34. 34. Voluntary Restraints!!
  35. 35. Okay, okay, let's try...
  36. 36. Forum with WebSock
  37. 37. Too many samples available... Not challenging enough
  38. 38. Create RESTfull API!
  39. 39. Like this DynamoDBNitrous.IOWeb Client ● All free of charge
  40. 40. RESTfull API ● Implemented very simple function – No BatchGetItem/BatchWriteItem Action Function Method Example of URL Create putItem POST /Group/Hash?p=1&q=A Read scan getItem GET /Groups /Groups/Hash Update updateItem PUT /Group/Hash?p=A&q=1 Delete deleteItem DELETE /Group/Hash
  41. 41. Create by POST ● curl -i -X POST -H "Content-Type: application/json" -d '{"datetime":"2013-03- 13T23:27:32.256Z" , "by":"Tokuno"}' http://your_host:8080/Group/JAWS-UG
  42. 42. Read all by GET ● curl -H "Content-Type: application/json" http://your_host:8080/Groups
  43. 43. Read by GET ● curl -H "Content-Type: application/json" http://your_host:8080/Group/JAWS-UG May need URI encode for Japanese
  44. 44. Update by PUT ● curl -i -X PUT -H "Content-Type: application/json" -d '{"datetime":"2021-03- 14T00:04:01.951Z" , "by":"No one tell me..."}' http://your_host:8080/Group/JAWS-UG
  45. 45. Delete by DELETE ● curl -i -X DELETE http://your_host:8080/Group/JAWS-UG
  46. 46. One more thing
  47. 47. Like this DynamoDBNitrous.IOPentaho ETL ● All free!
  48. 48. Pentaho for data processing ● Limited by DynamoDB throughput ● Pay more to boost Group LastPostDateTime LastPostBy あ 2013-03-13T23:27:32.256Z とくのう い 2013-03-13T23:27:32.256Z とくのう う 2013-03-13T23:27:32.256Z とくのう え 2013-03-13T23:27:32.256Z とくのう お 2013-03-13T23:27:32.256Z とくのう か 2013-03-13T23:27:32.256Z とくのう き 2013-03-13T23:27:32.256Z とくのう く 2013-03-13T23:27:32.256Z とくのう け 2013-03-13T23:27:32.256Z とくのう こ 2013-03-13T23:27:32.256Z とくのう あ1 2013-03-13T23:27:32.256Z とくのう い1 2013-03-13T23:27:32.256Z とくのう う1 2013-03-13T23:27:32.256Z とくのう え1 2013-03-13T23:27:32.256Z とくのう お1 2013-03-13T23:27:32.256Z とくのう か1 2013-03-13T23:27:32.256Z とくのう き1 2013-03-13T23:27:32.256Z とくのう
  49. 49. Summary ● Easy to maintaing since managed by Amazon! ● NoSQL is not 'Relational' but application links – Not easy to develop ● Unable to add/modify Index ● Not flexible to query. Need to query every 1MB ● Want method to actual and document ● Only suitable for disposable application and enterprise whom has big data?
  50. 50. Thank you!
  51. 51. Timeline ● 2014.03.08 Create account ● 2014.03.11 Attend Hack for Town 2014.03.11 Cross over in last 3 years ● 2014.03.12 Presentation for Game Lab ● 2014.03.13 Develop RESTful 2014.03.13 Play with Pentaho ● 2014.03.14 Presentation, today
  52. 52. Appendix
  53. 53. Region ● Find region used http://docs.aws.amazon.com/general/latest/gr/rande.html
  54. 54. Procedure to find Credentialss
  55. 55. IAM
  56. 56. Create a New Group of Users
  57. 57. Group
  58. 58. Permissions ● Select Power User Access for full access
  59. 59. Verify Policy ● Required to review although nonsense
  60. 60. Create User
  61. 61. User
  62. 62. Credentials ● Finally

×