Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Shop.com: Rapid data modeling and testing – Couchbase Connect 2016

378 views

Published on

NoSQL systems allow for rapid development using flexible models through schema-less documents. However, these models still need to be defined to enforce consistency within the domain. Conceptual, logical, and physical data modeling is important, and often overlooked in the development process, regardless of the database platform. Once the models for a domain have been defined, data is then needed to start building the application. Have you ever spent time writing throw away code to generate or import a dataset? In this session, Aaron will cover how Shop.com/Market America uses Couchbase Server and Couchbase Mobile, including learnings on data modeling fundamentals and different ways to rapidly generate vast amounts of random / fake data to test models.

Published in: Software

Shop.com: Rapid data modeling and testing – Couchbase Connect 2016

  1. 1. Rapid Data Modeling and Testing AARON BENTON
  2. 2. bit.ly/cbcommunity
  3. 3. +
  4. 4. OUR CAPABILITI ES 6
  5. 5. Enthusiasm Disillusionm ent Panic Search for the Guilty Punishment of the Praise for Non-
  6. 6. First, solve the problem. Then, write the code.“ ” John Johnson
  7. 7. Data Modeling
  8. 8. Phases • Conceptual • Logical • Physical
  9. 9. Conceptual Data Modeling Entity Names Entity Relationships Users Products Line Items Orders
  10. 10. N 1 Person Birthplace Location Person Location Born In Birthplace of Person Location (1, 1) Born In Birthplace of (0, N) << Entity >> Person 0. .N 1 << Entity >> Location << Relationship >> Born in > < Birthplace of Chen Person Location IDEF1x Person Location Born In Birthplace of Bachman Crow's Foot ISO UML
  11. 11. Logical Data Modeling Entity Names Entity Attributes Entity Relationships Users User ID (PK) Username Password First Name Last Name Address City State Zip Code Country Email Phone Orders Order ID (PK) Order Number User ID (FK) Order Date Address City State Zip Code Country Phone Email Credit Card Expiration Date Order Line Items Order ID (FK) Product ID (FK) Quantity Total Products Product ID (PK) Product Name Description Price
  12. 12. Physical Data ModelingEntity Names -> Table NamesEntity Attributes -> Field NamesEntity Relationships -> Keys Primary / Foreign Keys Data Types
  13. 13. NoSql Modeling • How do you represent a model? • What tools do you use? • How do you show relationships? • Does code define the model? • What key patterns should be used?
  14. 14. JSON
  15. 15. JSON Schema
  16. 16. YAML
  17. 17. Now What?
  18. 18. 20
  19. 19. Fake Data Generators• FakerJS • ChanceJS • json-generator.com • objgen.com/json • mockaroo.com • jsonschema.net
  20. 20. Relationships are Hard
  21. 21. FakeIT
  22. 22. Fakeit • CLI Tool inspired by Swagger • Models defined in YAML • Generates JSON, CSON, YAML, CSV • Exports to Couchbase, Sync Gateway, Zip, Console
  23. 23. Fakeit Data • Exposes FakerJS and ChanceJS • Custom Build Functions w/ Pre and Post • Multiple Models • Dependencies • Definitions *fakeit is for structure, cbworkloadgen is for sizing
  24. 24. npm install fakeit -g
  25. 25. Original YAML Model
  26. 26. Simple Model 30 Data Attribute
  27. 27. Console Output
  28. 28. File Output
  29. 29. Output Formats
  30. 30. CSV Output
  31. 31. Zip Archive
  32. 32. Couchbase
  33. 33. What about Mobile…?
  34. 34. Sync Gateway
  35. 35. Sync Gateway with Auth
  36. 36. How does it Work?
  37. 37. Property Data: value • A static value to be used • Executed at build time fake • A FakerJS template string • Executed at build time pre_build • Function body used to initialize a property • Property is set to the returned value build • Function body used to set a property • Property is set to the returned value post_build • Function body used to manipulate a property after it has been set • Property is set to the returned value pre_run • Function body that is executed before any models are generated post_run • Function body that is executed before after all models have been generated
  38. 38. Flow Model Pre_RUN Model PRE_BUILD Initialize Property Value (EACH) Start Document Run Property PRE_BUILD + BUILD (EACH) PROPERTY POST_BUIL D (EACH) MODEL POST_BUIL D MODEL POST_RUN
  39. 39. Flow
  40. 40. Flow
  41. 41. Multiple Models
  42. 42. Multiple Models
  43. 43. Dependencie s Users Products Reviews Orders
  44. 44. Dependencies
  45. 45. Dependencies
  46. 46. Definitions
  47. 47. Definitions
  48. 48. continents.json countries.csv Input
  49. 49. continents.json countries.csv Input
  50. 50. How we use FakeIt• YAML models store in the repository • Make commands to generate data • Docker Compose: • Nginx • Couchbase • Sync Gateway • Testing / QA
  51. 51. Demos
  52. 52. Fakeit Examples• Contacts • Ecommerce • Flat • Flight Data • Music • Simple • Social github.com/bentonam/fakeit-examples
  53. 53. Questions?
  54. 54. HOW TO CONTACT MEDon’t hesitate to contact with me. I will be happy to answer your questions. TWITTER: @bentonam github.com/bentonam EMAIL: aaron@shop.com THANK YOU

×