Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

mm-ADT: A Virtual Machine/An Economic Machine

0 views

Published on

The mm-ADT virtual machine is looking to be an economic engine for our open source developers in the data technologies landscape.

Published in: Technology
  • Be the first to comment

mm-ADT: A Virtual Machine/An Economic Machine

  1. 1. mm-ADT A Virtual Machine/An Economic Machine Dr. Marko A. Rodriguez DataDays Austin TX, January 2020 Database Keynote
  2. 2. It’s 2012 The Mayan Calendar predicts the end of the world. The OSS community predicts a Golden Age of Data Technologies. NoSQL
  3. 3. Document Database Graph Database Wide Column Store Index Store OLAP Processor
  4. 4. Open Source Software costs money and time for research and development. In order to provide the world free open source software many of our young innovators created companies to fund the development of their technology.
  5. 5. Apache
  6. 6. A Collaborative Ecosystem of Shared Technologies develops uses uses
  7. 7. uses develops develops A Collaborative Ecosystem of Shared Technologies
  8. 8. develops uses A Collaborative Ecosystem of Shared Technologies
  9. 9. develops uses A Collaborative Ecosystem of Shared Technologies
  10. 10. The NoSQL Summer of Love 2012uses uses uses uses uses develops develops develops develops develops
  11. 11. I compare us to the bridge, dam, and interstate builders of the 1930’s generation. This generation of developers built the modern world’s software data infrastructure. By making our software freely available, many institutions that would otherwise not pay for cutting-edge software have been able to advance their domain. We did something very good for this world.
  12. 12. And then came the parasites…
  13. 13. Amazon’s use of Apache TinkerPop for their internal operations is a perfect use of open source software. Amazon gets free software for their shipping logistics. Apache TinkerPop is tested at extreme scale on a world-class problem. DataStax, who were the primary developers at the time, is able to point (potential) customers to a success story. The Good
  14. 14. The Bad
  15. 15. Amazon “Acquires” OSS Companies for Free “Does Amazon contribute to these projects (benefitting users and the creators)?” Let’s see what the statistics say… The Ugly
  16. 16. The top contributors do not work for Amazon MongoDB MongoDB MongoDB MongoDB
  17. 17. The top contributors do not work for Amazon DataStax RReduX RReduX <unknown>
  18. 18. The top contributors do not work for Amazon DataStax DataStax DataStax Oracle
  19. 19. The top contributors do not work for Amazon DataBricks DataBricks DataBricks DataBricks
  20. 20. The top contributors do not work for Amazon Elastic Elastic Elastic Elastic
  21. 21. 1 2 5 10 20 50 100 5205020010005000 cassandra frequency commits 1 2 5 10 20 50 100 20502005002000 elasticsearch frequency commits 1 2 5 10 20 50 100 502005002000 mongodb frequency commits 1 2 5 10 20 50 100 501002005001000 spark frequency commits 1 2 5 10 20 50 100 15505005000 tinkerpop frequency commits α = 2.074 α = 1.69 α = 1.95 α = 2.19 α = 2.717 Smaller α means fewer people do most of the work. log/log-scale
  22. 22. Yes, AWS promises to contribute back to Cassandra. The question is whether those contributions be more than self- serving ones … there are no incentives to make real progress. To Amazon’s credit, they already made a single major contribution in the form of the Dynamo paper, 12 years ago. https://www.scylladb.com/2019/12/04/managed-cassandra-on-aws-our-take/
  23. 23. Let’s honor some other soldiers of the trade.
  24. 24. The top contributors do not work for Amazon RedisLabs RedisLabs CarrierDB Alibaba
  25. 25. The top contributors do not work for Amazon Cloudera Salesforce <unknown> Xiaomi
  26. 26. The top contributors do not work for Amazon Percona MariaDB MariaDB<unknown>
  27. 27. 1 2 5 10 20 50 100 5205020010005000 hbase frequency commits 1 2 5 10 20 50 100 5205020010005000 mariadb frequency commits 1 2 5 10 20 50 100 25201005002000 redis frequency commits α = 1.778 α = 1.713 α = 1.682
  28. 28. It used to be…
  29. 29. Contribute nothing, take everything. July 2016
  30. 30. Now it’s the…
  31. 31. MutuallyExclusive?
  32. 32. Amazon will continue to release products based on Apache Software Foundation technology. When Amazon has completed their exploitation and little competition exists they will use legal techniques to dismantle the Apache Software Foundation. Open source developers will use less liberal licenses. Innovation in our industry will come to a standstill. DataDays will be subsumed by re:Invent. The Apache Software Foundation had $1 million in their coffers for legal related issues in 2017. (prior to Amazon’s 2018 $1 million donation) Predictions
  33. 33. All popular open source technologies will be taken by Amazon. All popular closed source technologies will be cloned by Amazon. Small agile data technology companies will struggle. The majority of jobs in data technology systems engineering will be at Amazon. Salaries in data technology will drop preciptously. Amazon employees will try to create labor unions. Amazon will squash any unionization efforts. Their employees will be stuck in legal quagmires. Predictions U.S. tax payers will be left to foot the bill on the most expensive anti-trust suit the U.S. government has ever endured.
  34. 34. The systematic desertification of our industry in the pursuit of profits as opposed to the pursuit of innovation in our trade.
  35. 35. Oh father, love of mine, you have built such wonderful things with all your time. In honor of your greatness, we followed behind and now of age, we give not what was mine. May our software make your machines sing to the heavens for all time. A NoSQL Poem for Amazon/IBM/Microsoft
  36. 36. The Virtual Machine
  37. 37. Modern Data System Components Storage Processor Language
  38. 38. Synthetic Data Systems Storage Processor Language
  39. 39. mm-ADT Virtual Machine Storage Processor Language
  40. 40. Storage ProcessorLanguage mm-ADT Virtual Machine programs reads writes
  41. 41. Storage Processor Language mm-ADT Virtual Machine int bool strreal lst rec map filter reduce branch f.g(h.i).j.k Model Storage w/ Types obj ModelLanguagesw/Instructions Model Processors w/ Operators sideeffect initial terminal barrier
  42. 42. g.V(1).out(‘knows’).id() int{*} <=[=graph] [is,[get,’id’][eq,1]] [get,’outE’] [is,[get,’label’][eq,’knows’]] [get,’inV’] [get,’id’] graph -> [graph -> vertex{*} |vertex -> [‘id' :int, ’inE’ :edge{*}, ’outE’:edge{*}] |edge -> [‘label’:str, ’outV’ :vertex, ’inV’ :vertex]] Storage Processor Language mm-ADT Virtual Machine initial [=graph] filter [is,[get,’id’][eq,1]] map [get,’outE’] filter [is,[get,’label’][eq,’knows’]] map [get,’inV’] map [get,’id’] processor supports requisite operators language compiles to the graph model storage exposes graph model [ a->b | c->d | e->f ] pattern matching “if lhs, then rhs”
  43. 43. mm-ADT is about creating models in terms of other models. The base model is always mm. The ‘domain of discourse’ of the mm-ADT VM. Commonly used models are “standardize” by mm-ADT. mm : multi-model kv : key/value rdb: relational pg : property graph wc : wide column doc: documents rdf: resource description framework Storage providers expose their system via these models. Language providers write compilers to these models.
  44. 44. A user’s schema is a model… social -> [ person -> ['ssn' :int, 'friends':person{*}]] DSLs
  45. 45. social->[person -> ['ssn' :int~x, 'friends':person{*} A user’s schema is a model… …that can be embedded in other models. social->[person -> ['ssn' :int~x, 'friends':person{*} social model => relational model => mm-ADT model DSLs social -> [ person -> ['ssn' :int, 'friends':person{*}]] social model => graph model => mm-ADT model
  46. 46. social->[person -> ['ssn' :int~x, 'friends':person{*} <=(vertex{*} <=[=graph][is,[get,'id'][eq,x]] [get,'outE'] [is,[get,'label'][eq,'knows']] [get,'inV'])] <=(vertex <=[=graph][is,[get,'id'][eq,x]]) A user’s schema is a model… social model => graph model => mm-ADT model …that can be embedded in other models. social->[person -> ['ssn' :int~x, 'friends':person{*} <=(row{*} <=[=rdb][get,'people'][is,[get,'id'] [eq,[=rdb][get,'links'] [is,[get,'source'][eq,x]] [get,'sink']]])] <=(row <=[=rdb][get,'people'][is,[get,'id'][eq,x]]) social model => relational model => mm-ADT model DSLs social -> [ person -> ['ssn' :int, 'friends':person{*}]]
  47. 47. The Economic Machine
  48. 48. The mm-ADT virtual machine serves as a universal integrator for the technologies of our industry. The mm-ADT economic machine serves as a universal integrator for the creators in our industry. mm-ADT
  49. 49. Independent open source developers spend their time developing software that provides no financial compensation and, paradoxically, as the popularity of their project increases, success can lead to burnout and financial ruin. Entrepreneurial open source developers may form a company around their open source project. Unfortunately, such companies are easily obviated in the market because the core product is freely licensed and competing firms can offer the same product without having to invest in the research and development effort. Corporate open source developers are in the problematic situation where their work is dependent on their sponsoring organization who also controls the direction and pace of development as well as the interoperability and inclusion of any novel innovations within the project. Entrepreneurial Open Source Developers Lack a Competitive Edge Corporate Open Source Developers Sacrifice Autonomy Independent Open Source Developers Suffer Economically
  50. 50. The OSS Developer …is an artist, not an employee. …needs an agent, not a employer. …seeks to mentor, not to manage. …should be patronized, not invested in. …needs to be financially compensated for their work.
  51. 51. A Royalty-Based Open Source Model Data Technology Developers Think Etsy+Apache A Non-Growth Company contractor
  52. 52. A Royalty-Based Open Source Model mm-ADT mm-ADT mm-ADT design: Specify the component mm-ADT model. develop: Engineer the component binding. price: Decide the cost for licensing the component. promote: Write blog posts, academic articles, tutorials. maintain: Fix bugs and update over mm-ADT VM versioning. support: Mailing list Q&A, documentation, issue tracker. An mm-ADT component is created by an open source development team
  53. 53. A Royalty-Based Open Source Model mm-ADT Virtual Machine MicroCyberRobo Corp
  54. 54. A Royalty-Based Open Source Model mm-ADT Virtual Machine Design MicroCyberRobo Corp
  55. 55. A Royalty-Based Open Source Model mm-ADT Virtual Machine Develop MicroCyberRobo Corp
  56. 56. A Royalty-Based Open Source Model mm-ADT Virtual Machine Compose MicroCyberRobo Corp
  57. 57. A Royalty-Based Open Source Model mm-ADT Virtual Machine Purchase MicroCyberRobo Corp
  58. 58. A Royalty-Based Open Source Model mm-ADT Virtual Machine Payout MicroCyberRobo Corp
  59. 59. A Royalty-Based Open Source Model mm-ADT Virtual Machine Design Develop Compose Purchase Payout MicroCyberRobo Corp
  60. 60. Fin. http://mm-adt.org

×