Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.



Published on

Python asyncio REST Resource API

Published in: Engineering
  • Be the first to comment

  • Be the first to like this


  1. 1. GUILLOTINA An Async REST Resource DB to manage millions of objects
  2. 2. CoFounder/CTO Onna - SF/BCN Connect and search all knowledge
 inside an enterprise with ML Ramon Navarro Bosch Plone Software Foundation & FWT Member Web Framework Engineer 😱 !!
  4. 4. “Organizing data is where we spend more time and its boring.” Data Scientist Sysadmin / Engineers
  5. 5. Tables / CSV Docs / Unstructured data NOSQLSQL
  6. 6. WEB FRAMEWORKS • Angular/React : Server rendering frameworks are dead • Most sources of data comes from the web/api • Lots of experience on storing, distributing, managing resources
  7. 7. Web Framework
 Communities Data Scientist
 Communities Connect live data web framework with data scientist framework
  8. 8. Long time ago … 18 years … Zope and ZODB was created object oriented DB and web application server Then 16 years ago … Plone was created layer on top of Z stack to provide CMS Then 7 years ago … Pyramid was created merge pylons + repoze.bfg (zope fork) Then 2 years ago … Plone REST API was created Abstraction layer for creating resources on top of Plone 5 300 python packages Then 1 years ago … plone.server was created Rewrite from scratch of minimum Plone backend with py 3.6 and asyncio
  9. 9. TRANSACTION All operations are managed to be durable and confirmed, conflict resolution policies
  10. 10. TREE Information is
 organized in
 trees of objects
  11. 11. RESOURCES Objects are resources with schema attributes, annotations, OO inheritance and static/ dynamic behaviors. RESOURCE
  12. 12. SCHEMA JSON/ PYTHON Direct mapping of JSON schemas and python schemas
  13. 13. SECURITY Full definition of permissions / roles / principals with global and local inheritance of permissions on the tree.Allow, Deny, Unset,AllowSingle (no inheritance)
  14. 14. CRUD DynamicTraversal CRUD HTTP verbs mapping for each content type Custom endpoints for specific operations GET HEAD POST PUTPATCH
  15. 15. ASYNCIO All based on asyncio for network integrations with external indexers, db, services Based aioHTTP
  16. 16. CORS Cors configured globally and enabled by default
  17. 17. WEBSOCKET Websocket connection to apply operations throw frames. Mapping of REST API on aTCP async channel.
  18. 18. TUS Binary resumable file upload
  19. 19. EVENT Event based system to trigger operations in code
  20. 20. REGISTRY Configuration registry x main container
  21. 21. QUEUE Operational queue and after response tasks
  22. 22. MULTI DB Mount multiple DBs and partition objects pickles based onTree position POSTGRESQL COCKROACH
  23. 23. FILE CLOUD Support for S3/GCloud storage. ***local distributed FS soon***
  24. 24. INDEX Elasticsearch indexing
  25. 25. DIST CACHE Redis backend
  26. 26. SWAGGER Automatic API documentation generation
  27. 27. CONTAINERS Docker / K8s / Nomad out the box
  28. 28. EXPLICIT PY All configuration is defined on the code using decorators @configure.service()
  29. 29. guillotina(x) = argmin Zope/Plone(x) * zope.interface still used
  30. 30. Traversal /DB/MAIN_CONTAINER/OBJ1/OBJ2
  31. 31. • Only >= Python 3.5 • Designed to host millions of objects • Memory optimizations • Apply operations to contained objects in async • Authentication & authorization extensions (oauth2 flow supported) • Reusable UI JS components from Plone (Widgets/ SPA)
  32. 32. TRY IT !
  33. 33. { "databases": [{ "db": { "storage": "postgresql", "transaction_strategy": "resolve", "dsn": { "scheme": "postgres", "dbname": "guillotina", "user": "postgres", "host": "localhost", "password": "", "port": 5432 }, "pool_size": 40, "read_only": false } }], "host": "", "port": 8080, "static": [ {"favicon.ico": "static/favicon.ico"} ], "root_user": { "password": "root" }, "cors": { "allow_origin": ["*"], "allow_methods": ["GET", "POST", "DELETE", "HEAD", "PATCH"], "allow_headers": ["*"], "expose_headers": ["*"], "allow_credentials": true, "max_age": 3660 }, "utilities": [] }
  34. 34. docker run -p 5432:5432 -d postgres psql -h localhost -U postgres << EOF CREATE DATABASE guillotina; EOF pip install guillotina guillotina -c config.json curl -X GET http://localhost:8080 curl -u root:root http://localhost:8080 curl -X POST -u root:root -H "Content-Type: application/json" -d '{"@type":"Container","id":"mycontainer","title":"My Lovely Container"}' http://localhost:8080/db/ | jq . curl -X POST -u root:root -H "Content-Type: application/json" -d '{"@type":"Folder","id":"myfolder"}' http://localhost:8080/db/mycontainer | jq . curl -X POST -u root:root -H "Content-Type: application/json" -d '{"@type":"Folder"}' http://localhost:8080/db/mycontainer | jq . curl -X POST -u root:root -H "Content-Type: application/json" -d '{"@type":"Item","id":"myitem"}' http://localhost:8080/db/mycontainer/myfolder | jq . curl -X PATCH -u root:root -H "Content-Type: application/json" -d '{"title": "My new title"}' http://localhost:8080/db/mycontainer/myfolder/myitem | jq . curl -X GET -u root:root -H "Content-Type: application/json" -d '{"title": "My new title"}' http://localhost:8080/db/mycontainer/myfolder/myitem | jq . curl -X DELETE -u root:root -H "Content-Type: application/json" http://localhost:8080/db/mycontainer | jq .
  35. 35. DATA MODEL Resource & Container Interface Schema fields Static Behaviors Dynamic Behaviors
  36. 36. from guillotina import configure from guillotina.content import Item from guillotina.interfaces import IItem from guillotina import schema class ICustomType(IItem): foo = schema.Text() @configure.contenttype( type_name="CustomType", schema=ICustomType, behaviors=[ "guillotina.behaviors.dublincore.IDublinCore", "example.behaviors.ICustomBehavior", ]) class CustomType(Item): pass
  37. 37. @configure.subscriber(for_=(ICustomType, IObjectAddedEvent)) async def created_userfolder(obj, evnt): ... @configure.service( context=ICustomType, name='@myEndpoint', method='GET', permission='guillotina.AccessContent') class MyEndpoint(Service): async def __call__(self): ...
  38. 38. WIP
  39. 39. WIP : DISTRIBUTED HIVE Execute an operation to all objects in distributed execution Based on etcd Dynamic workers that are going to compute a task No aggregation callback Batch mass modification of the model guillotina_hive (thanks @vangheezy)
  40. 40. from guillotina.traversal import traverse async def my_task(task_info, root, request): data = path = data['path'] ob, end_path = await traverse(request, root, path.lstrip('/').split('/')) assert len(end_path) == 0 from guillotina.component import getUtility from guillotina_hive.interfaces import IHiveUtility hive = getUtility(IHiveUtility) task_info = TaskInfo('my_task', {'foo': 'bar'}) await hive.push_task(task_info)
  41. 41. WIP : INFERENCE Right now using tf serving Storing models as resources on the api TF loader adapter to get the model with cache Manages tf serving service provisioning on k8s guillotina_tf URI_RESOURCE/@applyModel?model=URI_MODEL Use event system on create/update apply model
  42. 42. WIP : DISTRIBUTEDTRAINING Right now using tf distributed Storing models as resources on the api Start workers and parameters servers with k8s Offer the tf operation to add guillotina REST API as source Reinforcement learning support ML architecture definition on guillotina with RestrictedPython guillotina_tflearn send_to_learn(FEED,ARCHITECTURE, MODEL_URL) Websocket feedback
  43. 43. “Why gRPC has no asyncio support?”
  44. 44. “Made in Barcelona, looking for contributions”
  45. 45. l October 16 22 2017 Come to the Digital Experience Conference