Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
GUILLOTINA
Async REST Resource API to manage millions of objects
Full Stack Engineer Onna - BCN
Connect and search all knowledge

inside an enterprise with ML
Nathan Van Gheem
Plone Softw...
CoFounder/CTO Onna - SF/BCN
Connect and search all knowledge

inside an enterprise with ML
Ramon Navarro Bosch
Plone Softw...
BACKGROUND
WEB FRAMEWORKS
• Angular/React : Server rendering frameworks are
dying
• Most sources of data comes from the web/api
• Lot...
LOVE
We love Plone but wanted to
be able to use it in high
performant situations with
modern web technologies
Long time ago … 18 years … Zope and ZODB was created
object oriented DB and web application server
Then 16 years ago … Plo...
EVOLUTION
The spirit of Plone lives
LESSONS
Taking lessons learned from
Plone, simplify and apply to an
AsyncIO context
INSPIRATION
Plone/Zope’s hierarchical data model
Pyramid’s decorator-based configuration
Django’s global application settin...
FORKS
parts of zodb data model
plone.behavior
zope.security
zope.schema
zope.component/zope.configuration
zope.dublincore
z...
WHAT IT IS
NOT
replacement for Plone
re-implementation of Plone
plone.restapi compatible API
guillotina_cms
FEATURES
TRANSACTION
All operations are managed to
be durable and confirmed,
conflict resolution policies
CONSCIENTIOUS
WRITER
Reduce conflict errors, better
performance
RELATIONAL
DATABASES
Use the best database
systems available
PostgreSQL
CockroachDB
TREE
Information is

organized in

trees of objects
RESOURCES
Objects are resources with
schema attributes, annotations,
OO inheritance and static/
dynamic behaviors.
RESOURCE
JSON
SCHEMA/
PYTHON
Serialization of python
interfaces to JSON Schema
and back again
SECURITY
Full definition of permissions /
roles / principals with global
and local inheritance of
permissions on the tree.A...
CRUD
DynamicTraversal CRUD
HTTP verbs mapping for each
content type
Custom endpoints for specific
operations
GET
HEAD
POST
...
ASYNCIO
All based on asyncio for
network integrations with
external indexers, db, caching,
services
Based on aioHTTP
SIMPLE
Easy to install,
Easy to develop
pip install guillotina
docker run guillotina/guillotina
CORS
Cors configured globally and
enabled by default
WEBSOCKET
Websocket connection to
apply operations through
frames. Mapping of REST API
on aTCP async channel.
MICRO-
SERVICES
Perfect for micro services
TUS
Binary resumable file upload
APPLICATIONS
Easily extendable with python packages
guillotina_swagger
guillotina_elasticsearch
guillotina_rediscache
guil...
EXTENSIBLE
Uses zope.interface and based
on zope.component



utilities and adapters!Yay!
COOKIE
CUTTER
Cookie cutter templates for
configuration and applications
QUEUE
Operational queue and after
response tasks
EVENT
Async event based system to
trigger operations in code



Async tasks
REGISTRY
Configuration registry for
each container
STATIC FILES
AND JS APPS
Support for serving JavaScript
applications
MULTI DB
Mount multiple DBs

Supports:
POSTGRESQL
COCKROACH
FILE CLOUD
Support for S3/GCloud
storage.
Database blob storage
otherwise
INDEX
ElasticSearch indexing
DIST CACHE
Redis backend
SWAGGER
Automatic API documentation
generation
CONTAINERS
Docker / K8s / Nomad out
the box
EXPLICIT PY
All configuration is defined on
the code using decorators
@configure.service()
guillotina(x) = argmin Zope/Plone(x)
* zope.interface still used
Traversal
/DB/MAIN_CONTAINER/OBJ1/OBJ2
• Only >= Python 3.6
• Designed to host millions of objects
• Memory optimizations
• Apply operations to contained objects...
TRY IT !
---
databases:
- db:
storage: postgresql
transaction_strategy: resolve
dsn:
scheme: postgres
dbname: guillotina
user: post...
docker run -p 5432:5432 -d postgres
psql -h localhost -U postgres << EOF
CREATE DATABASE guillotina;
EOF
pip install guill...
DATA MODEL
Resource &
Container
Interface
Schema fields
Static
Behaviors
Dynamic
Behaviors
from guillotina import configure
from guillotina.content import Item
from guillotina.interfaces import IItem
from guilloti...
@configure.subscriber(for_=(ICustomType, IObjectAddedEvent))
async def created_userfolder(obj, evnt):
...
@configure.servi...
DATA SCIENCE
“Organizing data is where we spend more time and its boring.”
Data Scientist Sysadmin / Engineers
Tables / CSV Docs / Unstructured data
NOSQLSQL
WIP : DISTRIBUTED HIVE
Execute an operation to all objects in distributed execution
Based on etcd
Dynamic workers that are...
from guillotina.traversal import traverse
async def my_task(task_info, root, request):
data = task_info.data
path = data['...
Try it, create issues, contribute
Go to the “Introduction to AsyncIO” talk!
PREGUNTES ?
GRÀCIES !
vangheem@gmail.com
ramon@plone.org
CORE : https://github.com/plone/guillotina
DOCS : http://guilloti...
Guillotina: The Asyncio REST Resource API
Upcoming SlideShare
Loading in …5
×

Guillotina: The Asyncio REST Resource API

451 views

Published on

Learn about the exciting new REST Resource API powered by Python's new asyncio library. In this talk you'll learn about some of the amazing things you can do with Guillotina and how you can leverage it to build your next JavaScript web application.

Published in: Technology
  • Be the first to comment

Guillotina: The Asyncio REST Resource API

  1. 1. GUILLOTINA Async REST Resource API to manage millions of objects
  2. 2. Full Stack Engineer Onna - BCN Connect and search all knowledge
 inside an enterprise with ML Nathan Van Gheem Plone Software Foundation,
 FWT Member, 
 UITEAM,
 SecurityTeam
  3. 3. CoFounder/CTO Onna - SF/BCN Connect and search all knowledge
 inside an enterprise with ML Ramon Navarro Bosch Plone Software Foundation & FWT Member
  4. 4. BACKGROUND
  5. 5. WEB FRAMEWORKS • Angular/React : Server rendering frameworks are dying • Most sources of data comes from the web/api • Lots of experience on storing, distributing, managing resources
  6. 6. LOVE We love Plone but wanted to be able to use it in high performant situations with modern web technologies
  7. 7. Long time ago … 18 years … Zope and ZODB was created object oriented DB and web application server Then 16 years ago … Plone was created layer on top of Z stack to provide CMS Then 7 years ago … Pyramid was created merge pylons + repoze.bfg (zope fork) Then 2 years ago … Plone REST API was created Abstraction layer for creating resources on top of Plone 5 300 python packages Then 1 years ago … plone.server was created Rewrite from scratch of minimum Plone backend with py 3.6 and asyncio
  8. 8. EVOLUTION The spirit of Plone lives
  9. 9. LESSONS Taking lessons learned from Plone, simplify and apply to an AsyncIO context
  10. 10. INSPIRATION Plone/Zope’s hierarchical data model Pyramid’s decorator-based configuration Django’s global application settings ZCA pluggability Plone/Zope’s security model JSON Schema
  11. 11. FORKS parts of zodb data model plone.behavior zope.security zope.schema zope.component/zope.configuration zope.dublincore zope.i18n zope.lifecycleevent zope.location zope.event
  12. 12. WHAT IT IS NOT replacement for Plone re-implementation of Plone plone.restapi compatible API guillotina_cms
  13. 13. FEATURES
  14. 14. TRANSACTION All operations are managed to be durable and confirmed, conflict resolution policies
  15. 15. CONSCIENTIOUS WRITER Reduce conflict errors, better performance
  16. 16. RELATIONAL DATABASES Use the best database systems available PostgreSQL CockroachDB
  17. 17. TREE Information is
 organized in
 trees of objects
  18. 18. RESOURCES Objects are resources with schema attributes, annotations, OO inheritance and static/ dynamic behaviors. RESOURCE
  19. 19. JSON SCHEMA/ PYTHON Serialization of python interfaces to JSON Schema and back again
  20. 20. SECURITY Full definition of permissions / roles / principals with global and local inheritance of permissions on the tree.Allow, Deny, Unset,AllowSingle (no inheritance)
  21. 21. CRUD DynamicTraversal CRUD HTTP verbs mapping for each content type Custom endpoints for specific operations GET HEAD POST PUTPATCH
  22. 22. ASYNCIO All based on asyncio for network integrations with external indexers, db, caching, services Based on aioHTTP
  23. 23. SIMPLE Easy to install, Easy to develop pip install guillotina docker run guillotina/guillotina
  24. 24. CORS Cors configured globally and enabled by default
  25. 25. WEBSOCKET Websocket connection to apply operations through frames. Mapping of REST API on aTCP async channel.
  26. 26. MICRO- SERVICES Perfect for micro services
  27. 27. TUS Binary resumable file upload
  28. 28. APPLICATIONS Easily extendable with python packages guillotina_swagger guillotina_elasticsearch guillotina_rediscache guillotina_s3storage guillotina_gcloudstorage guillotina_dbusers guillotina_mailer guillotina_pgcatalog
  29. 29. EXTENSIBLE Uses zope.interface and based on zope.component
 
 utilities and adapters!Yay!
  30. 30. COOKIE CUTTER Cookie cutter templates for configuration and applications
  31. 31. QUEUE Operational queue and after response tasks
  32. 32. EVENT Async event based system to trigger operations in code
 
 Async tasks
  33. 33. REGISTRY Configuration registry for each container
  34. 34. STATIC FILES AND JS APPS Support for serving JavaScript applications
  35. 35. MULTI DB Mount multiple DBs
 Supports: POSTGRESQL COCKROACH
  36. 36. FILE CLOUD Support for S3/GCloud storage. Database blob storage otherwise
  37. 37. INDEX ElasticSearch indexing
  38. 38. DIST CACHE Redis backend
  39. 39. SWAGGER Automatic API documentation generation
  40. 40. CONTAINERS Docker / K8s / Nomad out the box
  41. 41. EXPLICIT PY All configuration is defined on the code using decorators @configure.service()
  42. 42. guillotina(x) = argmin Zope/Plone(x) * zope.interface still used
  43. 43. Traversal /DB/MAIN_CONTAINER/OBJ1/OBJ2
  44. 44. • Only >= Python 3.6 • Designed to host millions of objects • Memory optimizations • Apply operations to contained objects in async • Authentication & authorization extensions • Reusable UI JS components from Plone (Widgets/SPA)
  45. 45. TRY IT !
  46. 46. --- databases: - db: storage: postgresql transaction_strategy: resolve dsn: scheme: postgres dbname: guillotina user: postgres host: localhost password: '' port: 5432 pool_size: 40 read_only: false host: 127.0.0.1 port: 8080 static: - favicon.ico: static/favicon.ico root_user: password: root cors: allow_origin: - "*" allow_methods: - GET - POST - DELETE - HEAD - PATCH allow_headers: - "*" expose_headers: - "*" allow_credentials: true max_age: 3660 utilities: []
  47. 47. docker run -p 5432:5432 -d postgres psql -h localhost -U postgres << EOF CREATE DATABASE guillotina; EOF pip install guillotina guillotina -c config.json curl -X GET http://localhost:8080 curl -u root:root http://localhost:8080 curl -X POST -u root:root -H "Content-Type: application/json" -d '{"@type":"Container","id":"mycontainer","title":"My Lovely Container"}' http://localhost:8080/db/ | jq . curl -X POST -u root:root -H "Content-Type: application/json" -d '{"@type":"Folder","id":"myfolder"}' http://localhost:8080/db/mycontainer | jq . curl -X POST -u root:root -H "Content-Type: application/json" -d '{"@type":"Folder"}' http://localhost:8080/db/mycontainer | jq . curl -X POST -u root:root -H "Content-Type: application/json" -d '{"@type":"Item","id":"myitem"}' http://localhost:8080/db/mycontainer/myfolder | jq . curl -X PATCH -u root:root -H "Content-Type: application/json" -d '{"title": "My new title"}' http://localhost:8080/db/mycontainer/myfolder/myitem | jq . curl -X GET -u root:root -H "Content-Type: application/json" -d '{"title": "My new title"}' http://localhost:8080/db/mycontainer/myfolder/myitem | jq . curl -X DELETE -u root:root -H "Content-Type: application/json" http://localhost:8080/db/mycontainer | jq .
  48. 48. DATA MODEL Resource & Container Interface Schema fields Static Behaviors Dynamic Behaviors
  49. 49. from guillotina import configure from guillotina.content import Item from guillotina.interfaces import IItem from guillotina import schema class ICustomType(IItem): foo = schema.Text() @configure.contenttype( type_name="CustomType", schema=ICustomType, behaviors=[ "guillotina.behaviors.dublincore.IDublinCore", "example.behaviors.ICustomBehavior", ]) class CustomType(Item): pass
  50. 50. @configure.subscriber(for_=(ICustomType, IObjectAddedEvent)) async def created_userfolder(obj, evnt): ... @configure.service( context=ICustomType, name='@myEndpoint', method='GET', permission='guillotina.AccessContent') async def my_service(context, request): ...
  51. 51. DATA SCIENCE
  52. 52. “Organizing data is where we spend more time and its boring.” Data Scientist Sysadmin / Engineers
  53. 53. Tables / CSV Docs / Unstructured data NOSQLSQL
  54. 54. WIP : DISTRIBUTED HIVE Execute an operation to all objects in distributed execution Based on etcd Dynamic workers that are going to compute a task No aggregation callback Batch mass modification of the model guillotina_hive (thanks @vangheezy)
  55. 55. from guillotina.traversal import traverse async def my_task(task_info, root, request): data = task_info.data path = data['path'] ob, end_path = await traverse(request, root, path.lstrip('/').split('/')) assert len(end_path) == 0 from guillotina.component import getUtility from guillotina_hive.interfaces import IHiveUtility hive = getUtility(IHiveUtility) task_info = TaskInfo('my_task', {'foo': 'bar'}) await hive.push_task(task_info)
  56. 56. Try it, create issues, contribute
  57. 57. Go to the “Introduction to AsyncIO” talk!
  58. 58. PREGUNTES ? GRÀCIES ! vangheem@gmail.com ramon@plone.org CORE : https://github.com/plone/guillotina DOCS : http://guillotina.readthedocs.io/en/latest/ DOCKER: https://hub.docker.com/r/guillotina/guillotina/ MODULES : https://github.com/guillotinaweb ONNA: https://github.com/onna

×