Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

2018 PyCon Korea - Ring

563 views

Published on

Ring: 프로그래밍 언어와 가까운 캐시 인터페이스

#
user에 item을 추가해야 한다고 생각해 봅시다.
클래스가 없는 언어라면 아마도 user_add_item(user, item) 같은 코드를 쓸 것입니다.
아마 user_delete_item도 있고 user_clear_items도 있겠지요.

하지만 우리는 파이썬 프로그래머니까 보통 user.add_item(item) 같은 코드를 씁니다.

#
user에 속한 item들을 가져오는 함수가 있고 이 함수는 결과를 캐시하고 있다고 생각해 봅시다.
user.get_items() 같은 코드를 쓸 수도 있고 user.get_cached_items(storage) 같은 코드를 쓸 수도 있겠지요.

item의 목록이 업데이트 되었습니다. 이제 캐시를 무효화해야 합니다.
아마도 user.invalidate_items()나 user.delete_cached_items(storage) 같은 코드를 만들어야 하겠지요.

Ring에서는 user.get_items.delete() 를 호출합니다.

#
Ring은 이 아이디어에서부터 출발합니다.

Published in: Software

2018 PyCon Korea - Ring

  1. 1. Ring
  2. 2. OOP user item .
 user_add_item(user, item) 
 . user_delete_item user_clear_items . user.add_item(item) .
  3. 3. user item 
 .
 user.get_items() 
 user.get_cached_items(storage) . item . .
 user.invalidate_items() user.delete_cached_items(storage)
 . Ring user.get_items.delete() .
  4. 4. • Ring . • .
 .
  5. 5. ?
  6. 6. ?
  7. 7. Ring ? ? =
  8. 8. ? • • • • :
  9. 9. 
 

  10. 10. • IO ...
  11. 11. • IO ...
  12. 12. • IO ...
  13. 13. decorator
  14. 14. decorator
  15. 15. decorator
  16. 16. decorator
  17. 17. decorator pure function
  18. 18. • “ 
 . (... ...) .” ( ) -
  19. 19. • “ 
 . (... ...) .” ( ) -
  20. 20. # 60 * 15 # cache_page: Django HTTP decorator
  21. 21. # 60 * 15 PyCon Django , ? # cache_page: Django HTTP decorator
  22. 22. # 60 * 15 PyCon Django , ? ..? # cache_page: Django HTTP decorator
  23. 23. Ring
  24. 24. Ring • Sub-function control • Methods & descriptor support • Key Consistency • Backend: memcache, redis, diskcache, shelve, dict • asyncio support • Fully customizable • Django integration
  25. 25. Sub-functions • import ring @ring.dict({}) def function(v): return ... import functools @functools.lru_cache() def function(v) return ... Ring lru_cache
  26. 26. Sub-functions • from django… import cache_page @cache_page(60) def view(request): return ... import ring @ring.dict( {}, expire=60) def function(v): return ... Ring Django per-view cache
  27. 27. Sub-functions • • & value = function(10) value = function(10) import ring @ring.dict({}) def function(v): return ... import functools @functools.lru_cache() def function(v) return ... Ring lru_cache
  28. 28. Sub-functions • • python-memcached function.delete(10) key = create_key(10) client.delete(key)
  29. 29. Sub-functions • • python-memcached key = create_key(10) value = function(10) client.set(key) value = function.update(10)
  30. 30. Sub-functions • & • python-memcached value = function(10)
 
 value = function .get_or_update(10) key = create_key(10) value = client.get(key) if value is None: value = function(10) client.set(key)
  31. 31. Sub-functions • & • • • • • ... value = function(10) function.delete(10) value = function.update(10) value = function.execute(10) value = function.get(10)
  32. 32. Sub-functions • Django
  33. 33. Sub-functions • Django
  34. 34. Sub-functions • get_many • set_many, update_many, get_or_update_many … @ring.memcache(...) def f(a, b): ... # getting data for f(1, 2), f(1, 3), f(a=2, b=2) data = f.get_many((1, 2), (1, 3), {'a': 2, 'b': 2}) # data sequence(list)
  35. 35. Sub-functions • get_many • set_many, update_many, get_or_update_many … @ring.memcache(...) def f(a, b): return a * 100 + b # getting data for f(1, 2), f(1, 3), f(a=2, b=2) data = f.get_many((1, 2), (1, 3), {'a': 2, 'b': 2}) assert data == [102, 103, 202]
  36. 36. Sub-functions • get_many • execute_many, set_many, update_many, get_or_update_many …, @ring.memcache(...) def f(a, b): return a * 100 + b # getting data for f(1, 2), f(1, 3), f(a=2, b=2) data = f.get_many((1, 2), (1, 3), {'a': 2, 'b': 2}) assert data == [102, 103, 202]
  37. 37. Sub-functions • ( ) • • ( ) namespace
  38. 38. Ring • Sub-function control • Methods & descriptor support • Key Consistency • Backend: memcache, redis, diskcache, shelve, dict • asyncio support • Fully customizable • Django integration
  39. 39. Methods & Descriptors class UserAPI(object): url_prefix = 'https://...' secret_key = '...' def __init__(self, user_id): self.user_id = user_id # cache? @classmethod def api_status(cls): return request( f'{cls.url_prefix}/status') # cache? def get_user(self, n): return request( f'{cls.url_prefix}/users/{n}')
  40. 40. Methods & Descriptors • ? • ( ) • • In-house 
 class UserAPI(object): url_prefix = 'https://...' secret_key = '...' def __init__(self, user_id): self.user_id = user_id # cache? @classmethod def api_status(cls): return request( f'{cls.url_prefix}/status') # cache? def get_user(self, n): return request( f'{cls.url_prefix}/users/{n}')
  41. 41. Methods & Descriptors • Ring: class UserAPI(object): url_prefix = 'https://...' secret_key = '...' def __init__(self, user_id): self.user_id = user_id @ring.dict({}, expire=5) @classmethod def api_status(cls): return request( f'{cls.url_prefix}/status') @ring.dict({}, expire=60) def get_user(self, n): return request( f'{cls.url_prefix}/users/{n}')
  42. 42. Methods & Descriptors • Ring: • (free) function • method • classmethod • staticmethod • property • custom descriptors • hybridmethod? hybridproperty? • descriptor class UserAPI(object): url_prefix = 'https://...' secret_key = '...' def __init__(self, user_id): self.user_id = user_id @ring.dict({}, expire=5) @classmethod def api_status(cls): return request( f'{cls.url_prefix}/status') @ring.dict({}, expire=60) def get_user(self, n): return request( f'{cls.url_prefix}/users/{n}')
  43. 43. Methods & Descriptors • Q: Ring method/descriptor ?
  44. 44. Methods & Descriptors • Q: Ring method/descriptor ? • A: https://github.com/youknowone/wirerope
  45. 45. Ring • Sub-function control • Methods & descriptor support • Key Consistency • Backend: memcache, redis, diskcache, shelve, dict • sync/asyncio support • Fully customizable • Django integration
  46. 46. Key consistency • • (expire, invalidate)
  47. 47. Key consistency • • (expire, invalidate) function(1) function(v=1) ? @functools.lru_cache(
 maxsize=64) def function(v): return ...
  48. 48. Key consistency • • (expire, invalidate) function(1) function(v=1)@functools.lru_cache(
 maxsize=64) def function(v): return ... ? ...
  49. 49. Key consistency • • (expire, invalidate) function.delete(1) function(v=1)@ring.dict({}) def function(v): return ... ?
  50. 50. Key consistency • • (expire, invalidate) function.delete(1) function(v=1)@ring.dict({}) def function(v): return ... ?
  51. 51. Key consistency • • (expire, invalidate) >>> import sample sample.f:1:2 sample.f:1:2 sample.f:1:2 import ring @ring.dict({}) def f(a, *, b): return ... print(f.key(1, b=2)) print(f.key(a=1, b=2)) print(f.key(**{'a': 1, 'b': 2}))
  52. 52. Key consistency • • (expire, invalidate) >>> import sample sample.A.f:A():1 sample.A.g:A:2 sample.A.g:A:3 import ring class A(object): def __ring_key__(self): return 'A()' @ring.dict({}) def f(self, a): pass @ring.dict({}) @classmethod def g(cls, a): pass a = A() print(a.f.key(a=1)) print(a.g.key(a=2)) print(A.g.key(a=3))
  53. 53. Key policy • 1 ( ) • ( ) • ( )
  54. 54. Ring • Sub-function control • Methods & descriptor support • Key Consistency • Backend: memcache, redis, diskcache, shelve, dict • asyncio support • Fully customizable • Django integration
  55. 55. Backends @ring.dict(s) @ring.shelve(s) @ring.memcache(s) @ring.redis(s) @ring.diskcache(s) @ring.django.cache()
  56. 56. Backends @ring.dict(s) @ring.shelve(s) @ring.memcache(s) @ring.redis(s) @ring.diskcache(s) @ring.django.cache() dict shelve.Shelf memcache.Client (python-memcached) pylibmc.Client pymemcache.client.Client aiomcache.Client redis.Client aioredis.Client diskcache.Cache django.core.cache.caches
  57. 57. Ring • Sub-function control • Methods & descriptor support • Key Consistency • Backend: memcache, redis, diskcache, shelve, dict • asyncio support • Fully customizable • Django integration
  58. 58. Backends @ring.dict(s) @ring.shelve(s) @ring.memcache(s) @ring.redis(s) @ring.diskcache(s) dict shelve.Shelf memcache.Client (python-memcached) pylibmc.Client pymemcache.client.Client aiomcache.Client redis.Client aioredis.Client diskcache.Cache
  59. 59. Backends • Q: ? • A: 30
  60. 60. Backends • Q: ? • A: 30 class MemcacheStorage( fbase.CommonMixinStorage, fbase.StorageMixin, BulkStorageMixin): def get_value(self, key): value = self.backend.get(key) if value is None: raise fbase.NotFound return value def set_value(self, key, value, expire): self.backend.set(key, value, expire) def delete_value(self, key): self.backend.delete(key) def touch_value(self, key, expire): self.backend.touch(key, expire) def get_many_values(self, keys): values = self.backend.get_multi(keys) return [values.get(k, fbase.NotFound) for k in keys] def set_many_values(self, keys, values, expire): self.backend.set_multi({k: v for k, v in zip(keys, values)}, expire) def delete_many_values(self, keys): return self.backend.delete_multi(keys)
  61. 61. Ring • Sub-function control • Methods & descriptor support • Key Consistency • Backend: memcache, redis, diskcache, shelve, dict • asyncio support • Fully customizable • Django integration
  62. 62. Storage Key • Q: Ring
  63. 63. Storage Key sample.article:42 @ring.memcache(client) def article(id): return ... print(article.key(42))
  64. 64. Storage Key sample.article:42 new_prefix:42 @ring.memcache(client) def article(id): return ... print(article.key(42)) @ring.memcache(client, key_prefix='new_prefix') def article(id): return ... print(article.key(42)) case1: prefix config
  65. 65. Storage Key sample.article:42 a42e @ring.memcache(client) def article(id): return ... print(article.key(42)) @article.ring.key def article_key(id): return f'a{id}e' print(article.key(42)) case2: key function override
  66. 66. Data Encoder • Q: , ? •
  67. 67. Data Encoder dict(id=42, name='Name') dict(id=42, name='Name') @ring.dict({}) def user1(id): return { 'id': id, 'name': 'Name'} print(user1(42)) @ring.dict( {}, coder='json') def user2(id): return { 'id': id, 'name': 'Name'} print(user2(42))
  68. 68. Data Encoder dict(id=42, name='Name') # = json.dumps(user1(42)) b'{"id": id, "name": "Name"}' d1 = {} @ring.dict(d1) def user1(id): return { 'id': id, 'name': 'Name'} print(d1[42]) d2 = {} @ring.dict(d2, coder='json') def user2(id): return { 'id': id, 'name': 'Name'} print(d2[42])
  69. 69. Data Encoder • Q: •
  70. 70. Data Encoder • json, pickle •
  71. 71. Data Encoder # = json.dumps(user1(42)) '{"id": id, "name": "Name"}' d1 = {} @ring.dict(d1) def user1(id): return { 'id': id, 'name': 'Name'} @user1.ring.encode def user_encode(v): return json.dumps(v) @user1.ring.decode def user_decode(v): return json.loads(v) print(d1[42]) case1: single function coder
  72. 72. Data Encoder import json import ring ring.coder.register( 'json', (json.dumps, json.loads)) import pickle import ring ring.coder.register( 'pickle', (pickle.dumps, pickle.loads)) case2: reusable coder
  73. 73. Strategy • __call__, get, update, delete ? • ? • ex: • ? • ex: expire hit •
  74. 74. Strategy • __call__, get, update, delete ? • ? • ex: • ? • ex: expire hit • UserInterface
  75. 75. Low-level Interface • Q: UserInterface ?
  76. 76. Ring ?
  77. 77. Ring ?
  78. 78. Low-level Interface @ring.memcache(client, key_prefix='article') def article(id): return ...
  79. 79. Low-level Interface @ring.memcache(client, key_prefix='article') def article(id): return ... key = article.key(42) # key encoded = client.get(key) # memcache get data = article.decode(encoded) #
  80. 80. Low-level Interface @ring.memcache(client, key_prefix='article') def article(id): return ... key = article.key(42) # key encoded = client.get(key) # memcache get data = article.decode(encoded) # data = article.get(42) # 3
  81. 81. Low-level Interface • : •
  82. 82. Ring
  83. 83. inspect.signature • parameters & return annotation • Python 3.3+ • Backport: https://pypi.org/project/inspect2/
  84. 84. inspect.signature In [1]: def f(a, b, *args, x, y=10, **kw): ...: pass ...: In [2]: import inspect In [3]: s = inspect.signature(f) Out[4]: <Signature (a, b, *args, x, y=10, **kw)> In [6]: for name, parameter in s.parameters.items(): ...: print(name, parameter.kind, parameter.default) ...: a POSITIONAL_OR_KEYWORD <class 'inspect._empty'> b POSITIONAL_OR_KEYWORD <class 'inspect._empty'> args VAR_POSITIONAL <class 'inspect._empty'> x KEYWORD_ONLY <class 'inspect._empty'> y KEYWORD_ONLY 10 kw VAR_KEYWORD <class 'inspect._empty'>
  85. 85. qualname • / 
 • 
 (py2 im_class !) • Python 3.3+ >>> class C: ... def f(): pass ... class D: ... def g(): pass ... >>> C.__qualname__ 'C' >>> C.f.__qualname__ 'C.f' >>> C.D.__qualname__ 'C.D' >>> C.D.g.__qualname__ 'C.D.g'
  86. 86. annotation • def f(a: int, b: int) -> int:
 return a + b
 print(f.__annotations__)
 {'a': <class 'int'>, 'b': <class 'int'>, 'return': <class 'int'>} • inspect.signature(f).return_annotation
 int • inspect.signature(f.get).return_annotation
 Optional[int] • inspect.signature(f.key).return_annotation
 str
  87. 87. pickle, shelve • pickle: serialize • shelve: pickle dict DB •
  88. 88. Methods & Descriptors • Q: Ring method/descriptor ? • A: https://github.com/youknowone/wirerope • Rope: 
 (unbound Rope ) • Wire: (method),
 (classmethod) 
 ( bind Wire ) • hybridmethod, hybridproperty
  89. 89. Descriptor analysis • : • ? • descriptor ? • method property ? • descriptor ?
 (= method classmethod - hybrid ) • descriptor ? • ?
  90. 90. Descriptor analysis • : • ? • descriptor ? • method property ? • descriptor ?
 (= method classmethod - hybrid ) • descriptor ? • ? 2.7, 3.4-3.7 / PyPy2 PyPy3 def _f(owner): return owner
  91. 91. Django app in single file • Django ... • ? • 30 Django app • https://github.com/youknowone/django-app-in-single-file

×