More Related Content Similar to 연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015 (20) More from Jeongkyu Shin (20) 연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 201517. :
▪
▪ MATLAB : matplotlib, ipython notebook
▪ Ultra-complex figures
▪
▪
▪
▪ /
19. It's time to go Python 3
▪ Python 3
▪ Django, numpy/scipy, …
▪ Python 3.3
▪ PEP-393: str
unicode codepoint
▪ narrow build UTF-16, wide build UTF-32
▪ UTF-8/16/32
▪ ASCII/Latin-1
▪ / ·
20. It's time to go Python 3
▪ Python 3.4
▪ PEP-3156: asyncio coroutine
▪ I/O
▪ GIL (global interpreter lock)
▪
▪ pip virtualenv
▪
22. – ZeroMQ
▪
▪ Request-Reply
▪ 1
( : HTTP-like server/client)
▪ Push-Pull
▪ N-to-1
( : logging system, task partitioning)
▪ Publish-Subscribe
▪ 1-to-N / N-to-M
( : broadcast)
23. – ZeroMQ
▪ :
▪ zero broker, zero latency, zero administration, ...
▪ : connection
▪ TCP / UDP / UNIX domain / local "in-process"
▪ zeromq transport header (ZMTP)
▪ payload (64+ )
▪ : persistent queue
▪ TCP reliability
24. – LogStash
▪ LogStash
▪ input / filter / output
▪ ZeroMQ, S3
▪ ( !)
Django
User Kernel
Logstash
Server
Database
ZeroMQ (push-pull)
+ JSON
AWS S3
msgpack
26. Docker containers
▪ docker container
▪
▪ VM deploy/destroy
▪ AWS EC2 Instance: ( )
▪ pooling
▪ Amazon ECS?
▪ container (run task) latency
▪ …
51. ▪ React / Flux / AngularJS / Polymer / ...
▪ +UI
▪ Polymer 0.9 API ....
▪ (?) Bootstrap …
▪ I/O 2015 Polymer 1.0
...
52. Polymer: it is too google to be true
▪
▪ 0.5 - 0.8rc2 - 0.9 - …
▪
▪ ( )
▪ Google I/O - 1.0 :
57. Polymer + Django + Security
▪ CORS (cross-origin resource scripting)
▪ Django – CORS header ( )
▪ Polymer – vulcanize Javascript CORS
▪ Crisper CORS Javascript
▪ -
▪ XSS
▪ Django - CSRF token
▪ Polymer – header iron-ajax
▪
61. : Django + RDS
▪ UTF-8 Python ?
▪ (cf., str / bytes / io / codecs)
▪ MySQL SQLite
▪ MySQL
▪ ForeignKey Django
▪ MySQL + Django + Korean = EPIC FAIL
▪ ( ) Postgres ...
▪ postgres ( / ) .
62. Python 3.4 – asyncio / coroutine
▪ .
▪ yield from ...
▪ AST static analysis coroutine
yield from
▪ , 100% static analysis
▪ duck typing + dynamic method generation
+ method proxy pattern ...
▪ Python 3.5 await/async
!
63. Python 3.4 – asyncio loop
▪ loop.close()
..
import asyncio
import asyncio_redis
loop = asyncio.get_event_loop()
conn = loop.run_until_complete(
asyncio_redis.Connection.create('localhost', 6379))
conn.close()
loop.close()
Task was destroyed but it is pending!
task: <Task pending coro=<_reader_coroutine() running at /...
/lib/python3.4/site-packages/asyncio_redis/protocol.py:919>
wait_for=<Future pending cb=[Task._wakeup()]>>
64. Python 3.4 – asyncio loop
▪ asyncio graceful shutdown ,
1. loop._run_once()
2. loop.run_until_complete(asyncio.sleep(0))
3. loop.run_until_complete(server.wait_closed())
▪ , wait_closed()
coroutine
.
▪ event loop .
65. Python 3.4 – asyncio loop
#! /usr/bin/env python3
import asyncio
loop = asyncio.get_event_loop()
@asyncio.coroutine
def my_timer():
i = 0
while True:
yield from asyncio.sleep(1)
print(i)
i += 1
try:
asyncio.async(my_timer(), loop=loop)
loop.run_forever()
except KeyboardInterrupt:
pass
finally:
loop.close()
^CTask was destroyed but it is pending!
task: <Task pending coro=<my_timer() done, defined at test.py:4>
wait_for=<Future pending cb=[Task._wakeup()]>>
66. Python 3.4 – asyncio loop
#! /usr/bin/env python3
import asyncio
loop = asyncio.get_event_loop()
@asyncio.coroutine
def my_timer():
i = 0
while True:
yield from asyncio.sleep(1)
print(i)
i += 1
try:
asyncio.async(my_timer(), loop=loop)
loop.run_forever()
except KeyboardInterrupt:
for t in asyncio.Task.all_tasks():
t.cancel()
try:
loop._run_once()
except asyncio.CancelledError:
pass
finally:
loop.close()
67. asyncio_redis
▪ Redis asyncio
▪ Heisenbug ! 🚨
▪ SCAN key ...
▪ while if ➞
▪ https://github.com/jonathanslenders/asyncio-
redis/issues/65
▪ Connection pool API
▪ API call connection
stateful API ( : SELECT)
▪ (aioredis)
68. : – ZeroMQ
▪ Asynchronous ? block ??
▪ socket connect
. ( ...)
▪ send/recv timeout
▪ aiozmq , blocking call
asyncio.wait_for timeout
.
▪ pyzmq , socket.poll(msec)
timeout .
69. : – LogStash
▪ zmq input plugin s3 output plugin
zmq recv_string
▪ 1: s3 output plugin “write”
, PutObject
DeleteObject .
▪ 2: config , output
plugin log level
· .
▪ 3: output plugin
input plugin receive timeout .
70. : Docker
▪
▪ docker registry "latest"
latest
▪ 2.0 2.0.1 tag
▪
▪
▪ docker 1.8 , CoreOS (rkt)
▪ /API
71. :
▪ Jupyter / ipython
▪ scalable
▪ Jupyter
▪ Unix
▪
▪ :
▪
▪ – Sorna
▪ .
73. ▪ Container Resource Consolidation
▪ Kubernetes
▪ paxos raft
agreement key-value
store .
▪ Ingen REPL (read-evaluate-print loop)
▪ stdout/stderr streaming
▪ interactive plot
▪ image, sound multimedia
74. ▪
▪
▪ User profiler + backtracer + ML with back propagation
▪
▪ d3.js fluid nonlinear navigation UI
▪
▪
▪
▪
▪