Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
/ (+ )
lablup.com
,
,
.
/ (+ )
lablup.com
,
,
.
▪
▪
▪ /
▪
▪
/
▪ TNF/ Needlworks
▪ …
▪
▪
▪
▪ NBA
▪
80 Gbps
▪
▪ …
http://www.lablup.com .
▪ 21 , 20 , 19
▪
▪ ,
21
?
▪
.
▪
.
.
..
▪
▪
▪ / / /
▪
▪ /
▪ Three keyboardists but…
▪
▪
▪
▪ resource consolidation
▪
▪
!
Python!
…
…
Python!
▪
▪ :
▪ PEP-8
▪ :
▪
▪
Python T_T
▪
▪
▪ PyPy / Pyston
▪ :
▪ C
▪ Cython
▪
Python /
▪
▪ CS101
▪ “ ”
▪
▪ indent
…
▪
▪ MATLAB numpy
:
▪
▪ MATLAB : matplotlib, ipython notebook
▪ Ultra-complex figures
▪
▪
▪
▪ /
:
▪
▪ NBA
▪
▪ configuration Python
▪ I/O
stdout asyncio
It's time to go Python 3
▪ Python 3
▪ Django, numpy/scipy, …
▪ Python 3.3
▪ PEP-393: str
unicode codepoint
▪ narrow build ...
It's time to go Python 3
▪ Python 3.4
▪ PEP-3156: asyncio coroutine
▪ I/O
▪ GIL (global interpreter lock)
▪
▪ pip virtuale...
▪ RabbitMQ, Kafka, Celery, ...
▪
▪ (broker) ,
▪ redis …
– ZeroMQ
▪
▪ Request-Reply
▪ 1
( : HTTP-like server/client)
▪ Push-Pull
▪ N-to-1
( : logging system, task partitioning)
▪ ...
– ZeroMQ
▪ :
▪ zero broker, zero latency, zero administration, ...
▪ : connection
▪ TCP / UDP / UNIX domain / local "in-pr...
– LogStash
▪ LogStash
▪ input / filter / output
▪ ZeroMQ, S3
▪ ( !)
Django
User Kernel
Logstash
Server
Database
ZeroMQ (pu...
On-premise vs. Hosting vs. Cloud
▪
▪ " " ...
▪ , ,
▪ AWS
▪
▪
▪ …
▪ 2012 R2 / /
.
Docker containers
▪ docker container
▪
▪ VM deploy/destroy
▪ AWS EC2 Instance: ( )
▪ pooling
▪ Amazon ECS?
▪ container (ru...
Docker containers
▪
▪ : instance
▪ Microsoft docker
▪ Azure Windows
Polymer library
▪ HTML5
▪ HTML imports / Custom elements / Shadow DOM
▪ Web components
▪ “Polyfill”
▪
▪ DOMelements
Polymer library
▪ ...
▪ !
▪ Bootstrap + theme?
▪
▪ HTML5 !
HTML imports!
Reusable web
components!
Shadow DOM!
Polyfill!
Polymer
▪
▪ 7 ? ?
Electron: Cross-platform Polymer
▪ : cross-platform webapp container
▪ Electron
▪ Github Node.js
▪ Node Chromium wrapping
...
▪
▪
▪
▪ Django UnitTest + Selenium
▪ Selenium webdriver:
▪
▪
▪
▪ sphinx
▪ reST
▪
▪
▪ !
▪
▪
▪
▪ sphinx
▪ reST
▪
▪
▪ !
▪
.
.
, ,
+
,
▪ +
.
▪ :
▪ –
▪ , , scaling, ...
➞ ++
▪ :
▪ –
▪
▪ .
▪ !
▪
▪
CodeON!
//codeonweb.com
!
…
, …
“ .”
▪ React / Flux / AngularJS / Polymer / ...
▪ +UI
▪ Polymer 0.9 API ....
▪ (?) Bootstrap …
▪ I/O 2015 Polymer 1.0
...
Polymer: it is too google to be true
▪
▪ 0.5 - 0.8rc2 - 0.9 - …
▪
▪ ( )
▪ Google I/O - 1.0 :
Polymer: vulcanize
▪ Vulcanize
▪ :
▪ IE – +
▪ CSS – +
▪ Crisper javascript map shadow DOM
root - +
Polymer: vulcanize
▪ Vulcanize
▪ :
▪ IE –
▪ CSS –
▪ Crisper javascript map shadow DOM
root -
Polymer: polyfill
▪
▪
▪
(html import
shadow DOM)
polyfill
▪
▪
Polymer + Django
▪
▪ {{ … }}
▪ Polymer –
▪ Django –
▪
▪
▪ …
Polymer + Django + Security
▪ CORS (cross-origin resource scripting)
▪ Django – CORS header ( )
▪ Polymer – vulcanize Java...
Polymer
▪ Production ready?!
▪
▪ “Do not swim in sandocean. Swim in the ocean.”
▪ HTML5
▪
▪ X-tag, react, polymer…
▪ ?
Polymer
▪ Production ready?!
▪
▪ “Do not swim in sandocean. Swim in the ocean.”
▪ HTML5
▪
▪ X-tag, react, polymer…
▪ ?
Electron + web
▪ ... !
▪
▪
(Noto )
▪
▪ CSS:
.
▪
?
: Django + RDS
▪ UTF-8 Python ?
▪ (cf., str / bytes / io / codecs)
▪ MySQL SQLite
▪ MySQL
▪ ForeignKey Django
▪ MySQL + Dj...
Python 3.4 – asyncio / coroutine
▪ .
▪ yield	from	 ...
▪ AST	 static	analysis coroutine
yield	from
▪ , 100% static analysi...
Python 3.4 – asyncio loop
▪ loop.close()
..
import asyncio
import asyncio_redis
loop = asyncio.get_event_loop()
conn = loo...
Python 3.4 – asyncio loop
▪ asyncio graceful shutdown ,
1. loop._run_once()
2. loop.run_until_complete(asyncio.sleep(0))
3...
Python 3.4 – asyncio loop
#! /usr/bin/env python3
import asyncio
loop = asyncio.get_event_loop()
@asyncio.coroutine
def my...
Python 3.4 – asyncio loop
#! /usr/bin/env python3
import asyncio
loop = asyncio.get_event_loop()
@asyncio.coroutine
def my...
asyncio_redis
▪ Redis asyncio
▪ Heisenbug ! 🚨
▪ SCAN key ...
▪ while if ➞
▪ https://github.com/jonathanslenders/asyncio-
r...
: – ZeroMQ
▪ Asynchronous ? block ??
▪ socket connect
. ( ...)
▪ send/recv timeout
▪ aiozmq , blocking call
asyncio.wait_f...
: – LogStash
▪ zmq input plugin s3 output plugin
zmq recv_string
▪ 1: s3 output plugin “write”
, PutObject
DeleteObject .
...
: Docker
▪
▪ docker registry "latest"
latest
▪ 2.0 2.0.1 tag
▪
▪
▪ docker 1.8 , CoreOS (rkt)
▪ /API
:
▪ Jupyter / ipython
▪ scalable
▪ Jupyter
▪ Unix	
▪
▪ :
▪
▪ – Sorna
▪ .
CodeON
▪ Container Resource Consolidation
▪ Kubernetes
▪ paxos raft
agreement key-value
store .
▪ Ingen REPL (read-evaluate-print...
▪
▪
▪ User profiler + backtracer + ML with back propagation
▪
▪ d3.js fluid nonlinear navigation UI
▪
▪
▪
▪
▪
▪
.
▪ ... + +
…
▪ .
▪ ?
Lablup Inc.
http://www.lablup.com
연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015
연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015
연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015
연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015
연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015
연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015
연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015
연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015
연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015
연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015
Upcoming SlideShare
Loading in …5
×

연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015

3,549 views

Published on

현대 과학 연구에는 컴퓨터를 이용한 계산 및 분석 작업이 필수적입니다. 그러나 거대 스케일의 계산 및 분석 작업을 수행할 경우 컴퓨팅 리소스의 적절한 관리 및 확장 용이성을 확보하는 것은 많은 리소스를 필요로 합니다. 우리는 컴퓨터 계산 작업 및 분석 작업을 표준화하고 클라우드에서 처리하는 파이썬3 기반의 오픈소스 플랫폼을 설계 및 개발하고 있습니다. 또한 이 플랫폼 위에서 돌아가는 교육 / 연구 플랫폼을 함께 설계하고 있습니다.

새로운 서비스를 변화하는 환경에 맞추어 개발하는 일은 즐거운 경험인 동시에, "무엇을" "어떻게" "왜" 로 이어지는 지뢰밭을 거니는 일이기도 합니다. "무엇을" 만들지 고민하고 토론하며 결정하고, 설계하고, 토론하고, 목표가 바뀌는 과정이 일어납니다. "어떻게" 만드느냐의 지뢰들로는 python 2에서 python 3 로의 전환, 웹 프레임웍인 Django와 프론트엔드 프레임웍들과의 충돌, 아마존 elastic computing cloud와 docker를 사용한 디플로이 시나리오 등 삽질 중에 발생하는 일들이 있습니다. "왜"에 대한 질문들은 무겁지만 피해갈 수 없습니다. "왜 하필 파이썬인가?" "왜 하필 그런 서비스를 만드려 하는가" 등의 질문은, 무엇인가를 만들기로 결심한 사람들에게 주어지는 가장 중요한 질문이자 보상이기도 합니다.

저희는 지난 2개월동안 이 과정을 통해 우리가 배운 것들을 공유하고자 합니다. 구체적으로는 개발 중인 플랫폼 구조, 설계 과정의 경험 및 python 3 기반의 플랫폼 개발시 주의할 점들에 대해 이야기하고, 그와 함께 지뢰밭을 걷게 만든 '원동력' 에 대해 함께 이야기할 수 있는 자리가 되었으면 합니다.

Published in: Software
  • Dating direct: ❶❶❶ http://bit.ly/2ZDZFYj ❶❶❶
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Sex in your area is here: ♥♥♥ http://bit.ly/2ZDZFYj ♥♥♥
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • ..............ACCESS that WEBSITE Over for All Ebooks ................ ......................................................................................................................... DOWNLOAD FULL PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Download or read that Ebooks here ... ......................................................................................................................... DOWNLOAD FULL PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • accessibility Books Library allowing access to top content, including thousands of title from favorite author, plus the ability to read or download a huge selection of books for your pc or smartphone within minutes DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... ...................................ALL FOR EBOOKS................................................. Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015

  1. 1. / (+ ) lablup.com , , .
  2. 2. / (+ ) lablup.com , , .
  3. 3. ▪ ▪ ▪ / ▪ ▪ / ▪ TNF/ Needlworks ▪ … ▪ ▪ ▪ ▪ NBA ▪ 80 Gbps ▪ ▪ … http://www.lablup.com .
  4. 4. ▪ 21 , 20 , 19 ▪ ▪ ,
  5. 5. 21 ?
  6. 6. ▪ . ▪ .
  7. 7. . ..
  8. 8. ▪ ▪ ▪ / / / ▪ ▪ /
  9. 9. ▪ Three keyboardists but…
  10. 10. ▪ ▪ ▪ ▪ resource consolidation ▪ ▪
  11. 11. ! Python! … …
  12. 12. Python! ▪ ▪ : ▪ PEP-8 ▪ : ▪ ▪
  13. 13. Python T_T ▪ ▪ ▪ PyPy / Pyston ▪ : ▪ C ▪ Cython ▪
  14. 14. Python / ▪ ▪ CS101 ▪ “ ” ▪ ▪ indent … ▪ ▪ MATLAB numpy
  15. 15. : ▪ ▪ MATLAB : matplotlib, ipython notebook ▪ Ultra-complex figures ▪ ▪ ▪ ▪ /
  16. 16. : ▪ ▪ NBA ▪ ▪ configuration Python ▪ I/O stdout asyncio
  17. 17. It's time to go Python 3 ▪ Python 3 ▪ Django, numpy/scipy, … ▪ Python 3.3 ▪ PEP-393: str unicode codepoint ▪ narrow build UTF-16, wide build UTF-32 ▪ UTF-8/16/32 ▪ ASCII/Latin-1 ▪ / ·
  18. 18. It's time to go Python 3 ▪ Python 3.4 ▪ PEP-3156: asyncio coroutine ▪ I/O ▪ GIL (global interpreter lock) ▪ ▪ pip virtualenv ▪
  19. 19. ▪ RabbitMQ, Kafka, Celery, ... ▪ ▪ (broker) , ▪ redis …
  20. 20. – ZeroMQ ▪ ▪ Request-Reply ▪ 1 ( : HTTP-like server/client) ▪ Push-Pull ▪ N-to-1 ( : logging system, task partitioning) ▪ Publish-Subscribe ▪ 1-to-N / N-to-M ( : broadcast)
  21. 21. – ZeroMQ ▪ : ▪ zero broker, zero latency, zero administration, ... ▪ : connection ▪ TCP / UDP / UNIX domain / local "in-process" ▪ zeromq transport header (ZMTP) ▪ payload (64+ ) ▪ : persistent queue ▪ TCP reliability
  22. 22. – LogStash ▪ LogStash ▪ input / filter / output ▪ ZeroMQ, S3 ▪ ( !) Django User Kernel Logstash Server Database ZeroMQ (push-pull) + JSON AWS S3 msgpack
  23. 23. On-premise vs. Hosting vs. Cloud ▪ ▪ " " ... ▪ , , ▪ AWS ▪ ▪ ▪ … ▪ 2012 R2 / / .
  24. 24. Docker containers ▪ docker container ▪ ▪ VM deploy/destroy ▪ AWS EC2 Instance: ( ) ▪ pooling ▪ Amazon ECS? ▪ container (run task) latency ▪ …
  25. 25. Docker containers ▪ ▪ : instance ▪ Microsoft docker ▪ Azure Windows
  26. 26. Polymer library ▪ HTML5 ▪ HTML imports / Custom elements / Shadow DOM ▪ Web components ▪ “Polyfill” ▪ ▪ DOMelements
  27. 27. Polymer library ▪ ... ▪ ! ▪ Bootstrap + theme? ▪ ▪ HTML5 !
  28. 28. HTML imports! Reusable web components! Shadow DOM! Polyfill!
  29. 29. Polymer ▪ ▪ 7 ? ?
  30. 30. Electron: Cross-platform Polymer ▪ : cross-platform webapp container ▪ Electron ▪ Github Node.js ▪ Node Chromium wrapping ▪ / / ▪ ATOM ▪ ? ▪ Apache Cordova
  31. 31. ▪ ▪ ▪ ▪ Django UnitTest + Selenium ▪ Selenium webdriver: ▪
  32. 32. ▪ ▪ ▪ sphinx ▪ reST ▪ ▪ ▪ ! ▪
  33. 33. ▪ ▪ ▪ sphinx ▪ reST ▪ ▪ ▪ ! ▪
  34. 34. . .
  35. 35. , , + ,
  36. 36. ▪ + . ▪ : ▪ – ▪ , , scaling, ... ➞ ++ ▪ : ▪ – ▪
  37. 37. ▪ . ▪ ! ▪ ▪
  38. 38. CodeON! //codeonweb.com !
  39. 39.
  40. 40. , …
  41. 41. “ .”
  42. 42. ▪ React / Flux / AngularJS / Polymer / ... ▪ +UI ▪ Polymer 0.9 API .... ▪ (?) Bootstrap … ▪ I/O 2015 Polymer 1.0 ...
  43. 43. Polymer: it is too google to be true ▪ ▪ 0.5 - 0.8rc2 - 0.9 - … ▪ ▪ ( ) ▪ Google I/O - 1.0 :
  44. 44. Polymer: vulcanize ▪ Vulcanize ▪ : ▪ IE – + ▪ CSS – + ▪ Crisper javascript map shadow DOM root - +
  45. 45. Polymer: vulcanize ▪ Vulcanize ▪ : ▪ IE – ▪ CSS – ▪ Crisper javascript map shadow DOM root -
  46. 46. Polymer: polyfill ▪ ▪ ▪ (html import shadow DOM) polyfill ▪ ▪
  47. 47. Polymer + Django ▪ ▪ {{ … }} ▪ Polymer – ▪ Django – ▪ ▪ ▪ …
  48. 48. Polymer + Django + Security ▪ CORS (cross-origin resource scripting) ▪ Django – CORS header ( ) ▪ Polymer – vulcanize Javascript CORS ▪ Crisper CORS Javascript ▪ - ▪ XSS ▪ Django - CSRF token ▪ Polymer – header iron-ajax ▪
  49. 49. Polymer ▪ Production ready?! ▪ ▪ “Do not swim in sandocean. Swim in the ocean.” ▪ HTML5 ▪ ▪ X-tag, react, polymer… ▪ ?
  50. 50. Polymer ▪ Production ready?! ▪ ▪ “Do not swim in sandocean. Swim in the ocean.” ▪ HTML5 ▪ ▪ X-tag, react, polymer… ▪ ?
  51. 51. Electron + web ▪ ... ! ▪ ▪ (Noto ) ▪ ▪ CSS: . ▪ ?
  52. 52. : Django + RDS ▪ UTF-8 Python ? ▪ (cf., str / bytes / io / codecs) ▪ MySQL SQLite ▪ MySQL ▪ ForeignKey Django ▪ MySQL + Django + Korean = EPIC FAIL ▪ ( ) Postgres ... ▪ postgres ( / ) .
  53. 53. Python 3.4 – asyncio / coroutine ▪ . ▪ yield from ... ▪ AST static analysis coroutine yield from ▪ , 100% static analysis ▪ duck typing + dynamic method generation + method proxy pattern ... ▪ Python 3.5 await/async !
  54. 54. Python 3.4 – asyncio loop ▪ loop.close() .. import asyncio import asyncio_redis loop = asyncio.get_event_loop() conn = loop.run_until_complete( asyncio_redis.Connection.create('localhost', 6379)) conn.close() loop.close() Task was destroyed but it is pending! task: <Task pending coro=<_reader_coroutine() running at /... /lib/python3.4/site-packages/asyncio_redis/protocol.py:919> wait_for=<Future pending cb=[Task._wakeup()]>>
  55. 55. Python 3.4 – asyncio loop ▪ asyncio graceful shutdown , 1. loop._run_once() 2. loop.run_until_complete(asyncio.sleep(0)) 3. loop.run_until_complete(server.wait_closed()) ▪ , wait_closed() coroutine . ▪ event loop .
  56. 56. Python 3.4 – asyncio loop #! /usr/bin/env python3 import asyncio loop = asyncio.get_event_loop() @asyncio.coroutine def my_timer(): i = 0 while True: yield from asyncio.sleep(1) print(i) i += 1 try: asyncio.async(my_timer(), loop=loop) loop.run_forever() except KeyboardInterrupt: pass finally: loop.close() ^CTask was destroyed but it is pending! task: <Task pending coro=<my_timer() done, defined at test.py:4> wait_for=<Future pending cb=[Task._wakeup()]>>
  57. 57. Python 3.4 – asyncio loop #! /usr/bin/env python3 import asyncio loop = asyncio.get_event_loop() @asyncio.coroutine def my_timer(): i = 0 while True: yield from asyncio.sleep(1) print(i) i += 1 try: asyncio.async(my_timer(), loop=loop) loop.run_forever() except KeyboardInterrupt: for t in asyncio.Task.all_tasks(): t.cancel() try: loop._run_once() except asyncio.CancelledError: pass finally: loop.close()
  58. 58. asyncio_redis ▪ Redis asyncio ▪ Heisenbug ! 🚨 ▪ SCAN key ... ▪ while if ➞ ▪ https://github.com/jonathanslenders/asyncio- redis/issues/65 ▪ Connection pool API ▪ API call connection stateful API ( : SELECT) ▪ (aioredis)
  59. 59. : – ZeroMQ ▪ Asynchronous ? block ?? ▪ socket connect . ( ...) ▪ send/recv timeout ▪ aiozmq , blocking call asyncio.wait_for timeout . ▪ pyzmq , socket.poll(msec) timeout .
  60. 60. : – LogStash ▪ zmq input plugin s3 output plugin zmq recv_string ▪ 1: s3 output plugin “write” , PutObject DeleteObject . ▪ 2: config , output plugin log level · . ▪ 3: output plugin input plugin receive timeout .
  61. 61. : Docker ▪ ▪ docker registry "latest" latest ▪ 2.0 2.0.1 tag ▪ ▪ ▪ docker 1.8 , CoreOS (rkt) ▪ /API
  62. 62. : ▪ Jupyter / ipython ▪ scalable ▪ Jupyter ▪ Unix ▪ ▪ : ▪ ▪ – Sorna ▪ .
  63. 63. CodeON
  64. 64. ▪ Container Resource Consolidation ▪ Kubernetes ▪ paxos raft agreement key-value store . ▪ Ingen REPL (read-evaluate-print loop) ▪ stdout/stderr streaming ▪ interactive plot ▪ image, sound multimedia
  65. 65. ▪ ▪ ▪ User profiler + backtracer + ML with back propagation ▪ ▪ d3.js fluid nonlinear navigation UI ▪ ▪ ▪ ▪ ▪
  66. 66. ▪ . ▪ ... + + … ▪ . ▪ ?
  67. 67. Lablup Inc. http://www.lablup.com

×