Pushing Python: Building a High Throughput, Low Latency System
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Pushing Python: Building a High Throughput, Low Latency System

  • 16,817 views
Uploaded on

 

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
16,817
On Slideshare
457
From Embeds
16,360
Number of Embeds
13

Actions

Shares
Downloads
5
Comments
0
Likes
0

Embeds 16,360

http://www.tellapart.com 11,080
http://tellaparteng.tumblr.com 5,184
http://tellapart.red8dev.com 62
http://tellapart.dev 16
https://assets.txmblr.com 6
http://feedly.com 3
https://twitter.com 3
http://www.tumblr.com 1
http://www.soso.com 1
http://webcache.googleusercontent.com 1
http://tellapart.wpengine.com 1
http://www.feedspot.com 1
http://www.tellaparteng.tumblr.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Kevin  Ballard   SFpython.org   2014-­‐03-­‐12  
  • 2. kevin@   tellapart.com   Introductions
  • 3. Taba •  Distributed event aggregation service import taba ... taba.RecordValue(‘winning_bid_price’, wincpm) ... $ taba-cli aggregate winning_bid_price {“name”: “winning_bid_price”, “10m”: {“count”: 14709, “total”: 5836.4}, “percentiles”: [0.07 0.16 0.32 0.84 1.33 8.03]}
  • 4. Taba +10,000,000   events/sec   +50,000   metrics   +1,000   clients   +100   processors  
  • 5. GET THE DATA MODEL RIGHT Lesson #1
  • 6. Data Model
  • 7. Data Model Event:  (‘bid_cpm’,  ‘Counter’,  time(),  0.233)       State:             Aggregate:  {“10m”:  43.9,  “1h”:  592.22}    
  • 8. Data Model
  • 9. Data Model
  • 10. Data Model
  • 11. STATE IS HARD Lesson #2
  • 12. Centralizing State
  • 13. GENERATORS + GREENLETS = AWESOME Lesson #3
  • 14. Asynchronous Iterator •  JIT processing •  Automatically switches through I/O
  • 15. CPYTHON SUFFERS FROM MEMORY FRAGMENTATION Lesson #4
  • 16. Fragmentation •  Fragmentation is when a process’s heap is inefficiently used. •  The GC may report a low memory footprint, but the OS reports a much larger RSS.
  • 17. Fragmentation
  • 18. Fragmentation
  • 19. Fragmentation
  • 20. Fragmentation
  • 21. Fragmentation
  • 22. Hybrid Memory Management •  Use Cython to allocate page-sized blocks of pointers into incoming chunk •  Hand-off the whole thing to the CPython memory manager •  Whole thing gets deallocated at once
  • 23. Hybrid Memory Management
  • 24. Hybrid Memory Management
  • 25. Hybrid Memory Management
  • 26. Ratcheting •  Ratcheting is a pathological case of Fragmentation, caused by the fact that the heap must be contiguous*: •  It’s a limitation of CPython that it cannot compact memory (mostly due to extensions).
  • 27. Ratcheting •  Ratcheting is a pathological case of Fragmentation, caused by the fact that the heap must be contiguous*: •  It’s a limitation of CPython that it cannot compact memory (mostly due to extensions).
  • 28. Ratcheting •  Ratcheting is a pathological case of Fragmentation, caused by the fact that the heap must be contiguous*: •  It’s a limitation of CPython that it cannot compact memory (mostly due to extensions).
  • 29. Ratcheting •  Ratcheting is a pathological case of Fragmentation, caused by the fact that the heap must be contiguous*: •  It’s a limitation of CPython that it cannot compact memory (mostly due to extensions).
  • 30. Ratcheting •  Avoid persistent objects •  Sockets are common offenders •  Anything that has to be persistent should be created at application startup, before processing data •  Avoid letting the heap grow in the first place
  • 31. fin. github.com/tellapart/taba       kevin@tellapart.com      |    @misterkgb     We’re  Hiring!      tellapart.com/careers