Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Python redis talk

5,576 views

Published on

This is a talk that I gave on July 20, 2012 at the Southern California Python Interest Group meetup at Cross Campus, with food and drinks provided by Graph Effect.

Published in: Technology
  • DOWNLOAD THAT BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book that can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer that is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story That Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money That the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths that Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Memory Improvement: How To Improve Your Memory In Just 30 Days, click here.. ➤➤ https://bit.ly/2GEWG9T
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Sharpen your mind with brain pill. learn more info.. ■■■ https://tinyurl.com/brainpill101
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Python redis talk

  1. Redis and Python by Josiah Carlson @dr_josiah dr-josiah.blogspot.com bit.ly/redis-in-action
  2. Redis and Python; Its PB & J time by Josiah Carlson @dr_josiah dr-josiah.blogspot.com bit.ly/redis-in-action
  3. What will be covered• Who am I?• What is Redis?• Why Redis with Python?• Cool stuff you can do by combining them
  4. Who am I?• A Python user for 12+ years• Former python-dev bike-shedder• Former maintainer of Python async sockets libraries• Author of a few small OS projects o rpqueue, parse-crontab, async_http, timezone-utils, PyPE• Worked at some cool places youve never heard of (Networks In Motion, Ad.ly)• Cool places you have (Google)• And cool places you will (ChowNow)• Heavy user of Redis• Author of upcoming Redis in Action
  5. What is Redis?• In-memory database/data structure server o Limited to main memory; vm and diskstore defunct• Persistence via snapshot or append-only file• Support for master/slave replication (multiple slaves and slave chaining supported) o No master-master, dont even try o Client-side sharding o Cluster is in-progress• Five data structures + publish/subscribe o Strings, Lists, Sets, Hashes, Sorted Sets (ZSETs)• Server-side scripting with Lua in Redis 2.6
  6. What is Redis? (compared to otherdatabases/caches)• Memcached o in-memory, no-persistence, counters, strings, very fast, multi-threaded• Redis o in-memory, optionally persisted, data structures, very fast, server-side scripting, single-threaded• MongoDB o on-disk, speed inversely related to data integrity, bson, master/slave, sharding, multi-master, server-side mapreduce, database-level locking• Riak o on-disk, pluggable data stores, multi-master sharding, RESTful API, server-side map-reduce, (Erlang + C)• MySQL/PostgreSQL o on-disk/in-memory, pluggable data stores, master/slave, sharding, stored procedures, ...
  7. What is Redis? (Strings)• Really scalars of a few different types o Character strings  concatenate values to the end  get/set individual bits  get/set byte ranges o Integers (platform long int)  increment/decrement  auto "casting" o Floats (IEEE 754 FP Double)  increment/decrement  auto "casting"
  8. What is Redis? (Lists)• Doubly-linked list of character strings o Push/pop from both ends o [Blocking] pop from multiple lists o [Blocking] pop from one list, push on another o Get/set/search for item in a list o Sortable
  9. What is Redis? (Sets)• Unique unordered sequence of character strings o Backed by a hash table o Add, remove, check membership, pop, random pop o Set intersection, union, difference o Sortable
  10. What is Redis? (Hashes)• Key-value mapping inside a key o Get/Set/Delete single/multiple o Increment values by ints/floats o Bulk fetch of Keys/Values/Both o Sort-of like a small version of Redis that only supports strings/ints/floats
  11. What is Redis? (Sorted Sets -ZSETs)• Like a Hash, with members and scores, scores limited to float values o Get, set, delete, increment o Can be accessed by the sorted order of the (score,member) pair  By score  By index
  12. What is Redis? (Publish/Subscribe)• Readers subscribe to "channels" (exact strings or patterns)• Writers publish to channels, broadcasting to all subscribers• Messages are transient
  13. Why Redis with Python?• The power of Python lies in: o Reasonably sane syntax/semantics o Easy manipulation of data and data structures o Large and growing community• Redis also has: o Reasonably sane syntax/semantics o Easy manipulation of data and data structures o Medium-sized and growing community o Available as remote server  Like a remote IPython, only for data  So useful, people have asked for a library version
  14. Per-hour and Per-day hit countersfrom itertools import imapimport redisdef process_lines(prefix, logfile): conn = redis.Redis() for log in imap(parse_line, open(logfile, rb)): time = log.timestamp.isoformat() hour = time.partition(:)[0] day = time.partition(T)[0] conn.zincrby(prefix + hour, log.path) conn.zincrby(prefix + day, log.path) conn.expire(prefix + hour, 7*86400) conn.expire(prefix + day, 30*86400)
  15. Per-hour and Per-day hit counters(with pipelines for speed)from itertools import imapimport redisdef process_lines(prefix, logfile): pipe = redis.Redis().pipeline(False) for i, log in enumerate(imap(parse_line, open(logfile, rb))): time = log.timestamp.isoformat() hour = time.partition(:)[0] day = time.partition(T)[0] pipe.zincrby(prefix + hour, log.path) pipe.zincrby(prefix + day, log.path) pipe.expire(prefix + hour, 7*86400) pipe.expire(prefix + day, 30*86400) if not i % 1000: pipe.execute() pipe.execute()
  16. Simple task queue - add/run itemsimport jsonimport redisdef add_item(queue, name, *args, **kwargs): redis.Redis().rpush(queue, json.dumps([name, args, kwargs]))def execute_one(queues): item = redis.Redis().blpop(queues, 30) name, args, kwargs = json.loads(item) REGISTRY[name](*args, **kwargs)
  17. Simple task queue - register tasksREGISTRY = {}def task(queue): def wrapper(function): def defer(*args, **kwargs): add_item(queue, name, *args, **kwargs) name = function.__name__ if name in REGISTRY: raise Exception( "Duplicate callback %s"%(name,)) REGISTRY[name] = function return defer if isinstance(queue, str): return wrapper function, queue = queue, default return wrapper(function)
  18. Simple task queue – register tasks@task(high)def do_something(arg): pass@taskdef do_something_else(arg): pass
  19. Cool stuff to do...• Reddit • Publish/Subscribe• Caching • Messaging• Cookies • Search engines• Analytics • Ad targeting• Configuration • Twitter management • Chat rooms• Autocomplete • Job search• Distributed locks • ...• Counting Semaphores• Task queues
  20. Thank you @dr_josiahdr-josiah.blogspot.com bit.ly/redis-in-action Questions?

×