Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

TensorFlow Encrypted + +Redis Streams + RedisAI - Advanced Spark and TensorFlow Meetup May 20 2019

65 views

Published on


Title

TF Encrypted + RedisAI + Redis Streams

Video: https://youtu.be/zYfS9SVeqSU

Agenda

1. Redis AI + TensorFlow + PyTorch + Redis Streams

Speaker: Dave Nielsen, Head of Community and Ecosystem @ Redis Labs
(https://www.linkedin.com/in/dnielsen)

2. TensorFlow Encrypted

Machine learning applied to healthcare or otherwise sensitive data may be blocked if privacy isn’t adequately addressed. In this presentation we give an overview of how the TF Encrypted open source library can be used to make predictions and fit models on encrypted data in TensorFlow, without first needing to master advanced cryptography.

Speaker: Morten Dahl, Research Scientist @ DropOut Labs
(https://www.linkedin.com/in/mortendahlcs)



Date/Time

9-10am US Pacific Time (Third Monday of Every Month)

** RSVP & LOGIN HERE **

Eventbrite:

https://www.eventbrite.com/e/1-hr-fre...

Meetup:

https://www.meetup.com/Advanced-Spark...

Zoom:

https://zoom.us/j/690414331
Webinar ID: 690 414 331

Phone:

+1 646 558 8656 (US Toll) or +1 408 638 0968 (US Toll)

Related Links

PipelineAI Home: https://pipeline.ai

PipelineAI Community Edition: https://community.pipeline.ai

PipelineAI GitHub: https://github.com/PipelineAI/pipeline

PipelineAI Quick Start: https://quickstart.pipeline.ai

Advanced Spark and TensorFlow Meetup (SF-based, Global Reach): https://www.meetup.com/Advanced-Spark...

YouTube Videos: https://youtube.pipeline.ai

SlideShare Presentations: https://slideshare.pipeline.ai

Slack Support:
https://joinslack.pipeline.ai

Web Support and Knowledge Base: https://support.pipeline.ai

Email Support: help@pipeline.ai

Published in: Software
  • Be the first to comment

  • Be the first to like this

TensorFlow Encrypted + +Redis Streams + RedisAI - Advanced Spark and TensorFlow Meetup May 20 2019

  1. 1. Redis Data Structures and Redis Streams MAY 2019 | DAVE NIELSEN
  2. 2. 2 • Why is Redis so popular? • Redis Data Structures (Cache, User Sessions, etc) • New Use Cases - Microservices Framework • What is a data stream? • What are the typical challenges you face while managing data streams? • What is Redis Stream? • How Redis Stream addresses the challenges? • How to use Redis Streams? Agenda
  3. 3. Who is Redis Labs? 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case redis.io The home of open source Redis, and commercial provider of Redis Enterprise technology, platform, products & services. redislabs.com/redis-enterprise/ Redis Enterprise Cloud redislabs.com/redis-cloud - Redis Cloud Essentials up to 30mb FREE! - Redis Cloud Pro
  4. 4. 4 Examples
  5. 5. Redis Top Redis Differentiators 5 Simplicity ExtensibilityPerformance NoSQL Benchmark 1 Redis Data Structures 2 3 Redis Modules Lists Hashes Bitmaps Strings Bit field Streams Hyperloglog Sorted Sets Sets Geospatial Indexes
  6. 6. 6 Possible Data Stream Use Cases Top Redis Use Cases
  7. 7. Simplicity: Data Structures - Redis’ Building Blocks Lists [ A → B → C → D → E ] Hashes { A: “foo”, B: “bar”, C: “baz” } Bitmaps 0011010101100111001010 Strings "I'm a Plain Text String!” Bit field {23334}{112345569}{766538} Key 7 2 ”Retrieve the e-mail address of the user with the highest bid in an auction that started on July 24th at 11:00pm PST” ZREVRANGE 07242015_2300 0 0= Streams à{id1=time1.seq1(A:“xyz”, B:“cdf”), d2=time2.seq2(D:“abc”, )}à Hyperloglog 00110101 11001110 Sorted Sets { A: 0.1, B: 0.3, C: 100 } Sets { A , B , C , D , E } Geospatial Indexes { A: (51.5, 0.12), B: (32.1, 34.7) }
  8. 8. 8 Redis Data Structures 1. Strings - Cache SQL queries, pages, fast data 2. Bitmaps - Store 1s/0s (Online/Offline) like Pinterest 3. Bit Fields - Store arrays of complex numbers 4. Hashes - Store record-like info (user sessions, favorites, etc.) 5. Lists - Store ordered data (for ingesting data) 6. Sets - Store unique/unordered data (tags, unique IPs, etc.) 7. Sorted Sets - Store real-time sorted data like a MMPORG Leaderboard 8. Geospacial Indexes - Store real-time location (Uber, Lyft, etc.) 9. Hyperloglog - Store probabilistic data (Google Search count) 10. Streams - Store and share event data (similar to Kafka) Lists Hashes Bitmaps Strings Bit fields Streams Hyperloglog Sorted Sets Sets Geospatial Indexes
  9. 9. What is a cache?
  10. 10. • GET mykey • If mykey = (nil) then – run SQL Query – SET mykey = “query results” EX 5 • GET mykey • For more, visit redis.io/commands/set Redis cache is very easy …
  11. 11. Redis Streams
  12. 12. 12 What is a Data Stream?
  13. 13. Simplest form of a Data Stream 13 ConsumerProducer
  14. 14. But Often You Have M Producers and N Consumers 14 Producer 2 Producer m Producer 1 Producer 3 Consumer 1 Consumer n Consumer 2 Consumer 3
  15. 15. It’s a bit more complex 15
  16. 16. Data Stream Components 16 Ingest Stream Message Broker or Stream Processor Output Stream Analytics Collect large volume of data arriving in high velocity Store the data temporarily Transform incoming data into one or more formats as required by the consumers Push the data to the consumers Allow backward looking queries to pull data Store the data and perform analytical operations such as predictive analytics, ML training, ML classification and categorization, recommendations, pattern matching, etc. IoT Activity logs Messages
  17. 17. 17
  18. 18. Redis Supports All the Functions of a Data Stream 18 Ingest Stream Message Broker or Stream Processor Output Stream Analytics Redis • Streams • Pub/Sub • Lists • Sorted Sets • Streams • Pub/Sub • Lists • Streams • Sorted Sets + • Spark Connector • ODBC/JDBC Connector • Sorted Sets • Sets • Geo • Bitfield + • RedisGraph • RediSearch
  19. 19. 19
  20. 20. Understanding Redis Streams 20
  21. 21. Let’s look at the challenges first 21
  22. 22. How to Support a Variety of Consumers 22 Analytics Data Backup Consumers Producer Messaging Real-time Real-time or Periodic Lookup Periodic Read
  23. 23. The Catch Up Problem 23 Image ProcessorProducer Consumption Rate: 100/secArrival Rate: 500/sec Redis Stream Fast Slow Backlog
  24. 24. How to Address The Catch Up Problem? 24 Producer Image Processor Arrival Rate: 500/sec Consumption Rate: 500/sec Image Processor Image Processor Image Processor Image Processor Redis Stream Scale Out New Problem: Mutual Exclusivity
  25. 25. Data Recovery from Consumer Failures 25 Producer Image Processor Arrival Rate: 500/sec Consumption Rate: 500/sec Image Processor Image Processor Image Processor Image Processor Redis Stream Scale Out Yet Another Problem: Failure Scenarios
  26. 26. Connection Failures 26 Producer Image Processor Image Processor Image Processor Image Processor Image Processor Redis Stream The solution must be resilient to failures
  27. 27. Here comes Redis Streams 27
  28. 28. What is Redis Streams? 28 Pub/Sub Lists Sorted Sets It is like Pub/Sub, but with persistence It is like Lists, but decouples producers and consumers It is like Sorted Sets, but asynchronous + • Lifecycle management of streaming data • Built-in support for timeseries data • A rich choice of options to the consumers to read streaming and static data • Super fast lookback queries powered by radix trees • Automatic eviction of data based on the upper limit
  29. 29. 1. It enables asynchronous data exchange between producers and consumers 29 MessagingProducer Consumer
  30. 30. 30 Analytics Data Backup Consumers Producer Messaging 2. You can consume data in real-time as it arrives or lookup historical data
  31. 31. 31 Producer Image Processor Arrival Rate: 500/sec Consumption Rate: 500/sec Image Processor Image Processor Image Processor Image Processor Redis Stream 3. With consumer groups, you can scale out and avoid backlogs
  32. 32. 32 Classifier 1 Classifier 2 Classifier n Consumer Group XREADGROUP XREAD Consumers Producer 2 Producer m Producer 1 Producer 3 XADD XACK Deep Learning-based Classification Analytics Data Backup Messaging 4. Simplify data collection, processing and distribution to support complex scenarios
  33. 33. Quick Reference Guide: https://redislabs.com/docs/getting-started-redis-streams/ Redis Streams Commands 33
  34. 34. Commands: XADD, XREAD 34 Asynchronous producer-consumer message transfer MessagingProducer Consumer XADD XREAD XADD mystream * name Anna XADD mystream * name Bert XADD mystream * name Cathy XREAD COUNT 100 STREAMS mystream 0 XREAD BLOCK 10000 STREAMS mystream $
  35. 35. Commands: XRANGE, XREVRANGE 35 Lookup historical data Analytics Data Backup Consumers Producer Messaging XADD XREAD XRANGE XREVRANGE XRANGE mystream 1518951123450-0 1518951123460-0 COUNT 10 XRANGE mystream - + COUNT 10
  36. 36. Scaling out using consumer groups 36 Producer Image Processor Arrival Rate: 500/sec Consumption Rate: 500/sec Image Processor Image Processor Image Processor Redis Stream
  37. 37. Scaling out using consumer groups 37 Producer Image Processor Arrival Rate: 500/sec Consumption Rate: 500/sec Image Processor Image Processor Image Processor Redis Stream Consumer Group Consumer 1 Consumer 2 Consumer 3 Consumer n Unconsumed List Consumers XADD XGROUP XREADGROUP XACK XPENDING XCLAIM
  38. 38. Command: XGROUP CREATE 38 Create a consumer group Unconsumed List mygroup Alice Bob edcba mystream edcba App A App B XGROUP CREATE XGROUP CREATE mystream mygroup $ MKSTREAM
  39. 39. Command: XREADGROUP 39 Read the data Unconsumed List mygroup Alice Bob App A App B a ed cb XREADGROUP XREADGROUP GROUP mygroup COUNT 2 Alice STREAMS mystream > XREADGROUP GROUP mygroup COUNT 2 Bob STREAMS mystream >
  40. 40. Command: XACK 40 Consumers acknowledge that they consumed the data Unconsumed List mygroup Alice Bob App A App B a cb XACK XACK mystream mygroup 1526569411111-0 1526569411112-0
  41. 41. Command: XREADGROUP 41 Repeat the cycle Unconsumed List mygroup Alice Bob App A App B a cb XREADGROUP XREADGROUP GROUP mygroup COUNT 2 Alice STREAMS mystream >
  42. 42. How to claim the data from a consumer that failed while processing the data? 42 Unconsumed List mygroup Alice Bob App A App B cb Unconsumed List mygroup Alice Bob App A App B cb XCLAIM
  43. 43. Commands: XPENDING, XCLAIM 43 Claim pending data from other consumers Unconsumed List mygroup Alice Bob App A App B cb XPENDING mystream mygroup - + 10 Bob XCLAIM mystream mygroup Alice 0 1526569411113-0 1526569411114-0
  44. 44. Now is the time to 44
  45. 45. Sample Use Case: Social Media Analytics 45 a b x Language classifier XREADGROUP Real-time reports XREAD Sentiment analyzers Consumer is x times slower than the producer Location classifier Influencer classifierRedis Streams
  46. 46. Demo 46 Redis Streams Twitter Ingest Stream Influencer Classifier https://github.com/redislabsdemo/IngestRedisStreams
  47. 47. Java Jedis https://github.com/xetorthio/jedis Lettuce https://lettuce.io/ Python redis-py https://github.com/andymccurdy/redis-py JavaScript ioredis https://github.com/luin/ioredis node_redis https://github.com/NodeRedis/node_redis .NET StackExchange.Redis https://github.com/StackExchange/StackExchange.Redis Go redigo https://github.com/gomodule/redigo go-redis https://github.com/go-redis/redis C Hiredis https://github.com/redis/hiredis PHP predis https://github.com/nrk/predis phpredis https://github.com/phpredis/phpredis Ruby redis-rb https://github.com/redis/redis-rb Redis Clients that Support Redis Streams 47
  48. 48. Questions? dave@redislabs.com @davenielsen Thanks to @roshankumar for the demo 48

×