AWS offers a broad range of purpose-built services, such as databases, analytics, search, and visualization. Attend this session to learn how you can integrate the fully-managed Redis service offered by AWS, which includes search and graph services to build modern applications.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Overview of Redis with Search and
Graph
Kevin McGehee
Sr. Software Development Engineer
AWS ElastiCache
D A T 3 3 4
Mohan Varthakavi
Sr. Manager, Software Development
AWS ElastiCache

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
Redis overview
Search Integration with Amazon Elasticsearch Service
Graph Integration with Amazon Neptune

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Purpose-built databases
• AWS builds purpose-built solutions to solve problems
• Some use cases can leverage multiple solutions together

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Redis engine
Fast
<1ms latency for most commands
Open source
Easy to learn
Highly available
Replication
Atomic operations
Supports transactions
In-memory key-value store
Powerful
~200 commands, Lua scripting,
Geospatial, Pub/Sub
Various data structures
Strings, lists, hashes, sets,
sorted sets, bitmaps, streams,
and HyperLogLogs
Persistence
Snapshots

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon ElastiCache for Redis
Fully-managed, Redis-compatible, in-memory data store in the cloud
Extreme performance
In-memory data store and
cache for sub-millisecond
response times
Fully managed
AWS manages all hardware
and software setup,
configuration, monitoring
Easily scalable
Read scaling with replicas
Write and memory
scaling with sharding
Non disruptive scaling
Redis-compatible
Redis 5 support
Redis clients compatible
Reliable
Multi-AZ
Deep monitoring
Automatic failover
Secure & compliant
Amazon Virtual Private Cloud
(Amazon VPC)
HIPAA, PCI, FedRAMP
Encryption at-rest and in-transit
Authentication

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Use Cases
Gaming
leaderboards
Chat apps
Caching
Session
store
Machine learning
Real-time
analytics
Media
streaming
Geospatial
Message queues

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Open source
Benefits of Amazon Elasticsearch Service
Drop-in replacement with no need to
learn new APIs or skills
Resize your cluster with a few
clicks or a single API call
Deploy into your VPC and
restrict access using security
groups and AWS Identity and Access
Management (IAM) policies
Replicate across Availability Zones,
with monitoring and
automated self-healing
Leverage other AWS services for
seamless data ingestion, security,
auditing and orchestration
Easy to use
Deploy a production-ready
Elasticsearch cluster in minutes
Scalable
Secure Highly available Integrated

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Use case #1: Log analytics ingestion
• Elasticsearch used to query structured application logs
• Ingestion pipeline can slow down during surge of log events
• Redis can be used as a fast, in-memory message buffer
• Reduces scaling requirements of downstream components

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Message queueing for log analytics
Logstash
Buffered
log data
( Redis lists )
Search & log analytics
Mobile client
Mobile client
Mobile client
Mobile client
Application
Frontend
Pull batches Processed batch
local batchsize = tonumber(ARGV[1])
local result = redis.call('lrange', KEYS[1], 0, batchsize)
redis.call('ltrim', KEYS[1], batchsize + 1, -1)
return result
Push log data
Administrators

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Use case #2: Application log analytics
• Process Elasticsearch logs and push application data into Redis
• Application can query Redis directly
• Advantage of sub-ms access latency with significant throughput

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Leaderboards using log analytics
Logstash
Application data
( sorted sets )
Search & log analytics
Mobile client
Mobile client
Mobile client
Mobile client
Application
Frontend
Existing
pipeline
Update leaders Poll event data
ZINCRBY topreviewers 5 bob123
ZINCRBY topreviewers 1 alicerules0
...
Query leaders
> ZREVRANGEBYSCORE topreviewers inf 0
1) bob123
2) alicerules0

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Use case #3: Query caching
• Reduce query latency
• Sub-ms latency for query caching of slower Elasticsearch queries
• Increase overall throughput
• Increase throughput capacity without provisioning more Elasticsearch nodes
• Reduce Elasticsearch provisioning if bottlenecked on throughput

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Query caching Elasticsearch results
Frontend
application
Cached
search
results
Search & log analytics
(1)
Check cache
Query
Elasticsearch
data
(2)
Cache miss:
Issue query
(3)
Cache miss:
Cache result
GET [query]
SET [query] [result]

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Neptune
Fast, reliable graph database built for the cloud
Open graph APIs
Gremlin and SPARQL
support
High performance &
scalability
Purpose-built high
performance graph database
engine
Automatically scaling storage
and easily scaling compute
and memory
Availability & durability
Multi-AZ deployments with
read replicas
Instance monitoring &
repair
Continuous Incremental
Backups
Cost-effective
Pay for what you use
Security
Amazon VPC network
isolation
IAM access control
Encryption at rest using
AWS Key Management
Service (AWS KMS)
Fully managed
Easy to use
Amazon CloudWatch integration
Automatic patching

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
User recommendation Graph
• Store restaurant recommendations and friendship connections
RATED
rating: 5
RATED
rating: 7
name: Bob
id: 20123194
User
name: Denny’s
id: 1125923
name: Fancy Grill
id: 2345678
Restaurant
name: Alice
Id: 11041235
User
name: Fish Please
id: 3315963
RATED
rating: 9
RATED
rating: 2
Restaurant Restaurant
FRIENDS_WITH

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Use case #1: Data enrichment
• User wants to see nearby restaurants that friends recommended
• Redis natively supports geospatial indexes while Neptune does not
• Query Neptune for friend relationships and Redis for geospatial
relationships

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Geospatial enrichment
> GEORADIUS geo:restaurantids 47.6062 122.3321 10 mi
1) "2345678"
2) "2788539"
3) "3315963"
Frontend
application
Restaurant
geospatial
data
Friend
relationship
data
Query friend
recommendations
Query
nearby
restaurants
Query nearby
restaurants

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Use case #2: Pagination of Neptune results
• User wants to paginate graph query
• Neptune supports pagination, but needs to rerun query
• Redis supports lists and sorted sets in-memory
• An entire Neptune result set can be computed once, then fetched from cache over time

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Neptune pagination offload
Application
Query set
cache
Query entire
result set
Cache
result set
(first request)
(first request)
Retrieve page
Paginated queries
LPUSH [session id] [data page 1]
LPUSH [session id] [data page 2]
...
EXPIRE [session id] [seconds]
LINDEX [session id] [n]

24. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kevin McGehee
mcgehee@amazon.com
Mohan Varthakavi
varthaka@amazon.com

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.