Near is an ambient intelligence platform that uses massive data and AI to understand consumer behavior in smart environments. It processes 150K events per second globally and manages 1.5 billion user profiles and nearly 3000 audience segments. Redis powers Near's ID system, gridstore, countstore, and real-time bidding engine by enabling scaling, exploration of data structures, and development of reliable real-time and batch pipelines. Near is evaluating enterprise Redis and reorganizing some data stores to further optimize its platform.

1. Title
Visualize, Engage and Analyze Audience
https://near.co/products/allspark
2.0
REDIS POWERS NEXT-GEN
AMBIENT INTELLIGENCE PLATFORM
REDISCONF 2017
Madhu Therani, CTO
madhu@near.co

2. www.near.co | CONFIDENTIAL**
Company Overview
www.near.co
Unify streaming and static data from multiple
sources to map out “consumer” journeys
Near is a “Ambient Intelligence” platform
that uses massive data and artificial intelligence
to understand consumers in smart
environments
Near’s Mission

3. www.near.co | CONFIDENTIAL**
Company Overview
www.near.co
Varied Data Sources – Connected via Location
Bringing massive data into a unified platform
for the most accurate understanding of consumer behavior
Location Data
SINGLE IDENTIFIER

4. Title
Visualize, Engage and Analyze Audience
https://near.co/products/allspark
2.0
Our Current Platform
Ad
Engine
Places
API
Near
Places
DB
ID System
Location
Refinement
Batch
Analytics
To External
Systems
Profile
Cache
Count
Store
Audience
Analytics
API
Audience
Rules
API
Campaign
Analytics
API
Sources
Profile
Store
Product
API
People
API
ALLSPARK
DaaS APIs
Deduping
ID Linkage
Realtime
Audience
Segmentation
Spatial
Counts
Model-based
Profile
Realtime
Analytics

5. Title
Visualize, Engage and Analyze Audience
https://near.co/products/allspark
2.0
Our SaaS Product - AllSpark
Enables Definition, Curation of and Engagement with “Audiences”

6. Title
Visualize, Engage and Analyze Audience
https://near.co/products/allspark
2.0
Our requirements - 24 months ago
Scale from 20K events per sec to 200K events per sec with & without location
Map to the physical world - Gather/index geo information and associated activity
to “physical spaces”
Maintain “user” level aggregated identity - accumulate, summarize, analyze,
expire
Evaluate utility of inferences in realtime and batch mode

7. Title
Visualize, Engage and Analyze Audience
https://near.co/products/allspark
2.0
Where are we now?
Process 150K events per sec globally, 6-10 TB of data everyday
Global footprint - Washington DC, Amsterdam, Hong Kong, Singapore, Tokyo
Geo-data from 44 countries
1.5 Billion profiles actively managed
Allspark actively manages nearly 3000 audience segments, 1000+ campaigns
per quarter

8. Title
Visualize, Engage and Analyze Audience
https://near.co/products/allspark
2.0
Role of Redis
Powered incremental development
Scaling of key subsystems
Allowed “exploration” of data structures as platform requirements evolved
Enabled the development of a reliable realtime + batch pipeline - Storm/MR

9. Title
Visualize, Engage and Analyze Audience
https://near.co/products/allspark
2.0
Key Subsystem - ID System
Powers ID generation for profiles - across sources at different data centers
Deduplication
Opt-out management
ID unification - maintain multiple channel-specific IDs for system-wide Allspark ID
Metrics - what is the traffic like ? location vs non-location? Source mix ?

10. Title
Visualize, Engage and Analyze Audience
https://near.co/products/allspark
2.0
Key Subsystem - GridStore
Measuring physical pings at scale
Every point is an HLL
HLL by hour/days
For all countries
At multiple resolutions

11. Title
Visualize, Engage and Analyze Audience
https://near.co/products/allspark
2.0
Key Subsystem - CountStore
Counts about Audience Groups
Powers varied kind of demographic and behavior analysis - what
interests/behaviors
Spatio-temporal distribution - What other locales were visited in a given time
period?

12. Title
Visualize, Engage and Analyze Audience
https://near.co/products/allspark
2.0
Key Subsystem - RealTime Bidding Engine
Powering Programmatic Advertising
Metrics on a billion+ events in a day
Caches - bid cache, event cache, ad cache
Realtime status for various kinds of optimization

13. Title
Visualize, Engage and Analyze Audience
https://near.co/products/allspark
2.0
Other Usecases
Realtime analytics across pipelines
Caches across pipeline - Audience analytics
Powering model-as-a-service - metrics for data science
Shared state across both realtime and batch pipelines

14. Title
Visualize, Engage and Analyze Audience
https://near.co/products/allspark
2.0
Open source Redis deployment scale
60+ high-end servers
RAM requirements range from 128 GB to 512 GB
DBs are clustered per sub-system
Backup and maintenance with manual processes

15. Title
Visualize, Engage and Analyze Audience
https://near.co/products/allspark
2.0
Ad hoc Issues
Backups - snapshots versus AOF - how to evaluate
Replication across DCs
Key semantics and management - Key definition embeds a lot of ad hoc
meaning

16. Title
Visualize, Engage and Analyze Audience
https://near.co/products/allspark
2.0
Redis Benefits
Enabled incremental development
Externalization of core data structures - developers of varying ability can develop
scalable systems
Developing systems in multiple languages
Sweet spot between - relational dbs (mysql) and nosql (couch/mongo) - build
using Redis - then decide

17. Title
Visualize, Engage and Analyze Audience
https://near.co/products/allspark
2.0
What next
Evaluating Enterprise Redis
Reorganizing some of other datastores - KyotoTycoon, ElasticSearch, Mongo,
Cassandra, Hbase
ML on the edge - Realtime analytics on samples - alert generation

18. Title
Thank You
Acknowledgements
NearNEar Tech Team
NEar Tech Team
Near’s Tech Team in Bangalore & SFO