Presented at All Things Open 2022 Presented by Garth Henson Title: Telling Stories with Big Data Abstract: In a world where data is king, we are constantly bombarded with infographics, charts, numbers, and other data that may or may not actually make sense. Access to massive volumes of data is at our fingertips, but it isn't always clear how to tell a story that is both accurate and relevant with it. Join us for this session as we build some tools to help us manage our big data streams using open-source technologies. We will discuss important data topics such as partitioning, proper indexing, and redundancy to enable you to confidently tell a story with both accurate and relevant data you have collected!

1. Garth Henson, November 2022
Telling Stories with Big Data
Striving for relevance and accuracy

2. Telling Stories with Big Data Striving for relevance and accuracy
Garth Henson, November 2022
Prerequisites
• GitHub access
• Docker (with docker-compose) installed
• Favorite IDE
Setup
Clone the project repo
https://github.com/guahanweb/twitter-relay-consumer
Create an environment
fi
le at .docker.env
LOG_LEVEL=debug
REDIS_HOST=redis
REDIS_PORT=6379
RELAY_AUTH_TOKEN=<token>
RELAY_AUTH_URL=http://<ip-address>:3000/connect
RELAY_WEBSOCKET_URL=ws://<ip-address>:3000/ws
Run the consumer locally
$ docker-compose up

3. Lucas
fi
lm
Disney Streaming
Microsoft
The Walt Disney Company
Amazon
Synacor, Inc.
@guahanweb
Garth Henson

4. Understanding Data

5. Relevant? Accurate?

6. High Level Data Analysis

7. Data Partitioning (i.e., by time)

8. Processing Data in Real Time

9. Our Use Case: Twitter data processing at scale

10. Twitter Stream
a study on data discovery

11. Redis Data Type: Sorted Sets

12. Preprocessing data: partitions and indexes
• Step 1: register rules with Twitter (watch the @Lucas
fi
lm user)
• Step 2: connect consumer to Twitter data stream (stream only
contains matching data sequentially)
• Step 3: monitor each event and process data accordingly
•Metadata for unique authors and tags
•Time partitions split into tweets, retweets, quotes, and replies
• Step 4: store partitioned data in Redis

13. Why time-based partitions?
• Finite partition life (aids in cleaning up old data)
• Supports multiple inbound data streams
• Real-time feed
• Slow or latent event handling
• Manual back
fi
ll of data queries
• Heavy query caching potential (most recent X hours)
• Reduced overhead of aggregate queries
• Cache summary results by partition
• Purge partition cache(s) on write
• Data loss (even a full partition) is manageable

14. Indexes are driven by your data retrieval model
(What do you need to know?)
Aggregate by author Aggregate by hashtag
Indexes may contain redundant data;
data normalization is another talk...

15. Querying the Data
What were the top 10 daily hashtags for the first week of May 2022?

16. Step 1: union hourly partitions by date
ZUNIONSTORE
20220501:tweet:tags
24
2022050100:tweet:tags
2022050101:tweet:tags
…
• Identify the time periods (daily)
• Calculate the partitions within each data point
(hours 00-23)
• UNION the values across all partitions for a
count

17. Step 2: create a scoring algorithm (what does TOP mean?)
• Hashtags used across tweets, retweets,
quotes, and replies
• Our algorithm assigns a value to each type:
• Tweets: 100 points
• Replies: 50 points
• Quotes: 25 points
• Retweets: 1 point
• UNION your previously calculated daily
summary across indexes for the
fi
nal daily
score

18. Step 3: script it together with Node.js + Lua for Redis
Create a function specifying algorithmic weights and pass it to a Lua script

19. Step 3: script it together with Node.js + Lua for Redis
Retrieve the arguments passed in via Node.js Set up a table with all matching partitions, by type

20. Step 3: script it together with Node.js + Lua for Redis
UNION all partitions, weighted by type JSON encode the response to the client

21. Redis Considerations
Why bother with Lua scripting?
• Atomic actions - all operations within the script are synchronous
• Performance - Lua scripts are evaluated in memory and can be pre-cached
• Separation of concerns - complex Redis interactions in one place
• ...

22. Redis Considerations
What should we keep in mind?
• Data types and design
• Our example in this talk uses primarily SORTED SETS
• GEOSEARCH available since 6.2
• Be aware when running in cluster mode
• Index hashing is unique to your shard
• Cannot dynamically reference keys from Lua
• Consider redundancy and syncing

23. Telling Stories with Big Data Striving for relevance and accuracy
Garth Henson, November 2022
Prerequisites
• GitHub access
• Docker (with docker-compose) installed
• Favorite IDE
Setup
Clone the project repo
https://github.com/guahanweb/twitter-relay-consumer
Create an environment
fi
le at .docker.env
LOG_LEVEL=debug
REDIS_HOST=redis
REDIS_PORT=6379
RELAY_AUTH_TOKEN=<token>
RELAY_AUTH_URL=http://<ip-address>:3000/connect
RELAY_WEBSOCKET_URL=ws://<ip-address>:3000/ws
Run the consumer locally
$ docker-compose up

24. High Level Architecture

25. Twitter Event Structure

26. Hands-on
Distribute access key and IP address

27. Hands-on
Overview of currently configured rules and message handling

28. Hands-on
Current state of reporting (and how to extend it)

29. Hands-on
How could we track and report ATO rule matches by minute?

30. Thank you
Questions, comments, recommendations, corrections...