The slides we used at the first meetup hosted at Redis Labs' TLV offices :)
Touches on some of the more notable user-facing functionality in the newest Redis version, as well as interesting internal optimizations with major gains.
#RedisTLV: www.meetup.com/Tel-Aviv-Redis-Meetup/events/227594422/
Leveraging Probabilistic Data Structures for Real Time Analytics with Redis M...Itamar Haber
An introduction to Redis, Redis' new modules API and probabilistic data structures (PDS). Budding data scientists and big-data gurus use PDSs for estimating that which is difficult to be accurately counted.
Starting with v4, modules hold a promise for changing how Redis is used and developed for. Enabling custom data types and commands, Redis Modules build upon and extend the core functionality to handle any use case.
The video of the webinar given with these slides is at: https://youtu.be/EglSYFodaqw
An overview and discussion on indexing data in Redis to facilitate fast and efficient data retrieval. Presented on September 22nd, 2014 to the Redis Tel Aviv Meetup.
A list of all URLs in the deck is at: https://gist.github.com/itamarhaber/87e8c8c7126fbfb3f722
A lightening talk filled to the brim with knowledge and tips about Redis, data structures, performance, RAM and tips to take Redis to the max
An insight into NoSQL solutions implemented at RTV Slovenia and elsewhere, what problems we are trying to solve and an introduction to solving them with Redis.
Talk given at #wwwh @ Ljubljana, 30.1.2013 by me, Tit Petric
Slides from my talk on Redis at Human talks nantes http://humantalks.com/cities/nantes/events/136/talks/414-introduction-a-redis
Redsmin - Fully featured GUI for Redis: https://redsmin.com @redsmin
RedisWeekly - http://redisweekly.com @redisweekly
Developing a Redis Module - Hackathon KickoffItamar Haber
Slides deck for kicking off Redis Labs' Modules Hackathon - https://www.hackerearth.com/sprints/redislabs-hackathon-global
Video of the webinar is at: https://youtu.be/LPxx4QPyUPw
Leveraging Probabilistic Data Structures for Real Time Analytics with Redis M...Itamar Haber
An introduction to Redis, Redis' new modules API and probabilistic data structures (PDS). Budding data scientists and big-data gurus use PDSs for estimating that which is difficult to be accurately counted.
Starting with v4, modules hold a promise for changing how Redis is used and developed for. Enabling custom data types and commands, Redis Modules build upon and extend the core functionality to handle any use case.
The video of the webinar given with these slides is at: https://youtu.be/EglSYFodaqw
An overview and discussion on indexing data in Redis to facilitate fast and efficient data retrieval. Presented on September 22nd, 2014 to the Redis Tel Aviv Meetup.
A list of all URLs in the deck is at: https://gist.github.com/itamarhaber/87e8c8c7126fbfb3f722
A lightening talk filled to the brim with knowledge and tips about Redis, data structures, performance, RAM and tips to take Redis to the max
An insight into NoSQL solutions implemented at RTV Slovenia and elsewhere, what problems we are trying to solve and an introduction to solving them with Redis.
Talk given at #wwwh @ Ljubljana, 30.1.2013 by me, Tit Petric
Slides from my talk on Redis at Human talks nantes http://humantalks.com/cities/nantes/events/136/talks/414-introduction-a-redis
Redsmin - Fully featured GUI for Redis: https://redsmin.com @redsmin
RedisWeekly - http://redisweekly.com @redisweekly
Developing a Redis Module - Hackathon KickoffItamar Haber
Slides deck for kicking off Redis Labs' Modules Hackathon - https://www.hackerearth.com/sprints/redislabs-hackathon-global
Video of the webinar is at: https://youtu.be/LPxx4QPyUPw
Provides an overview of Redis which is a Key Value NoSQL database and the different data types it supports. Also shows how to use Redis Client API from node.
Redis - for duplicate detection on real time streamCodemotion
Roberto "frank" Franchini presenta a Codemotion Techmeetup Torino Redis, un data structure server che può utilizzare come chiavi stringhe, hashes, lists, sets, sorted sets, bitmaps e hyperloglogs
.
Redis Use Patterns (DevconTLV June 2014)Itamar Haber
An introduction to Redis for the SQL practitioner, covering data types and common use cases.
The video of this session can be found at: https://www.youtube.com/watch?v=8Unaug_vmFI
Power to the People: Redis Lua ScriptsItamar Haber
Redis is the Sun.
Earth is your application.
Imagine that the Moon is stuck in the middle of the Sun.
You send non-melting rockets (scripts) with robots
(commands) and cargo (data) back and forth…
Introduction to Sqoop | Big Data Hadoop Spark Tutorial | CloudxLabCloudxLab
Big Data with Hadoop & Spark Training: http://bit.ly/2kBMiEt
This CloudxLab Understanding Sqoop tutorial helps you to understand Sqoop in detail. Below are the topics covered in this tutorial:
1) Introduction to Sqoop
2) Sqoop Import - MySQL to HDFS
3) Sqoop Import - MySQL to Hive
4) Sqoop Import - MySQL to HBase
5) Sqoop Export - Hive to MySQL
New Indexing and Aggregation Pipeline Capabilities in MongoDB 4.2Antonios Giannopoulos
MongoDB 4.2 comes GA soon delivering some amazing new features on multiple areas. In this talk, we will focus on the new capabilities of the aggregation framework. We are going to cover the new operators and expressions. At the same time, we will explore how updates commands can now use the aggregation framework operators. We are also going to present aggregation framework improvements focusing on the on-demand materialized views. Finally, we are going to explore the wildcard indexes introduced in MongoDB 4.2 and how they change the way we design documents and build queries/aggregations. We will also make a reference to the new index build system.
Provides an overview of Redis which is a Key Value NoSQL database and the different data types it supports. Also shows how to use Redis Client API from node.
Redis - for duplicate detection on real time streamCodemotion
Roberto "frank" Franchini presenta a Codemotion Techmeetup Torino Redis, un data structure server che può utilizzare come chiavi stringhe, hashes, lists, sets, sorted sets, bitmaps e hyperloglogs
.
Redis Use Patterns (DevconTLV June 2014)Itamar Haber
An introduction to Redis for the SQL practitioner, covering data types and common use cases.
The video of this session can be found at: https://www.youtube.com/watch?v=8Unaug_vmFI
Power to the People: Redis Lua ScriptsItamar Haber
Redis is the Sun.
Earth is your application.
Imagine that the Moon is stuck in the middle of the Sun.
You send non-melting rockets (scripts) with robots
(commands) and cargo (data) back and forth…
Introduction to Sqoop | Big Data Hadoop Spark Tutorial | CloudxLabCloudxLab
Big Data with Hadoop & Spark Training: http://bit.ly/2kBMiEt
This CloudxLab Understanding Sqoop tutorial helps you to understand Sqoop in detail. Below are the topics covered in this tutorial:
1) Introduction to Sqoop
2) Sqoop Import - MySQL to HDFS
3) Sqoop Import - MySQL to Hive
4) Sqoop Import - MySQL to HBase
5) Sqoop Export - Hive to MySQL
New Indexing and Aggregation Pipeline Capabilities in MongoDB 4.2Antonios Giannopoulos
MongoDB 4.2 comes GA soon delivering some amazing new features on multiple areas. In this talk, we will focus on the new capabilities of the aggregation framework. We are going to cover the new operators and expressions. At the same time, we will explore how updates commands can now use the aggregation framework operators. We are also going to present aggregation framework improvements focusing on the on-demand materialized views. Finally, we are going to explore the wildcard indexes introduced in MongoDB 4.2 and how they change the way we design documents and build queries/aggregations. We will also make a reference to the new index build system.
Lua: the world's most infuriating languagejgrahamc
Slides from a talk I gave at the Lua London Meetup on October 17: "When first confronted with Lua an experienced programmer (like me!) finds themselves infuriated by the languages little differences and hopes that can dismiss it as not worth learning :-) Later they find themselves infuriated to learn that they can't dismiss it: Lua is just too fast, too useful and too flexible. This talk will look at my experience of learning Lua and using it to send a high-altitude balloon into the stratosphere and build CloudFlare's new low latency WAF."
This talk gives details about Spark internals and an explanation of the runtime behavior of a Spark application. It explains how high level user programs are compiled into physical execution plans in Spark. It then reviews common performance bottlenecks encountered by Spark users, along with tips for diagnosing performance problems in a production application.
Web-Scale Graph Analytics with Apache Spark with Tim HunterDatabricks
Graph analytics has a wide range of applications, from information propagation and network flow optimization to fraud and anomaly detection. The rise of social networks and the Internet of Things has given us complex web-scale graphs with billions of vertices and edges. However, in order to extract the hidden gems of understanding and information within those graphs, you need tools to analyze the graphs easily and efficiently.
At Spark Summit 2016, Databricks introduced GraphFrames, which implements graph queries and pattern matching on top of Spark SQL to simplify graph analytics. In this talk, we’ll discuss the work that has made graph algorithms in GraphFrames faster and more scalable. For example, new implementations of connected components have received algorithm improvements based on recent research, as well as performance improvements from Spark DataFrames. Discover lessons learned from scaling the implementation from millions to billions of nodes; see its performance in the context of other popular graph libraries; and hear about real-world applications.
Managing your black friday logs - Code EuropeDavid Pilato
Monitoring an entire application is not a simple task, but with the right tools it is not a hard task either. However, events like Black Friday can push your application to the limit, and even cause crashes. As the system is stressed, it generates a lot more logs, which may crash the monitoring system as well. In this talk I will walk through the best practices when using the Elastic Stack to centralize and monitor your logs. I will also share some tricks to help you with the huge increase of traffic typical in Black Fridays.
Topics include:
* monitoring architectures
* optimal bulk size
* distributing the load
* index and shard size
* optimizing disk IO
Takeaway: best practices when building a monitoring system with the Elastic Stack, advanced tuning to optimize and increase event ingestion performance.
Challenging Web-Scale Graph Analytics with Apache Spark with Xiangrui MengDatabricks
Graph analytics has a wide range of applications, from information propagation and network flow optimization to fraud and anomaly detection. The rise of social networks and the Internet of Things has given us complex web-scale graphs with billions of vertices and edges. However, in order to extract the hidden gems within those graphs, you need tools to analyze the graphs easily and efficiently.
At Spark Summit 2016, Databricks introduced GraphFrames, which implemented graph queries and pattern matching on top of Spark SQL to simplify graph analytics. In this talk, you'll learn about work that has made graph algorithms in GraphFrames faster and more scalable. For example, new implementations like connected components have received algorithm improvements based on recent research, as well as performance improvements from Spark DataFrames. Discover lessons learned from scaling the implementation from millions to billions of nodes; compare its performance with other popular graph libraries; and hear about real-world applications.
Challenging Web-Scale Graph Analytics with Apache SparkDatabricks
Graph analytics has a wide range of applications, from information propagation and network flow optimization to fraud and anomaly detection. The rise of social networks and the Internet of Things has given us complex web-scale graphs with billions of vertices and edges. However, in order to extract the hidden gems within those graphs, you need tools to analyze the graphs easily and efficiently.
At Spark Summit 2016, Databricks introduced GraphFrames, which implemented graph queries and pattern matching on top of Spark SQL to simplify graph analytics. In this talk, you’ll learn about work that has made graph algorithms in GraphFrames faster and more scalable. For example, new implementations like connected components have received algorithm improvements based on recent research, as well as performance improvements from Spark DataFrames. Discover lessons learned from scaling the implementation from millions to billions of nodes; compare its performance with other popular graph libraries; and hear about real-world applications.
Managing your Black Friday Logs NDC OsloDavid Pilato
Monitoring an entire application is not a simple task, but with the right tools it is not a hard task either. However, events like Black Friday can push your application to the limit, and even cause crashes. As the system is stressed, it generates a lot more logs, which may crash the monitoring system as well. In this talk I will walk through the best practices when using the Elastic Stack to centralize and monitor your logs. I will also share some tricks to help you with the huge increase of traffic typical in Black Fridays.
Topics include:
* monitoring architectures
* optimal bulk size
* distributing the load
* index and shard size
* optimizing disk IO
Takeaway: best practices when building a monitoring system with the Elastic Stack, advanced tuning to optimize and increase event ingestion performance.
FOSDEM 2019: M3, Prometheus and Graphite with metrics and monitoring in an in...Rob Skillington
The world in which we monitor software is growing more complex every year. There are increasingly more ways to run server-side software, with many more independent services and more points of failures, the list goes on! On the plus side, there’s a lot of great tools and patterns being developed to try and make things simple to assess and understand. This talk covers how metrics and monitoring can be leveraged in a variety of different ways, auto-discovering applications and their usage of databases, caches, load balancers, etc, setting up and tearing down dashboards and monitoring automatically for services and instances, and more.
We’ll also talk about how you can accomplish all this with a global view of your systems using both Prometheus and Graphite with M3, our open source metrics platform. We’ll take a deep dive look at how we use M3DB, distributed aggregation with the M3 aggregator and the M3 Kubernetes operator to horizontally scale a metrics platform in a way that doesn’t cost outrageous amounts to run with a system that’s still sane to operate with petabytes of metrics data.
Spark SQL Catalyst Code Optimization using Function Outlining with Kavana Bha...Databricks
Spark SQL Catalyst optimizer, post query plan optimization, compiles the SQL query to Java code. Without code generation, such query expressions would have to be interpreted for each row of data, by walking down a tree of nodes. This introduces large amounts of branches and virtual function calls that slow down execution. With code generation, a query is collapsed into a single optimized function that eliminates multiple function calls and leverages CPU registers for intermediate data.
This code is then compiled in runtime to Java bytecode using Janino compiler. This presentation focuses on further catalyst code generation optimizations possible using function outlining. Automatic code generation tools generally tend to generate huge optimized functions. Large functions that are frequently executed might degrade runtime performance by preventing JVM optimizations such as function inlining. To avoid this, code generation tools should try to contain independent logic into separate functions.
This presentation will take the audience through the Spark Catalyst Code generation, how automatic split of large functions into smaller functions was achieved and the performance benefits associated with it
After more than 5 years of doing this, I think I managed to capture the essence of the beast quite neatly. Here's what matters about Redis, the open source in-memory data structure store, IMO.
How I Implemented the #1 Requested Feature In Redis In Less than 1 Hour with ...Itamar Haber
These are the slides from my RedisConf 18 session about developing Redis modules. It focuses on the new keyspace notifications API and the blocking API, by showing how to develop the Ze POP module (https://github.com/itamarhaber/zpop).
P.S. no pineapples or bananas were hurt.
An introduction and status update on Redis' upcoming new data structure - Stream - that is not unlike a log, has some Apache Kafka-like thingamagigs and can be also used for time series data
Recording: https://www.youtube.com/watch?v=qHkXVY2LpwU
External links: https://gist.github.com/itamarhaber/dddc3d4d9c19317b1477
Applications today are required to process massive amounts of data and return responses in real time. Simply storing Big Data is no longer enough; insights must be gleaned and decisions made as soon as data rushes in. In-memory databases like Redis provide the blazing fast speeds required for sub-second application response times. Using a combination of in-memory Redis and disk-based MongoDB can significantly reduce the “digestive” challenge associated with processing high velocity data.
Redis & MongoDB: Stop Big Data Indigestion Before It StartsItamar Haber
Efficiently digesting data in large volumes can prove to be challenging for any database. The challenges are compounded when this influx must be analyzed on the fly, or "tasted", to satisfy the sophisticated palates of modern apps. Luckily, there are several proven remedies you can concoct with Redis to help with potential indigestion.
The URLs from the presentation are also available at: https://gist.github.com/itamarhaber/325e515c1715a12ef132
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Welcome to the first live UiPath Community Day Dubai! Join us for this unique occasion to meet our local and global UiPath Community and leaders. You will get a full view of the MEA region's automation landscape and the AI Powered automation technology capabilities of UiPath. Also, hosted by our local partners Marc Ellis, you will enjoy a half-day packed with industry insights and automation peers networking.
📕 Curious on our agenda? Wait no more!
10:00 Welcome note - UiPath Community in Dubai
Lovely Sinha, UiPath Community Chapter Leader, UiPath MVPx3, Hyper-automation Consultant, First Abu Dhabi Bank
10:20 A UiPath cross-region MEA overview
Ashraf El Zarka, VP and Managing Director MEA, UiPath
10:35: Customer Success Journey
Deepthi Deepak, Head of Intelligent Automation CoE, First Abu Dhabi Bank
11:15 The UiPath approach to GenAI with our three principles: improve accuracy, supercharge productivity, and automate more
Boris Krumrey, Global VP, Automation Innovation, UiPath
12:15 To discover how Marc Ellis leverages tech-driven solutions in recruitment and managed services.
Brendan Lingam, Director of Sales and Business Development, Marc Ellis
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
4. About our sponsor
- Home of open source Redis
- The commercial provider for managed and
downloadable Redis solutions
- HQ @ Mountain View, R&D @ Italy & Israel
- ~70 employees & growing fast - we're hiring :)
7. OSS crossover:
Redis 2.x -> ardb ->
2D spatial index ->
Matt Stancliff @mattsta ->
Salvatore Sanfilippo @antirez ->
Redis 3.2
Geospatial indices redhistory
8. The Geohash
- A geocoding system, hierarchical spatial grid
- The hash value maps to a location (lon, lat)
- Is usually base-32 encoded (.e.g sv8y8v66bt0)
- By Gustavo Niemeyer, in the public's domain
since 28 Feb. 2009
14. Geospatial Indices in Redis
- Redis' Geohashes are 52-bit integers (~0.6m)
- Redis' Sorted Sets' scores are IEEE 754
floats, i.e. 53-bit integer precision…
BINGO!
15. New! GEO API, part 1
Add a point:
GEOADD key longitude latitude member [...]
Get longitude & latitude / geohash:
GEOPOS|GEOHASH key member [...]
Get the distance between two points:
GEODIST key member1 member2 [unit]
16. GEO API - "Demo"
127.0.0.1:6379> GEOADD g 34.84076 32.10942 RL@TLV
(integer) 1
127.0.0.1:6379> GEOADD g -122.0678325 37.3775256 RL@MV
(integer) 1
127.0.0.1:6379> GEODIST g RL@TLV RL@MV km
"11928.692170353959"
127.0.0.1:6379> GEOADD g 34.8380433 32.1098095 Hudson
(integer) 1
17. Size (almost) doesn't matter
127.0.0.1:6379> GEOHASH g RL@TLV Hudson
1) "sv8y8v66bt0"
2) "sv8y8v2m1n0"
- Shorter hashes -> "same" location, bigger area
- Close spatial proximity usually means a
shared hash prefix
18. New! GEO API, part 2
Search for members in a radial area:
GEORADIUS key longitude latitude radius unit ...
GEORADIUSBYMEMBER key member radius unit ...
Overthrows ZREVRANGEBYSCORE!!! #RedisTrivia
Delete a point - no GEOREM for you:
ZREM key member [...]
21. Multi-dimensional queries
SELECT id
FROM users
WHERE age > 35 AND
salary BETWEEN 250 AND 350
http://stackoverflow.
com/questions/32911604/intersection-of-two-or-
more-sorted-sets
22. The Redis way - "ZQUERY"
ZUNIONSTORE t 1 age WEIGHTS 1
ZREMRANGEBYSCORE t -inf (25
ZREMRANGEBYSCORE t (35 +inf
ZINTERSTORE t 2 t salary WEIGHTS 0 1
ZRANGEBYSCORE t 250 350
DEL t
Works, but not too efficient.
23. Would indexing the data help?
rqtih.lua: Another Redis Way
https://gist.github.
com/itamarhaber/c1ffda42d86b314ea701
rqtih.lua is about 32.5 times faster than
ZQUERY on 100K users (age & salary)
24. rqtih.lua?!? A PoC for
R - Redis, duh
QT - Quadtree
IH - In Hash
.LUA - "object oriented", JiT reads, delayed
writes
26. A new Redis data structure?
- Discussions in proximity to Redis Developers
Day 2015 (London)
- k-d tree: similar principles for k dimensions,
but complex complexity
- Outcomes: topics/indexes & experimental API
that uses existing data types (Zorted & Hash)
27. Redimension: k-d query API
@antirez's idea: interleave the dimensions, store
"score"+data in a Zorted Set for lexicographical
ranges, maintain a Hash for lookups
redimension.rb - implementation by @antirez
redimension.lua - port by @itamarhaber
28. Redimension "Demo"
~/src/lua-redimension$ redis-cli SCRIPT LOAD
"$(cat redimension.lua)"
"4abdad23c459145cbd658c991c0c8ad93d984d91"
~/src/lua-redimension$ redis-cli EVALSHA
4abdad23c459145cbd658c991c0c8ad93d984d91 0
1) "KEYS[1] - index sorted set key"
2) "KEYS[2] - index hash key"
3) "ARGV[1] - command. Can be:"
29. Redimension "Demo", 2
4) " create - create an index with ARGV
[2] as dimension and ARGV[3] as precision"
5) " drop - drops an index"
6) " index - index an element ARGV[2]
with ARGV[3]..ARGV[3+dimension] values"
7) " unindex - unindex an element ARGV
[2] with ARGV[3]..ARGV[3+dimension] values"
30. Redimension "Demo", 3
9) " update - update an element ARGV[2]
with ARGV[3]..ARGV[3+dimension] values"
10) " query - query using ranges ARGV
[2], ARGV[3]..ARGV[2+dimension-1], ARGV
[2+dimension]"
11) " fuzzy_test - fuzzily tests the library
on ARGV[2] dimension with ARGV[3] items using
ARGV[4] queries"
31. redimension.next()
- Currently just an experiment
- Many improvements still needed
- Planned to become a part of the core project
- Need more feedback WRT functionality & API
- Any ideas?
33. changes that made it (or didn’t) to OSS redis
● merged into 3.0
○ Fix a race condition in processCommand() with freeMemoryIfNeeded()
○ diskless replication fixes
○ psync fixes
○ fixes in LRU eviction (dict random keys during rehasing)
● merged into 3.2
○ sds optimizations
○ jemalloc size class optimization
● changes not merged yet
○ diskless slave replication
○ dict.c improvements
● other changes i didn’t get to push yet
○
35. diskless replication
● how normal replication works.
master->fork->rdb on disk->main process streams to slave
slave->save to disk while serving clients->flushdb->load rdb
● disadvantages of diskless replication
○ slaves must connect together
○ slave side flush before RDB was fully received
○ on slow network, longer fork duration
● a word about fork() and CoW?
36. diskless replication benchmark (replication time)
two instances of r3.2xl (60GB ram, with 160GB SSD),
4,000,000 string keys of 1k random data.
(consuming 52GB of RAM), 19GB RDB file.
fully disk based: 513 seconds
only master diskless: 365 seconds
fully diskless: 231 seconds
only salve is diskless: 360 seconds
37. ● we all know what fragmentation is
● history: on the search for the ultimate allocator
● how an allocator works (bins) to overcome that
● a word about virtual address space vs OS pages
○ RSS = VM pages mapped to physical RAM
● what’s internal fragmentation / used_memory
(maxmemory) includes internal frag
● RSS = used_memory (+external frag)
○ external frag are unused bins, and pages
●
42. 4k page
4k page
4k page
4k page
can be returned to os
(won’t be rss anymore)
4k page
43. ABCDEFn
char*
used unusedfree
4 bytes 4 bytes
old sds header
● grows in place(sometimes no need for realloc)
○ although realloc may nop instead of give new pointer and do
memcpy
● no need for strlen (search for null terminator)
● can be used in normal string functions like printf
struct sdshdr {
unsigned int len;
unsigned int free;
char buf[];
};
45. sds size classes
debug populate 10000000
used_memory of original code: 1,254,709,872
used_memory with new code: 1,078,723,024
memory optimization: 16%
49. Script replication before v3.2
Lua scripts are pushed down to the slaves for
local execution. This reduces wire traffic in
cases such as:
for i = 1, 1000000 do
redis.call('LPUSH', KEYS[1], i)
end
50. Script replication caveats
Compute-intensive scripts (e.g. ZQUERY) waste
CPU time because they are run:
- 1+number of slaves times: wasteful
- When recovering from AOF: really bad
And then there's also...
51. Free will
vs.
> EVAL "redis.call('SET', KEYS[1], redis.call
('TIME')[1])" 1 foo
(error) ... Write commands
not allowed after non
deterministic commands
52. Script replication in v3.2
- Same defaults
- NEW! redis.replicate_commands()
causes the script's effects to be replicated
- NEW! redis.set_repl(...)
redis.REPL_[ALL|NONE|AOF|SLAVE]
55. "A few things about Redis security"
"The Redis security model is: it’s
totally insecure to let untrusted
clients access the system, please
protect it from the outside world yourself...
Let’s crack Redis for fun and no profit…"
HOWTO: http://antirez.com/news/96
57. 3 critical points about security
Honesty is always the best option. That said:
1. Never leave an unprotected server open to
the outside world
2. If your server has been compromised, burn it
3. Always read the documentation
58. NEW! protected-mode directive
By default is enabled -> a breaking upgrade!
When (protected-mode && !requirepass && !bind):
- Allow only 127.0.0.1, ::1 or socket connections
- DENY (with the longest message ever!) others
59. Protection in Action - "Demo"
-DENIED Redis is running in protected mode because protected mode is enabled, no
bind address was specified, no authentication password is requested to clients. In
this mode connections are only accepted from the loopback interface. If you want to
connect from external computers to Redis you may adopt one of the following
solutions: 1) Just disable protected mode sending the command 'CONFIG SET protected-
mode no' from the loopback interface by connecting to Redis from the same host the
server is running, however MAKE SURE Redis is not publicly accessible from internet
if you do so. Use CONFIG REWRITE to make this change permanent. 2) Alternatively you
can just disable the protected mode by editing the Redis configuration file, and
setting the protected mode option to 'no', and then restarting the server. 3) If you
started the server manually just for testing, restart it with the '--protected-mode
no' option. 4) Setup a bind address or an authentication password. NOTE: You only
need to do one of the above things in order for the server to start accepting
connections from the outside.