Charla sobre Redis impartida en el grupo de Ruby Zaragoza explicando qué es Redis, para qué se usa en sitios como Twitter o Pinterest y cómo se está usando en teowaki
(WEB401) Optimizing Your Web Server on AWS | AWS re:Invent 2014Amazon Web Services
Tuning your EC2 web server will help you to improve application server throughput and cost-efficiency as well as reduce request latency. In this session we will walk through tactics to identify bottlenecks using tools such as CloudWatch in order to drive the appropriate allocation of EC2 and EBS resources. In addition, we will also be reviewing some performance optimizations and best practices for popular web servers such as Nginx and Apache in order to take advantage of the latest EC2 capabilities.
(SDD419) Amazon EC2 Networking Deep Dive and Best Practices | AWS re:Invent 2014Amazon Web Services
Amazon EC2 instances give customers a variety of high-bandwidth networking choices. In this session, we discuss how to choose among Amazon EC2 networking technologies and examine how to get the best performance out of Amazon EC2 enhanced networking and cluster networking. We also share best practices and useful tips for success.
Every company likes to brag about their successes, but not many are willing to talk about their failures. At PagerDuty we have been rigorously tracking downtime in order to analyze it and learn from our mistakes - we even blog about these failures publicly.
Despite being a highly available system, we have had three outages caused by problems with our production Cassandra clusters over the past year. We'll take a look at each of these outages: what we saw from the inside, the actions we took to recover, and most importantly the procedures and monitoring that will help prevent it from happening to you.
API analytics with Redis and Google Bigquery. NoSQL matters editionjavier ramirez
At teowaki we have a system for API use analytics using Redis as a fast intermediate store and bigquery as a big data backend. As a result, we can launch aggregated queries on our traffic/usage data in a few seconds and we can try and find for usage patterns that wouldn’t be obvious otherwise. In this session I will speak of the alternatives we evaluated and how we are using Redis and Bigquery to solve our problem.
Big data lambda architecture - Streaming Layer Hands Onhkbhadraa
This presentation describes Hands on guide BIG Data Streaming Pipeline AWS Cloud Platform using Apache Kafka, Apache Hadoop, Apache Spark and Apache Cassandra.
(WEB401) Optimizing Your Web Server on AWS | AWS re:Invent 2014Amazon Web Services
Tuning your EC2 web server will help you to improve application server throughput and cost-efficiency as well as reduce request latency. In this session we will walk through tactics to identify bottlenecks using tools such as CloudWatch in order to drive the appropriate allocation of EC2 and EBS resources. In addition, we will also be reviewing some performance optimizations and best practices for popular web servers such as Nginx and Apache in order to take advantage of the latest EC2 capabilities.
(SDD419) Amazon EC2 Networking Deep Dive and Best Practices | AWS re:Invent 2014Amazon Web Services
Amazon EC2 instances give customers a variety of high-bandwidth networking choices. In this session, we discuss how to choose among Amazon EC2 networking technologies and examine how to get the best performance out of Amazon EC2 enhanced networking and cluster networking. We also share best practices and useful tips for success.
Every company likes to brag about their successes, but not many are willing to talk about their failures. At PagerDuty we have been rigorously tracking downtime in order to analyze it and learn from our mistakes - we even blog about these failures publicly.
Despite being a highly available system, we have had three outages caused by problems with our production Cassandra clusters over the past year. We'll take a look at each of these outages: what we saw from the inside, the actions we took to recover, and most importantly the procedures and monitoring that will help prevent it from happening to you.
API analytics with Redis and Google Bigquery. NoSQL matters editionjavier ramirez
At teowaki we have a system for API use analytics using Redis as a fast intermediate store and bigquery as a big data backend. As a result, we can launch aggregated queries on our traffic/usage data in a few seconds and we can try and find for usage patterns that wouldn’t be obvious otherwise. In this session I will speak of the alternatives we evaluated and how we are using Redis and Bigquery to solve our problem.
Big data lambda architecture - Streaming Layer Hands Onhkbhadraa
This presentation describes Hands on guide BIG Data Streaming Pipeline AWS Cloud Platform using Apache Kafka, Apache Hadoop, Apache Spark and Apache Cassandra.
Fun with ruby and redis, arrrrcamp edition, javier_ramirez, teowakijavier ramirez
In this talk I make an introduction to Redis, then I explain how some big names (twitter, pinterest...) are using it, then I describe some pitfalls, then I explain how we are using redis at teowaki
Fun with ruby and redis, arrrrcamp edition, javier_ramirez, teowakijavier ramirez
In this talk I make an introduction to Redis, then I explain how some big names (twitter, pinterest...) are using it, then I describe some pitfalls, then I explain how we are using redis at teowaki
Link to the full talk - https://youtu.be/2Rf5t2Eh6IQ
https://go.dok.community/slack
https://dok.community
ABSTRACT OF THE TALK
This talk will provide a high-level overview of Kubernetes, Helm charts and how they can be used to deploy Apache Druid clusters of any size.
We'll review how Kubernetes functionality enables resilience and self-healing, historical tiers through node group affinity, middle manager scaling through Kubernetes autoscaling to optimize ingestion capacity and some of the gotchas along the way.
BIO
Sergio Ferragut is a database veteran turned Developer Advocate at Imply. His experience includes 16 years at Teradata in professional services and engineering roles.
He has direct experience in building analytics applications spanning the retail, supply chain, pricing optimization and IoT spaces.
Sergio has worked at multiple technology start-ups including APL and Splice Machine where he helped guide product design and field messaging.
Amazon Aurora services are MySQL and PostgreSQL -compatible relational database engines with the speed, reliability, and availability of high-end commercial databases at one-tenth the cost. This session introduces you to Amazon Aurora, explores the capabilities and features of Aurora, explains common use cases, and helps you get started with Aurora.
Redis is an open source, in-memory data store that delivers sub-millisecond response times enabling millions of requests per second to power real-time applications. It can be used as a fast database, cache, message broker, and queue. Amazon ElastiCache delivers the ease-of-use and power of Redis along with the availability, reliability, scalability, security, and performance suitable for the most demanding applications. We’ll take a close look at Redis and how to use it to power different use cases.
Bigdata for small pockets, by Javier Ramirez from teowaki. RubyC Kiev 2014javier ramirez
This is the story of how https://teowaki.com added bigdata analytics using a very economic approach.
Bigdata is amazing. You can get insights from your users, find interesting patterns and have lots of geek fun. Problem is big data usually means many servers, a complex set up, intensive monitoring and a steep learning curve. All those things cost money. If you don't have the money, you are losing all the fun. In my talk I will show you how you can use Redis, Google Bigquery and Apps Script to manage big data from your application for under $1 per month. Don't you feel like running a RegExp over 300 million rows in just 5 seconds?
by Jeff Duffy, Database Specialist Solutions Architect, AWS
Database Week at the AWS Loft is an opportunity to learn about Amazon’s broad and deep family of managed database services. These services provide easy, scalable, reliable, and cost-effective ways to manage your data in the cloud. We explain the fundamentals and take a technical deep dive into Amazon RDS and Amazon Aurora relational databases, Amazon DynamoDB non-relational databases, Amazon Neptune graph databases, and Amazon ElastiCache managed Redis, along with options for database migration, caching, search and more. You'll will learn how to get started, how to support applications, and how to scale.
At teowaki we have a system for API usage analytics, with Redis as a fast intermediate store and bigquery as a big data backend. As a result, we can launch aggregated queries on our traffic/usage data in just a few seconds and we can try and find for usage patterns that wouldn’t be obvious otherwise.
In this session I will talk about how we entered the Big Data world, which alternatives we evaluated, and how we are using Redis and Bigquery to solve our problem.
1- Introduction about Database Mirroring Concept
2- Reference (8 Blogs )
3- Note
4- Database mirroring operation mode
5- Database Mirroring Requirement
6- Advantage of Database Mirroring
7- Disadvantage of Database Mirroring
8- Database Mirroring Enhancement in SQL Server 2008
9- Database Mirroring Installation Step by Step
10- High Availability Mode [Automatic Failover]
11- High Availability Mode [Manual Failover]
12- High Safety Mode Without witness server [Manual Failover]
13- Stander listener port in database mirroring
14- Check SQL server mirroring availability
15- Add or replace witness server to an existing mirroring database
16- How to monitor Database Mirroring
17- Mirroring in workshop not in DC (Domain Controller)
Serverless is a new framework that allows developers to easily harness AWS Lambda and Api Gateway to build and deploy full fledged API services without needing to deal with any ops level overhead or paying for servers when they're not in use. It's kinda like Heroku on-demand for single functions.
What are some of the performance implications of using lambdas and what strategies can be used to address these. When might be want an alternative to using a lambda and how can we design our APIs to be flexible in this regard. What are the principles of writing low latency code in Java? How do we tune and optimize our code for low latency? When don’t we optimize our code? Where does the JVM help and where does it get in our way? How does this apply to lambdas? How can we design our APIs to use lambdas and minimize garbage?
Securing Big Data at rest with encryption for Hadoop, Cassandra and MongoDB o...Big Data Spain
This session shows how to secure different Big Data sensitive data items such as log files, metastore databases, control files, config files, data directories or data files for different Big Data technologies.
As Hadoop, MongoDB, Cassandra and other massively distributed Big Data stores grow in popularity, so too does the volume of sensitive regulatory data that gets captured for analysis. Cloudera Navigator Encrypt gives peace of mind, knowing the sensitive information used to run massive-scale queries and analytics is secure. Navigator Encrypt works as a last line of defense for protecting data, by providing a transparent layer between the application and file system and securing information as it gets written to disk, ensuring minimal performance lag in the encryption or decryption process. The solution also includes robust key management and process-based access controls, while simultaneously preventing admins or super users like root from accessing data that they don’t need to see allowing users to store their cryptographic keys separate from the encrypted data.
Session presented at Big Data Spain 2015 Conference
15th Oct 2015
Kinépolis Madrid
http://www.bigdataspain.org
Event promoted by: http://www.paradigmatecnologico.com
Abstract: http://www.bigdataspain.org/program/thu/slot-13.html
API Analytics with Redis and Bigquery. NoSQLmatters Cologne '14 edition. Javi...javier ramirez
At teowaki we have a system for API usage analytics, with Redis as a fast intermediate store and bigquery as a big data backend. As a result, we can launch aggregated queries on our traffic/usage data in just a few seconds and we can try and find for usage patterns that wouldn’t be obvious otherwise.In this session I will talk about how we entered the Big Data world, which alternatives we evaluated, and how we are using Redis and Bigquery to solve our problem.
Similar to Por que deberias haberle pedido redis a los reyes magos (20)
The Building Blocks of QuestDB, a Time Series Databasejavier ramirez
Talk Delivered at Valencia Codes Meetup 2024-06.
Traditionally, databases have treated timestamps just as another data type. However, when performing real-time analytics, timestamps should be first class citizens and we need rich time semantics to get the most out of our data. We also need to deal with ever growing datasets while keeping performant, which is as fun as it sounds.
It is no wonder time-series databases are now more popular than ever before. Join me in this session to learn about the internal architecture and building blocks of QuestDB, an open source time-series database designed for speed. We will also review a history of some of the changes we have gone over the past two years to deal with late and unordered data, non-blocking writes, read-replicas, or faster batch ingestion.
Hubo un tiempo en el que casi cualquier componente de software requería pagar una licencia. Afortunadamente, hoy en día gracias al software libre y de código abierto, se puede desarrollar prácticamente cualquier aplicación usando componentes gratuitos.
Pero, si el software es gratis, ¿Quién lo desarrolla? ¿Trabaja la comunidad de software libre de forma altruista? ¿Se puede desarrollar software libre de forma profesional? De hecho, hay quien dice que el código abierto tal y como lo conocimos ya no existe, y que lo que hay hoy en día es otra cosa.
En esta charla hablaré de cómo se puede monetizar el código libre, y de algunos posibles conflictos que puedes encontrarte en el camino.
Además, te contaré cómo hacemos desde QuestDB para desarrollar una base de datos de código abierto y mantener un equipo estable viviendo de ello. Comentaré también algunas situaciones problemáticas a las que proyectos muy destacados se han enfrentado, o que se enfrentan a día de hoy.
QuestDB: The building blocks of a fast open-source time-series databasejavier ramirez
(talk delivered at OSA CON 23)
Traditionally, databases have treated timestamps just as another data type. However, when performing real-time analytics, timestamps should be first class citizens and we need rich time semantics to get the most out of our data. We also need to deal with ever growing datasets while keeping performant, which is as fun as it sounds.
It is no wonder time-series databases are now more popular than ever before. Join me in this session to learn about the internal architecture and building blocks of QuestDB, an open source time-series database designed for speed.
We will learn how it deals with data ingestion, and which SQL extensions it implements for working with time-series efficiently.
We will also review a history of some of the changes we have gone over the past two years to deal with late and unordered data, non-blocking writes, read-replicas, or data deduplication.
Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...javier ramirez
QuestDB es una base de datos open source de alto rendimiento. Mucha gente nos comentaba que les gustaría usarla como servicio, sin tener que gestionar las máquinas. Así que nos pusimos manos a la obra para desarrollar una solución que nos permitiese lanzar instancias de QuestDB con provisionado, monitorización, seguridad o actualizaciones totalmente gestionadas.
Unos cuantos clusters de Kubernetes más tarde, conseguimos lanzar nuestra oferta de QuestDB Cloud. Esta charla es la historia de cómo llegamos ahí. Hablaré de herramientas como Calico, Karpenter, CoreDNS, Telegraf, Prometheus, Loki o Grafana, pero también de retos como autenticación, facturación, multi-nube, o de a qué tienes que decir que no para poder sobrevivir en la nube.
Ingesting Over Four Million Rows Per Second With QuestDB Timeseries Database ...javier ramirez
How would you build a database to support sustained ingestion of several hundreds of thousands rows per second while running near real-time queries on top?
In this session I will go over some of the technical decisions and trade-offs we applied when building QuestDB, an open source time-series database developed mainly in JAVA, and how we can achieve over four million row writes per second on a single instance without blocking or slowing down the reads. There will be code and demos, of course.
We will also review a history of some of the changes we have gone over the past two years to deal with late and unordered data, non-blocking writes, read-replicas, or faster batch ingestion.
Deduplicating and analysing time-series data with Apache Beam and QuestDBjavier ramirez
Time series data pipelines tend to prioritise speed and freshness over completeness and integrity. In such scenarios, it is very common to ingest duplicate data, which may be fine for many analytical use cases, but is very inconvenient for others.
There are many open source databases built specifically for the speed and query semantics of time series, and most of them lack automatic deduplication of events in near real-time. One such database is QuestDB, which requires a manual batch process to deduplicate ingested data.
In this talk, we will see how we can successfully use Apache Beam to deduplicate streaming time series, which can then be analysed by a time series database.
Relational databases were created a long time ago for a simpler world. Even if they are still awesome tools for generic workloads, there are some things they cannot do well.
In this session I will speak about purpose-built databases that you can use for specific business scenarios. We will see the type of queries you can run on a Graph database, a Document Database, and a Time-Series database. We will then see how a relational database could also be used for the same use cases, just in a much more complex way.
Your Timestamps Deserve Better than a Generic Databasejavier ramirez
If you are storing records with a timestamp in your database, it is very likely a time series database can make your life easier.
However, time series databases are still the great unknown for a large part of the tech community.
In this talk, I will show you what use cases they are good for, what they give you that you cannot get from a traditional database, and when it is a good idea (and when it is not) to use them.
For the demos, we will be using QuestDB, the fastest open-source time series database.
Cómo se diseña una base de datos que pueda ingerir más de cuatro millones de ...javier ramirez
En esta sesión voy a contar las decisiones técnicas que tomamos al desarrollar QuestDB, una base de datos Open Source para series temporales compatible con Postgres, y cómo conseguimos escribir más de cuatro millones de filas por segundo sin bloquear o enlentecer las consultas.
Hablaré de cosas como (zero) Garbage Collection, vectorización de instrucciones usando SIMD, reescribir en lugar de reutilizar para arañar microsegundos, aprovecharse de los avances en procesadores, discos duros y sistemas operativos, como por ejemplo el soporte de io_uring, o del balance entre experiencia de usuario y rendimiento cuando se plantean nuevas funcionalidades.
Processing and analysing streaming data with Python. Pycon Italy 2022javier ramirez
Data used to be a batch thing, but more and more we get unbounded streams of data, fast or slow, that we need to process and analyse in near real time.
In this talk I’ll show you how you can use Apache Flink and QuestDB to build reliable streaming data pipelines that can grow as much as you need.
QuestDB: ingesting a million time series per second on a single instance. Big...javier ramirez
In this session I will show you the technical decisions we made when building QuestDB, the open source, Postgres compatible, time-series database, and how we can achieve a million row writes per second without blocking or slowing down the reads.
Servicios e infraestructura de AWS y la próxima región en Aragónjavier ramirez
AWS está montando una región de infraestructura en Aragón. Vale, pero ¿Qué significa eso? ¿Es tan diferente de un centro de datos convencional o de otros proveedores de nube? (Spoiler: Sí). En esta sesión te cuento por qué. Hay video en https://catedrasamcadt.unizar.es/noticias/el-momento-tecnologico-actual-contado-por-trabajadores-de-amazon-web-services/
¿Qué es eso del desarrollo sin servidores? ¿Qué lenguajes puedo utilizar? ¿Cómo hago cosas como autenticación, o guardar en base de datos, o enviar notificaciones? ¿Esto escala? A todas estas preguntas, y a alguna más, intentaré dar respuesta en esta sesión, donde haré una pequeña demo de montar una app muy sencilla y desplegarla en la nube sin preocuparnos de gestionar infraestructura. Charla realizada por primera vez para AlcarriaConf 2021
AWS launched publicly on March 2006 with just one service, starting the age of the public cloud. You might think after 15 years everything in cloud has already been invented, but that's simply not the case.
In this session I want to show you how AWS is reinventing the cloud in areas like computing, machine learning, databases and analytics, or cloud infrastructure.
Analitica de datos en tiempo real con Apache Flink y Apache BEAMjavier ramirez
Trabajar en tiempo real con datos que se mueven muy rápido no es trivial, sobre todo con volúmenes de datos elevados. Apache Flink y Apache BEAM están específicamente diseñadas para ese caso de uso. En esta charla te contaré los retos de la analítica en tiempo real, cuál es la arquitectura de Apache Flink, qué es Apace BEAM, y cómo usan estas herramientas empresas para hacer desde procesos triviales hasta gestionar billones de eventos al día con latencias de milisegundos. Por supuesto, haremos una demo :)
In this webinar we explain which are some of the problems of streaming analytics, and why they are different to batch/big data analytics. Then we go into introducing some basic streaming concepts, like event queues, event processors, event vs processing time, and delivery guarantees. We end this first part of the series presenting a few of the most common open source components for streaming (Kafka, Spark, Flink, Cassandra, or ElasticSearch) and we mention the different options you have to run them on AWS.
Getting started with streaming analytics: Setting up a pipelinejavier ramirez
In this session I will show you how to create a simple streaming analytics pipeline, first using open source tools and developing locally, then moving to a VM, then moving to fully managed AWS services. The session will serve as an introduction to some details of Apache Kafka, Apache Flink, ElasticSearch, Amazon Managed Streaming for Kafka, Kinesis Data Analytics, and Amazon ElasticSearch. It will be an almost slideless presentation, as I will spent most of the time at the command line and the IDE.
Getting started with streaming analytics: Deep Divejavier ramirez
Now that we know how to create simple streaming analytics pipelines, it is time to learn something more interesting. In this session I will show you how to add Complex Event Processing to your Apache Flink (or Kinesis Data Analytics) application using JAVA. For those of you that prefer SQL, I will show you how to run streaming analytics using only SQL.
Getting started with streaming analytics: streaming basics (1 of 3)javier ramirez
In this webinar we explain which are some of the problems of streaming analytics, and why they are different to batch/big data analytics. Then we go into introducing some basic streaming concepts, like event queues, event processors, event vs processing time, and delivery guarantees. We end this first part of the series presenting a few of the most common open source components for streaming (Kafka, Spark, Flink, Cassandra, or ElasticSearch) and we mention the different options you have to run them on AWS.
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024Neo4j
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Communications Mining Series - Zero to Hero - Session 1DianaGray10
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
Communications Mining Series - Zero to Hero - Session 1
Por que deberias haberle pedido redis a los reyes magos
1. Por qué deberías haberle
pedido Redis a los Reyes
Magos
javier ramirez
@supercoco9
2.
3. In the beginner's
mind there are many
possibilities, in the
expert's mind there
are few.
Shunryu Suzuki
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
5. Intel(R) Xeon(R) CPU E5520 @ 2.27GHz (with pipelining)
$ ./redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop -P 16 -q
SET: 552,028 requests per second
GET: 707,463 requests per second
LPUSH: 767,459 requests per second
LPOP: 770,119 requests per second
Intel(R) Xeon(R) CPU E5520 @ 2.27GHz (without pipelining)
$ ./redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop -q
SET: 122,556 requests per second
GET: 123,601 requests per second
LPUSH: 136,752 requests per second
LPOP: 132,424 requests per second
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
6. myth: Redis is just like
memcached
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
7. myth: Redis is for queues
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
8. open source, BSD licensed, advanced
key-value store. It is often referred to as a
data structure server since keys can contain
strings, hashes, lists, sets and sorted sets.
http://redis.io
started in 2009 by Salvatore Sanfilippo @antirez
103 contributors at
https://github.com/antirez/redis
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
10. The Redis Manifesto
1.A DSL for Abstract Data Types
2.Memory storage is #1
3.Fundamental data structures for a
fundamental API
4.Two levels of API
5.Code is like a poem; it's not just something
we write to reach some practical result
6.We're against complexity
7.We optimize for joy
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
15. basics: sets
cool example about sunionstore
http://robots.thoughtbot.com/post/46335890055/redis-set-intersection-using-sets-to-filter-data
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
16. Redis as a PUBSUB system
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
18. Persistance: RDB
Compact binary format
Saves snapshots every few minutes
Good for backups and synchronizing
If Redis crashes, a few minutes worth of
data will be lost
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
19. Durability: AOF
Log text format
Configurable durability
Large file, can slow down startup
If Redis crashes, typically one second of
data could be lost
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
20. replication & scaling out
one master, several
read-only slaves
sharding
Twemproxy & redis cluster
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
21. atomicity
single threaded, so no
concurrency problems
transactions and lua
scripts to run multiple
operations atomically
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
22. Scripting with lua
You can use Lua for scripting Redis when you need to
atomically execute a sequence of commands in which
the output of a command is used as input for another
It reduces the need to use complex lock mechanisms
and simplifies dependencies between clients
You can even extend the functionality of Redis by using
Lua scripts
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
23. what's being used for
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
24. twitter
Every time line (800 tweets
per user) is on redis
5000 writes per second avg
300K reads per second
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
25. twitter
write API (from browser or client app)
follo
e per
on
user id
tweet id
wer
fanout (flockDB)
metadata
rpushx to Redis
user info from
gizmoduck
(memcached)
javier ramirez
@supercoco9
tweet info from
tweetypie
(memcached + mysql)
https://teowaki.com
your twitter
timeline
madrid rb 2013
26. pinterest object graph
from mysql+memcached
to redis
30% save on IOps
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
27. per user
Redis SortedSet, with timestamp as the score, is used to
store the users followed explicitly.
Redis SortedSet, with timestamp as the score, is used to
store the users followed implicitly
Redis SortedSet, with timestamp as the score, is used to
store the user’s explicit followers
Redis SortedSet, with timestamp as the score, is used to
store the user’s implicit followers
Redis Set is used to store boards followed explicitly
Redis Set is used to store boards unfollowed explicitly
per board
Redis Hash is used to store a board’s explicit followers
Redis Set is used to store a board’s explicit unfollowers
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
28. viacom
Object dependency graph. Cache on steroids
Redis as a queue for background jobs
Activity tracking and view counts buffering before saving
to mysql
Lua script working on a slave to recalculate ranking and
popularity of contents, then send the data to master.
The new process takes 1/60th less than the old one in
mysql
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
29. openredis
redis as a service on AWS
serving over 1000
customers with a single
machine
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
30. when things go wrong
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
31. the instagram case
moving from redis to
cassandra: 75% savings on
servers
lesson learnt:
know when redis is not appropriate
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
32. the twilio case
credit card hell
lesson learnt:
know what you are doing.
Don't change config on the fly
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
33. how teowaki is using redis
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
34. Abusing sidekiq/resque
keep track of every activity in the system, even if you
don't need them all right now:
- every page view
- every API request
- every time a record is created/updated/deleted
benefits:
- highly decoupled system
- easier to divide into services
- you can add behaviour without changing your app
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
35. intermediate cache
* As a very fast lightweight storage for analytics data before
sending them to our google bigquery based solution
* As a cache for AR attributes frequently “plucked” or included
and plucked (names, nicknames, guids, delegated or
included model names...)
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
36. Some of our uses of Lua
Expiring attributes inside a Redis hash
Inserting notifications into a list only if there are not
pending notifications from the same user for the
same scope
Paginating a list by an attribute
Manipulating JSON directly at the Redis layer
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
37. counters
Atomic counters can be safely invoked concurrently
from anywhere, so you can implement “like”
features, global sequences or usage monitoring
systems in highly concurrent applications for free.
You can share your counters with any other internal
application and still be sure they won't collide.
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
38. Temporary data
Redis allows us to self expire keys after a time has
passed. You can use this mechanism for a simple
cache
If you operate on a key with expiration time, you
can change its value and still keep the expiration
going. By combining a decrementing counter with
an expiration time, implementing usage quotas is
trivial
Also, you can inspect which keys you have in your
server efficiently using SCAN commands
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
39. bloom filters
bloom filter: space-efficient probabilistic data
structure that is used to test whether an element is
a member of a set. False positive matches are
possible, but false negatives are not.
Redis bit operations make easy to implement bloom
filters
We are using bloom filters for checking uniqueness
of user names and reserved words without going to
postgresql
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
40. nginx + lua + redis
apache + mruby + redis
Multiple levels of cache by using Redis on the webserver/
middleware layer
http://wiki.nginx.org/HttpRedis
https://github.com/jodosha/redis-store
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
41. summarizing
* Redis is more powerful than it seems
* Very fast, easy to use, simple, good documentation
* In-memory data structures, distributed, shared and persisted
* Good as data store, intermediate data store, cache or queue
* Lots of use cases, both in huge and smaller systems
You should probably use it a lot more
javier ramirez
@supercoco9
https://teowaki.com
madrid rb 2013
42. Find related links at
https://teowaki.com/teams/javier-community/link-categories/redis
¡Gracias!
Javier Ramírez
@supercoco9
madrid rb 2013