This document summarizes a presentation about how Spil Games achieves high availability for their globally distributed databases. It discusses using master-slave replication, multi-master replication, database clustering, and geographic redundancy. It also covers scaling out through horizontal partitioning and federated partitioning. The key points are abstracting the storage layer using a platform built with Erlang, utilizing MySQL and other databases, and sharding data using a bucket model.
Percona Live London 2014: Serve out any page with an HA Sphinx environmentspil-engineering
At Spil Games we have been using Sphinx for over five years now. At first we used it to offload full text search queries to our MySQL databases, but one year ago we started to use it in a different way: we now serve out any page to any of our 26M daily active users dynamically. This means on every visitor for every pageview he/she makes we will make at least one invocation to Sphinx Search. This session will describe on how we need to pick the right content for our users dynamically based upon their browser capabilities (Flash, HTML5, WebGL,etc) in millisecond ranges using the attribute filtering capabilities from Sphinx.
Using Sphinx as one of the main building blocks of our architectural foundations means we have to set it up as highly available as possible and be smart with data loading. Using the background indexing method we relied on for years makes our Sphinx instances respond slower during indexing and graceful index reloading. To overcome this we now use the real-time indexes whenever we update our content database. I will cover the various scenarios of HA and index population we went through with their pros and cons.
Johnny Miller – Cassandra + Spark = Awesome- NoSQL matters Barcelona 2014NoSQLmatters
Johnny Miller – Cassandra + Spark = Awesome
This talk will discuss how Cassandra and Spark can work together to deliver real-time analytics. This is a technical discussion that will introduce the attendees to the basic principals on Cassandra and Spark, why they work well together and examples usecases.
Percona Live London 2014: Serve out any page with an HA Sphinx environmentspil-engineering
At Spil Games we have been using Sphinx for over five years now. At first we used it to offload full text search queries to our MySQL databases, but one year ago we started to use it in a different way: we now serve out any page to any of our 26M daily active users dynamically. This means on every visitor for every pageview he/she makes we will make at least one invocation to Sphinx Search. This session will describe on how we need to pick the right content for our users dynamically based upon their browser capabilities (Flash, HTML5, WebGL,etc) in millisecond ranges using the attribute filtering capabilities from Sphinx.
Using Sphinx as one of the main building blocks of our architectural foundations means we have to set it up as highly available as possible and be smart with data loading. Using the background indexing method we relied on for years makes our Sphinx instances respond slower during indexing and graceful index reloading. To overcome this we now use the real-time indexes whenever we update our content database. I will cover the various scenarios of HA and index population we went through with their pros and cons.
Johnny Miller – Cassandra + Spark = Awesome- NoSQL matters Barcelona 2014NoSQLmatters
Johnny Miller – Cassandra + Spark = Awesome
This talk will discuss how Cassandra and Spark can work together to deliver real-time analytics. This is a technical discussion that will introduce the attendees to the basic principals on Cassandra and Spark, why they work well together and examples usecases.
This presentation can help you to apply partioning when appropriate, and to avoid problems when using it. The oneliner is: Simple Works Best. The illustrating demos are on Postgres12 (maybe -13 by the time of presenting) and show some of the problems and solutions that Partitioning can provide. Some of this “experience” is quite old and the demo runs near-identical on Oracle…
These problems are the same on any database.
These are the slides from my talk at Hulu in March 2015 discussing Apache Spark & Cassandra. I cover the evolution of data from a single machine to RDBMS (MySQL is the primary example) to big data systems.
On the Spark side, I covered batch jobs, streaming, Apache Kafka, an introduction to machine learning, clustering, logistic regression and recommendations systems (collaborative filtering).
The talk was recorded and is available on youtube: https://www.youtube.com/watch?v=_gFgU3phogQ
Unified Batch & Stream Processing with Apache SamzaDataWorks Summit
The traditional lambda architecture has been a popular solution for joining offline batch operations with real time operations. This setup incurs a lot of developer and operational overhead since it involves maintaining code that produces the same result in two, potentially different distributed systems. In order to alleviate these problems, we need a unified framework for processing and building data pipelines across batch and stream data sources.
Based on our experiences running and developing Apache Samza at LinkedIn, we have enhanced the framework to support: a) Pluggable data sources and sinks; b) A deployment model supporting different execution environments such as Yarn or VMs; c) A unified processing API for developers to work seamlessly with batch and stream data. In this talk, we will cover how these design choices in Apache Samza help tackle the overhead of lambda architecture. We will use some real production use-cases to elaborate how LinkedIn leverages Apache Samza to build unified data processing pipelines.
Speaker
Navina Ramesh, Sr. Software Engineer, LinkedIn
Large-Scale Stream Processing in the Hadoop EcosystemGyula Fóra
Distributed stream processing is one of the hot topics in big data analytics today. An increasing number of applications are shifting from traditional static data sources to processing the incoming data in real-time. Performing large scale stream processing or analysis requires specialized tools and techniques which have become publicly available in the last couple of years.
This talk will give a deep, technical overview of the top-level Apache stream processing landscape. We compare several frameworks including Spark, Storm, Samza and Flink. Our goal is to highlight the strengths and weaknesses of the individual systems in a project-neutral manner to help selecting the best tools for the specific applications. We will touch on the topics of API expressivity, runtime architecture, performance, fault-tolerance and strong use-cases for the individual frameworks.
Mysql NDB Cluster's Asynchronous Parallel Design for High PerformanceBernd Ocklin
MySQL's NDB Cluster is a partitioned distributed database engine that is entirely build around a parallel virtual machine with an event driven asynchronous design. Using this design NDB can execute even single queries in parallel and scales linearly handling terabytes of sharded data in a real-time fashion.
Using Riak for Events storage and analysis at Booking.comDamien Krotkine
At Booking.com, we have a constant flow of events coming from various applications and internal subsystems. This critical data needs to be stored for real-time, medium and long term analysis. Events are schema-less, making it difficult to use standard analysis tools.This presentation will explain how we built a storage and analysis solution based on Riak. The talk will cover: data aggregation and serialization, Riak configuration, solutions for lowering the network usage, and finally, how Riak's advanced features are used to perform real-time data crunching on the cluster nodes.
Real-time data analytics with Cassandra at ilandJulien Anguenot
This presentation, from the 2014 Datastax Cassandra Tech Day in Houston, TX, is a quick overview on how iland Internet Solutions leverages Cassandra as its sole data storage platform for performance metrics as well as configuration across multiple-data centers in the US, EU and Asia.
iland exposes these metrics to its customers trough its ECS portal application which is covering a wealth of functionality including offering visibility into resource consumption, billing, performance, alerts, the impact of change and other key areas. The platform also provides predictive analytics that help companies monitor performance, achieve consistency and anticipate growth requirements.
Cassandra Meetup: Real-time Analytics using Cassandra, Spark and Shark at OoyalaDataStax Academy
What You Will Learn At This Meetup:
• Review of Cassandra analytics landscape: Hadoop & HIVE
• Custom input formats to extract data from Cassandra
• How Spark & Shark increase query speed & productivity over standard solutions
Abstract
This session covers our experience with using the Spark and Shark frameworks for running real-time queries on top of Cassandra data.We will start by surveying the current Cassandra analytics landscape, including Hadoop and HIVE, and touch on the use of custom input formats to extract data from Cassandra. We will then dive into Spark and Shark, two memory-based cluster computing frameworks, and how they enable often dramatic improvements in query speed and productivity, over the standard solutions today.
About Evan Chan
Evan Chan is a Software Engineer at Ooyala. In his own words: I love to design, build, and improve bleeding edge distributed data and backend systems using the latest in open source technologies. I am a big believer in GitHub, open source, and meetups, and have given talks at conferences such as the Cassandra Summit 2013.
South Bay Cassandra Meetup URL: http://www.meetup.com/DataStax-Cassandra-South-Bay-Users/events/147443722/
Scaling out Tensorflow-as-a-Service on Spark and Commodity GPUsJim Dowling
Scaling out Tensorflow-as-a-Service on Spark and Commodity GPUs, including AllReduce, Horovod, and how commodity GPU servers, such as DeepLearning11, will gain adoption.
MongoDB has taken a clear lead in adoption among the new generation of databases, including the enormous variety of NoSQL offerings. A key reason for this lead has been a unique combination of agility and scalability. Agility provides business units with a quick start and flexibility to maintain development velocity, despite changing data and requirements. Scalability maintains that flexibility while providing fast, interactive performance as data volume and usage increase. We'll address the key organizational, operational, and engineering considerations to ensure that agility and scalability stay aligned at increasing scale, from small development instances to web-scale applications. We will also survey some key examples of highly-scaled customer applications of MongoDB.
This presentation can help you to apply partioning when appropriate, and to avoid problems when using it. The oneliner is: Simple Works Best. The illustrating demos are on Postgres12 (maybe -13 by the time of presenting) and show some of the problems and solutions that Partitioning can provide. Some of this “experience” is quite old and the demo runs near-identical on Oracle…
These problems are the same on any database.
These are the slides from my talk at Hulu in March 2015 discussing Apache Spark & Cassandra. I cover the evolution of data from a single machine to RDBMS (MySQL is the primary example) to big data systems.
On the Spark side, I covered batch jobs, streaming, Apache Kafka, an introduction to machine learning, clustering, logistic regression and recommendations systems (collaborative filtering).
The talk was recorded and is available on youtube: https://www.youtube.com/watch?v=_gFgU3phogQ
Unified Batch & Stream Processing with Apache SamzaDataWorks Summit
The traditional lambda architecture has been a popular solution for joining offline batch operations with real time operations. This setup incurs a lot of developer and operational overhead since it involves maintaining code that produces the same result in two, potentially different distributed systems. In order to alleviate these problems, we need a unified framework for processing and building data pipelines across batch and stream data sources.
Based on our experiences running and developing Apache Samza at LinkedIn, we have enhanced the framework to support: a) Pluggable data sources and sinks; b) A deployment model supporting different execution environments such as Yarn or VMs; c) A unified processing API for developers to work seamlessly with batch and stream data. In this talk, we will cover how these design choices in Apache Samza help tackle the overhead of lambda architecture. We will use some real production use-cases to elaborate how LinkedIn leverages Apache Samza to build unified data processing pipelines.
Speaker
Navina Ramesh, Sr. Software Engineer, LinkedIn
Large-Scale Stream Processing in the Hadoop EcosystemGyula Fóra
Distributed stream processing is one of the hot topics in big data analytics today. An increasing number of applications are shifting from traditional static data sources to processing the incoming data in real-time. Performing large scale stream processing or analysis requires specialized tools and techniques which have become publicly available in the last couple of years.
This talk will give a deep, technical overview of the top-level Apache stream processing landscape. We compare several frameworks including Spark, Storm, Samza and Flink. Our goal is to highlight the strengths and weaknesses of the individual systems in a project-neutral manner to help selecting the best tools for the specific applications. We will touch on the topics of API expressivity, runtime architecture, performance, fault-tolerance and strong use-cases for the individual frameworks.
Mysql NDB Cluster's Asynchronous Parallel Design for High PerformanceBernd Ocklin
MySQL's NDB Cluster is a partitioned distributed database engine that is entirely build around a parallel virtual machine with an event driven asynchronous design. Using this design NDB can execute even single queries in parallel and scales linearly handling terabytes of sharded data in a real-time fashion.
Using Riak for Events storage and analysis at Booking.comDamien Krotkine
At Booking.com, we have a constant flow of events coming from various applications and internal subsystems. This critical data needs to be stored for real-time, medium and long term analysis. Events are schema-less, making it difficult to use standard analysis tools.This presentation will explain how we built a storage and analysis solution based on Riak. The talk will cover: data aggregation and serialization, Riak configuration, solutions for lowering the network usage, and finally, how Riak's advanced features are used to perform real-time data crunching on the cluster nodes.
Real-time data analytics with Cassandra at ilandJulien Anguenot
This presentation, from the 2014 Datastax Cassandra Tech Day in Houston, TX, is a quick overview on how iland Internet Solutions leverages Cassandra as its sole data storage platform for performance metrics as well as configuration across multiple-data centers in the US, EU and Asia.
iland exposes these metrics to its customers trough its ECS portal application which is covering a wealth of functionality including offering visibility into resource consumption, billing, performance, alerts, the impact of change and other key areas. The platform also provides predictive analytics that help companies monitor performance, achieve consistency and anticipate growth requirements.
Cassandra Meetup: Real-time Analytics using Cassandra, Spark and Shark at OoyalaDataStax Academy
What You Will Learn At This Meetup:
• Review of Cassandra analytics landscape: Hadoop & HIVE
• Custom input formats to extract data from Cassandra
• How Spark & Shark increase query speed & productivity over standard solutions
Abstract
This session covers our experience with using the Spark and Shark frameworks for running real-time queries on top of Cassandra data.We will start by surveying the current Cassandra analytics landscape, including Hadoop and HIVE, and touch on the use of custom input formats to extract data from Cassandra. We will then dive into Spark and Shark, two memory-based cluster computing frameworks, and how they enable often dramatic improvements in query speed and productivity, over the standard solutions today.
About Evan Chan
Evan Chan is a Software Engineer at Ooyala. In his own words: I love to design, build, and improve bleeding edge distributed data and backend systems using the latest in open source technologies. I am a big believer in GitHub, open source, and meetups, and have given talks at conferences such as the Cassandra Summit 2013.
South Bay Cassandra Meetup URL: http://www.meetup.com/DataStax-Cassandra-South-Bay-Users/events/147443722/
Scaling out Tensorflow-as-a-Service on Spark and Commodity GPUsJim Dowling
Scaling out Tensorflow-as-a-Service on Spark and Commodity GPUs, including AllReduce, Horovod, and how commodity GPU servers, such as DeepLearning11, will gain adoption.
MongoDB has taken a clear lead in adoption among the new generation of databases, including the enormous variety of NoSQL offerings. A key reason for this lead has been a unique combination of agility and scalability. Agility provides business units with a quick start and flexibility to maintain development velocity, despite changing data and requirements. Scalability maintains that flexibility while providing fast, interactive performance as data volume and usage increase. We'll address the key organizational, operational, and engineering considerations to ensure that agility and scalability stay aligned at increasing scale, from small development instances to web-scale applications. We will also survey some key examples of highly-scaled customer applications of MongoDB.
NoSQL - MongoDB. Agility, scalability, performance. I am going to talk about the basis of NoSQL and MongoDB. Why some projects requires RDBMs and another NoSQL databases? What are the pros and cons to use NoSQL vs. SQL? How data are stored and transefed in MongoDB? What query language is used? How MongoDB supports high availability and automatic failover with the help of the replication? What is sharding and how it helps to support scalability?. The newest level of the concurrency - collection-level and document-level.
Getting started with Spark & Cassandra by Jon Haddad of DatastaxData Con LA
Massively scalable, always on, and ridiculously fast. Apache Cassandra is the database chosen by Apple, Netflix, and 30 of the Fortune 100 to power their critical infrastructure. How do we analyze petabytes of data, whether it be massive batching or as it’s ingested via streaming with Apache Kafka? Enter Apache Spark. Challenging MapReduce head on, Apache Spark offers powerful constructs that make it possible to slice and dice your data, whether it be through machine learning, graph queries, as well as transformations familiar to people with functional programming backgrounds such as map, filter, and reduce. Step away ready to rock with the most powerful distributed database, scalable messaging, and analytics platform on the planet.
Watch the video here
https://www.youtube.com/watch?v=X-FKmKc9hkI
Tugdual Grall - Real World Use Cases: Hadoop and NoSQL in ProductionCodemotion
What’s important about a technology is what you can use it to do. I’ve looked at what a number of groups are doing with Apache Hadoop and NoSQL in production, and I will relay what worked well for them and what did not. Drawing from real world use cases, I show how people who understand these new approaches can employ them well in conjunction with traditional approaches and existing applications. Thread Detection, Datawarehouse optimization, Marketing Efficiency, Biometric Database are some examples exposed during this presentation.
Using Graph Analysis and Fraud Detection in the Fintech IndustryStanka Dalekova
Paysafe provides simple and secure payment solutions to businesses of all sizes around the world, processing billions of payment dollars a year. This, combined with the focus of flawless customer experience and real-time money transfer, makes it a candidate for the “dark side” of the payments industry: fraudsters, money launderers, etc. With traditional data storage techniques such as relational technologies, it is almost impossible to see beyond individual accounts to the connections between them. In this session see how Paysafe implemented the property graph technologies in Oracle Spatial and Graph and Oracle Database, including its fast, built-in, in-memory graph analytics, to perform fast graph queries that identify patterns of fraud.
Using Graph Analysis and Fraud Detection in the Fintech IndustryStanka Dalekova
Paysafe provides simple and secure payment solutions to businesses of all sizes around the world, processing billions of payment dollars a year. This, combined with the focus of flawless customer experience and real-time money transfer, makes it a candidate for the “dark side” of the payments industry: fraudsters, money launderers, etc. With traditional data storage techniques such as relational technologies, it is almost impossible to see beyond individual accounts to the connections between them. In this session see how Paysafe implemented the property graph technologies in Oracle Spatial and Graph and Oracle Database, including its fast, built-in, in-memory graph analytics, to perform fast graph queries that identify patterns of fraud.
MySQL performance monitoring using Statsd and Graphite (PLUK2013)spil-engineering
MySQL performance monitoring using Statsd and Graphite (PLUK2013)
Note: this is a placeholder for the presentation next Tuesday at the Percona Live London
Every year the financial industry loses billions because of fraud while in the meantime fraudsters are coming up with more and more sophisticated patterns.
Financial institutions have to find the balance between fraud protection and negative customer experience. Fraudsters bury their patterns in lots of data, but the traditional technologies are not designed to detect fraud in real-time or to see patterns beyond the individual account.
Analyzing relations with graph databases helps uncover these larger complex patterns and speeds up suspicious behavior identification.
Furthermore, graph databases enable fast and effective real-time link queries and passing context to machine learning models.
The earlier fraud pattern or network is identified, the faster the activity is blocked. As a result, losses and fines are minimized.
In KDD2011, Vijay Narayanan (Yahoo!) and Milind Bhandarkar (Greenplum Labs, EMC) conducted a tutorial on "Modeling with Hadoop". This is the first half of the tutorial.
Active Record 4.0 includes all sorts of exciting support for PostgreSQL! In this presentation, I show many of these improvements, and discuss why these are important for Web developers. If you haven't yet adopted PostgreSQL, now might be a great time and chance to do so.
Description of some of the elements that go in to creating a PostgreSQL-as-a-Service for organizations with many teams and a diverse ecosystem of applications and teams.
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024Neo4j
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
4. 4
• Company
founded
in
2001
• 350+
employees
world
wide
• 180M+
unique
visitors
per
month
• 45
portals
in
19
languages
• Casual
games
• Social
games
• Real
=me
mul=player
games
• Mobile
games
• 35+
MySQL
clusters
• 60k
queries
per
second
(3.5
billion
qpd)
Facts
5. 5
Geographic Reach
180
Million
Monthly
Ac=ve
Users(*)
Source:
(*)
Google
Analy3cs,
August
2012
• Over
45
localized
portals
in
19
languages
• Mul=
pla9orm:
web,
mobile,
tablet
• Focus
on
casual
and
social
games
• 180M
MAU
per
month
(30M
YoY
growth)
• Over
50M
registered
users
6. 6
Girls,
Teens
and
Family
spielen.com
juegos.com
gamesgames.com
games.co.uk
Brands
8. 8
• What
does
it
exactly
mean?
Retaining globally distributed HA
9. 9
Wikipedia:
High
availability
is
a
system
design
approach
and
associated
service
implementa=on
that
ensures
a
prearranged
level
of
opera=onal
performance
will
be
met
during
a
contractual
measurement
period.
Oracle:
• Availability
of
resources
in
a
computer
system
What is high availability?
10. 10
• Master
with
(many)
slave(s)
How do we reach HA with MySQL?
Master
Slave Slave Slave
11. 11
• Master
with
(many)
slave(s)
• Mul=
Master
How do we reach HA with MySQL?
Master
Slave
Master
Slave
12. 12
• Master
with
(many)
slave(s)
• Mul=
Master
• Clustering
How do we reach HA with MySQL?
MysqldMysqld
ndbd
ndbd ndbd
ndbd
ndbd
mgmt
13. 13
• Master
with
(many)
slave(s)
• Mul=
Master
• Clustering
• Geographical
redundancy
How do we reach HA with MySQL?
Master
local DC
Slave local
DC
Slave Asia Slave US
14. 14
• Scale
up
• Ver=cal
• Faster
CPU/Memory/disks
• Expensive
• Costs
mul=ply
in
same
rate
as
#
of
nodes
• Scale
out
• Horizontal
• More
(small)
machines
• Inexpensive
• Par==oning/federa=ng
(sharding)
What if we keep growing?
15. 15
• Func=onal
• Shard
your
database
func=onally
• Reads
• Add
more
slaves
(keep
them
coming!)
• Writes
• More
disks
• Horizontal
par==oning
• Federated
par==ons
Scale out
16. 16
• Breaking
up
tables
in
small
parts
on
the
same
host
• Par==oned
on
a
column
• Infinite
growth
(as
long
as
you
add
diskspace)
• Less
used
data
to
slower
(cheaper)
disks
• No
stored
procedures,
func=ons,
etc
• Uneven
usage
of
par==ons
(hash
par==on
may
help)
• Once
wrihen,
data
remains
on
the
par==on
Horizontal partitioning
17. 17
• Breaking
up
your
table
in
parts
on
mul=ple
hosts
• Par==oned
on
a
column
• Infinite
growth
(as
long
as
you
add
hosts)
• Less
used
data
on
slower
hosts
• Not
supported
in
(standard)
MySQL
• Par==oning
on
applica=on
level
(or
proxy)
• Alterna=vely:
NDB
• Uneven
usage
of
par==ons
• Once
wrihen
data
(mostly)
remains
on
the
par==on
• Parallel
queries
to
retrieve
data
from
all
shards
Federated partitions (sharding)
18. 18
• Parallel
execu=on
of
sequen=al
jobs
• Limited
by
the
weakest
link
• As
fast
as
the
slowest
node
• Fix:
nonsequen=al
(asynchronous)
execu=on
Amdahl's law
22. 22
• Dependent
on
one
storage
pla9orm
• No
more
pla9orm-‐specific
query
language
• Differen=ate
writes
• Op=mis=c
(asynchronous)
• Pessimis=c
(synchronous)
• Shard
data
beher
• Par==on
on
user
and
func=on
• Cluster
informa=on
by
users,
not
by
func=on
• Global
expansion
• Par==on
on
geographic
loca=on
• Solve
uneven
usage
of
data
storage
• Move
data
from
shard
to
shard
• Anything
may/could/will
fail
eventually
• Not
designed
for
the
“happy”
flow
What was our wishlist?
25. 25
New architecture overview
Server API
Application Model
Storage platform
Client-side API
Presentation layer
Physical storage
26. 26
• Everything
wrihen
in
Erlang
• Piqi
as
protocol
• binary
• JSON
• XML
• SSP
u=lizes
local
caching
(memcache)
• Flexible
(persistent)
storage
layer
• MySQL
(various
flavors)
• Membase/Couchbase
• Could
be
any
other
storage
product
• MQs
(DWH
updates)
Our building blocks
28. 28
• Func=onal
language
• High
availability:
designed
for
telecom
solu=ons
• Excels
at
concurrency,
distribu=on,
fault
tolerance
• Do
more
with
less!
• Other
companies
using
Erlang:
Why Erlang?
29. 29
• What
is
the
bucket
model?
• Each
record
has
one
unique
owner
ahribute
(GID)
• GID
(Global
IDen=fier)
iden=fying
different
types
• Bucket(s)
per
func=onality
• Bucket
is
structured
data
• Ahributes
contain
data
of
records
• Ahributes
do
not
have
to
correspond
to
schema
How do we shard?
36. 36
• Nearest
datacenter
(DC)
to
the
end
user
• Satellite
DC
• Processing
and
caching
• Do
not
own/store
data
• Storage
DC
• Processing,
caching
and
persistent
storage
• Store
all
same
user
data
in
same
DC
• Par==on
on
user
globally
• Global
IDen=fier
per
user
Global distribution
37. 37
• Contains
GIDs
and
their
master
DC
• GIDs
master
DC
predefined
• Migrated
GIDs
get
updated
The lookup server
38. 38
• Globally
sharded
on
GID
• (local)
GID
Lookup
How does this work?
GID
lookup
Shard 1 Shard 2
Persistent
storage
40. 40
• Spread
data
even
on
shards
• Migra=on
of
buckets
between
shards
• GID
migra=on
between
DCs
• Crea=ng
a
new
storage
DC
needs
data
migra=on
• Users
will
automa=cally
be
migrated
a‚er
visi=ng
another
DC
many
=mes
Why do we need data migration?
41. 41
• Versioning
on
bucket
defini=ons
• GIDs
are
assigned
to
a
bucket
version
• Data
in
old
bucket
versions
remain
(read
only)
• New
data
only
gets
wrihen
to
new
bucket
version
• Updates
migrate
data
to
new
bucket
version
• Migrates
can
be
triggered
Seamless schema upgrades
42. 42
Seamless schema upgrades
Demobucket
v1
GID
1234
1235
1236
1237
1238
1239
name
Roy
Moss
Jen
Douglas
Denholm
Richmond
Demobucket
v2
GID
name
gender
GID
1241
name
Patricia
gender
f
GID
1241
1235
name
Patricia
Moss
gender
f
m
GID
1234
1236
1237
1238
1239
name
Roy
Jen
Douglas
Denholm
Richmond
GID
1234
1237
1238
1239
name
Roy
Douglas
Denholm
Richmond
GID
1241
1235
1236
name
Patricia
Moss
Jen
gender
f
m
f
43. 43
• Every
cluster
(two
masters)
will
contain
two
shards
• Data
wrihen
interleaved
• HA
for
both
shards
• No
warmup
needed
• Both
masters
ac=ve
and
“warmed
up”
• Slaves
added
(other
DC)
for
HA
and
backup
Multi Master writes
SSP
Shard
1
Shard
2
44. 44
• SPAPI
is
in
place
• SSP
is
(mostly)
running
in
shadow
mode
• GID
buckets
running
in
produc=on
• Ac=vity
feed
system
first
to
produc=on
• Satellite
DC
in
early
2013!
Where do we stand now?
47. 47
• Presenta=on
can
be
found
at:
hhp://spil.com/perconalondon2012
• If
you
wish
to
contact
me:
art@spilgames.com
• Don’t
forget
to
rate
my
talk!
Thank you!