Twitter is a good example for next generation real-time web applications, but building such an application imposes challenges such as handling an every growing volume of tweets and responses, as well as a large number of concurrent users, who continually *listen* for tweets from users (or topics) they follow. During this session we will review some of the key design principles addressing these challenges, including alternatives *NoSQL* alternatives and blackboard patterns. We will be using Twitter as a use case, while learning how to apply these to any real-time we application
Revitalizing Walmart's Aging Architecture for Web ScaleKevin Webber
Learn how Walmart embraced the concepts of reactive programming, microservices, and domain-driven design to achieve results impossible only a decade ago.
Extracting Value from IOT using Azure Cosmos DB, Azure Synapse Analytics and ...HostedbyConfluent
Due to explosion of IoT, we have streaming data that needs to be processed in real-time. This needs to be made available for applications as well as analytics scenarios such as anomaly detection. This workshop presents a solution using Confluent Cloud on Azure, Azure Cosmos DB and Azure Synapse Analytics which can be connected in a secure way within Azure VNET using Azure Private link configured on Kafka clusters.
Web Scale GIS has become the de-facto collaboration tool to accommodate Multi-agency/govt/org responses to major disasters (sometimes concurrent events). We describe a real-world DevOps journey to facilitate flexible IaaS/PaaS hybrid system capable of elastic accommodation of need within tight budgetary constraints. We start at genesis when our team inherited a prototype quality geospatial disaster management system in response to Katrina. Overnight this became a production system for Sandy. Providing the discipline to maturely scale, a DevOps managed cloud continues to be our approach to stay ahead of the problem. Along the way we had to define, tailor, engineer, educate and learn quickly on the job. Through this process we have built out a more robust and reliable stack that has significantly evolved from where it started. Hear about how we fashioned our DevOps journey and the valuable miles we earned along the way.
Peter Kessler (GIS SME at eGlobalTech)
(http://www.eglobaltech.com/services/cloud-computing/)
Peter Kessler has been a architect of major Enterprise Geographic Information Systems and spatial data expert for nearly 20 years. More recently he found what he's hoped for all of his career: a way to empower his most demanding customers with rapid and flexible solutions...That would be DevOps :-) With a contagious enthusiasm, he enjoys sharing the benefits of a DevOps Managed hosting approach for Web GIS infrastructure.
Event Driven Architecture with a RESTful Microservices Architecture (Kyle Ben...confluent
Tinder’s Quickfire Pipeline powers all things data at Tinder. It was originally built using AWS Kinesis Firehoses and has since been extended to use both Kafka and other event buses. It is the core of Tinder’s data infrastructure. This rich data flow of both client and backend data has been extended to service a variety of needs at Tinder, including Experimentation, ML, CRM, and Observability, allowing backend developers easier access to shared client side data. We perform this using many systems, including Kafka, Spark, Flink, Kubernetes, and Prometheus. Many of Tinder’s systems were natively designed in an RPC first architecture.
Things we’ll discuss decoupling your system at scale via event-driven architectures include:
– Powering ML, backend, observability, and analytical applications at scale, including an end to end walk through of our processes that allow non-programmers to write and deploy event-driven data flows.
– Show end to end the usage of dynamic event processing that creates other stream processes, via a dynamic control plane topology pattern and broadcasted state pattern
– How to manage the unavailability of cached data that would normally come from repeated API calls for data that’s being backfilled into Kafka, all online! (and why this is not necessarily a “good” idea)
– Integrating common OSS frameworks and libraries like Kafka Streams, Flink, Spark and friends to encourage the best design patterns for developers coming from traditional service oriented architectures, including pitfalls and lessons learned along the way.
– Why and how to avoid overloading microservices with excessive RPC calls from event-driven streaming systems
– Best practices in common data flow patterns, such as shared state via RocksDB + Kafka Streams as well as the complementary tools in the Apache Ecosystem.
– The simplicity and power of streaming SQL with microservices
Revitalizing Walmart's Aging Architecture for Web ScaleKevin Webber
Learn how Walmart embraced the concepts of reactive programming, microservices, and domain-driven design to achieve results impossible only a decade ago.
Extracting Value from IOT using Azure Cosmos DB, Azure Synapse Analytics and ...HostedbyConfluent
Due to explosion of IoT, we have streaming data that needs to be processed in real-time. This needs to be made available for applications as well as analytics scenarios such as anomaly detection. This workshop presents a solution using Confluent Cloud on Azure, Azure Cosmos DB and Azure Synapse Analytics which can be connected in a secure way within Azure VNET using Azure Private link configured on Kafka clusters.
Web Scale GIS has become the de-facto collaboration tool to accommodate Multi-agency/govt/org responses to major disasters (sometimes concurrent events). We describe a real-world DevOps journey to facilitate flexible IaaS/PaaS hybrid system capable of elastic accommodation of need within tight budgetary constraints. We start at genesis when our team inherited a prototype quality geospatial disaster management system in response to Katrina. Overnight this became a production system for Sandy. Providing the discipline to maturely scale, a DevOps managed cloud continues to be our approach to stay ahead of the problem. Along the way we had to define, tailor, engineer, educate and learn quickly on the job. Through this process we have built out a more robust and reliable stack that has significantly evolved from where it started. Hear about how we fashioned our DevOps journey and the valuable miles we earned along the way.
Peter Kessler (GIS SME at eGlobalTech)
(http://www.eglobaltech.com/services/cloud-computing/)
Peter Kessler has been a architect of major Enterprise Geographic Information Systems and spatial data expert for nearly 20 years. More recently he found what he's hoped for all of his career: a way to empower his most demanding customers with rapid and flexible solutions...That would be DevOps :-) With a contagious enthusiasm, he enjoys sharing the benefits of a DevOps Managed hosting approach for Web GIS infrastructure.
Event Driven Architecture with a RESTful Microservices Architecture (Kyle Ben...confluent
Tinder’s Quickfire Pipeline powers all things data at Tinder. It was originally built using AWS Kinesis Firehoses and has since been extended to use both Kafka and other event buses. It is the core of Tinder’s data infrastructure. This rich data flow of both client and backend data has been extended to service a variety of needs at Tinder, including Experimentation, ML, CRM, and Observability, allowing backend developers easier access to shared client side data. We perform this using many systems, including Kafka, Spark, Flink, Kubernetes, and Prometheus. Many of Tinder’s systems were natively designed in an RPC first architecture.
Things we’ll discuss decoupling your system at scale via event-driven architectures include:
– Powering ML, backend, observability, and analytical applications at scale, including an end to end walk through of our processes that allow non-programmers to write and deploy event-driven data flows.
– Show end to end the usage of dynamic event processing that creates other stream processes, via a dynamic control plane topology pattern and broadcasted state pattern
– How to manage the unavailability of cached data that would normally come from repeated API calls for data that’s being backfilled into Kafka, all online! (and why this is not necessarily a “good” idea)
– Integrating common OSS frameworks and libraries like Kafka Streams, Flink, Spark and friends to encourage the best design patterns for developers coming from traditional service oriented architectures, including pitfalls and lessons learned along the way.
– Why and how to avoid overloading microservices with excessive RPC calls from event-driven streaming systems
– Best practices in common data flow patterns, such as shared state via RocksDB + Kafka Streams as well as the complementary tools in the Apache Ecosystem.
– The simplicity and power of streaming SQL with microservices
RightScale Webinar: How RightScale Architects Its Databases (for Worldwide Sc...RightScale
Is your database holding back your application? Find out how we at RightScale use SQL and NoSQL databases such as MySQL and Cassandra to provide a scalable, distributed, and highly available service around the world, that is designed to recover from failures of a whole cloud region.
In this webinar, we will:
- Share the data taxonomy for specific RightScale systems
- Give you insights on how to think through your own data taxonomy
- Go deep into RightScale's distributed database architecture
Join RightScale's VP of Engineering and Chief Architect and learn directly from the team who architected RightScale's databases for scale, HA and DR.
Amazon Redshift, Customer Acquisition Cost & Advertising ROI presented with A...Amazon Web Services
In today's world, consumer habits change fast and marketing decisions need to be made within seconds, not days. Delivering engaging advertising experiences requires real time, high performing architectures that provide digital advertisers the ability to measure and improve the performance of their campaigns and tie them more closely to corporate goals. The insights gleaned from the massive amounts of data collected can then be used to dynamically adjust media spend and creative execution for optimal performance. The AWS Cloud enables you to deliver marketing content and advertisements with the levels of availability, performance, and personalization that your customers expect. Plus, AWS lowers your costs. Join us to learn about how big data and low latency / high performing architectures are changing the game for digital advertising.
This paper purpose for give knowledge about Cloud Computing. By studied in definition, evolution, feature, advantage and disadvantage and type of service of Cloud Computing. These topics will be the basis data before talk to topic of research in Cloud Computing.
Introducing the Hub for Data OrchestrationAlluxio, Inc.
Data Orchestration Summit 2020 organized by Alluxio
https://www.alluxio.io/data-orchestration-summit-2020/
Introducing the Hub for Data Orchestration
Danny Linden, Chapter Lead Software Engineer (Ryte)
About Alluxio: alluxio.io
Engage with the open source community on slack: alluxio.io/slack
AWS is hosting the first FSI Cloud Symposium in Hong Kong, which will take place on Thursday, March 23, 2017 at Grand Hyatt Hotel. The event will bring together FSI customers, industry professional and AWS experts, to explore how to turn the dream of transformation, innovation and acceleration into reality by exploiting Cloud, Voice to Text and IoT technologies. The packed agenda includes expert sessions on a host of pressing issues, such as security and compliance, as well as customer experience sharing on how cloud computing is benefiting the industry.
Speaker: Olivier Klein, Solutions Architect, AWS
If you implement a microservice architecture correctly, you will end up with a proliferation of different microservices; with multiple instances of each one for redundancy. Find out how you to get microservices to automatically discover each other, share a configuration with real-time updates. See how to eliminate server management altogether with "serverless" microservice frameworks.
Getting started with azure event hubs and stream analytics servicesEastBanc Tachnologies
Author: Vladimir Bychkov, www.eastbanctech.com
The total amount of data in the world almost doubles every 2 years. Storing data for offline processing is no longer a viable business model. In the past few years, new technologies for real-time data processing emerged. Microsoft Azure offers a comprehensive set of tools to ingest and process data in motion. In this presentation we will go over and learn how to collect data from devices, how to process data in real time using Azure Stream Analytic jobs, and how to produce and handle actionable insights.
Creating a Data Driven Culture with Amazon QuickSight - Technical 201Amazon Web Services
Data drives good business decisions and a data-driven culture can help organisations increase profitability and reduce costs.
Amazon QuickSight is a very fast, cloud-powered Business Intelligence (BI) service that makes it easy for all employees to build visualisations, perform ad-hoc analysis, and quickly get business insights from their data.
Speaker: David McAmis, Consultant, Amazon Web Services
DataStax: How to Roll Cassandra into Production Without Losing your Health, M...DataStax Academy
You know Cassandra works and can solve a lot of problems, but then you try to design it into your application and things start falling apart. Stop! This is where we need to have some real talk. I've been helping organizations implement Cassandra for years. True story. I can help! It's easy to get lost in the details, but making the switch to Cassandra is a journey of many steps. This will be a system of the next 30 years so take your time, do it right and feel the happiness. It's all there for you. Your health and sanity will be intact and most importantly, your job will be better!
Kubernetes for Sales Engineers & Solutions Engineers–You Too Can Leverage Kub...AWS Chicago
AWS Community Day | Midwest 2018
Track 3
Kubernetes for Sales Engineers & Solutions Engineers–You Too Can Leverage Kubernetes To Build A Flexible Self-Service Product Demo Environment - Frank Reno, Milwaukee
Troubleshooting as Your Kafka Clusters Grow (Krunal Vora, Tinder) Kafka Summi...confluent
The graphical curve of ‘getting things done with Kafka’ vs. ‘time spent with Kafka’ rises pretty quickly before starting to plateau. Kafka, as we know, is great to get started out of the box. Eventually though, anyone who has used Kafka for a reasonable amount of time would come across problems, both trivial and hardcore. More often than not, these problems feel tedious and repetitive which could be addressed better by the Kafka community. Issues like skewed partitions and brokers, partition reassignment for a large number of topics, working with messed up consumer offsets, distributing workload evenly to partitions, monitor and view the Kafka cluster better, etc. sound familiar among the community. Come, learn from our experiences trying to overcome and automate some of the most common problems surrounding the scaling Kafka clusters at Tinder. Let’s create an environment of sharing the best practices for usage of tools around Kafka.
Kafka as an Event Store (Guido Schmutz, Trivadis) Kafka Summit NYC 2019confluent
Event Sourcing and CQRS are two popular patterns for implementing a Microservices architectures. With Event Sourcing we do not store the state of an object, but instead store all the events impacting its state. Then to retrieve an object state, we have to read the different events related to a certain object and apply them one by one. CQRS (Command Query Responsibility Segregation) on the other hand is a way to dissociate writes (Command) and reads (Query). Event Sourcing and CQRS are frequently grouped and used together to form something bigger. While it is possible to implement CQRS without Event Sourcing, the opposite is not necessarily correct. In order to implement Event Sourcing, an efficient Event Store is needed. But is that also true when combining Event Sourcing and CQRS? And what is an event store in the first place and what features should it implement? This presentation will first discuss what functionalities an event store should offer and then present how Apache Kafka can be used to implement an event store. But is Kafka good enough or do specific event store solutions such as AxonDB or Event Store provide a better solution?
AWS re:Invent 2016: Earth on AWS—Next-Generation Open Data Platforms (STG203)Amazon Web Services
Making earth observation data available by using Amazon S3 is accelerating scientific discovery and enabling the creation of new products. Attend and learn how the scale and performance of Amazon S3 lets earth scientists, researchers, startups, and GIS professionals gather and analyze planetary-scale data without worrying about limitations of bandwidth, storage, memory, or processing power. Learn how AWS is being used to combine satellite imagery, social data, and telemetry data to produce new products and services. Learn also how Amazon S3 provides much more than storage, and how an open geospatial data lake on Amazon S3 can be used as the basis for planetary-scale applications built with Amazon EMR, Amazon API Gateway, and AWS Lambda. As part of this talk, AWS customer Digital Globe demonstrates how they use open data stored in S3 to distribute high-resolution satellite imagery to their customers around the world.
Using Apache Cassandra and Apache Kafka to Scale Next Gen ApplicationsData Con LA
Adoption of open source software (OSS) at the enterprise level has flourished, as more businesses discover the considerable advantages that open source solutions hold over their proprietary counterparts, and as the enterprise mentality around open source continues to shift. We will discuss how to identify good application candidates for Apache Cassandra and Kafka as well as best practices and common pitfalls.
This presentation will also cover:
The origins of Apache Cassandra and Kafka and how these technologies have come to shape how next-gen applications are built.
Production use cases of Cassandra and Kafka: Real-time payments and buying a house (Lendi and Worldpay)
Core concepts that make the magic; Explaining the technical attributes that make your project a good fit for these technologies and the architectural patterns that make the best use of it’s capability.
Speaker: Adam Zegelin, SVP Engineering and Co-Founder, Instaclustr
As Instaclustr's founding software engineer, Adam provides the foundation knowledge of Instaclustr's capability and engineering environment. Adam is also focused on providing Instaclustr's contribution to the broader open source community on which our products and the services rely, including Apache Cassandra, Apache Spark, and other core technologies such as CoreOS and Docker. Prior to founding Instaclustr, Adam worked on large-scale big data projects with Australian Government agencies.
RightScale Webinar: How RightScale Architects Its Databases (for Worldwide Sc...RightScale
Is your database holding back your application? Find out how we at RightScale use SQL and NoSQL databases such as MySQL and Cassandra to provide a scalable, distributed, and highly available service around the world, that is designed to recover from failures of a whole cloud region.
In this webinar, we will:
- Share the data taxonomy for specific RightScale systems
- Give you insights on how to think through your own data taxonomy
- Go deep into RightScale's distributed database architecture
Join RightScale's VP of Engineering and Chief Architect and learn directly from the team who architected RightScale's databases for scale, HA and DR.
Amazon Redshift, Customer Acquisition Cost & Advertising ROI presented with A...Amazon Web Services
In today's world, consumer habits change fast and marketing decisions need to be made within seconds, not days. Delivering engaging advertising experiences requires real time, high performing architectures that provide digital advertisers the ability to measure and improve the performance of their campaigns and tie them more closely to corporate goals. The insights gleaned from the massive amounts of data collected can then be used to dynamically adjust media spend and creative execution for optimal performance. The AWS Cloud enables you to deliver marketing content and advertisements with the levels of availability, performance, and personalization that your customers expect. Plus, AWS lowers your costs. Join us to learn about how big data and low latency / high performing architectures are changing the game for digital advertising.
This paper purpose for give knowledge about Cloud Computing. By studied in definition, evolution, feature, advantage and disadvantage and type of service of Cloud Computing. These topics will be the basis data before talk to topic of research in Cloud Computing.
Introducing the Hub for Data OrchestrationAlluxio, Inc.
Data Orchestration Summit 2020 organized by Alluxio
https://www.alluxio.io/data-orchestration-summit-2020/
Introducing the Hub for Data Orchestration
Danny Linden, Chapter Lead Software Engineer (Ryte)
About Alluxio: alluxio.io
Engage with the open source community on slack: alluxio.io/slack
AWS is hosting the first FSI Cloud Symposium in Hong Kong, which will take place on Thursday, March 23, 2017 at Grand Hyatt Hotel. The event will bring together FSI customers, industry professional and AWS experts, to explore how to turn the dream of transformation, innovation and acceleration into reality by exploiting Cloud, Voice to Text and IoT technologies. The packed agenda includes expert sessions on a host of pressing issues, such as security and compliance, as well as customer experience sharing on how cloud computing is benefiting the industry.
Speaker: Olivier Klein, Solutions Architect, AWS
If you implement a microservice architecture correctly, you will end up with a proliferation of different microservices; with multiple instances of each one for redundancy. Find out how you to get microservices to automatically discover each other, share a configuration with real-time updates. See how to eliminate server management altogether with "serverless" microservice frameworks.
Getting started with azure event hubs and stream analytics servicesEastBanc Tachnologies
Author: Vladimir Bychkov, www.eastbanctech.com
The total amount of data in the world almost doubles every 2 years. Storing data for offline processing is no longer a viable business model. In the past few years, new technologies for real-time data processing emerged. Microsoft Azure offers a comprehensive set of tools to ingest and process data in motion. In this presentation we will go over and learn how to collect data from devices, how to process data in real time using Azure Stream Analytic jobs, and how to produce and handle actionable insights.
Creating a Data Driven Culture with Amazon QuickSight - Technical 201Amazon Web Services
Data drives good business decisions and a data-driven culture can help organisations increase profitability and reduce costs.
Amazon QuickSight is a very fast, cloud-powered Business Intelligence (BI) service that makes it easy for all employees to build visualisations, perform ad-hoc analysis, and quickly get business insights from their data.
Speaker: David McAmis, Consultant, Amazon Web Services
DataStax: How to Roll Cassandra into Production Without Losing your Health, M...DataStax Academy
You know Cassandra works and can solve a lot of problems, but then you try to design it into your application and things start falling apart. Stop! This is where we need to have some real talk. I've been helping organizations implement Cassandra for years. True story. I can help! It's easy to get lost in the details, but making the switch to Cassandra is a journey of many steps. This will be a system of the next 30 years so take your time, do it right and feel the happiness. It's all there for you. Your health and sanity will be intact and most importantly, your job will be better!
Kubernetes for Sales Engineers & Solutions Engineers–You Too Can Leverage Kub...AWS Chicago
AWS Community Day | Midwest 2018
Track 3
Kubernetes for Sales Engineers & Solutions Engineers–You Too Can Leverage Kubernetes To Build A Flexible Self-Service Product Demo Environment - Frank Reno, Milwaukee
Troubleshooting as Your Kafka Clusters Grow (Krunal Vora, Tinder) Kafka Summi...confluent
The graphical curve of ‘getting things done with Kafka’ vs. ‘time spent with Kafka’ rises pretty quickly before starting to plateau. Kafka, as we know, is great to get started out of the box. Eventually though, anyone who has used Kafka for a reasonable amount of time would come across problems, both trivial and hardcore. More often than not, these problems feel tedious and repetitive which could be addressed better by the Kafka community. Issues like skewed partitions and brokers, partition reassignment for a large number of topics, working with messed up consumer offsets, distributing workload evenly to partitions, monitor and view the Kafka cluster better, etc. sound familiar among the community. Come, learn from our experiences trying to overcome and automate some of the most common problems surrounding the scaling Kafka clusters at Tinder. Let’s create an environment of sharing the best practices for usage of tools around Kafka.
Kafka as an Event Store (Guido Schmutz, Trivadis) Kafka Summit NYC 2019confluent
Event Sourcing and CQRS are two popular patterns for implementing a Microservices architectures. With Event Sourcing we do not store the state of an object, but instead store all the events impacting its state. Then to retrieve an object state, we have to read the different events related to a certain object and apply them one by one. CQRS (Command Query Responsibility Segregation) on the other hand is a way to dissociate writes (Command) and reads (Query). Event Sourcing and CQRS are frequently grouped and used together to form something bigger. While it is possible to implement CQRS without Event Sourcing, the opposite is not necessarily correct. In order to implement Event Sourcing, an efficient Event Store is needed. But is that also true when combining Event Sourcing and CQRS? And what is an event store in the first place and what features should it implement? This presentation will first discuss what functionalities an event store should offer and then present how Apache Kafka can be used to implement an event store. But is Kafka good enough or do specific event store solutions such as AxonDB or Event Store provide a better solution?
AWS re:Invent 2016: Earth on AWS—Next-Generation Open Data Platforms (STG203)Amazon Web Services
Making earth observation data available by using Amazon S3 is accelerating scientific discovery and enabling the creation of new products. Attend and learn how the scale and performance of Amazon S3 lets earth scientists, researchers, startups, and GIS professionals gather and analyze planetary-scale data without worrying about limitations of bandwidth, storage, memory, or processing power. Learn how AWS is being used to combine satellite imagery, social data, and telemetry data to produce new products and services. Learn also how Amazon S3 provides much more than storage, and how an open geospatial data lake on Amazon S3 can be used as the basis for planetary-scale applications built with Amazon EMR, Amazon API Gateway, and AWS Lambda. As part of this talk, AWS customer Digital Globe demonstrates how they use open data stored in S3 to distribute high-resolution satellite imagery to their customers around the world.
Using Apache Cassandra and Apache Kafka to Scale Next Gen ApplicationsData Con LA
Adoption of open source software (OSS) at the enterprise level has flourished, as more businesses discover the considerable advantages that open source solutions hold over their proprietary counterparts, and as the enterprise mentality around open source continues to shift. We will discuss how to identify good application candidates for Apache Cassandra and Kafka as well as best practices and common pitfalls.
This presentation will also cover:
The origins of Apache Cassandra and Kafka and how these technologies have come to shape how next-gen applications are built.
Production use cases of Cassandra and Kafka: Real-time payments and buying a house (Lendi and Worldpay)
Core concepts that make the magic; Explaining the technical attributes that make your project a good fit for these technologies and the architectural patterns that make the best use of it’s capability.
Speaker: Adam Zegelin, SVP Engineering and Co-Founder, Instaclustr
As Instaclustr's founding software engineer, Adam provides the foundation knowledge of Instaclustr's capability and engineering environment. Adam is also focused on providing Instaclustr's contribution to the broader open source community on which our products and the services rely, including Apache Cassandra, Apache Spark, and other core technologies such as CoreOS and Docker. Prior to founding Instaclustr, Adam worked on large-scale big data projects with Australian Government agencies.
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010Bhupesh Bansal
Jan 22nd, 2010 Hadoop meetup presentation on project voldemort and how it plays well with Hadoop at linkedin. The talk focus on Linkedin Hadoop ecosystem. How linkedin manage complex workflows, data ETL , data storage and online serving of 100GB to TB of data.
Running a Megasite on Microsoft Technologiesgoodfriday
MySpace and Microsoft.com are two of the most-visited Web sites on the planet. Come to this session to hear about lessons learned using Microsoft technologies to run Web applications on a massive scale. Representatives from Microsoft.com talk about lessons learned using an all-Microsoft datacenter. Representatives from MySpace talk about the realities of using Microsoft technologies in a scalable, federated environment using SQL Server 2005, .NET 2.0 and IIS 6 on Windows Server 2003 64-bit editions. This session features an open Q&A with a panel of technical managers and engineers from MySpace and Microsoft.com.
Architecting and Tuning IIB/eXtreme Scale for Maximum Performance and Reliabi...Prolifics
Abstract: Recent projects have stressed the "need for speed" while handling large amounts of data, with near zero downtime. An analysis of multiple environments has identified optimizations and architectures that improve both performance and reliability. The session covers data gathering and analysis, discussing everything from the network (multiple NICs, nearby catalogs, high speed Ethernet), to the latest features of extreme scale. Performance analysis helps pinpoint where time is spent (bottlenecks) and we discuss optimization techniques (MQ tuning, IIB performance best practices) as well as helpful IBM support pacs. Log Analysis pinpoints system stress points (e.g. CPU starvation) and steps on the path to near zero downtime.
This is the course that was presented by James Liddle and Adam Vile for Waters in September 2008.
The book of this course can be found at: http://www.lulu.com/content/4334860
These are slides of the session that Jim Liddle gave at GigaSpaces Cloud Crowd event in the UK on 11th November 2009.
These slides concentrate on GigaSpaces VMWARE integration and the value proposition for using GigaSpaces for Private Clouds.
DM Radio Webinar: Adopting a Streaming-Enabled ArchitectureDATAVERSITY
Architecture matters. That's why today's innovators are taking a hard look at streaming data, an increasingly attractive option that can transform business in several ways: replacing aging data ingestion techniques like ETL; solving long-standing data quality challenges; improving business processes ranging from sales and marketing to logistics and procurement; or any number of activities related to accelerating data warehousing, business intelligence and analytics.
Register for this DM Radio Deep Dive Webinar to learn how streaming data can rejuvenate or supplant traditional data management practices. Host Eric Kavanagh will explain how streaming-first architectures can relieve data engineers from time-consuming, error-prone processes, ideally bidding farewell to those unpleasant batch windows. He'll be joined by Kevin Petrie of Attunity, who will explain why (with real-world story successes) streaming data solutions can keep the business fueled with trusted data in a timely, efficient manner for improved business outcomes.
Windows Azure - Uma Plataforma para o Desenvolvimento de AplicaçõesComunidade NetPonto
A plataforma Windows Azure abre espaço a desenvimento de aplicações utilizando o novo paradigma: "A Nuvem". Aplicações escaláveis, redundantes, e mais próximas do utilizador final. Isto tudo utilizando como base os conhecimentos que já tem e o novo Visual Studio 2010.
Monitoring as an entry point for collaborationJulien Pivotto
In the last years, we have been building complex stacks, made from lots of components. All of this backed by multiple teams. This talk will present how you can use monitoring to look at the business side and have everyone looking at the same dashboards, making cooperation a reality.
Desktop, Embedded and Mobile Apps with Vortex CaféAngelo Corsaro
In the past few years we have been experiencing an amazing proliferation of mobile and embedded platforms. Contemporary developers are increasingly faced with the challenge of writing applications that can run on desktop, mobile (e.g. Android), and on low-cost embedded platforms (e.g. Raspberry-Pi and Beaglebone). This is causing a rejuvenated interest in the Java platform as the mean to achieve the holy grail of write-once and run-everywhere. With the availability of Java environments supporting almost any kind of device in several different form factors, the missing element to the picture is an effective way of enabling communication between them.
Vortex Café is a pure Java implementation of the OMG Data Distribution Service (DDS) that enables seamless, efficient and timely data sharing across many-core machines, mobile and embedded devices.
This presentation will (1) introduce the main abstractions provided by Vortex Café, (2) provide an overview of its architecture and explain how it exploits Staged Event Driven Architectures to optimize its runtime depending of the target hardware, (3) provide an overview of the typical performance delivered by Vortex Café, and (3) get you started developing distributed Java and Scala applications with Vortex Café.
Similar to Designing a Scalable Twitter - Patterns for Designing Scalable Real-Time Web Applications (20)
This presentation provides an introduction to the Cloudify integration plugin with Terraform.
This integration allows Terraform users to use Cloudify to manage configuration and workflow of applications ontop of an infrastructure that was created by Terraform.
What A No Compromises Hybrid Cloud Looks Like Nati Shalom
Expectation vs. reality of a typical enterprise cloud journey
Lesson learned on how to set a cloud native strategy without compromising on the least common denominator, nor going through a complete rewrite
It has long been debated whether OpenStack is production ready. In this session you will learn how a major bank has gone to production with more than 5000 VMs that delivered the results of a 40% decrease in cost, reduced deployment time to hours not weeks, 56 new technologies introduced, 7 new platforms launched - all in under a year. Learn how their platform built on Rackspace and RHEL, coupled with best of breed open source tooling - SaltStack, Jenkins, Cloudify, and Nexus are the enablers for production-grade OpenStack.
http://sched.co/7fH1
Orchestration tool roundup kubernetes vs. docker vs. heat vs. terra form vs...Nati Shalom
Video recording: https://www.youtube.com/watch?v=tGlIgUeoGz8
It’s no news that containers represent a portable unit of deployment, and OpenStack has proven an ideal environment for running container workloads. However, where it usually becomes more complex is that many times an application is often built out of multiple containers. What’s more, setting up a cluster of container images can be fairly cumbersome because you need to make one container aware of another and expose intimate details that are required for them to communicate which is not trivial especially if they’re not on the same host.
These scenarios have instigated the demand for some kind of orchestrator. The list of container orchestrators is growing fairly fast. This session will compare the different orchestation projects out there - from Heat to Kubernetes to TOSCA - and help you choose the right tool for the job.
Session link from teh summit: https://openstacksummitmay2015vancouver.sched.org/event/abd484e0dedcb9774edda1548ad47518#.VV5eh5NViko
Real World Application Orchestration Made Easy on VMware vCloud Air, vSphere ...Nati Shalom
Looking for application orchestration in a hybrid or multi-cloud environment? You’ve got to hear about TOSCA orchestration. TOSCA (Topology and Orchestration Specification for Cloud Applications), brought to you by the same people who brought us XML, enables you to seamlessly migrate your workloads across environments or build a hybrid deployment that runs simultaneously across the VMware cloud offering.
Join our Cloud Online Meetup to learn how Cloudify’s TOSCA-compliant orchestration can be your common management interface across the VMware cloud offering, OpenStack and heterogeneous cloud environments.
Speakers:
Nati Shalom, Founder and CTO at GigaSpaces, is a thought leader in Cloud Computing and Big Data Technologies. Shalom was recently recognized as a Top Cloud Computing Blogger for CIOs by The CIO Magazine and his blog is listed as an excellent blog by YCombinator. Shalom is the founder and also one of leaders of OpenStack Israel group, and is a frequent presenter at industry conferences.
Paco Gomez, Senior Solution Architect at VMware vCloud Air. Paco evaluates and integrates strategic solutions that help vCloud Air clients benefit from VMware's hybrid cloud and application services. Paco is a seasoned technologist, having extensive experience in diverse fields including mainframes, distributed systems, enterprise development, cloud computing, mobile, assistive technology, electrical engineering and embedded systems. Across his career, Paco has held positions in consulting, sales engineering
OpenStack Juno The Complete Lowdown and Tales from the SummitNati Shalom
This presentation covers the main points from the summit and the OpenStack Juno release
It also covers how users use OpenStack based on the recent survey
Application and Network Orchestration using Heat & ToscaNati Shalom
The buzzwords Neutron, Heat, and TOSCA are spoken about quite often when it comes to the OpenStack - and many of us are still trying to make sense of the terminology and its place in the OpenStack world.
Where OpenStack Neutron provides APIs for creating network elements, OpenStack Heat provides an orchestration engine for automating the setup and configuration of OpenStack infrastructure, while TOSCA is a standard for templating and defining application topology and policies (that form the basis for Heat). In this context, it really makes sense to put these all together to achieve application and network automation for OpenStack on steroids.
In this session we will learn how we can use the robust combination of Heat and TOSCA to configure and control resources on Nova and Neutron in order to automate the network configuration as part of the application deployment.
The session will include a demo and code examples that show how you can configure virtual networks, attach public IPs, set up security groups, set up load balancing and automatically scale up/down and more. You will leave this session understanding where Neutron meets Heat and TOSCA.
This talk was delivered as part of OpenStack Paris summit - 2014 - http://openstacksummitnovember2014paris.sched.org/event/2b85b682ccaf3a5961e463b61e2403f8#.VFeuG_TF8mc
During the past few years we’ve seen how our entire data-center becomes software defined. This include the Compute, Storage, Network and also Configuration. This new data centre is the cloud.
The missing piece in the puzzle:
While this is pretty much old news there is one big thing that is missing in this puzzle and that is the operator itself.
The operator is responsible for running processes such:
* Installation of new apps
* Upgrades and update of new features or patches
* Performance tuning
* Handling failure
* Managing the capacity to meet the scaling demand.
Most of those tasks today involves lots of human intervention. Users who realised that gap try to mitigate that by putting their own custom automation - usually that comes in a form of scripts on-top of the configuration management. Those custom scripts tend to grow fairly quickly to the point where they become unmanageable.
This presentation will introduce how we can use an orchestrator to automate those tasks and by that create a software defined Operator.
Complex Analytics with NoSQL Data Store in Real TimeNati Shalom
NOSQL are often limited in the type of queries that they can support due to the distributed nature of the data. In this session we would learn patterns on how we can overcome this limitation and combine multiple query semantics with NoSQL based engines.
We will demonstrate specifically a combination of key/value, SQL like, Document model and Graph based queries as well as more advanced topic such as handling partial update and query through projection. We will also demonstrate how we can create a meshaup between those API's i.e. write fast through Key/Value API and execute complex queries on that same data through SQL query.
- See more at: http://nosql2014.dataversity.net/sessionPop.cfm?confid=81&proposalid=6335#sthash.PNSZi5TJ.dpuf
Is Orchestration the Next Big Thing in DevOpsNati Shalom
DevOps processes (such as continuous deployment and delivery) often involve writing many custom scripts that are triggered by the build system. With that approach, it is relatively hard to trace the deployment process and troubleshoot when something goes wrong. Additionally, custom scripts are often not written in an easily understood manner. In this session we will walk through specific DevOps workflows (such as install, update, etc) using Riemann as the framework in subject and see the steps required to automate those processes. We will also discuss how Cloudify uses Riemann to provide simple execution and monitoring of those workflow processes. We will share how one customer, PaddyPower, was able to leverage Cloudify to transition their traditional IT into a DevOps environment, bridging the gap betwe
When networks meets apps (open stack atlanta)Nati Shalom
Recent advancements in OpenStack capabilities have made the cloud better tuned to enterprise needs by introducing much more flexible network designs and networking services, with the tradeoff of making the cloud more complex.
In this session we will describe how we can leverage the power of the new networking advancement without exposing the complexity to the end user. We will present alternative approaches and their tradeoffs for automating the deployment of a typical n-tier enterprise application that include multi-tenant environment, separate network for admin and applications, cross region network, attach a floating IP, setup security groups etc. all through a combination of Heat, TOSCA, Chef, Puppet, and more.
The experience of automating continuous delivery processes with Chef and Cloudify through an application-centric approach to DevOps, and how this model transformed PaddyPower's traditional IT into DevOps, keeping their Devs and their Ops happy.
References:
---------------
- Cloudify & Chef : http://www.cloudifysource.org/guide/2.7/integrations/chef_documentation
- Blog Post: http://www.cloudifysource.org/2013/10/27/application_centric_approach_to_devops.html
- Earlier Video Presentation : http://www.youtube.com/watch?v=YhDNKyP_s7U
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Designing a Scalable Twitter - Patterns for Designing Scalable Real-Time Web Applications
1. Designing a scalable twitter Nati Shalom, CTO & Founder GigaSpaces John D. Mitchell Mad Scientist of Friendster.
2. About GigaSpaces Technologies Enabling applications to run a distributed cluster as if it was a single machine… 75+ Cloud Customers Among Top 50 Cloud Vendors 300+ Direct Customers 2 2
3. Introduction – Why twitter as an example Everybody knows twitter Twitter exposes really difficult scaling challenges Twitter can serve as a generic example for any real-time/social web application If you can built a scalable twitter your already half way through.
4. Why twitter is different then other web apps? Twitter makes many to many relationship first class citizen Twitter behaves like a living organism Every post gets to number of followers Every user follows number of users Followers Following User User
5. Scaling challenges 1 - Fluctuating growth Every user network (Following/Followers) grows constantly Traffic tends to fluctuate randomly Followers Following User User
6. Scaling challenges 2 – The crowd effect Re-tweet can lead to a perfect storm Every Scaling has a physical limit Followers User
11. Users Approach 1 scalability challenge Application Application Application Application Read Service Read Service Read Service Read Service The database becomes the bottleneck Web Data Base Publish Service Publish Service Web Publish Service Publish Service Load Balancer Web Web IIS Web
12. Reader – Database Implementation public ICollection<Post> GetUserPosts(string userID, DateTime fromDate) { // Create command: IDbCommand dbCommand = _dbConnection.CreateCommand(); dbCommand.CommandText = String.Format( "SELECT * FROM Post WHERE UserID='{0}' AND PostedOn > {1}", userID, fromDate); // Execute command: IDataReader dataReader = dbCommand.ExecuteReader(); // Translate results from db records to .NET objects: List<Post> result = ReadPostsFromDataReader(dataReader); // Return results: return result; }
13. Eliminate database bottlenecks – a questions of alternatives Solving the database bottleneck requires change Difference approaches offers various degree of change Database clusters- read-only databases Requires significant data access design changes Does not improve performance In Memory DataGrid – Change at the application data not the database Memcache – Small change/Small value fairly limited not transactional, doesn’t support SQL query, non clustered Google App Engine Persistency (On top of Big Table) Provides standard JPA access (Smaller change) Many model limitations Does not provide database integration into existing model Application Partitioning (several different databases for the same application) Requires significant data access design changes Need to synchronize between the databases for transactional integrity
17. In-Memory vs Filesystem based approaches In-memory Pros: Complementary – the database can be kept unchanged Designed for real time performance (Not limited to disk I/O) Enable execution of the code with the data (stored procedure) Cons Memory is more expensive then file-system (Cost of per GB storage) File-System Pros: Can manage terra bytes of data Low cost per GB storage Cons Fairly radical change -> requires complete re-write Best of both worlds Memory based storage for real time access Filesystem for long-term data
18. Choosing the right solution for our twitter app Performance analysis: Twitter allows 150 req/hour per user For 1M users that means 40,000 req/sec Capacity The actual data that needs to be accessed in real-time is only the window of time between interactions (Last 10/30 min should be more then sufficient) The rest of the data can be stored in file-system storage (For users who just logged in) Assuming a ratio of 20% writes and the size per post is 128 bytes: Data produced = 40k*20% * 128 = 1Mb/Sec We can keep a buffer of an hour in less then 5G Solution: Use In Memory Data Grid for the real-time access and files-ystem storage for long term, search and other analytics requirements
29. In-Memory Space-Based Architecture (SBA) is a software architecture pattern for achieving linear scalability of stateful, high-performance applications, based on the Yale’s Tuple-Space Model (Source Wikipedia)
32. Reduced Network hops (if in-process)Application Read Service Publish Service Users Load Balancer NoSQL In-Memory Data Grid Web
33. Route calls Based on @userid Reader (Proxy) Users Load Balancer SBA (Step II) – Linear Scalability: Partitioning and Collocation Writer (Proxy) Data Base Space Space Space Space Space Reader Reader Reader Reader Reader Writer Writer Writer Writer Writer
34. Scaling read/write Partition the data based on @user-id Read uses Map/Reduce pattern to read the data from all relevant partitions Space Reader Writer Read all tweets Space Space Space Space Space Reader Reader Reader Reader Reader Post a tweet Web Writer Writer Writer Writer Writer Web space.write(post); Read- Map/Reduce Write
39. GigaSpaces Home Page:http://www.gigaspaces.com/ GigaSpaces XAP Product Overview: http://www.gigaspaces.com/wiki/display/XAP7/Concepts GigaSpaces XAP for the Cloud: http://www.gigaspaces.com/cloud