The Marketplace data team at Uber has built a scalable complex event processing platform to solve many challenging real-time data needs for various Uber products. This platform has been in production for more than a year and supports over 100 real-time data use cases with a team of 3. In this talk, we will share the detail of the design and our experience, and how we employ Siddhi, Kafka and Samza at scale.
Tokyo Azure Meetup #5 - Microservices and Azure Service FabricTokyo Azure Meetup
Azure Service Fabric is now Generally Available!
In this meetup we will start from the beginning and define what is microservice.
Next we will have a deep dive in Azure Service Fabric. Azure Service Fabric is one of the most interesting Azure service. Used internally in Microsoft for 5 years and backing up one of the most demanding Azure services today such as Azure SQL, Document DB, Cortana and Skype for Business.
We will be talking about the two models that are supported by Azure Service Fabric:
- Reliable Services (We will explore the reasons for having both stateful and stateless offerings in this model)
- Reliable Actors
Then we will talk how you can create Azure Service Fabric cluster on premise or in another cloud.
We will demo deployments in Azure for the various models.
Azure Service Fabric is the most advanced and complete offering for developing and hosting microservices in Azure. It builds on years experience Microsoft acquired running one of the most demanding services such as Azure SQL. Moreover, Azure Service Fabric solves very difficult distributed computing problems such as data synchronization, zero downtime deployment, update and rollback operations at large scale.
Join us to learn more about Azure Service Fabric and start using it immediately after the meetup!
A high level overview of how a web application like Magento 2 CE/EE can be split up into a set of manageable MicroServices using Docker Containers. The resultant application will be fast, platform independent, portable, repeatable/scale-able and manageable, resulting in an immensely simplified DevOps.
Tokyo Azure Meetup #5 - Microservices and Azure Service FabricTokyo Azure Meetup
Azure Service Fabric is now Generally Available!
In this meetup we will start from the beginning and define what is microservice.
Next we will have a deep dive in Azure Service Fabric. Azure Service Fabric is one of the most interesting Azure service. Used internally in Microsoft for 5 years and backing up one of the most demanding Azure services today such as Azure SQL, Document DB, Cortana and Skype for Business.
We will be talking about the two models that are supported by Azure Service Fabric:
- Reliable Services (We will explore the reasons for having both stateful and stateless offerings in this model)
- Reliable Actors
Then we will talk how you can create Azure Service Fabric cluster on premise or in another cloud.
We will demo deployments in Azure for the various models.
Azure Service Fabric is the most advanced and complete offering for developing and hosting microservices in Azure. It builds on years experience Microsoft acquired running one of the most demanding services such as Azure SQL. Moreover, Azure Service Fabric solves very difficult distributed computing problems such as data synchronization, zero downtime deployment, update and rollback operations at large scale.
Join us to learn more about Azure Service Fabric and start using it immediately after the meetup!
A high level overview of how a web application like Magento 2 CE/EE can be split up into a set of manageable MicroServices using Docker Containers. The resultant application will be fast, platform independent, portable, repeatable/scale-able and manageable, resulting in an immensely simplified DevOps.
Event Bus as Backbone for Decoupled Microservice Choreography (Oracle Code, A...Lucas Jellema
Microservices are independent, encapsulated entities that produce meaningful results and business functionality in tentative collaboration. Events and pub/sub are great for allowing such decoupled interaction. Using Apache Kafka as robust, distributed, real-time, high volume event bus, this session demonstrates how microservices implemented in Java, Node, Python and SQL collaborate unknowingly. The microservices respond to social (media) events - courtesy of IFTTT - and publish results to multiple channels. The event bus operates across cloud services and on premises platforms: both the bus and the microservices can run anywhere.
JDD 2016 - Jacek Bukowski - "Flying To Clouds" - Can It Be Easy?PROIDEA
Nowadays "cloud" and "microservice" terms are used all the time, even overused. Does any system must be the "microservices" deployed in the "cloud"? Definitely not! However once you see that your system may benefit from that architecture, the next question is how to get there - how to fly to the clouds?
Spring was always about simplifying the complicated aspects of your enterprise system. Netflix went to microservice architecture long before this term even was created. Both are very much contributed to open source software. How can you benefit from joint forces of the both?
Flying to clouds - can it be easy? Cloud Native ApplicationsJacek Bukowski
Nowadays "cloud" and "microservice" terms are used all the time, even overused. Does any system must be the "microservices" deployed in the "cloud"? Definitely not! However once you see that your system may benefit from that architecture, the next question is how to get there - how to fly to the clouds?
Spring was always about simplifying the complicated aspects of your enterprise system. Netflix went to microservice architecture long before this term even was created. Both are very much contributed to open source software. How can you benefit from joint forces of the both?
AWS RDS Oracle - What is missing for a fully managed service?DanielHillinger
With the Relational Database Service (RDS) Amazon Web Services (AWS) offers a managed service for many database products (e.g. Oracle, Postges and MYSQL).
AWS takes over many of the standard DBA tasks and has automated them. But what is missing, so that you really don't have to take care of anything anymore?
Which topics are fully managed and where do you have to actively work on solutions yourself?
In a world where an automatic backup is just a checkmark in a web interface, it is worth taking a closer look.
Using Kafka as a Database For Real-Time Transaction Processing | Chad Preisle...HostedbyConfluent
You have learned about Kafka event sourcing with streams and using Kafka as a database, but you may be having a tough time wrapping your head around what that means and what challenges you will face. Kafka’s exactly once semantics, data retention rules, and stream DSL make it a great database for real-time transaction processing. This talk will focus on how to use Kafka events as a database. We will talk about using KTables vs GlobalKTables, and how to apply them to patterns we use with traditional databases. We will go over a real-world example of joining events against existing data and some issues to be aware of. We will finish covering some important things to remember about state stores, partitions, and streams to help you avoid problems when your data sets become large.
MVC 6 - the new unified Web programming modelAlex Thissen
Presentation for Dutch Microsoft TechDays 2015:
With ASP.NET 5 comes MVC 6 with a programming model that unifies Web Pages, MVC and Web API. Each of these has been rebuilt to reflect Microsoft's vision of lean and composable web applications. In this session you will see the changes that have been made to the programming model. We will cover topics such as the new POCO controllers, View Components, dependency injection and much more. Plus, you are going to see the significant changes to the ASP.NET runtime on which MVC 6 is built.
Lessons from the field: Catalog of Kafka Deployments | Joseph Niemiec, ClouderaHostedbyConfluent
Streaming architectures have been on the rise steadily and as a result, we have seen the adoption of Kafka go up too. With the diverse spread of use cases across multiple industries, we have seen a variety of Kafka deployments across our hundreds of Kafka customers. Along the way, we have learnt some best practices as well as what not to do in mission-critical architectures. Join Joe Niemiec, Sr. Product Manager at Cloudera, as he shares these insights in this session that covers topics such as - The many ways that Kafka has been deployed in the field Standalone clusters, multiple clusters in a single data center and multiple clusters geographically distributed performing replication Clusters of all sizes small and large, few messages to hundreds of thousands per second Discussion about architecture failure domains Configurations tuned and used in specific deployments
Hello All,
Let's meet and discuss what are the new announcements from Build 2016 and how we can best leverage them in our business!
Here are some of the topics we will cover this time:
- Azure Functions
- Service Fabric
- Azure Storage
- Document DB
- Azure Container Services
- Power BI Embedded
- ASP.NET Core
- Virtual Machine Scale Sets
I will be happy to share my experience from the conference, especially the session I visited and also the conversations I had with various Microsoft representatives.
Azure is developing faster than ever and Microsoft is driving the platform in very interesting direction that require us to know and work with more and more new technologies!
Come and join us to learn more about Azure!
I am arranging the venue but my plan for the meetup is to be on April 25-th or April 27-th from 19:30. I will keep you updated on that!
Thank you!
Kanio
Scalable complex event processing on samza @UBERShuyi Chen
The Marketplace data team at Uber has built a scalable complex event processing platform to solve many challenging real time data needs for various Uber products. This platform has been in production for almost a year and it has proven to be very flexible to solve many use cases. In this talk, we will share in detail the design and architecture of the platform, and how we employ Samza, Kafka, and Siddhi at scale.
This slides was presented at Stream Processing Meetup @ LinkedIn on June 15 2016.
Event Bus as Backbone for Decoupled Microservice Choreography (Oracle Code, A...Lucas Jellema
Microservices are independent, encapsulated entities that produce meaningful results and business functionality in tentative collaboration. Events and pub/sub are great for allowing such decoupled interaction. Using Apache Kafka as robust, distributed, real-time, high volume event bus, this session demonstrates how microservices implemented in Java, Node, Python and SQL collaborate unknowingly. The microservices respond to social (media) events - courtesy of IFTTT - and publish results to multiple channels. The event bus operates across cloud services and on premises platforms: both the bus and the microservices can run anywhere.
JDD 2016 - Jacek Bukowski - "Flying To Clouds" - Can It Be Easy?PROIDEA
Nowadays "cloud" and "microservice" terms are used all the time, even overused. Does any system must be the "microservices" deployed in the "cloud"? Definitely not! However once you see that your system may benefit from that architecture, the next question is how to get there - how to fly to the clouds?
Spring was always about simplifying the complicated aspects of your enterprise system. Netflix went to microservice architecture long before this term even was created. Both are very much contributed to open source software. How can you benefit from joint forces of the both?
Flying to clouds - can it be easy? Cloud Native ApplicationsJacek Bukowski
Nowadays "cloud" and "microservice" terms are used all the time, even overused. Does any system must be the "microservices" deployed in the "cloud"? Definitely not! However once you see that your system may benefit from that architecture, the next question is how to get there - how to fly to the clouds?
Spring was always about simplifying the complicated aspects of your enterprise system. Netflix went to microservice architecture long before this term even was created. Both are very much contributed to open source software. How can you benefit from joint forces of the both?
AWS RDS Oracle - What is missing for a fully managed service?DanielHillinger
With the Relational Database Service (RDS) Amazon Web Services (AWS) offers a managed service for many database products (e.g. Oracle, Postges and MYSQL).
AWS takes over many of the standard DBA tasks and has automated them. But what is missing, so that you really don't have to take care of anything anymore?
Which topics are fully managed and where do you have to actively work on solutions yourself?
In a world where an automatic backup is just a checkmark in a web interface, it is worth taking a closer look.
Using Kafka as a Database For Real-Time Transaction Processing | Chad Preisle...HostedbyConfluent
You have learned about Kafka event sourcing with streams and using Kafka as a database, but you may be having a tough time wrapping your head around what that means and what challenges you will face. Kafka’s exactly once semantics, data retention rules, and stream DSL make it a great database for real-time transaction processing. This talk will focus on how to use Kafka events as a database. We will talk about using KTables vs GlobalKTables, and how to apply them to patterns we use with traditional databases. We will go over a real-world example of joining events against existing data and some issues to be aware of. We will finish covering some important things to remember about state stores, partitions, and streams to help you avoid problems when your data sets become large.
MVC 6 - the new unified Web programming modelAlex Thissen
Presentation for Dutch Microsoft TechDays 2015:
With ASP.NET 5 comes MVC 6 with a programming model that unifies Web Pages, MVC and Web API. Each of these has been rebuilt to reflect Microsoft's vision of lean and composable web applications. In this session you will see the changes that have been made to the programming model. We will cover topics such as the new POCO controllers, View Components, dependency injection and much more. Plus, you are going to see the significant changes to the ASP.NET runtime on which MVC 6 is built.
Lessons from the field: Catalog of Kafka Deployments | Joseph Niemiec, ClouderaHostedbyConfluent
Streaming architectures have been on the rise steadily and as a result, we have seen the adoption of Kafka go up too. With the diverse spread of use cases across multiple industries, we have seen a variety of Kafka deployments across our hundreds of Kafka customers. Along the way, we have learnt some best practices as well as what not to do in mission-critical architectures. Join Joe Niemiec, Sr. Product Manager at Cloudera, as he shares these insights in this session that covers topics such as - The many ways that Kafka has been deployed in the field Standalone clusters, multiple clusters in a single data center and multiple clusters geographically distributed performing replication Clusters of all sizes small and large, few messages to hundreds of thousands per second Discussion about architecture failure domains Configurations tuned and used in specific deployments
Hello All,
Let's meet and discuss what are the new announcements from Build 2016 and how we can best leverage them in our business!
Here are some of the topics we will cover this time:
- Azure Functions
- Service Fabric
- Azure Storage
- Document DB
- Azure Container Services
- Power BI Embedded
- ASP.NET Core
- Virtual Machine Scale Sets
I will be happy to share my experience from the conference, especially the session I visited and also the conversations I had with various Microsoft representatives.
Azure is developing faster than ever and Microsoft is driving the platform in very interesting direction that require us to know and work with more and more new technologies!
Come and join us to learn more about Azure!
I am arranging the venue but my plan for the meetup is to be on April 25-th or April 27-th from 19:30. I will keep you updated on that!
Thank you!
Kanio
Scalable complex event processing on samza @UBERShuyi Chen
The Marketplace data team at Uber has built a scalable complex event processing platform to solve many challenging real time data needs for various Uber products. This platform has been in production for almost a year and it has proven to be very flexible to solve many use cases. In this talk, we will share in detail the design and architecture of the platform, and how we employ Samza, Kafka, and Siddhi at scale.
This slides was presented at Stream Processing Meetup @ LinkedIn on June 15 2016.
Extending Spark Streaming to Support Complex Event ProcessingOh Chan Kwon
In this talk, we introduce the extensions of Spark Streaming to support (1) SQL-based query processing and (2) elastic-seamless resource allocation. First, we explain the methods of supporting window queries and query chains. As we know, last year, Grace Huang and Jerry Shao introduced the concept of “StreamSQL” that can process streaming data with SQL-like queries by adapting SparkSQL to Spark Streaming. However, we made advances in supporting complex event processing (CEP) based on their efforts. In detail, we implemented the sliding window concept to support a time-based streaming data processing at the SQL level. Here, to reduce the aggregation time of large windows, we generate an efficient query plan that computes the partial results by evaluating only the data entering or leaving the window and then gets the current result by merging the previous one and the partial ones. Next, to support query chains, we made the result of a query over streaming data be a table by adding the “insert into” query. That is, it allows us to apply stream queries to the results of other ones. Second, we explain the methods of allocating resources to streaming applications dynamically, which enable the applications to meet a given deadline. As the rate of incoming events varies over time, resources allocated to applications need to be adjusted for high resource utilization. However, the current Spark's resource allocation features are not suitable for streaming applications. That is, the resources allocated will not be freed when new data are arriving continuously to the streaming applications even though the quantity of the new ones is very small. In order to resolve the problem, we consider their resource utilization. If the utilization is low, we choose victim nodes to be killed. Then, we do not feed new data into the victims to prevent a useless recovery issuing when they are killed. Accordingly, we can scale-in/-out the resources seamlessly.
Guide to Application Performance: Planning to Continued OptimizationMuleSoft
Supporting everything from mobile apps with thousands of concurrent users to global deployments processing millions of requests daily, Anypoint Platform has been put to test. In this session, MuleSoft experts will talk through case studies from our most demanding deployments and provide a best practice approach to designing and tuning applications for optimal performance.
Azure Event Hubs - Behind the Scenes With Kasun Indrasiri | Current 2022HostedbyConfluent
Azure Event Hubs - Behind the Scenes With Kasun Indrasiri | Current 2022
Azure Event Hubs is a hyperscale PaaS event stream broker with protocol support for HTTP, AMQP, and Apache Kafka RPC that accepts and forwards several trillion (!) events per day and is available in all global Azure regions. This session is a look behind the curtain where we dive deep into the architecture of Event Hubs and look at the Event Hubs cluster model, resource isolation, and storage strategies and also review some performance figures.
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with K...confluent
Microservices, events, containers, and orchestrators are dominating our vernacular today. As operations teams adapt to support these technologies in production, cloud-native platforms like Cloud Foundry and Kubernetes have quickly risen to serve as force multipliers of automation, productivity and value. Kafka is providing developers a critically important component as they build and modernize applications to cloud-native architecture. This talk will explore:
• Why cloud-native platforms and why run Kafka on Kubernetes?
• What kind of workloads are best suited for this combination?
• Tips to determine the path forward for legacy monoliths in your application portfolio
• Running Kafka as a Streaming Platform on Container Orchestration
This talk focuses on how we used Amazon Kinesis to build the pub-sub infra at Lyft, that ingests more than a 100 billion events per day. We'll review the strengths and weaknesses of Kinesis as a choice for streaming events in realtime, at Lyft's scale; as well as the best practices and lessons learnt over time.
Speaker: Hafiz Hamid (Lyft)
Hafiz Hamid is a software engineer on the Pub-Sub/Streaming Platform team at Lyft. He has built some of the key pieces in the messaging & streaming infrastructure at Lyft. Previously, Hafiz was a technical lead at Bing Search where he worked on data pipelines, relevance and web crawlers.
Low latency high throughput streaming using Apache Apex and Apache KuduDataWorks Summit
True streaming is fast becoming a necessity for many business use cases. On the other hand the data set sizes and volumes are also growing exponentially compounding the complexity of data processing pipelines.There exists a need for true low latency streaming coupled with very high throughput data processing. Apache Apex as a low latency and high throughput data processing framework and Apache Kudu as a high throughput store form a nice combination which solves this pattern very efficiently.
This session will walk through a use case which involves writing a high throughput stream using Apache Kafka,Apache Apex and Apache Kudu. The session will start with a general overview of Apache Apex and capabilities of Apex that form the foundation for a low latency and high throughput engine with Apache kafka being an example input source of streams. Subsequently we walk through Kudu integration with Apex by walking through various patterns like end to end exactly once, selective column writes and timestamp propagations for out of band data. The session will also cover additional patterns that this integration will cover for enterprise level data processing pipelines.
The session will conclude with some metrics for latency and throughput numbers for the use case that is presented.
Speaker
Ananth Gundabattula, Senior Architect, Commonwealth Bank of Australia
Software Architecture for Cloud InfrastructureTapio Rautonen
Distributed systems are hard to build. Software architecture must be carefully crafted to suit cloud infrastructure.
Design for failure. Learn from failure. Adopt new cloud compatible design patterns and follow the guidelines during the journey of building cloud native applications.
More and more, the scalable on-demand infrastructure provided by AWS is being used by researchers, scientists and engineers in Life Sciences, Finance and Engineering to solve bigger problems, answer complex questions and run larger simulations. In this session we start by talking about the supercomputing class performance and high performance storage available to the scientists and engineers at their fingertips. We will go over examples of how startups are innovating and large enterprises are extending their HPC environments. Finally, we walk through some of the common questions that come up as organizations start leveraging AWS for their high performance computing needs.
Hhm 3474 mq messaging technologies and support for high availability and acti...Pete Siddall
Active-Active is the target messaging model for the modern data center. But its successful adoption must encompass not only the mainframe, but also heterogeneous and peripherally distributed platforms, which makes it much more complex to implement. Data synchronization is at the heart of the various Active-Active technologies, and the right messaging technology must therefore be chosen for its implementation. This session gives an overview of Active-Active technologies on both z Systems and distributed platforms. It highlights how Active-Active provides the benefits of both high availability and workload balancing. We will also discuss customer cases on how to implement messaging-based Active-Active.
Realtime streaming architecture in INFINARIOJozo Kovac
About our experience with realtime analyses on never-ending stream of user events. Discuss Lambda architecture, Kappa, Apache Kafka and our own approach.
Mtc learnings from isv & enterprise interactionGovind Kanshi
This is one of the dated presentation for which I keep getting requests for, please do reach out to me for status on various things as Azure keeps fixing/innovating whole of things every day.
There are bunch of other things I can help you on to ensure you can take advantage of Azure platform for oss, .net frameworks and databases.
Mtc learnings from isv & enterprise (dated - Dec -2014)Govind Kanshi
This is little dated deck for our learnings - I keep getting multiple requests for it. I have removed one slide for access permissions (RBAC -which are now available).
Similar to WSO2Con USA 2017: Scalable Real-time Complex Event Processing at Uber (20)
Accelerate Enterprise Software Engineering with PlatformlessWSO2
Key takeaways:
Challenges of building platforms and the benefits of platformless.
Key principles of platformless, including API-first, cloud-native middleware, platform engineering, and developer experience.
How Choreo enables the platformless experience.
How key concepts like application architecture, domain-driven design, zero trust, and cell-based architecture are inherently a part of Choreo.
Demo of an end-to-end app built and deployed on Choreo.
Less Is More: Utilizing Ballerina to Architect a Cloud Data PlatformWSO2
At its core, the challenge of managing Human Resources data is an integration challenge: estimates range from 2-3 HR systems in use at a typical SMB, up to a few dozen systems implemented amongst enterprise HR departments, and these systems seldom integrate seamlessly between themselves. Providing a multi-tenant, cloud-native solution to integrate these hundreds of HR-related systems, normalize their disparate data models and then render that consolidated information for stakeholder decision making has been a substantial undertaking, but one significantly eased by leveraging Ballerina. In this session, we’ll cover:
The overall software architecture for VHR’s Cloud Data Platform
Critical decision points leading to adoption of Ballerina for the CDP
Ballerina’s role in multiple evolutionary steps to the current architecture
Roadmap for the CDP architecture and plans for Ballerina
WSO2’s partnership in bringing continual success for the CD
The integration landscape is changing rapidly with the introduction of technologies like GraphQL, gRPC, stream processing, iPaaS, and platformless. However, not all existing applications and industries can keep up with these new technologies. Certain industries, like manufacturing, logistics, and finance, still rely on well-established EDI-based message formats. Some applications use XML or CSV with file-based communications, while others have strict on premises deployment requirements. This talk focuses on how Ballerina's built-in integration capabilities can bridge the gap between "old" and "new" technologies, modernizing enterprise applications without disrupting business operations.
Platformless Horizons for Digital AdaptabilityWSO2
In this keynote, Asanka Abeysinghe, CTO,WSO2 will explore the shift towards platformless technology ecosystems and their importance in driving digital adaptability and innovation. We will discuss strategies for leveraging decentralized architectures and integrating diverse technologies, with a focus on building resilient, flexible, and future-ready IT infrastructures. We will also highlight WSO2's roadmap, emphasizing our commitment to supporting this transformative journey with our evolving product suite.
Quantum computers are rapidly evolving and are promising significant advantages in domains like machine learning or optimization, to name but a few areas. In this keynote we sketch the underpinnings of quantum computing, show some of the inherent advantages, highlight some application areas, and show how quantum applications are built.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
14. Can we use declarative semantics to specify these stream
processing logics?
15. Complex event processing
• Combines data from multiple sources to infer events or patterns that
suggest more complicated circumstances
• CEP is used across many industries for various use cases, including:
– Finance: Trade analysis, fraud detection
– Airlines: Operations monitoring
– Healthcare: Claims processing, patient monitoring
– Energy and Telecommunications: Outage detection
• CEP uses declarative rule/query language to specify event processing
logic
16. WSO2/Siddhi: Complex event processing engine
• Lightweight, extensible, open source, released as a Java library
• Features supported
– Filter
– Join
– Aggregation
– Group by
– Window
– Pattern processing
– Sequence processing
– Event tables
– Event-time processing
– UDF
– Extensions
– Declarative query language: SiddhiQL
18. How Siddhi works
• Query is parsed at runtime into an execution plan runtime
• As events flow in, the execution plan runtime process events inside
the CEP engine according the query logic
20. Apache Samza
• A distributed stream processing framework
– Distributed and Scalable
– Built-in State management
– Built-in fault tolerant
– At-least-once message processing
21. How can we make the stream processing output useful?
22. Actions
• Generalize a set of common action templates to make it easy for
services and human to harness the power of realtime stream
processing
• Currently we support
– Make an RPC call
– Invoke a Webhook endpoint
– Index to ElasticSearch
– Index to Cassandra
– Kafka
– Statsd
– Chat service
– Email
– Push notification
27. Partitioner
• Re-partition events based on key
• Support predicate pushdown through query analysis
• Support column pruning through query analysis (WIP)
28. Query processor
• Parse Siddhi queries into execution plan runtime
• Process events in Siddhi execution plan runtime
• Checkpoint state regularly to ensure recovery upon crash/restart
using RocksDB
29. Action processor
• Execute actions upon the complex event processing output
• Support various kinds of actions for easy integration
• Implement action retry mechanism using RocksDB to provide
at-least-once delivery
30. How do we translate a query into psychical plan that
runs?
31. DAG (Directed Acyclic Graph) generation
• Analyze Siddhi query to automatically generate the stream
processing DAG in Samza using the processors
Filter, transformation
35. REST API backend
• All queries, actions are stored externally in database.
• RESTFUL API for CRUD operations
• If query/action logic changed
– Redeploy the Samza DAG if needed
– Otherwise, the updated queries/actions will be loaded at runtime w/o
interruption
36. Unified management and monitoring
• Every use case
– share the same set of processors
– Use queries and actions to describe its processing logic
• A single monitoring template can be reused across different use
cases
40. Out-of-order event handling
• Not a big concern
– Events of the same rider/partner are usually seconds aparts
• K-slack extension in Siddhi for out-of-order event processing
41. Auto-scaling
• Manually re-partition kafka topics to increase parallelism
• Manually tune container memory if needed
• Future
– Use CPU/memory/IO stats to auto-scale the data pipelines
43. Large checkpointing state
• Samza use Kafka to log state changes
• Siddhi engine snapshot can be large
• Kafka message size limit to 1MB by default
• Solution: we build logics to slice state into smaller pieces and
checkpoint them.
44. Synchronous checkpointing
• If state is large, time to checkpoint can be long
• Samza uses single-threaded model, unsafe to do it asynchronously
(SAMZA-863)
45. Exactly once state processing?
• Can not commit state and offset atomically
• No exactly once state processing
46. Custom business logic
• Common logic implemented as Siddhi extensions
• Ad-hoc logic implemented as UDF in javascript or scala
47. Intermediate Kafka messages
• Samza uses Kafka as message queue for intermediate processing
output
– This can create large load on Kafka if a heave topic is partitioned multiple
times
– Encode the intermediate messages to reduce footprint
48. Multi-tenancy
• Older Siddhi version process events using a thread pool
– Bad for multi-tenancy in YARN
– Consume more CPU resource than claimed
• Newer version still use thread pool for scheduled task, but main
processing in single thread
– Good: CPU consumption per YARN container is bounded
49. Upgrading Samza jobs
• Upgrade Samza jobs require a full restart, and can take minutes due
to
– Offset checkpointing topic too large → set retention to hours
– Changelog topic too large → set retention or enable compaction in
Kafka or host affinity (SAMZA-617)
• To minimize the interruption during upgrade, it would be nice to
have
– Rolling restart
– Per container restart
50. Our solution: non-interrupted handoff
• For critical jobs, we use replication during upgrade
– Start a shadow job
– Upgrade shadow
– Switch primary and shadow
– Upgrade primary
– Switch back
• Downside: require 2x capacity during upgrade