Using schedulers like Marathon and Aurora help to get your applications scheduled and executing on Mesos. In many cases it makes sense to build a framework and integrate directly. This talk will breakdown what is involved in building a framework, how to-do this with examples and why you would want to-do this. Frameworks are not only for generally available software applications (like Kafka, HDFS, Spark ,etc) but can also be used for custom internal R&D built software applications too.
Get started with Developing Frameworks in Go on Apache MesosJoe Stein
Apache Mesos provides a platform for building distributed systems. Mesos is built using the same principles as the Linux kernel, only at a different level of abstraction. The Mesos kernel runs on every machine and provides applications (e.g., Hadoop, Spark, Kafka, Elastic Search) with API’s for resource management and scheduling across entire datacenter and cloud environments. How to use that platform and what to make of it becomes a complex task requiring not only understanding of where the system has been but also where it is going. Using schedulers like Marathon and Aurora help to get your applications scheduled and executing on Mesos. In many cases it makes sense to build a framework and integrate directly. This talk will breakdown what is involved in building a framework in Go, how to-do this with examples and why you would want to-do this. Frameworks are not only for generally available software applications (like Kafka, HDFS, Spark ,etc) but should also be used for custom internal R&D built software applications too.
Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively.
Using schedulers like Marathon and Aurora help to get your applications scheduled and executing on Mesos. In many cases it makes sense to build a framework and integrate directly. This talk will breakdown what is involved in building a framework, how to-do this with examples and why you would want to-do this. Frameworks are not only for generally available software applications (like Kafka, HDFS, Spark ,etc) but can also be used for custom internal R&D built software applications too.
Get started with Developing Frameworks in Go on Apache MesosJoe Stein
Apache Mesos provides a platform for building distributed systems. Mesos is built using the same principles as the Linux kernel, only at a different level of abstraction. The Mesos kernel runs on every machine and provides applications (e.g., Hadoop, Spark, Kafka, Elastic Search) with API’s for resource management and scheduling across entire datacenter and cloud environments. How to use that platform and what to make of it becomes a complex task requiring not only understanding of where the system has been but also where it is going. Using schedulers like Marathon and Aurora help to get your applications scheduled and executing on Mesos. In many cases it makes sense to build a framework and integrate directly. This talk will breakdown what is involved in building a framework in Go, how to-do this with examples and why you would want to-do this. Frameworks are not only for generally available software applications (like Kafka, HDFS, Spark ,etc) but should also be used for custom internal R&D built software applications too.
Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively.
Real-Time Distributed and Reactive Systems with Apache Kafka and Apache AccumuloJoe Stein
In this talk we will walk through how Apache Kafka and Apache Accumulo can be used together to orchestrate a de-coupled, real-time distributed and reactive request/response system at massive scale. Multiple data pipelines can perform complex operations for each message in parallel at high volumes with low latencies. The final result will be inline with the initiating call. The architecture gains are immense. They allow for the requesting system to receive a response without the need for direct integration with the data pipeline(s) that messages must go through. By utilizing Apache Kafka and Apache Accumulo, these gains sustain at scale and allow for complex operations of different messages to be applied to each response in real-time.
Introduction of mesos persistent storageZhou Weitao
1. How to run stateful service against current Mesos-0.22
2. Disk isolation and monitoring
3. Persistent Volumes
4. Dynamic Reservations
5. What we can contribute for Mesos persistent storage
Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...DataStax
Traditionally, machines were statically partitioned across the different services at Uber. In an effort to increase the machine utilization, Uber has recently started transitioning most of its services, including the storage services, to run on top of Mesos. This presentation will describe the initial experience building and operating a framework for running Cassandra on top of Mesos running across multiple datacenters at Uber. This framework automates several Cassandra operations such as node repairs, addition of new nodes and backup/restore. It improves efficiency by co-locating CPU-intensive services as well as multiple Cassandra nodes on the same Mesos agent. It handles failure and restart of Mesos agents by using persistent volumes and dynamic reservations. This talk includes statistics about the number of Cassandra clusters in production, time taken to start a new cluster, add a new node, detect a node failure; and the observed Cassandra query throughput and latency.
About the Speaker
Abhishek Verma Software Engineer, Uber
Dr. Abhishek Verma is currently working on running Cassandra on top of Mesos at Uber. Prior to this, he worked on BorgMaster at Google and was the first author of the Borg paper published in Eurosys 2015. He received an MS in 2010 and a PhD in 2012 in Computer Science from the University of Illinois at Urbana-Champaign, during which he authored more than 20 publications in conferences, journals and books and presented tens of talks.
Presented at MesosCon EU 2015.
Linux containers, popularized by Docker, have been a game-changer in data center computing in recent years. Mesos has supported container isolation since its early days and has been supporting Docker since 0.20. This talk gives an overview of the evolution of Mesos containerization and an introduction to an upcoming Mesos feature that provisions container (filesystem) images such as Appc and Docker and does filesystem isolation all natively through one unified containerizer without requiring any additional container image runtime. Lastly, it includes a case study for introducing container images to large running clusters both in terms of the number of hosts and the size of their host images and what we have learned along the way.
Adobe has packaged HBase in Docker containers and uses Marathon and Mesos to schedule them—allowing them to decouple the HBase RegionServer from the host, express resource requirements declaratively, and open the door for unassisted real-time deployments, elastic (up and down) real-time scalability, and more.
Making Distributed Data Persistent Services Elastic (Without Losing All Your ...C4Media
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1L2FXLC.
Joe Stein introduces Mesos and managing data services on it, presenting use cases for replacing classic solutions (like cold storage) with new functionality based on these technology. Filmed at qconnewyork.com.
Joe Stein is the CEO of Elodina, a startup focusing on the support & maintenance of third party open source software (like Mesos frameworks) as well as its own open source products & SaaS solutions. He is also the Founder and Principal Consultant of Big Data Open Source Security.
This presentation introduces people to Cassandra and Column Family Datastores in general. I will discuss what Cassandra is, how and when it is useful, and how it integrates with Rails. I will also go in to lessons learned during our 3-month project, and the useful patterns that emerged. The discussion will be very technical, but targeted at developers who are not familiar with, or have not done a project with Cassandra.
What is Apache Mesos and how to use it. A short introduction to distributed fault-tolerant systems with using ZooKeeper and Mesos. #installfest Prague 2014
As a company starts dealing with large amounts of data, operation engineers are challenged with managing the influx of information while ensuring the resilience of data. Hadoop HDFS, Mesos and Spark help reduce issues with a scheduler that allows data cluster resources to be shared. It provides a common ground where data scientists and engineers can meet, develop high performance data processing applications and deploy their own tools.
Evolution of MongoDB Replicaset and Its Best PracticesMydbops
There are several exciting and long-awaited features released from MongoDB 4.0. He will focus on the prime features, the kind of problem it solves, and the best practices for deploying replica sets.
Apache Mesos is the first open source cluster manager that handles the workload efficiently in a distributed environment through dynamic resource sharing and isolation.
Big Data Open Source Security LLC: Realtime log analysis with Mesos, Docker, ...DataStax Academy
We will be talking about the solution we developed for using Mesos, Docker, Kafka, Spark, Cassandra and Solr (DataStax Enterprise Edition) all developed in Go for doing realtime log analysis at scale. Many organizations either need or want log analysis in real time where you can see within a second what is happening within your entire infrastructure. Today, with the hardware available and software systems we have in place, you can develop, build and use as a service these solutions.
Real-Time Distributed and Reactive Systems with Apache Kafka and Apache AccumuloJoe Stein
In this talk we will walk through how Apache Kafka and Apache Accumulo can be used together to orchestrate a de-coupled, real-time distributed and reactive request/response system at massive scale. Multiple data pipelines can perform complex operations for each message in parallel at high volumes with low latencies. The final result will be inline with the initiating call. The architecture gains are immense. They allow for the requesting system to receive a response without the need for direct integration with the data pipeline(s) that messages must go through. By utilizing Apache Kafka and Apache Accumulo, these gains sustain at scale and allow for complex operations of different messages to be applied to each response in real-time.
Introduction of mesos persistent storageZhou Weitao
1. How to run stateful service against current Mesos-0.22
2. Disk isolation and monitoring
3. Persistent Volumes
4. Dynamic Reservations
5. What we can contribute for Mesos persistent storage
Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...DataStax
Traditionally, machines were statically partitioned across the different services at Uber. In an effort to increase the machine utilization, Uber has recently started transitioning most of its services, including the storage services, to run on top of Mesos. This presentation will describe the initial experience building and operating a framework for running Cassandra on top of Mesos running across multiple datacenters at Uber. This framework automates several Cassandra operations such as node repairs, addition of new nodes and backup/restore. It improves efficiency by co-locating CPU-intensive services as well as multiple Cassandra nodes on the same Mesos agent. It handles failure and restart of Mesos agents by using persistent volumes and dynamic reservations. This talk includes statistics about the number of Cassandra clusters in production, time taken to start a new cluster, add a new node, detect a node failure; and the observed Cassandra query throughput and latency.
About the Speaker
Abhishek Verma Software Engineer, Uber
Dr. Abhishek Verma is currently working on running Cassandra on top of Mesos at Uber. Prior to this, he worked on BorgMaster at Google and was the first author of the Borg paper published in Eurosys 2015. He received an MS in 2010 and a PhD in 2012 in Computer Science from the University of Illinois at Urbana-Champaign, during which he authored more than 20 publications in conferences, journals and books and presented tens of talks.
Presented at MesosCon EU 2015.
Linux containers, popularized by Docker, have been a game-changer in data center computing in recent years. Mesos has supported container isolation since its early days and has been supporting Docker since 0.20. This talk gives an overview of the evolution of Mesos containerization and an introduction to an upcoming Mesos feature that provisions container (filesystem) images such as Appc and Docker and does filesystem isolation all natively through one unified containerizer without requiring any additional container image runtime. Lastly, it includes a case study for introducing container images to large running clusters both in terms of the number of hosts and the size of their host images and what we have learned along the way.
Adobe has packaged HBase in Docker containers and uses Marathon and Mesos to schedule them—allowing them to decouple the HBase RegionServer from the host, express resource requirements declaratively, and open the door for unassisted real-time deployments, elastic (up and down) real-time scalability, and more.
Making Distributed Data Persistent Services Elastic (Without Losing All Your ...C4Media
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1L2FXLC.
Joe Stein introduces Mesos and managing data services on it, presenting use cases for replacing classic solutions (like cold storage) with new functionality based on these technology. Filmed at qconnewyork.com.
Joe Stein is the CEO of Elodina, a startup focusing on the support & maintenance of third party open source software (like Mesos frameworks) as well as its own open source products & SaaS solutions. He is also the Founder and Principal Consultant of Big Data Open Source Security.
This presentation introduces people to Cassandra and Column Family Datastores in general. I will discuss what Cassandra is, how and when it is useful, and how it integrates with Rails. I will also go in to lessons learned during our 3-month project, and the useful patterns that emerged. The discussion will be very technical, but targeted at developers who are not familiar with, or have not done a project with Cassandra.
What is Apache Mesos and how to use it. A short introduction to distributed fault-tolerant systems with using ZooKeeper and Mesos. #installfest Prague 2014
As a company starts dealing with large amounts of data, operation engineers are challenged with managing the influx of information while ensuring the resilience of data. Hadoop HDFS, Mesos and Spark help reduce issues with a scheduler that allows data cluster resources to be shared. It provides a common ground where data scientists and engineers can meet, develop high performance data processing applications and deploy their own tools.
Evolution of MongoDB Replicaset and Its Best PracticesMydbops
There are several exciting and long-awaited features released from MongoDB 4.0. He will focus on the prime features, the kind of problem it solves, and the best practices for deploying replica sets.
Apache Mesos is the first open source cluster manager that handles the workload efficiently in a distributed environment through dynamic resource sharing and isolation.
Big Data Open Source Security LLC: Realtime log analysis with Mesos, Docker, ...DataStax Academy
We will be talking about the solution we developed for using Mesos, Docker, Kafka, Spark, Cassandra and Solr (DataStax Enterprise Edition) all developed in Go for doing realtime log analysis at scale. Many organizations either need or want log analysis in real time where you can see within a second what is happening within your entire infrastructure. Today, with the hardware available and software systems we have in place, you can develop, build and use as a service these solutions.
Data Pipeline with Kafka, This slide include
Kafka Introduction, Topic / Partitions, Produce / Consumer, Quick Start, Offset Monitoring, Example Code, Camus
Developing Realtime Data Pipelines With Apache KafkaJoe Stein
Developing Realtime Data Pipelines With Apache Kafka. Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients. Kafka is designed to allow a single cluster to serve as the central data backbone for a large organization. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of co-ordinated consumers. Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages without performance impact. Kafka has a modern cluster-centric design that offers strong durability and fault-tolerance guarantees.
Detail behind the Apache Cassandra 2.0 release and what is new in it including Lightweight Transactions (compare and swap) Eager retries, Improved compaction, Triggers (experimental) and more!
• CQL cursors
Developing Real-Time Data Pipelines with Apache KafkaJoe Stein
Developing Real-Time Data Pipelines with Apache Kafka http://kafka.apache.org/ is an introduction for developers about why and how to use Apache Kafka. Apache Kafka is a publish-subscribe messaging system rethought of as a distributed commit log. Kafka is designed to allow a single cluster to serve as the central data backbone. A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of coordinated consumers. Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages. For the Spring user, Spring Integration Kafka and Spring XD provide integration with Apache Kafka.
SMACK Stack 1.0 has been Spark, Mesos, Akka, Cassandra and Kafka working into different cohesive systems delivering different solutions for different use cases. Haven't heard about it before? Oh man! Where have you been? https://www.google.com/search?q=smack+stack+1.0
SMACK Stack 1.1 we go a step further Streaming, Mesos, Analytics, Cassandra and Kafka and Joe Stein will walk through in detail some of the different viable options for Streaming and Analytics with Mesos, Kafka and Cassandra.
Microxchg Analyzing Response Time Distributions for MicroservicesAdrian Cockcroft
Research oriented presentation @Microxchg Berlin Feb 5th 2016. New code to collect histograms of response time and export them to monte-carlo simulation spreadsheet via getguesstimate.com
Real-Time Log Analysis with Apache Mesos, Kafka and CassandraJoe Stein
Slides for our solution we developed for using Mesos, Docker, Kafka, Spark, Cassandra and Solr (DataStax Enterprise Edition) all developed in Go for doing realtime log analysis at scale. Many organizations either need or want log analysis in real time where you can see within a second what is happening within your entire infrastructure. Today, with the hardware available and software systems we have in place, you can develop, build and use as a service these solutions.
A straight-forward explanation with an example of how JSR-88 aka Deployment Plans can be used in WebLogic Server to make changes to values in deployment descriptors without modifying application archives.
Real-time streaming and data pipelines with Apache KafkaJoe Stein
Get up and running quickly with Apache Kafka http://kafka.apache.org/
* Fast * A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients.
* Scalable * Kafka is designed to allow a single cluster to serve as the central data backbone for a large organization. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of co-ordinated consumers
* Durable * Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages without performance impact.
* Distributed by Design * Kafka has a modern cluster-centric design that offers strong durability and fault-tolerance guarantees.
Introducing Kafka Streams, the new stream processing library of Apache Kafka,...Michael Noll
Video recording: https://www.youtube.com/watch?v=o7zSLNiTZbA
Slides of my talk at Berlin Buzzwords in June 2016.
Abstract:
"In the past few years Apache Kafka has established itself as the world's most popular real-time, large-scale messaging system. It is used across a wide range of industries by thousands of companies such as Netflix, Cisco, PayPal, Twitter, and many others.
In this session I am introducing the audience to Kafka Streams, which is the latest addition to the Apache Kafka project. Kafka Streams is a stream processing library natively integrated with Kafka. It has a very low barrier to entry, easy operationalization, and a high-level DSL for writing stream processing applications. As such it is the most convenient yet scalable option to process and analyze data that is backed by Kafka. We will provide the audience with an overview of Kafka Streams including its design and API, typical use cases, code examples, and an outlook of its upcoming roadmap. We will also compare Kafka Streams' light-weight library approach with heavier, framework-based tools such as Apache Storm and Spark Streaming, which require you to understand and operate a whole different infrastructure for processing real-time data in Kafka."
OSDC 2015: Bernd Mathiske | Why the Datacenter Needs an Operating SystemNETWAYS
Developers are moving away from their host-based patterns and adopting a new mindset around the idea that the datacenter is the computer. It?s quickly becoming a mainstream model that you can view a warehouse full of servers as a single computer (with terabytes of memory and tens of thousands of cores). There is a key missing piece, which is an operating system for the datacenter (DCOS), which would provide the same OS functionality and core OS abstractions across thousands of machines that an OS provides on a single machine today. In this session, we will discuss:
How the abstraction of an OS has evolved over time and can cleanly scale to spand thousands of machines in a datacenter.
How key open source technologies like the Apache Mesos distributed systems kernel provide the key underpinnings for a DCOS.
How developers can layer core system services on top of a distributed systems kernel, including an init system (Marathon), cron (Chronos), service discovery (DNS), and storage (HDFS)
What would the interface to the DCOS look like? How would you use it?
How you would install and operate datacenter services, including Apache Spark, Apache Cassandra, Apache Kafka, Apache Hadoop, Apache YARN, Apache HDFS, and Google's Kubernetes.
How will developers build datacenter-scale apps, programmed against the datacenter OS like it?s a single machine?
As CloudStack cannot work with any mysql clustering solution, it is time to explore a new locking service, manager and pluggable interface which would allow CloudStack DB to be HA enabled with multi-master read/write. Talk will focus on, - Need for a locking service and challenges with existing CloudStack architecture - Different possible clustering solution that can be adopted - Showcasing a PoC for future implementation with minimal changes to existing architecture using percona xtradb or any other clustering solution - Additionally, explore the idea of getting rid of mshost table, and use locking service to find about other management servers.
DevOps Fest 2020. Сергій Калінець. Building Data Streaming Platform with Apac...DevOps_Fest
Apache Kafka зараз на хайпі. Все більше компаній починають використовувати її, як message bus. Проте Kafka може набагато більше, аніж бути просто транспортом. Її реальна міць і краса розкриваються, коли Kafka стає центральною нервовою системою вашої архітектури. Вона швидка, надійна і доволі гнучка для різних сценаріїв використання.
На цій доповіді Сергій поділитися досвідом побудови data streaming платформи. Ми поговоримо про те, як Kafka працює, як її потрібно конфігурувати і в які халепи можна потрапити, якщо Kafka використовується неоптимально.
Kafka Connect is a framework which connects Kafka with external Systems. It helps to move the data in and out of the Kafka. Connect makes it simple to use existing connector configuration for common source and sink Connectors.
Putting Kafka In Jail – Best Practices To Run Kafka On Kubernetes & DC/OSLightbend
Apache Kafka–part of Lightbend Fast Data Platform–is a distributed streaming platform that is best suited to run close to the metal on dedicated machines in statically defined clusters. For most enterprises, however, these fixed clusters are quickly becoming extinct in favor of mixed-use clusters that take advantage of all infrastructure resources available.
In this webinar by Sean Glover, Fast Data Engineer at Lightbend, we will review leading Kafka implementations on DC/OS and Kubernetes to see how they reliably run Kafka in container orchestrated clusters and reduce the overhead for a number of common operational tasks with standard cluster resource manager features. You will learn specifically about concerns like:
* The need for greater operational knowhow to do common tasks with Kafka in static clusters, such as applying broker configuration updates, upgrading to a new version, and adding or decommissioning brokers.
* The best way to provide resources to stateful technologies while in a mixed-use cluster, noting the importance of disk space as one of Kafka’s most important resource requirements.
* How to address the particular needs of stateful services in a model that natively favors stateless, transient services.
What is Mesos? How does it works? In the following slides we make an interesting review of this open-source software project to manage computer clusters.
Cloud Infrastructures Slide Set 8 - More Cloud Technologies - Mesos, Spark | ...anynines GmbH
Beside IaaS and PaaS there is a growing number of Cluster-Managers for maintaining spezialised Compute Frameworks. In this set of slides you will find a short introduction of the Cluster-Manager Apache Mesos and the Compute Framework Apache Spark.
JCConf 2016 - Cloud Computing Applications - Hazelcast, Spark and IgniteJoseph Kuo
This session aims to establish applications running against distributed and scalable system, or as we know cloud computing system. We will introduce you not only briefing of Hazelcast but also deeper kernel of it, and how it works with Spark, the most famous Map-reduce library. Furthermore, we will introduce another in-memory cache called Apache Ignite and compare it with Hazelcast to see what's the difference between them. In the end, we will give a demonstration showing how Hazelcast and Spark work together well to form a cloud-base service which is distributed, flexible, reliable, available, scalable and stable. You can find demo code here: https://github.com/CyberJos/jcconf2016-hazelcast-spark
https://cyberjos.blog/java/seminar/jcconf-2016-cloud-computing-applications-hazelcast-spark-and-ignite/
To facilitate a variety of usage scenarios and gradually scale to larger number of users, Galaxy supports deployment on systems ranging from a laptop to a supercomputer to clouds. In this talk, real-world examples of two different models for harnessing a variety of resources will be presented: (1) a centralized Galaxy utilizing a set of geographically distributed resources in support of a large user base, and (2) a model of easily deploying multiple standalone instances of Galaxy to support high resource demands or customizations by a smaller groups. Together, these models showcase the capacity of Galaxy to support a variety of usage scenarios and a variable number of users with a variety of needs.
DataStax | Building a Spark Streaming App with DSE File System (Rocco Varela)...DataStax
In this talk, we review a real-world use case that tested the Cassandra+Spark stack on Datastax Enterprise (DSE). We also cover implementation details around application high availability and fault tolerance using the new DSE File System (DSEFS). From a field and testing perspective, we discuss the strategies we can leverage to meet our requirements. Such requirements include (but not limited to) functional coverage, system integration, usability, and performance. We will discuss best practices and lessons we learned covering everything from application development to DSE setup and tuning.
About the Speaker
Rocco Varela Software Engineer in Test, DataStax
After earning his PhD in bioinformatics from UCSF, Rocco Varela took his passion for technology to DataStax. At DataStax he works on several aspects of performance and test automation around DataStax Enterprise (DSE) integrated offerings such as Apache Spark, Hadoop, Solr, and more recently DSE Graph.
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
2. CEO of Elodina, Inc. Elodina http://www.elodina.net/ is a startup focusing
on the support & maintenance of third party open source software (like
Mesos frameworks) and offering SaaS based solutions for those
systems. Elodina started as Big Data Open Source Security
http://stealth.ly and has been working for the last couple of years on
implementing and assisting organizations with their Kafka, Mesos,
Hadoop, Cassandra, Accumulo, Storm, Spark, etc, Big Data systems.
Twitter: https://twitter.com/allthingshadoop
LinkedIn: https://www.linkedin.com/in/charmalloc
Joe Stein
5. Papers
◉ Mesos: A Platform for Fine-Grained Resource Sharing in the
Data Center
https://www.cs.berkeley.edu/~alig/papers/mesos.pdf
◉ Omega: flexible, scalable schedulers for large compute clusters
http://eurosys2013.tudos.org/wp-
content/uploads/2013/paper/Schwarzkopf.pdf
◉ Large-scale cluster management at Google with Borg
http://research.google.com/pubs/pub43438.html
12. Mesos
● Scalability to 10,000s of nodes
● Fault-tolerant replicated master and slaves using ZooKeeper
● Support for Docker containers
● Native isolation between tasks with Linux Containers
● Multi-resource scheduling (memory, CPU, disk, and ports)
● Java, Python and C++ APIs for developing new parallel
applications
● Web UI for viewing cluster state
18. Roles
Total consumable resources per slave, in the form 'name(role):value;name(role):value...'. This value can be set
to limit resources per role, or to overstate the number of resources that are available to the slave.
--resources="cpus(*):8; mem(*):15360; disk(*):710534; ports(*):[31000-32000]"
--resources="cpus(prod):8; cpus(stage):2 mem(*):15360; disk(*):710534; ports(*):[31000-32000]"
All * roles will be detected, so you can specify only the resources that are not all roles (*). --
resources="cpus(prod):8; cpus(stage)"
Frameworks bind a specific roles or any. A default roll (instead of *) can also be configured.
Roles can be used to isolate and segregate frameworks.
19. Attributes
The Mesos system has two basic methods to describe the
slaves that comprise a cluster. One of these is managed
by the Mesos master, the other is simply passed onwards
to the frameworks using the cluster.
--attributes='disks:sata;raid:jbod;dc:1;rack:3'
20. Constraints
Constraints control where apps run to
allow optimizing for fault tolerance or
locality. Constraints are made up of
three parts: a field name, an operator,
and an optional parameter. The field
can be the slave hostname or any
Mesos slave attribute.
◉ UNIQUE
◉ CLUSTER
◉ GROUP_BY
◉ LIKE
◉ UNLIKE
22. Future release(s) to make things even better!
MESOS-2018 Dynamic Reservations
MESOS-1554 Persistent resources support for storage-like
services
MESOS-1279 Add resize task primitive
MESOS-1607 Optimistic Offers
27. Goals we set out with
● smart broker.id assignment.
● preservation of broker placement (through constraints
and/or new features).
● ability to-do configuration changes.
● rolling restarts (for things like configuration changes).
● scaling the cluster up and down with automatic,
programmatic and manual options.
● smart partition assignment via constraints visa vi
roles, resources and attributes.
28. Scheduler
● Provides the operational automation for a Kafka Cluster.
● Manages the changes to the broker's configuration.
● Exposes a REST API for the CLI to use or any other
client.
● Runs on Marathon for high availability.
Executor
● The executor interacts with the kafka broker as an
intermediary to the scheduler
Scheduler & Executor
29. CLI & REST API
● scheduler - starts the scheduler.
● add - adds one more more brokers to the cluster.
● update - changes resources, constraints or broker properties one or more
brokers.
● remove - take a broker out of the cluster.
● start - starts a broker up.
● stop - this can either a graceful shutdown or will force kill it (./kafka-mesos.sh
help stop)
● rebalance - allows you to rebalance a cluster either by selecting the brokers
or topics to rebalance. Manual assignment is still possible using the Apache
Kafka project tools. Rebalance can also change the replication factor on a
topic.
● help - ./kafka-mesos.sh help || ./kafka-mesos.sh help {command}
33. Cassandra on Mesos
https://github.com/mesosphere/cassandra-mesos
The Mesos scheduler is the component with the most
high-level intelligence in the framework. It will need to
possess the ability to bootstrap a ring and distribute
the correct configuration to all subsequently started
nodes. The Scheduler will also be responsible for
orchestrating all tasks with regard to restarting nodes
and triggering and monitoring periodic administrative
tasks required by a node.
34. Cassandra Scheduler
◉ Bootstrapping a ring
◉ Adding nodes to a ring
◉ Restarting a node that has crashed
◉ Providing configuration to nodes
o Seed nodes, Snitch Class, JVM
OPTS
◉ Scheduling and running administrative
utilities
o nodetool repair
o nodetool cleanup
◉ Registers with a failover timeout
◉ Supports framework authentication
◉ Declines offers to resources it
doesn't need
◉ Only use necessary fraction of
offers
◉ Deal with lost tasks
◉ Does not rely on in-memory state
◉ Verifies supported Mesos Version
◉ Supports roles
◉ Able to provide set of ports to be
used by Nodes
◉ Initial implementation will be for a
static set of ports with a potential
for longer term dynamic port usage.
35. Cassandra Executor
◉ Monitor health of running node
◉ Use JMX Mbeans for interfacing
with Cassandra Server Process
◉ Communicate results of
administrative actions via
StatusUpdates to scheduler
when necessary
◉ Does not rely on file system
state outside sandbox
◉ Pure libprocess
communication with Scheduler
leveraging StatusUpdate
◉ Does not rely on running on a
particular slave node
◉ Data directories will be
created and managed by
Mesos leveraging the features
provided in MESOS-1554
37. HDFS on Mesos
https://github.com/mesosphere/hdfs
◉ 3 journal nodes
◉ 2 name nodes (active/standby)
◉ data nodes, lots of them!
◉ Fault tolerance more
than just what Hadoop
gives.
◉ Ease of configuration and
distributing nodes.
◉ Elastic DFS
◉ Run multiple frameworks
at a time for new
solutions
39. MySQL on Mesos (Apache Incubating)
◉ Open sourced by Twitter https://github.com/twitter/mysos
◉ Moving to Apache https://twitter.com/ApacheMysos
◉ Dramatically simplifies the management of a MySQL cluster:
o Efficient hardware utilization through multi-tenancy (in performance-
isolated containers)
o High reliability through preserving the MySQL state during failure and
automatic backing up to/restoring from HDFS
o An automated self-service option for bringing up new MySQL clusters
o High availability through automatic MySQL master failover
o An elastic solution that allows users to easily scale up and down a MySQL
cluster by changing the number of slave instances