This document discusses distributed tracing standards and microservices simulations. It introduces OpenZipkin and OpenTracing as open source distributed tracing projects. It also discusses Pivot Tracing and the OpenTracing initiative to standardize instrumentation. The document proposes using a microservices simulator called Spigo to generate test data and visualize traces. It provides an example of defining a LAMP stack architecture in JSON to simulate with Spigo.
A rough and researchy presentation where I tried out some new material in front of a local audience. Skipped the usual introduction and talked about some of the problems people run into when they do microservices and miss a few things. More refined version of this talk to be shown at O'Reilly Software Architecture Conference in New York in April.
GameDay - Achieving resilience through Chaos EngineeringDiUS
http://dius.com.au/resources/game-day/
Agility has brought us iterative software development, independent feature teams, nimble architectures and distributed, scalable infrastructure. But how do you maintain confidence in these systems in the face of this emergent complexity and fast paced change? The answer is to anticipate and practice failure!
In this session we explore GameDays, a collaborative exercise where teams safely introduce chaos into their systems, in order to make them better.
Opening talk at Monitorama, talks about the problems of monitoring, challenges of creating monitoring tools and why monitoring vendors keep getting disrupted. Ended with a discussion of simulation testing and serverless architectures - Monitorless.
A rough and researchy presentation where I tried out some new material in front of a local audience. Skipped the usual introduction and talked about some of the problems people run into when they do microservices and miss a few things. More refined version of this talk to be shown at O'Reilly Software Architecture Conference in New York in April.
GameDay - Achieving resilience through Chaos EngineeringDiUS
http://dius.com.au/resources/game-day/
Agility has brought us iterative software development, independent feature teams, nimble architectures and distributed, scalable infrastructure. But how do you maintain confidence in these systems in the face of this emergent complexity and fast paced change? The answer is to anticipate and practice failure!
In this session we explore GameDays, a collaborative exercise where teams safely introduce chaos into their systems, in order to make them better.
Opening talk at Monitorama, talks about the problems of monitoring, challenges of creating monitoring tools and why monitoring vendors keep getting disrupted. Ended with a discussion of simulation testing and serverless architectures - Monitorless.
It's clear that Docker speeds up development and makes testing and deployment more efficient. As Docker moves into production new use cases and patterns are emerging that address availability and security concerns. With microservices, safety is part of the architecture that developers need to understand and build for. It's no longer good enough to wrap a firewall around an entire app when it goes to production, and have a cold standby in case it breaks.
Discussion of how microservices are being applied across both web scale and enterprise/government use cases to help speed up development.
Video available at http://www.ustream.tv/recorded/86151804
Businesses are speeding up development and automating operations to remain competitive and to get large organizations to scale. Project based monolithic application updates are replaced by product teams owning containerized microservices. This puts developers on call, responsible for pushing code to production, fixing it when it breaks, and managing the cost and security aspects of running their microservices. In this world operations skill-sets are either embedded in the microservices development teams, or building and operating API driven platforms. The platform automates stress testing, canary based deployment, penetration testing and enforces availability and security requirements. There are no meetings or tickets to file in the delivery process for updating a containerized microservice, which can happen many times a day, and takes seconds to complete. The role of site reliability engineering moves from firefighting and fixing outages to buiding tools for finding problems and routing those problems to the right developers. SREs manage the incident lifecycle for customer visible problems, and measure and publish availability metrics. This may sound futuristic but Werner Vogels described this as “You build it, you run it” in 2006.
Slides originally written in April 2013 for a private conference and internal use at Netflix. Publishing now since Heartbleed is another example of an epidemic failure mode.
Sildes of an internal talk given at Twitter similar to a previous webinar for Redhat with the same title.
Speeding up development is a key concern, cloud and technology improvements like Docker speed up key steps that make continuous delivery possible. Breaking up the work into many separate microservices and datastores with stable APIs allows teams to make progress independently so that the organization scales. Monolithic apps are preferred for small projects, built by small teams and when very low latency and high efficiency is the primary requirement. Monitoring microservices is currently a challenge with solutions starting to emerge.
Microxchg Analyzing Response Time Distributions for MicroservicesAdrian Cockcroft
Research oriented presentation @Microxchg Berlin Feb 5th 2016. New code to collect histograms of response time and export them to monte-carlo simulation spreadsheet via getguesstimate.com
Containerizing your Security Operations CenterJimmy Mesta
AppSec USA 2016 talk on using containers and Kubernetes to manage a variety of security tools. Includes best practices for securing Kubernetes implementations.
You just got “done” with the transformation of your organization (or parts of it) to leverage more DevOps practices, and now the next hot thing is taking over the industry: containers, Cloud Native, SRE, GitOps, Kubernetes, etc. What’s a DevOps Manager to do? Throw away the last few years and rebrand the team as Cloud Native SREs?
Technological advancement not only provides advancement in “what” a modern technology architecture looks like, it can also provide advancement in the processes and the day to day of an organization’s technology teams. We’ve seen this before in the move from mainframe to client-server, and client-server to Cloud.
In this presentation I’ll talk about the relationship of DevOps to Cloud Native technologies, and help make sense of all the jargon - containers, microservices, orchestration (and Kubernetes), SRE, GitOps, etc. I’ll also talk about how some processes & practices in the world of DevOps change when leveraging these technologies. Attendees will leave with a base understanding of what a DevOps operating model looks like when leveraging modern Cloud Native technologies.
The Emergent Cloud Security Toolchain for CI/CDJames Wickett
The Emergent Cloud Security Toolchain for CI/CD given at RSA Conference 2018 in San Francisco.
All organizations want to go faster and decrease friction in their cloud software delivery pipeline. Infosec has an opportunity to change their classic approach from blocker to enabler. This talk will discuss hallmarks of CI/CD and some practical examples for adding security testing across different organizations. The talk will cover emergent patterns, practices and toolchains that bring security to the table.
Learning Objectives:
1: Learn the emerging patterns for security in CI/CD pipelines.
2: Receive a pragmatic security toolchain for CI/CD to use in your organization.
3: Understand the real meaning of DevSecOps is without all the hype.
From Zero to DevSecOps in 60 Minutes - DevTalks Romania - Cluj-Napocajerryhargrove
Whether you’re building an application in a DevOps + Security culture, or have already bridged the gap with DevSecOps, the task remains the same: How do you ensure that security best practices are understood, architected for and integrated into your application from day 1 AND remain relevant year 1. During this talk I’ll focus on how to achieve these goals amidst the ever changing landscape of people, process, and technology in the cloud, in the context of various compute environments like instances, containers and serverless functions. and how to do so using off-the-shelf AWS services and features. I’ll complete the story by accompanying this discussion with a reference application architecture and examples. Attendees of this talk will receive actionable best practices and guidance, with specific implementation details for AWS
Lattice: A Cloud-Native Platform for Your Spring ApplicationsMatt Stine
As presented at SpringOne2GX 2015 in Washington, DC.
Lattice is a cloud-native application platform that enables you to run your applications in containers like Docker, on your local machine via Vagrant. Lattice includes features like:
Cluster scheduling
HTTP load balancing
Log aggregation
Health management
Lattice does this by packaging a subset of the components found in the Cloud Foundry elastic runtime. The result is an open, single-tenant environment suitable for rapid application development, similar to Kubernetes and Mesos Applications developed using Lattice should migrate unchanged to full Cloud Foundry deployments.
Lattice can be used by Spring developers to spin up powerful micro-cloud environments on their desktops, and can be useful for developing and testing cloud-native application architectures. Lattice already has deep integration with Spring Cloud and Spring XD, and you’ll have the opportunity to see deep dives into both at this year’s SpringOne 2GX. This session will introduce the basics:
Installing Lattice
Lattice’s Architecture
How Lattice Differs from Cloud Foundry
How to Package and Run Your Spring Apps on Lattice
Serverless Security: Doing Security in 100 millisecondsJames Wickett
Talk on serverless security with a brief history of cloud, containers and now serverless. This talk also features serverless patterns, and security considerations needed in this new environment. This talk was given at AppSecUSA 2016.
DevOps Days Boston 2017: Developer first workflows for KubernetesAmbassador Labs
Kubernetes is a powerful, operational platform for containerized applications. However, the developer workflow on Kubernetes – how you code, deploy, update, and monitor your services – is much less mature.
How should you lay out your Git repo? How do you create loosely coupled services? How do you support deploying your service at any time?
In this talk, we’ll talk about these questions and more. We’ll discuss the journey towards a rapid development workflow, discuss best practices, and, talk about the process we followed to get to a rapid development workflow.
DevOps Days Boston 2017
Microservices: What's Missing - O'Reilly Software Architecture New YorkAdrian Cockcroft
Assuming you have already figured out microservices, what else do you need to figure out to get them to work properly. This talk skips my usual introduction to why and what, and goes deeper on how.
It's clear that Docker speeds up development and makes testing and deployment more efficient. As Docker moves into production new use cases and patterns are emerging that address availability and security concerns. With microservices, safety is part of the architecture that developers need to understand and build for. It's no longer good enough to wrap a firewall around an entire app when it goes to production, and have a cold standby in case it breaks.
Discussion of how microservices are being applied across both web scale and enterprise/government use cases to help speed up development.
Video available at http://www.ustream.tv/recorded/86151804
Businesses are speeding up development and automating operations to remain competitive and to get large organizations to scale. Project based monolithic application updates are replaced by product teams owning containerized microservices. This puts developers on call, responsible for pushing code to production, fixing it when it breaks, and managing the cost and security aspects of running their microservices. In this world operations skill-sets are either embedded in the microservices development teams, or building and operating API driven platforms. The platform automates stress testing, canary based deployment, penetration testing and enforces availability and security requirements. There are no meetings or tickets to file in the delivery process for updating a containerized microservice, which can happen many times a day, and takes seconds to complete. The role of site reliability engineering moves from firefighting and fixing outages to buiding tools for finding problems and routing those problems to the right developers. SREs manage the incident lifecycle for customer visible problems, and measure and publish availability metrics. This may sound futuristic but Werner Vogels described this as “You build it, you run it” in 2006.
Slides originally written in April 2013 for a private conference and internal use at Netflix. Publishing now since Heartbleed is another example of an epidemic failure mode.
Sildes of an internal talk given at Twitter similar to a previous webinar for Redhat with the same title.
Speeding up development is a key concern, cloud and technology improvements like Docker speed up key steps that make continuous delivery possible. Breaking up the work into many separate microservices and datastores with stable APIs allows teams to make progress independently so that the organization scales. Monolithic apps are preferred for small projects, built by small teams and when very low latency and high efficiency is the primary requirement. Monitoring microservices is currently a challenge with solutions starting to emerge.
Microxchg Analyzing Response Time Distributions for MicroservicesAdrian Cockcroft
Research oriented presentation @Microxchg Berlin Feb 5th 2016. New code to collect histograms of response time and export them to monte-carlo simulation spreadsheet via getguesstimate.com
Containerizing your Security Operations CenterJimmy Mesta
AppSec USA 2016 talk on using containers and Kubernetes to manage a variety of security tools. Includes best practices for securing Kubernetes implementations.
You just got “done” with the transformation of your organization (or parts of it) to leverage more DevOps practices, and now the next hot thing is taking over the industry: containers, Cloud Native, SRE, GitOps, Kubernetes, etc. What’s a DevOps Manager to do? Throw away the last few years and rebrand the team as Cloud Native SREs?
Technological advancement not only provides advancement in “what” a modern technology architecture looks like, it can also provide advancement in the processes and the day to day of an organization’s technology teams. We’ve seen this before in the move from mainframe to client-server, and client-server to Cloud.
In this presentation I’ll talk about the relationship of DevOps to Cloud Native technologies, and help make sense of all the jargon - containers, microservices, orchestration (and Kubernetes), SRE, GitOps, etc. I’ll also talk about how some processes & practices in the world of DevOps change when leveraging these technologies. Attendees will leave with a base understanding of what a DevOps operating model looks like when leveraging modern Cloud Native technologies.
The Emergent Cloud Security Toolchain for CI/CDJames Wickett
The Emergent Cloud Security Toolchain for CI/CD given at RSA Conference 2018 in San Francisco.
All organizations want to go faster and decrease friction in their cloud software delivery pipeline. Infosec has an opportunity to change their classic approach from blocker to enabler. This talk will discuss hallmarks of CI/CD and some practical examples for adding security testing across different organizations. The talk will cover emergent patterns, practices and toolchains that bring security to the table.
Learning Objectives:
1: Learn the emerging patterns for security in CI/CD pipelines.
2: Receive a pragmatic security toolchain for CI/CD to use in your organization.
3: Understand the real meaning of DevSecOps is without all the hype.
From Zero to DevSecOps in 60 Minutes - DevTalks Romania - Cluj-Napocajerryhargrove
Whether you’re building an application in a DevOps + Security culture, or have already bridged the gap with DevSecOps, the task remains the same: How do you ensure that security best practices are understood, architected for and integrated into your application from day 1 AND remain relevant year 1. During this talk I’ll focus on how to achieve these goals amidst the ever changing landscape of people, process, and technology in the cloud, in the context of various compute environments like instances, containers and serverless functions. and how to do so using off-the-shelf AWS services and features. I’ll complete the story by accompanying this discussion with a reference application architecture and examples. Attendees of this talk will receive actionable best practices and guidance, with specific implementation details for AWS
Lattice: A Cloud-Native Platform for Your Spring ApplicationsMatt Stine
As presented at SpringOne2GX 2015 in Washington, DC.
Lattice is a cloud-native application platform that enables you to run your applications in containers like Docker, on your local machine via Vagrant. Lattice includes features like:
Cluster scheduling
HTTP load balancing
Log aggregation
Health management
Lattice does this by packaging a subset of the components found in the Cloud Foundry elastic runtime. The result is an open, single-tenant environment suitable for rapid application development, similar to Kubernetes and Mesos Applications developed using Lattice should migrate unchanged to full Cloud Foundry deployments.
Lattice can be used by Spring developers to spin up powerful micro-cloud environments on their desktops, and can be useful for developing and testing cloud-native application architectures. Lattice already has deep integration with Spring Cloud and Spring XD, and you’ll have the opportunity to see deep dives into both at this year’s SpringOne 2GX. This session will introduce the basics:
Installing Lattice
Lattice’s Architecture
How Lattice Differs from Cloud Foundry
How to Package and Run Your Spring Apps on Lattice
Serverless Security: Doing Security in 100 millisecondsJames Wickett
Talk on serverless security with a brief history of cloud, containers and now serverless. This talk also features serverless patterns, and security considerations needed in this new environment. This talk was given at AppSecUSA 2016.
DevOps Days Boston 2017: Developer first workflows for KubernetesAmbassador Labs
Kubernetes is a powerful, operational platform for containerized applications. However, the developer workflow on Kubernetes – how you code, deploy, update, and monitor your services – is much less mature.
How should you lay out your Git repo? How do you create loosely coupled services? How do you support deploying your service at any time?
In this talk, we’ll talk about these questions and more. We’ll discuss the journey towards a rapid development workflow, discuss best practices, and, talk about the process we followed to get to a rapid development workflow.
DevOps Days Boston 2017
Microservices: What's Missing - O'Reilly Software Architecture New YorkAdrian Cockcroft
Assuming you have already figured out microservices, what else do you need to figure out to get them to work properly. This talk skips my usual introduction to why and what, and goes deeper on how.
Microservices, Kubernetes and Istio - A Great Fit!Animesh Singh
Microservices and containers are now influencing application design and deployment patterns. Sixty percent of all new applications will use cloud-enabled continuous delivery microservice architectures and containers. Service discovery, registration, and routing are fundamental tenets of microservices. Kubernetes provides a platform for running microservices. Kubernetes can be used to automate the deployment of Microservices and leverage features such as Kube-DNS, Config Maps, and Ingress service for managing those microservices. This configuration works fine for deployments up to a certain size. However, with complex deployments consisting of a large fleet of microservices, additional features are required to augment Kubernetes.
Just about all of my current technical content in one 364 slide mega-deck. Source files at https://github.com/adrianco/slides
Sections on:
Scene Setting
State of the Cloud
What Changes?
Product Processes
Microservices
State of the Art
Segmentation
What’s Missing?
Monitoring
Challenges
Migration
Response Times
Serverless
Lock-In
Teraservices
Wrap-Up
Microservices are small services with independent lifecycles that work together. There is an underlying tension in that definition – how independent can you be when you have to be part of a whole? I’ve spent much of the last couple of years trying to understand how to find the right balance, and in this talk/tutorial I’ll be presenting the core seven principles that I think represent what makes microservices tick.
After a brief introduction of what microservices are and why they are important, we’ll spend the bulk of the time looking at the principles themselves, wherever possible covering real-world examples and technology:
- Modelled around business domain – using techniques from domain-driven design to find service boundaries leads to better team alignment and more stable service boundaries, avoiding expensive cross-service changes.
- Culture of automation – all organisations that use microservices at scale have strong cultures of automation. We’ll look at some of their stories and think about which sort of automation is key.
- Hide implementation details – how do you hide the detail inside each service to avoid coupling, and ensure each service retains its autonomous nature?
- Decentralize all the things! – we have to push power down as far as we can, and this goes for both the system and organisational architecture. We’ll look at everything from autonomous self-contained teams and internal open source, to using choreographed systems to handle long-lived business transactions.
- Deploy independently – this is all about being able to deploy safely. So we’ll cover everything from deployment models to consumer-driven contracts and the importance of separating deployment from release.
- Isolate failure – just making a system distributed doesn’t make it more stable than a monolithic application. So what do you need to look for?
- Highly observable – we need to understand the health of a single service, but also the whole ecosystem. How?
In terms of learning outcomes, beginners will get a sense of what microservices are and what makes them different, whereas more experienced practitioners will get insight and practical advice into how to implement them.
Top 5 Deep Learning and AI Stories - October 6, 2017NVIDIA
Read this week's top 5 news updates in deep learning and AI: Gartner predicts top 10 strategic technology trends for 2018; Oracle adds GPU Accelerated Computing to Oracle Cloud Infrastructure; chemistry and physics Nobel Prizes are awarded to teams supported by GPUs; MIT uses deep learning to help guide decisions in ICU; and portfolio management firms are using AI to seek alpha.
Genomic Computation at Scale with Serverless, StackStorm and Docker SwarmDmitri Zimine
Presented on SuperComputing SC17 on Nov 14/2017 by Dmitri Zimine.
This talk is a story of bio-tech meeting DevOps to produce genomic computations, economically, and at scale.
Genomic computation is growing in demand as it comes to the mainstream practices of bio-technology, agriculture, and personal medicine. It also explodes the demand for compute resources. In fact, with inexpensive next-gen sequencing, some labs sequence over 1,000,000 billion bases per year. Genetic data banks are growing over 10x annually. How to compute the genomic data at massive scale, and do it in a cost-efficient way?
In the presentation, we describe and demonstrate a serverless solution built with Docker, Docker Swarm, StackStorm and other tools from the DevOps toolchain on AWS. The solution offers a new take on creating and computing a bio-informatic pipelines that can run at high scale and at optimal cost.
http://sc17.supercomputing.org/presentation/?id=exforum106&sess=sess150
Docker moves very fast, with an edge channel released every month and a stable release every 3 months. Patrick will talk about how Docker introduced Docker EE and a certification program for containers and plugins with Docker CE and EE 17.03 (from March), the announcements from DockerCon (April), and the many new features planned for Docker CE 17.05 in May.
This talk will be about what's new in Docker and what's next on the roadmap
Ultra Fast Deep Learning in Hybrid Cloud Using Intel Analytics Zoo & AlluxioAlluxio, Inc.
Alluxio Global Online Meetup
Apr 23, 2020
For more Alluxio events: https://www.alluxio.io/events/
Speakers:
Jiao (Jennie) Wang, Intel
Tsai Louie, Intel
Bin Fan, Alluxio
Today, many people run deep learning applications with training data from separate storage such as object storage or remote data centers. This presentation will demo the Intel Analytics Zoo + Alluxio stack, an architecture that enables high performance while keeping cost and resource efficiency balanced without network being I/O bottlenecked.
Intel Analytics Zoo is a unified data analytics and AI platform open-sourced by Intel. It seamlessly unites TensorFlow, Keras, PyTorch, Spark, Flink, and Ray programs into an integrated pipeline, which can transparently scale from a laptop to large clusters to process production big data. Alluxio, as an open-source data orchestration layer, accelerates data loading and processing in Analytics Zoo deep learning applications.
This talk, we will go over:
- What is Analytics Zoo and how it works
- How to run Analytics Zoo with Alluxio in deep learning applications
- Initial performance benchmark results using the Analytics Zoo + Alluxio stack
Новый InterSystems: open-source, митапы, хакатоныTimur Safin
Presentation for the 1st InterSystems Meetup in the Minsk:
- New and better InterSystems changes their practice.
- open-source repositories, meetups, and hackathon;
- CPM (package manager) as a good example of open-source project
IoTWorld 2016 OSS Keynote Param Singh, Ian SkerrettParam Singh
Emergent Open Source IoT Ecosystem
There is a vibrant open source ecosystem developing around all layers of the IoT software stack. These technologies, when woven together, have the potential of propelling the Internet of things forward exponentially. Open source provides a trusted space where device vendors and software companies can reliably share components essential to interconnect the currently splintered IoT ecosystem.
Come see what is happening and how you can leverage open source IoT software right now.
Ian Skerrett, VP of Marketing, Eclipse Foundation
Param Singh, CEO, iotracks; IoT Advisor, City of San Francisco
https://iotworldevent.com/iot-open-source-summit/
This work is licensed under a Creative Commons Attribution 4.0 International License.
Practical Chaos Engineering will show how to start running chaos experiments in your infrastructure and will try to guide your through the principles of chaos.
Running Emerging AI Applications on Big Data Platforms with Ray On Apache SparkDatabricks
With the rapid evolution of AI in recent years, we need to embrace advanced and emerging AI technologies to gain insights and make decisions based on massive amounts of data. Ray (https://github.com/ray-project/ray) is a fast and simple framework open-sourced by UC Berkeley RISELab particularly designed for easily building advanced AI applications in a distributed fashion.
Erik Skytthe - Monitoring Mesos, Docker, Containers with Zabbix | ZabConf2016Zabbix
At DBC we are running docker and other container types in a mesos/marathon cluster environment. I will demonstrate how we collect statistics, logs etc. and monitor this environment, showing configuration examples, data flows and templates.
Some of the covered topics:
- Mesos master and agents
- Marathon Framework
- Docker engine
- Containers
- Zookeeper
- Elasticserach/ELK
The aim of the EU FP 7 Large-Scale Integrating Project LarKC is to develop the Large Knowledge Collider (LarKC, for short, pronounced “lark”), a platform for massive distributed incomplete reasoning that will remove the scalability barriers of currently existing reasoning systems for the Semantic Web. The LarKC platform is available at larkc.sourceforge.net. This talk, is part of a tutorial for early users of the LarKC platform, and introduces the platform and the project in general.
Running Accurate, Scalable, and Reproducible Simulations of Distributed Syste...Rafael Ferreira da Silva
Scientific workflows are used routinely in numerous scientific domains, and Workflow Management Systems (WMSs) have been developed to orchestrate and optimize workflow executions on distributed platforms. WMSs are complex software systems that interact with complex software infrastructures. Most WMS research and development activities rely on empirical experiments conducted with full-fledged software stacks on actual hardware platforms. Such experiments, however, are limited to hardware and software infrastructures at hand and can be labor- and/or time-intensive. As a result, relying solely on real- world experiments impedes WMS research and development. An alternative is to conduct experiments in simulation.
In this work we present WRENCH, a WMS simulation framework, whose objectives are (i) accurate and scalable simula- tions; and (ii) easy simulation software development. WRENCH achieves its first objective by building on the SimGrid framework. While SimGrid is recognized for the accuracy and scalability of its simulation models, it only provides low-level simulation abstractions and thus large software development efforts are required when implementing simulators of complex systems. WRENCH thus achieves its second objective by providing high- level and directly re-usable simulation abstractions on top of SimGrid. After describing and giving rationales for WRENCH’s software architecture and APIs, we present a case study in which we apply WRENCH to simulate the Pegasus production WMS. We report on ease of implementation, simulation accuracy, and simulation scalability so as to determine to which extent WRENCH achieves its two above objectives. We also draw both qualitative and quantitative comparisons with a previously proposed workflow simulator.
Cloud Expo East 2013: Essential Open Source Software for Building the Open CloudMark Hinkle
Cloud computing is more than a buzz-phrase it’s a transformative IT paradigm shift. The emphasis in the cloud is on elasticity, scalability, agility and open. Not just open standards but open APIs and open source. The delivery of software is also going through a paradigm shift. Open source software was often a commoditization of a market leader; Unix to Linux or Oracle to MySQL what’s changing is that the iterative nature, user context and the motto of releasing early and often are driving real innovation in open source.
This session will cover those essential open source technologies for delivering cloud computing in the enterprise.
Speaker Bio:
Mark Hinkle is the Senior Director, Open Source Solutions at Citrix Systems Inc. He joined Citrix as a result of their July 2011 acquisition of Cloud.com where he was their Vice President of Community. He is currently responsible for Citrix open source efforts around the open source cloud computing platform, Apache CloudStack and the Xen Hypervisor. Previously he was the VP of Community at Zenoss Inc., a producer of the open source application, server, and network management software, where he grew the Zenoss Core project to over 100,000 users and 20,000 organizations on all seven continents. He also is a longtime open source expert and author having served as Editor-in-Chief for both LinuxWorld Magazine and Enterprise Open Source Magazine. His blog on open source, technology, and new media can be found at http://www.socializedsoftware.com.
Ultra Fast Deep Learning in Hybrid Cloud using Intel Analytics Zoo & AlluxioAlluxio, Inc.
Data Orchestration Summit 2020 organized by Alluxio
https://www.alluxio.io/data-orchestration-summit-2020/
Ultra Fast Deep Learning in Hybrid Cloud using Intel Analytics Zoo & Alluxio
Jennie Wang, Software Engineer (Intel)
Tsai Louie, Software Engineer (Intel)
About Alluxio: alluxio.io
Engage with the open source community on slack: alluxio.io/slack
Historical view of process and channel oriented programming idioms: CSP 1978, Occam 1983, Pi-Calculus 1993 etc. How they map to Go and some examples of dynamic channel routing using Go to simulate peer-to-peer networks and microservices networks.
Full slide deck for day long discussion of microservices topics. Why use microservices, what options exist and how to migrate to them and address common problems.
There are many ways to manage whether a service can talk to another service. It can be tempting to over-use one segmentation mechanism to implement policy when the real problem is how to coordinate and manage many mechanisms in the physical, cloud and container spaces. This talk summarizes the problem space and opportunities rather than offers solutions.
Presented at the Docker Palo Alto meetup Feb 16th 2016 http://www.meetup.com/Docker-Palo-Alto/events/228277181/
Updated slides for 2016 presentation on innovation in large organizations, why microservices and Docker can be useful, thoughts on monitoring for large complex architectures, some discussion of new topics - serverless architectures AWS Lambda and teraservices.
Summary of fast development and cloud native architecture along with cost optimization techniques. Presented as opening keynote at the Utility and Cloud Computing 2014 as part of the Cloud Control Workshop.
Keynote at Dockercon Europe Amsterdam Dec 4th, 2014.
Speeding up development with Docker.
Summary of some interesting web scale microservice architectures.
Please send me updates and corrections to the architecture summaries @adrianco
Thanks Adrian
Monitorama - Please, no more Minutes, Milliseconds, Monoliths or Monitoring T...Adrian Cockcroft
Monitorama opening keynote talk on the challenges of Monitoring in a world where we need to deal with continuous delivery, cloud, and automated control feedback loops.
Hack Kid Con - Learn to be a Data Scientist for $1Adrian Cockcroft
Attempt to inspire some kids to pay attention in Math and Science classes so they can get a good job and help fill the skills gap in the years to come.
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Globus
The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.
Globus Connect Server Deep Dive - GlobusWorld 2024Globus
We explore the Globus Connect Server (GCS) architecture and experiment with advanced configuration options and use cases. This content is targeted at system administrators who are familiar with GCS and currently operate—or are planning to operate—broader deployments at their institution.
Quarkus Hidden and Forbidden ExtensionsMax Andersen
Quarkus has a vast extension ecosystem and is known for its subsonic and subatomic feature set. Some of these features are not as well known, and some extensions are less talked about, but that does not make them less interesting - quite the opposite.
Come join this talk to see some tips and tricks for using Quarkus and some of the lesser known features, extensions and development techniques.
SOCRadar Research Team: Latest Activities of IntelBrokerSOCRadar
The European Union Agency for Law Enforcement Cooperation (Europol) has suffered an alleged data breach after a notorious threat actor claimed to have exfiltrated data from its systems. Infamous data leaker IntelBroker posted on the even more infamous BreachForums hacking forum, saying that Europol suffered a data breach this month.
The alleged breach affected Europol agencies CCSE, EC3, Europol Platform for Experts, Law Enforcement Forum, and SIRIUS. Infiltration of these entities can disrupt ongoing investigations and compromise sensitive intelligence shared among international law enforcement agencies.
However, this is neither the first nor the last activity of IntekBroker. We have compiled for you what happened in the last few days. To track such hacker activities on dark web sources like hacker forums, private Telegram channels, and other hidden platforms where cyber threats often originate, you can check SOCRadar’s Dark Web News.
Stay Informed on Threat Actors’ Activity on the Dark Web with SOCRadar!
Understanding Globus Data Transfers with NetSageGlobus
NetSage is an open privacy-aware network measurement, analysis, and visualization service designed to help end-users visualize and reason about large data transfers. NetSage traditionally has used a combination of passive measurements, including SNMP and flow data, as well as active measurements, mainly perfSONAR, to provide longitudinal network performance data visualization. It has been deployed by dozens of networks world wide, and is supported domestically by the Engagement and Performance Operations Center (EPOC), NSF #2328479. We have recently expanded the NetSage data sources to include logs for Globus data transfers, following the same privacy-preserving approach as for Flow data. Using the logs for the Texas Advanced Computing Center (TACC) as an example, this talk will walk through several different example use cases that NetSage can answer, including: Who is using Globus to share data with my institution, and what kind of performance are they able to achieve? How many transfers has Globus supported for us? Which sites are we sharing the most data with, and how is that changing over time? How is my site using Globus to move data internally, and what kind of performance do we see for those transfers? What percentage of data transfers at my institution used Globus, and how did the overall data transfer performance compare to the Globus users?
Accelerate Enterprise Software Engineering with PlatformlessWSO2
Key takeaways:
Challenges of building platforms and the benefits of platformless.
Key principles of platformless, including API-first, cloud-native middleware, platform engineering, and developer experience.
How Choreo enables the platformless experience.
How key concepts like application architecture, domain-driven design, zero trust, and cell-based architecture are inherently a part of Choreo.
Demo of an end-to-end app built and deployed on Choreo.
Check out the webinar slides to learn more about how XfilesPro transforms Salesforce document management by leveraging its world-class applications. For more details, please connect with sales@xfilespro.com
If you want to watch the on-demand webinar, please click here: https://www.xfilespro.com/webinars/salesforce-document-management-2-0-smarter-faster-better/
Cyaniclab : Software Development Agency Portfolio.pdfCyanic lab
CyanicLab, an offshore custom software development company based in Sweden,India, Finland, is your go-to partner for startup development and innovative web design solutions. Our expert team specializes in crafting cutting-edge software tailored to meet the unique needs of startups and established enterprises alike. From conceptualization to execution, we offer comprehensive services including web and mobile app development, UI/UX design, and ongoing software maintenance. Ready to elevate your business? Contact CyanicLab today and let us propel your vision to success with our top-notch IT solutions.
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...Anthony Dahanne
Les Buildpacks existent depuis plus de 10 ans ! D’abord, ils étaient utilisés pour détecter et construire une application avant de la déployer sur certains PaaS. Ensuite, nous avons pu créer des images Docker (OCI) avec leur dernière génération, les Cloud Native Buildpacks (CNCF en incubation). Sont-ils une bonne alternative au Dockerfile ? Que sont les buildpacks Paketo ? Quelles communautés les soutiennent et comment ?
Venez le découvrir lors de cette session ignite
Developing Distributed High-performance Computing Capabilities of an Open Sci...Globus
COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among public health practitioners, mathematical modelers, and scientific computing specialists, while revealing critical gaps in exploiting advanced computing systems to support urgent decision making. Informed by our team’s work in applying high-performance computing in support of public health decision makers during the COVID-19 pandemic, we present how Globus technologies are enabling the development of an open science platform for robust epidemic analysis, with the goal of collaborative, secure, distributed, on-demand, and fast time-to-solution analyses to support public health.
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTier1 app
Even though at surface level ‘java.lang.OutOfMemoryError’ appears as one single error; underlyingly there are 9 types of OutOfMemoryError. Each type of OutOfMemoryError has different causes, diagnosis approaches and solutions. This session equips you with the knowledge, tools, and techniques needed to troubleshoot and conquer OutOfMemoryError in all its forms, ensuring smoother, more efficient Java applications.
May Marketo Masterclass, London MUG May 22 2024.pdfAdele Miller
Can't make Adobe Summit in Vegas? No sweat because the EMEA Marketo Engage Champions are coming to London to share their Summit sessions, insights and more!
This is a MUG with a twist you don't want to miss.
We describe the deployment and use of Globus Compute for remote computation. This content is aimed at researchers who wish to compute on remote resources using a unified programming interface, as well as system administrators who will deploy and operate Globus Compute services on their research computing infrastructure.
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoamtakuyayamamoto1800
In this slide, we show the simulation example and the way to compile this solver.
In this solver, the Helmholtz equation can be solved by helmholtzFoam. Also, the Helmholtz equation with uniformly dispersed bubbles can be simulated by helmholtzBubbleFoam.
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus
As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...informapgpstrackings
Keep tabs on your field staff effortlessly with Informap Technology Centre LLC. Real-time tracking, task assignment, and smart features for efficient management. Request a live demo today!
For more details, visit us : https://informapuae.com/field-staff-tracking/
Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics.
To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/
Microservices Application Tracing Standards and Simulators - Adrians at OSCON
1. Microservices Application Tracing Standards
and Simulators
From Zipkin to Greater Tracing: Involving a wider group of people in
distributed tracing
@adrianfcole
@adrianco
#oscon
5. Distributed Tracing commoditizes knowledge
Distributed tracing systems collect end-to-end latency graphs
(traces) in near real-time.
You can compare traces to understand why certain requests
take longer than others.
6. Zipkin is like Chrome DevTool’s network panel!
http://zipkin.io/
• .. except you see your whole architecture
8. commit 92c941890c2009a401b777093342dc4f28955640
Author: Johan Oskarsson <johan@oskarsson.nu>
Date: Tue Nov 15 10:09:47 2011 -0800
[split] Enable B3 tracing for TFE. Filter out finagle-http headers from incoming requests
BigBrotherBird is silently born
9. Zipkin is less silently born
commit 2b7acead044e71c744f39804abe564383eb5f846
Author: Johan Oskarsson <johan@oskarsson.nu>
Date: Wed Jun 6 11:28:34 2012 -0700
Initial commit
12. So what happened?
Zipkin development at Twitter was in short bursts, centered on other work
Many experienced Zipkin engineers don’t work at Twitter (or in the Bay Area)
Platform diversity is a reality for many
Having the same goals was our opportunity
13. How’s OpenZipkin doing now?
Zipkin's now releasable (maybe too releasable)
We’re working on understandability on usability
We’re making the community easier to find
We hit bumps, and sometimes reverse change
15. The “greater” tracing
Many groups are solving similar problems
Some focus on stacks, others on instrumentation
By collaborating more, we can make tracing greater
16. Instrumentation portability
Interop through shared trace pipelines.
Practical matters, like categorization and tactical designs
Moving R&D to implementation
Simulation and system testing
distributed-tracing google group
Distributed Tracing Workgroup
17. OpenTracing is an effort to clean-up and de-risk
distributed tracing instrumentation
OpenTracing Interfaces decouple instrumentation from
vendor-specific dependencies and terminology. This
allows applications to switch products with less effort.
http://opentracing.io/
OpenTracing: Go, Python, Java, JavaScript..
18. A single configuration change to bind a Tracer
implementation in main() or similar
import opentracing "github.com/opentracing/opentracing-go"
import "github.com/tracer_x/tracerimpl"
func main() {
// Bind tracerimpl to the opentracing system
opentracing.InitGlobalTracer(
tracerimpl.New(kTracerImplAccessToken))
... normal main() stuff ...
}
How does it work?
Clean, vendor-neutral instrumentation code that
naturally tells the story of a distributed operation
import opentracing "github.com/opentracing/opentracing-go"
func AddContact(c *Contact) {
sp := opentracing.StartSpan("AddContact")
defer sp.Finish()
sp.LogEventWithPayload("Added contact: ", *c)
subRoutine(sp, ...)
...
}
func subRoutine(parentSpan opentracing.Span, ...) {
...
sp := opentracing.StartChildSpan(parentSpan, "subRoutine")
defer sp.Finish()
sp.Info("deferred work to subroutine")
...
}
Thanks, @el_bhs for
the slide!
19. Pivot Tracing is applied research from Brown University
(the one that brought us X-Trace).
Pivot tracing allows you to dynamically query systems at
runtime, grouping on “Baggage” which propagates across
service boundaries.
pivottracing.io
Pivot Tracing
20. Start writing queries including the fancy
happened-before join
From incr In DataNodeMetrics.incrBytesRead
Join cl In First(ClientProtocols) On cl -> incr
GroupBy cl.procName
Select cl.procName, SUM(incr.delta)
How does it work?
Services need to be in Java and be able to talk to
a provided PubSub broker.
// Add a library
<dependency>
<groupId>edu.brown.cs.systems</groupId>
<artifactId>pivottracing-agent</artifactId>
<version>4.0</version>
</dependency>
// Initialize it on bootstrap
PivotTracing.initialize(); @brownsys_jmace
made this!
22. What does @adrianco do?
@adrianco
Technology Due
Diligence on Deals
Presentations at
Conferences
Presentations at
Companies
Technical
Advice for Portfolio
Companies
Program
Committee for
Conferences
Networking with
Interesting PeopleTinkering with
Technologies
Maintain
Relationship with
Cloud Vendors
Previously: Netflix, eBay, Sun Microsystems, CCL, TCU London BSc Applied Physics
23. @adrianco
Testing Flow Monitors
Monitoring tools often “explode on impact” with
real world use cases at scale
Interestingly large complex environments are
expensive to create or hard to get access to
Free, open source tools don’t have a budget…
24. OSS Microservice Simulator
Model and visualize microservices
Simulate interesting architectures
Generate large scale configurations
Stress test real tools like Zipkin
Code: github.com/adrianco/spigo
Simulate Protocol Interactions in Go
Visualize with D3, Neo4j or Guesstimate
See for yourself: http://simianviz.surge.sh
Follow @simianviz for updates
ELB Load Balancer
Zuul
API Proxy
Karyon
Business Logic
Staash
Data Access Layer
Priam
Cassandra Datastore
Three
Availability
Zones
Denominator
DNS Endpoint
25. POST Spigo flows to zipkin
# collect flows, duration 2 seconds, architecture lamp
$ ./spigo -c —d 2 -a lamp
—snip—
# clean out zipkin database and post newly created data
$ misc/zipkin.sh lamp
26. Spigo Nanoservice Structure
func Start(listener chan gotocol.Message) {
...
for {
select {
case msg := <-listener:
flow.Instrument(msg, name, hist)
switch msg.Imposition {
case gotocol.Hello: // get named by parent
...
case gotocol.NameDrop: // someone new to talk to
...
case gotocol.Put: // upstream request handler
...
outmsg := gotocol.Message{gotocol.Replicate, listener, time.Now(),
msg.Ctx.NewParent(), msg.Intention}
flow.AnnotateSend(outmsg, name)
outmsg.GoSend(replicas)
}
case <-eurekaTicker.C: // poll the service registry
...
}
}
}
Skeleton code for sideways replicating a Put message
Instrument incoming requests
Instrument outgoing requests
update trace context
32. Migrating to Microservices
See for yourself: http://simianviz.surge.sh/migration
Endpoint
ELB
PHP
MySQL
MySQL
Next step Controls node
placement distance
Select models
36. Single Region Riak IoT
IoT Ingestion Endpoint
Stream Endpoint
Analytics Endpoint
See for yourself: http://simianviz.surge.sh/riak
37. Single Region Riak IoT
IoT Ingestion Endpoint
Stream Endpoint
Analytics Endpoint
Load Balancer
Load Balancer
Load Balancer
See for yourself: http://simianviz.surge.sh/riak
38. Single Region Riak IoT
IoT Ingestion Endpoint
Stream Endpoint
Analytics Endpoint
Load Balancer
Normalization Services
Load Balancer
Load Balancer
Stream Service
Analytics Service
See for yourself: http://simianviz.surge.sh/riak
39. Single Region Riak IoT
IoT Ingestion Endpoint
Stream Endpoint
Analytics Endpoint
Load Balancer
Normalization Services
Enrich Message Queue
Riak KV
Enricher Services
Load Balancer
Load Balancer
Stream Service
Analytics Service
See for yourself: http://simianviz.surge.sh/riak
40. Single Region Riak IoT
IoT Ingestion Endpoint
Stream Endpoint
Analytics Endpoint
Load Balancer
Normalization Services
Enrich Message Queue
Riak KV
Enricher Services
Ingest Message Queue
Load Balancer
Load Balancer
Stream Service
Analytics Service
See for yourself: http://simianviz.surge.sh/riak
41. Single Region Riak IoT
IoT Ingestion Endpoint
Stream Endpoint
Analytics Endpoint
Load Balancer
Normalization Services
Enrich Message Queue
Riak KV
Enricher Services
Ingest Message Queue
Load Balancer
Load Balancer
Stream Service Riak TS
Analytics Service
Ingester Service
See for yourself: http://simianviz.surge.sh/riak
42. Two Region Riak IoT
IoT Ingestion Endpoint
Stream Endpoint
Analytics Endpoint
East Region Ingestion
West Region Ingestion
Multi Region TS Analytics
See for yourself: http://simianviz.surge.sh/riak
46. @adrianco
Conclusion
Monitoring tools can be stressed at scale by
simulating their inputs with metrics, dependency
graphs and flows
Spigo can be extended to produce any format
output at very large scale from a laptop