Azure Functions make it easy to create and host webhook interfaces without maintaining a server. You can quickly setup an endpoint to receive data and act on it. Being able to ingest, process, and respond to data from a variety of sources without building out an infrastructure gives you time to focus on building functionality.
In this presentation, Nick Zimmerman, Sr. Site Reliability Engineer at SparkPost, will show you how to setup an Azure Function, accept webhook data, process that data with C#, and integrate that data into an application in real time.
The Rounds Project: Growing from thousands to millions - Berry Ventura & Yoah...DroidConTLV
The document discusses tips for building a great mobile app from the Rounds Android team's experience with their app. It emphasizes supporting many devices and languages, having a robust and crash-free app, using responsive UI design, and surprising and delighting users. Technical strategies discussed include using an executor service for background threads, an IntentService for asynchronous tasks, and events via broadcasts to communicate between components. The goal is to scale the app while keeping it fast, reliable and a great experience for users.
The signs are on the wall, scripting and coding is becoming increasingly important for any role in IT. PowerShell is a crucial component in your toolkit and in this session we will cover the basics of PowerShell and how you can get started and be successful, how we can secure systems running PowerShell and what benefits can it bring.
In this session Jaap will talk about what PowerShell is, the language, the console, the editors and how PowerShell came to be. Recently PowerShell has been made open source and multi-platform. We will look at what the implications are and what benefits this brings along.
As security is increasingly a more important consideration for every company, we will take a look at what capabilities PowerShell has and how we can use these tools to create a more secure scripting environment.
This session contains multiple real-world examples and applications of this technology. There will be time for questions during the session and if you have any specific PowerShell related questions that are not answered in the presentation Jaap will be available for questions all day.
Practical Continuous Deployment - Atlassian - London AUG 18 Feb 2014Matthew Cobby
The document discusses practical approaches to implementing continuous deployment. It describes converting an organization's internal systems to continuous delivery and deployment over six months to address integration issues. Continuous deployment aims to release features, not unfinished work, through automation that makes releasing repeatable. Stakeholders benefit from faster delivery of features to customers and clearer progress signals. The document outlines a development workflow involving tracking requests, branching per feature, automated testing, code reviews, merging to a release branch, and deploying to staging and production. It also addresses challenges of automation and coordination across servers for the "last mile" of deployment.
Apply chat automation today - work smarter tomorrowJaap Brasser
Today we will look at something different, Chat Automation often also referred to as ChatOps. What is this and why should we care about it? In this session I will take you through the basics, discuss why this is worth our time and how it can help you get more out of your investment of PowerShell scripts and modules. Aside from that we are going to take a challenge, get a fully customizable PowerShell Slack bot up-and-running within 10 minutes.
LONDON SDET MEETUP - Difference between Selenium and Cypress presentationSriram Angajala
This document summarizes a presentation comparing the automated testing tools Cypress and Selenium. It outlines key features of each tool such as browser support, speed, and capabilities. Cypress is described as a frontend testing tool that supports only Chrome and JavaScript, while Selenium supports multiple browsers and languages but requires more configuration. The presentation demonstrates Cypress for integration and end-to-end testing and concludes that Cypress may replace Selenium if it gains support for more browsers, remote execution, behavior-driven development, and additional programming languages.
Philly CocoaHeads 20160414 - Building Your App SDK With SwiftJordan Yaker
This document provides a summary of the steps taken to build an iOS SDK for the Supportify app over multiple iterations:
1. The initial SDK was created in Objective-C using a base framework but was difficult to implement and had code quality issues.
2. An updated SDK was created using Swift, but usability problems remained as it was still difficult for developers to implement.
3. Further iterations incorporated iOS design patterns like Clean Swift to reduce dependencies and modularize components. However, problems with the underlying API specification and tests emerged.
4. The final SDK version incorporated a Swagger-based proxy, models, authentication, and other improvements to address issues, resulting in a more full-featured and easier
Chat automation in a modern it environmentJaap Brasser
In this session we will cover a range of topics regarding Chat automation and the possibilities this gives us. We will dive into the concept of ChatOps, what it is, how we can implement it and what the benefits are. After this brief introduction we will dive into real-world examples of setting up and configuring your first chatbot, configuring security and resolving real-world incidents using this chatbot.
The Rounds Project: Growing from thousands to millions - Berry Ventura & Yoah...DroidConTLV
The document discusses tips for building a great mobile app from the Rounds Android team's experience with their app. It emphasizes supporting many devices and languages, having a robust and crash-free app, using responsive UI design, and surprising and delighting users. Technical strategies discussed include using an executor service for background threads, an IntentService for asynchronous tasks, and events via broadcasts to communicate between components. The goal is to scale the app while keeping it fast, reliable and a great experience for users.
The signs are on the wall, scripting and coding is becoming increasingly important for any role in IT. PowerShell is a crucial component in your toolkit and in this session we will cover the basics of PowerShell and how you can get started and be successful, how we can secure systems running PowerShell and what benefits can it bring.
In this session Jaap will talk about what PowerShell is, the language, the console, the editors and how PowerShell came to be. Recently PowerShell has been made open source and multi-platform. We will look at what the implications are and what benefits this brings along.
As security is increasingly a more important consideration for every company, we will take a look at what capabilities PowerShell has and how we can use these tools to create a more secure scripting environment.
This session contains multiple real-world examples and applications of this technology. There will be time for questions during the session and if you have any specific PowerShell related questions that are not answered in the presentation Jaap will be available for questions all day.
Practical Continuous Deployment - Atlassian - London AUG 18 Feb 2014Matthew Cobby
The document discusses practical approaches to implementing continuous deployment. It describes converting an organization's internal systems to continuous delivery and deployment over six months to address integration issues. Continuous deployment aims to release features, not unfinished work, through automation that makes releasing repeatable. Stakeholders benefit from faster delivery of features to customers and clearer progress signals. The document outlines a development workflow involving tracking requests, branching per feature, automated testing, code reviews, merging to a release branch, and deploying to staging and production. It also addresses challenges of automation and coordination across servers for the "last mile" of deployment.
Apply chat automation today - work smarter tomorrowJaap Brasser
Today we will look at something different, Chat Automation often also referred to as ChatOps. What is this and why should we care about it? In this session I will take you through the basics, discuss why this is worth our time and how it can help you get more out of your investment of PowerShell scripts and modules. Aside from that we are going to take a challenge, get a fully customizable PowerShell Slack bot up-and-running within 10 minutes.
LONDON SDET MEETUP - Difference between Selenium and Cypress presentationSriram Angajala
This document summarizes a presentation comparing the automated testing tools Cypress and Selenium. It outlines key features of each tool such as browser support, speed, and capabilities. Cypress is described as a frontend testing tool that supports only Chrome and JavaScript, while Selenium supports multiple browsers and languages but requires more configuration. The presentation demonstrates Cypress for integration and end-to-end testing and concludes that Cypress may replace Selenium if it gains support for more browsers, remote execution, behavior-driven development, and additional programming languages.
Philly CocoaHeads 20160414 - Building Your App SDK With SwiftJordan Yaker
This document provides a summary of the steps taken to build an iOS SDK for the Supportify app over multiple iterations:
1. The initial SDK was created in Objective-C using a base framework but was difficult to implement and had code quality issues.
2. An updated SDK was created using Swift, but usability problems remained as it was still difficult for developers to implement.
3. Further iterations incorporated iOS design patterns like Clean Swift to reduce dependencies and modularize components. However, problems with the underlying API specification and tests emerged.
4. The final SDK version incorporated a Swagger-based proxy, models, authentication, and other improvements to address issues, resulting in a more full-featured and easier
Chat automation in a modern it environmentJaap Brasser
In this session we will cover a range of topics regarding Chat automation and the possibilities this gives us. We will dive into the concept of ChatOps, what it is, how we can implement it and what the benefits are. After this brief introduction we will dive into real-world examples of setting up and configuring your first chatbot, configuring security and resolving real-world incidents using this chatbot.
Careful - APIs Inside: Testing and Monitoring for App Development3scale
This document discusses the importance of testing and monitoring APIs that power applications. It notes that apps often depend on backend APIs and problems with APIs can waste development time or even kill apps. It then provides recommendations for tools to help with API development and operations testing, including HTTP sniffers, OAuth libraries, documentation sites, monitoring services, and API-specific analytics platforms. The goal is to make working with APIs easier so developers and providers can have more fun and be more profitable.
From Code to the Monkeys: Continuous Delivery at NetflixDianne Marsh
At Netflix, we continue to improve upon our continuous delivery process. We thrive in a hybrid environment, where every developer is able to deploy code, and with that freedom comes the responsibility for ensuring that our customers are not negatively impacted. We have constructed Open Source tools toward a Continuous Delivery solution. In this presentation, from QConSF 2013, you will learn about our tool chain so that you can determine which make sense in your environment.
Make Your UI Tests Resilient with the Next Generation of FrameworksSatyajit Malugu
A big problem with test automation on any platform or operating system is synchronizing test automation interactions with the UI. It is challenging to know when the UI is ready for the next automated click(). Traditional black box tools try to address this problem by explicit or implicit waiting, but this technique is slow and error-prone. A new generation of test frameworks, starting with Espresso, understands the internals of the app and synchronizes interactions only when the view is ready, making the framework very fast and reliable. This same technique is making Cypress and EarlGrey popular. Join Satyajit Malugu to discover how peeking below the surface of the system under test makes tests less flaky. You'll leave with an understanding of synchronization, black box versus gray box testing, and how to implement them in your own frameworks.
This document discusses monitoring system and application metrics using tools like Collectd and Logster to send data to Graphite. It provides examples of key metrics to monitor for resources like CPU, memory, disk, network and Nginx. These include utilization, saturation and errors. Graph examples are given showing how to visualize request rates, response times, and resource usage over time in Graphite. The document also discusses configuring Collectd and Logster and creating a custom Logster parser to send detailed request timing data rather than averages.
This document summarizes a meetup about serverless technologies. It introduces the organizer and details that the meetup group was started as an experiment and has grown to 140 members who meet monthly. The key aspects of serverless technologies highlighted are that they are fast evolving and poorly documented. The most important things about the meetup are to join online discussions, present topics, provide contributions and ideas, and help spread awareness of serverless technologies.
The document discusses the growth of Site Reliability Engineering (SRE) at Squarespace from a team of 2 people in New York to a global organization with teams in New York, Portland, and Dublin. It describes how the initial SRE team focused on three pillars: monitoring and alerting, configuration management, and builds and deploys. It then explains how the SRE organization expanded to include additional teams focused on areas like provisioning, release engineering, developer productivity, and observability while also embedding SREs within product teams.
The document discusses serverless architectures and compares them to traditional architectures like monoliths, 3-tier, and microservices. It defines serverless as having functions that are triggered by events to execute code without needing to manage servers. Serverless architectures offer benefits like automatic scaling, pay-per-use billing, and not needing to manage servers directly. However, they may increase complexity for large systems and have limited runtime options currently. Examples of serverless use cases and platforms are also provided.
Enabling your DevOps culture with AWS-webinarAaron Walker
In this presentation shows you how the benefits of AWS technologies can be combined with a new approach to Development and Operations.
It’s all about delivering new features and functionality faster, without compromising reliability, stability and performance.
* Understand the challenges faced by traditional Development and Operations teams
* Apply Continuous Integration/Delivery processes and tools to enable change
* Appreciate how various AWS technologies can be used to facilitate DevOps
The document discusses options for setting up continuous integration for native iOS applications. It explores unit testing libraries like Sentest and gh-unit, functional testing options like UIAutomation, iCuke, and Frank, and static analysis tools like scan-build. While many existing tools were found lacking or difficult to use in a CI environment, Frank was identified as an actively maintained and easy to run functional testing option. The document also mentions concerns about testing push notifications and API interactions that may break with changes.
This document discusses new features in Puppet Enterprise 2016.5 that improve high availability, including eliminating single points of failure to ensure Puppet continues running. It introduces simple agent-side failover that happens at the application level for enterprise customers, promoting a replica when the primary fails. During failovers in Puppet Enterprise 2016.5, scheduled Puppet runs, catalog compilation, and reporting can still occur, but deploying new code, editing node classifier data, using the console, certificate functionality, most CLI tools and application orchestration are not possible.
Building software by feature with immutable infrastructures on AWSNicolas Mas
How serv.sg deliver features using AWS infrastructure: programatically building and provisioning a stack before destroying it. This talk will explain what they do, why they do it and the tools they use (packer.io, terraform.io, ansible etc.). Includes a live demo of the whole cycle.
Video of the talk: http://engineers.sg/v/474
Top 8 mistakes developer teams make in their first serverless projectPaul Swail
Your development team is about to embark on its first fully serverless project. You want to get productive quickly and extract the most benefit from the AWS serverless ecosystem you are moving into. Find out the common pitfalls to avoid and ensure you get off on the best possible footing.
Applying Roles and Profiles method to compliance codePuppet
This document discusses adapting the roles and profiles design pattern to writing compliance code in Puppet modules. It begins by noting the challenges of writing compliance code, such as it touching many parts of nodes and leading to sprawling code. It then provides an overview of the roles and profiles pattern, which uses simple "front-end" roles/interfaces and more complex "back-end" profiles/implementations. The rest of the document discusses how to apply this pattern when authoring Puppet modules for compliance - including creating interface and implementation classes, using Hiera for configuration, and tools for reducing boilerplate code. It aims to provide a maintainable structure and simplify adapting to new compliance frameworks or requirements.
MonkeyTalk is an open source automation tool that allows testing of mobile, web, and desktop applications across platforms using object-based recording. It features cross-platform scripting and the ability to run tests interactively or in a headless automated manner without requiring app jailbreaking. MonkeyTalk uses a client-server model where an IDE installs an agent on the target device to record and playback scripts consisting of component objects, actions, and parameters.
Innovation dank DevOps (DevOpsCon Berlin 2015)Wooga
“You build it, you run it!” - Wenn Du als Entwickler weisst, dass Du Deine Software selbst betreiben musst, was bist bereit zu tun, um den späteren Betrieb zu vereinfach?
Bei Wooga haben Dutzende von Teams ihre eigene Antwort auf die Frage gesucht und dabei von den Erfahrungen der anderen Teams gelernt. Herausgekommen ist ein großes Experimentierfeld beim Betrieb von Web Services - und eine technologische Innovation, die uns innerhalb weniger Iterationen von einem simplen LAMP-Stack zu lastabhängig skalierenden stateful Servern auf Basis von Erlang oder Akka gebracht hat.
This document discusses applying security automation principles through a SecDevOps approach. It begins by highlighting lessons from other companies that deployed features in a disabled state using feature flags and integrated security testing in continuous integration. The document then outlines how Kenna applies SecDevOps principles through automation, with examples like using Chef for configuration management and testing security at each code check. It also presents a use case where Kenna loads security scanning results from various tools into its platform via API to enable continuous security testing.
- Build automation helps ensure consistent builds, prevents errors, and speeds up the release process. It helps development teams integrate and deliver changes continuously.
- Common tools for build automation include MSBuild, Team Foundation Server, CruiseControl.NET, and Hudson. These tools help with continuous integration (CI), running tests, code analysis, versioning, and deploying builds.
- Best practices include CI on every code check-in, running unit tests as part of the build to prevent bugs, and continuously delivering integrated builds to environments for testing. This supports rapid and reliable software delivery.
Ike Ellis gave a presentation on the 14 habits of great SQL developers. Some of the most important habits discussed were using source control, extensive testing, questioning assumptions, and fighting dependencies. Great SQL developers also work as a team, code for resiliency, and constantly improve code quality before moving on to new tasks. The goal is to deliver value and leave applications better organized and more maintainable than when development began.
SparkOscope: Enabling Apache Spark Optimization through Cross Stack Monitorin...Databricks
During the last year, the team at IBM Research at Ireland has been using Apache Spark to perform analytics on large volumes of sensor data. These applications need to be executed on a daily basis, therefore, it was essential for them to understand Spark resource utilization. They found it cumbersome to manually consume and efficiently inspect the CSV files for the metrics generated at the Spark worker nodes.
Although using an external monitoring system like Ganglia would automate this process, they were still plagued with the inability to derive temporal associations between system-level metrics (e.g. CPU utilization) and job-level metrics (e.g. job or stage ID) as reported by Spark. For instance, they were not able to trace back the root cause of a peak in HDFS Reads or CPU usage to the code in their Spark application causing the bottleneck.
To overcome these limitations, they developed SparkOScope. Taking advantage of the job-level information available through the existing Spark Web UI and to minimize source-code pollution, they use the existing Spark Web UI to monitor and visualize job-level metrics of a Spark application (e.g. completion time). More importantly, they extend the Web UI with a palette of system-level metrics of the server/VM/container that each of the Spark job’s executor ran on. Using SparkOScope, you can navigate to any completed application and identify application-logic bottlenecks by inspecting the various plots providing in-depth timeseries for all relevant system-level metrics related to the Spark executors, while also easily associating them with stages, jobs and even source code lines incurring the bottleneck.
They have made Sparkoscope available as a standalone module, and also extended the available Sinks (mongodb, mysql).
Careful - APIs Inside: Testing and Monitoring for App Development3scale
This document discusses the importance of testing and monitoring APIs that power applications. It notes that apps often depend on backend APIs and problems with APIs can waste development time or even kill apps. It then provides recommendations for tools to help with API development and operations testing, including HTTP sniffers, OAuth libraries, documentation sites, monitoring services, and API-specific analytics platforms. The goal is to make working with APIs easier so developers and providers can have more fun and be more profitable.
From Code to the Monkeys: Continuous Delivery at NetflixDianne Marsh
At Netflix, we continue to improve upon our continuous delivery process. We thrive in a hybrid environment, where every developer is able to deploy code, and with that freedom comes the responsibility for ensuring that our customers are not negatively impacted. We have constructed Open Source tools toward a Continuous Delivery solution. In this presentation, from QConSF 2013, you will learn about our tool chain so that you can determine which make sense in your environment.
Make Your UI Tests Resilient with the Next Generation of FrameworksSatyajit Malugu
A big problem with test automation on any platform or operating system is synchronizing test automation interactions with the UI. It is challenging to know when the UI is ready for the next automated click(). Traditional black box tools try to address this problem by explicit or implicit waiting, but this technique is slow and error-prone. A new generation of test frameworks, starting with Espresso, understands the internals of the app and synchronizes interactions only when the view is ready, making the framework very fast and reliable. This same technique is making Cypress and EarlGrey popular. Join Satyajit Malugu to discover how peeking below the surface of the system under test makes tests less flaky. You'll leave with an understanding of synchronization, black box versus gray box testing, and how to implement them in your own frameworks.
This document discusses monitoring system and application metrics using tools like Collectd and Logster to send data to Graphite. It provides examples of key metrics to monitor for resources like CPU, memory, disk, network and Nginx. These include utilization, saturation and errors. Graph examples are given showing how to visualize request rates, response times, and resource usage over time in Graphite. The document also discusses configuring Collectd and Logster and creating a custom Logster parser to send detailed request timing data rather than averages.
This document summarizes a meetup about serverless technologies. It introduces the organizer and details that the meetup group was started as an experiment and has grown to 140 members who meet monthly. The key aspects of serverless technologies highlighted are that they are fast evolving and poorly documented. The most important things about the meetup are to join online discussions, present topics, provide contributions and ideas, and help spread awareness of serverless technologies.
The document discusses the growth of Site Reliability Engineering (SRE) at Squarespace from a team of 2 people in New York to a global organization with teams in New York, Portland, and Dublin. It describes how the initial SRE team focused on three pillars: monitoring and alerting, configuration management, and builds and deploys. It then explains how the SRE organization expanded to include additional teams focused on areas like provisioning, release engineering, developer productivity, and observability while also embedding SREs within product teams.
The document discusses serverless architectures and compares them to traditional architectures like monoliths, 3-tier, and microservices. It defines serverless as having functions that are triggered by events to execute code without needing to manage servers. Serverless architectures offer benefits like automatic scaling, pay-per-use billing, and not needing to manage servers directly. However, they may increase complexity for large systems and have limited runtime options currently. Examples of serverless use cases and platforms are also provided.
Enabling your DevOps culture with AWS-webinarAaron Walker
In this presentation shows you how the benefits of AWS technologies can be combined with a new approach to Development and Operations.
It’s all about delivering new features and functionality faster, without compromising reliability, stability and performance.
* Understand the challenges faced by traditional Development and Operations teams
* Apply Continuous Integration/Delivery processes and tools to enable change
* Appreciate how various AWS technologies can be used to facilitate DevOps
The document discusses options for setting up continuous integration for native iOS applications. It explores unit testing libraries like Sentest and gh-unit, functional testing options like UIAutomation, iCuke, and Frank, and static analysis tools like scan-build. While many existing tools were found lacking or difficult to use in a CI environment, Frank was identified as an actively maintained and easy to run functional testing option. The document also mentions concerns about testing push notifications and API interactions that may break with changes.
This document discusses new features in Puppet Enterprise 2016.5 that improve high availability, including eliminating single points of failure to ensure Puppet continues running. It introduces simple agent-side failover that happens at the application level for enterprise customers, promoting a replica when the primary fails. During failovers in Puppet Enterprise 2016.5, scheduled Puppet runs, catalog compilation, and reporting can still occur, but deploying new code, editing node classifier data, using the console, certificate functionality, most CLI tools and application orchestration are not possible.
Building software by feature with immutable infrastructures on AWSNicolas Mas
How serv.sg deliver features using AWS infrastructure: programatically building and provisioning a stack before destroying it. This talk will explain what they do, why they do it and the tools they use (packer.io, terraform.io, ansible etc.). Includes a live demo of the whole cycle.
Video of the talk: http://engineers.sg/v/474
Top 8 mistakes developer teams make in their first serverless projectPaul Swail
Your development team is about to embark on its first fully serverless project. You want to get productive quickly and extract the most benefit from the AWS serverless ecosystem you are moving into. Find out the common pitfalls to avoid and ensure you get off on the best possible footing.
Applying Roles and Profiles method to compliance codePuppet
This document discusses adapting the roles and profiles design pattern to writing compliance code in Puppet modules. It begins by noting the challenges of writing compliance code, such as it touching many parts of nodes and leading to sprawling code. It then provides an overview of the roles and profiles pattern, which uses simple "front-end" roles/interfaces and more complex "back-end" profiles/implementations. The rest of the document discusses how to apply this pattern when authoring Puppet modules for compliance - including creating interface and implementation classes, using Hiera for configuration, and tools for reducing boilerplate code. It aims to provide a maintainable structure and simplify adapting to new compliance frameworks or requirements.
MonkeyTalk is an open source automation tool that allows testing of mobile, web, and desktop applications across platforms using object-based recording. It features cross-platform scripting and the ability to run tests interactively or in a headless automated manner without requiring app jailbreaking. MonkeyTalk uses a client-server model where an IDE installs an agent on the target device to record and playback scripts consisting of component objects, actions, and parameters.
Innovation dank DevOps (DevOpsCon Berlin 2015)Wooga
“You build it, you run it!” - Wenn Du als Entwickler weisst, dass Du Deine Software selbst betreiben musst, was bist bereit zu tun, um den späteren Betrieb zu vereinfach?
Bei Wooga haben Dutzende von Teams ihre eigene Antwort auf die Frage gesucht und dabei von den Erfahrungen der anderen Teams gelernt. Herausgekommen ist ein großes Experimentierfeld beim Betrieb von Web Services - und eine technologische Innovation, die uns innerhalb weniger Iterationen von einem simplen LAMP-Stack zu lastabhängig skalierenden stateful Servern auf Basis von Erlang oder Akka gebracht hat.
This document discusses applying security automation principles through a SecDevOps approach. It begins by highlighting lessons from other companies that deployed features in a disabled state using feature flags and integrated security testing in continuous integration. The document then outlines how Kenna applies SecDevOps principles through automation, with examples like using Chef for configuration management and testing security at each code check. It also presents a use case where Kenna loads security scanning results from various tools into its platform via API to enable continuous security testing.
- Build automation helps ensure consistent builds, prevents errors, and speeds up the release process. It helps development teams integrate and deliver changes continuously.
- Common tools for build automation include MSBuild, Team Foundation Server, CruiseControl.NET, and Hudson. These tools help with continuous integration (CI), running tests, code analysis, versioning, and deploying builds.
- Best practices include CI on every code check-in, running unit tests as part of the build to prevent bugs, and continuously delivering integrated builds to environments for testing. This supports rapid and reliable software delivery.
Ike Ellis gave a presentation on the 14 habits of great SQL developers. Some of the most important habits discussed were using source control, extensive testing, questioning assumptions, and fighting dependencies. Great SQL developers also work as a team, code for resiliency, and constantly improve code quality before moving on to new tasks. The goal is to deliver value and leave applications better organized and more maintainable than when development began.
SparkOscope: Enabling Apache Spark Optimization through Cross Stack Monitorin...Databricks
During the last year, the team at IBM Research at Ireland has been using Apache Spark to perform analytics on large volumes of sensor data. These applications need to be executed on a daily basis, therefore, it was essential for them to understand Spark resource utilization. They found it cumbersome to manually consume and efficiently inspect the CSV files for the metrics generated at the Spark worker nodes.
Although using an external monitoring system like Ganglia would automate this process, they were still plagued with the inability to derive temporal associations between system-level metrics (e.g. CPU utilization) and job-level metrics (e.g. job or stage ID) as reported by Spark. For instance, they were not able to trace back the root cause of a peak in HDFS Reads or CPU usage to the code in their Spark application causing the bottleneck.
To overcome these limitations, they developed SparkOScope. Taking advantage of the job-level information available through the existing Spark Web UI and to minimize source-code pollution, they use the existing Spark Web UI to monitor and visualize job-level metrics of a Spark application (e.g. completion time). More importantly, they extend the Web UI with a palette of system-level metrics of the server/VM/container that each of the Spark job’s executor ran on. Using SparkOScope, you can navigate to any completed application and identify application-logic bottlenecks by inspecting the various plots providing in-depth timeseries for all relevant system-level metrics related to the Spark executors, while also easily associating them with stages, jobs and even source code lines incurring the bottleneck.
They have made Sparkoscope available as a standalone module, and also extended the available Sinks (mongodb, mysql).
The Project Chess team at IBM developed the original IBM PC in about a year using an agile approach, bypassing normal development processes. They built it with off-the-shelf parts and software from various OEMs. This rapid development approach helped Microsoft launch with their MS-DOS operating system. The author's alma mater FAU was influenced by IBM and the legacy of Project Chess. The author has helped migrate CU's Faculty Information System to newer technologies while adopting agile development practices and launching minimum viable products like VIVO CU-Boulder and UCCS FRPA to validate concepts with customers.
This document provides an overview of a guided hackathon to build a single page application using the MEAN stack (MongoDB, Express, AngularJS, Node.js) in 2 hours. It outlines the concepts that will be covered, including API testing, DOM integration testing, build systems, and more. Attendees will build a package manager for the Go programming language, creating the server with Express and Mongoose, and the client with AngularJS and Browserify. Testing will be done with Mocha, Karma, and other tools.
Build software like a bag of marbles, not a castle of LEGO®Hannes Lowette
If you have ever played with LEGO®, you will know that adding, removing or changing features of a completed castle isn’t as easy as it seems. You will have to deconstruct large parts to get to where you want to be, to build it all up again afterwards. Unfortunately, our software is often built the same way. Wouldn’t it be better if our software behaved like a bag of marbles? So you can just add, remove or replace them at will?
Most of us have taken different approaches to building software: a big monolith, a collection of services, a bus architecture, etc. But whatever your large scale architecture is, at the granular level (a single service or host), you will probably still end up with tightly couple code. Adding functionality means making changes to every layer, service or component involved. It gets even harder if you want to enable or disable features for certain deployments: you’ll need to wrap code in feature flags, write custom DB migration scripts, etc. There has to be a better way!
So what if you think of functionality as loose feature assemblies? We can construct our code in such a way that adding a feature is as simple as adding the assembly to your deployment, and removing it is done by just deleting the file. We would open the door for so many scenarios!
In this talk, I will explain how to tackle the following parts of your application to achieve this goal: WebAPI, Entity Framework, Onion Architecture, IoC and database migrations. And most of all, when you would want to do this. Because… ‘it depends’.
The Cowardly Test-o-Phobe's Guide To TestingTim Duckett
The document discusses testing iOS applications. It begins by addressing why testing can be scary and introduces test-driven development as an approach to make testing less scary. It then covers different tools for testing iOS apps, including frameworks for unit testing and UI testing. It emphasizes the importance of mocking and stubbing dependencies to isolate parts of the code for testing. The document also discusses approaches for testing network requests, such as using a tool called OHHTTPStubs to stub network responses from files. It concludes by encouraging testing user interfaces in code and using mocking/stubbing to handle dependencies.
Acing application lifecycle management in SharePointJeremy Thake
This document discusses application lifecycle management (ALM) in SharePoint. It defines ALM and its three aspects of governance, development, and operations. It then discusses approaches to ALM including source control, coding standards, testing, tracking, and release management. It presents an ALM maturity model and provides recommendations for getting started with ALM in SharePoint using tools like Visual Studio and Team Foundation Server. A case study is also presented of implementing ALM on a SharePoint site using Mercurial and DocAve.
Spark Summit EU talk by Yiannis GkoufasSpark Summit
This document introduces SparkOscope, a tool that enables optimization of Apache Spark through cross-stack monitoring and visualization. SparkOscope extends Spark's monitoring framework by collecting operating system-level metrics using Hyperic Sigar library. It enhances Spark's web UI to plot all available metrics, including the new OS metrics. SparkOscope provides an easy way to monitor Spark jobs and identify bottlenecks by associating metrics with source code locations. The author demonstrates SparkOscope and discusses plans to improve its storage mechanism and enable infrastructure recommendations based on job resource usage patterns.
Driving application development through behavior driven developmentEinar Ingebrigtsen
This document discusses Behavior Driven Development (BDD) and how it can be used to drive application development. It introduces BDD, focusing on behaviors of the system rather than tests. It discusses key aspects of BDD like Gherkin, units, test doubles, writing testable code, frameworks like SpecFlow and recommended reading. The overall message is that BDD changes the way software is developed by shifting the focus to behaviors and improving collaboration.
Getting your mobile test automation process in place - using Cucumber and Cal...Niels Frydenholm
Taking your mobile development process cycle, and the quality of the apps, from good to great.
See how focusing on automated tests can improve app quality, time to market and much more, and learn some best practices to avoid too much trouble getting started
Presented at Xamarin Evolve 2014
Topics of this presentation:
- Basics and best practices of developing single-page applications (SPA) and Web API Services on Microsoft .NET -
- Core with Docker and Linux.
- PowerShell Core automated builds.
- Markdown/PDF documentation.
- Documentation of public interfaces with Swagger/OAS/YAML.
- Automated testing of SPA on Protractor and testing the Web API on Postman/Newman.
This presentation by Sergii Fradkov (Consultant, Engineering), Andrii Zarharov (Lead Software Engineer, Consultant), Igor Magdich (Lead Test Engineer, Consultant) was delivered at GlobalLogic Kharkiv .NET TechTalk #1 on May 24, 2019.
Visual Studio LightSwitch (Beta 1) OverviewSteve Lange
Visual Studio LightSwitch allows developers to quickly create professional, data-centric business applications. It provides pre-configured screen templates and pre-written code to build the basic application functionality faster. LightSwitch applications can access existing data sources like SQL Server and SharePoint. The applications are made up of projects containing data entities, screens, queries, and other resources. Developers can customize and extend LightSwitch applications through code. LightSwitch applications can be deployed to the desktop, browser, or cloud.
[Rodrigo Pinto] Being an enterprise consultant in many companies across the globe, the most common phrase I come across each client\supplier I've helped last year is "each team as a different way to do it"In small\medium\huge companies, this is a major overkill.In this session learn how you can structure your teams, processes, tools, and development for SharePoint Enterprise solutions.
deliver:agile - Enable your Agile Team with Continuous Delivery PipelinesEsteban Garcia
Continuous Delivery session from deliver:Agile
As your Agile team looks to shorten the cycle time from idea to production, it is important to give them the tools that will enable continuous feedback, collaboration with stakeholders, and most importantly, a way to get the product in front of the customer and enable a feedback loop.
This session will teach you how to create an effective release pipeline that incorporates Continuous Integration, automated testing, cloud deployment with Infrastructure as Code, Instrumentation, load testing, and more.
We will go from zero to Production in less than an hour and you will go back to work on Monday ready to deploy!
Learning Outcomes:
Continuous Integration
Continuous Deployment
Automation
Create Microservice with Spring Boot and DockerSivaprakash
Spring Boot makes it easy to create standalone and production-ready Spring applications with minimal configuration. It aims to provide a faster development experience and common features like embedded servers and health checks. Spring Boot 2.0.1 requires Java 8/9 and works with embedded containers like Tomcat, Jetty and Undertow. Docker is a containerization platform that virtualizes applications and their dependencies to run on any OS without conflicts. It was developed by Docker Inc. and written in Go, with community and enterprise editions available.
Building High Quality Android ApplicationsLeif Janzik
The slides from my talk "Building High Quality Android Applications with Continuous Integration and Automated device tests" from April 9th at DroidCon 2013 in Berlin
This document discusses Pinterest's use of open source software and their PINRemoteImage project. It summarizes that Pinterest uses many open source projects in their iOS app due to scale needs, speed of development, and reliability. It then details their PINRemoteImage project which provides thread-safe, performant image downloading and processing for iOS. Key features of PINRemoteImage include support for formats like JPEG, PNG, WebP and GIFs, progressive downloading, adaptive quality based on network conditions, and custom processing of images.
Similar to Webhooks with Azure Functions - Live 360 Conference (20)
Unlocking Email’s Hidden Opportunities to Create a Competitive AdvantageSparkPost
Learn how to leverage the power of analytics and competitive intelligence to uncover opportunities to gain an edge
on competition and capture subscriber attention.
SparkPost Celebrates International Women's Day 2020SparkPost
In honor of Women's History Month and International Women's Day, we asked our employees to share with us who their biggest female role model is as well as what they do to champion women at SparkPost. Check out their inspiring responses!
PowerMTA Integration Experiences and SolutionsSparkPost
This document discusses PowerMTA integration and solutions. It covers PowerMTA building blocks like SMTP configuration and accounting. It also addresses integration challenges and solutions for email submission via SMTP, APIs, and pickup folders. Additionally, it provides examples of mail merge support, email generation flows, and processing inbound email and accounting data through log tailing, pickup folders, and pipes to external systems. Finally, it discusses potential features for PowerMTA 5.X like APIs for email, configuration, and delivering events through webhooks.
Get Ahead of the Game! Our Journey to Rebranding and SuccessSparkPost
In his welcome speech at the PMTA Summit 2019, Joal Barbehenn, SVP, Head of On Premises Software Vertical, presented the Business Unit's rebranding story.
Beyond the Norm: Email and the Innovation EthosSparkPost
Assumptions, acceptance of norms and bad best practices all conspire to keep email marketers tied to past practices. In this keynote, Kath Pay, CEO at Holistic Email Marketing, challenges you to begin to question all of these and inspires and equips you with a desire to innovate.
SparkPost's Director of Support Scott Habicht talks through the new features in PMTA 5.0 and explains how companies can make the most of the new version.
Amie Durr, VP of Product at SparkPost, together with Technical Product Manager Harold Vass, review the company's roadmap on deliverability and analytics, and discuss the future of the industry.
At the Wheel: An Evolution Story for Email Strategists (and Cyclists)SparkPost
SparkPost's CEO, Rich Harris, discussed how email has evolved, SparkPost's vision for the future of email and how the company's strategy works in tandem with the market's needs at the 2019 PowerMTA Summit. He threw in some thoughts on bicycles as well!
Webinar: Retail Banking - Optimizing the Customer Deposit LifecycleSparkPost
Marketing has become a dominant force for revenue growth in the battle for retail bank relationships. As consumer preferences shift rapidly to digital channels, there’s both a great challenge and opportunity to cultivate a customer journey that is more personalized and engaging than ever.
SparkPost is so excited to celebrate National Pride Month! It's a time where people come together to honor, celebrate and support the LGBTQ community. There are local events and celebrations that recognize how far we've come, in our country and across the globe. It's also a time to consider the work still needed to be done to support these individuals in our families, friendship circles and in the workplace.
How LendingTree is Growing Rapidly with Email and Customer Engagement SparkPost
These slides cover:
- The concept of the customer engagement maturity curve
- Tips for mastering 1:1 personalization
- Business factors that lead to a focus on growing email volumes
- Challenges you’ll encounter as sending volumes increase
- Best practices for optimizing high volume sending
On-Premises and Cloud - Putting the Pieces TogetherSparkPost
This document summarizes a panel discussion on hybrid email delivery solutions using SparkPost, PowerMTA, and Momentum. The panelists discuss SparkPost's offerings for on-premises and cloud implementations. They describe how a hybrid solution can provide failover redundancy, handle peak loads by offloading to the cloud, and provide analytics and engagement insights. Implementing a hybrid approach allows leveraging the strengths of both on-premises and cloud-based email delivery.
Scaling Your Product with Key Growth SurfacesSparkPost
The panelists discussed how companies can identify intrinsic growth surfaces within their own products to drive user engagement and business growth. A growth surface is a feature that encourages users to spend more time, broaden their use of new interactions, and engage with additional product lines. Drift uses its conversational marketing product as a growth surface through features like a test drive option, which results in 20% of leads scheduling meetings. The key is leveraging a product's natural interactions to deliver value to both current and prospective users.
When utilizing email to communicate with customers, organizations are faced with the challenge of sending to the right people, at the right time, and with the right message. All of these factors contribute to the success of email sending; however, it can be a massive undertaking to tie all these strategies together.
In this webinar replay with Ongage, we discuss the value of email segmentation and how it can be used to effectively engage with your users, improve deliverability and optimize internal operations.
State of Transactional Email 2018 (Benchmark report)SparkPost
Transactional email is a fundamental part of the customer experience and is increasing in importance in our app and cloud-based economy. It now plays a critical role in acquiring and retaining customers. To understand how organizations are utilizing this important communications channel SparkPost surveyed more than 1,800 organizations to gain insight into their use of transactional email. From overall strategy to specific tactics – such as sending patterns, deliverability, technologies used and people involved – the results are both eye-opening and instructive.
Learn how GDPR will affect your organization and how you can ensure that your company’s email sending is compliant with the new regulation. Taking action now will help you avoid the significant fines for noncompliance. Hear practical advice from SparkPost and 250ok on steps your organization should take to get your email sending practices in line with GDPR’s requirements.
High performance Serverless Java on AWS- GoTo Amsterdam 2024Vadym Kazulkin
Java is for many years one of the most popular programming languages, but it used to have hard times in the Serverless community. Java is known for its high cold start times and high memory footprint, comparing to other programming languages like Node.js and Python. In this talk I'll look at the general best practices and techniques we can use to decrease memory consumption, cold start times for Java Serverless development on AWS including GraalVM (Native Image) and AWS own offering SnapStart based on Firecracker microVM snapshot and restore and CRaC (Coordinated Restore at Checkpoint) runtime hooks. I'll also provide a lot of benchmarking on Lambda functions trying out various deployment package sizes, Lambda memory settings, Java compilation options and HTTP (a)synchronous clients and measure their impact on cold and warm start times.
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/temporal-event-neural-networks-a-more-efficient-alternative-to-the-transformer-a-presentation-from-brainchip/
Chris Jones, Director of Product Management at BrainChip , presents the “Temporal Event Neural Networks: A More Efficient Alternative to the Transformer” tutorial at the May 2024 Embedded Vision Summit.
The expansion of AI services necessitates enhanced computational capabilities on edge devices. Temporal Event Neural Networks (TENNs), developed by BrainChip, represent a novel and highly efficient state-space network. TENNs demonstrate exceptional proficiency in handling multi-dimensional streaming data, facilitating advancements in object detection, action recognition, speech enhancement and language model/sequence generation. Through the utilization of polynomial-based continuous convolutions, TENNs streamline models, expedite training processes and significantly diminish memory requirements, achieving notable reductions of up to 50x in parameters and 5,000x in energy consumption compared to prevailing methodologies like transformers.
Integration with BrainChip’s Akida neuromorphic hardware IP further enhances TENNs’ capabilities, enabling the realization of highly capable, portable and passively cooled edge devices. This presentation delves into the technical innovations underlying TENNs, presents real-world benchmarks, and elucidates how this cutting-edge approach is positioned to revolutionize edge AI across diverse applications.
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectorsDianaGray10
Join us to learn how UiPath Apps can directly and easily interact with prebuilt connectors via Integration Service--including Salesforce, ServiceNow, Open GenAI, and more.
The best part is you can achieve this without building a custom workflow! Say goodbye to the hassle of using separate automations to call APIs. By seamlessly integrating within App Studio, you can now easily streamline your workflow, while gaining direct access to our Connector Catalog of popular applications.
We’ll discuss and demo the benefits of UiPath Apps and connectors including:
Creating a compelling user experience for any software, without the limitations of APIs.
Accelerating the app creation process, saving time and effort
Enjoying high-performance CRUD (create, read, update, delete) operations, for
seamless data management.
Speakers:
Russell Alfeche, Technology Leader, RPA at qBotic and UiPath MVP
Charlie Greenberg, host
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfChart Kalyan
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
Discover top-tier mobile app development services, offering innovative solutions for iOS and Android. Enhance your business with custom, user-friendly mobile applications.
AppSec PNW: Android and iOS Application Security with MobSFAjin Abraham
Mobile Security Framework - MobSF is a free and open source automated mobile application security testing environment designed to help security engineers, researchers, developers, and penetration testers to identify security vulnerabilities, malicious behaviours and privacy concerns in mobile applications using static and dynamic analysis. It supports all the popular mobile application binaries and source code formats built for Android and iOS devices. In addition to automated security assessment, it also offers an interactive testing environment to build and execute scenario based test/fuzz cases against the application.
This talk covers:
Using MobSF for static analysis of mobile applications.
Interactive dynamic security assessment of Android and iOS applications.
Solving Mobile app CTF challenges.
Reverse engineering and runtime analysis of Mobile malware.
How to shift left and integrate MobSF/mobsfscan SAST and DAST in your build pipeline.
5th LF Energy Power Grid Model Meet-up SlidesDanBrown980551
5th Power Grid Model Meet-up
It is with great pleasure that we extend to you an invitation to the 5th Power Grid Model Meet-up, scheduled for 6th June 2024. This event will adopt a hybrid format, allowing participants to join us either through an online Mircosoft Teams session or in person at TU/e located at Den Dolech 2, Eindhoven, Netherlands. The meet-up will be hosted by Eindhoven University of Technology (TU/e), a research university specializing in engineering science & technology.
Power Grid Model
The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services.
Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability.
Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization.
What to expect
For the upcoming meetup we are organizing, we have an exciting lineup of activities planned:
-Insightful presentations covering two practical applications of the Power Grid Model.
-An update on the latest advancements in Power Grid -Model technology during the first and second quarters of 2024.
-An interactive brainstorming session to discuss and propose new feature requests.
-An opportunity to connect with fellow Power Grid Model enthusiasts and users.
The Department of Veteran Affairs (VA) invited Taylor Paschal, Knowledge & Information Management Consultant at Enterprise Knowledge, to speak at a Knowledge Management Lunch and Learn hosted on June 12, 2024. All Office of Administration staff were invited to attend and received professional development credit for participating in the voluntary event.
The objectives of the Lunch and Learn presentation were to:
- Review what KM ‘is’ and ‘isn’t’
- Understand the value of KM and the benefits of engaging
- Define and reflect on your “what’s in it for me?”
- Share actionable ways you can participate in Knowledge - - Capture & Transfer
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-EfficiencyScyllaDB
Freshworks creates AI-boosted business software that helps employees work more efficiently and effectively. Managing data across multiple RDBMS and NoSQL databases was already a challenge at their current scale. To prepare for 10X growth, they knew it was time to rethink their database strategy. Learn how they architected a solution that would simplify scaling while keeping costs under control.
Introduction of Cybersecurity with OSS at Code Europe 2024Hiroshi SHIBATA
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
Fueling AI with Great Data with Airbyte WebinarZilliz
This talk will focus on how to collect data from a variety of sources, leveraging this data for RAG and other GenAI use cases, and finally charting your course to productionalization.
"Choosing proper type of scaling", Olena SyrotaFwdays
Imagine an IoT processing system that is already quite mature and production-ready and for which client coverage is growing and scaling and performance aspects are life and death questions. The system has Redis, MongoDB, and stream processing based on ksqldb. In this talk, firstly, we will analyze scaling approaches and then select the proper ones for our system.
Main news related to the CCS TSI 2023 (2023/1695)Jakub Marek
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...DanBrown980551
This LF Energy webinar took place June 20, 2024. It featured:
-Alex Thornton, LF Energy
-Hallie Cramer, Google
-Daniel Roesler, UtilityAPI
-Henry Richardson, WattTime
In response to the urgency and scale required to effectively address climate change, open source solutions offer significant potential for driving innovation and progress. Currently, there is a growing demand for standardization and interoperability in energy data and modeling. Open source standards and specifications within the energy sector can also alleviate challenges associated with data fragmentation, transparency, and accessibility. At the same time, it is crucial to consider privacy and security concerns throughout the development of open source platforms.
This webinar will delve into the motivations behind establishing LF Energy’s Carbon Data Specification Consortium. It will provide an overview of the draft specifications and the ongoing progress made by the respective working groups.
Three primary specifications will be discussed:
-Discovery and client registration, emphasizing transparent processes and secure and private access
-Customer data, centering around customer tariffs, bills, energy usage, and full consumption disclosure
-Power systems data, focusing on grid data, inclusive of transmission and distribution networks, generation, intergrid power flows, and market settlement data
Monitoring and Managing Anomaly Detection on OpenShift.pdfTosin Akinosho
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
5. 5@SparkPost
Azure Functions
Pluses
• No infrastructure to maintain
• Scalable
- Infrequent workloads
- Hard to predict demand
• Quick to create, deploy, and update via a single point of deployment
• Developer friendly
- Easy learning curve for developers
- Interact with other Azure services such as CosmosDB, SQL database,
etc.
- C#, JavaScript, F#, Python, PowerShell, Bash are all supported
languages
Minuses
• Expense
• Hard to introspect and debug complex processes
• Not suitable for long running processes
9. 9@SparkPost
Azure Function Console Overview
Benefits
• No need to have Visual Studio.
• Download Visual Studio Project
• Easier to understand all the discreet bits
- Build, Integrate, Manage, Monitor
- Quickstart templates
• Easy testing interface
• Immediate deployment
Risks and Negatives
• Immediate deployment.
• Editor is not rich and helpful IDE.
• Easy to delete things without recovery path
15. 15@SparkPost
Azure Functions with Visual Studio
Benefits
• The same solid Visual Studio experience
• Local testing and debugging
- Step through code
- Avoid deploying untested code
• Source control integration
• Deploy from Visual Studio directly to Azure
Risks
• Relies on good developer habits
• Failure to account the for all the discreet parameters
Welcome to Azure Functions Webhook Interface
Serverless infrastructure is all the rage lately.It represents a paradigm shift that is on the same magnitude as the move from on-premises data center to the cloud.The shift to the cloud allowed us to stop worrying about physical hardware. This quickly extended to providing services for unwieldy things like databases, web servers, and queuing systems. Now the concepts have been extended to individual functions through services like Azure Functions.At the end of this presentation, you will be prepared to venture forth and start building serverless functions.
Thank you all for coming by, and before we dive into the fun stuff, let me tell you a bit about me.
In short, I am a lifelong computer geek. I have been fiddling about with computers for most of my life thanks to my father.These days, I get paid to be a Senior Site Reliability Engineer for SparkPost. I really enjoy the challenges of building, monitoring, and maintaining the complex infrastructure that provides our customers with a great experience and spectacular results.
One of the challenges facing IT organizations is controlling server cost. Amazon helped address this issue by starting AWS to monetize the excess capacity. That still meant virtual machines and the overhead that is associated with managing them does not drop significantly.
Maintaining IT infrastructure is hard and expensive. Keeping things patched, running, spare parts, and preventing bad actors all take resources. Most organizations are not able to hire specialist in all of these fields, so there is some amount of compromise in where resources are allocated. Accommodating bursts of activity and growth mean allocating a certain amount of resources to stand idle just waiting to be needed. This is largely a guessing game that everyone hopes they have made all the right assumptions about.Complex deployment systems are often the source of pain and suffering for many organizations. Trying to orchestrate the deployment of servers, operating systems, needed dependencies, and custom software is a hard problem. There are a plethora of tools and techniques to accomplish this management, but the advent of simple to use serverless functions and cloud services make things much simpler. You get simpler deployments and have to worry much less about the underlying bits and pieces, and instead you get to focus on your core business and competencies.There are some negatives to serverless architecture. Care must be taken in construction, as long running functions can become extremely expensive. It can also become difficult to introspect and debug complex processes. It is incumbent that developers think through how these complex systems will be instrumented to insure there is transparency into the systems. A negative that often causes great expense is the long running process. Developers need to rethink the way things are architected when dealing with serverless functions. Short, quick executing functions are the best. Breaking apart complex processes is actually more cost effective and efficient than having a large, long running process. This is a bit counter-intuitive to developers not experienced with scalable enterprise architectures.
What are these webhooks you speak of and what can they do for me?
In simplest terms, webhooks are event driven interactions. Many products support webhooks. By employing webhooks, you can create rich, complex systems that provide meaningful interactions for users.Webhooks are also useful for monitoring activity, creating notifications, and aggregating data for analysis.
In some cases, the console for cloud services is in some cases an afterthought leaving a lot to be desired.This is not the case for the Azure console.
The Azure Function App console is extremely useful, and something every developer should spend some time studying. It is not just an interface, but it is a great guide to how the bits of Azure interact. Many of the deep details of what can be done within Azure can be unearthed inside of the console.One of the best things is that you can develop new functions without having Visual Studio, and if you do have Visual Studio, you can download the created function as a solution file.By working in the console, you can get a better understanding of all the discreet things that can be done with Azure Functions, and the quickstart templates make it very easy to get a basic framework that you just need to flesh out with the details of your use-case.The test interface makes it easy to test a function and see that it is working properly.When you update a function in the console, it is immediately deployed. No need to build any sort of deployment infrastructure. This can be very risky though since if you make a mistake in the console, it has immediate impact.It is challenging to know when you have made a mistake since there is not the rich and helpful IDE experience that you get when working with Visual Studio.Once you delete something in the Azure Function console, it is gone forever. There is no backup or undo.
Let’s dive right in and build our first function.
I like to learn things the first time in the console to help me understand the requirements and general structure. I tend to be quite visual, so seeing how the pieces relate following the interface is very helpful to me.Most of what is listed here in the Create dialog is not critical to creating a function. You really just need to choose a name. Everything else can be automatic. Once you get the basics down, you can dive into the fine details of Resource Groups, Hosting Plans, and Storage.For now, just know that a Resource Group is a way to provision, manage, control access, and monitor a group of resources. Compartmentalization of resources helps control costs, assist in accounting, and limit noisy neighbor effects.The Pin to dashboard check box is unchecked by default. It is a good idea to tick the box to get your newly created function to show up on the dashboard. Once you are more familiar with the interface, you can adjust your dashboard to suit your needs, but to start, I like to easily be able to find what I just created without rooting around for it.
Once creation is completed, you can see all the details of the Function App, and you can manage your Function App. A few things to take note of here. The details of the app, including the access URL is shown. Downloading the app content will get you a Visual Studio solution file, and you can add Functions, Proxies, and Slots. Of special note are the Function App settings and the Application settings.
These links will open tabs for their respective activities. Within each, you can control many of the fine details of the Function App. You can change the underlying platform for 32-bit or 64-bit, tweak access controls, manage deployment options, explore resources, and even access a development console.
The Function app settings are important if you want to set some limitations on the usage of the app or avoid accidental updates.
That lays the foundation, but the real magic happens when you click the plus next to the Functions in the console and create the actual code that is doing the work. Just give the function a name, and start to lay out the code within the framework selected.The construction interface gives you an editor, a log, and click on the side tab to pop out the test pane, if it happens to be collapsed.The coding itself is straightforward C# just like Visual Studio.Note that in the testing pane you can create complex query strings, add headers, and edit your request body.You can also grab the URL of the function if you want to test with an external tool.
Now that we have a taste of things in the console, let’s move on from the common experience to the extraordinary.
Once you have added the Azure Workload to your installation of Visual Studio 2017, you have access to the full power of an operational cloud!You get the familiar Visual Studio experience while able to local test and debug your Function Apps. As a good practice, you can put your Azure solutions in the source control of your choice.Unlike tradition deployment situations where you need Team Services or some other deployment method, you can deploy directly from Visual Studio to an Azure account, assuming you have the account info and deployment permissions.This leads to the same risk of the console that you could deploy untested code directly to production.The other thing that is less obvious in Visual Studio is all the discreet parameters that you can control with Azure Function Apps. You can still access all of these things and control them, but it takes a little more effort to make the tweaks in Visual Studio. A little exploring and time, and it is completely manageable through the host.json file.
When creating a new project, you should see the option for Visual C#, Cloud, and Azure Functions. Give it a name, and click OK. Just like creating any other Visual Studio solution.This is where there is a slight disconnect here, as this is what the console calls a Function App. Small difference, but it can be a little confusing to some.
Right click the project folder, select New Item, Azure Function.This is equivalent to clicking the plus in the console and creating a new function within your Function app.
Just like the console, now we have a template for the function.Just start writing your code.When you are ready to test, press F5, and you will be able to debug locally just like you would other Visual Studio projects.
The first time you try to debug an Azure Function locally, you will be prompted to install the Azure CLI.After that, it will go directly into starting the debugging console.
The debugger console shows every request that is triggered. This can be useful for validating behavior and insure the function is behaving as expected.You can test the function through the URL in the debugging console using your favorite way to interact with a REST API. Personally, I prefer using curl through Bash for most things, and when things get complex, Postman works very well for interacting with RESTful endpoints.
Right click the project node in Solution Explorer, choose Publish, and you will see this screen.A little tip here is that even if you did not download an existing solution from the console, you can update an existing function through Visual Studio.
Wait for the creation to finish, and you will see the publish details in Visual Studio. You get essentially the same details as you get in the console.
Assuming you have connected your Visual Studio to your Azure account, you just need to create the new function.Name the app, choose the description, the resource group, and click Create.It is that simple.Note that you can have multiple accounts attached, for example if you separate development and production.