Faster Time to Science - Scaling BioMedical Research in the Cloud with SciOps...Amazon Web Services
Medical Researchers are constantly looking for ways to be able to conduct more experiments, innovate at a faster rate and derive meaningful research outcomes more quickly. One of the major barriers to achieving this is long processing times due to giant datasets. A combined industry and research partnership, large-scale on-demand compute and the cloud has been key to making inroads to solving this very common challenge.
DiUS and the Walter Eliza Hall Institute of Medical Research (WEHI) have been working on approaches to accelerate the capture, processing and analysis of bioimagery and microscopy data used in the research labs at WEHI. In this talk, Pavi and Lachlan will share a case study starting with a background on microscope development and a synopsis of state-of-the-art microscopy techniques requiring large scale compute. The session will then launch into a discussion of scaling complex image analysis using Fiji, a bio-science image analysis package and dealing with ever-growing bioimaging datasets.
You will learn about the development of tailored high performance compute (HPC) platforms on AWS to enable this kind of research as well as the 'convention-over-configuration' framework developed by DiUS as a repeatable solution. Lower level technical considerations around network integration, efficient data movement and cluster compute approaches using CfnCluster on AWS will also be discussed in detail.
Speakers: Lachlan Whitehead, PhD, BioImage Analyst and Microscopy Walter and Eliza Hall Institute of Medical Research & Pavi De Alwis, Snr.Software Engineer, DiUS
Currently, a breadth of AWS training opportunities are available worldwide, both led by AWS and through community-driven training platforms. In this session, community leaders sort through the different training resources, discuss the resources they used to help them become AWS experts, and explain how different training solutions can complement one another.
AWS re:Invent 2016: Serverless Computing Patterns at Expedia (SVR306) )Amazon Web Services
In the middle of 2015, Expedia started using AWS Lambda for serverless computing. We built boilerplate templates in Node.js, Java, and Python so development teams could build and deploy serverless applications into AWS. Currently, we have 300 AWS Lambda functions processing 40 million invocations per day.
In this session, we will discuss how development teams use boilerplate templates to create serverless applications with Amazon API Gateway and AWS Lambda and how they deploy them to AWS. We will cover patterns, architectural design choices, and the benefits --- like cost, scale, availability, and operations --- of running serverless applications.
Chris Munns, DevOps @ Amazon: Microservices, 2 Pizza Teams, & 50 Million Depl...TriNimbus
Keynote presentation from Vancouver's 2016 Canadian Executive DevOps & Cloud Summit on Thursday, May 5th.
Speaker: Chris Munns, Business Development Manager, DevOps at Amazon Web Services
Title: DevOps @ Amazon: Microservices, 2 Pizza Teams, & 50 Million Deploys a Year
Application Delivery Patterns for Developers - Technical 401Amazon Web Services
Every developer has gone through the frustration of creating new features, fixing bugs, or refactoring beautiful code, and then wait for it to reach the promise land of production. Come and learn how to get your changes in the hands of your customers with more speed, reliability, security and quality.
We will dive deep into architectures for continuous delivery pipelines, apply lean principles, and build intelligence into your pipeline.
Speaker: Shiva Narayanaswamy, Solutions Architect, Amazon Web Services
Featured Customer - REA Group
Faster Time to Science - Scaling BioMedical Research in the Cloud with SciOps...Amazon Web Services
Medical Researchers are constantly looking for ways to be able to conduct more experiments, innovate at a faster rate and derive meaningful research outcomes more quickly. One of the major barriers to achieving this is long processing times due to giant datasets. A combined industry and research partnership, large-scale on-demand compute and the cloud has been key to making inroads to solving this very common challenge.
DiUS and the Walter Eliza Hall Institute of Medical Research (WEHI) have been working on approaches to accelerate the capture, processing and analysis of bioimagery and microscopy data used in the research labs at WEHI. In this talk, Pavi and Lachlan will share a case study starting with a background on microscope development and a synopsis of state-of-the-art microscopy techniques requiring large scale compute. The session will then launch into a discussion of scaling complex image analysis using Fiji, a bio-science image analysis package and dealing with ever-growing bioimaging datasets.
You will learn about the development of tailored high performance compute (HPC) platforms on AWS to enable this kind of research as well as the 'convention-over-configuration' framework developed by DiUS as a repeatable solution. Lower level technical considerations around network integration, efficient data movement and cluster compute approaches using CfnCluster on AWS will also be discussed in detail.
Speakers: Lachlan Whitehead, PhD, BioImage Analyst and Microscopy Walter and Eliza Hall Institute of Medical Research & Pavi De Alwis, Snr.Software Engineer, DiUS
Currently, a breadth of AWS training opportunities are available worldwide, both led by AWS and through community-driven training platforms. In this session, community leaders sort through the different training resources, discuss the resources they used to help them become AWS experts, and explain how different training solutions can complement one another.
AWS re:Invent 2016: Serverless Computing Patterns at Expedia (SVR306) )Amazon Web Services
In the middle of 2015, Expedia started using AWS Lambda for serverless computing. We built boilerplate templates in Node.js, Java, and Python so development teams could build and deploy serverless applications into AWS. Currently, we have 300 AWS Lambda functions processing 40 million invocations per day.
In this session, we will discuss how development teams use boilerplate templates to create serverless applications with Amazon API Gateway and AWS Lambda and how they deploy them to AWS. We will cover patterns, architectural design choices, and the benefits --- like cost, scale, availability, and operations --- of running serverless applications.
Chris Munns, DevOps @ Amazon: Microservices, 2 Pizza Teams, & 50 Million Depl...TriNimbus
Keynote presentation from Vancouver's 2016 Canadian Executive DevOps & Cloud Summit on Thursday, May 5th.
Speaker: Chris Munns, Business Development Manager, DevOps at Amazon Web Services
Title: DevOps @ Amazon: Microservices, 2 Pizza Teams, & 50 Million Deploys a Year
Application Delivery Patterns for Developers - Technical 401Amazon Web Services
Every developer has gone through the frustration of creating new features, fixing bugs, or refactoring beautiful code, and then wait for it to reach the promise land of production. Come and learn how to get your changes in the hands of your customers with more speed, reliability, security and quality.
We will dive deep into architectures for continuous delivery pipelines, apply lean principles, and build intelligence into your pipeline.
Speaker: Shiva Narayanaswamy, Solutions Architect, Amazon Web Services
Featured Customer - REA Group
(WEB301) Operational Web Log Analysis | AWS re:Invent 2014Amazon Web Services
Log data contains some of the most valuable raw information you can gather and analyze about your infrastructure and applications. Amid the mess of confusing lines of seemingly random text can be hints about performance, security, flaws in code, user access patterns, and other operational data. Without the proper tools, finding insights in these logs can be like searching for a hay-colored needle in a haystack. In this session you learn what practices and patterns you can easily implement that can help you better understand your log files. You see how you can customize web logs to add more information to them, how to digest logs from around your infrastructure, and how to analyze your log files in near real time.
AWS re:Invent 2016: Moving Mission Critical Apps from One Region to Multi-Reg...Amazon Web Services
In gaming, low latencies and connectivity are bare minimum expectations users have while playing online on PlayStation Network. Alex and Dustin share key architectural patterns to provide low latency, multi-region services to global users. They discuss the testing methodologies and how to programmatically map out a large dependency multi-region deployment with data-driven techniques. The patterns shared show how to adapt to changing bottlenecks and sudden, several million request spikes. You’ll walk away with several key architectural patterns that can service users at global scale while being mindful of costs.
AWS re:Invent 2016: Deploying and Managing .NET Pipelines and Microsoft Workl...Amazon Web Services
In this session, we’ll look at the AWS services that customers are using to build and deploy Microsoft-based solutions that use technologies like Windows, .NET, SQL Server, and PowerShell. We’ll start by showing you how to build a Windows-based CI/CD pipeline on AWS using AWS CodeDeploy, AWS CodePipeline, AWS CloudFormation, and PowerShell using an AWS Quick Start. We’ll also cover best practices for how you can create templates that let you automatically deploy ready-to-use Windows products by leveraging services and tools like AWS CloudFormation, PowerShell, and Git. Woot, an online retailer for electronics, will share how it moved from using a complex mix of custom PowerShell code for its DevOps processes to using services like Amazon EC2 Simple Systems Manager (SSM), AWS CodeDeploy, and AWS Directory Service. This migration eliminated the need for complex PowerShell scripts and reduced the operational complexity of performing operational tasks like renaming servers, joining domains, and securely handling keys.
ENT310 Microservices? Dynamic Infrastructure? - Adventures in Keeping Your Ap...Amazon Web Services
Keeping an application running at scale can be a daunting task. When do you need to add more capacity? Larger databases? Additional servers? These questions get harder as the complexity of your application grows. Microservice based architectures and cloud-based dynamic infrastructures are technologies that help you keep your application running with high availability, even during times of extreme scaling. We will discuss some of the best practices we’ve learned working with New Relic customers on how you can manage your applications running at scale, and how technologies such as microservices and dynamic infrastructure can help you with this challenge. This session is brought to you by AWS Summit San Francisco Platinum Sponsor New Relic.
Getting from scale to profitability, the question is: How to run lean and continue to grow revenue? How to get to profitability? This webinar, the last installment of a four-part webinar series for start-ups, will focus on how to run lean.
Amazon WorkSpaces is a secure, managed, virtual desktop service running on the AWS cloud. The service helps organizations support a modern mobile workforce, improve information security, and save money with a pay-as-you-go model. In this session, we'll cover how cloud desktops can benefit your organization, what's new with Amazon WorkSpaces, and some of the top technical considerations like user identity and access management, VPC design, network traffic flow, and application delivery. This session is for IT professionals and business decision makers interested in learning how to simplify desktop management and productivity for their organizations.
Navigating PCI Compliance in the Cloud (SEC206) | AWS re:Invent 2013Amazon Web Services
People assume that implementing the Payment Card Industry Data Security Standard (PCI DSS) on AWS is more difficult than in a traditional data center, but that's simply not true. Come learn how PaymentSpring implemented a PCI DSS level 1 compliant gateway running entirely on AWS. Learn how they designed the system to make PCI DSS validation easier, what they could depend on AWS to provide, and what they still had to take care of. The session covers some of the things PaymentSpring did to significantly reduce costs and increase the overall security of the system. But most importantly, learn why it's easier to maintain compliance over time. Jesse Angell, CTO of PaymentSpring, shares his first-hand experiences with implementing PCI DSS on AWS at his organization.
Crunch Your Data in the Cloud with Elastic Map Reduce - Amazon EMR HadoopAdrian Cockcroft
A introductory discussion of cloud computing and capacity planning implications is followed by a step by step guide to running a Hadoop job in EMR, and finally a discussion of how to write your own Hadoop queries.
Key Steps for Setting up your AWS Journey for Success - BusinessAmazon Web Services
When building anything, it's longevity begins with establishing a solid foundation, on AWS you will need to ensure your application is built on top of best practices. We will help you make the best use out of AWS Support and Training, correct account set up and strategies to help you optimise your AWS spend.
Speakers: David Ly, Account Manager and Nathan Besh, Technical Account Manager, Amazon Web Services
Featured Customer - Domain
AWS re:Invent 2016: How News UK Centralized Cloud Governance Through Policy M...Amazon Web Services
When you run a complex AWS environment with thousands of Amazon EC2 instances, more than half a petabyte of object storage, and support the largest daily newspapers in the UK, you need a world-class cloud management strategy. For companies like News Corp, implementing policies that automate infrastructure schedules, right-size workloads, and manage and modify reservations is critical. As you scale your cloud infrastructure, defining centralized governance rules while enabling decentralized management is key to running an optimized cloud.
This session is designed for advanced operations, infrastructure, and engineering teams to improve/deploy optimization strategies. It covers the five best cloud management practices, including automating Reserved Instance modifications, setting policies to ensure proper tagging, and scheduling lights-on/lights-off policies. Session sponsored by CloudHealth Technologies.
With AWS Lambda, you can easily build scalable microservices for mobile, web, and IoT applications or respond to events from other AWS services without managing infrastructure. In this session, you’ll see demonstrations and hear more about newly launched features. We’ll show you how to use Lambda to build web, mobile, or IoT backends and voice-enabled apps, and we'll show you how to extend both AWS and third party services by triggering Lambda functions. We’ll also provide productivity and performance tips for getting the most out of your Lambda functions and show how cloud native architectures use Lambda to eliminate “cold servers” and excess capacity without sacrificing scalability or responsiveness.
(SPOT205) 5 Lessons for Managing Massive IT Transformation ProjectsAmazon Web Services
Choice Hotels is undertaking a multiyear, $20 million project to recreate our core business engines on AWS. In trying to approach this complex undertaking, we determined that the project itself is a system too. You can apply principles of good architecture and design work in how you approach the project structure and management. Come to this talk by Choice Hotels’ CTO to learn five key lessons and 20 concrete takeaways that you can implement today to help your AWS projects succeed.
SEC307 Automating Security and Compliance Response in the CloudAmazon Web Services
"Your enterprise cloud requires oversight and continuous management that can be more effectively addressed through security and compliance automation. In this session, learn how to ensure your cloud is compliant with standards critical to your business, and how you can replace outdated processes and tools that slow down delivery of new (and secure) code. We will look closely at issues that can make your environment vulnerable, like dead accounts and other inappropriate access-enabling risks, and misconfigurations that can create unwanted access points. Our session explains how security and compliance automation is used to eradicate these types of issues and many more by leveraging Amazon Simple Notification Service (SNS) and AWS Lambda workflows, integrations with DevOps tools and more.
This session is brought to you by AWS Summit New York City sponsor, Evident.io."
AWS re:Invent 2016: Building and Growing a Successful AWS User Group (DCS203)Amazon Web Services
Our panel of experts lead AWS user groups in San Francisco, Yokohama, Munich, and Shanghai. In this session they share the stories behind how their groups began, list best practices for sustaining a technical meetup over time, and offer advice for AWS enthusiasts who are considering starting up a new user group in their city.
AWS re:Invent 2016: How Pitney Bowes is transforming their business in the cl...Amazon Web Services
Pitney Bowes is reinventing its business based on a SaaS and a cloud-based model to deliver services for their clients globally centered on the Pitney Bowes Commerce Cloud. The Pitney Bowes Commerce Cloud is a commerce enabler, providing access to solutions, analytics and APIs across the full commerce continuum with speed and agility to help clients identify customers, locate opportunities, enable communications, power shipping from anywhere to everywhere, and manage payments. During this session, the Pitney Bowes team will discuss the strategy behind the Commerce Cloud, how we accelerated the speed of innovation and creation of new product solutions by taking full advantage of the AWS platform, from Cloud Infrastructure Services (Compute, Big Data, Storage and Content Delivery, Databases and Networking) , to Analytics and Internet of Things Services. Pitney Bowes’ applications are deployed in Docker containers using AWS Elastic Beanstalk, and are utilizing S3, Amazon RDS, SQS, SNS, and ElastiCache. CloudWatch and CloudFormation are being used to manage the solutions that Pitney Bowes’ brings to the market. Additionally, the Pitney Bowes Data and Analytics platform is powered by Elastic Map Reduce (EMR), Spark, Aurora, DynamoDB and Postgres.
The session will also discuss Pitney Bowes’ partnership with AWS to deploy Pitney Bowes’ APIs for Location Intelligence via the AWS Marketplace, allowing developers, customers and partners to access innovative location intelligence data services in an easy to consume and highly reliable manner.
Considerations for Operating An OpenStack CloudMark Voelker
My talk from All Things Open 2014
Over the past four years, OpenStack has become a widely adopted cloud operating system. Cloud computing has made many tasks like creating new servers and networks easy for end users by creating abstractions above the infrastructure. However, cloud operators need to maintain not only the cloud operating system itself, but all of the underpinning systems beneath it. The challenges of managing a set of distributed systems isn’t small, but with proper tooling is well within reach. This talk will discuss considerations for cloud operators such as logging, storage, monitoring, high availability, configuration management with a focus on OpenStack clouds with a focus on open source solutions for common issues encountered when operating an OpenStack cloud. We’ll consider data gathered from the community and discuss “day 1″ and “day 2″ concerns as well as established patterns and technology choices among OpenStack deployers today.
(WEB301) Operational Web Log Analysis | AWS re:Invent 2014Amazon Web Services
Log data contains some of the most valuable raw information you can gather and analyze about your infrastructure and applications. Amid the mess of confusing lines of seemingly random text can be hints about performance, security, flaws in code, user access patterns, and other operational data. Without the proper tools, finding insights in these logs can be like searching for a hay-colored needle in a haystack. In this session you learn what practices and patterns you can easily implement that can help you better understand your log files. You see how you can customize web logs to add more information to them, how to digest logs from around your infrastructure, and how to analyze your log files in near real time.
AWS re:Invent 2016: Moving Mission Critical Apps from One Region to Multi-Reg...Amazon Web Services
In gaming, low latencies and connectivity are bare minimum expectations users have while playing online on PlayStation Network. Alex and Dustin share key architectural patterns to provide low latency, multi-region services to global users. They discuss the testing methodologies and how to programmatically map out a large dependency multi-region deployment with data-driven techniques. The patterns shared show how to adapt to changing bottlenecks and sudden, several million request spikes. You’ll walk away with several key architectural patterns that can service users at global scale while being mindful of costs.
AWS re:Invent 2016: Deploying and Managing .NET Pipelines and Microsoft Workl...Amazon Web Services
In this session, we’ll look at the AWS services that customers are using to build and deploy Microsoft-based solutions that use technologies like Windows, .NET, SQL Server, and PowerShell. We’ll start by showing you how to build a Windows-based CI/CD pipeline on AWS using AWS CodeDeploy, AWS CodePipeline, AWS CloudFormation, and PowerShell using an AWS Quick Start. We’ll also cover best practices for how you can create templates that let you automatically deploy ready-to-use Windows products by leveraging services and tools like AWS CloudFormation, PowerShell, and Git. Woot, an online retailer for electronics, will share how it moved from using a complex mix of custom PowerShell code for its DevOps processes to using services like Amazon EC2 Simple Systems Manager (SSM), AWS CodeDeploy, and AWS Directory Service. This migration eliminated the need for complex PowerShell scripts and reduced the operational complexity of performing operational tasks like renaming servers, joining domains, and securely handling keys.
ENT310 Microservices? Dynamic Infrastructure? - Adventures in Keeping Your Ap...Amazon Web Services
Keeping an application running at scale can be a daunting task. When do you need to add more capacity? Larger databases? Additional servers? These questions get harder as the complexity of your application grows. Microservice based architectures and cloud-based dynamic infrastructures are technologies that help you keep your application running with high availability, even during times of extreme scaling. We will discuss some of the best practices we’ve learned working with New Relic customers on how you can manage your applications running at scale, and how technologies such as microservices and dynamic infrastructure can help you with this challenge. This session is brought to you by AWS Summit San Francisco Platinum Sponsor New Relic.
Getting from scale to profitability, the question is: How to run lean and continue to grow revenue? How to get to profitability? This webinar, the last installment of a four-part webinar series for start-ups, will focus on how to run lean.
Amazon WorkSpaces is a secure, managed, virtual desktop service running on the AWS cloud. The service helps organizations support a modern mobile workforce, improve information security, and save money with a pay-as-you-go model. In this session, we'll cover how cloud desktops can benefit your organization, what's new with Amazon WorkSpaces, and some of the top technical considerations like user identity and access management, VPC design, network traffic flow, and application delivery. This session is for IT professionals and business decision makers interested in learning how to simplify desktop management and productivity for their organizations.
Navigating PCI Compliance in the Cloud (SEC206) | AWS re:Invent 2013Amazon Web Services
People assume that implementing the Payment Card Industry Data Security Standard (PCI DSS) on AWS is more difficult than in a traditional data center, but that's simply not true. Come learn how PaymentSpring implemented a PCI DSS level 1 compliant gateway running entirely on AWS. Learn how they designed the system to make PCI DSS validation easier, what they could depend on AWS to provide, and what they still had to take care of. The session covers some of the things PaymentSpring did to significantly reduce costs and increase the overall security of the system. But most importantly, learn why it's easier to maintain compliance over time. Jesse Angell, CTO of PaymentSpring, shares his first-hand experiences with implementing PCI DSS on AWS at his organization.
Crunch Your Data in the Cloud with Elastic Map Reduce - Amazon EMR HadoopAdrian Cockcroft
A introductory discussion of cloud computing and capacity planning implications is followed by a step by step guide to running a Hadoop job in EMR, and finally a discussion of how to write your own Hadoop queries.
Key Steps for Setting up your AWS Journey for Success - BusinessAmazon Web Services
When building anything, it's longevity begins with establishing a solid foundation, on AWS you will need to ensure your application is built on top of best practices. We will help you make the best use out of AWS Support and Training, correct account set up and strategies to help you optimise your AWS spend.
Speakers: David Ly, Account Manager and Nathan Besh, Technical Account Manager, Amazon Web Services
Featured Customer - Domain
AWS re:Invent 2016: How News UK Centralized Cloud Governance Through Policy M...Amazon Web Services
When you run a complex AWS environment with thousands of Amazon EC2 instances, more than half a petabyte of object storage, and support the largest daily newspapers in the UK, you need a world-class cloud management strategy. For companies like News Corp, implementing policies that automate infrastructure schedules, right-size workloads, and manage and modify reservations is critical. As you scale your cloud infrastructure, defining centralized governance rules while enabling decentralized management is key to running an optimized cloud.
This session is designed for advanced operations, infrastructure, and engineering teams to improve/deploy optimization strategies. It covers the five best cloud management practices, including automating Reserved Instance modifications, setting policies to ensure proper tagging, and scheduling lights-on/lights-off policies. Session sponsored by CloudHealth Technologies.
With AWS Lambda, you can easily build scalable microservices for mobile, web, and IoT applications or respond to events from other AWS services without managing infrastructure. In this session, you’ll see demonstrations and hear more about newly launched features. We’ll show you how to use Lambda to build web, mobile, or IoT backends and voice-enabled apps, and we'll show you how to extend both AWS and third party services by triggering Lambda functions. We’ll also provide productivity and performance tips for getting the most out of your Lambda functions and show how cloud native architectures use Lambda to eliminate “cold servers” and excess capacity without sacrificing scalability or responsiveness.
(SPOT205) 5 Lessons for Managing Massive IT Transformation ProjectsAmazon Web Services
Choice Hotels is undertaking a multiyear, $20 million project to recreate our core business engines on AWS. In trying to approach this complex undertaking, we determined that the project itself is a system too. You can apply principles of good architecture and design work in how you approach the project structure and management. Come to this talk by Choice Hotels’ CTO to learn five key lessons and 20 concrete takeaways that you can implement today to help your AWS projects succeed.
SEC307 Automating Security and Compliance Response in the CloudAmazon Web Services
"Your enterprise cloud requires oversight and continuous management that can be more effectively addressed through security and compliance automation. In this session, learn how to ensure your cloud is compliant with standards critical to your business, and how you can replace outdated processes and tools that slow down delivery of new (and secure) code. We will look closely at issues that can make your environment vulnerable, like dead accounts and other inappropriate access-enabling risks, and misconfigurations that can create unwanted access points. Our session explains how security and compliance automation is used to eradicate these types of issues and many more by leveraging Amazon Simple Notification Service (SNS) and AWS Lambda workflows, integrations with DevOps tools and more.
This session is brought to you by AWS Summit New York City sponsor, Evident.io."
AWS re:Invent 2016: Building and Growing a Successful AWS User Group (DCS203)Amazon Web Services
Our panel of experts lead AWS user groups in San Francisco, Yokohama, Munich, and Shanghai. In this session they share the stories behind how their groups began, list best practices for sustaining a technical meetup over time, and offer advice for AWS enthusiasts who are considering starting up a new user group in their city.
AWS re:Invent 2016: How Pitney Bowes is transforming their business in the cl...Amazon Web Services
Pitney Bowes is reinventing its business based on a SaaS and a cloud-based model to deliver services for their clients globally centered on the Pitney Bowes Commerce Cloud. The Pitney Bowes Commerce Cloud is a commerce enabler, providing access to solutions, analytics and APIs across the full commerce continuum with speed and agility to help clients identify customers, locate opportunities, enable communications, power shipping from anywhere to everywhere, and manage payments. During this session, the Pitney Bowes team will discuss the strategy behind the Commerce Cloud, how we accelerated the speed of innovation and creation of new product solutions by taking full advantage of the AWS platform, from Cloud Infrastructure Services (Compute, Big Data, Storage and Content Delivery, Databases and Networking) , to Analytics and Internet of Things Services. Pitney Bowes’ applications are deployed in Docker containers using AWS Elastic Beanstalk, and are utilizing S3, Amazon RDS, SQS, SNS, and ElastiCache. CloudWatch and CloudFormation are being used to manage the solutions that Pitney Bowes’ brings to the market. Additionally, the Pitney Bowes Data and Analytics platform is powered by Elastic Map Reduce (EMR), Spark, Aurora, DynamoDB and Postgres.
The session will also discuss Pitney Bowes’ partnership with AWS to deploy Pitney Bowes’ APIs for Location Intelligence via the AWS Marketplace, allowing developers, customers and partners to access innovative location intelligence data services in an easy to consume and highly reliable manner.
Considerations for Operating An OpenStack CloudMark Voelker
My talk from All Things Open 2014
Over the past four years, OpenStack has become a widely adopted cloud operating system. Cloud computing has made many tasks like creating new servers and networks easy for end users by creating abstractions above the infrastructure. However, cloud operators need to maintain not only the cloud operating system itself, but all of the underpinning systems beneath it. The challenges of managing a set of distributed systems isn’t small, but with proper tooling is well within reach. This talk will discuss considerations for cloud operators such as logging, storage, monitoring, high availability, configuration management with a focus on OpenStack clouds with a focus on open source solutions for common issues encountered when operating an OpenStack cloud. We’ll consider data gathered from the community and discuss “day 1″ and “day 2″ concerns as well as established patterns and technology choices among OpenStack deployers today.
Orient Me is the first Connections service which is built on the new Connections Pink stack. Nico will talk about the installation, integration and administration of Orient Me. He will also provide useful insights around the used backend tools. Walk away with knowledge how to successfully run Orient Me in your own Connections environment!
Complex realtime event analytics using BigQuery @Crunch WarmupMárton Kodok
Complex event analytics solutions require massive architecture, and Know-How to build a fast real-time computing system. Google BigQuery solves this problem by enabling super-fast, SQL-like queries against append-only tables, using the processing power of Google’s infrastructure.In this presentation we will see how Bigquery solves our ultimate goal: Store everything accessible by SQL immediately at petabyte-scale. We will discuss some common use cases: funnels, user retention, affiliate metrics.
Game On! (@gameontext – http://game-on.org) is an awesome throwback text-based adventure built with microservices. Completely open source, it enables everyone to choose their own adventure to learn about microservices concepts while extending the game. One of the core services is the Map, which maintains a two-dimensional map containing all the registered rooms. The Map started with a document store as a back end, but as the Map changed over time, tombstones started to accrue. And then people started to ask how to manage three dimensions, and dragons appeared. Come to this session to find out why the decision was made to change the NoSQL back end, how it was done, and the result of the change with a new NoSQL API (http://jnosql.org/).
Microservices mit Java EE - am Beispiel von IBM LibertyMichael Hofmann
Viele Unternehmen versprechen sich derzeit einiges vom aktuellen Architektur-Trend: Microservices. Unter anderem verbinden sie damit die Hoffnung bestimmte Architektur-Probleme in den Griff zu bekommen: Stichwort Monolith. Dabei stellen sich Entwicklungsorganisationen mit einem Fokus auf Java EE-Technologien die Frage, ob und wie sie mit ihren Java EE-Mitteln optimal Microservices implementieren können. Im Gegenzug erweitern oder verändern Java EE-Hersteller ihre Produkte, um den Trend der Microservices gerecht zu werden. Ziel des Vortrages soll es sein, am Beispiel von IBM's WebSphere Liberty Profile Server zu verdeutlichen, welche Vorteile bzw. Nachteile der Java EE-Ansatz bringen kann. Dabei wird nicht nur auf technologische Aspekte, sondern auch auf organisatorische Problemstellungen eingegangen. Themen wie DevOps und Continous Delivery werden dabei am Rande auch betrachtet. Abgerundet wird das Ganze mit Hinweisen auf bekannte Fallbeispiele, wie z.B. Netflix, um weitere Denkanstöße zu geben.
In front-end software development it's still rare that data is collected on the client side besides some analytics data where developers usually don't have access to.
Imagine what you can do when you have front-end log data, you see how many ajax calls hitting your servers and you finally know whether the single page application is used like you expected or not. I will briefly talk about projects I was part of where we used these kind of data to improve our product and surprisingly reduced AWS costs by changing front-end code.
https://docs.google.com/presentation/d/1kGK8P7Ll2H4Z_1UUdBneAbNzUEDmpj8g2Mxj_Z-F5u8/pub?start=false&loop=false&delayms=3000
Performance monitoring and call tracing in microservice environmentsMartin Gutenbrunner
Performance analysis can easily be done with on-board tools of nearly any programming language. In microservice environments, the real challenge is not in single, high-performing services, but in resiliently running a complex ecosystem of many services.This talk will introduce open-source tools for analysis and call tracing. Concluding, we will briefly get to know Dynatrace Ruxit - a commercial alternative. After this session, the audience will know about how to get started in performance analysis and call-tracing and some according tools.
Automated Infrastructure Security: Monitoring using FOSSSonatype
Madhu Akula, Automation Ninja
We can see attacks happening in real time using a dashboard. By collecting logs from various sources we will monitor & analyse. Using data gleaned from the logs, we can apply defensive rules against the attackers. We will use AWS for managing and securing the infrastructure discussed in our talk.
For most network engineers who monitor the perimeter for malicious content, it is very important to respond to an imminent threat originating from outside the boundaries of their network. Having to crunch through all the logs that the various devices (firewalls, routers, security appliances etc.) spit out, correlating that data and in real time making the right choices can prove to be a nightmare. Even with the solutions already available in the market.
As I have experienced this myself, as part of the Internal DevOps and Incident Response Teams, in several cases, I would want to create a space for interested folks to design, build, customise and deploy their very own FOSS based centralised visual attack monitoring dashboard. This setup would be able to perform real time analysis using the trusted ELK stack and visually denote what popular attack hotspots exist on a network.
CPU and RAM costs continue to plummet. Multi-core systems are ubiquitous. Writing code is easier than it has ever been. Why, then, is it still so darn hard to make a scalable system?
The impact of emerging IoT Technology and BigData. This is the slide presentation I did at the http://globalbigdatabootcamp.com/speakers/sanjay-sabnis/
DevDay 2013 - Building Startups and Minimum Viable ProductsBen Hall
DevDay (http://devday.pl),
20th of September 2013, Kraków
Video at http://www.youtube.com/watch?v=L4eTOvq2WmM&feature=c4-overview-vl&list=PLBMFXMTB7U74NdDghygvBaDcp67owVUUF
Minimum Viable Architecture - Good Enough is Good EnoughRandy Shoup
The “right” architecture and organization depends on the size and scale of your company. The only constant is change, and what works for 5 engineers does not work for 5000. Based upon lessons from Google and eBay, learn how to evolve both technology and organization together successfully.
This presentation is based on many hard-won lessons by the speaker, who led large-scale engineering teams at Google and eBay, but also co-founded a tiny startup and tried (unsuccessfully) to apply the same techniques. This session hopes to help others from making the same mistakes by introducing the concept of “Minimal Viable Architecture”. It outlines the common architectural evolution of a company or project through the search, execution, and scaling phases, and discusses the appropriate technologies, disciplines, and organizational structures at each phase. You'll start with a monolith, and end up with microservices, and that's completely and entirely appropriate.
When going into the development of a software product, a possible source of mistake is the incorrect evaluation of the complexity that lies behind an idea , as well as a clutter coming from the massive amounts of technologies enabled. This presentation explains a possible way to deal with such issues.
In his recent book, Clean Agile, Robert C "Uncle Bob" Martin chooses Extreme Programming (XP) for the basis of his explanation of Agile because "of all the Agile processes, XP is the best defined, the most complete, and the least muddled."
So why is it that in my professional life I only hear us speaking about Agile in terms of Scrum, Sprints, and possibly Kanban? Often I mention XP and people are not sure what I mean. Am I sure myself?
Coined in 1999 by Kent Beck and Ward Cunningham, XP has been with us for twenty years, but may of its practices have been with us for much longer. Many of them will be familar to you, but did you know they came from XP?
This talk aims to take us back to what XP is, how it fits in the Agile world, how it sits alongside other methodologies, and why, like Uncle Bob, I believe it is the best defined methodology, and what we should all be talking about.
The talk is based on a heavily refactored talk that Mike gave previously at Agile on the Beach conference, updated for 2020.
Given at Ox:Agile Meetup on February 11th 2020: https://www.meetup.com/OXAGILE/events/nxrdmrybcdbpb/
Machine learning applications are typically stitched together from hopes and dreams, shell scripts, cron jobs, home-grown schedulers, snippets of configuration clipped from multiple blog posts, thousands of hard-coded business rules, a.k.a. "our SQL corpus," and a few lines of training and testing code. Organizing all the moving parts into something maintainable and supportive of ongoing development is a challenge most teams have on their TODO list, roadmap, or tech debt pile. Getting ahead of the day-to-day demands and settling into a sane architecture often seems like an unattainable goal. The past several years have seen an explosion of tool-building in the data engineering and analytics area, including in Apache projects spanning the areas of search and information retrieval, job orchestration, file and stream formats, and machine learning libraries. In this talk we will cover our product and development teams' choices of architecture and tools, from data ingestion and storage, through transformations and processing, to presentation of results and publishing to web services, reports, and applications.
Learn more about enterprise frameworks and why your technology business and you need to be thinking about your software application architecture at scale.
Agile Development Overview (with a bit about builds)David Benjamin
I gave this presentation to our dev team when i started at Hannan IT back in October. Its a quick run through the Agile basics, with a bit of extra discussion on continuous integration.
I experimented here with scripting in two tangential sections in the hopes that it would avoid many more spontaneous tangents. It worked!
Serverless Architectures enable scalable and cost-effective apps to be built faster, so they can dramatically increase the odds of Your Startup's Success!
In "Startups + Serverless = Match made in Heaven" meetup, www.ServerlessToronto.org members discussed how to help Entrepreneurs push their businesses up to "other side of the teeterboard" (without failing) using the Serverless technologies: https://www.youtube.com/watch?v=1SqfJo47kMA
We have a lot to do on the cybersecurity side, and we are almost always lacking people, or budget, or both. Can we take lessons and approaches from entrepreneurship to apply to our cybersecurity programs? Can we do more with what we have, or for each addition can we make sure it has a large impact?
We’ll explore some entrepreneurship principles and then dive into some ways to improve security without large increases in headcount or budget.
Confoo-Montreal-2016: Controlling Your Environments using Infrastructure as CodeSteve Mercier
Slides from my talk at ConFoo Montreal, February 2016. A presentation on how to apply configuration management (CM) principles for your various environments, to control changes made to them. You apply CM on your code, why not on your environments content? This presentation will present the infrastructure as code principles using Chef and/or Ansible. Topics discussed include Continuous Integration, Continuous Delivery/Deployment principles, Infrastructure As Code and DevOps.
We’re all doing Agile nowadays, aren’t we? We’ll all delivering software in an Agile way. But what does that mean? Does it mean sprints and stand-ups? Kanban even? But what about Extreme Programming? If as a development team we’re not using pair programming, test driven development, continuous integration, and other XP practices, then we’re not really doing Agile software development and we may be on a march to frustration, or even failure.
I’m going to look at why the current trend of companies and projects adopting Scrum, calling themselves Agile, but not transitioning their development to XP, is a recipe for disaster. I’d like to cover the main practices of XP as well as other good practices that can really help a team deliver quality software, whether they’re doing two-week sprints, Kanban, or even Waterfall.
https://www.youtube.com/watch?v=aZgnY9fAHOA
Beyond Technical Debt: Unconventional techniques to uncover technical and soc...Juraj Martinka
* Traditional static code analysis tools (Sonar et al.) are notoriously bad at producing actionable insights (what does the "4000 years of accumulated technical debt" really mean?)
* CodeScene takes a different approach and focuses instead on how code evolves over time using a treasure trove we all posses - version control history.
* Come and meet CodeScene, a unique behavioral code analysis tool that looks for patterns in version control data to understand the history and evolution of a codebase: unraveling things like hotspots, change coupling between modules and interesting social aspects of the code.
* I'll describe the ideas behind CodeScene, how it works and demonstrate the techniques on analyses of real projects.
Comment choisir entre Parse, Heroku et AWS ?TheFamily
Par Laurent Cerveau, CTO at Zen.ly (https://twitter.com/laurentc27)
Inscrivez-vous au prochain meetup! — http://www.meetup.com/StartupWorkshop
Pour ne pas rater les prochains évènements, c'est ici : http://www.thefamily.co/education/
Entreprendre n’est pas inné et 80% des erreurs peuvent être évitées. Ne perds pas de temps, offre-toi Koudetat : http://bit.ly/koudetat-youtube
So you have finally this great product in mind with all its variations. It will be cool, slick, social have notification and million of people will use it! One problem : it all works because there is a server behind this great future client. But the more you talk with people around, the more you are confused on what to use to develop your server technology, how much investment (both in money and work) will be required, and how long will this investment last. Here we’ll talk about three possible ways (Facebook Parse , use of Heroku and do everything directly on AWS) and how they compare to each other.
Presentation done at the "CTO Crunch" event by France Digitale, Paris, 24/02/2015.
Based on his experience (VP Eng @ Digiplug, CTO @ Pixmania, VP Eng @ Criteo, CTO @ Aldebaran Robotics and now CTO @ Viadeo), Julien shares some hard-learned, bullshit-free lessons on what it means to be a CTO.
Hiring, Tools, Methodology, Technology, Politics: welcome to Hell :)
Similar to The Lost Tales of Platform Design (February 2017) (20)
An introduction to computer vision with Hugging FaceJulien SIMON
In this code-level talk, Julien will show you how to quickly build and deploy computer vision applications based on Transformer models. Along the way, you'll learn about the portfolio of open source and commercial Hugging Face solutions, and how they can help you deliver high-quality solutions faster than ever before.
Starting your AI/ML project right (May 2020)Julien SIMON
In this talk, we’ll see how you can put your AI/ML project on the right track from the get-go. Applying common sense and proven best practices, we’ll discuss skills, tools, methods, and more. We’ll also look at several real-life projects built by AWS customers in different industries and startups.
Building Machine Learning Inference Pipelines at Scale (July 2019)Julien SIMON
Talk at OSCON, Portland, 18/07/2019
Real-life Machine Learning applications require more than a single model. Data may need pre-processing: normalization, feature engineering, dimensionality reduction, etc. Predictions may need post-processing: filtering, sorting, combining, etc.
Our goal: build scalable ML pipelines with open source (Spark, Scikit-learn, XGBoost) and managed services (Amazon EMR, AWS Glue, Amazon SageMaker)
Optimize your Machine Learning Workloads on AWS (July 2019)Julien SIMON
Talk at Floor 28, Tel Aviv.
Infrastructure, tips to speed up training, hyperparameter optimization, model compilation, Amazon SageMaker Neo, cost optimization, Amazon Elastic Inference
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Assuring Contact Center Experiences for Your Customers With ThousandEyes
The Lost Tales of Platform Design (February 2017)
1. The Lost Tales of Platform Design!
Wisdom from the Masters, learned the Hard Way!
!
Julien Simon!
julien@julien.org!
@julsimon
2. Who am I ?
• Software Engineer
• 20+ years in R&D teams, from smartcards to web platforms
• VP Eng/CTO @ Digiplug, Pixmania, Criteo, Aldebaran Robotics, Viadeo
• Now working for Amazon Web Services
3. Agenda
• The Only Thing that Matters
• A Method to the Madness
• Infrastructure
• Architecture
• Code Hygiene
• Optimization
• The Bleeding Edge
• Summing things up
5. What Founders & Investors obsess about
“Do whatever is required to get to product/market fit. Including
changing out people, rewriting your product, moving into a
different market, telling customers no when you don't want to,
telling customers yes when you don't want to, raising that
fourth round of highly dilutive venture capital - whatever is
required. When you get right down to it, you can ignore almost
everything else”
Marc Andreessen, June 2007
http://web.archive.org/web/20070701074943/http://blog.pmarca.com/2007/06/the-pmarca-gu-2.html
7. “I’m a dev, that’s not my problem”
EXCUSE ME?
Are you a pro, or just playing with computers?
What do you think you’re getting paid for?
What do you think happens when the company runs out of cash?
Do you want to work for shit companies or great ones?
9. The Trilemma of Project Management!
aka The Iron Triangle (Martin Barnes, 1969)
https://en.wikipedia.org/wiki/Project_management_triangle
https://www.pmi.org/pmbok-guide-standards
10. Less than 1/3 of projects are successful!
and it’s not getting better
https://www.infoq.com/articles/standish-chaos-2015
11. Kaizen (1950)
Spend no money
Add no people
Add no space
https://en.wikipedia.org/wiki/Kaizen
https://en.wikipedia.org/wiki/Toyota_Production_System
Reduce waste
12. Boyd’s Law (1950)
• Boyd was a US Air Force Colonel, trying to understand why
the highly superior MIG-15 was consistently losing dogfights
to the F-86
• Answer: the hydraulic controls of the F-86 allowed its pilot to
repeatedly observe, orient, plan and act with less effort than
the MIG-15 pilot
• Hence, Boyd’s Law: in analyzing complexity, fast iteration
almost always produces better results than in-depth analysis
“A Better Path to Enterprise Architectures” (2006) https://msdn.microsoft.com/en-us/library/aa479371.aspx
13. No Silver Bullet — Essence and Accidents !
of Software Engineering (Fred Brooks, 1987)
“The essence of a software entity is a construct of interlocking concepts: !
data sets, relationships among data items, algorithms, and invocations of !
functions […] I believe the hard part of building software to be the specification, design,
and testing of this conceptual construct, not the labor of representing it and testing the
fidelity of the representation […] If this is true, building software will always be hard. There
is inherently no silver bullet.”
• Exploiting the mass market to avoid constructing what can be bought.
• Using rapid prototyping as part of a planned iteration in establishing software
requirements.
• Growing software organically, adding more and more function to systems as they are
run, used, and tested.
• Identifying and developing the great conceptual designers of the rising generation
http://faculty.salisbury.edu/~xswang/Research/Papers/SERelated/no-silver-bullet.pdf
17. Caveats
• Agile has become a cult, like ISO9001 30 years ago.
• Be a practitioner, not a priest.
• Yes, it’s OK to adapt Scrum/Kanban to your own context. !
Whatever works.
• The Agile Manifesto is the light in the dark. !
Stay on the path, you’ll be fine.
• Lean, MVP, walking skeleton: fine, but make sure you eventually deliver
something consistent. Sum of demos and PoCs != Product
• « Fail fast », « move fast, break stuff », « trial and error »: fine too, but
make sure you have tests and metrics, or how will you know you failed?
• Which brings me to…
18.
19.
20. In summary, you are expected to…
• Deliver new, working code at the drop of a hat
• Build it AND run it (DevOps, remember?)
• Iterate, learn & adapt as fast as possible
• Manage all necessary data for company-wide decisions
• Preserve cash while doing it: survival starts with not spending foolishly
• Be prepared to scale non-stop when you’re told to
27. Disaster Recovery (or not)
IBM datacenter (Dec’11, Tokyo)
Verizon facilities (Nov’12, Sandy in NYC)
Personal collection
28. Do you see the light?
Reminder:
• Deliver new, working code at the drop of a hat
• Build it AND run it (DevOps, remember?)
• Iterate, learn & adapt as fast as possible
• Manage all necessary data for company-wide decisions
• Preserve cash while doing it: survival starts with not spending foolishly
• Be prepared to scale like Hell when you’re told to
Ask yourself: !
does physical infrastructure help with these goals? !
Or does it stand in the way?
Yeah, thought so. The Cloud, then.
29. Minimum Viable Infrastructure
• Infrastructure should be iterative as well
• Early days
– Take the fastest route to customer discovery
– PaaS platforms rock: Heroku, Elastic Beanstalk, Convox, etc.
• As you grow
– Immutable infrastructure (and Docker isn’t the only way)
– No plumbing! Use Managed Services as much as you can
– Automate everything
• Infrastructure Deployment: CloudFormation, Terraform, etc.
• Application Deployment: Chef, Puppet, etc.
– Scale-out
http://www.slideshare.net/nzoschke/minimum-viable-infrastructure
31. A generation lost in the Bazaar (PHK, 2011)
That is the sorry reality of the bazaar Raymond praised in his book: a pile of
old festering hacks, endlessly copied and pasted by a clueless generation
of IT "professionals" who wouldn't recognize sound IT architecture if you hit
them over the head with it.
https://en.wikipedia.org/wiki/Poul-Henning_Kamp
http://queue.acm.org/detail.cfm?id=2349257
“Quality happens only
when someone is
responsible for it”
32. Gall’s Law (1975)
A complex system that works is invariably found
to have evolved from a simple system that
worked.
A complex system designed from scratch never
works and cannot be patched up to make it
work. You have to start over with a working
simple system.
https://en.wikipedia.org/wiki/John_Gall_(author)
33. Amdahl’s Law (1967)
• Theoretical speedup of the
execution of a task - at fixed
workload - that can be
expected of a system
w h o s e re s o u rc e s a re
improved.
• In other words: hardware
parallelism only helps as far
a s s o f t w a r e c a n b e
parallelized. Moore’s Law
doesn’t come for free!
https://en.wikipedia.org/wiki/Amdahl's_law
34. “You are not paid to write code” (2016)
Many people grumble or reject the notion of using proven tools or
techniques. It’s boring. It requires investing time to learn at the expense of
shipping code. It doesn’t do this one thing that we need it to do. It won’t
work for us. For some reason—and I continue to be completely baffled by
this—everyone sees their situation as a unique snowflake despite the fact
that a million other people have probably done the same thing […]
I was able to come to terms with this once I internalized something a
colleague once said: you are not paid to write code. You have never been
paid to write code. In fact, code is a nasty byproduct of being a software
engineer.
http://bravenewgeek.com/you-are-not-paid-to-write-code/
35. I’ll say it again: don’t write code unless you must
• Every single line of code is technical debt
• Design Patterns: only if they make sense. Don’t play with yourself.
• What’s your self-delusional story for not using:
– Off-the-shelf libs & tools
• Standard language libs, Boost, Guava, etc.
• Logging, parsing, math, crypto, etc.
– SaaS platforms
• Authentication, SSO
• Tags, trackers, A/B testing, dashboards
• Complex but non-core business processes: media transcoding,
document processing, etc.
36. Monitoring is a first-class citizen
• Design for monitoring from day 1.
• Don’t send monitoring information by email. No one reads it.
• Logging to local files is not enough and may create more problems
– Centralize log storage and processing (ELK, Splunk, etc.)
– If something is really urgent, it shouldn’t be written to a file!
• Monitor all services carefully
– Inside view (Nagios) and outside view (Catchpoint, PagerDuty)
– Don’t forget 3rd party services
• Real-time technical & business metrics are priceless (Graphite, Kibana)
37. Do it to them before they do it to you
• Make sure you understand how things fail (periodic testing)
• Make sure you understand why and how they failed (post mortems)
• Implement circuit breakers and feature toggles to limit blast radius
• Find out how your service behaves in degraded mode
• Break stuff, see what happens. Most of your assumptions will be wrong.
38. Minimum Viable Architecture
• Early days: take the fastest route to customer discovery. !
No points for style, but you should be aware of the shortcuts.
• As you grow: APIs and microservices, but there is no silver bullet J
– State management
– Dependency Hell
– Impedance mismatch between services and backends
– Deployment & monitoring
– Authentication across services
– Etc.
• Good news: scalability is almost free when your design is clean
http://www.slideshare.net/RandyShoup/minimum-viable-architecture-good-enough-is-good-enough-in-a-startup
40. Coding vs Debugging vs WTF
• How much of your time is spent
– working on product features?
– investigating or fixing bugs?
– doing something else?
• From my experience: 25% / 50% / 25%
• You only get a day’s worth of coding per week. Make it count.
41. Half your time is spent fixing bugs
http://www.newelectronics.co.uk/article-images/65147%5CVector_PDF.pdf
42. Spending less time & money on bugfixing
• Write less code
• Find bugs faster and sooner
– Deploy the best CI/CD you can get
– And don’t go building it! Lots of SaaS options: Travis, CircleCI, etc.
• Fix bugs faster
– Deploy the best real-time monitoring you can get
– If you have budget for APM tools like New Relic or Dynatrace, spend it!
• Don’t fix all bugs: “Fixing bugs is only important when the value of having the
bug fixed exceeds the cost of the fixing it” - Joel Spolsky
http://www.joelonsoftware.com/articles/fog0000000014.html
43. The Joel Test – 12 Steps to Better Code (2001)
Your real score is the one I’d find if I audited you ;)
http://www.joelonsoftware.com/articles/fog0000000043.html
https://simpleprogrammer.com/2015/02/16/joel-test-programmers-simple-programmer-test/
47. Premature optimization is the root of all evil !
(Knuth, 1974)
Programmers waste enormous amounts of time thinking
about, or worrying about, the speed of noncritical parts of
their programs, and these attempts at efficiency actually
have a strong negative impact when debugging and
maintenance are considered.
We should forget about small efficiencies, say about 97%
of the time: premature optimization is the root of all evil.
Yet we should not pass up our opportunities in that critical
3%
Structured Programming with Goto Statements, 1971!
48. Mature Optimization Handbook !
(Carlos Bueno @ Facebook, 2013)
Performance optimization is, or should be,a cost/benefit
decision
In my experience, it makes most sense on mature systems
whose architectures have settled down. New code is almost by
definition slow code, but it’s also likely to be ripped out and
replaced as a young program slouches towards beta-test.
Unless your optimizations are going to stick around long
enough to pay for the time you spend making them, plus the
opportunity cost of not doing something else, it’s a net loss
https://www.facebook.com/notes/facebook-engineering/the-mature-optimization-handbook/10151784131623920/
49. What’s the ROI of saving x ms on a feature?
You don’t know? My point exactly!
Find out before launching into a !
useless, risky optimization crusade
50. Latency Numbers Every Programmer Should Know
https://people.eecs.berkeley.edu/~rcs/research/interactive_latency.html
Actually, the numbers don’t matter, the ratios do!
51. Numbers only 0.1% of Programmers Should Know
http://ithare.com/infographics-operation-costs-in-cpu-clock-cycles /
• Are you writing
critical embedded
code?
• Or compilers?
• Or HFT code?
• No? Keep out, then
52. What I think really matters
• Make sure that critical product features are “fast enough”
• Don’t wait after product launch to find out that they !
they’re not
• Performance monitoring is part on the dev process
– Unit testing for raw performance
– Load testing for throughput
– Profiling: gprof, Callgrind, Dtrace, APM tools, etc.
• Most of the issues will be design or configuration, so start
there instead of chasing mythical deep-sea monsters
53. Silent but Deadly (in no particular order)
1. Stupid code (StringBuilder, not String)
2. Naive Memory / Thread Allocation (pooling, dammit)
3. 1+2 inside a loop
4. Database contention, misbehaving connection pools
5. In-memory cache contention (memcached, etc.)
6. Blocking calls and locks in general
7. Garbage Collection in “Workstation Mode”
8. Stupid SQL, too few or too many indexes, ORMs
9. Additional A/B tests or logs “that someone needed”
10. Parsing data (XML, JSON, etc) without a parser “because it’s simpler”
11. ...
55. Death march projects (Yourdon, 1996)
What really contributes to the death march nature of such
projects […] is the attempt to use bleeding-edge
technology for an industrial-strength application. Even if
the technology is basically usable, it often does not scale
up well for large-scale usage; and nobody knows how to
exploit its strengths and avoid its weaknesses; and the
vendors don't know how to support it properly; and on
and on…
Is it any wonder that things degenerate into a death
march project, with everyone working late nights and long
weekends in order to coax the experimental new
technology into some semblance of working order?
56. Choose boring technology (McKinley, 2015)
What counts as boring? That's a little tricky. "Boring" should not be conflated
with "bad” […] There are many choices of technology that are boring and good,
or at least good enough. MySQL is boring. Postgres is boring. PHP is boring.
Python is boring. Memcached is boring. Squid is boring. Cron is boring.
The nice thing about boringness (so constrained) is that the capabilities of
these things are well understood. But more importantly, their failure modes are
well understood.
New technology choices might be purely additive (for example: "we don't have
caching yet, so let's add memcached"). But they might also overlap or replace
things you are already using. If that's the case, you should set clear
expectations about migrating old functionality to the new system.
http://mcfunley.com/choose-boring-technology
57. Hype Driven Development (Kirejczyk, 2017)
Software development teams often make decisions about
software architecture or technological stack based on
inaccurate opinions, social media, and in general on what is
considered to be “hot”, rather than solid research and any
serious consideration of expected impact on their projects. I
call this trend Hype Driven Development, perceive it harmful
and advocate for a more professional approach I call “Solid
Software Engineering”.
https://blog.daftcode.pl/hype-driven-development-3469fc2e9b22
58. Horror Stories Are Just Stories
• Every single piece of tech has horror stories
• They’re fun to read and may teach you a thing or two
• However, most of the time they’re not relevant to your platform
– Different version
– Different workload
– Different scale
– etc.
• So do not base any technology choice on horror stories
– One exception: ESBs and other shit Enterprise middleware. Just say no.
• Remember: it’s easier to blame tech than to expose your own mistakes
59. And please don’t be one of these
• Trolls: « Java is for pussies. Real men use C++ »
• Lunatics: « Erlang is the future ». Also works with Scala, Haskell, Clojure, etc.
• Old fart: « SQL Server has always worked for us » (and they buy us lunch)
• PhD egghead: « I think we can reduce latency by tweaking the Ethernet driver »
• And the worst of all, fanboys & hipsters: !
« guys, HackyLib v0.1 has just been pushed to Github. !
It’s totally awesome. Spotify, Netflix and Valve are already !
using it in production. Let’s use it too! »
Make your own choices, build your own platform.
60. Use common sense
1. Focus on business needs
2. Identify top challenges: time to market? UI? Perf? Security? Don’t know?
3. List candidate technologies, expecting them to last at least a year (think 10x)
4. KPIs, benchmarks, PoC: educated guess is OK, random decision isn’t!
5. Implement, deploy and monitor
6. Anything on fire?
– Can it be fixed by refactoring or optimization?
– If not (are you really sure?), can it be fixed with new technology?
• Yes: you need a new building block in your stack, GOTO 2.
• No: WTF? Are you scared? Man up! Not moving = death
– If there is absolutely no other way, add servers… !
but it won’t work forever!
62. • Your project will die unless it finds product-market fit fast enough
• Unless you’re GAFA / FANG / NATU, you have the same ideas, tech and
algos as everyone else: this isn’t how you’ll win the fight
• Turn ideas into MVPs as fast and frugally as possible
– Lean & Agile methods
– Flexible design, well-understood technology choices
– Best development tools possible
– Solid data stack to collect, store, process and visualize
• Keep a critical eye out for new tech, but think twice (or more) before adding
it to your stack
• When the Big Boys push the button, be ready to scale