PyconJP: Building a data preparation pipeline with Pandas and AWS LambdaFabian Dubois
Building a data preparation pipeline with Pandas and AWS Lambda
What is data preparation and why it is required.
How to prepare data with pandas.
How to set up a pipeline with AWS Lambda
https://youtu.be/pc0Xn0uAm34?t=9m15s
Building a Machine Learning App with AWS LambdaSri Ambati
Ludi Rehaks' meetup on 03.17.16
- Powered by the open source machine learning software H2O.ai. Contributors welcome at: https://github.com/h2oai
- To view videos on H2O open source machine learning software, go to: https://www.youtube.com/user/0xdata
Machine learning in the physical world by Kip Larson from AWS IoTBill Liu
Presented at AI NEXTCon Seattle 1/17-20, 2018
http://aisea18.xnextcon.com
join our free online AI group with 50,000+ tech engineers to learn and practice AI technology, including: latest AI news, tech articles/blogs, tech talks, tutorial videos, and hands-on workshop/codelabs, on machine learning, deep learning, data science, etc..
AWS re:Invent 2016: NEW LAUNCH! Lambda Everywhere (IOT309)Amazon Web Services
You can now execute Lambda’s almost anywhere – originating in the cloud, and on connected devices with AWS Greengrass. This advanced technical session explores Lambda Functions and what it means to use them across these diverse environments. We will treat the cloud as the ‘brain’, using local Lambda’s for local executions. This way devices can react instinctively, much like the autonomic nervous system, operating in the periphery and responsible for collecting and filtering information, implementing simple and time-sensitive local actions reflexively.
PyconJP: Building a data preparation pipeline with Pandas and AWS LambdaFabian Dubois
Building a data preparation pipeline with Pandas and AWS Lambda
What is data preparation and why it is required.
How to prepare data with pandas.
How to set up a pipeline with AWS Lambda
https://youtu.be/pc0Xn0uAm34?t=9m15s
Building a Machine Learning App with AWS LambdaSri Ambati
Ludi Rehaks' meetup on 03.17.16
- Powered by the open source machine learning software H2O.ai. Contributors welcome at: https://github.com/h2oai
- To view videos on H2O open source machine learning software, go to: https://www.youtube.com/user/0xdata
Machine learning in the physical world by Kip Larson from AWS IoTBill Liu
Presented at AI NEXTCon Seattle 1/17-20, 2018
http://aisea18.xnextcon.com
join our free online AI group with 50,000+ tech engineers to learn and practice AI technology, including: latest AI news, tech articles/blogs, tech talks, tutorial videos, and hands-on workshop/codelabs, on machine learning, deep learning, data science, etc..
AWS re:Invent 2016: NEW LAUNCH! Lambda Everywhere (IOT309)Amazon Web Services
You can now execute Lambda’s almost anywhere – originating in the cloud, and on connected devices with AWS Greengrass. This advanced technical session explores Lambda Functions and what it means to use them across these diverse environments. We will treat the cloud as the ‘brain’, using local Lambda’s for local executions. This way devices can react instinctively, much like the autonomic nervous system, operating in the periphery and responsible for collecting and filtering information, implementing simple and time-sensitive local actions reflexively.
Crunch Your Data in the Cloud with Elastic Map Reduce - Amazon EMR HadoopAdrian Cockcroft
A introductory discussion of cloud computing and capacity planning implications is followed by a step by step guide to running a Hadoop job in EMR, and finally a discussion of how to write your own Hadoop queries.
4 years ago, mid 2013, we have identified a gap in the cloud echo-system. The landscape of IaaS, PaaS and SaaS provides solutions for VMs, Container and Networking, platforms of different types for backend developers, Backends for mobile developers and ready made software for individuals and enterprises. What is missing in the middle is the platform for web-sites and web-apps.
4 years down the line, with the emergence of Serverless, there are still no players in this gap. We will talk about what makes a platform for web-sites and web-apps. Things frontend optimized javascript, SEO, visual builder, web methods & backend javascript as well as request time container boot.
We have built Wix Code over the last 4 years targeting this exact gap – a serverless platform for website and web applications, and so …
Wix is taking the risk of predicting the future of serverless computing and where it should be 4 years from now.
Resilient microservices with Kubernetes - Mete AtamelITCamp
Creating a single microservice is a well understood problem. Creating a cluster of load-balanced microservices that are resilient and self-healing is not so easy. Managing that cluster with rollouts and rollbacks, scaling individual services on demand, securely sharing secrets and configuration among services is even harder. Kubernetes, an open-source container management system, can help with this. In this talk, we will start with a simple microservice, containerize it using Docker, and scale it to a cluster of resilient microservices managed by Kubernetes. Along the way, we will learn what makes Kubernetes a great system for automating deployment, operations, and scaling of containerized applications.
Portable batch and streaming pipelines with Apache Beam (Big Data Application...Malo Denielou
Apache Beam is a top-level Apache project which aims at providing a unified API for efficient and portable data processing pipeline. Beam handles both batch and streaming use cases and neatly separates properties of the data from runtime characteristics, allowing pipelines to be portable across multiple runtimes, both open-source (e.g., Apache Flink, Apache Spark, Apache Apex, ...) and proprietary (e.g., Google Cloud Dataflow). This talk will cover the basics of Apache Beam, describe the main concepts of the programming model and talk about the current state of the project (new python support, first stable version). We'll illustrate the concepts with a use case running on several runners.
Presented at AI NEXTCon Seattle 1/17-20, 2018
http://aisea18.xnextcon.com
join our free online AI group with 50,000+ tech engineers to learn and practice AI technology, including: latest AI news, tech articles/blogs, tech talks, tutorial videos, and hands-on workshop/codelabs, on machine learning, deep learning, data science, etc..
Video: https://youtu.be/T0L0JxDaPkc
RSVP Here: https://www.eventbrite.com/e/full-day-workshop-kubeflow-kerastensorflow-20-tf-extended-tfx-kubernetes-pytorch-xgboost-airflow-tickets-63362929227
Description
In this workshop, we build real-world machine learning pipelines using TensorFlow Extended (TFX), KubeFlow, Airflow, and MLflow.
Described in the 2017 paper, TFX is used internally by thousands of Google data scientists and engineers across every major product line within Google.
KubeFlow is a modern, end-to-end pipeline orchestration framework that embraces the latest AI best practices including hyper-parameter tuning, distributed model training, and model tracking.
Airflow is the most-widely used pipeline orchestration framework in machine learning and data engineering.
MLflow is a lightweight experiment-tracking system recently open-sourced by Databricks, the creators of Apache Spark. MLflow supports Python, Java/Scala, and R - and offers native support for TensorFlow, Keras, and Scikit-Learn.
Pre-requisites
Modern browser - and that's it!
Every attendee will receive a cloud instance
Nothing will be installed on your local laptop
Everything can be downloaded at the end of the workshop
Location
Online Workshop
The link will be sent a few hours before the start of the workshop.
Only registered users will receive the link.
If you do not receive the link a few hours before the start of the workshop, please send your Eventbrite registration confirmation to support@pipeline.ai for help.
Agenda
1. Create a Kubernetes cluster
2. Install KubeFlow, Airflow, TFX, and Jupyter
3. Setup ML Training Pipelines with KubeFlow and Airflow
4. Transform Data with TFX Transform
5. Validate Training Data with TFX Data Validation
6. Train Models with Jupyter, Keras/TensorFlow 2.0, PyTorch, XGBoost, and KubeFlow
7. Run a Notebook Directly on Kubernetes Cluster with KubeFlow
8. Analyze Models using TFX Model Analysis and Jupyter
9. Perform Hyper-Parameter Tuning with KubeFlow
10. Select the Best Model using KubeFlow Experiment Tracking
11. Run Multiple Experiments with MLflow Experiment Tracking
12. Reproduce Model Training with TFX Metadata Store
13. Deploy the Model to Production with TensorFlow Serving and Istio
14. Save and Download your Workspace
Key Takeaways
Attendees will gain experience training, analyzing, and serving real-world Keras/TensorFlow 2.0 models in production using model frameworks and open-source tools.
RSVP Here: https://www.eventbrite.com/e/full-day-workshop-kubeflow-kerastensorflow-20-tf-extended-tfx-kubernetes-pytorch-xgboost-airflow-tickets-63362929227
https://youtu.be/T0L0JxDaPkc
AWS re:Invent 2016 Recap: What Happened, What It MeansRightScale
Get behind the hype and headlines from AWS re:Invent 2016 and find out what it all means to you. We’ll share what’s working for AWS users and highlight which new features and services you’ll want to look at. Whether or not you attended re:Invent, this wrap-up will help you develop your 2017 cloud to-do list.
Serverless architectures are promising and will play an important role in the coming years but the ecosystem around serverless is still pretty young. We have been operating Lambda based applications for about a year and faced several challenges. In this presentation we share these challenges and propose some solutions to work around them.
Zeppelin Interpreters
PSQL (to became JDBC in 0.6.x)
Geode
SpringXD
Apache Ambari
Zeppelin Service
Geode, HAWQ and Spring XD services
Webpage Embedder View
AWS re:Invent 2016: Running Batch Jobs on Amazon ECS (CON310)Amazon Web Services
Batch computing is a common way for developers, scientists and engineers to run a series of jobs on a large pool of shared compute resources, such as servers, virtual machines, and containers. Amazon ECS makes it easy to run and manage Docker-enabled applications across a cluster of Amazon EC2 instances. In this session will show you how to run batch jobs using Amazon ECS and together with other AWS services, such as AWS Lambda and Amazon SQS. We will see how you can leverage Amazon EC2 Spot Instances to power your ECS cluster and easily scale your batch workloads. You'll hear from Mapbox on how they use ECS to power their entire batch processing architecture to collect and process over 100 million miles of sensor data per day that they use for powering their maps. Mapbox will also discuss how they optimize their batch processing framework on ECS using Spot Instances and demo their open source framework that will help you get up and running with ECS in minutes.
AWS re:Invent 2016: Get Technically Inspired by Container-Powered Migrations ...Amazon Web Services
This session is a technical journey through application migration and refactoring using containerized technologies. Flux 7 recently worked with Rent-a-Center to perform a Hybris migration from their datacenter to AWS and you can hear how they used Amazon ECS, the new Application Load Balancer, and Auto Scaling to meet the customers' business objectives.
Data Summer Conf 2018, “Building unified Batch and Stream processing pipeline...Provectus
Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing pipelines, and also data ingestion and integration flows, supporting for both batch and streaming use cases. In presentation I will provide a general overview of Apache Beam and programming model comparison Apache Beam vs Apache Spark.
An MPI-IO Cloud Cluster Bioinformatics Summer Project (BDT205) | AWS re:Inven...Amazon Web Services
Researchers at Clemson University assigned a student summer intern to explore bioinformatics cloud solutions that leverage MPI, the OrangeFS parallel file system, AWS CloudFormation templates, and a Cluster Scheduler. The result was an AWS cluster that runs bioinformatics code optimized using MPI-IO. We give an overview of the process and show how easy it is to create clusters in AWS.
Building Serverless Data Infrastructure in the AWS CloudRyan Plant
Presentation given at the Utah Big Mountain Data & Developer Conference in November 2017. Describes the new data economy, a reference architecture for Big Data infrastructure, and its application to Amazon Web Services serverless services.
Automated Testing for Terraform, Docker, Packer, Kubernetes, and MoreC4Media
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2rm4hFD.
Yevgeniy Brikman talks about how to write automated tests for infrastructure code, including the code written for use with tools such as Terraform, Docker, Packer, and Kubernetes. Topics covered include: unit tests, integration tests, end-to-end tests, dependency injection, test parallelism, retries and error handling, static analysis, property testing and CI / CD for infrastructure code. Filmed at qconsf.com.
Yevgeniy Brikman is the co-founder of Gruntwork, a company that provides DevOps as a Service. He is the author of two books published by O'Reilly Media: Hello, Startup and Terraform: Up & Running. Previously, he worked as a software engineer at LinkedIn, TripAdvisor, Cisco Systems, and Thomson Financial.
Batch Processing with Containers on AWS - June 2017 AWS Online Tech TalksAmazon Web Services
Learning Objectives:
- Learn about the options for running batch workloads on AWS
- Learn how to architect a containerized batch processing service on Amazon ECS
- Learn best practices for optimizing and scaling complex batch workload requirements
Batch processing is useful when you need to periodically analyze large amounts of data, but configuring and scaling a cluster of virtual machines to process complex batch jobs can be difficult. Containers provide a great solution for running batch jobs by providing easily managed, scalable, and portable code environments.
In this tech talk, we’ll show you how to use containers on AWS for batch processing jobs that can scale quickly and cost-effectively. We’ll discuss AWS Batch, our fully managed batch-processing service, and show you how to architect your own batch processing service using the Amazon EC2 Container Service. We’ll also discuss best practices for ensuring efficient and opportunistic scheduling, fine-grained monitoring, compute resource auto-scaling, and security for your batch jobs.
Listen up, developers. You are not special. Your infrastructure is not a beautiful and unique snowflake. You have the same tech debt as everyone else. This is a talk about a better way to build and manage infrastructure: Terraform Modules. It goes over how to build infrastructure as code, package that code into reusable modules, design clean and flexible APIs for those modules, write automated tests for the modules, and combine multiple modules into an end-to-end techs tack in minutes.
You can find the video here: https://www.youtube.com/watch?v=LVgP63BkhKQ
Crunch Your Data in the Cloud with Elastic Map Reduce - Amazon EMR HadoopAdrian Cockcroft
A introductory discussion of cloud computing and capacity planning implications is followed by a step by step guide to running a Hadoop job in EMR, and finally a discussion of how to write your own Hadoop queries.
4 years ago, mid 2013, we have identified a gap in the cloud echo-system. The landscape of IaaS, PaaS and SaaS provides solutions for VMs, Container and Networking, platforms of different types for backend developers, Backends for mobile developers and ready made software for individuals and enterprises. What is missing in the middle is the platform for web-sites and web-apps.
4 years down the line, with the emergence of Serverless, there are still no players in this gap. We will talk about what makes a platform for web-sites and web-apps. Things frontend optimized javascript, SEO, visual builder, web methods & backend javascript as well as request time container boot.
We have built Wix Code over the last 4 years targeting this exact gap – a serverless platform for website and web applications, and so …
Wix is taking the risk of predicting the future of serverless computing and where it should be 4 years from now.
Resilient microservices with Kubernetes - Mete AtamelITCamp
Creating a single microservice is a well understood problem. Creating a cluster of load-balanced microservices that are resilient and self-healing is not so easy. Managing that cluster with rollouts and rollbacks, scaling individual services on demand, securely sharing secrets and configuration among services is even harder. Kubernetes, an open-source container management system, can help with this. In this talk, we will start with a simple microservice, containerize it using Docker, and scale it to a cluster of resilient microservices managed by Kubernetes. Along the way, we will learn what makes Kubernetes a great system for automating deployment, operations, and scaling of containerized applications.
Portable batch and streaming pipelines with Apache Beam (Big Data Application...Malo Denielou
Apache Beam is a top-level Apache project which aims at providing a unified API for efficient and portable data processing pipeline. Beam handles both batch and streaming use cases and neatly separates properties of the data from runtime characteristics, allowing pipelines to be portable across multiple runtimes, both open-source (e.g., Apache Flink, Apache Spark, Apache Apex, ...) and proprietary (e.g., Google Cloud Dataflow). This talk will cover the basics of Apache Beam, describe the main concepts of the programming model and talk about the current state of the project (new python support, first stable version). We'll illustrate the concepts with a use case running on several runners.
Presented at AI NEXTCon Seattle 1/17-20, 2018
http://aisea18.xnextcon.com
join our free online AI group with 50,000+ tech engineers to learn and practice AI technology, including: latest AI news, tech articles/blogs, tech talks, tutorial videos, and hands-on workshop/codelabs, on machine learning, deep learning, data science, etc..
Video: https://youtu.be/T0L0JxDaPkc
RSVP Here: https://www.eventbrite.com/e/full-day-workshop-kubeflow-kerastensorflow-20-tf-extended-tfx-kubernetes-pytorch-xgboost-airflow-tickets-63362929227
Description
In this workshop, we build real-world machine learning pipelines using TensorFlow Extended (TFX), KubeFlow, Airflow, and MLflow.
Described in the 2017 paper, TFX is used internally by thousands of Google data scientists and engineers across every major product line within Google.
KubeFlow is a modern, end-to-end pipeline orchestration framework that embraces the latest AI best practices including hyper-parameter tuning, distributed model training, and model tracking.
Airflow is the most-widely used pipeline orchestration framework in machine learning and data engineering.
MLflow is a lightweight experiment-tracking system recently open-sourced by Databricks, the creators of Apache Spark. MLflow supports Python, Java/Scala, and R - and offers native support for TensorFlow, Keras, and Scikit-Learn.
Pre-requisites
Modern browser - and that's it!
Every attendee will receive a cloud instance
Nothing will be installed on your local laptop
Everything can be downloaded at the end of the workshop
Location
Online Workshop
The link will be sent a few hours before the start of the workshop.
Only registered users will receive the link.
If you do not receive the link a few hours before the start of the workshop, please send your Eventbrite registration confirmation to support@pipeline.ai for help.
Agenda
1. Create a Kubernetes cluster
2. Install KubeFlow, Airflow, TFX, and Jupyter
3. Setup ML Training Pipelines with KubeFlow and Airflow
4. Transform Data with TFX Transform
5. Validate Training Data with TFX Data Validation
6. Train Models with Jupyter, Keras/TensorFlow 2.0, PyTorch, XGBoost, and KubeFlow
7. Run a Notebook Directly on Kubernetes Cluster with KubeFlow
8. Analyze Models using TFX Model Analysis and Jupyter
9. Perform Hyper-Parameter Tuning with KubeFlow
10. Select the Best Model using KubeFlow Experiment Tracking
11. Run Multiple Experiments with MLflow Experiment Tracking
12. Reproduce Model Training with TFX Metadata Store
13. Deploy the Model to Production with TensorFlow Serving and Istio
14. Save and Download your Workspace
Key Takeaways
Attendees will gain experience training, analyzing, and serving real-world Keras/TensorFlow 2.0 models in production using model frameworks and open-source tools.
RSVP Here: https://www.eventbrite.com/e/full-day-workshop-kubeflow-kerastensorflow-20-tf-extended-tfx-kubernetes-pytorch-xgboost-airflow-tickets-63362929227
https://youtu.be/T0L0JxDaPkc
AWS re:Invent 2016 Recap: What Happened, What It MeansRightScale
Get behind the hype and headlines from AWS re:Invent 2016 and find out what it all means to you. We’ll share what’s working for AWS users and highlight which new features and services you’ll want to look at. Whether or not you attended re:Invent, this wrap-up will help you develop your 2017 cloud to-do list.
Serverless architectures are promising and will play an important role in the coming years but the ecosystem around serverless is still pretty young. We have been operating Lambda based applications for about a year and faced several challenges. In this presentation we share these challenges and propose some solutions to work around them.
Zeppelin Interpreters
PSQL (to became JDBC in 0.6.x)
Geode
SpringXD
Apache Ambari
Zeppelin Service
Geode, HAWQ and Spring XD services
Webpage Embedder View
AWS re:Invent 2016: Running Batch Jobs on Amazon ECS (CON310)Amazon Web Services
Batch computing is a common way for developers, scientists and engineers to run a series of jobs on a large pool of shared compute resources, such as servers, virtual machines, and containers. Amazon ECS makes it easy to run and manage Docker-enabled applications across a cluster of Amazon EC2 instances. In this session will show you how to run batch jobs using Amazon ECS and together with other AWS services, such as AWS Lambda and Amazon SQS. We will see how you can leverage Amazon EC2 Spot Instances to power your ECS cluster and easily scale your batch workloads. You'll hear from Mapbox on how they use ECS to power their entire batch processing architecture to collect and process over 100 million miles of sensor data per day that they use for powering their maps. Mapbox will also discuss how they optimize their batch processing framework on ECS using Spot Instances and demo their open source framework that will help you get up and running with ECS in minutes.
AWS re:Invent 2016: Get Technically Inspired by Container-Powered Migrations ...Amazon Web Services
This session is a technical journey through application migration and refactoring using containerized technologies. Flux 7 recently worked with Rent-a-Center to perform a Hybris migration from their datacenter to AWS and you can hear how they used Amazon ECS, the new Application Load Balancer, and Auto Scaling to meet the customers' business objectives.
Data Summer Conf 2018, “Building unified Batch and Stream processing pipeline...Provectus
Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing pipelines, and also data ingestion and integration flows, supporting for both batch and streaming use cases. In presentation I will provide a general overview of Apache Beam and programming model comparison Apache Beam vs Apache Spark.
An MPI-IO Cloud Cluster Bioinformatics Summer Project (BDT205) | AWS re:Inven...Amazon Web Services
Researchers at Clemson University assigned a student summer intern to explore bioinformatics cloud solutions that leverage MPI, the OrangeFS parallel file system, AWS CloudFormation templates, and a Cluster Scheduler. The result was an AWS cluster that runs bioinformatics code optimized using MPI-IO. We give an overview of the process and show how easy it is to create clusters in AWS.
Building Serverless Data Infrastructure in the AWS CloudRyan Plant
Presentation given at the Utah Big Mountain Data & Developer Conference in November 2017. Describes the new data economy, a reference architecture for Big Data infrastructure, and its application to Amazon Web Services serverless services.
Automated Testing for Terraform, Docker, Packer, Kubernetes, and MoreC4Media
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2rm4hFD.
Yevgeniy Brikman talks about how to write automated tests for infrastructure code, including the code written for use with tools such as Terraform, Docker, Packer, and Kubernetes. Topics covered include: unit tests, integration tests, end-to-end tests, dependency injection, test parallelism, retries and error handling, static analysis, property testing and CI / CD for infrastructure code. Filmed at qconsf.com.
Yevgeniy Brikman is the co-founder of Gruntwork, a company that provides DevOps as a Service. He is the author of two books published by O'Reilly Media: Hello, Startup and Terraform: Up & Running. Previously, he worked as a software engineer at LinkedIn, TripAdvisor, Cisco Systems, and Thomson Financial.
Batch Processing with Containers on AWS - June 2017 AWS Online Tech TalksAmazon Web Services
Learning Objectives:
- Learn about the options for running batch workloads on AWS
- Learn how to architect a containerized batch processing service on Amazon ECS
- Learn best practices for optimizing and scaling complex batch workload requirements
Batch processing is useful when you need to periodically analyze large amounts of data, but configuring and scaling a cluster of virtual machines to process complex batch jobs can be difficult. Containers provide a great solution for running batch jobs by providing easily managed, scalable, and portable code environments.
In this tech talk, we’ll show you how to use containers on AWS for batch processing jobs that can scale quickly and cost-effectively. We’ll discuss AWS Batch, our fully managed batch-processing service, and show you how to architect your own batch processing service using the Amazon EC2 Container Service. We’ll also discuss best practices for ensuring efficient and opportunistic scheduling, fine-grained monitoring, compute resource auto-scaling, and security for your batch jobs.
Listen up, developers. You are not special. Your infrastructure is not a beautiful and unique snowflake. You have the same tech debt as everyone else. This is a talk about a better way to build and manage infrastructure: Terraform Modules. It goes over how to build infrastructure as code, package that code into reusable modules, design clean and flexible APIs for those modules, write automated tests for the modules, and combine multiple modules into an end-to-end techs tack in minutes.
You can find the video here: https://www.youtube.com/watch?v=LVgP63BkhKQ
Pycon Colombia 2018
One year ago I joined a team that favours Serverless, since then I’ve been building and maintaining lots of services using Serverless. With a pinch of Skepticism, I sailed through some of the challenges and tooling, I want to share with the community the pains and glory of it.
● Fundamentals
● Key Components
● Best practices
● Spring Boot REST API Deployment
● CI with Ansible
● Ansible for AWS
● Provisioning a Docker Host
● Docker&Ansible
https://github.com/maaydin/ansible-tutorial
Adopt DevOps philosophy on your Symfony projects (Symfony Live 2011)Fabrice Bernhard
This is the presentation given at the Symfony Live 2011 conference. It is an introduction to the new agile movement spreading in the technical operations community called DevOps and how to adopt it on web development projects, in particular Symfony projects.
Plan of the slides :
- Configuration Management
- Development VM
- Scripted deployment
- Continuous deployment
Tools presented in the slides:
- Puppet
- Vagrant
- Fabric
- Jenkins / Hudson
Kubernetes is exploding in popularity right now and has all the buzz and cargo-culting that Docker enjoyed just a few years ago. But what even is Kubernetes? How do I run my PHP apps in it? Should I run my PHP apps in it ?
AWS has improved Lambda cold starts by leaps and bounds in the last year. But for performance-sensitive applications such as user-facing APIs, Lambda cold starts are still a thorn in one’s side, especially when working with languages such as Java and .Net Core.
In this webinar, we will dive into strategies for improving cold start latency and how to mitigate them altogether with Provisioned Concurrency, and how Lumigo helps you optimize your use of Provisioned Concurrency.
DevOps Fest 2020. immutable infrastructure as code. True story.Vlad Fedosov
In this talk I’ll explain how we went from classic Pet servers to immutable infrastructure, fully described as code, with Cattle instances. I’ll also share which tools we use and how we evolved our experience with them.
Serverless in production, an experience report (IWOMM)Yan Cui
AWS Lambda has changed the way we deploy and run software, but this new serverless paradigm has created new challenges to old problems - how do you test a cloud-hosted function locally? How do you monitor them? What about logging and config management? And how do we start migrating from existing architectures?
In this talk Yan and Domas will discuss solutions to these challenges by drawing from real-world experience running Lambda in production and migrating from an existing monolithic architecture.
Integrating Infrastructure as Code into a Continuous Delivery Pipeline | AWS ...Amazon Web Services
Ansible is a simple, but powerful automation tool with an agentless footprint that allows for the definition of architecture, intent, and policy as code that can be deployed across both on-prem and cloud infrastructure. This enables customers to extend their enterprise and applications into AWS in a way that maintains a consistent, secure posture as part of a continuous delivery pipeline. Customers can then natively integrate with AWS to seamlessly configure and deploy a range of AWS services such as Amazon Aurora, Amazon Redshift, Amazon EMR, Amazon Athena, Amazon CloudFront, Amazon Route 53, and Elastic Load Balancing from within Red Hat OpenShift across a secure, consistent hybrid cloud infrastructure. In this session, we will demonstrate how infrastructure can be instantiated with code as part of a continuous delivery pipeline and describe how that integrates with an OpenShift hybrid cloud deployment. Learn More: https://aws.amazon.com/government-education/
This presentation explains what serverless is all about, explaining the context from Devs & Ops points of view, and presenting the various ways to achieve serverless (Functions a as Service, BaaS....). It also presents the various competitors on the market and demo one of them, openfaas. Finally, it enlarges the pictures, positionning serverless, combined with Edge computing & IoT, as a valuable triptic cloud vendors are leveraging on top of, to create end-to-end offers.
AWS Summit 2013 | Auckland - Continuous Deployment Practices, with Production...Amazon Web Services
With AWS companies now have the ability to develop and run their applications with speed and flexibility like never before. Working with an infrastructure that can be 100% API driven enables businesses to use lean methodologies and realize these benefits. This in turn leads to greater success for those who make use of these practices. In this session we'll talk about some key concepts and design patterns for Continuous Deployment and Continuous Integration, two elements of lean development of applications and infrastructures.
Globus Connect Server Deep Dive - GlobusWorld 2024Globus
We explore the Globus Connect Server (GCS) architecture and experiment with advanced configuration options and use cases. This content is targeted at system administrators who are familiar with GCS and currently operate—or are planning to operate—broader deployments at their institution.
top nidhi software solution freedownloadvrstrong314
This presentation emphasizes the importance of data security and legal compliance for Nidhi companies in India. It highlights how online Nidhi software solutions, like Vector Nidhi Software, offer advanced features tailored to these needs. Key aspects include encryption, access controls, and audit trails to ensure data security. The software complies with regulatory guidelines from the MCA and RBI and adheres to Nidhi Rules, 2014. With customizable, user-friendly interfaces and real-time features, these Nidhi software solutions enhance efficiency, support growth, and provide exceptional member services. The presentation concludes with contact information for further inquiries.
Cyaniclab : Software Development Agency Portfolio.pdfCyanic lab
CyanicLab, an offshore custom software development company based in Sweden,India, Finland, is your go-to partner for startup development and innovative web design solutions. Our expert team specializes in crafting cutting-edge software tailored to meet the unique needs of startups and established enterprises alike. From conceptualization to execution, we offer comprehensive services including web and mobile app development, UI/UX design, and ongoing software maintenance. Ready to elevate your business? Contact CyanicLab today and let us propel your vision to success with our top-notch IT solutions.
Enhancing Project Management Efficiency_ Leveraging AI Tools like ChatGPT.pdfJay Das
With the advent of artificial intelligence or AI tools, project management processes are undergoing a transformative shift. By using tools like ChatGPT, and Bard organizations can empower their leaders and managers to plan, execute, and monitor projects more effectively.
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns
Unlocking Business Potential: Tailored Technology Solutions by Prosigns
Discover how Prosigns, a leading technology solutions provider, partners with businesses to drive innovation and success. Our presentation showcases our comprehensive range of services, including custom software development, web and mobile app development, AI & ML solutions, blockchain integration, DevOps services, and Microsoft Dynamics 365 support.
Custom Software Development: Prosigns specializes in creating bespoke software solutions that cater to your unique business needs. Our team of experts works closely with you to understand your requirements and deliver tailor-made software that enhances efficiency and drives growth.
Web and Mobile App Development: From responsive websites to intuitive mobile applications, Prosigns develops cutting-edge solutions that engage users and deliver seamless experiences across devices.
AI & ML Solutions: Harnessing the power of Artificial Intelligence and Machine Learning, Prosigns provides smart solutions that automate processes, provide valuable insights, and drive informed decision-making.
Blockchain Integration: Prosigns offers comprehensive blockchain solutions, including development, integration, and consulting services, enabling businesses to leverage blockchain technology for enhanced security, transparency, and efficiency.
DevOps Services: Prosigns' DevOps services streamline development and operations processes, ensuring faster and more reliable software delivery through automation and continuous integration.
Microsoft Dynamics 365 Support: Prosigns provides comprehensive support and maintenance services for Microsoft Dynamics 365, ensuring your system is always up-to-date, secure, and running smoothly.
Learn how our collaborative approach and dedication to excellence help businesses achieve their goals and stay ahead in today's digital landscape. From concept to deployment, Prosigns is your trusted partner for transforming ideas into reality and unlocking the full potential of your business.
Join us on a journey of innovation and growth. Let's partner for success with Prosigns.
We describe the deployment and use of Globus Compute for remote computation. This content is aimed at researchers who wish to compute on remote resources using a unified programming interface, as well as system administrators who will deploy and operate Globus Compute services on their research computing infrastructure.
Experience our free, in-depth three-part Tendenci Platform Corporate Membership Management workshop series! In Session 1 on May 14th, 2024, we began with an Introduction and Setup, mastering the configuration of your Corporate Membership Module settings to establish membership types, applications, and more. Then, on May 16th, 2024, in Session 2, we focused on binding individual members to a Corporate Membership and Corporate Reps, teaching you how to add individual members and assign Corporate Representatives to manage dues, renewals, and associated members. Finally, on May 28th, 2024, in Session 3, we covered questions and concerns, addressing any queries or issues you may have.
For more Tendenci AMS events, check out www.tendenci.com/events
Gamify Your Mind; The Secret Sauce to Delivering Success, Continuously Improv...Shahin Sheidaei
Games are powerful teaching tools, fostering hands-on engagement and fun. But they require careful consideration to succeed. Join me to explore factors in running and selecting games, ensuring they serve as effective teaching tools. Learn to maintain focus on learning objectives while playing, and how to measure the ROI of gaming in education. Discover strategies for pitching gaming to leadership. This session offers insights, tips, and examples for coaches, team leads, and enterprise leaders seeking to teach from simple to complex concepts.
Quarkus Hidden and Forbidden ExtensionsMax Andersen
Quarkus has a vast extension ecosystem and is known for its subsonic and subatomic feature set. Some of these features are not as well known, and some extensions are less talked about, but that does not make them less interesting - quite the opposite.
Come join this talk to see some tips and tricks for using Quarkus and some of the lesser known features, extensions and development techniques.
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Globus
The U.S. Geological Survey (USGS) has made substantial investments in meeting evolving scientific, technical, and policy driven demands on storing, managing, and delivering data. As these demands continue to grow in complexity and scale, the USGS must continue to explore innovative solutions to improve its management, curation, sharing, delivering, and preservation approaches for large-scale research data. Supporting these needs, the USGS has partnered with the University of Chicago-Globus to research and develop advanced repository components and workflows leveraging its current investment in Globus. The primary outcome of this partnership includes the development of a prototype enterprise repository, driven by USGS Data Release requirements, through exploration and implementation of the entire suite of the Globus platform offerings, including Globus Flow, Globus Auth, Globus Transfer, and Globus Search. This presentation will provide insights into this research partnership, introduce the unique requirements and challenges being addressed and provide relevant project progress.
Check out the webinar slides to learn more about how XfilesPro transforms Salesforce document management by leveraging its world-class applications. For more details, please connect with sales@xfilespro.com
If you want to watch the on-demand webinar, please click here: https://www.xfilespro.com/webinars/salesforce-document-management-2-0-smarter-faster-better/
Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics.
To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...informapgpstrackings
Keep tabs on your field staff effortlessly with Informap Technology Centre LLC. Real-time tracking, task assignment, and smart features for efficient management. Request a live demo today!
For more details, visit us : https://informapuae.com/field-staff-tracking/
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
Enhancing Research Orchestration Capabilities at ORNL.pdfGlobus
Cross-facility research orchestration comes with ever-changing constraints regarding the availability and suitability of various compute and data resources. In short, a flexible data and processing fabric is needed to enable the dynamic redirection of data and compute tasks throughout the lifecycle of an experiment. In this talk, we illustrate how we easily leveraged Globus services to instrument the ACE research testbed at the Oak Ridge Leadership Computing Facility with flexible data and task orchestration capabilities.
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoamtakuyayamamoto1800
In this slide, we show the simulation example and the way to compile this solver.
In this solver, the Helmholtz equation can be solved by helmholtzFoam. Also, the Helmholtz equation with uniformly dispersed bubbles can be simulated by helmholtzBubbleFoam.
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...Juraj Vysvader
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
Developing Distributed High-performance Computing Capabilities of an Open Sci...Globus
COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among public health practitioners, mathematical modelers, and scientific computing specialists, while revealing critical gaps in exploiting advanced computing systems to support urgent decision making. Informed by our team’s work in applying high-performance computing in support of public health decision makers during the COVID-19 pandemic, we present how Globus technologies are enabling the development of an open science platform for robust epidemic analysis, with the goal of collaborative, secure, distributed, on-demand, and fast time-to-solution analyses to support public health.
3. Tensorflow in production with AWS lambda
What Will You Learn?
▸ What can you do with your trained model: MLOPS
▸ Why AWS lambda can be a solution
▸ AWS lambda with tensor flow: how it works
4. Tensorflow in production with AWS lambda
About Me
▸ Freelance Data Products Developper and Consultant
(data visualization, machine learning)
▸ Former Orange Labs and Locarise
(connected sensors data processing and visualization)
▸ Current side project denryoku.io an API for electric grid
power demand and capacity prediction.
6. Tensorflow in production with AWS lambda
It is a product, not an ad-hoc analysis
Live Data
Historical Data
" "
Trained model Deployed model Prediction
Model selection and training
Production
▸ Needs to run on live data
7. Tensorflow in production with AWS lambda
Many things may need to be done in production
▸ Batch processing
▸ Stream / event processing
▸ A prediction API
▸ Update and maintain the model
8. Tensorflow in production with AWS lambda
This needs to be scalable, resilient
And also:
▸ maintainable
▸ versioned
▸ easy to integrate
ML+DevOps = MLOPS
10. Tensorflow in production with AWS lambda
Some deployment solutions
▸ Tensor flow Serving:
▸ Forces you to create dedicated code if you have more than
a pure Tensorflow model
▸ doesn’t solve scalability issues
▸ forces you to manage servers
▸ Google CloudML
▸ Private Beta
▸ Likely limitations
11. Tensorflow in production with AWS lambda
Serverless architectures with AWS Lambda
▸ Serverless offer by AWS
▸ No lifecycle to manage or shared state => resilient
▸ Auto-scaling
▸ Pay for actual running time: low cost
▸ No server, infra management: reduced dev / devops cost
…events
lambda function
output
15. Tensorflow in production with AWS lambda
Event / microbatch processing
▸ event based: db/stream update, new file on s3, web hook
▸ classify the incoming data or update your prediction
16. Tensorflow in production with AWS lambda
Batch processing
▸ cron scheduling
▸ let your function get some data and process it at regular interval
17. Tensorflow in production with AWS lambda
An API
▸ on API call
▸ returned response is your function return value
▸ manage API keys, rate limits, etc on AWS gateway
19. Tensorflow in production with AWS lambda
How to save a TF model
▸ Use a saver object.
▸ It will save on disk:
▸ the graph model (‘filename.meta’)
▸ the variable values (‘filename’)
▸ Need to identify the placeholders that will be accessed later
saver = tf.train.Saver()
#
# do the training
#
tf.add_to_collection('output', pred)
tf.add_to_collection('input', x)
save_path = saver.save(sess, "model-name.ckpt")
python
20. Tensorflow in production with AWS lambda
How to restore a TF model
▸ Restore the graph and variable values with a saver object
saver = tf.train.import_meta_graph(filename + '.meta')
with tf.Session() as sess:
# Restore variables from disk.
saver.restore(sess, filename)
pred = tf.get_collection('output')[0]
x = tf.get_collection('input')[0]
print("Model restored.")
# Do some work with the model
prediction = pred.eval({x: test_data})
python
21. Tensorflow in production with AWS lambda
Setting up AWS Lambda for Tensorflow
Tensorflow needs to be compiled for the right environment
# install compilation environment
sudo yum -y update
sudo yum -y upgrade
sudo yum groupinstall "Development Tools"
# create and activate virtual env
virtualenv tfenv
source tfenv/bin/activate
# install tensorflow
export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/
linux/cpu/tensorflow-0.10.0-cp27-none-linux_x86_64.whl
pip install --upgrade $TF_BINARY_URL
# zip the environment content
touch ~/tfenv/lib/python2.7/site-packages/google/__init__.py
cd ~/tfenv/lib/python2.7/site-packages/
zip -r ~/tf-env.zip . --exclude *.pyc
cd ~/tfenv/lib64/python2.7/site-packages/
1. Launch an
EC2 instance
and connect
to it
2. Install
TensorFlow in
a virtualenv
3. Zip the
installed
libraries
shell
22. Tensorflow in production with AWS lambda
A tensorflow calling lambda function
▸ Accepts a list of input vectors: multiple predictions
▸ Returns a list of predictions
import tensorflow as tf
filename = 'model-name.ckpt'
def lambda_handler(event, context):
saver = tf.train.import_meta_graph(filename + '.meta')
inputData = event['data']
with tf.Session() as sess:
# Restore variables from disk.
saver.restore(sess, filename)
pred = tf.get_collection('pred')[0]
x = tf.get_collection('x')[0]
# Apply the model to the input data
predictions = pred.eval({x: inputData})
return {'result': predictions.tolist()}
python
23. Tensorflow in production with AWS lambda
upload and test
▸ add your lambda function code and TF model to the environment zip.
▸ upload your function
24. Tensorflow in production with AWS lambda
Where to put the model?
▸ with the function: easy, in particular when testing
▸ on s3: ease update or allows for multiple models to be used in
parallel.
▸ function could be called with model ref as argument
"…
lambda function
tensor flowlive data
prediction
$model
25. Tensorflow in production with AWS lambda
caveats
▸ No GPU support at the moment
▸ model loading time: better to increase machine RAM
(hence CPU) for fast API response time
▸ python 2.7 (python 3 doable with more work)
▸ request limit increase to AWS for more than 100
concurrent executions