• What is AWS Elastic Beanstalk (EB)?
• What are the advantages of using EB over managing EC2 instances / Load-balancing / Auto-scaling myself?
• What are some common issues I might run into when deploying my Django app to EB?
The AWS SDK for Go is now live and ready for developers to build productive applications with using the Go programming language. This session will show you how to get started using the AWS SDK for Go. We’ll introduce the best practices for how to use the SDK, and how it integrates with your Go application. We will explore an example application which uses async programming with Amazon S3 and Amazon SQS to distribute tasks across multiple concurrent goroutines, and use Amazon Elastic Beanstalk to deploy and scale the application.
Automating aws infrastructure and code deployments using Ansible @WebEngageVishal Uderani
In this talk , we’ll cover how and why Ansible was leveraged to automate routine management of EC2 instances/EBS/EIP/ELB etc and why the Ansible approach towards automation is key for code and system deployments across 100’s of nodes and how we achieved this at Webengage. We will provide an overview of the deployment process and give a demonstration as an example
Outlines :
How ansible is a straightforward , easy way to manage multiple cloud resources
Intended Audience :
Previous experience with configuration management systems
Previous experience with AWS and Ansible
The AWS SDK for Go is now live and ready for developers to build productive applications with using the Go programming language. This session will show you how to get started using the AWS SDK for Go. We’ll introduce the best practices for how to use the SDK, and how it integrates with your Go application. We will explore an example application which uses async programming with Amazon S3 and Amazon SQS to distribute tasks across multiple concurrent goroutines, and use Amazon Elastic Beanstalk to deploy and scale the application.
Automating aws infrastructure and code deployments using Ansible @WebEngageVishal Uderani
In this talk , we’ll cover how and why Ansible was leveraged to automate routine management of EC2 instances/EBS/EIP/ELB etc and why the Ansible approach towards automation is key for code and system deployments across 100’s of nodes and how we achieved this at Webengage. We will provide an overview of the deployment process and give a demonstration as an example
Outlines :
How ansible is a straightforward , easy way to manage multiple cloud resources
Intended Audience :
Previous experience with configuration management systems
Previous experience with AWS and Ansible
How to use Ansible to automate your applications in AWS. What is Ansible and why is it different? How to control cloud deployments securely and how to control AWS resources using dynamic inventory and tags.
A lot has happened this year in the world of hosting Plone sites. This 2014 Plone Conference session aims to provide a forum for sharing information and debating approaches. We will begin with brief presentations from our panelists, followed by questions and discussion.
- Steve McMahon: Ansible
- Cris Ewing: AWS OpsWorks
- Sven Strack: Nix, Docker, OpenVZ
- Nejc Zupan: Heroku
- Nate Aune: OpenShift, Dotcloud, and other PaaS providers
(WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014Amazon Web Services
Migrating from a hosted environment to AWS is a good opportunity to streamline deployment and site operations. This session shows how FillZ used AWS OpsWorks with other tools to automate site operations and deliver a highly available site that is used by large numbers of customers. Through code and examples, this session shows you how to automate deployments across an entire fleet, configure a patching strategy, use common tools to create useful alarms and monitor system performance, and employ security best-practices in AWS.
Ansible Intro - June 2015 / Ansible Barcelona User GroupOrestes Carracedo
Brief intro to Ansible for the first Ansible Barcelona User Group meetup in June 2015.
http://www.meetup.com/Ansible-Barcelona/events/222305386/
http://ansible-barcelona.github.io
Ansible: How to Get More Sleep and Require Less CoffeeSarah Z
Why you need automation, configuration management and remote execution in your life. An intro to Ansible and how it can make your life in Ops infinitely easier.
DevOps for Humans - Ansible for Drupal Deployment Victory!Jeff Geerling
Everyone knows it's a Good Idea™ to use a configuration management system (e.g. Puppet, Chef) to manage your Drupal infrastructure. But many people (myself included) have run into a wall of #wtfmoments when trying to learn the vagaries of traditional CM systems and their vendor-specific syntaxes.
In 2012, Ansible was released, enabling normal human beings to manage their servers with an easy, but powerful, CM system that uses YAML (just like Drupal 8!) to define configuration and Jinja2 (very much like Twig!) for templates. Not only that, but Ansible is also an incredibly simple and very flexible Drupal deployment and continuous delivery tool.
Learn how you can use Ansible to manage your infrastructure—including local development environments—and stop letting servers and deployments get in the way of development.
Chasing AMI - Building Amazon machine images with Puppet, Packer and JenkinsTomas Doran
Using puppet when configuring EC2 machines seems a natural fit. However bringing up new machines from a community image with puppet is not trivial and can be slow, and so not useful for auto-scaling.
The cloud also offers a solution to ongoing server maintenance, allowing you to launch fresh instances whenever you upgrade your applications (Immutable or Phoenix servers). However to predictably succeed, you need to freeze the puppet code alongside the application version for deployment.
The solution to these issues is generating custom machine images (AMIs) with your software inlined. This talk will cover Yelp's use of a Packer, Jenkins and Puppet for generating AMIs. This will include how we deal with issues like bootstrapping, getting canonical information about a machine's environment and cluster state at launch time, as well as supporting immutable/phoenix servers in combination with more traditional long lived servers inside our hybrid cloud infrastructure.
Drupal VM for Drupal 8 Dev - MidCamp 2017Jeff Geerling
These slides were used in my presentation "Developing for Drupal 8 with Drupal VM", given at MidCamp in Chicago, IL on 2017-04-01.
Learn how to build a modern Drupal 8 website using Composer and Drupal VM for local and prod!
Presented at All Things Open, Raleigh NC, October 2014. Why do people love Ansible for automation? Good question! We walked through several Ansible use cases.
This presentation covers deploying Django in production server. The topics that are mainly covered includes
O Environment setup
O Server configuration
O Deploying in AWS
How to use Ansible to automate your applications in AWS. What is Ansible and why is it different? How to control cloud deployments securely and how to control AWS resources using dynamic inventory and tags.
A lot has happened this year in the world of hosting Plone sites. This 2014 Plone Conference session aims to provide a forum for sharing information and debating approaches. We will begin with brief presentations from our panelists, followed by questions and discussion.
- Steve McMahon: Ansible
- Cris Ewing: AWS OpsWorks
- Sven Strack: Nix, Docker, OpenVZ
- Nejc Zupan: Heroku
- Nate Aune: OpenShift, Dotcloud, and other PaaS providers
(WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014Amazon Web Services
Migrating from a hosted environment to AWS is a good opportunity to streamline deployment and site operations. This session shows how FillZ used AWS OpsWorks with other tools to automate site operations and deliver a highly available site that is used by large numbers of customers. Through code and examples, this session shows you how to automate deployments across an entire fleet, configure a patching strategy, use common tools to create useful alarms and monitor system performance, and employ security best-practices in AWS.
Ansible Intro - June 2015 / Ansible Barcelona User GroupOrestes Carracedo
Brief intro to Ansible for the first Ansible Barcelona User Group meetup in June 2015.
http://www.meetup.com/Ansible-Barcelona/events/222305386/
http://ansible-barcelona.github.io
Ansible: How to Get More Sleep and Require Less CoffeeSarah Z
Why you need automation, configuration management and remote execution in your life. An intro to Ansible and how it can make your life in Ops infinitely easier.
DevOps for Humans - Ansible for Drupal Deployment Victory!Jeff Geerling
Everyone knows it's a Good Idea™ to use a configuration management system (e.g. Puppet, Chef) to manage your Drupal infrastructure. But many people (myself included) have run into a wall of #wtfmoments when trying to learn the vagaries of traditional CM systems and their vendor-specific syntaxes.
In 2012, Ansible was released, enabling normal human beings to manage their servers with an easy, but powerful, CM system that uses YAML (just like Drupal 8!) to define configuration and Jinja2 (very much like Twig!) for templates. Not only that, but Ansible is also an incredibly simple and very flexible Drupal deployment and continuous delivery tool.
Learn how you can use Ansible to manage your infrastructure—including local development environments—and stop letting servers and deployments get in the way of development.
Chasing AMI - Building Amazon machine images with Puppet, Packer and JenkinsTomas Doran
Using puppet when configuring EC2 machines seems a natural fit. However bringing up new machines from a community image with puppet is not trivial and can be slow, and so not useful for auto-scaling.
The cloud also offers a solution to ongoing server maintenance, allowing you to launch fresh instances whenever you upgrade your applications (Immutable or Phoenix servers). However to predictably succeed, you need to freeze the puppet code alongside the application version for deployment.
The solution to these issues is generating custom machine images (AMIs) with your software inlined. This talk will cover Yelp's use of a Packer, Jenkins and Puppet for generating AMIs. This will include how we deal with issues like bootstrapping, getting canonical information about a machine's environment and cluster state at launch time, as well as supporting immutable/phoenix servers in combination with more traditional long lived servers inside our hybrid cloud infrastructure.
Drupal VM for Drupal 8 Dev - MidCamp 2017Jeff Geerling
These slides were used in my presentation "Developing for Drupal 8 with Drupal VM", given at MidCamp in Chicago, IL on 2017-04-01.
Learn how to build a modern Drupal 8 website using Composer and Drupal VM for local and prod!
Presented at All Things Open, Raleigh NC, October 2014. Why do people love Ansible for automation? Good question! We walked through several Ansible use cases.
This presentation covers deploying Django in production server. The topics that are mainly covered includes
O Environment setup
O Server configuration
O Deploying in AWS
Docker containers are an excellent solution to many problems: they can greatly simplify your deployment process, they provide an alternative approach to configuration management, and they offer excellent consistency and portability. However, running production environments on Docker containers can be challenging and requires careful consideration of various aspects such as high availability, scalability, storage persistence and more. In this webinar we will address some of the prominent issues you might encounter when designing Docker-based production environments on AWS, and present possible solutions to these issues.
In this session, we will discuss the difficulties of running Docker in production and how Amazon EC2 Container Service (ECS) can be used to reduce the operational burdens. We will give an overview of the core architectural principles underlying Amazon ECS, and we will walk through a number of patterns used by our customers to run their microservices platforms and batch applications.
Apache Spark Clusters for Everyone | AWS Public Sector Summit 2016Amazon Web Services
Easy Access to Amazon EMR Spark Clusters Using R and Python in the Browser Using systems like Apache Spark, big data analysis is becoming more accessible from high-level languages like R and Python. However, many analysts are unprepared for the challenges of setting up a big data analytical environment. In this talk, we outline a process that allows anyone in an organization to quickly spin up elastic Spark clusters, then analyze data through RStudio and SparkR, or alternatively pySpark and Jupyter Notebooks. The resulting system is affordable, powerful, and incredibly accessible: It takes just two clicks and a 15-minute wait time for analysts to each have their own cluster. This session will cover the following: the Amazon EMR bootstrap process for installations of high-level languages to work on top of Spark (specifically SparkR and pySpark); dynamic port forwarding with SSH and Foxy Proxy for browser access to RStudio and Jupyter; convenient data loading from Amazon S3 to EMR without leaving RStudio or Jupyter; and automating the startup process for nontechnical data analysts and researchers. Finally, we will share some short studies that demonstrate the power of Spark with EMR to interactively analyze massive datasets and discern public policy insights.
(APP313) NEW LAUNCH: Amazon EC2 Container Service in Action | AWS re:Invent 2014Amazon Web Services
Container technology, particularly Docker, is all the rage these days. At AWS, our customers have been running Linux containers at scale for several years, and we are increasingly seeing customers adopt Docker, especially as they build loosely coupled distributed applications. However, to do so they have to run their own cluster management solutions, deal with configuration management, and manage their containers and associated metadata. We believe that those capabilities should be a core building block technology, just like EC2. Today, we are announcing the preview of Amazon EC2 Container Service, a new AWS service that makes is easy to run and manage Docker-enabled distributed applications using powerful APIs that allow you to launch and stop containers, get complete cluster state information, and manage linked containers. In this session we will discuss why we built the EC2 Container Service, some of the core concepts, and walk you through how you can use the service for your applications.
Apache Kafka, and the Rise of Stream ProcessingGuozhang Wang
For a long time, a substantial portion of data processing that companies did ran as big batch jobs. But businesses operate in real-time and the software they run is catching up. Today, processing data in a streaming fashion becomes more and more popular in many companies over the more "traditional" way of batch-processing big data sets available as a whole.
Demystifying Stream Processing with Apache Kafkaconfluent
www.confluent.io/online-talks/
This talk will introduce Kafka Streams and help you understand how to map practical data problems to stream processing and how to write applications that process streams of data at scale using Kafka Streams. We will also cover what is stream processing, why one should care about stream processing, where Apache Kafka and Kafka Streams fit in, the hard parts of stream processing, and how Kafka Streams solves those problems; along with a concrete example of how these ideas tie together in Kafka Streams and in the big picture of your data center.
In this presentation we describe the design and implementation of Kafka Connect, Kafka’s new tool for scalable, fault-tolerant data import and export. First we’ll discuss some existing tools in the space and why they fall short when applied to data integration at large scale. Next, we will explore Kafka Connect’s design and how it compares to systems with similar goals, discussing key design decisions that trade off between ease of use for connector developers, operational complexity, and reuse of existing connectors. Finally, we’ll discuss how standardizing on Kafka Connect can ultimately lead to simplifying your entire data pipeline, making ETL into your data warehouse and enabling stream processing applications as simple as adding another Kafka connector.
AWS re:Invent 2016: Workshop: Deploy a Swift Web Application on Amazon ECS (C...Amazon Web Services
In this workshop, you learn how to develop a mobile front-end using Swift, and develop a Swift microservices-based web application to be deployed on Amazon ECS.
Workshop Requirements:
Mac Laptop
XCode 7 or 8
AWS Account
IAM admin privileges for AWS account with full access to EC2, ECS, ECR, Code Pipeline, Mobile Hub and Device Farm. (If you choose to create a new account, please do it 24 hrs ahead.)
Apache Kafka is an open-source message broker project developed by the Apache Software Foundation written in Scala. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.
This is the first part of the presentation.
Here is the 2nd part of this presentation:-
http://www.slideshare.net/knoldus/introduction-to-apache-kafka-part-2
5 Reasons Why You Should Consider Migrating Web Apps to the Cloud on AWSrivetlogic
More and more, organizations are turning to the Cloud to reduce cost and liability, increase performance and develop greater flexibility to provide services for their mission critical enterprise deployments.
This presentation covers the benefits of migrating web applications to the Cloud leveraging Amazon Web Services (AWS), and the positive impact it can have on your business, including:
- Reduced costs
- Increased operational efficiency
- Flexibility and scalability
- Increased security and reduced risk
- Freeing up IT resources to focus on strategic projects
- Benefits of Managed Services
You will learn:
• How to use Elastic Beanstalk
• How to run Single and Multi-Container Docker with AWS Elastic Beanstalk
• Best Practices running AWS Elastic Beanstalk
O serviço Elastic Beanstalk da AWS permite que voce automatize o lançamento de aplicativos e sites inteiros na nuvem da AWS. Nesta apresentação, feita durante o AWS Summit 2015, mostramos como usá-lo.
AD113 Speed Up Your Applications w/ Nginx and PageSpeededm00se
My slide deck from my session, AD113: Speed Up Your Applications with Nginx + PageSpeed, at MWLUG 2015 in Atlanta, GA at the Ritz-Carlton.
For more, see:
- https://edm00se.io/self-promotion/mwlug-ad113-success
- https://github.com/edm00se/AD113-Speed-Up-Your-Apps-with-Nginx-and-PageSpeed
AWS July Webinar Series-Deploying and Scaling Web Application with AWS Elasti...Amazon Web Services
Want to easily deploy your Node.js, Ruby, Python, .NET, Tomcat, PHP or Docker web applications to AWS?
AWS Elastic Beanstalk is a service that makes it easy to deploy, scale, and manage your web applications on AWS by providing preconfigured application stacks and managing the underlying infrastructure on your behalf. You simply select the application stack you want and upload your code to get started.
This webinar will familiarize you with how to deploy your web application to AWS using AWS Elastic Beanstalk (EB). We will walk through the steps necessary to debug, test and scale your application to handle millions of web requests. Also, you will learn about deployment options, cost management, and ongoing monitoring and maintenance.
Learning Objectives:
Understand the benefits of AWS Elastic Beanstalk versus do-it-yourself
Deploy a sample Node.js application using the Elastic Beanstalk command line interface
Modify application stack configuration and extend your application to use additional AWS resources (e.g.: DynamoDB, SNS, etc.)
Debug, load test, and scale the sample application to handle millions of web requests
Use deployment options available for zero downtime deployments (in-place and blug/green)
Use tags for cost management
Azure Templates for Consistent DeploymentJosé Maia
Presented at NetPonto Porto 09/September/2017, by José Maia, at Farfetch.
A talk about how to use Azure Resource Manager to follow good DevOps practices.
Deploy, Scale and Manage your Application with AWS Elastic BeanstalkAmazon Web Services
AWS Elastic Beanstalk provides an easy way to quickly deploy, manage, and scale applications in the AWS cloud. Through interactive demos, this session will discuss the best practices for deploying and scaling your application, provisioning additional AWS resources and performance tuning. We will also do a deep dive into the recently launched Elastic Beanstalk features and cover some of best practices for using Elastic Beanstalk. This session will benefit both new and experienced users of Elastic Beanstalk.
Experience our free, in-depth three-part Tendenci Platform Corporate Membership Management workshop series! In Session 1 on May 14th, 2024, we began with an Introduction and Setup, mastering the configuration of your Corporate Membership Module settings to establish membership types, applications, and more. Then, on May 16th, 2024, in Session 2, we focused on binding individual members to a Corporate Membership and Corporate Reps, teaching you how to add individual members and assign Corporate Representatives to manage dues, renewals, and associated members. Finally, on May 28th, 2024, in Session 3, we covered questions and concerns, addressing any queries or issues you may have.
For more Tendenci AMS events, check out www.tendenci.com/events
Listen to the keynote address and hear about the latest developments from Rachana Ananthakrishnan and Ian Foster who review the updates to the Globus Platform and Service, and the relevance of Globus to the scientific community as an automation platform to accelerate scientific discovery.
Understanding Globus Data Transfers with NetSageGlobus
NetSage is an open privacy-aware network measurement, analysis, and visualization service designed to help end-users visualize and reason about large data transfers. NetSage traditionally has used a combination of passive measurements, including SNMP and flow data, as well as active measurements, mainly perfSONAR, to provide longitudinal network performance data visualization. It has been deployed by dozens of networks world wide, and is supported domestically by the Engagement and Performance Operations Center (EPOC), NSF #2328479. We have recently expanded the NetSage data sources to include logs for Globus data transfers, following the same privacy-preserving approach as for Flow data. Using the logs for the Texas Advanced Computing Center (TACC) as an example, this talk will walk through several different example use cases that NetSage can answer, including: Who is using Globus to share data with my institution, and what kind of performance are they able to achieve? How many transfers has Globus supported for us? Which sites are we sharing the most data with, and how is that changing over time? How is my site using Globus to move data internally, and what kind of performance do we see for those transfers? What percentage of data transfers at my institution used Globus, and how did the overall data transfer performance compare to the Globus users?
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTier1 app
Even though at surface level ‘java.lang.OutOfMemoryError’ appears as one single error; underlyingly there are 9 types of OutOfMemoryError. Each type of OutOfMemoryError has different causes, diagnosis approaches and solutions. This session equips you with the knowledge, tools, and techniques needed to troubleshoot and conquer OutOfMemoryError in all its forms, ensuring smoother, more efficient Java applications.
Quarkus Hidden and Forbidden ExtensionsMax Andersen
Quarkus has a vast extension ecosystem and is known for its subsonic and subatomic feature set. Some of these features are not as well known, and some extensions are less talked about, but that does not make them less interesting - quite the opposite.
Come join this talk to see some tips and tricks for using Quarkus and some of the lesser known features, extensions and development techniques.
Check out the webinar slides to learn more about how XfilesPro transforms Salesforce document management by leveraging its world-class applications. For more details, please connect with sales@xfilespro.com
If you want to watch the on-demand webinar, please click here: https://www.xfilespro.com/webinars/salesforce-document-management-2-0-smarter-faster-better/
Accelerate Enterprise Software Engineering with PlatformlessWSO2
Key takeaways:
Challenges of building platforms and the benefits of platformless.
Key principles of platformless, including API-first, cloud-native middleware, platform engineering, and developer experience.
How Choreo enables the platformless experience.
How key concepts like application architecture, domain-driven design, zero trust, and cell-based architecture are inherently a part of Choreo.
Demo of an end-to-end app built and deployed on Choreo.
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
Developing Distributed High-performance Computing Capabilities of an Open Sci...Globus
COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among public health practitioners, mathematical modelers, and scientific computing specialists, while revealing critical gaps in exploiting advanced computing systems to support urgent decision making. Informed by our team’s work in applying high-performance computing in support of public health decision makers during the COVID-19 pandemic, we present how Globus technologies are enabling the development of an open science platform for robust epidemic analysis, with the goal of collaborative, secure, distributed, on-demand, and fast time-to-solution analyses to support public health.
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...Globus
Large Language Models (LLMs) are currently the center of attention in the tech world, particularly for their potential to advance research. In this presentation, we'll explore a straightforward and effective method for quickly initiating inference runs on supercomputers using the vLLM tool with Globus Compute, specifically on the Polaris system at ALCF. We'll begin by briefly discussing the popularity and applications of LLMs in various fields. Following this, we will introduce the vLLM tool, and explain how it integrates with Globus Compute to efficiently manage LLM operations on Polaris. Attendees will learn the practical aspects of setting up and remotely triggering LLMs from local machines, focusing on ease of use and efficiency. This talk is ideal for researchers and practitioners looking to leverage the power of LLMs in their work, offering a clear guide to harnessing supercomputing resources for quick and effective LLM inference.
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...Juraj Vysvader
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
SOCRadar Research Team: Latest Activities of IntelBrokerSOCRadar
The European Union Agency for Law Enforcement Cooperation (Europol) has suffered an alleged data breach after a notorious threat actor claimed to have exfiltrated data from its systems. Infamous data leaker IntelBroker posted on the even more infamous BreachForums hacking forum, saying that Europol suffered a data breach this month.
The alleged breach affected Europol agencies CCSE, EC3, Europol Platform for Experts, Law Enforcement Forum, and SIRIUS. Infiltration of these entities can disrupt ongoing investigations and compromise sensitive intelligence shared among international law enforcement agencies.
However, this is neither the first nor the last activity of IntekBroker. We have compiled for you what happened in the last few days. To track such hacker activities on dark web sources like hacker forums, private Telegram channels, and other hidden platforms where cyber threats often originate, you can check SOCRadar’s Dark Web News.
Stay Informed on Threat Actors’ Activity on the Dark Web with SOCRadar!
Large Language Models and the End of ProgrammingMatt Welsh
Talk by Matt Welsh at Craft Conference 2024 on the impact that Large Language Models will have on the future of software development. In this talk, I discuss the ways in which LLMs will impact the software industry, from replacing human software developers with AI, to replacing conventional software with models that perform reasoning, computation, and problem-solving.
How to Position Your Globus Data Portal for Success Ten Good PracticesGlobus
Science gateways allow science and engineering communities to access shared data, software, computing services, and instruments. Science gateways have gained a lot of traction in the last twenty years, as evidenced by projects such as the Science Gateways Community Institute (SGCI) and the Center of Excellence on Science Gateways (SGX3) in the US, The Australian Research Data Commons (ARDC) and its platforms in Australia, and the projects around Virtual Research Environments in Europe. A few mature frameworks have evolved with their different strengths and foci and have been taken up by a larger community such as the Globus Data Portal, Hubzero, Tapis, and Galaxy. However, even when gateways are built on successful frameworks, they continue to face the challenges of ongoing maintenance costs and how to meet the ever-expanding needs of the community they serve with enhanced features. It is not uncommon that gateways with compelling use cases are nonetheless unable to get past the prototype phase and become a full production service, or if they do, they don't survive more than a couple of years. While there is no guaranteed pathway to success, it seems likely that for any gateway there is a need for a strong community and/or solid funding streams to create and sustain its success. With over twenty years of examples to draw from, this presentation goes into detail for ten factors common to successful and enduring gateways that effectively serve as best practices for any new or developing gateway.
Unleash Unlimited Potential with One-Time Purchase
BoxLang is more than just a language; it's a community. By choosing a Visionary License, you're not just investing in your success, you're actively contributing to the ongoing development and support of BoxLang.
How Recreation Management Software Can Streamline Your Operations.pptxwottaspaceseo
Recreation management software streamlines operations by automating key tasks such as scheduling, registration, and payment processing, reducing manual workload and errors. It provides centralized management of facilities, classes, and events, ensuring efficient resource allocation and facility usage. The software offers user-friendly online portals for easy access to bookings and program information, enhancing customer experience. Real-time reporting and data analytics deliver insights into attendance and preferences, aiding in strategic decision-making. Additionally, effective communication tools keep participants and staff informed with timely updates. Overall, recreation management software enhances efficiency, improves service delivery, and boosts customer satisfaction.
Software Engineering, Software Consulting, Tech Lead.
Spring Boot, Spring Cloud, Spring Core, Spring JDBC, Spring Security,
Spring Transaction, Spring MVC,
Log4j, REST/SOAP WEB-SERVICES.
1. Scaling Django Apps using
AWS Elastic Beanstalk
Oct 22nd, 2015
Lushen Wu
Founder, Bebop
http://bebop.com
2. What we will cover
• What is AWS Elastic Beanstalk (EB)?
• What are the advantages of using EB over
managing EC2 instances / Load-balancing / Auto-
scaling myself?
• What are some common issues I might run into
when deploying my Django app to EB?
4. Scaling option 1: manage it yourself
Amazon RDS (database)
Amazon S3 (static assets)
Amazon EC2
(cloud computing)
Amazon EC2
(cloud computing)
Amazon EC2
(cloud computing)
User User User User User User
Instances: either manually
boot up instances or configure
an Auto-scaling policy
Load-balancing: point your
DNS host to the Load
Balancer’s IP address instead
of an EC2 public IP
Use a deployment tool like
Fabric to deploy new
application code to multiple
EC2 instances (e.g. run git pull
on every instance)
+
Load-balancing
Auto-scaling
5. Core concepts primer
What is AWS auto-scaling?
• AWS monitors your EC2 instances
based on metrics that you specify
(e.g. CPU usage, RAM usage, # of
HTTP requests per minute)
• AWS adds or terminates EC2
instances based on your policy
thresholds (e.g. add instance if
average CPU >60% for 5 minutes,
remove if average CPU < 20% for 5
minutes)
What is AWS load-balancing?
• Distributes incoming requests
among multiple EC2 instances
• Usually routes traffic to instance
with lowest CPU load but you can
specify other policies (e.g. latency)
6. Scaling option 2: AWS Elastic Beanstalk
Amazon RDS (database)
Amazon S3 (static assets)
Amazon EC2
(cloud computing)
Amazon EC2
(cloud computing)
Amazon EC2
(cloud computing)
User User User User User User
EB web console allows
you to easily configure
Load Balancer and
Autoscaling policy
EB command line tool lets
you easily deploy an
application (in this case,
Django code) to all EC2
instances+
Auto-scaling
Load-balancing
Amazon EB
7. Quick aside: thoughts on “scaling”
Why is scaling necessary?
• AWS EB/EC2 supports horizontal
scaling (adding more instances) as
well as vertical scaling (using more
powerful instances)
• For speed: You want your web app
to load quickly, and avoid downtime
• For availability: if one datacenter
goes down, people can still access
your web app
Other things to keep in mind:
• Scaling can be expensive, and many site speed
bottlenecks are not solved by adding or upgrading
(e.g. render-blocking javascript/css,
uncompressed or unnecessarily large static
assets, too many HTTP requests)
• Make sure you optimize the basics, e.g.:
minified/gzipped assets, backend caching (e.g.
memcached), frontend caching (e.g. varnish) with
selective AJAX fetching
• We are not using EB to scale the database!
8. Summary of scaling options
AWS Elastic Beanstalk
• Easier to configure than manually setting up Load Balancer / Autoscaling group
• EB handles source code deployment so you don’t have to use tools like Fabric
Docker
• Briefly looked into it and seemed like EB was sufficient for our needs at the moment
• AWS also supports Docker containers
• Docker just acquired Tutum, potentially easier integrated deployment process
Heroku
• Google search for deploying / scaling Django on Heroku returned some worrying forum posts
Others … ?
9. EB Overview
1. Creating an EB Environment
2. Configuring an EB Environment
• Instances and scaling
• Environment properties
3. Deployment
4. Recap & tricky snags
10. 1. Creating an EB environment
pip install awsebcli
# check we’re good to go
eb --version
# configure eb with your AWS credentials (Access Key ID, Secret Access Key, default
region, etc.)
eb init
# create an EB environment (will add a .elasticbeanstalk and .ebextensions folder in
your project root)
eb create
Getting started with EB
11. 1. Creating an EB environment
Environment type
• Use ‘Web Server’ environment type
for an external-facing web application
• Make sure the Configuration you use
has a matching PostgreSQL version
with your RDS instance (otherwise
you may run into trouble executing
Postgres commands like ‘pgdump’)
12. 1. You can have more than one environment!
• Testing / Staging -> use smallest EC2 size, 1 or 2 instances in group
• Production -> can be larger EC2 size, at least 3 instances in group
• Workers -> integrate with Amazon SQS, do cronjobs, etc.
13. 2. Instances -> scaling -> deployment
EC2 instance management with Elastic Beanstalk
• Pick the type/size of the EC2 instances that EB will automatically launch
• All instances launched by the auto-scaling policy will be of this type/size!
• You can still manage all EC2 instances as you normally would in the AWS web console for EC2
What size should I pick?
• Remember that you are paying for EC2 instances by the hour, and (AFAIK), even a ping or some background
task using CPU will make that instance count as used
• If you spin up a ton of large instances thinking “Just in case I get a ton of traffic” … you’re gonna get a big bill
How many should I get?
• Having 3 means that you can have one instance down/overwhelmed, one instance deploying new application
version, and still have 1 for failover
14. 2. Instances -> scaling -> deployment
Scaling policy
• Specify minimum and maximum # of instances
• Specify triggers to add or terminate EC2 instances (typically based on CPU usage,
RAM usage, or # of HTTP requests)
• Cooldown: minimum amount of time after a scaling event before another scaling
event can happen… prevents “see-sawing” due to traffic micro spikes
Deployment
• Specify batch size (what # or % of instances to deploy to at the same time)
• Maximum # of instances to deploy to simultaneously
• Minimum # of instances in service
• What could be an issue here?
15. 2. EB Environment Properties
A great way to:
• Avoid committing sensitive
credentials (e.g. social API keys,
DB passwords) into source code
• Deploy same source to different
EB environments (e.g. testing /
staging / production may each
target different databases)
16. 3. Deployment Process
pip install awsebcli
# specify which EB environment to target with subsequent commands
eb use {environment-name}
# output status (e.g. whether currently deploying) and health of target EB environment
eb status
# deploy the latest commit in the current git repository
eb deploy
# dumps logs for whole environment (all instances) into a gzipped file in your project folder
eb logs -z
Using EB command line to deploy
17. 3. What actually happens when you deploy?
What happens in your environment during deployment?
1. ‘eb deploy’ creates a zip file of the latest git commit
• The version you are deploying doesn’t need to be pushed to github, it just
needs to be locally committed
• You can use another revision control platform (e.g. mercurial hg) if you want
2. EB uploads zip file to an S3 bucket associated with your Elastic
Beanstalk account, so it’s accessible by the EB environment
3. EB cycles through your instances and deploys the application to
each ‘deployment batch’ (according to your deployment policy)
18. 3. What actually happens when you deploy?
What happens on an instance during deployment?
• EB runs any server commands you specified (e.g. yum install)
• Extracts source of new version into /opt/python/ondeck
• EB runs any container_commands you specified (e.g. manage.py syncdb)
• If an error is encountered, the deployment aborts, no other instances are
attempted, and the current application version remains active
• If no errors are encountered, EB proceed with deployment:
• Links /opt/python/current/ to the /ondeck/ folder above, WSGI updated
• Django project root is always /opt/python/current/app
19. 3. Helping EB find your Django app
• Options specified in .config
files stored in .ebextensions
folder in project root
• Adds application root to
PYTHONPATH
• Points EB to WSGI
application and configure
threading
option_settings:
"aws:elasticbeanstalk:application:environment":
DJANGO_SETTINGS_MODULE: "musicserver.settings"
"PYTHONPATH": "/opt/python/current/app:$PYTHONPATH"
"aws:elasticbeanstalk:container:python":
WSGIPath: musicserver/wsgi.py
NumProcesses: 3
NumThreads: 20
"aws:elasticbeanstalk:container:python:staticfiles":
"/static/": "static/"
Option Settings (Django WSGI config)
20. 3. Customizing server
01_packages.config
Server Commands (this page) and Container Commands (next page)
• Executed in alphabetical order (name them whatever you want)
• You can run arbitrary bash commands, manage linux packages & services, run arbitrary
python code…
More: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html
commands:
set_time_zone:
command: ln -f -s /usr/share/zoneinfo/US/Eastern /etc/localtime
02_timezone.config
packages:
yum:
gcc: []
git: []
postgresql93-devel: []
21. 3. Customizing server
Container Commands
• leader_only means only the first
instance in a deployment will run
the command
• Other useful parameters:
ignoreErrors (true|false), env
(name/value pairs), cwd
• This is also the time to set up
dependencies like running chmod
on executables (targeting the
/ondeck/app folder)
03_django.config
container_commands:
01_migrate:
command: "source /opt/python/run/venv/bin/activate &&
python manage.py migrate --noinput"
leader_only: true
02_collectstatic:
command: "source /opt/python/run/venv/bin/activate &&
python manage.py collectstatic --noinput"
leader_only: true
03_compress:
command: "source /opt/python/run/venv/bin/activate &&
python manage.py compress --force"
22. 4. Recap & tricky snags
• Scaling EC2 instances is only one part of the solution!
• Select the right EC2 image version for EB (with same PostgreSQL version as RDS)
• Watch out for devtools-related deployment errors (e.g. collectstatic or compress)
• Get familiar with the deployment flow
• Zip & upload -> run server commands -> extract to /ondeck folder -> run container
commands -> link from /current folder
• Watch out for deployment batch size and minimum instances settings (avoid
“stalled” deployments)
23. Sources
# SSH invoke Django shell with virtualenv
sudo su
cd /opt/python/current/app
source /opt/python/current/env
source /opt/python/run/venv/bin/activate
python manage.py shell
Bonus snippet:• Google / various blogs
• AWS walkthrough on EB and Django
• About 3 days of head-banging and a
dozen Monster energy drinks in
August 2015