At a blistering pace and for a variety of reasons, companies are migrating their on-premise database infrastructures to cloud-based solutions - to save costs on hardware, tame the impact of disaster recovery, or even to improve security. Zalando is not an exception: more than two years ago we migrated our first production services to AWS.
In addition to the fully managed database services like RDS and Aurora, Amazon offers a wide spectra of EC2 Instances with different types of performance and price. Without a lot of experience in running cloud databases it’s not easy to make the right choice, and as a result you will either have pure database performance or will overpay for over-provisioned resources.
In this talk I will compare different ways of running PostgreSQL on AWS, explain why we decided to run most of our databases on EC2 Instances instead of RDS, how we chose EC2 Instance types and EBS Volumes, which AWS CloudWatch metrics MUST be monitored (and why), and what problems we hit plus how to avoid them.
- 동영상 보기: https://www.youtube.com/watch?v=Rq4I57eqIp4
Amazon RDS 프록시는 Amazon Relational Database Service (RDS)를 위한 완전 관리형 고가용성 데이터베이스 프록시로, 애플리케이션의 확장 성, 데이터베이스 장애에 대한 탄력성 및 보안 성을 향상시킬 수 있습니다. (2020년 6월 서울 리전 출시)
Amazon Kinesis Data Analytics는 실시간으로 스트리밍 데이터를 처리하고 분석할 수 있는 서버리스 서비스입니다. Kinesis Data Analytics를 사용하면 로그 분석, 클릭스트림 분석, 사물 인터넷(IoT), 광고 기술, 게임 등의 대규모의 스트림을 처리할 수 있는 애플리케이션을 신속하고 유연하게 구축할 수 있으며 유지관리의 어려움에서 벗어날 수 있습니다. 이 세션에서는 Kinesis Data Analytics의 동작과 기능, 운영상의 모범 사례에 대한 설명을 바탕으로 Streaming Application 개발, Studio Notebook 활용하는 방법을 데모를 통해 알아봅니다.
In this popular session, discover how Amazon EBS can take your application deployments on Amazon EC2 to the next level. Learn about Amazon EBS features and benefits, how to identify applications that are appropriate for use with Amazon EBS, best practices, snapshots, and details about its performance and volume types. The target audience is storage administrators, application developers, applications owners, and anyone who wants to understand how to optimize performance for Amazon EC2 using the power of Amazon EBS.
AWS 상의 컨테이너 서비스 소개 ECS, EKS - 이종립 / Principle Enterprise Evangelist @베스핀글로벌BESPIN GLOBAL
지난 11월 Bespin Gaming Day 행사의 발표 자료를 통해 AWS에서 컨테이너를 활용하는 방법을 알아보겠습니다.
AWS에서는 사용 목적에 따라 다양한 컨테이너 서비스를 제공합니다.
- Management 스케쥴링, 스케일링, 배포, 전략: Amazon ECS, Amazon EKS
- Hosting 컨테이너가 수행되는 곳: Amazon EC2, AWS Fargate
- Image Registry 컨테이너 이미지 저장소: Amazon ECR
컨테이너 및 쿠버네티스 관련 서비스 중 Amazon EKS, Amazon ECS, AWS Fargate에 대해 보다 자세히 살펴보실 수 있습니다.
Redis is an open source, in-memory data store that delivers sub-millisecond response times enabling millions of requests per second to power real-time applications. It can be used as a fast database, cache, message broker, and queue. Amazon ElastiCache delivers the ease-of-use and power of Redis along with the availability, reliability, scalability, security, and performance suitable for the most demanding applications. We’ll take a close look at Redis and how to use it to power different use cases.
Speaker: Samir Karande - Sr. Manager, Solutions Architecture, AWS
- 동영상 보기: https://www.youtube.com/watch?v=Rq4I57eqIp4
Amazon RDS 프록시는 Amazon Relational Database Service (RDS)를 위한 완전 관리형 고가용성 데이터베이스 프록시로, 애플리케이션의 확장 성, 데이터베이스 장애에 대한 탄력성 및 보안 성을 향상시킬 수 있습니다. (2020년 6월 서울 리전 출시)
Amazon Kinesis Data Analytics는 실시간으로 스트리밍 데이터를 처리하고 분석할 수 있는 서버리스 서비스입니다. Kinesis Data Analytics를 사용하면 로그 분석, 클릭스트림 분석, 사물 인터넷(IoT), 광고 기술, 게임 등의 대규모의 스트림을 처리할 수 있는 애플리케이션을 신속하고 유연하게 구축할 수 있으며 유지관리의 어려움에서 벗어날 수 있습니다. 이 세션에서는 Kinesis Data Analytics의 동작과 기능, 운영상의 모범 사례에 대한 설명을 바탕으로 Streaming Application 개발, Studio Notebook 활용하는 방법을 데모를 통해 알아봅니다.
In this popular session, discover how Amazon EBS can take your application deployments on Amazon EC2 to the next level. Learn about Amazon EBS features and benefits, how to identify applications that are appropriate for use with Amazon EBS, best practices, snapshots, and details about its performance and volume types. The target audience is storage administrators, application developers, applications owners, and anyone who wants to understand how to optimize performance for Amazon EC2 using the power of Amazon EBS.
AWS 상의 컨테이너 서비스 소개 ECS, EKS - 이종립 / Principle Enterprise Evangelist @베스핀글로벌BESPIN GLOBAL
지난 11월 Bespin Gaming Day 행사의 발표 자료를 통해 AWS에서 컨테이너를 활용하는 방법을 알아보겠습니다.
AWS에서는 사용 목적에 따라 다양한 컨테이너 서비스를 제공합니다.
- Management 스케쥴링, 스케일링, 배포, 전략: Amazon ECS, Amazon EKS
- Hosting 컨테이너가 수행되는 곳: Amazon EC2, AWS Fargate
- Image Registry 컨테이너 이미지 저장소: Amazon ECR
컨테이너 및 쿠버네티스 관련 서비스 중 Amazon EKS, Amazon ECS, AWS Fargate에 대해 보다 자세히 살펴보실 수 있습니다.
Redis is an open source, in-memory data store that delivers sub-millisecond response times enabling millions of requests per second to power real-time applications. It can be used as a fast database, cache, message broker, and queue. Amazon ElastiCache delivers the ease-of-use and power of Redis along with the availability, reliability, scalability, security, and performance suitable for the most demanding applications. We’ll take a close look at Redis and how to use it to power different use cases.
Speaker: Samir Karande - Sr. Manager, Solutions Architecture, AWS
Amazon Aurora Storage Demystified: How It All Works (DAT363) - AWS re:Invent ...Amazon Web Services
Amazon Aurora is a high performance, highly scalable database service with MySQL- and PostgreSQL-compatibility. One of its key components is an innovative storage system that is optimized for database workloads and specifically designed to take advantage of modern cloud technology. Hear from the team that built Amazon Aurora's storage system on how the system is designed, how it works, and what you need to know to get the most out of it.
Oracle Real Application Clusters 19c- Best Practices and Internals- EMEA Tour...Sandesh Rao
In this session, I will cover under-the-hood features that power Oracle Real Application Clusters (Oracle RAC) 19c specifically around Cache Fusion and Service management. Improvements in Oracle RAC helps in integration with features such as Multitenant and Data Guard. In fact, these features benefit immensely when used with Oracle RAC. Finally we will talk about changes to the broader Oracle RAC Family of Products stack and the algorithmic changes that helps quickly detect sick/dead nodes/instances and the reconfiguration improvements to ensure that the Oracle RAC Databases continue to function without any disruption
Amazon Aurora is a fully managed relational database engine that combines the speed and availability of high-end commercial databases with the simplicity and cost-effectiveness of open source databases. It is purpose-built for the cloud using a new architectural model and distributed systems techniques to provide far higher performance, availability and durability than previously possible using conventional monolithic database architectures. Amazon Aurora packs a lot of innovations in the engine and storage layers. In this session, we will do a deep-dive into some of the key innovations behind Amazon Aurora, new improvements to Aurora's performance, availability and cost-effectiveness and discuss best practices and optimal configurations.
발표 영상 다시보기: https://kr-resources.awscloud.com/data-databases-and-analytics/%EC%A7%80%EA%B8%88-%EB%8B%B9%EC%9E%A5-dynamo-db-%ED%99%9C%EC%9A%A9%ED%95%98%EA%B8%B0-%EA%B0%95%EB%AF%BC%EC%84%9D-aws-database-modernization-day-%EC%98%A8%EB%9D%BC%EC%9D%B8-2
DynamoDB는 대량의 트래픽에 대해 빠른 응답시간을 보장하는 AWS의 NOSQL Database 서비스 입니다. 본 세션에서는 DynamoDB를 생성하고 테이블 디자인 후 데이터 입력, 삭제, 업데이트 및 성능에 관련된 설정에 대해서 진행합니다. 이 세션후 참석자들은 DyanmoDB에 대한 이해하며 직접 구성 및 사용할 수 있습니다.
Resiliency-and-Availability-Design-Patterns-for-the-CloudAmazon Web Services
We have traditionally built robust software systems by trying to avoid mistakes and by dodging failures when they occur in production or by testing parts of the system in isolation from one another. Modern methods and techniques take a very different approach based on resiliency, which promotes embracing failure instead of trying to avoid it. Resilient architectures enhance observability, leverage well-known patterns such as graceful degradation, timeouts and circuit breakers. In this session, will review the most useful patterns for building resilient software systems and especially show the audience how they can benefit from the patterns.
MySQL Database Architectures - MySQL InnoDB ClusterSet 2021-11Kenny Gryp
Oracle's MySQL solutions make it easy to setup various database architectures and achieve high availability with the introduction MySQL InnoDB Cluster and MySQL InnoDB ReplicaSet meeting various high availability requirements. MySQL InnoDB ClusterSet provides a popular disaster recovery solution.
Completely built in-house and supported by Oracle, many enterprises large and small have adopted these solutions into business critical applications.
In this presentation the various database architecture solutions for high availability and disaster recovery will be covered and help you choose the right solutions based on your business requirements.
A closer look at the MySQL and PostgreSQL compatible relational database built for the cloud that combines the performance and availability of high-end commercial databases with the simplicity and cost-effectiveness of open source databases. We’ll explore how Aurora uses the AWS cloud to provide high reliability, high durability, and high throughput.
Speakers:
Steve Abraham - Principal Database Specialist Solutions Architect, AWS
Peter Dachnowicz - Sr. Technical Account Manager, AWS
by Mikhail Prudnikov, Sr. Solutions Architect, AWS
Redis is an open source, in-memory data store that delivers sub-millisecond response times enabling millions of requests per second to power real-time applications. It can be used as a fast database, cache, message broker, and queue. Amazon ElastiCache delivers the ease-of-use and power of Redis along with the availability, reliability, scalability, security, and performance suitable for the most demanding applications. We’ll take a close look at Redis and how to use it to power different use cases.
Group Replication went Generally Available end of 2016, it introduces a 'synchronous' active:active multi-master eplication, in addition to asynchronous and semi-synchronous replication, the latter 2 being available in in MySQL for longtime.
As with any new feature, and especially with introducing active:active multi-master replication, it takes a while before companies are adopting the software in production database environment.
For example, even though MySQL 5.7 has been GA for more than a year, adoption is only starting to increase recently.
We can, and should, expect the same from Group Replication. As with every release, bugs will be found, and with new features, best practises still need to formed out of practical experience.
After giving a short introduction on what Group Replication is, I will cover my experience so far in evaluating Group Replication.
AWS - an introduction to bursting (GP2 - T2)Rasmus Ekman
An introduction to bursting on AWS. The presentation includes an introductory explanation of the new EC2 family, T2, and the new EBS volume type GP2.
Learn about the new services on AWS to get the most bang for your buck.
Amazon Aurora Storage Demystified: How It All Works (DAT363) - AWS re:Invent ...Amazon Web Services
Amazon Aurora is a high performance, highly scalable database service with MySQL- and PostgreSQL-compatibility. One of its key components is an innovative storage system that is optimized for database workloads and specifically designed to take advantage of modern cloud technology. Hear from the team that built Amazon Aurora's storage system on how the system is designed, how it works, and what you need to know to get the most out of it.
Oracle Real Application Clusters 19c- Best Practices and Internals- EMEA Tour...Sandesh Rao
In this session, I will cover under-the-hood features that power Oracle Real Application Clusters (Oracle RAC) 19c specifically around Cache Fusion and Service management. Improvements in Oracle RAC helps in integration with features such as Multitenant and Data Guard. In fact, these features benefit immensely when used with Oracle RAC. Finally we will talk about changes to the broader Oracle RAC Family of Products stack and the algorithmic changes that helps quickly detect sick/dead nodes/instances and the reconfiguration improvements to ensure that the Oracle RAC Databases continue to function without any disruption
Amazon Aurora is a fully managed relational database engine that combines the speed and availability of high-end commercial databases with the simplicity and cost-effectiveness of open source databases. It is purpose-built for the cloud using a new architectural model and distributed systems techniques to provide far higher performance, availability and durability than previously possible using conventional monolithic database architectures. Amazon Aurora packs a lot of innovations in the engine and storage layers. In this session, we will do a deep-dive into some of the key innovations behind Amazon Aurora, new improvements to Aurora's performance, availability and cost-effectiveness and discuss best practices and optimal configurations.
발표 영상 다시보기: https://kr-resources.awscloud.com/data-databases-and-analytics/%EC%A7%80%EA%B8%88-%EB%8B%B9%EC%9E%A5-dynamo-db-%ED%99%9C%EC%9A%A9%ED%95%98%EA%B8%B0-%EA%B0%95%EB%AF%BC%EC%84%9D-aws-database-modernization-day-%EC%98%A8%EB%9D%BC%EC%9D%B8-2
DynamoDB는 대량의 트래픽에 대해 빠른 응답시간을 보장하는 AWS의 NOSQL Database 서비스 입니다. 본 세션에서는 DynamoDB를 생성하고 테이블 디자인 후 데이터 입력, 삭제, 업데이트 및 성능에 관련된 설정에 대해서 진행합니다. 이 세션후 참석자들은 DyanmoDB에 대한 이해하며 직접 구성 및 사용할 수 있습니다.
Resiliency-and-Availability-Design-Patterns-for-the-CloudAmazon Web Services
We have traditionally built robust software systems by trying to avoid mistakes and by dodging failures when they occur in production or by testing parts of the system in isolation from one another. Modern methods and techniques take a very different approach based on resiliency, which promotes embracing failure instead of trying to avoid it. Resilient architectures enhance observability, leverage well-known patterns such as graceful degradation, timeouts and circuit breakers. In this session, will review the most useful patterns for building resilient software systems and especially show the audience how they can benefit from the patterns.
MySQL Database Architectures - MySQL InnoDB ClusterSet 2021-11Kenny Gryp
Oracle's MySQL solutions make it easy to setup various database architectures and achieve high availability with the introduction MySQL InnoDB Cluster and MySQL InnoDB ReplicaSet meeting various high availability requirements. MySQL InnoDB ClusterSet provides a popular disaster recovery solution.
Completely built in-house and supported by Oracle, many enterprises large and small have adopted these solutions into business critical applications.
In this presentation the various database architecture solutions for high availability and disaster recovery will be covered and help you choose the right solutions based on your business requirements.
A closer look at the MySQL and PostgreSQL compatible relational database built for the cloud that combines the performance and availability of high-end commercial databases with the simplicity and cost-effectiveness of open source databases. We’ll explore how Aurora uses the AWS cloud to provide high reliability, high durability, and high throughput.
Speakers:
Steve Abraham - Principal Database Specialist Solutions Architect, AWS
Peter Dachnowicz - Sr. Technical Account Manager, AWS
by Mikhail Prudnikov, Sr. Solutions Architect, AWS
Redis is an open source, in-memory data store that delivers sub-millisecond response times enabling millions of requests per second to power real-time applications. It can be used as a fast database, cache, message broker, and queue. Amazon ElastiCache delivers the ease-of-use and power of Redis along with the availability, reliability, scalability, security, and performance suitable for the most demanding applications. We’ll take a close look at Redis and how to use it to power different use cases.
Group Replication went Generally Available end of 2016, it introduces a 'synchronous' active:active multi-master eplication, in addition to asynchronous and semi-synchronous replication, the latter 2 being available in in MySQL for longtime.
As with any new feature, and especially with introducing active:active multi-master replication, it takes a while before companies are adopting the software in production database environment.
For example, even though MySQL 5.7 has been GA for more than a year, adoption is only starting to increase recently.
We can, and should, expect the same from Group Replication. As with every release, bugs will be found, and with new features, best practises still need to formed out of practical experience.
After giving a short introduction on what Group Replication is, I will cover my experience so far in evaluating Group Replication.
AWS - an introduction to bursting (GP2 - T2)Rasmus Ekman
An introduction to bursting on AWS. The presentation includes an introductory explanation of the new EC2 family, T2, and the new EBS volume type GP2.
Learn about the new services on AWS to get the most bang for your buck.
DevOps for ETL processing at scale with MongoDB, Solr, AWS and ChefGaurav "GP" Pal
Large scale data processing for Extract Transform and Loading (ETL) jobs is a very common practice. The stackArmor DevOps team developed a Chef based automation solution to automate the AWS environment provisioning, code deployment and data ingestion processing to ingest and process over 2 TB of Data.
This presentation covers the technologies used, the planning phase, AWS instance selection and optimizing the ETL processing for not only performance but also cost.
The target was to process 500 million rows within 72 hours with a processing rate of 5 million transactions per hour.
The presentation also provides pitfalls and automation optimizations performed to accomplish the targeted processing rates.
The presentation was delivered at the DevOpsDC Meetup on May 17, 2016
Training for AWS Solutions Architect at http://zekelabs.com/courses/amazon-web-services-training-bangalore/.This slide describes about features of EC2, EC2 Options, family type, storage, EBS Volumes, EC2 Instance Store, Security Groups, Volumes and Snapshots, Amazon Machine Image (AMI), Elastic load balancer, Classic load balancer, Application load balancer, Network load balancer, AWS CLI and EC2 Metadata
___________________________________________________
zekeLabs is a Technology training platform. We provide instructor led corporate training and classroom training on Industry relevant Cutting Edge Technologies like Big Data, Machine Learning, Natural Language Processing, Artificial Intelligence, Data Science, Amazon Web Services, DevOps, Cloud Computing and Frameworks like Django,Spring, Ruby on Rails, Angular 2 and many more to Professionals.
Reach out to us at www.zekelabs.com or call us at +91 8095465880 or drop a mail at info@zekelabs.com
AWS Webcast - Cost and Performance Optimization in Amazon RDSAmazon Web Services
Amazon RDS makes it easy to set up, operate, and scale relational databases in the cloud. The service offers a variety of options for optimizing the performance level delivered, as well as optimizing your spending. In this webinar, we will show a variety of techniques for implementing the right performance level for your application.
Learning Objectives:
• Understand the Amazon RDS options that change database performance and cost
• Select the appropriate performance and cost level for your specific application Who Should Attend:
• Technical Amazon RDS customers and prospective customers
Deep Dive on Amazon EC2 Instances - AWS Summit Cape Town 2017Amazon Web Services
Amazon EC2 provides a broad selection of instance types to accommodate a diverse mix of workloads. In this session, we provide an overview of the Amazon EC2 instance platform, key platform features, and the concept of instance generations. We dive into the current generation design choices of the different instance families, including the General Purpose, Compute Optimized, Storage Optimized, Memory Optimized, and Accelerated Computing (GPU and FPGA) instance families. We also detail best practices and share performance tips for getting the most out of your Amazon EC2 instances.
AWS Speaker: Ian Massingham, Sr Mgr, Technical Evangelist - Amazon Web Services
Customer Speaker: Andrew Mori, Konga, Technical Director
Azure Native Qumulo scales elastically for common High Performance Compute (HPC) workloads based on application requirements for: Financial Services, Automotive, Genomics / Life Sciences, Media and Entertainment, Energy, Oil and Gas, and more. Performance can be increased (and elastically decreased) much higher than the examples shown here. These slides offer a glimpse into ANQ's HPC capabilities, although at a smaller scale. We invite YOU to do your own testing (with a free ANQ trial) and work with us to test your HPC workloads in Azure.
Getting Started with AWS EC2. From Zero to HeroAWSBulgaria
Presentation by George Yanev during the #AWSBulgaria User Group Meeting: http://www.meetup.com/AWS-Bulgaria/events/229990095/
#AWSBulgaria is a community for anyone interested in Amazon Web Services, cloud computing, DevOps.
In this session, Kevin will dive into the unique challenges of keeping your Kubernetes workloads highly available while keeping costs low. You will learn about how to leverage cloud-native autoscaling, pod requirement right-sizing, resource buffer definition, cost allocation and more.
This report is executed by StorPool Storage and compares the block storage
offerings of well-known public clouds
(AWS, Digital Ocean, OVH, DreamHost)
with a number of StorPool-based public
cloud offerings.
Amazon EC2 provides resizable compute capacity in the cloud, making web scale computing easier. It offers a wide variety of compute instances and is well suited to every imaginable use case, from static websites to on-demand, high-performance supercomputing--all with flexible pricing options. In this session, learn about the latest Amazon EC2 features and capabilities, including Approved instance families, the differences among their hardware types and capabilities, and their optimal use cases. Also discover best practices for optimizing your expenditure and getting the most benefit from your EC2 instances while saving time and money.
Amazon EC2 provides a broad selection of instance types to deliver high performance for a diverse mix of applications. In this session, we overview the drivers of system performance and discuss in depth how Amazon EC2 instances deliver system performance while also providing elasticity and complete control over your infrastructure. We also detail best practices and share performance tips for getting the most out of your Amazon EC2 instances.
Deep Dive on Amazon EC2 Instances - January 2017 AWS Online Tech TalksAmazon Web Services
Amazon EC2 provides a broad selection of instance types to accommodate a diverse mix of workloads. In this session, we provide an overview of the Amazon EC2 instance platform, key platform features, and the concept of instance generations. We dive into the current generation design choices of the different instance families, including the General Purpose, Compute Optimized, Storage Optimized, Memory Optimized, and GPU instance families. We will also provide an overview of the newest instances announced at re:Invent, including the latest generation of Memory and Compute Optimized Instances R4 and C5 instances, new Storage Optimized High I/O I3 instances, and new larger T2 instances. We also detail best practices and share performance tips for getting the most out of your Amazon EC2 instances.
Learning Objectives:
• Get an overview of the EC2 instance platform, key platform features, and the concept of instance generations
• Learn about the latest generation of Amazon EC2 Instances
• Learn best practices around instance selection to optimize performance
Benchmarking your cloud performance with top 4 global public cloudsdata://disrupted®
In this presentation, we will present the performance measurement metrics of leading cloud providers - AWS, Google Cloud, Microsoft Azure, and Digital Ocean. We’ll give you useful tools to measure your own cloud performance and a handy guide on how to calculate cloud TCO (total cost of ownership). In addition, you’ll learn how to estimate correctly your market positioning and perform better than the cloud giants.
Boyan Krosnov is a Co-Founder and Chief Product Officer of StorPool Storage. He has been part of the technical teams building 5 service providers from scratch in 4 countries. In most of these projects, he has designed the architecture, led the technical teams, and managed the implementation of projects in the millions.
We have compiled the most important slides from each speaker's presentation. This year’s compilation, available for free, captures the key insights and contributions shared during the DfMAy 2024 conference.
Understanding Inductive Bias in Machine LearningSUTEJAS
This presentation explores the concept of inductive bias in machine learning. It explains how algorithms come with built-in assumptions and preferences that guide the learning process. You'll learn about the different types of inductive bias and how they can impact the performance and generalizability of machine learning models.
The presentation also covers the positive and negative aspects of inductive bias, along with strategies for mitigating potential drawbacks. We'll explore examples of how bias manifests in algorithms like neural networks and decision trees.
By understanding inductive bias, you can gain valuable insights into how machine learning models work and make informed decisions when building and deploying them.
Final project report on grocery store management system..pdfKamal Acharya
In today’s fast-changing business environment, it’s extremely important to be able to respond to client needs in the most effective and timely manner. If your customers wish to see your business online and have instant access to your products or services.
Online Grocery Store is an e-commerce website, which retails various grocery products. This project allows viewing various products available enables registered users to purchase desired products instantly using Paytm, UPI payment processor (Instant Pay) and also can place order by using Cash on Delivery (Pay Later) option. This project provides an easy access to Administrators and Managers to view orders placed using Pay Later and Instant Pay options.
In order to develop an e-commerce website, a number of Technologies must be studied and understood. These include multi-tiered architecture, server and client-side scripting techniques, implementation technologies, programming language (such as PHP, HTML, CSS, JavaScript) and MySQL relational databases. This is a project with the objective to develop a basic website where a consumer is provided with a shopping cart website and also to know about the technologies used to develop such a website.
This document will discuss each of the underlying technologies to create and implement an e- commerce website.
6th International Conference on Machine Learning & Applications (CMLA 2024)ClaraZara1
6th International Conference on Machine Learning & Applications (CMLA 2024) will provide an excellent international forum for sharing knowledge and results in theory, methodology and applications of on Machine Learning & Applications.
Hierarchical Digital Twin of a Naval Power SystemKerry Sado
A hierarchical digital twin of a Naval DC power system has been developed and experimentally verified. Similar to other state-of-the-art digital twins, this technology creates a digital replica of the physical system executed in real-time or faster, which can modify hardware controls. However, its advantage stems from distributing computational efforts by utilizing a hierarchical structure composed of lower-level digital twin blocks and a higher-level system digital twin. Each digital twin block is associated with a physical subsystem of the hardware and communicates with a singular system digital twin, which creates a system-level response. By extracting information from each level of the hierarchy, power system controls of the hardware were reconfigured autonomously. This hierarchical digital twin development offers several advantages over other digital twins, particularly in the field of naval power systems. The hierarchical structure allows for greater computational efficiency and scalability while the ability to autonomously reconfigure hardware controls offers increased flexibility and responsiveness. The hierarchical decomposition and models utilized were well aligned with the physical twin, as indicated by the maximum deviations between the developed digital twin hierarchy and the hardware.
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdffxintegritypublishin
Advancements in technology unveil a myriad of electrical and electronic breakthroughs geared towards efficiently harnessing limited resources to meet human energy demands. The optimization of hybrid solar PV panels and pumped hydro energy supply systems plays a pivotal role in utilizing natural resources effectively. This initiative not only benefits humanity but also fosters environmental sustainability. The study investigated the design optimization of these hybrid systems, focusing on understanding solar radiation patterns, identifying geographical influences on solar radiation, formulating a mathematical model for system optimization, and determining the optimal configuration of PV panels and pumped hydro storage. Through a comparative analysis approach and eight weeks of data collection, the study addressed key research questions related to solar radiation patterns and optimal system design. The findings highlighted regions with heightened solar radiation levels, showcasing substantial potential for power generation and emphasizing the system's efficiency. Optimizing system design significantly boosted power generation, promoted renewable energy utilization, and enhanced energy storage capacity. The study underscored the benefits of optimizing hybrid solar PV panels and pumped hydro energy supply systems for sustainable energy usage. Optimizing the design of solar PV panels and pumped hydro energy supply systems as examined across diverse climatic conditions in a developing country, not only enhances power generation but also improves the integration of renewable energy sources and boosts energy storage capacities, particularly beneficial for less economically prosperous regions. Additionally, the study provides valuable insights for advancing energy research in economically viable areas. Recommendations included conducting site-specific assessments, utilizing advanced modeling tools, implementing regular maintenance protocols, and enhancing communication among system components.
Water billing management system project report.pdfKamal Acharya
Our project entitled “Water Billing Management System” aims is to generate Water bill with all the charges and penalty. Manual system that is employed is extremely laborious and quite inadequate. It only makes the process more difficult and hard.
The aim of our project is to develop a system that is meant to partially computerize the work performed in the Water Board like generating monthly Water bill, record of consuming unit of water, store record of the customer and previous unpaid record.
We used HTML/PHP as front end and MYSQL as back end for developing our project. HTML is primarily a visual design environment. We can create a android application by designing the form and that make up the user interface. Adding android application code to the form and the objects such as buttons and text boxes on them and adding any required support code in additional modular.
MySQL is free open source database that facilitates the effective management of the databases by connecting them to the software. It is a stable ,reliable and the powerful solution with the advanced features and advantages which are as follows: Data Security.MySQL is free open source database that facilitates the effective management of the databases by connecting them to the software.
Immunizing Image Classifiers Against Localized Adversary Attacksgerogepatton
This paper addresses the vulnerability of deep learning models, particularly convolutional neural networks
(CNN)s, to adversarial attacks and presents a proactive training technique designed to counter them. We
introduce a novel volumization algorithm, which transforms 2D images into 3D volumetric representations.
When combined with 3D convolution and deep curriculum learning optimization (CLO), itsignificantly improves
the immunity of models against localized universal attacks by up to 40%. We evaluate our proposed approach
using contemporary CNN architectures and the modified Canadian Institute for Advanced Research (CIFAR-10
and CIFAR-100) and ImageNet Large Scale Visual Recognition Challenge (ILSVRC12) datasets, showcasing
accuracy improvements over previous techniques. The results indicate that the combination of the volumetric
input and curriculum learning holds significant promise for mitigating adversarial attacks without necessitating
adversary training.
The Internet of Things (IoT) is a revolutionary concept that connects everyday objects and devices to the internet, enabling them to communicate, collect, and exchange data. Imagine a world where your refrigerator notifies you when you’re running low on groceries, or streetlights adjust their brightness based on traffic patterns – that’s the power of IoT. In essence, IoT transforms ordinary objects into smart, interconnected devices, creating a network of endless possibilities.
Here is a blog on the role of electrical and electronics engineers in IOT. Let's dig in!!!!
For more such content visit: https://nttftrg.com/
Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...Dr.Costas Sachpazis
Terzaghi's soil bearing capacity theory, developed by Karl Terzaghi, is a fundamental principle in geotechnical engineering used to determine the bearing capacity of shallow foundations. This theory provides a method to calculate the ultimate bearing capacity of soil, which is the maximum load per unit area that the soil can support without undergoing shear failure. The Calculation HTML Code included.
Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)MdTanvirMahtab2
This presentation is about the working procedure of Shahjalal Fertilizer Company Limited (SFCL). A Govt. owned Company of Bangladesh Chemical Industries Corporation under Ministry of Industries.
4. 4
WHY WE USE AWS
● Fast hardware (service) provisioning
● Easy scale up/down/in/out
● Pay only for resources you need
5. 5
GLOSSARY
● RDS - Relational Database Service
● EC2 - Elastic Compute Cloud
● EBS - Elastic Block Store
● S3 - Simple Storage Service
● AZ - Availability Zone
● ASG - Auto Scaling Group
6. 6
POSTGRESQL HA ON AWS
● Shared storage - EBS, attach it to a different EC2 Instance
○ Works within one AZ
● Storage mirroring - Multi-AZ RDS (EBS is replicated to another AZ)
○ Works between AZ
○ Replicas can’t execute queries
● Streaming replication - you can start additional RDS replicas
○ Replicas can execute queries
○ Automatic Failover problem (if not Multi-AZ deployment)
● Amazon Aurora
○ Keeps 6 copies of data in 3 AZ
○ Low latency read replicas
○ Failover typically takes less than 30 seconds.
7. 7
EC2 VS. RDS VS. AURORA: MONTHLY PRICING*
Instance Type EC2 RDS Aurora
(db.)t2.micro $9.78 $18.25 n/a
(db.)m4.large $87.60 $158.41 n/a
(db.)m5.large $83.95 n/a n/a
(db.)r4.large $116.80 $222.65 $255.50
(db.)r4.xlarge $233.60 $445.30 $511.00
* storage price is not included
8. 8
EC2 VS. RDS VS. AURORA: MONTHLY PRICING*
Instance Type 2 x EC2 3 x EC2 RDS Multi-AZ 2 x Aurora
(db.)t2.micro $19.56 $29.34 $36.50 n/a
(db.)m4.large $175.20 $262.80 $316.82 n/a
(db.)m5.large $167.90 $251.85 n/a n/a
(db.)r4.large $233.60 $350.40 $445.30 $511.00
(db.)r4.xlarge $467.20 $700.80 $890.60 $1022.00
* storage price and price for Aurora I/O Rate are not included
9. 9
● No superuser access
● No replication connection
● No custom extensions
● Multi-AZ RDS doubles the price, but replica can’t execute queries
● Aurora works only on R4 instances and price for “I/O Rate” can
become really high
RDS SUMMARY
10. 10
● Patroni - High-Availability and automatic failover
● Spilo - Docker package of Patroni and WAL-E for AWS or Kubernetes
● Use CloudFormation stacks and ASG for deployments
● One Docker container per EC2 Instance
SPILO AND PATRONI
16. 16
● Designed to provide moderate baseline performance and the capability to burst to significantly higher
performance as required by your workload.
● If your account is less than 12 months old, you can use a t2.micro instance for free within certain usage
limits.
T2 INSTANCES
Instance Type Initial CPU
credit
Credits
earned per
hour
vCPUs Base
performance
(CPU utilization)
Max CPU
credit
balance
t2.micro 30 6 1 10% 144
t2.small 30 12 1 20% 288
t2.medium 60 24 2 40% 578
t2.large 60 36 2 60% 864
17. 17
● One CPU credit is equal to one vCPU running at 100% utilization for
one minute.
● When a T2 instance uses fewer CPU resources than its base
performance level allows (such as when it is idle), the unused CPU
credits (or the difference between what was earned and what was
spent) are stored in the credit balance for up to 24 hours, building CPU
credits for bursting.
● When a T2 instance requires more CPU resources than its base
performance level allows, it uses credits from the CPU credit balance to
burst up to 100% utilization.
CPU CREDITS
18. 18
● WAL-E wal-fetch/wal-prefetch is terribly slow
○ Prefetch spawns 8 worker processes (by default)
and can burn all CPU Credits
How we solved it:
1. use “-p 0” to disable prefetch
2. reimplemented ‘wal-fetch’ in bash + curl + openssl
HORROR STORY
19. 19
● CPUCreditUsage metric indicates the number of CPU credits used
during the measurement period
● CPUCreditBalance metric indicates the number of unused CPU credits
a T2 instance has earned
MONITOR CPU CREDITS t2.small
20. 20
● 1 year or 3 year contracts
● Significant discount compared to On-Demand instance pricing
● Capacity reservation
● Customers using both Reserved and On-Demand instances will have
Reserved Instance rates applied first to minimize costs
● Discounts up to 70%
RESERVED INSTANCES
21. 21
● Bidding on “AWS overcapacity”
● Variable price point, save up to 90% vs. on-demand
*Risks*
Unavailability or loss of instance if outbid!!!
SPOT INSTANCES
23. 23
● Async processing
● Reporting
● CI
● Staging systems
● Testing of backups
USE CASES
Everything that can fail or be unavailable for short duration
24. 24
● There were only 2 AZ in eu-central-1 until June 2017
● ASG is trying to do its best to distribute resources across different AZ, but when Spot Price
is high, it might happen that two EC2 Instances will run in the same AZ
● When price will go down, ASG will rebalance resources, i.e. spawn a new instance in another
AZ and terminate one of the running instances
● With 50% probability it will kill the “master”
How we solved it:
AutoScalingGroup -> “TerminationPolicies”: [“NewestInstance”, “Default”]
HORROR STORY
25. 25
ELASTIC BLOCK STORE
Name io1 gp2 st1 sc1
Size 4GB - 16TB 1GB - 16TB 500GB - 16TB 500GB - 16TB
Max IOPS/Volume 20000 10000 500 250
Max Throughput/Volume 320MB/s 160MB/s 500MB/s 250MB/s
26. 26
● ST1 and SC1 - HDD, good throughput, but low IOPS
● GP2 and IO1 - SSD, usually good choice for databases
● EBS volumes are created in a specific AZ, and can then be attached to
any instances in that AZ
● IOPS and throughput depends on Volume Size
ELASTIC BLOCK STORE
28. 28
● IO1 gives better IOPS guarantees
● But you have to pay for it – the same size costs 25% more (or 9%
more for RDS). Plus you have to pay for provisioned IOPS
● For RDS minimum size of IO1 volume is 100 GB + 1000 provisioned
IOPS. It will cost you $14.90 + $119 per month
● You can get 1000 IOPS with 334 GB GP2 volume for $45.76/month
IO1 VS. GP2
32. 32
I/O CREDITS AND BURST PERFORMANCE
● The performance of gp2 volumes is tied to volume size (3 IOPS/GiB)
● The maximum and initial I/O credit balance for a volume is 5.4 million
● When your volume requires more than the baseline performance I/O
level, it draws on I/O credits in the credit balance to burst to the required
performance level, up to a maximum of 3,000 IOPS
● When your volume uses fewer I/O credits than it earns in a second,
unused I/O credits are added to the I/O credit balance
34. 34
● Especially BurstBalance if GP2 volume is smaller than 1TB
MONITOR I/O WITH CLOUDWATCH
500 GB GP2 Volume
2500
1250
35. 35
● Dedicated bandwidth to Amazon EBS, with options between 500 Mbps
and 12,000 Mbps, depending on the instance type you use
● Provides the best performance for your EBS volumes by minimizing
contention between Amazon EBS I/O and other traffic from your
instance
EBS-OPTIMIZED INSTANCES
36. 36
EBS THROUGHPUT
InstanceType vCPU Memory Max IOPS Throughput (Mb/s) Price (per month)
m4.large 2 8 GB 3600 56.25 $87.6
r4.large 2 15 GB 3000 54 $116.8
m5.large* 2 8 GB Up to 16000 Up to 265 $83.95
m4.xlarge 4 16 GB 6000 93.75 $175.2
r4.xlarge 4 30 GB 6000 106.25 $233.6
m5.xlarge* 4 16 GB Up to 16000 Up to 265 $167.9
37. 37
● Next generation
● Slightly cheaper than M4
● Better networking and storage performance
● EBS burst capability on smaller instance sizes (large,
xlarge, 2xlarge)
○ can support maximum performance for 30 minutes
at least once every 24 hours
M5 INSTANCES
39. 39
● If you run m5.large, m5.xlarge or m5.2xlarge, you should monitor
○ EBSIOBalance% - percentage of I/O credits remaining in
the burst bucket.
○ EBSByteBalance% - percentage of throughput credits
remaining in the burst bucket
● Instances with a consistently low balance percentage are
candidates for upsizing
● Instances where the balance percentage never drops below
100% are candidates for downsizing
MORE CLOUDWATCH METRICS
40. 40
● t2.micro, t2.small and t2.medium are showing up to ~60 MB/s on EBS
● t2.large, t2.xlarge and t2.2xlarge are showing up to ~120 MB/s on EBS
● I don’t have any numbers about Max. IOPS for T2 instances
● There is no guaranteed throughput for T2 instances!
EBS PERFORMANCE OF T2 INSTANCES
41. 41
● GP2 is MUCH cheaper than IO1
● Choose an EC2 Instance with enough bandwidth
● Build a RAID-0 from multiple GP2 volumes to get
more than 10000 IOPS or 160MB/s
EBS TIPS
42. 42
HORROR STORY
● AWS didn’t provide a means to
monitor EBS Burst Balance
until November 2016
● RDS still doesn’t provide
information about GP2 Volume
Burst Balance
43. 43
● Pros:
○ Located on disks that are physically attached to the host computer
○ Amazing throughput and latencies compared to EBS
● Cons:
○ Provides only temporary block-level storage
○ Data in the instance store is lost under the following circumstances:
■ The underlying disk drive fails
■ The instance stops or terminates
INSTANCE STORE VOLUMES
45. 45
● For high-intensive OLTP, i3 instances are a rescue:
○ r4.2xlarge + 3.3TB EBS ($863.75/month) wasn’t able to keep up
○ The switch to i3.2xlarge solved all problems and saved 37% of
costs
We run ~40 clusters on i3 instances and only two i3 instances has failed
during last year.
INSTANCE STORE SUMMARY
47. 47
● We observed 4 failures of EBS during past year
● EC2 instances fail a few times more often than EBS
○ Sometimes they just fail
○ But usually AWS notifies you about degradation
● Please do continuous archiving to S3!
○ wal-e, wal-g, pgBackRest
ANYTHING CAN FAIL
48. 48
● Start with small Instances and Volumes, it’s easy to
scale up later
● Some of the resources (CPUCreditUsage,
CPUCreditBalance, BurstBalance,
EBSIOBalance%, EBSByteBalance%) it’s possible
to monitor only with CloudWatch
● Always do backups and test them
SUMMARY