This document provides an overview of performance tuning best practices for Scala applications. It discusses motivations for performance tuning such as resolving issues or reducing infrastructure costs. Some common bottlenecks are identified as databases, asynchronous/thread operations, and I/O. Best practices covered include measuring metrics, identifying bottlenecks, and avoiding premature optimization. Microbenchmarks and optimization examples using Scala collections are also presented.
Prespective analytics with DOcplex and pandasPyDataParis
The document discusses prescriptive analytics and optimization modeling using Python. It introduces DOcplex, an IBM product that allows users to formulate and solve optimization problems in Python. Key points include:
- Prescriptive analytics makes recommendations to optimize outcomes based on constraints and past events.
- DOcplex allows optimization problems to be modeled in Python and solved using local or cloud solvers like CPLEX.
- Pandas can improve performance for DOcplex models by efficiently handling slicing and aggregation operations on large datasets.
Highly-scalable Reinforcement Learning RLlib for Real-world ApplicationsBill Liu
This document discusses Ray and RLlib, a reinforcement learning library built on Ray. It provides three key points:
1. Ray is a framework for building distributed applications and services with shared memory abstraction. It allows ML workloads to scale beyond a single machine.
2. RLlib is a scalable reinforcement learning library that uses Ray. It supports a wide range of algorithms and execution models. This allows for easy implementation and comparison of RL techniques.
3. The Ray community is growing rapidly and provides resources like tutorials and Slack support to help users adopt Ray and RLlib for their distributed Python and reinforcement learning applications.
This document provides an introduction to training machine learning models using cloud computing. It discusses renting a single virtual machine and optimizing costs. It also covers distributed training using horizontal and vertical scaling with TensorFlow. Data preparation, deploying code to virtual machines, using GPUs and TPUs, and data parallelization techniques are described for efficient distributed training in the cloud.
Since it’s introduction over 20 years ago, Java developers have had plenty of strong and often opposing ways of doing things.
Modern day JVM development has only added more options, more divisiveness, and even more heated arguments.
* Constructor or Setter Injection ?
* Dynamic or Static typing ?
* Monolithic or Micro-service application design ?
* Java or Scala or Kotlin ?
* JVM or .NET ?
* Reactive ?
* Are Mutable types and Threads inherently Evil?
* Is Kubernetes / Docker the modern way of JVM deployments?
Let’s dive into these and other issues .. while also stepping back and looking at the Bigger Picture:
What is it really about these choices that improve the bottom line ?
Most importantly .. let’s also settle once and for all:
Maven or Gradle?
Some resources how to navigate in the hardware space in order to build your own workstation for training deep learning models.
Alternative download link: https://www.dropbox.com/s/o7cwla30xtf9r74/deepLearning_buildComputer.pdf?dl=0
This document discusses machine learning with Hadoop. It begins with an agenda that covers why big data has become important, what can be done with big data, and how machine learning works with big data. It then discusses why now is the right time for big data, explaining that improvements in algorithms, data practices, and hardware have enabled new opportunities. Specific machine learning techniques like classification, clustering, and recommendations using Hadoop and Mahout are then outlined.
This document discusses deep learning, including its relationship to artificial intelligence and machine learning. It describes deep learning techniques like artificial neural networks and how GPUs are useful for deep learning. Applications mentioned include computer vision, speech recognition, and bioinformatics. Both benefits like robustness and weaknesses like long training times are outlined. Finally, common deep learning algorithms, libraries and tools are listed.
Prespective analytics with DOcplex and pandasPyDataParis
The document discusses prescriptive analytics and optimization modeling using Python. It introduces DOcplex, an IBM product that allows users to formulate and solve optimization problems in Python. Key points include:
- Prescriptive analytics makes recommendations to optimize outcomes based on constraints and past events.
- DOcplex allows optimization problems to be modeled in Python and solved using local or cloud solvers like CPLEX.
- Pandas can improve performance for DOcplex models by efficiently handling slicing and aggregation operations on large datasets.
Highly-scalable Reinforcement Learning RLlib for Real-world ApplicationsBill Liu
This document discusses Ray and RLlib, a reinforcement learning library built on Ray. It provides three key points:
1. Ray is a framework for building distributed applications and services with shared memory abstraction. It allows ML workloads to scale beyond a single machine.
2. RLlib is a scalable reinforcement learning library that uses Ray. It supports a wide range of algorithms and execution models. This allows for easy implementation and comparison of RL techniques.
3. The Ray community is growing rapidly and provides resources like tutorials and Slack support to help users adopt Ray and RLlib for their distributed Python and reinforcement learning applications.
This document provides an introduction to training machine learning models using cloud computing. It discusses renting a single virtual machine and optimizing costs. It also covers distributed training using horizontal and vertical scaling with TensorFlow. Data preparation, deploying code to virtual machines, using GPUs and TPUs, and data parallelization techniques are described for efficient distributed training in the cloud.
Since it’s introduction over 20 years ago, Java developers have had plenty of strong and often opposing ways of doing things.
Modern day JVM development has only added more options, more divisiveness, and even more heated arguments.
* Constructor or Setter Injection ?
* Dynamic or Static typing ?
* Monolithic or Micro-service application design ?
* Java or Scala or Kotlin ?
* JVM or .NET ?
* Reactive ?
* Are Mutable types and Threads inherently Evil?
* Is Kubernetes / Docker the modern way of JVM deployments?
Let’s dive into these and other issues .. while also stepping back and looking at the Bigger Picture:
What is it really about these choices that improve the bottom line ?
Most importantly .. let’s also settle once and for all:
Maven or Gradle?
Some resources how to navigate in the hardware space in order to build your own workstation for training deep learning models.
Alternative download link: https://www.dropbox.com/s/o7cwla30xtf9r74/deepLearning_buildComputer.pdf?dl=0
This document discusses machine learning with Hadoop. It begins with an agenda that covers why big data has become important, what can be done with big data, and how machine learning works with big data. It then discusses why now is the right time for big data, explaining that improvements in algorithms, data practices, and hardware have enabled new opportunities. Specific machine learning techniques like classification, clustering, and recommendations using Hadoop and Mahout are then outlined.
This document discusses deep learning, including its relationship to artificial intelligence and machine learning. It describes deep learning techniques like artificial neural networks and how GPUs are useful for deep learning. Applications mentioned include computer vision, speech recognition, and bioinformatics. Both benefits like robustness and weaknesses like long training times are outlined. Finally, common deep learning algorithms, libraries and tools are listed.
Flavius Ștef: Big Rewrites Without Big Risks at I T.A.K.E. UnconferenceMozaic Works
This document discusses strategies for incrementally rewriting software architecture without big risks. It suggests splitting large rewrites into smaller chunks and using techniques like faking functionality until it is implemented, incremental refactoring, and rewriting in small bites. Specific strategies discussed include researching prerequisites, creating proofs of concept, adding unit and performance tests incrementally, and removing dependencies one by one. Choosing the right refactoring is important, and catalogues of refactorings can provide guidance. An incremental process of analyzing, planning, refactoring, testing and integrating changes in small batches is recommended to validate progress toward goals at each step.
Incremental development is easy when we are talking about functionality. Story splitting has become quite popular as a technique lately.
But what about those cases when you need to do an architectural refactoring? Could incremental development be applied?
(Talk delivered during I T.A.K.E. Unconference 2015)
Dyn delivers exceptional Internet Performance. Enabling high quality services requires data centers around the globe. In order to manage services, customers need timely insight collected from all over the world. Dyn uses DataStax Enterprise (DSE) to deploy complex clusters across multiple datacenters to enable sub 50 ms query responses for hundreds of billions of data points. From granular DNS traffic data, to aggregated counts for a variety of report dimensions, DSE at Dyn has been up since 2013 and has shined through upgrades, data center migrations, DDoS attacks and hardware failures. In this webinar, Principal Engineers Tim Chadwick and Rick Bross cover the requirements which led them to choose DSE as their go-to Big Data solution, the path which led to SPARK, and the lessons that we’ve learned in the process.
The document discusses emerging trends in software and services including:
1) Software as a Service and cloud computing which allows software to be delivered and consumed "as a service" with service level agreements.
2) The growth of massive data centers which are becoming large physical assets requiring significant capital expenditures.
3) The rise of "Dev-signers" or designer-developers who are combining development and design skills.
4) The integration of software and services will be key as local software interacts with internet services to provide combined capabilities.
Maximize Big Data ROI via Best of Breed Patterns and PracticesJeff Bertman
This presentation discusses maximizing ROI from big data technologies and architectures. It introduces the concept of a fitness technology landscape (FiTL) to evaluate different data platform options based on factors like cost. The presentation advocates using a polyglot or best-of-breed approach using multiple technologies to address diverse use cases. This includes using different technologies for extraction, loading, and transformation of data in integrated architectures. Maximizing ROI requires balancing factors like functionality, cost, scalability and other considerations for each specific use case.
The document provides a checklist for front-end performance optimization. It includes recommendations to establish performance metrics and goals, optimize assets like images, videos, fonts and JavaScript, choose frameworks and CDNs wisely, and set priorities to optimize the core experience for all users. Key metrics to target include a Time to Interactive under 5 seconds on 3G and First Input Delay below 100ms.
Highway to heaven - Microservices Meetup MunichChristian Deger
The document summarizes AutoScout24's transition from a monolithic architecture to microservices in the cloud. Some key points:
- They moved from an on-premise Microsoft-based stack to AWS and a microservices architecture using JVM and Linux.
- This was a major technical transformation to become "mobile first" and reduce costs and time to market while attracting new talent.
- They established architectural principles like event sourcing, autonomous teams, infrastructure as code, and shared-nothing.
- DynamoDB is now used as the "atom feed" between services while eliminating tight coupling.
- Teams are organized around business capabilities rather than projects to improve agility.
Questions Log: Dynamic Cubes – Set to Retire Transformer?Senturus
This document contains a questions log from a webinar about optimizing Cognos performance. It includes questions from webinar attendees about topics like using virtual cubes and dynamic cubes to address large data volumes, optimizing in-memory aggregates, hardware sizing requirements for dynamic cubes, and configuration considerations when using dynamic cubes. The questions are answered in detail to help attendees understand how to best implement and optimize dynamic cubes in Cognos.
The document discusses deep learning techniques for financial technology (FinTech) applications. It begins with examples of current deep learning uses in FinTech like trading algorithms, fraud detection, and personal finance assistants. It then covers topics like specialized compute hardware for deep learning training and inference, optimization techniques for CPUs and GPUs, and distributed training approaches. Finally, it discusses emerging areas like FPGA and quantum computing and provides resources for practitioners to start with deep learning for FinTech.
Dataiku productive application to production - pap is may 2015 Dataiku
This document discusses the development of predictive applications and outlines a vision for a platform called "Blue Box" that could help address many of the challenges in building and deploying these applications at scale. It notes that building predictive applications currently requires integrating multiple separate components. The document then describes desired features for the Blue Box platform, such as data cleansing, external data integration, model updating, decision logic, auditing, and serving predictions in real-time. It poses questions about how such a platform could be created, whether through open source or a commercial offering.
Flink Forward San Francisco 2018: David Reniz & Dahyr Vergara - "Real-time m...Flink Forward
“Customer experience is the next big battle ground for telcos,” proclaimed recently Amit Akhelikar, Global Director of Lynx Analytics at TM Forum Live! Asia in Singapore. But, how to fight in this battle? A common approach has been to keep “under control” some well-known network quality indicators, like dropped calls, radio access congestion, availability, and so on; but this has proven not to be enough to keep customers happy, like a siege weapon is not enough to conquer a city. But, what if it were possible to know how customers perceive services, at least most demanded ones, like web browsing or video streaming? That would be like a squad of archers ready to battle. And even having that, how to extract value of it and take actions in no time, giving our skilled archers the right targets? Meet CANVAS (Customer And Network Visualization and AnaltyticS), one of the first LATAM implementations of a Flink-based stream processing use case for a telco, which successfully combines leading and innovative technologies like Apache Hadoop, YARN, Kafka, Nifi, Druid and advanced visualizations with Flink core features like non-trivial stateful stream processing (joins, windows and aggregations on event time) and CEP capabilities for alarm generation, delivering a next-generation tool for SOC (Service Operation Center) teams.
The document provides an overview of the key topics and objectives covered in a computer architecture course. It discusses trends in technology that have led to changes in computer systems over time, including exponential increases in processor performance, memory capacity and speeds, and network bandwidth. It introduces various performance metrics and challenges in benchmarking systems. Important principles of computer design are outlined, such as Amdahl's law, exploiting parallelism, and making common cases fast. Quantitative analysis tools like simulations and queueing theory are also summarized.
This document provides an overview of the key topics and objectives covered in a computer architecture course. The course aims to evaluate instruction set design tradeoffs, advanced pipelining techniques, solutions to increasing memory latency, and qualitative and quantitative tradeoffs in modern computer system design. Key areas covered include instruction set architecture, memory hierarchies, pipelining, parallelism, and performance evaluation metrics.
Capacity Planning Infrastructure for Web Applications (Drupal)Ricardo Amaro
In this session we will try to solve a couple of recurring problems:
Site Launch and User expectations
Imagine a customer that provides a set of needs for hardware, sets a date and launches the site, but then he forgets to warn that they have sent out some (thousands of) emails to half the world announcing their new website launch! What do you think it will happen?
Of course launching a Drupal Site involves a lot of preparation steps and there are plenty of guides out there about common Drupal Launch Readiness Checklists which is not a problem anymore.
What we are really missing here is a Plan for Capacity.
Data Engineer's Lunch #85: Designing a Modern Data StackAnant Corporation
What are the design considerations that go into architecting a modern data warehouse? This presentation will cover some of the requirements analysis, design decisions, and execution challenges of building a modern data lake/data warehouse.
ClickHouse Paris Meetup. ClickHouse at ContentSquare, by Christophe Kalenzaga...Altinity Ltd
Christophe Kalenzaga and Vianney Foucault from ContentSquare summarize their experience using ClickHouse as their new backend database. They were seeking to replace their ElasticSearch clusters due to high storage, compute and scaling costs. Through a series of benchmarks, they found ClickHouse to be significantly faster than ElasticSearch or other alternatives, with response times up to 85% better. It also helped reduce their infrastructure costs. While ClickHouse requires effort to master and lacks some tooling, they found it stable and fast enough to recommend it for other companies seeking to replace expensive ElasticSearch deployments.
Enterprise application performance - Understanding & LearningsDhaval Shah
This document discusses enterprise application performance, including:
- Performance basics like response time, throughput, and availability
- Common metrics like response time, transactions per second, and concurrent users
- Factors that affect performance such as software issues, configuration settings, and hardware resources
- Case studies where the author analyzed memory leaks, optimized services, and addressed an inability to meet non-functional requirements
- Learnings around heap dump analysis, hotspot identification, and database monitoring
Design and optimization of ion propulsion dronebjmsejournal
Electric propulsion technology is widely used in many kinds of vehicles in recent years, and aircrafts are no exception. Technically, UAVs are electrically propelled but tend to produce a significant amount of noise and vibrations. Ion propulsion technology for drones is a potential solution to this problem. Ion propulsion technology is proven to be feasible in the earth’s atmosphere. The study presented in this article shows the design of EHD thrusters and power supply for ion propulsion drones along with performance optimization of high-voltage power supply for endurance in earth’s atmosphere.
Advanced control scheme of doubly fed induction generator for wind turbine us...IJECEIAES
This paper describes a speed control device for generating electrical energy on an electricity network based on the doubly fed induction generator (DFIG) used for wind power conversion systems. At first, a double-fed induction generator model was constructed. A control law is formulated to govern the flow of energy between the stator of a DFIG and the energy network using three types of controllers: proportional integral (PI), sliding mode controller (SMC) and second order sliding mode controller (SOSMC). Their different results in terms of power reference tracking, reaction to unexpected speed fluctuations, sensitivity to perturbations, and resilience against machine parameter alterations are compared. MATLAB/Simulink was used to conduct the simulations for the preceding study. Multiple simulations have shown very satisfying results, and the investigations demonstrate the efficacy and power-enhancing capabilities of the suggested control system.
More Related Content
Similar to Adtech scala-performance-tuning-150323223738-conversion-gate01
Flavius Ștef: Big Rewrites Without Big Risks at I T.A.K.E. UnconferenceMozaic Works
This document discusses strategies for incrementally rewriting software architecture without big risks. It suggests splitting large rewrites into smaller chunks and using techniques like faking functionality until it is implemented, incremental refactoring, and rewriting in small bites. Specific strategies discussed include researching prerequisites, creating proofs of concept, adding unit and performance tests incrementally, and removing dependencies one by one. Choosing the right refactoring is important, and catalogues of refactorings can provide guidance. An incremental process of analyzing, planning, refactoring, testing and integrating changes in small batches is recommended to validate progress toward goals at each step.
Incremental development is easy when we are talking about functionality. Story splitting has become quite popular as a technique lately.
But what about those cases when you need to do an architectural refactoring? Could incremental development be applied?
(Talk delivered during I T.A.K.E. Unconference 2015)
Dyn delivers exceptional Internet Performance. Enabling high quality services requires data centers around the globe. In order to manage services, customers need timely insight collected from all over the world. Dyn uses DataStax Enterprise (DSE) to deploy complex clusters across multiple datacenters to enable sub 50 ms query responses for hundreds of billions of data points. From granular DNS traffic data, to aggregated counts for a variety of report dimensions, DSE at Dyn has been up since 2013 and has shined through upgrades, data center migrations, DDoS attacks and hardware failures. In this webinar, Principal Engineers Tim Chadwick and Rick Bross cover the requirements which led them to choose DSE as their go-to Big Data solution, the path which led to SPARK, and the lessons that we’ve learned in the process.
The document discusses emerging trends in software and services including:
1) Software as a Service and cloud computing which allows software to be delivered and consumed "as a service" with service level agreements.
2) The growth of massive data centers which are becoming large physical assets requiring significant capital expenditures.
3) The rise of "Dev-signers" or designer-developers who are combining development and design skills.
4) The integration of software and services will be key as local software interacts with internet services to provide combined capabilities.
Maximize Big Data ROI via Best of Breed Patterns and PracticesJeff Bertman
This presentation discusses maximizing ROI from big data technologies and architectures. It introduces the concept of a fitness technology landscape (FiTL) to evaluate different data platform options based on factors like cost. The presentation advocates using a polyglot or best-of-breed approach using multiple technologies to address diverse use cases. This includes using different technologies for extraction, loading, and transformation of data in integrated architectures. Maximizing ROI requires balancing factors like functionality, cost, scalability and other considerations for each specific use case.
The document provides a checklist for front-end performance optimization. It includes recommendations to establish performance metrics and goals, optimize assets like images, videos, fonts and JavaScript, choose frameworks and CDNs wisely, and set priorities to optimize the core experience for all users. Key metrics to target include a Time to Interactive under 5 seconds on 3G and First Input Delay below 100ms.
Highway to heaven - Microservices Meetup MunichChristian Deger
The document summarizes AutoScout24's transition from a monolithic architecture to microservices in the cloud. Some key points:
- They moved from an on-premise Microsoft-based stack to AWS and a microservices architecture using JVM and Linux.
- This was a major technical transformation to become "mobile first" and reduce costs and time to market while attracting new talent.
- They established architectural principles like event sourcing, autonomous teams, infrastructure as code, and shared-nothing.
- DynamoDB is now used as the "atom feed" between services while eliminating tight coupling.
- Teams are organized around business capabilities rather than projects to improve agility.
Questions Log: Dynamic Cubes – Set to Retire Transformer?Senturus
This document contains a questions log from a webinar about optimizing Cognos performance. It includes questions from webinar attendees about topics like using virtual cubes and dynamic cubes to address large data volumes, optimizing in-memory aggregates, hardware sizing requirements for dynamic cubes, and configuration considerations when using dynamic cubes. The questions are answered in detail to help attendees understand how to best implement and optimize dynamic cubes in Cognos.
The document discusses deep learning techniques for financial technology (FinTech) applications. It begins with examples of current deep learning uses in FinTech like trading algorithms, fraud detection, and personal finance assistants. It then covers topics like specialized compute hardware for deep learning training and inference, optimization techniques for CPUs and GPUs, and distributed training approaches. Finally, it discusses emerging areas like FPGA and quantum computing and provides resources for practitioners to start with deep learning for FinTech.
Dataiku productive application to production - pap is may 2015 Dataiku
This document discusses the development of predictive applications and outlines a vision for a platform called "Blue Box" that could help address many of the challenges in building and deploying these applications at scale. It notes that building predictive applications currently requires integrating multiple separate components. The document then describes desired features for the Blue Box platform, such as data cleansing, external data integration, model updating, decision logic, auditing, and serving predictions in real-time. It poses questions about how such a platform could be created, whether through open source or a commercial offering.
Flink Forward San Francisco 2018: David Reniz & Dahyr Vergara - "Real-time m...Flink Forward
“Customer experience is the next big battle ground for telcos,” proclaimed recently Amit Akhelikar, Global Director of Lynx Analytics at TM Forum Live! Asia in Singapore. But, how to fight in this battle? A common approach has been to keep “under control” some well-known network quality indicators, like dropped calls, radio access congestion, availability, and so on; but this has proven not to be enough to keep customers happy, like a siege weapon is not enough to conquer a city. But, what if it were possible to know how customers perceive services, at least most demanded ones, like web browsing or video streaming? That would be like a squad of archers ready to battle. And even having that, how to extract value of it and take actions in no time, giving our skilled archers the right targets? Meet CANVAS (Customer And Network Visualization and AnaltyticS), one of the first LATAM implementations of a Flink-based stream processing use case for a telco, which successfully combines leading and innovative technologies like Apache Hadoop, YARN, Kafka, Nifi, Druid and advanced visualizations with Flink core features like non-trivial stateful stream processing (joins, windows and aggregations on event time) and CEP capabilities for alarm generation, delivering a next-generation tool for SOC (Service Operation Center) teams.
The document provides an overview of the key topics and objectives covered in a computer architecture course. It discusses trends in technology that have led to changes in computer systems over time, including exponential increases in processor performance, memory capacity and speeds, and network bandwidth. It introduces various performance metrics and challenges in benchmarking systems. Important principles of computer design are outlined, such as Amdahl's law, exploiting parallelism, and making common cases fast. Quantitative analysis tools like simulations and queueing theory are also summarized.
This document provides an overview of the key topics and objectives covered in a computer architecture course. The course aims to evaluate instruction set design tradeoffs, advanced pipelining techniques, solutions to increasing memory latency, and qualitative and quantitative tradeoffs in modern computer system design. Key areas covered include instruction set architecture, memory hierarchies, pipelining, parallelism, and performance evaluation metrics.
Capacity Planning Infrastructure for Web Applications (Drupal)Ricardo Amaro
In this session we will try to solve a couple of recurring problems:
Site Launch and User expectations
Imagine a customer that provides a set of needs for hardware, sets a date and launches the site, but then he forgets to warn that they have sent out some (thousands of) emails to half the world announcing their new website launch! What do you think it will happen?
Of course launching a Drupal Site involves a lot of preparation steps and there are plenty of guides out there about common Drupal Launch Readiness Checklists which is not a problem anymore.
What we are really missing here is a Plan for Capacity.
Data Engineer's Lunch #85: Designing a Modern Data StackAnant Corporation
What are the design considerations that go into architecting a modern data warehouse? This presentation will cover some of the requirements analysis, design decisions, and execution challenges of building a modern data lake/data warehouse.
ClickHouse Paris Meetup. ClickHouse at ContentSquare, by Christophe Kalenzaga...Altinity Ltd
Christophe Kalenzaga and Vianney Foucault from ContentSquare summarize their experience using ClickHouse as their new backend database. They were seeking to replace their ElasticSearch clusters due to high storage, compute and scaling costs. Through a series of benchmarks, they found ClickHouse to be significantly faster than ElasticSearch or other alternatives, with response times up to 85% better. It also helped reduce their infrastructure costs. While ClickHouse requires effort to master and lacks some tooling, they found it stable and fast enough to recommend it for other companies seeking to replace expensive ElasticSearch deployments.
Enterprise application performance - Understanding & LearningsDhaval Shah
This document discusses enterprise application performance, including:
- Performance basics like response time, throughput, and availability
- Common metrics like response time, transactions per second, and concurrent users
- Factors that affect performance such as software issues, configuration settings, and hardware resources
- Case studies where the author analyzed memory leaks, optimized services, and addressed an inability to meet non-functional requirements
- Learnings around heap dump analysis, hotspot identification, and database monitoring
Similar to Adtech scala-performance-tuning-150323223738-conversion-gate01 (20)
Design and optimization of ion propulsion dronebjmsejournal
Electric propulsion technology is widely used in many kinds of vehicles in recent years, and aircrafts are no exception. Technically, UAVs are electrically propelled but tend to produce a significant amount of noise and vibrations. Ion propulsion technology for drones is a potential solution to this problem. Ion propulsion technology is proven to be feasible in the earth’s atmosphere. The study presented in this article shows the design of EHD thrusters and power supply for ion propulsion drones along with performance optimization of high-voltage power supply for endurance in earth’s atmosphere.
Advanced control scheme of doubly fed induction generator for wind turbine us...IJECEIAES
This paper describes a speed control device for generating electrical energy on an electricity network based on the doubly fed induction generator (DFIG) used for wind power conversion systems. At first, a double-fed induction generator model was constructed. A control law is formulated to govern the flow of energy between the stator of a DFIG and the energy network using three types of controllers: proportional integral (PI), sliding mode controller (SMC) and second order sliding mode controller (SOSMC). Their different results in terms of power reference tracking, reaction to unexpected speed fluctuations, sensitivity to perturbations, and resilience against machine parameter alterations are compared. MATLAB/Simulink was used to conduct the simulations for the preceding study. Multiple simulations have shown very satisfying results, and the investigations demonstrate the efficacy and power-enhancing capabilities of the suggested control system.
Introduction- e - waste – definition - sources of e-waste– hazardous substances in e-waste - effects of e-waste on environment and human health- need for e-waste management– e-waste handling rules - waste minimization techniques for managing e-waste – recycling of e-waste - disposal treatment methods of e- waste – mechanism of extraction of precious metal from leaching solution-global Scenario of E-waste – E-waste in India- case studies.
Applications of artificial Intelligence in Mechanical Engineering.pdfAtif Razi
Historically, mechanical engineering has relied heavily on human expertise and empirical methods to solve complex problems. With the introduction of computer-aided design (CAD) and finite element analysis (FEA), the field took its first steps towards digitization. These tools allowed engineers to simulate and analyze mechanical systems with greater accuracy and efficiency. However, the sheer volume of data generated by modern engineering systems and the increasing complexity of these systems have necessitated more advanced analytical tools, paving the way for AI.
AI offers the capability to process vast amounts of data, identify patterns, and make predictions with a level of speed and accuracy unattainable by traditional methods. This has profound implications for mechanical engineering, enabling more efficient design processes, predictive maintenance strategies, and optimized manufacturing operations. AI-driven tools can learn from historical data, adapt to new information, and continuously improve their performance, making them invaluable in tackling the multifaceted challenges of modern mechanical engineering.
Batteries -Introduction – Types of Batteries – discharging and charging of battery - characteristics of battery –battery rating- various tests on battery- – Primary battery: silver button cell- Secondary battery :Ni-Cd battery-modern battery: lithium ion battery-maintenance of batteries-choices of batteries for electric vehicle applications.
Fuel Cells: Introduction- importance and classification of fuel cells - description, principle, components, applications of fuel cells: H2-O2 fuel cell, alkaline fuel cell, molten carbonate fuel cell and direct methanol fuel cells.
Rainfall intensity duration frequency curve statistical analysis and modeling...bijceesjournal
Using data from 41 years in Patna’ India’ the study’s goal is to analyze the trends of how often it rains on a weekly, seasonal, and annual basis (1981−2020). First, utilizing the intensity-duration-frequency (IDF) curve and the relationship by statistically analyzing rainfall’ the historical rainfall data set for Patna’ India’ during a 41 year period (1981−2020), was evaluated for its quality. Changes in the hydrologic cycle as a result of increased greenhouse gas emissions are expected to induce variations in the intensity, length, and frequency of precipitation events. One strategy to lessen vulnerability is to quantify probable changes and adapt to them. Techniques such as log-normal, normal, and Gumbel are used (EV-I). Distributions were created with durations of 1, 2, 3, 6, and 24 h and return times of 2, 5, 10, 25, and 100 years. There were also mathematical correlations discovered between rainfall and recurrence interval.
Findings: Based on findings, the Gumbel approach produced the highest intensity values, whereas the other approaches produced values that were close to each other. The data indicates that 461.9 mm of rain fell during the monsoon season’s 301st week. However, it was found that the 29th week had the greatest average rainfall, 92.6 mm. With 952.6 mm on average, the monsoon season saw the highest rainfall. Calculations revealed that the yearly rainfall averaged 1171.1 mm. Using Weibull’s method, the study was subsequently expanded to examine rainfall distribution at different recurrence intervals of 2, 5, 10, and 25 years. Rainfall and recurrence interval mathematical correlations were also developed. Further regression analysis revealed that short wave irrigation, wind direction, wind speed, pressure, relative humidity, and temperature all had a substantial influence on rainfall.
Originality and value: The results of the rainfall IDF curves can provide useful information to policymakers in making appropriate decisions in managing and minimizing floods in the study area.
artificial intelligence and data science contents.pptxGauravCar
What is artificial intelligence? Artificial intelligence is the ability of a computer or computer-controlled robot to perform tasks that are commonly associated with the intellectual processes characteristic of humans, such as the ability to reason.
› ...
Artificial intelligence (AI) | Definitio
1. ×
∼ Best Practice for Better Performance ∼
Scala Days 2015 San Francisco Un-conference
2015-03-19 @mogproject
Ad Tech Performance
Tuning
Scala
×
2. Agenda
About Demand Side Science
Introduction to Performance Tuning
Best Practice in Development
Japanese language version here:
http://www.slideshare.net/mogproject/scala-41799241
3. Yosuke Mizutani (@mogproject)
Joined Demand Side Science in April 2013
(thanks to Scala Conference in Japan 2013)
Full-stack engineer (want to be…)
Background: 9-year infrastructure engineer
About Me
10. Advertiser’s side of realtime ads bidding (RTB)
What is DSP
Supply Side Platform
11. Dec 2013
Moved into the group of Opt, the e-marketing agency
Oct 2014
Released dynamic creative tool unis
Brief History of DSS
× ×
12. unis is a third-party ad server which creates
dynamic and/or personalized ads under the rules.
http://www.opt.ne.jp/news/pr/detail/id=2492
unis
items on sale most popular
items
fixed items
re-targeting
13. With venture mind + advantage of Opt group …
Future of DSS
Demand
×
Side
Science×
14. We will create various products based on Science!
Future of DSS
???
×
???
Science×
22. Application goes wrong with high load
Bad latency under the specific condition
Slow batch execution than expectations
Slow development tools
Resolve an Issue
23. Very important especially in ad tech industry
Cost tends to go bigger and bigger
High traffic
Need to response in few milli seconds
Big database, big log data
Business requires
Benefit from mass delivery > Infra Investment
Reduce Infrastructure Cost
24. You need to care about
cost (≒ engineer’s time) and
risk (possibility to cause new trouble)
for performance tuning itself.
Don’t lose you goal
Scaling up/out of Infra can be the best
solution, naively
Don’t want to be perfect
25. We iterate
Basic of Performance Tuning
Measure metrics
× Find bottleneck
Try with hypothesis×
Don't take erratic steps.
27. ※CAUTION: This is my own impression
Bottle Neck in My Experience
others
1%
Network
4%
JVM parameter
5%
Library
5%
OS
10%
Scala
10%
Async・Thread
15%
Database
(RDBMS/NOSQL)
50%
29. Approximate timing for various operations
http://norvig.com/21-days.html#answers
execute typical instruction 1/1,000,000,000 sec = 1 nanosec
fetch from L1 cache memory 0.5 nanosec
branch misprediction 5 nanosec
fetch from L2 cache memory 7 nanosec
Mutex lock/unlock 25 nanosec
fetch from main memory 100 nanosec
send 2K bytes over 1Gbps network 20,000 nanosec
read 1MB sequentially from memory 250,000 nanosec
fetch from new disk location (seek) 8,000,000 nanosec
read 1MB sequentially from disk 20,000,000 nanosec
send packet US to Europe and back 150 milliseconds = 150,000,000 nanosec
30. If Typical Instruction Takes 1 second…
https://www.coursera.org/course/reactive week3-2
execute typical instruction 1 second
fetch from L1 cache memory 0.5 seconds
branch misprediction 5 seconds
fetch from L2 cache memory 7 seconds
Mutex lock/unlock ½ minute
fetch from main memory 1½ minute
send 2K bytes over 1Gbps network 5½ hours
read 1MB sequentially from memory 3 days
fetch from new disk location (seek) 13 weeks
read 1MB sequentially from disk 6½ months
send packet US to Europe and back 5 years
31. A batch
reads 1,000,000 files of 10KB
from disk
for each time.
Data size:
10KB × 1,000,000 ≒ 10GB
Horrible and True Story
32. Assuming 1,000,000 seeks are needed,
Estimated time:
8ms × 106
+ 20ms × 10,000 ≒ 8,200 sec ≒ 2.5 h
If there is one file of 10GB and only one seek is
needed,
Estimated time:
8ms × 1 + 20ms × 10,000 ≒ 200 sec ≒ 3.5 min
Horrible and True Story
33. Have Respect for the Disk Head
http://en.wikipedia.org/wiki/Hard_disk_drive
35. In the other words…
JVM Performance Triangle
Compactness
Throughput Responsiveness
36. C × T × R = a
JVM Performance Triangle
Tuning: vary C, T, R for fixed a
Optimization: increase a
Reference:
Everything I ever learned about JVM performance tuning
@twitter by Attila Szegedi
http://www.beyondlinux.com/files/pub/qconhangzhou2011/Everything%20I%20ever%20learned
%20about%20JVM%20performance%20tuning%20@twitter%28Attila%20Szegedi%29.pdf
37. Agenda
About Demand Side Science
Introduction to Performance Tuning
Best Practice in Development
38. 1. Requirement Definition / Feasibility
2. Basic Design
3. Detailed Design
4. Building Infrastructure / Coding
5. System Testing
6. System Operation / Maintenance
Development Process
Only topics related to performance will be covered.
39. Make the agreement with stakeholders
about performance requirement
Requirement Definition / Feasibility
How many user IDs
internet users in Japan: 100 million
unique browsers: 200 ~ x00 million
will increase?
data expiration cycle?
type of devices / browsers?
opt-out rate?
40. Requirement Definition / Feasibility
Number of deliver requests for ads
Number of impressions per month
In case 1 billion / month
=> mean: 400 QPS (Query Per Second)
=> if peak rate = 250%, then 1,000 QPS
For RTB, bid rate? win rate?
Goal response time? Content size?
Plans for increasing?
How about Cookie Sync?
41. Requirement Definition / Feasibility
Number of receiving trackers
Timing of firing tracker
Click rate?
Conversion(*) rate?
* A conversion occurs when the user performs the
specific action that the advertiser has defined as the
campaign goal.
e.g. buying a product in an online store
42. Requirement Definition / Feasibility
Requirement for aggregation
Indicates to be aggregated
Is unique counting needed?
Any exception rules?
Who and when
secondary processing by ad agency?
Update interval
Storage period
43. Requirement Definition / Feasibility
Hard limit by business side
Sales plan
Christmas selling?
Annual sales target?
Total budget
44. The most important thing is to provide numbers,
although it is extremely difficult to approximate
precisely in the turbulent world of ad tech.
Requirement Definition / Feasibility
Architecture design needs assumed value
Performance testing needs numeric goal
46. Threading model design
Reduce blocking
Future based
Callback & function composition
Actor based
Message passing
Thread pool design
We can’t know the appropriate thread pool
size unless we complete performance
testing in production.
Basic Design
47. Database design
Access pattern / Number of lookup
Data size per one record
Create model of distribution when the size
is not constant
Number of records
Rate of growth / retention period
Memory usage
At first, measure the performance of the
database itself
Detailed Design
48. Log design
Consider compression ratio for disk usage
Cache design
Some software needs the double of capacity
for processing backup (e.g. Redis)
Detailed Design
49. Simplicity and clarity come first
“It is far, far easier to make a correct
program fast than it is to make a fast
program correct”
— C++ Coding Standards: 101 Rules, Guidelines, and Best Practices (C
++ in-depth series)
Building Infrastructure / Coding
52. Avoid the algorithm which is worse than linear
as possible
Measure, don’t guess
http://en.wikipedia.org/wiki/Unix_philosophy
Building Infrastructure / Coding
53. SBT Plugin for running OpenJDK JMH
(Java Microbenchmark Harness: Benchmark tool for Java)
https://github.com/ktoso/sbt-jmh
Micro Benchmark: sbt-jmh
54. addSbtPlugin("pl.project13.scala" % "sbt-jmh" % "0.1.6")
Micro Benchmark: sbt-jmh
plugins.sbt
jmhSettings
build.sbt
import org.openjdk.jmh.annotations.Benchmark
class YourBench {
@Benchmark
def yourFunc(): Unit = ??? // write code to measure
}
YourBench.scala
Just put an annotation
55. > run -i 3 -wi 3 -f 1 -t 1
Micro Benchmark: sbt-jmh
Run benchmark in the sbt console
Number of
measurement
iterations to do
Number of warmup
iterations to do
How many times to forks
a single benchmark
Number of worker
threads to run with
56. [info] Benchmark Mode Samples Score Score error Units
[info] c.g.m.u.ContainsBench.listContains thrpt 3 41.033 25.573 ops/s
[info] c.g.m.u.ContainsBench.setContains thrpt 3 6.810 1.569 ops/s
Micro Benchmark: sbt-jmh
Result (excerpted)
By default, throughput score
will be displayed.
(larger is better)
http://mogproject.blogspot.jp/2014/10/micro-benchmark-in-scala-using-sbt-jmh.html
57. Scala Optimization Example
Use Scala collection correctly
Prefer recursion to function call
by Prof. Martin Odersky in Scala Matsuri 2014
Try optimization libraries
58. def f(xs: List[Int], acc: List[Int] = Nil): List[Int] = {
if (xs.length < 4) {
(xs.sum :: acc).reverse
} else {
val (y, ys) = xs.splitAt(4)
f(ys, y.sum :: acc)
}
}
Horrible and True Story pt.2
Group by 4 elements of List[Int], then
calculate each sum respectively
scala> f((1 to 10).toList)
res1: List[Int] = List(10, 26, 19)
Example
59. Horrible and True Story pt.2
List#length takes time proportional to the
length of the sequence
When the length of the parameter xs is n,
time complexity of List#length is O(n)
Implemented in LinearSeqOptimized#length
https://github.com/scala/scala/blob/v2.11.4/src/library/scala/collection/
LinearSeqOptimized.scala#L35-43
60. Horrible and True Story pt.2
In function f,
xs.length will be evaluated n / 4 + 1 times,
so number of execution of f is also
proportional to n
Therefore,
time complexity of function f is O(n2)
It becomes too slow with big n
61. Horrible and True Story pt.2
For your information, the following one-liner does
same work using built-in method
scala> (1 to 10).grouped(4).map(_.sum).toList
res2: List[Int] = List(10, 26, 19)
63. Library for optimising Scala collection
(by using macro)
http://scala-blitz.github.io/
Presentation in Scala Days 2014
https://parleys.com/play/
53a7d2c6e4b0543940d9e549/chapter0/
about
ScalaBlitz
64. System feature testing
Interface testing
Performance testing
Reliability testing
Security testing
Operation testing
System Testing
65. Simple load testing
Scenario load testing
mixed load with typical user operations
Aging test (continuously running test)
Performance Testing
66. Apache attached
Simple benchmark tool
http://httpd.apache.org/docs/2.2/programs/ab.html
Adequate for naive requirements
Latest version recommended
(Amazon Linux pre-installed version’s bug made me sick)
Example
ab - Apache Bench
ab -C <CookieName=Value> -n <NumberOfRequests> -c <Concurrency> “<URL>“
67. Result example (excerpted)
ab - Apache Bench
Benchmarking example.com (be patient)
Completed 1200 requests
Completed 2400 requests
(略)
Completed 10800 requests
Completed 12000 requests
Finished 12000 requests
(略)
Concurrency Level: 200
Time taken for tests: 7.365 seconds
Complete requests: 12000
Failed requests: 0
Write errors: 0
Total transferred: 166583579 bytes
HTML transferred: 160331058 bytes
Requests per second: 1629.31 [#/sec] (mean)
Time per request: 122.751 [ms] (mean)
Time per request: 0.614 [ms] (mean, across all concurrent requests)
Transfer rate: 22087.90 [Kbytes/sec] received
(略)
Percentage of the requests served within a certain time (ms)
50% 116
66% 138
75% 146
80% 150
90% 161
95% 170
98% 185
99% 208
100% 308 (longest request)
Requests per second
= QPS
69. An era of Apache JMeter has finished
Say good bye to scenario making with GUI
With Gatling,
You load write scenario with Scala DSL
Gatling
70. Care for the resource of stressor side
Resource of server (or PC)
Network router (CPU) can be bottleneck
Don’t tune two or more parameters at one
time
Leave change log and log files
Days for Testing and Tuning
72. Day-to-day logging and monitoring
Application log
GC log
Profiler
Anomaly detection from several metrics
Server resource (CPU, memory, disk, etc.)
abnormal response code
Latency
Trends visualization from several metrics
System Operation / Maintenance
76. stdout / stderr
Should redirect to file
Should NOT throw away to /dev/null
Result of thread dump
(kill - 3 <PROCESS_ID>) will be written
here
JVM Settings
77. SLF4J + Profiler
http://www.slf4j.org/extensions.html
Coding example
Profiler
import org.slf4j.profiler.Profiler
val profiler: Profiler = new Profiler(this.getClass.getSimpleName)
profiler.start(“A”)
doA()
profiler.start(“B”)
doB()
profiler.stop()
logger.warn(profiler.toString)
78. SLF4J + Profiler
Output example
Example:
Log the result of the profiler when
timeout occurs
Profiler
+ Profiler [BASIC]
|-- elapsed time [A] 220.487 milliseconds.
|-- elapsed time [B] 2499.866 milliseconds.
|-- elapsed time [OTHER] 3300.745 milliseconds.
|-- Total [BASIC] 6022.568 milliseconds.
79. For catching trends, not for anomaly detection
Operation is also necessary not to look over
the sign of change
Not only for infrastructure /application, but
business indicates
Who uses the console?
System user
System administrator
Application developer
Business manager
Trends Visualization