Deep learning optimization at alibaba by zhenliang zhang from Alibaba
•
1 like•422 views
Optimization problems are found everywhere at Alibaba, from e-commerce and logistics to cloud computing and machine learning. Some key optimization challenges at Alibaba include predicting user needs, effectively matching users with products, online advertising matching users with ads, scheduling jobs on computing resources, and warehouse and transportation management in logistics. The speaker's team focuses on developing optimization techniques to solve large-scale, non-convex problems for applications across Alibaba's various business units.
Report
Share
Report
Share
Recommended
Natural Intelligence the human factor in AI
Presented at AI NEXTCon Seattle 1/17-20, 2018
http://aisea18.xnextcon.com
join our free online AI group with 50,000+ tech engineers to learn and practice AI technology, including: latest AI news, tech articles/blogs, tech talks, tutorial videos, and hands-on workshop/codelabs, on machine learning, deep learning, data science, etc..
Explainable Fact Checking with Humans in-the-loop
Invited Keynote at KDD 2021 TrueFact Workshop: Making a Credible Web for Tomorrow, August 15, 2021.
https://www.microsoft.com/en-us/research/event/kdd-2021-truefact-workshop-making-a-credible-web-for-tomorrow/#!program-schedule
Personalized News and Video Recomendation System at LinkSure
In recent years, the Internet industry has shifted more and more towards digital content distribution through online services. This presentation provides an overview of the overall system design and architecture of LinkSure News and Video Recommendations, the challenges encountered in practice, and the lessons learned from the production deployment of these systems at LinkSure. Specifically, we will highlight how news selection and personalisation of recommendations are formulated and addressed at LinkSure. By presenting our experiences in applying techniques at the intersection of recommender systems, information retrieval, machine learning, and statistical modelling in a large-scale industrial setting and highlighting the open problems, we hope to stimulate further research and collaborations.
E-commerce企業におけるビッグデータ活用の取り組みと今後の展望
□Author
楽天株式会社 楽天技術研究所
平手 勇宇
□Description
約4万店舗、1億5,000万点を超える商品で構成される楽天市場では、様々な大規模データが生成されており、これらの大規模データを有効に活用し、事業を加速させることを目的とした様々な取り組みが行われている。本資料では、商品検索サポートシステム、商品データの整備、商品レビューを利用したナビゲーション等のビッグデータを活用した多様な取り組みを紹介する。
SMART Seminar Series: "From Big Data to Smart data"
"From Big Data to Smart data"
Jie (Jack) Yang, Associate Research Fellow, SMART Infrastructure Facility, presented a summary of his research as part of the SMART Seminar Series on 28 April 2016.
For more information, visit the event page at: http://smart.uow.edu.au/events/UOW212890.html.
Deep learning for e-commerce: current status and future prospects
Deep learning is the prime avenue for Artificial Intelligence, with spectacular accomplishments in diverse fields such as computer vision, natural language processing, and board games such as Go. Its impact on e-commerce is already significant and will continue to grow in future years. In this talk, we will review some of the successful deep learning algorithms in light of their current and expected impact on e-commerce.
Data Science and Machine Learning for eCommerce and Retail
This document discusses how machine learning and data science can be applied to ecommerce and retail. It outlines several machine learning problems that are relevant, such as search ranking, typeahead, spell correction, cold start recommendations, and inventory management. It also discusses the large data volumes and processing needs, as well as hardware requirements, for deploying machine learning solutions at large retailers.
The New Model
(Big) Data Processing for Next Generation Business Value. Presented at the Leaders Buildings Leaders Conference, held at Union College on April 3, 2015.
https://www.ucollege.edu/academics/business-and-computer-science/leaders-building-leaders
This presentation is a semi-technical overview of big data and related use-cases, the Apache Hadoop software stack, and some example data-science / analysis models.
Recommended
Natural Intelligence the human factor in AI
Presented at AI NEXTCon Seattle 1/17-20, 2018
http://aisea18.xnextcon.com
join our free online AI group with 50,000+ tech engineers to learn and practice AI technology, including: latest AI news, tech articles/blogs, tech talks, tutorial videos, and hands-on workshop/codelabs, on machine learning, deep learning, data science, etc..
Explainable Fact Checking with Humans in-the-loop
Invited Keynote at KDD 2021 TrueFact Workshop: Making a Credible Web for Tomorrow, August 15, 2021.
https://www.microsoft.com/en-us/research/event/kdd-2021-truefact-workshop-making-a-credible-web-for-tomorrow/#!program-schedule
Personalized News and Video Recomendation System at LinkSure
In recent years, the Internet industry has shifted more and more towards digital content distribution through online services. This presentation provides an overview of the overall system design and architecture of LinkSure News and Video Recommendations, the challenges encountered in practice, and the lessons learned from the production deployment of these systems at LinkSure. Specifically, we will highlight how news selection and personalisation of recommendations are formulated and addressed at LinkSure. By presenting our experiences in applying techniques at the intersection of recommender systems, information retrieval, machine learning, and statistical modelling in a large-scale industrial setting and highlighting the open problems, we hope to stimulate further research and collaborations.
E-commerce企業におけるビッグデータ活用の取り組みと今後の展望
□Author
楽天株式会社 楽天技術研究所
平手 勇宇
□Description
約4万店舗、1億5,000万点を超える商品で構成される楽天市場では、様々な大規模データが生成されており、これらの大規模データを有効に活用し、事業を加速させることを目的とした様々な取り組みが行われている。本資料では、商品検索サポートシステム、商品データの整備、商品レビューを利用したナビゲーション等のビッグデータを活用した多様な取り組みを紹介する。
SMART Seminar Series: "From Big Data to Smart data"
"From Big Data to Smart data"
Jie (Jack) Yang, Associate Research Fellow, SMART Infrastructure Facility, presented a summary of his research as part of the SMART Seminar Series on 28 April 2016.
For more information, visit the event page at: http://smart.uow.edu.au/events/UOW212890.html.
Deep learning for e-commerce: current status and future prospects
Deep learning is the prime avenue for Artificial Intelligence, with spectacular accomplishments in diverse fields such as computer vision, natural language processing, and board games such as Go. Its impact on e-commerce is already significant and will continue to grow in future years. In this talk, we will review some of the successful deep learning algorithms in light of their current and expected impact on e-commerce.
Data Science and Machine Learning for eCommerce and Retail
This document discusses how machine learning and data science can be applied to ecommerce and retail. It outlines several machine learning problems that are relevant, such as search ranking, typeahead, spell correction, cold start recommendations, and inventory management. It also discusses the large data volumes and processing needs, as well as hardware requirements, for deploying machine learning solutions at large retailers.
The New Model
(Big) Data Processing for Next Generation Business Value. Presented at the Leaders Buildings Leaders Conference, held at Union College on April 3, 2015.
https://www.ucollege.edu/academics/business-and-computer-science/leaders-building-leaders
This presentation is a semi-technical overview of big data and related use-cases, the Apache Hadoop software stack, and some example data-science / analysis models.
Dubbo and Weidian's practice on micro-service architecture
Weidian is a mobile app that helps sellers open shops. It has gone through several architectural stages: monolithic, multi-module, and microservices using Dubbo. Challenges in the microservices stage included module splitting, automated deployment, call tracing, monitoring and alarms, flow control, and high availability during data center migration. Solutions involved domain-driven design, DevOps practices like Docker and configuration management, and modifying Dubbo components. Future plans include contributing back to the Dubbo community, supporting multiple languages through agents, and exploring reactive and functional programming approaches.
e-Minggu 1 Pengantar e-Business & e-Commerce.ppt
Here are two Indonesian e-business/e-commerce implementers and an assessment of whether they are primarily e-business or e-commerce implementers:
1. Tokopedia - Tokopedia is primarily an e-commerce platform that allows buyers and sellers to transact online. It would be considered an e-commerce implementer as the transactions are between businesses/individuals and consumers.
2. Gojek - Gojek provides a platform for online transportation, food delivery, logistics and more. While consumers can use the platform, Gojek also facilitates business processes between partners (drivers, merchants). It has elements of both e-business and e-commerce so could be considered a hybrid implementer.
Kai Wähner – Real World Use Cases for Realtime In-Memory Computing - NoSQL ma...
Kai Wähner – Real World Use Cases for Realtime In-Memory Computing
NoSQL is not just about different storage alternatives such as document store, key value store, graphs or column-based databases. The hardware is also getting much more important. Besides common disks and SSDs, enterprises begin to use in-memory storages more and more because a distributed in-memory data grid provides very fast data access and update. While its performance will vary depending on multiple factors, it is not uncommon to be 100 times faster than corresponding database implementations. For this reason and others described in this session, in-memory computing is a great solution for lifting the burden of big data, reducing reliance on costly transactional systems, and building highly scalable, fault-tolerant applications.The session begins with a short introduction to in-memory computing. Afterwards, different frameworks and product alternatives are discussed for implementing in-memory solutions. Finally, the main part of this session shows several different real world uses cases where in-memory computing delivers business value by supercharging the infrastructure.
Yodle Ecosystem Trends
Presentation by the Chief Product and Marketing Officer of Yodle.com, Louis Gagnon, going over the state of the hyperlocalized marketing industry, how Yodle is playing in the market, and the implications for success in the marketplace.
Smart DataCenters
All datacenters, regardless of their size, have specific objectives. Until now, balancing datacenter
objectives for efficiency, capacity, space utilization and availability has been difficult without making
sacrifices. That's why NxtGen has developed the Smart Solutions family of infrastructure offerings.
Each Smart Solutions offering addresses specific needs with rapidly deployable solutions that cost effectively increase efficiency, add datacenter capacity, and improve IT control.
These solutions contain the industry's leading power, precision cooling and management systems to
achieve efficiency without compromise in IT environments of all sizes.
SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telec...
SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telec...IBM India Smarter Computing
The document summarizes a technical white paper produced jointly by SAP and IBM that describes a proof of concept project testing the performance of SAP Convergent Invoicing software handling large data volumes. The project used an IBM enterprise architecture and demonstrated meeting key performance indicators for a telecommunications scenario, including uploading 1.5 billion items, billing 2.5 million customers, and invoicing in under 18 hours. Using IBM Easy Tier storage software reduced the processing time by over 30% and IBM Storwize V7000 storage eliminated performance bottlenecks.SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telec...
SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telec...IBM India Smarter Computing
This technical white paper describes a project to test the performance of the SAP Convergent Invoicing package in the scenario of a company having 50 million active customers. To know more about IBM’s portfolio of products for big data, visit http://bit.ly/T0NnOs.
Customer Story: Elastic Stack을 이용한 게임 서비스 통합 로깅 플랫폼
https://www.elastic.co/elasticon/tour/2019/seoul/devsisters-game-service-integration-logging-platform-using-elastic-stack
데브시스터즈에서 서비스하고 있는 모든 게임에서 생성된 각종 로그들은 하나의 통합 로깅 플랫폼으로 수집되어 데이터 분석, 서버 운영 및 트러블슈팅, 고객 문의 대응 등 다양한 용도로 사용하고 있습니다. 본 발표에서는 이 통합 로깅 플랫폼에서 Elastic Stack이 어떻게 사용되는지 다룹니다. 구체적으로, Filebeat를 이용한 Kubernetes와 AWS EC2 환경에서의 로그 수집, Elasticsearch를 이용한 로그 조회 서비스 구성에 대해 살펴보며, 서비스 구축 및 운영 과정에서 발생한 이슈들의 해결 과정, 그리고 앞으로의 미래에 대해 이야기합니다.
Monitoring aws finance spend
Slides of the talk I gave at the Dublin DevOps meetup on April 12.
https://www.meetup.com/Dublin-DevOps-Meetup/events/238425127/
Enterprise architectsview 2015-apr
In the world of big data, legacy modernization, siloed organizations, empowered customers, and mobile devices, making informed choices about your enterprise infrastructure has become more important than ever. The alternatives are abundant, and the successful Enterprise Architect must constantly discern which new technology is just a shiny object and which will add true business value.
Accounts Receivable Management Software Buyers Guide
There are many accounts receivable management software solutions available on the market today- but which one will best fit your business needs? This slide deck will provide you with everything you need to know about selecting accounts receivable management software to help you improve invoice collection, reduce days sales outstanding, improve cash flow, and more.
Anytime Collect is a proven credit and collection management software solution that helps automate the entire invoice management and collection process. Learn more at www.AnytimeCollect.com
Digital Strategy for future business
Digital strategies require cross-functional teams to maximize business benefits from data and technology. They involve understanding customers, competitors, innovation, data, and partnerships. Customers can provide competence and shape personalized experiences. Competitors may redefine industries or transform products to platforms. Data is an asset when converted to meaningful information and visualizations to guide decisions. Innovation starts by discovering customer needs and testing minimum viable products.
Enabling real interactive BI on Hadoop
Providing true interactive and scalable BI on Hadoop is proven to be one of the biggest challenges that is preventing completion of legacy EDW OLAP system transit to Hadoop. While we have all seen many benchmarks running consecutive queries claiming success, having thousands of concurrent business users sending complicated generated queries from their dashboards over billions of records while delivering interactive speed is yet to be seen.
In this session we will discuss how an architecture that replaces full-scan brute-force approach with adaptive indexing and auto-generated cubes can dramatically reduce the resources and effort per query, resulting in interactive performance for high concurrency workloads and explain how this is achieved with minimum data engineering efforts. We will also discuss how this architecture can be seamlessly integrated with Hive to provide a complete OLAP-on-Hadoop solution.
Session will include live demo of complex business dashboards connected to Hive and accessing billions of rows at interactive speed.
Speaker
Boaz Raufman, CTO and Co-Founder, JethroData
GraphTour - Popular Use Cases
This document provides an overview of Neo4j, including:
- Neo4j is a graph database company with over 260 employees and $80M in funding.
- It has over 10M downloads and 275+ enterprise customers, including top retail, financial, and software firms.
- Neo4j is used for common use cases like recommendations, fraud detection, knowledge graphs, and master data management by customers like NASA, eBay, Walmart, and Marriott.
- The document describes several customer case studies and how Neo4j helped organizations like NASA, ICIJ, and others solve problems and gain insights.
Epgp(one year) 2009-10_it strategy_groupassignment_case-dell_final
Dell Computer adopted an internet-based direct sales model in 1996. This allowed them to eliminate inventory costs and achieve 60x inventory turnover compared to competitors. Dell customized PCs to customer requirements and coordinated its entire value network through the internet with suppliers and partners. This virtual corporation model placed Dell at the center of all communication. Dell leveraged e-commerce to improve customer reach and enable online ordering. However, Dell struggled to convert its e-marketplace into success and lacked integration between its IT tools. Its future strategy focuses on increasing e-commerce usage in developing countries through education and incentives, providing more value-added services through business analytics, and improving its partner network and local support for geographical expansion.
Hadoop Boosts Profits in Media and Telecom Industry
1) The document discusses 21 use cases for using Hadoop in the telecommunications industry across network infrastructure, service and security, sales and marketing, and new business functions.
2) It provides details on specific use cases such as using Hadoop for network capacity planning, customer experience analytics based on call detail records, and improving contact center and field service productivity.
3) The document also outlines the typical journey an organization takes to become data-driven and the roles needed in a center of excellence at different stages of the journey.
Project presentation.pptx
This document summarizes a minor project on e-commerce. It discusses the team members, introduces e-commerce and how it works. It identifies problems like building trust, payments and navigation. It covers the feasibility study, objectives, literature review, proposed methodology including functional and non-functional requirements. It also discusses hardware, software requirements and includes class, DFD, use case, ER diagrams and a Gantt chart. The project aims to reach 40% traffic while minimizing costs and improving customer experience through an effective e-commerce website.
Big data in telecom
Big data is generated from a variety of sources like web data, purchases, social networks, sensors, and IoT devices. Telecom companies process exabytes and zettabytes of data daily, including call detail records, network configuration data, and customer information. This big data is analyzed to enhance customer experience through personalization, predict churn, and optimize networks. Analytics also helps with operations, data monetization through services, and identifying new revenue streams from IoT and M2M data. Frameworks like Hadoop and MapReduce are used to analyze this distributed big data across clusters in a distributed manner for faster insights.
How to Achieve Digital World Class Performance with AR Automation
How to Achieve Digital World Class Performance with AR Automation
https://www.emagia.com/resources/ebooks/how-to-achieve-digital-world-class-performance-with-ar-automation/
Zeng Ming of Alibaba: "Making Sense of Big Data"
Ming Zeng discusses the challenges of managing big data and how Alibaba addresses it. Traditional data collection has well-defined populations and expensive collection, while web data has undefined populations and is generated automatically. Alibaba Cloud Computing provides scalable infrastructure to store and analyze vast amounts of data from multiple sources. This enables new datadriven businesses like microloans through modeling of Alibaba's consumer data. Big data pushes technology innovation in cloud computing which then enables business innovation.
Walk Through a Real World ML Production Project
Success in productionizing ML models is difficult to achieve due to tools, processes and operational procedures. In this session, we demonstrate how data scientists and ML engineers collaborate and efficiently deploy models to production with the Wallaroo platform.
Using a real world scenario we will click down into the ML production journey that Data Scientists and ML engineers go through to take ML models into production. In this session you will learn:
The current pain points and blockers to production
The 2 persona roles in the ML production process. Data Scientist (DS) and ML Engineer
How the ML engineer creates a workspace in Wallaroo, and invites the DS to collaborate
How the DS uploads and deploys models to WL performing simple validation checks on output
How the ML Engineer can check model health (inference speed, etc)
How the DS checks logs, looks for anomalies
How the DS switches model in the pipeline
Speakers: Nina Zumel, Martin Bald
Redefining MLOps with Model Deployment, Management and Observability in Produ...
Tech talk: https://www.aicamp.ai/event/eventdetails/W2022052410
What happens after your machine learning models are deployed in production? How do you make sure that your model performance does not degrade as data and the world change?
The constantly changing data creates challenges for data scientists and engineering teams on how to detect which models have been affected and how to get their ML applications up and running seamlessly.
In this session we will take a deep dive into the new ML model monitoring and drift detection technology. We will discuss:
- How to track the ongoing accuracy of their models in production
- How to immediately detect drift before it causes significant damage to the business
- How to locate the cause of model drifting in live environments.
We will also discuss how data scientists and ML engineers can collaborate effectively using their respective tools to identify issues and take the necessary actions with a live demo and a real world use case.
Speaker: Younes Amar, Head of Product Wallaroo AI.
Resources: https://docs.wallaroo.ai/
More Related Content
Similar to Deep learning optimization at alibaba by zhenliang zhang from Alibaba
Dubbo and Weidian's practice on micro-service architecture
Weidian is a mobile app that helps sellers open shops. It has gone through several architectural stages: monolithic, multi-module, and microservices using Dubbo. Challenges in the microservices stage included module splitting, automated deployment, call tracing, monitoring and alarms, flow control, and high availability during data center migration. Solutions involved domain-driven design, DevOps practices like Docker and configuration management, and modifying Dubbo components. Future plans include contributing back to the Dubbo community, supporting multiple languages through agents, and exploring reactive and functional programming approaches.
e-Minggu 1 Pengantar e-Business & e-Commerce.ppt
Here are two Indonesian e-business/e-commerce implementers and an assessment of whether they are primarily e-business or e-commerce implementers:
1. Tokopedia - Tokopedia is primarily an e-commerce platform that allows buyers and sellers to transact online. It would be considered an e-commerce implementer as the transactions are between businesses/individuals and consumers.
2. Gojek - Gojek provides a platform for online transportation, food delivery, logistics and more. While consumers can use the platform, Gojek also facilitates business processes between partners (drivers, merchants). It has elements of both e-business and e-commerce so could be considered a hybrid implementer.
Kai Wähner – Real World Use Cases for Realtime In-Memory Computing - NoSQL ma...
Kai Wähner – Real World Use Cases for Realtime In-Memory Computing
NoSQL is not just about different storage alternatives such as document store, key value store, graphs or column-based databases. The hardware is also getting much more important. Besides common disks and SSDs, enterprises begin to use in-memory storages more and more because a distributed in-memory data grid provides very fast data access and update. While its performance will vary depending on multiple factors, it is not uncommon to be 100 times faster than corresponding database implementations. For this reason and others described in this session, in-memory computing is a great solution for lifting the burden of big data, reducing reliance on costly transactional systems, and building highly scalable, fault-tolerant applications.The session begins with a short introduction to in-memory computing. Afterwards, different frameworks and product alternatives are discussed for implementing in-memory solutions. Finally, the main part of this session shows several different real world uses cases where in-memory computing delivers business value by supercharging the infrastructure.
Yodle Ecosystem Trends
Presentation by the Chief Product and Marketing Officer of Yodle.com, Louis Gagnon, going over the state of the hyperlocalized marketing industry, how Yodle is playing in the market, and the implications for success in the marketplace.
Smart DataCenters
All datacenters, regardless of their size, have specific objectives. Until now, balancing datacenter
objectives for efficiency, capacity, space utilization and availability has been difficult without making
sacrifices. That's why NxtGen has developed the Smart Solutions family of infrastructure offerings.
Each Smart Solutions offering addresses specific needs with rapidly deployable solutions that cost effectively increase efficiency, add datacenter capacity, and improve IT control.
These solutions contain the industry's leading power, precision cooling and management systems to
achieve efficiency without compromise in IT environments of all sizes.
SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telec...
SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telec...IBM India Smarter Computing
The document summarizes a technical white paper produced jointly by SAP and IBM that describes a proof of concept project testing the performance of SAP Convergent Invoicing software handling large data volumes. The project used an IBM enterprise architecture and demonstrated meeting key performance indicators for a telecommunications scenario, including uploading 1.5 billion items, billing 2.5 million customers, and invoicing in under 18 hours. Using IBM Easy Tier storage software reduced the processing time by over 30% and IBM Storwize V7000 storage eliminated performance bottlenecks.SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telec...
SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telec...IBM India Smarter Computing
This technical white paper describes a project to test the performance of the SAP Convergent Invoicing package in the scenario of a company having 50 million active customers. To know more about IBM’s portfolio of products for big data, visit http://bit.ly/T0NnOs.
Customer Story: Elastic Stack을 이용한 게임 서비스 통합 로깅 플랫폼
https://www.elastic.co/elasticon/tour/2019/seoul/devsisters-game-service-integration-logging-platform-using-elastic-stack
데브시스터즈에서 서비스하고 있는 모든 게임에서 생성된 각종 로그들은 하나의 통합 로깅 플랫폼으로 수집되어 데이터 분석, 서버 운영 및 트러블슈팅, 고객 문의 대응 등 다양한 용도로 사용하고 있습니다. 본 발표에서는 이 통합 로깅 플랫폼에서 Elastic Stack이 어떻게 사용되는지 다룹니다. 구체적으로, Filebeat를 이용한 Kubernetes와 AWS EC2 환경에서의 로그 수집, Elasticsearch를 이용한 로그 조회 서비스 구성에 대해 살펴보며, 서비스 구축 및 운영 과정에서 발생한 이슈들의 해결 과정, 그리고 앞으로의 미래에 대해 이야기합니다.
Monitoring aws finance spend
Slides of the talk I gave at the Dublin DevOps meetup on April 12.
https://www.meetup.com/Dublin-DevOps-Meetup/events/238425127/
Enterprise architectsview 2015-apr
In the world of big data, legacy modernization, siloed organizations, empowered customers, and mobile devices, making informed choices about your enterprise infrastructure has become more important than ever. The alternatives are abundant, and the successful Enterprise Architect must constantly discern which new technology is just a shiny object and which will add true business value.
Accounts Receivable Management Software Buyers Guide
There are many accounts receivable management software solutions available on the market today- but which one will best fit your business needs? This slide deck will provide you with everything you need to know about selecting accounts receivable management software to help you improve invoice collection, reduce days sales outstanding, improve cash flow, and more.
Anytime Collect is a proven credit and collection management software solution that helps automate the entire invoice management and collection process. Learn more at www.AnytimeCollect.com
Digital Strategy for future business
Digital strategies require cross-functional teams to maximize business benefits from data and technology. They involve understanding customers, competitors, innovation, data, and partnerships. Customers can provide competence and shape personalized experiences. Competitors may redefine industries or transform products to platforms. Data is an asset when converted to meaningful information and visualizations to guide decisions. Innovation starts by discovering customer needs and testing minimum viable products.
Enabling real interactive BI on Hadoop
Providing true interactive and scalable BI on Hadoop is proven to be one of the biggest challenges that is preventing completion of legacy EDW OLAP system transit to Hadoop. While we have all seen many benchmarks running consecutive queries claiming success, having thousands of concurrent business users sending complicated generated queries from their dashboards over billions of records while delivering interactive speed is yet to be seen.
In this session we will discuss how an architecture that replaces full-scan brute-force approach with adaptive indexing and auto-generated cubes can dramatically reduce the resources and effort per query, resulting in interactive performance for high concurrency workloads and explain how this is achieved with minimum data engineering efforts. We will also discuss how this architecture can be seamlessly integrated with Hive to provide a complete OLAP-on-Hadoop solution.
Session will include live demo of complex business dashboards connected to Hive and accessing billions of rows at interactive speed.
Speaker
Boaz Raufman, CTO and Co-Founder, JethroData
GraphTour - Popular Use Cases
This document provides an overview of Neo4j, including:
- Neo4j is a graph database company with over 260 employees and $80M in funding.
- It has over 10M downloads and 275+ enterprise customers, including top retail, financial, and software firms.
- Neo4j is used for common use cases like recommendations, fraud detection, knowledge graphs, and master data management by customers like NASA, eBay, Walmart, and Marriott.
- The document describes several customer case studies and how Neo4j helped organizations like NASA, ICIJ, and others solve problems and gain insights.
Epgp(one year) 2009-10_it strategy_groupassignment_case-dell_final
Dell Computer adopted an internet-based direct sales model in 1996. This allowed them to eliminate inventory costs and achieve 60x inventory turnover compared to competitors. Dell customized PCs to customer requirements and coordinated its entire value network through the internet with suppliers and partners. This virtual corporation model placed Dell at the center of all communication. Dell leveraged e-commerce to improve customer reach and enable online ordering. However, Dell struggled to convert its e-marketplace into success and lacked integration between its IT tools. Its future strategy focuses on increasing e-commerce usage in developing countries through education and incentives, providing more value-added services through business analytics, and improving its partner network and local support for geographical expansion.
Hadoop Boosts Profits in Media and Telecom Industry
1) The document discusses 21 use cases for using Hadoop in the telecommunications industry across network infrastructure, service and security, sales and marketing, and new business functions.
2) It provides details on specific use cases such as using Hadoop for network capacity planning, customer experience analytics based on call detail records, and improving contact center and field service productivity.
3) The document also outlines the typical journey an organization takes to become data-driven and the roles needed in a center of excellence at different stages of the journey.
Project presentation.pptx
This document summarizes a minor project on e-commerce. It discusses the team members, introduces e-commerce and how it works. It identifies problems like building trust, payments and navigation. It covers the feasibility study, objectives, literature review, proposed methodology including functional and non-functional requirements. It also discusses hardware, software requirements and includes class, DFD, use case, ER diagrams and a Gantt chart. The project aims to reach 40% traffic while minimizing costs and improving customer experience through an effective e-commerce website.
Big data in telecom
Big data is generated from a variety of sources like web data, purchases, social networks, sensors, and IoT devices. Telecom companies process exabytes and zettabytes of data daily, including call detail records, network configuration data, and customer information. This big data is analyzed to enhance customer experience through personalization, predict churn, and optimize networks. Analytics also helps with operations, data monetization through services, and identifying new revenue streams from IoT and M2M data. Frameworks like Hadoop and MapReduce are used to analyze this distributed big data across clusters in a distributed manner for faster insights.
How to Achieve Digital World Class Performance with AR Automation
How to Achieve Digital World Class Performance with AR Automation
https://www.emagia.com/resources/ebooks/how-to-achieve-digital-world-class-performance-with-ar-automation/
Zeng Ming of Alibaba: "Making Sense of Big Data"
Ming Zeng discusses the challenges of managing big data and how Alibaba addresses it. Traditional data collection has well-defined populations and expensive collection, while web data has undefined populations and is generated automatically. Alibaba Cloud Computing provides scalable infrastructure to store and analyze vast amounts of data from multiple sources. This enables new datadriven businesses like microloans through modeling of Alibaba's consumer data. Big data pushes technology innovation in cloud computing which then enables business innovation.
Similar to Deep learning optimization at alibaba by zhenliang zhang from Alibaba (20)
Dubbo and Weidian's practice on micro-service architecture
Dubbo and Weidian's practice on micro-service architecture
Kai Wähner – Real World Use Cases for Realtime In-Memory Computing - NoSQL ma...
Kai Wähner – Real World Use Cases for Realtime In-Memory Computing - NoSQL ma...
SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telec...
SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telec...
SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telec...
SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telec...
Customer Story: Elastic Stack을 이용한 게임 서비스 통합 로깅 플랫폼
Customer Story: Elastic Stack을 이용한 게임 서비스 통합 로깅 플랫폼
Accounts Receivable Management Software Buyers Guide
Accounts Receivable Management Software Buyers Guide
Epgp(one year) 2009-10_it strategy_groupassignment_case-dell_final
Epgp(one year) 2009-10_it strategy_groupassignment_case-dell_final
Hadoop Boosts Profits in Media and Telecom Industry
Hadoop Boosts Profits in Media and Telecom Industry
How to Achieve Digital World Class Performance with AR Automation
How to Achieve Digital World Class Performance with AR Automation
More from Bill Liu
Walk Through a Real World ML Production Project
Success in productionizing ML models is difficult to achieve due to tools, processes and operational procedures. In this session, we demonstrate how data scientists and ML engineers collaborate and efficiently deploy models to production with the Wallaroo platform.
Using a real world scenario we will click down into the ML production journey that Data Scientists and ML engineers go through to take ML models into production. In this session you will learn:
The current pain points and blockers to production
The 2 persona roles in the ML production process. Data Scientist (DS) and ML Engineer
How the ML engineer creates a workspace in Wallaroo, and invites the DS to collaborate
How the DS uploads and deploys models to WL performing simple validation checks on output
How the ML Engineer can check model health (inference speed, etc)
How the DS checks logs, looks for anomalies
How the DS switches model in the pipeline
Speakers: Nina Zumel, Martin Bald
Redefining MLOps with Model Deployment, Management and Observability in Produ...
Tech talk: https://www.aicamp.ai/event/eventdetails/W2022052410
What happens after your machine learning models are deployed in production? How do you make sure that your model performance does not degrade as data and the world change?
The constantly changing data creates challenges for data scientists and engineering teams on how to detect which models have been affected and how to get their ML applications up and running seamlessly.
In this session we will take a deep dive into the new ML model monitoring and drift detection technology. We will discuss:
- How to track the ongoing accuracy of their models in production
- How to immediately detect drift before it causes significant damage to the business
- How to locate the cause of model drifting in live environments.
We will also discuss how data scientists and ML engineers can collaborate effectively using their respective tools to identify issues and take the necessary actions with a live demo and a real world use case.
Speaker: Younes Amar, Head of Product Wallaroo AI.
Resources: https://docs.wallaroo.ai/
Productizing Machine Learning at the Edge
These days, training of the Machine Learning models at the device Edge is still a risky endeavor. It is frequently considered a purely academic subject with little value for real-life product development.
In her talk, Vera will challenge this misconception, talk about the advantages of learning at the Edge and guide you through the Edge learning decision-making framework and design principles.
https://www.aicamp.ai/event/eventdetails/W2021102210
Transformers in Vision: From Zero to Hero
Attention Is All You Need.
With these simple words, the Deep Learning industry was forever changed. Transformers were initially introduced in the field of Natural Language Processing to enhance language translation, but they demonstrated astonishing results even outside language processing. In particular, they recently spread in the Computer Vision community, advancing the state-of-the-art on many vision tasks. But what are Transformers? What is the mechanism of self-attention, and do we really need it? How did they revolutionize Computer Vision? Will they ever replace convolutional neural networks?
These and many other questions will be answered during the talk.
In this tech talk, we will discuss:
- A piece of history: Why did we need a new architecture?
- What is self-attention, and where does this concept come from?
- The Transformer architecture and its mechanisms
- Vision Transformers: An Image is worth 16x16 words
- Video Understanding using Transformers: the space + time approach
- The scale and data problem: Is Attention what we really need?
- The future of Computer Vision through Transformers
Speaker: Davide Coccomini, Nicola Messina
Website: https://www.aicamp.ai/event/eventdetails/W2021101110
Deep AutoViML For Tensorflow Models and MLOps Workflows
deep_autoviml is a powerful new deep learning library with a very simple design goal: Make it as easy as possible for novices and experts alike to experiment with and build tensorflow.keras preprocessing pipelines and models in as few lines of code as possible.
deep_autoviml will enable data scientists, ML engineers and data engineers to fast prototype tensorflow models and data pipelines for MLOps workflows using the latest TF 2.4+ and keras preprocessing layers. You can now upload your saved model to any Cloud provider and make predictions out of the box since all the data preprocessing layers are attached to the model itself!
In this webinar, we will discuss the problems that deep_AutoViML can solve, its architecture design and demo how to build powerful TF.Keras models on structured data, NLP and Image data domains.
https://www.aicamp.ai/event/eventdetails/W2021080918
Metaflow: The ML Infrastructure at Netflix
Metaflow was started at Netflix to answer a pressing business need: How to enable an organization of data scientists, who are not software engineers by training, build and deploy end-to-end machine learning workflows and applications independently. We wanted to provide the best possible user experience for data scientists, allowing them to focus on parts they like (modeling using their favorite off-the-shelf libraries) while providing robust built-in solutions for the foundational infrastructure: data, compute, orchestration, and versioning.
Today, the open-source Metaflow powers hundreds of business-critical ML projects at Netflix and other companies from bioinformatics to real estate.
In this talk, you will learn about:
- What to expect from a modern ML infrastructure stack.
- Using Metaflow to boost the productivity of your data science organization, based on lessons learned from Netflix.
- Deployment strategies for a full stack of ML infrastructure that plays nicely with your existing systems and policies.
https://www.aicamp.ai/event/eventdetails/W2021080510
Practical Crowdsourcing for ML at Scale
Daria Baidakova is the Director of Educational Programs at Toloka, an open crowdsourcing platform. She manages crowdsourcing courses and organizes tutorials and hackathons. Toloka provides labeled data via crowdsourcing which is a missing pillar of AI. It offers data annotation, content collection, and business decision making through crowdsourcing. Toloka manages quality through controls at the project level, performer selection, and result aggregation rather than direct management of individuals.
Building large scale transactional data lake using apache hudi
Data is a critical infrastructure for building machine learning systems. From ensuring accurate ETAs to predicting optimal traffic routes, providing safe, seamless transportation and delivery experiences on the Uber platform requires reliable, performant large-scale data storage and analysis. In 2016, Uber developed Apache Hudi, an incremental processing framework, to power business critical data pipelines at low latency and high efficiency, and helps distributed organizations build and manage petabyte-scale data lakes.
In this talk, I will describe what is APache Hudi and its architectural design, and then deep dive to improving data operations by providing features such as data versioning, time travel.
We will also go over how Hudi brings kappa architecture to big data systems and enables efficient incremental processing for near real time use cases.
Speaker: Satish Kotha (Uber)
Apache Hudi committer and Engineer at Uber. Previously, he worked on building real time distributed storage systems like Twitter MetricsDB and BlobStore.
website: https://www.aicamp.ai/event/eventdetails/W2021043010
Deep Reinforcement Learning and Its Applications
What is the most exciting AI news in recent years? AlphaGo!
What are key techniques for AlphaGo? Deep learning and reinforcement learning (RL)!
What are application areas for deep RL? A lot! In fact, besides games, deep RL has been making tremendous achievements in diverse areas like recommender systems and robotics.
In this talk, we will introduce deep reinforcement learning, present several applications, and discuss issues and potential solutions for successfully applying deep RL in real life scenarios.
https://www.aicamp.ai/event/eventdetails/W2021042818
Big Data and AI in Fighting Against COVID-19
This document discusses how big data and AI can help fight Covid-19. It describes supercomputers being used for scientific research on Covid-19. An open data lake has been created containing various Covid-19 datasets for analysis. Natural language processing and BERT are being used to answer scientific questions from the Covid-19 literature by generating summaries and highlighting relevant text passages. Challenges are being conducted on the Covid-19 Open Research Dataset to further advance research.
Highly-scalable Reinforcement Learning RLlib for Real-world Applications
This document discusses Ray and RLlib, a reinforcement learning library built on Ray. It provides three key points:
1. Ray is a framework for building distributed applications and services with shared memory abstraction. It allows ML workloads to scale beyond a single machine.
2. RLlib is a scalable reinforcement learning library that uses Ray. It supports a wide range of algorithms and execution models. This allows for easy implementation and comparison of RL techniques.
3. The Ray community is growing rapidly and provides resources like tutorials and Slack support to help users adopt Ray and RLlib for their distributed Python and reinforcement learning applications.
Build computer vision models to perform object detection and classification w...
event: https://learn.xnextcon.com/event/eventdetails/W20042918
video:
description: Computer Vision has received significant attention over the recent years, both within academia, and industry. As the state-of-the-art rapidly improves, the art-of-the-possible follows , offering innovative forms of computer vision applications for different scenarios.
In this talk, Ramine will cover the background and development of computer vision, and demonstrate how to use AWS to build robust, computer vision models to perform object detection and classification.
Key Takeaways:
Understand the history of Computer Vision
Learn how to use Amazon SageMaker to build and Deploy Computer Vision Models
How to orchestrate multiple models for implementing a real-world use case
Causal Inference in Data Science and Machine Learning
Event: https://learn.xnextcon.com/event/eventdetails/W20042010
Video: https://www.youtube.com/channel/UCj09XsAWj-RF9kY4UvBJh_A
Modern machine learning techniques are able to learn highly complex associations from data, which has led to amazing progress in computer vision, NLP, and other predictive tasks. However, there are limitations to inference from purely probabilistic or associational information. Without understanding causal relationships, ML models are unable to provide actionable recommendations, perform poorly in new, but related environments, and suffer from a lack of interpretability.
In this talk, I provide an introduction to the field of causal inference, discuss its importance in addressing some of the current limitations in machine learning, and provide some real-world examples from my experience as a data scientist at Brex.
Weekly #106: Deep Learning on Mobile
https://learn.xnextcon.com/event/eventdetails/W20040610
This talk explains how to practically bring the power of convolutional neural networks and deep learning to memory and power-constrained devices like smartphones. You will learn various strategies to circumvent obstacles and build mobile-friendly shallow CNN architectures that significantly reduce the memory footprint and therefore make them easier to store on a smartphone;
The talk also dives into how to use a family of model compression techniques to prune the network size for live image processing, enabling you to build a CNN version optimized for inference on mobile devices. Along the way, you will learn practical strategies to preprocess your data in a manner that makes the models more efficient in the real world.
Weekly #105: AutoViz and Auto_ViML Visualization and Machine Learning
https://learn.xnextcon.com/event/eventdetails/W20040310
I will describe what is available in terms of Open Source and Proprietary tools for automating Data Science tasks and introduce 2 new tools: one to visualize any sized data set with one click, another: to try multiple ML models and techniques with a single call. I will provide the Github Repos for both for free in the talk.
AISF19 - On Blending Machine Learning with Microeconomics
The document discusses how machine learning and economics can inform each other. It describes how markets can be viewed as decentralized algorithms that solve complex tasks. Recommendation systems are discussed as an example of machine learning success, but it is noted that recommending the same items to everyone is problematic. As an alternative, the document proposes creating markets with recommendation systems on both sides, like a market connecting diners and restaurants. The rest of the document discusses several specific examples at the intersection of machine learning and economics research, like bandits that compete and finding Nash equilibria with gradient-based algorithms.
AISF19 - Travel in the AI-First World
Travel is a force for good economically and socially. Expedia Group is building towards an "AI-First World" of travel where AI is used throughout the entire travel planning, booking, and experience process. They are developing massive multi-dimensional AI models and an AI platform to deliver highly personalized and targeted experiences. Overcoming challenges like data quality, scalability, and control will be key to realizing an AI-First vision where travel is optimized for each individual user.
AISF19 - Unleash Computer Vision at the Edge
This document discusses the key drivers enabling computer vision at the edge, including new machine learning approaches, optimized model architectures, hardware innovations, and improved software tools. It describes how machine learning has advanced computer vision by enabling end-to-end learning without predefined features. Edge-optimized models like GoogleNet and ShuffleNet are discussed. The proliferation of cameras, embedded processors, and AI accelerators is enabling computer vision everywhere. Open-source tools like OpenCV and frameworks like TensorFlow are supporting development, along with platforms to speed application creation.
AISF19 - Building Scalable, Kubernetes-Native ML/AI Pipelines with TFX, KubeF...
This document discusses modern machine learning pipelines and popular open source tools to build them. It defines key characteristics of ML pipelines like experiment tracking, hyperparameter optimization, distributed execution, and metadata/data versioning. Popular tools covered are KubeFlow for Kubernetes+TensorFlow, Airflow for data and feature engineering, MLflow for experiment tracking, and TensorFlow Extended (TFX) libraries. The document demonstrates these tools and argues that while the field is emerging, simplicity is important and one should only use necessary components of different tools.
Toronto meetup 20190917
This document discusses elastic distributed deep learning training at scale on-premises and in the cloud. It introduces the architecture of elastic distributed training, which combines high performance synchronization techniques like distributed data parallel with session scheduling and elastic scaling to provide flexibility. This allows training jobs to automatically scale up and down resources based on policies while maintaining high performance. It aims to make distributed training transparent to frameworks like TensorFlow and PyTorch.
More from Bill Liu (20)
Redefining MLOps with Model Deployment, Management and Observability in Produ...
Redefining MLOps with Model Deployment, Management and Observability in Produ...
Deep AutoViML For Tensorflow Models and MLOps Workflows
Deep AutoViML For Tensorflow Models and MLOps Workflows
Building large scale transactional data lake using apache hudi
Building large scale transactional data lake using apache hudi
Highly-scalable Reinforcement Learning RLlib for Real-world Applications
Highly-scalable Reinforcement Learning RLlib for Real-world Applications
Build computer vision models to perform object detection and classification w...
Build computer vision models to perform object detection and classification w...
Causal Inference in Data Science and Machine Learning
Causal Inference in Data Science and Machine Learning
Weekly #105: AutoViz and Auto_ViML Visualization and Machine Learning
Weekly #105: AutoViz and Auto_ViML Visualization and Machine Learning
AISF19 - On Blending Machine Learning with Microeconomics
AISF19 - On Blending Machine Learning with Microeconomics
AISF19 - Building Scalable, Kubernetes-Native ML/AI Pipelines with TFX, KubeF...
AISF19 - Building Scalable, Kubernetes-Native ML/AI Pipelines with TFX, KubeF...
Recently uploaded
Taking AI to the Next Level in Manufacturing.pdf
Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as:
1. How quickly AI is being implemented in manufacturing.
2. Which barriers stand in the way of AI adoption.
3. How data quality and governance form the backbone of AI.
4. Organizational processes and structures that may inhibit effective AI adoption.
6. Ideas and approaches to help build your organization's AI strategy.
Presentation of the OECD Artificial Intelligence Review of Germany
Consult the full report at https://www.oecd.org/digital/oecd-artificial-intelligence-review-of-germany-609808d6-en.htm
TrustArc Webinar - 2024 Global Privacy Survey
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
Artificial Intelligence for XMLDevelopment
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for life science domain, where you retriever information from biomedical knowledge graphs using LLMs to increase the accuracy and performance of generated answers
Skybuffer SAM4U tool for SAP license adoption
Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool.
SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Simplify your search for a reliable Python development partner! This list presents the top 10 trusted US providers offering comprehensive Python development services, ensuring your project's success from conception to completion.
20240609 QFM020 Irresponsible AI Reading List May 2024
Everything I found interesting about the irresponsible use of machine intelligence in May 2024
Digital Marketing Trends in 2024 | Guide for Staying Ahead
https://www.wask.co/ebooks/digital-marketing-trends-in-2024
Feeling lost in the digital marketing whirlwind of 2024? Technology is changing, consumer habits are evolving, and staying ahead of the curve feels like a never-ending pursuit. This e-book is your compass. Dive into actionable insights to handle the complexities of modern marketing. From hyper-personalization to the power of user-generated content, learn how to build long-term relationships with your audience and unlock the secrets to success in the ever-shifting digital landscape.
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on integration of Salesforce with Bonterra Impact Management.
Interested in deploying an integration with Salesforce for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean Lotus cyber threat actors represent a sophisticated, persistent, and politically motivated group that poses a significant risk to organizations and individuals in the Southeast Asian region. Their continuous evolution and adaptability underscore the need for robust cybersecurity measures and international cooperation to identify and mitigate the threats posed by such advanced persistent threat groups.
5th LF Energy Power Grid Model Meet-up Slides
5th Power Grid Model Meet-up
It is with great pleasure that we extend to you an invitation to the 5th Power Grid Model Meet-up, scheduled for 6th June 2024. This event will adopt a hybrid format, allowing participants to join us either through an online Mircosoft Teams session or in person at TU/e located at Den Dolech 2, Eindhoven, Netherlands. The meet-up will be hosted by Eindhoven University of Technology (TU/e), a research university specializing in engineering science & technology.
Power Grid Model
The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services.
Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability.
Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization.
What to expect
For the upcoming meetup we are organizing, we have an exciting lineup of activities planned:
-Insightful presentations covering two practical applications of the Power Grid Model.
-An update on the latest advancements in Power Grid -Model technology during the first and second quarters of 2024.
-An interactive brainstorming session to discuss and propose new feature requests.
-An opportunity to connect with fellow Power Grid Model enthusiasts and users.
Choosing The Best AWS Service For Your Website + API.pptx
Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API?
Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose?
Which one is cheapest? Which one is fastest? Which one will scale to meet our needs?
Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!
Webinar: Designing a schema for a Data Warehouse
Are you new to data warehouses (DWH)? Do you need to check whether your data warehouse follows the best practices for a good design? In both cases, this webinar is for you.
A data warehouse is a central relational database that contains all measurements about a business or an organisation. This data comes from a variety of heterogeneous data sources, which includes databases of any type that back the applications used by the company, data files exported by some applications, or APIs provided by internal or external services.
But designing a data warehouse correctly is a hard task, which requires gathering information about the business processes that need to be analysed in the first place. These processes must be translated into so-called star schemas, which means, denormalised databases where each table represents a dimension or facts.
We will discuss these topics:
- How to gather information about a business;
- Understanding dictionaries and how to identify business entities;
- Dimensions and facts;
- Setting a table granularity;
- Types of facts;
- Types of dimensions;
- Snowflakes and how to avoid them;
- Expanding existing dimensions and facts.
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
Imagine a world where machines not only perform tasks but also learn, adapt, and make decisions. This is the promise of Artificial Intelligence (AI), a technology that's not just enhancing our lives but revolutionizing entire industries.
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
Recently uploaded (20)
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Deep learning optimization at alibaba by zhenliang zhang from Alibaba
- 2. Optimization at Alibaba Zhenliang Zhang iDST at Alibaba
- 3. Alibaba Ecosystem - -- -- $ - - D F ) F 1B F B DB: EE B 1 DH DBH DE F ( F DF F ,B EF E -B DBIE D ,B F B E1B -
- 4. Optimization Problems Found Everywhere Data Center Optimization Resource Optimization Large-scale Nonconvex Optimization E-commerce Logistics Smart City Digital Entertainment Computing Infrastructure Computing Platform Machine Learning Platform Business
- 5. E-commerce • Connecting consumers with sellers/products – Efficiency is the key of any e-commerce platform – Alibaba: 0.5 Bn users, 1 Bn products, GMV 459Bn • Two key problems – Prediction: accurately predict users' needs – Matching: effectively match users' needs with products
- 6. Predicting User Needs Trans. Behavior LBS User Profiles Real-time data Contextual data Prediction
- 7. Effectively Matching User Needs Ease case: supplies ³ demands • A simple greedy matching works best • Match each user with the best prediction
- 8. Difficult case: supplies < demands • How to decide which users will get which items ? • Greedy algorithm does not work, leading to assignment problem Effectively Matching User Needs
- 9. Assignment Problem • Given a bipartite graph – m agents & n tasks – An award ci,j is given when agent i is assigned to perform task j • Optimal task assignment – Goal: maximize overall awards – Constraints: each agent/task can be assigned to a limited number of tasks/agents 0.8 0.1 ci,j
- 10. Applications in Alibaba • Online advertisement – Match users with different ads – Awards: the number of clicks – Constraints: budgeted number of impressions for individual ads • Significance of optimization – Improved revenue by ~50 m RMB per day Users Ads
- 11. Ask-All ( LM) Background • Allow customers to raise questions about products before making purchases • System distributes questions to users who could provide answers within an hour Goal • Maximize the number of questions answered Constraints • For better experience, each user will only be asked for a small number of questions QuestionsUsers
- 13. • Mobile message push • Online traffic allocation • Online distribution of coupons • Stock aware online recommendation • Online assignment of delivery requests • … Other Applications in Alibaba
- 14. Cloud Computing • The largest cloud computing company in China – 1Million users, ~100% YoY revenue growth • Need to improve resource management – Reliability: service-level agreement – Pricing & cost reduction
- 15. Computing Infrastructure at Alibaba • 6 data centers in mainland China • 8 data centers globally, 3 more are coming • Power management – US data centers consumed ~70 billion kilowatt-hours of electricity • Network planning – Minimize the latency
- 16. Example: Scheduling Problem • Online multi-resource allocation problem, e.g. CPU, Mem, Disk • Generally NP hard • Need to make online decision Compute Resources Scheduler Pending Jobs
- 17. Example: Traffic Balancing • Assign projects to clusters with minimal cost – Dependent projects need to communicate, – But traffic between clusters are expensive • Constraints – Network band restriction – Some project migration are infeasible
- 18. Logistics in Alibaba • Cainiao network – 80 major warehouse networks, 4000 Cainiao partners – 0.5 Bn parcels/day, next day delivery service in 150 cities • Key areas Supply chain Warehouse management Transportation management
- 19. Applications of Optimization in Logistics • 3D bin packing – Find smallest number of boxes to include all items from an order • Warehouse management – New retailer Hema • Vehicle route planning – Planning for order pickup – Planning for delivery service
- 20. & B( B D ,B EF E:D EFD F D A A &B C F F B D EB D F E . EE F A A BI D F F F D FIBD C 0B F I D B E F C H DB F : F B F B C Focus of Our Team
- 21. Optimization at Alibaba - -- -- $ - - D F ) F 1B F B DB: EE B 1 DH DBH DE F ( F DF F ,B EF E -B DBIE D ,B F B E1B - n .CF F B E ( (0 (0( n & E n F DF F EE n . BCF F B n , D E F n E EE B E