Computational Advertising has been an important topical area in information retrieval and knowledge management. This tutorial will be focused on real-time advertising, aka Real-Time Bidding (RTB), the fundamental shift in the field of computational advertising. It is strongly related to CIKM areas such as user log analysis and modelling, information retrieval, text mining, knowledge extraction and management, behaviour targeting, recommender systems, personalization, and data management platform.
This tutorial aims to provide not only a comprehensive and systemic introduction to RTB and computational advertising in general, but also the emerging research challenges and research tools and datasets in order to facilitate the research. Compared to previous Computational Advertising tutorials in relevant top-tier conferences, this tutorial takes a fresh, neutral, and the latest look of the field and focuses on the fundamental changes brought by RTB.
We will begin by giving a brief overview of the history of online advertising and present the current eco-system in which RTB plays an increasingly important part. Based on our field study and the DSP optimisation contest organised by iPinyou, we analyse optimization problems both from the demand side (advertisers) and the supply side (publishers), as well as the auction mechanism design challenges for Ad exchanges. We discuss how IR, DM and ML techniques have been applied to these problems. In addition, we discuss why game theory is important in this area and how it could be extended beyond the auction mechanism design.
CIKM is an ideal venue for this tutorial because RTB is an area of multiple disciplines, including information retrieval, data mining, knowledge discovery and management, and game theory, most of which are traditionally the key themes of the conference. As an illustration of practical application in the real world, we shall cover algorithms in the iPinyou global DSP optimisation contest on a production platform; for the supply side, we also report experiments of inventory management, reserve price optimisation, etc. in production systems.
We expect the audience, after attending the tutorial, to understand the real-time online advertising mechanisms and the state of the art techniques, as well as to grasp the research challenges in this field. Our motivation is to help the audience acquire domain knowledge and obtain relevant datasets, and to promote research activities in RTB and computational advertising in general.
(Presented at the Deep Learning Re-Work SF Summit on 01/25/2018)
In this talk, we go through the traditional recommendation systems set-up, and show that deep learning approaches in that set-up don't bring a lot of extra value. We then focus on different ways to leverage these techniques, most of which relying on breaking away from that traditional set-up; through providing additional data to your recommendation algorithm, modeling different facets of user/item interactions, and most importantly re-framing the recommendation problem itself. In particular we show a few results obtained by casting the problem as a contextual sequence prediction task, and using it to model time (a very important dimension in most recommendation systems).
Reinforcement Learning 2. Multi-armed BanditsSeung Jae Lee
A summary of Chapter 2: Multi-armed Bandits of the book 'Reinforcement Learning: An Introduction' by Sutton and Barto. You can find the full book in Professor Sutton's website: http://incompleteideas.net/book/the-book-2nd.html
Learning to Rank for Recommender Systems - ACM RecSys 2013 tutorialAlexandros Karatzoglou
The slides from the Learning to Rank for Recommender Systems tutorial given at ACM RecSys 2013 in Hong Kong by Alexandros Karatzoglou, Linas Baltrunas and Yue Shi.
Recurrent Neural Networks for Recommendations and Personalization with Nick P...Databricks
In the last few years, RNNs have achieved significant success in modeling time series and sequence data, in particular within the speech, language, and text domains. Recently, these techniques have been begun to be applied to session-based recommendation tasks, with very promising results.
This talk explores the latest research advances in this domain, as well as practical applications. I will provide an overview of RNNs, covering common architectures and applications, before diving deeper into RNNs for session-based recommendations. I will pay particular attention to the challenges inherent in common personalization tasks and the specific adjustments to models and optimization techniques required for success.
(Presented at the Deep Learning Re-Work SF Summit on 01/25/2018)
In this talk, we go through the traditional recommendation systems set-up, and show that deep learning approaches in that set-up don't bring a lot of extra value. We then focus on different ways to leverage these techniques, most of which relying on breaking away from that traditional set-up; through providing additional data to your recommendation algorithm, modeling different facets of user/item interactions, and most importantly re-framing the recommendation problem itself. In particular we show a few results obtained by casting the problem as a contextual sequence prediction task, and using it to model time (a very important dimension in most recommendation systems).
Reinforcement Learning 2. Multi-armed BanditsSeung Jae Lee
A summary of Chapter 2: Multi-armed Bandits of the book 'Reinforcement Learning: An Introduction' by Sutton and Barto. You can find the full book in Professor Sutton's website: http://incompleteideas.net/book/the-book-2nd.html
Learning to Rank for Recommender Systems - ACM RecSys 2013 tutorialAlexandros Karatzoglou
The slides from the Learning to Rank for Recommender Systems tutorial given at ACM RecSys 2013 in Hong Kong by Alexandros Karatzoglou, Linas Baltrunas and Yue Shi.
Recurrent Neural Networks for Recommendations and Personalization with Nick P...Databricks
In the last few years, RNNs have achieved significant success in modeling time series and sequence data, in particular within the speech, language, and text domains. Recently, these techniques have been begun to be applied to session-based recommendation tasks, with very promising results.
This talk explores the latest research advances in this domain, as well as practical applications. I will provide an overview of RNNs, covering common architectures and applications, before diving deeper into RNNs for session-based recommendations. I will pay particular attention to the challenges inherent in common personalization tasks and the specific adjustments to models and optimization techniques required for success.
Presentation at the Netflix Expo session at RecSys 2020 virtual conference on 2020-09-24. It provides an overview of recommendation and personalization at Netflix and then highlights some of the things we’ve been working on as well as some important open research questions in the field of recommendations.
SaaS Marketing Plan: 5 Ways to Get your B2B App to Sell ItselfLincoln Murphy
Your B2B SaaS app will not sell itself unless it is designed to do so. This is a guide to getting your B2B SaaS to 'sell itself' was created by noted SaaS Marketing expert and Growth Hacker Lincoln Murphy of Sixteen Ventures.
This guide is full of actionable, unique, and thought-provoking ideas - Growth Hacks if you will - that are designed to allow your product to sell itself.
From attracting the right audience and driving user engagement, to making it easy to buy and easy to share through orchestrated virality, this guide covers all aspects of what's necessary to drive growth with SaaS companies of any stage and serving any market.
Whether you have a self-service sales model or one that requires outside sales reps to drive business, the tips and techniques contained in this guide and the source blog post will help you achieve your goals of efficiently scaling your sales process.
A Multi-Armed Bandit Framework For Recommendations at NetflixJaya Kawale
In this talk, we present a general multi-armed bandit framework for recommendations on the Netflix homepage. We present two example case studies using MABs at Netflix - a) Artwork Personalization to recommend personalized visuals for each of our members for the different titles and b) Billboard recommendation to recommend the right title to be watched on the Billboard.
Misha Bilenko, Principal Researcher, Microsoft at MLconf SEA - 5/01/15MLconf
Many Shades of Scale: Big Learning Beyond Big Data: In the machine learning research community, much of the attention devoted to ‘big data’ in recent years has been manifested as development of new algorithms and systems for distributed training on many examples. This focus has led to significant advances in the field, from basic but operational implementations on popular platforms to highly sophisticated prototypes in the literature. In the meantime, other aspects of scaling up learning have received relatively little attention, although they are often more pressing in practice. The talk will survey these less-studied facets of big learning: scaling to an extremely large number of features, to many components in predictive pipelines, and to multiple data scientists collaborating on shared experiments.
Talk with Yves Raimond at the GPU Tech Conference on Marth 28, 2018 in San Jose, CA.
Abstract:
In this talk, we will survey how Deep Learning methods can be applied to personalization and recommendations. We will cover why standard Deep Learning approaches don't perform better than typical collaborative filtering techniques. Then we will survey we will go over recently published research at the intersection of Deep Learning and recommender systems, looking at how they integrate new types of data, explore new models, or change the recommendation problem statement. We will also highlight some of the ways that neural networks are used at Netflix and how we can use GPUs to train recommender systems. Finally, we will highlight promising new directions in this space.
Netflix talk at ML Platform meetup Sep 2019Faisal Siddiqi
In this talk at the Netflix Machine Learning Platform Meetup on 12 Sep 2019, Fernando Amat and Elliot Chow from Netflix talk about the Bandit infrastructure for Personalized Recommendations
Recommendation Systems - Why How and Real Life ApplicationsLiron Zighelnic
These slides were created for a presentation at MIT - Massachusetts Institution of Technology
- Data Analytics Club
Recommendations become very popular in almost every field of our lives, from movies, to news to dating. Many systems try to give us personal recommendations.
In this presentation we will examine:
- Why recommendations are important?
- What are the main methods and algorithms being used?
- Real life applications & who use it? (the question should be: who doesn’t?)
About CurtainApp:
CurtainApp is an intelligent mobile app that learns your taste and gives you personal fashion recommendations, making shopping fun and efficient
Visit: www.curtainapp.com
Join us on Facebook: facebook.com/CurtainApp
Follow us on Twitter: twitter.com/thecurtainapp
#MIT #mobileapp #recommendation #fashion #recommendersystems #paradoxofchoice #Google #Netflix #OkCupid #Pandora #Curtain
Recommender Systems: Beyond the user-item matrixEugene Yan Ziyou
Recommendation systems. They're a pretty old topic that started way back in the 1990s.
A meetup on it sounds like it'll be boring... if we only talked about the standard user-item matrix collaborative filtering on big data systems.
Thankfully, for this meetup, we'll be sharing on how we can adopt some more recent techniques to recommend products, including social media graphs (and random walks), sequences (and NLP), and PyTorch. The sharing will cover everything starting from data acquisition and preparation, implementation of multiple techniques, and result comparisons. Some familiarity with Python and PyTorch would be useful; minimal math required.
Past, Present & Future of Recommender Systems: An Industry PerspectiveJustin Basilico
Slides from our talk at the RecSys 2016 conference in Boston, MA 2016-09-18 on our perspective for what are important areas for future work in recommender systems.
This presentation introduces CP Optimizer a model & run optimization engine for solving discrete combinatorial problems with a particular focus on scheduling problems.
Talk from QCon SF on 2018-11-05
For many years, the main goal of the Netflix personalized recommendation system has been to get the right titles in front each of our members at the right time. With a catalog spanning thousands of titles and a diverse member base spanning over a hundred million accounts, recommending the titles that are just right for each member is crucial. But the job of recommendation does not end there. Why should you care about any particular title we recommend? What can we say about a new and unfamiliar title that will pique your interest? How do we convince you that a title is worth watching? Answering these questions is critical in helping our members discover great content, especially for unfamiliar titles. One way to do this is to consider the artwork or imagery we use to visually portray each title. If the artwork representing a title captures something compelling to you, then it acts as a gateway into that title and gives you some visual “evidence” for why the title might be good for you. Selecting good artwork is important because it may be the first time a member becomes aware of a title (and sometimes the only time), so it must speak to them in a meaningful way. In this talk, we will present an approach for personalizing the artwork we show for each title on the Netflix homepage. We will look at how to frame this as a machine learning problem using contextual multi-armed bandits in a recommendation system setting. We will also describe the algorithmic and system challenges involved in getting this type of approach for artwork personalization to succeed at Netflix scale. Finally, we will discuss some of the future opportunities that we see to expand and improve upon this approach.
In display and mobile advertising, the most significant development in recent years is the Real-Time Bidding (RTB), which allows selling and buying in real-time one ad impression at a time. The ability of making impression level bid decision and targeting to an individual user in real-time has fundamentally changed the landscape of the digital media. The further demand for automation, integration and optimisation in RTB brings new research opportunities in the IR fields, including information matching with economic constraints, CTR prediction, user behaviour targeting and profiling, personalised advertising, and attribution and evaluation methodologies. In this tutorial, teamed up with presenters from both the industry and academia, we aim to bring the insightful knowledge from the real-world systems, and to provide an overview of the fundamental mechanism and algorithms with the focus on the IR context. We will also introduce to IR researchers a few datasets recently made available so that they can get hands-on quickly and enable the said research.
Presentation at the Netflix Expo session at RecSys 2020 virtual conference on 2020-09-24. It provides an overview of recommendation and personalization at Netflix and then highlights some of the things we’ve been working on as well as some important open research questions in the field of recommendations.
SaaS Marketing Plan: 5 Ways to Get your B2B App to Sell ItselfLincoln Murphy
Your B2B SaaS app will not sell itself unless it is designed to do so. This is a guide to getting your B2B SaaS to 'sell itself' was created by noted SaaS Marketing expert and Growth Hacker Lincoln Murphy of Sixteen Ventures.
This guide is full of actionable, unique, and thought-provoking ideas - Growth Hacks if you will - that are designed to allow your product to sell itself.
From attracting the right audience and driving user engagement, to making it easy to buy and easy to share through orchestrated virality, this guide covers all aspects of what's necessary to drive growth with SaaS companies of any stage and serving any market.
Whether you have a self-service sales model or one that requires outside sales reps to drive business, the tips and techniques contained in this guide and the source blog post will help you achieve your goals of efficiently scaling your sales process.
A Multi-Armed Bandit Framework For Recommendations at NetflixJaya Kawale
In this talk, we present a general multi-armed bandit framework for recommendations on the Netflix homepage. We present two example case studies using MABs at Netflix - a) Artwork Personalization to recommend personalized visuals for each of our members for the different titles and b) Billboard recommendation to recommend the right title to be watched on the Billboard.
Misha Bilenko, Principal Researcher, Microsoft at MLconf SEA - 5/01/15MLconf
Many Shades of Scale: Big Learning Beyond Big Data: In the machine learning research community, much of the attention devoted to ‘big data’ in recent years has been manifested as development of new algorithms and systems for distributed training on many examples. This focus has led to significant advances in the field, from basic but operational implementations on popular platforms to highly sophisticated prototypes in the literature. In the meantime, other aspects of scaling up learning have received relatively little attention, although they are often more pressing in practice. The talk will survey these less-studied facets of big learning: scaling to an extremely large number of features, to many components in predictive pipelines, and to multiple data scientists collaborating on shared experiments.
Talk with Yves Raimond at the GPU Tech Conference on Marth 28, 2018 in San Jose, CA.
Abstract:
In this talk, we will survey how Deep Learning methods can be applied to personalization and recommendations. We will cover why standard Deep Learning approaches don't perform better than typical collaborative filtering techniques. Then we will survey we will go over recently published research at the intersection of Deep Learning and recommender systems, looking at how they integrate new types of data, explore new models, or change the recommendation problem statement. We will also highlight some of the ways that neural networks are used at Netflix and how we can use GPUs to train recommender systems. Finally, we will highlight promising new directions in this space.
Netflix talk at ML Platform meetup Sep 2019Faisal Siddiqi
In this talk at the Netflix Machine Learning Platform Meetup on 12 Sep 2019, Fernando Amat and Elliot Chow from Netflix talk about the Bandit infrastructure for Personalized Recommendations
Recommendation Systems - Why How and Real Life ApplicationsLiron Zighelnic
These slides were created for a presentation at MIT - Massachusetts Institution of Technology
- Data Analytics Club
Recommendations become very popular in almost every field of our lives, from movies, to news to dating. Many systems try to give us personal recommendations.
In this presentation we will examine:
- Why recommendations are important?
- What are the main methods and algorithms being used?
- Real life applications & who use it? (the question should be: who doesn’t?)
About CurtainApp:
CurtainApp is an intelligent mobile app that learns your taste and gives you personal fashion recommendations, making shopping fun and efficient
Visit: www.curtainapp.com
Join us on Facebook: facebook.com/CurtainApp
Follow us on Twitter: twitter.com/thecurtainapp
#MIT #mobileapp #recommendation #fashion #recommendersystems #paradoxofchoice #Google #Netflix #OkCupid #Pandora #Curtain
Recommender Systems: Beyond the user-item matrixEugene Yan Ziyou
Recommendation systems. They're a pretty old topic that started way back in the 1990s.
A meetup on it sounds like it'll be boring... if we only talked about the standard user-item matrix collaborative filtering on big data systems.
Thankfully, for this meetup, we'll be sharing on how we can adopt some more recent techniques to recommend products, including social media graphs (and random walks), sequences (and NLP), and PyTorch. The sharing will cover everything starting from data acquisition and preparation, implementation of multiple techniques, and result comparisons. Some familiarity with Python and PyTorch would be useful; minimal math required.
Past, Present & Future of Recommender Systems: An Industry PerspectiveJustin Basilico
Slides from our talk at the RecSys 2016 conference in Boston, MA 2016-09-18 on our perspective for what are important areas for future work in recommender systems.
This presentation introduces CP Optimizer a model & run optimization engine for solving discrete combinatorial problems with a particular focus on scheduling problems.
Talk from QCon SF on 2018-11-05
For many years, the main goal of the Netflix personalized recommendation system has been to get the right titles in front each of our members at the right time. With a catalog spanning thousands of titles and a diverse member base spanning over a hundred million accounts, recommending the titles that are just right for each member is crucial. But the job of recommendation does not end there. Why should you care about any particular title we recommend? What can we say about a new and unfamiliar title that will pique your interest? How do we convince you that a title is worth watching? Answering these questions is critical in helping our members discover great content, especially for unfamiliar titles. One way to do this is to consider the artwork or imagery we use to visually portray each title. If the artwork representing a title captures something compelling to you, then it acts as a gateway into that title and gives you some visual “evidence” for why the title might be good for you. Selecting good artwork is important because it may be the first time a member becomes aware of a title (and sometimes the only time), so it must speak to them in a meaningful way. In this talk, we will present an approach for personalizing the artwork we show for each title on the Netflix homepage. We will look at how to frame this as a machine learning problem using contextual multi-armed bandits in a recommendation system setting. We will also describe the algorithmic and system challenges involved in getting this type of approach for artwork personalization to succeed at Netflix scale. Finally, we will discuss some of the future opportunities that we see to expand and improve upon this approach.
In display and mobile advertising, the most significant development in recent years is the Real-Time Bidding (RTB), which allows selling and buying in real-time one ad impression at a time. The ability of making impression level bid decision and targeting to an individual user in real-time has fundamentally changed the landscape of the digital media. The further demand for automation, integration and optimisation in RTB brings new research opportunities in the IR fields, including information matching with economic constraints, CTR prediction, user behaviour targeting and profiling, personalised advertising, and attribution and evaluation methodologies. In this tutorial, teamed up with presenters from both the industry and academia, we aim to bring the insightful knowledge from the real-world systems, and to provide an overview of the fundamental mechanism and algorithms with the focus on the IR context. We will also introduce to IR researchers a few datasets recently made available so that they can get hands-on quickly and enable the said research.
RTB tutorial Version 2.
In display and mobile advertising, the most significant development in recent years is the Real-Time Bidding (RTB), which allows selling and buying in real-time one ad impression at a time. Since then, RTB has fundamentally changed the landscape of the digital marketing by scaling the buying process across a large number of available inventories. The demand for automation, integration and optimisation in RTB brings new research opportunities in the IR/DM/ML fields. However, despite its rapid growth and huge potential, many aspects of RTB remain unknown to the research community for many reasons. In this tutorial, together with invited distinguished speakers from online advertising industry, we aim to bring the insightful knowledge from the real-world systems to bridge the gaps and provide an overview of the fundamental infrastructure, algorithms, and technical and research challenges of this new frontier of computational advertising. We will also introduce to researchers the datasets, tools, and platforms which are publicly available thus they can get hands-on quickly.
This tutorial aims to provide not only a comprehensive and systematic introduction to RTB and computational advertising in general, but also the emerging research challenges and research tools and datasets in order to facilitate the research. Compared to previous Computational Advertising tutorials in relevant top-tier conferences, this tutorial takes a fresh, neutral, and the latest look of the field and focuses on the fundamental changes brought by RTB. We expect the audience, after attending the tutorial, to understand the real-time online advertising mechanisms and the state of the art techniques, as well as to grasp the research challenges in this field. Our motivation is to help the audience acquire domain knowledge and obtain relevant datasets, and to promote research activities in RTB and computational advertising in general.
"Real Time Bidding on advertising exchanges is growing rapidly as advertisers are looking for cost effective ways to acquire valuable customers.
Running bidding infrastructure cost effectively presents a challenging problem and can be addressed with the latest advancements in EC2 Spot market.
We will discuss examples on how the bidder fleet can be scaled economically and can also be connected to low latency caches like DynamoDB. Low latency caches typically contain user specific information (such as audience segments) which can be use to form a bidding decision. Low latency and low jitter networking is another important consideration building an RTB business.
We will review partner solutions for connecting to advertising exchanges outside of AWS."
Study of Influencing Factors of Tender Evaluation: An Evidential Reasoning Ap...IJORCS
Selection of tender is a multi-criteria decision making process in which project performance is influenced by time, cost and quality. The appropriate tender selection can ensure a smooth delivery process and eliminate several complexities during construction. In this paper, the evidential reasoning (ER) approach which is capable of processing both quantitative and qualitative data is applied to find out the influencing factors as a means of solving the tender evaluation problem. The process of building a multiple criteria decision model of a hierarchical structure is presented, in which both quantitative and qualitative information is represented in a unified manner. By using a case study of Bangladesh the tender evaluation problem is then fully investigated using the ER approach. Finally we show the rank of influencing factors of best tender.
Real-time bidding (RTB) is revolutionizing the world of display advertising. sociomantic is making it easy for large eCommerce advertisers to get started in this powerful new performance channel.
Ranking Objects by Following Paths in Entity-Relationship Graphs (PhD Worksho...Minsuk Kahng
Minsuk Kahng, Sangkeun Lee, and Sang-goo Lee, "Ranking Objects by Following Paths in Entity-Relationship Graphs", Proceedings of the 4th ACM Workshop for Ph.D. Students in Information and Knowledge Management (PhD Workshop at CIKM 2011), 2011.
CUbRIK Research at CIKM 2012: Efficient Jaccard-based Diversity Analysis of L...CUbRIK Project
Presentation at CIKM 2013 of the CUbRIK research paper: "Efficient Jaccard-based Diversity Analysis of Large
Document Collections" authored by Fan Deng, Stefan Siersdorfer and Sergej Zerr of L3S Research Center, partner of the CUbRIK Consortium.
Semantic Tags Generation and Retrieval for Online Advertising - CIKM 2010Roku
One of the main problems in online advertising is to display ads which are relevant and appropriate \wrt what the user is looking for. Often search engines fail to reach this goal as they do not consider semantics attached to keywords. In this paper we propose a system that tackles the problem by two different angles: help (i) advertisers to create more efficient ads campaigns and (ii) ads providers to properly match ads content to keywords in search engines.
We exploit semantic relations stored in the DBpedia dataset and use an hybrid ranking system to rank keywords and to expand queries formulated by the user. Inputs of our ranking system are (i) the DBpedia dataset; (ii) external information sources such as classical search engine results and social tagging systems.
We compare our approach with other RDF similarity measures, proving the validity of our algorithm with an extensive evaluation involving real users.
Leveraging Joint Interactions for Credibility Analysis in News CommunitiesSubhabrata Mukherjee
Leveraging Joint Interactions for Credibility Analysis in News Communities,
Subhabrata Mukherjee and Gerhard Weikum,
Max Planck Institute for Informatics,
CIKM 2015
This deck introduces the ABX ad testing system, a paradigm shift in how companies can evaluate their paid and pre-test media. Their model conforms to the rules of faster, cheaper, deeper and better.
Elias Gagas, the Managing Director of TailWind, was one of the keynote speakers at "Media Research Day 2014" in Bucharest. On his presentation titled "The Programmatic (R)evolution: causes, effects & a framework to thrive in it", Elias delved into the fascinating world of Programmatic.
The Programmatic (R)evolution - Causes, Effects & a Framework to thrive in it!Elias Gagas
Presented in Bucharest at "Ziua Cercetarii Media 2014: Tendinte in online: audienta, modele de afaceri, continut" organized by BRAT [ http://www.brat.ro/evenimente/ziua-cercetarii-media-2014-tendinte-in-online-audienta-modele-de-afaceri-continut.html ]
Digital Advertising & Marketing Entrepreneurship: current building blocks and...Elias Gagas
Presented at Hellenic American college on April 28th ( http://haec.gr/en/aboutus/past-events/elias-gagas )
Additional links to material contained within:
- http://www.lumapartners.com/resource-center/lumascapes-2/ (these are the Lumascapes - some were included in the presentation)
- http://www.conversationprism.com/ (this is the social map)
- https://www.youtube.com/watch?v=-Glgi9RRuJs (the video showed)
What is Programmatic Advertising and how does it work? This talk by Webrepublic's Tech Lead Dorian Kind explains the basic principles and discusses the importance of creatives, media buying and tracking in a shifting advertising world.
The talk was given at the University of Zurich on November 4, 2016.
Embracing GenAI - A Strategic ImperativePeter Windle
Artificial Intelligence (AI) technologies such as Generative AI, Image Generators and Large Language Models have had a dramatic impact on teaching, learning and assessment over the past 18 months. The most immediate threat AI posed was to Academic Integrity with Higher Education Institutes (HEIs) focusing their efforts on combating the use of GenAI in assessment. Guidelines were developed for staff and students, policies put in place too. Innovative educators have forged paths in the use of Generative AI for teaching, learning and assessments leading to pockets of transformation springing up across HEIs, often with little or no top-down guidance, support or direction.
This Gasta posits a strategic approach to integrating AI into HEIs to prepare staff, students and the curriculum for an evolving world and workplace. We will highlight the advantages of working with these technologies beyond the realm of teaching, learning and assessment by considering prompt engineering skills, industry impact, curriculum changes, and the need for staff upskilling. In contrast, not engaging strategically with Generative AI poses risks, including falling behind peers, missed opportunities and failing to ensure our graduates remain employable. The rapid evolution of AI technologies necessitates a proactive and strategic approach if we are to remain relevant.
A Strategic Approach: GenAI in EducationPeter Windle
Artificial Intelligence (AI) technologies such as Generative AI, Image Generators and Large Language Models have had a dramatic impact on teaching, learning and assessment over the past 18 months. The most immediate threat AI posed was to Academic Integrity with Higher Education Institutes (HEIs) focusing their efforts on combating the use of GenAI in assessment. Guidelines were developed for staff and students, policies put in place too. Innovative educators have forged paths in the use of Generative AI for teaching, learning and assessments leading to pockets of transformation springing up across HEIs, often with little or no top-down guidance, support or direction.
This Gasta posits a strategic approach to integrating AI into HEIs to prepare staff, students and the curriculum for an evolving world and workplace. We will highlight the advantages of working with these technologies beyond the realm of teaching, learning and assessment by considering prompt engineering skills, industry impact, curriculum changes, and the need for staff upskilling. In contrast, not engaging strategically with Generative AI poses risks, including falling behind peers, missed opportunities and failing to ensure our graduates remain employable. The rapid evolution of AI technologies necessitates a proactive and strategic approach if we are to remain relevant.
How to Make a Field invisible in Odoo 17Celine George
It is possible to hide or invisible some fields in odoo. Commonly using “invisible” attribute in the field definition to invisible the fields. This slide will show how to make a field invisible in odoo 17.
The French Revolution, which began in 1789, was a period of radical social and political upheaval in France. It marked the decline of absolute monarchies, the rise of secular and democratic republics, and the eventual rise of Napoleon Bonaparte. This revolutionary period is crucial in understanding the transition from feudalism to modernity in Europe.
For more information, visit-www.vavaclasses.com
Read| The latest issue of The Challenger is here! We are thrilled to announce that our school paper has qualified for the NATIONAL SCHOOLS PRESS CONFERENCE (NSPC) 2024. Thank you for your unwavering support and trust. Dive into the stories that made us stand out!
Model Attribute Check Company Auto PropertyCeline George
In Odoo, the multi-company feature allows you to manage multiple companies within a single Odoo database instance. Each company can have its own configurations while still sharing common resources such as products, customers, and suppliers.
June 3, 2024 Anti-Semitism Letter Sent to MIT President Kornbluth and MIT Cor...Levi Shapiro
Letter from the Congress of the United States regarding Anti-Semitism sent June 3rd to MIT President Sally Kornbluth, MIT Corp Chair, Mark Gorenberg
Dear Dr. Kornbluth and Mr. Gorenberg,
The US House of Representatives is deeply concerned by ongoing and pervasive acts of antisemitic
harassment and intimidation at the Massachusetts Institute of Technology (MIT). Failing to act decisively to ensure a safe learning environment for all students would be a grave dereliction of your responsibilities as President of MIT and Chair of the MIT Corporation.
This Congress will not stand idly by and allow an environment hostile to Jewish students to persist. The House believes that your institution is in violation of Title VI of the Civil Rights Act, and the inability or
unwillingness to rectify this violation through action requires accountability.
Postsecondary education is a unique opportunity for students to learn and have their ideas and beliefs challenged. However, universities receiving hundreds of millions of federal funds annually have denied
students that opportunity and have been hijacked to become venues for the promotion of terrorism, antisemitic harassment and intimidation, unlawful encampments, and in some cases, assaults and riots.
The House of Representatives will not countenance the use of federal funds to indoctrinate students into hateful, antisemitic, anti-American supporters of terrorism. Investigations into campus antisemitism by the Committee on Education and the Workforce and the Committee on Ways and Means have been expanded into a Congress-wide probe across all relevant jurisdictions to address this national crisis. The undersigned Committees will conduct oversight into the use of federal funds at MIT and its learning environment under authorities granted to each Committee.
• The Committee on Education and the Workforce has been investigating your institution since December 7, 2023. The Committee has broad jurisdiction over postsecondary education, including its compliance with Title VI of the Civil Rights Act, campus safety concerns over disruptions to the learning environment, and the awarding of federal student aid under the Higher Education Act.
• The Committee on Oversight and Accountability is investigating the sources of funding and other support flowing to groups espousing pro-Hamas propaganda and engaged in antisemitic harassment and intimidation of students. The Committee on Oversight and Accountability is the principal oversight committee of the US House of Representatives and has broad authority to investigate “any matter” at “any time” under House Rule X.
• The Committee on Ways and Means has been investigating several universities since November 15, 2023, when the Committee held a hearing entitled From Ivory Towers to Dark Corners: Investigating the Nexus Between Antisemitism, Tax-Exempt Universities, and Terror Financing. The Committee followed the hearing with letters to those institutions on January 10, 202
Biological screening of herbal drugs: Introduction and Need for
Phyto-Pharmacological Screening, New Strategies for evaluating
Natural Products, In vitro evaluation techniques for Antioxidants, Antimicrobial and Anticancer drugs. In vivo evaluation techniques
for Anti-inflammatory, Antiulcer, Anticancer, Wound healing, Antidiabetic, Hepatoprotective, Cardio protective, Diuretics and
Antifertility, Toxicity studies as per OECD guidelines
Instructions for Submissions thorugh G- Classroom.pptxJheel Barad
This presentation provides a briefing on how to upload submissions and documents in Google Classroom. It was prepared as part of an orientation for new Sainik School in-service teacher trainees. As a training officer, my goal is to ensure that you are comfortable and proficient with this essential tool for managing assignments and fostering student engagement.
2024.06.01 Introducing a competency framework for languag learning materials ...Sandy Millin
http://sandymillin.wordpress.com/iateflwebinar2024
Published classroom materials form the basis of syllabuses, drive teacher professional development, and have a potentially huge influence on learners, teachers and education systems. All teachers also create their own materials, whether a few sentences on a blackboard, a highly-structured fully-realised online course, or anything in between. Despite this, the knowledge and skills needed to create effective language learning materials are rarely part of teacher training, and are mostly learnt by trial and error.
Knowledge and skills frameworks, generally called competency frameworks, for ELT teachers, trainers and managers have existed for a few years now. However, until I created one for my MA dissertation, there wasn’t one drawing together what we need to know and do to be able to effectively produce language learning materials.
This webinar will introduce you to my framework, highlighting the key competencies I identified from my research. It will also show how anybody involved in language teaching (any language, not just English!), teacher training, managing schools or developing language learning materials can benefit from using the framework.
CLASS 11 CBSE B.St Project AIDS TO TRADE - INSURANCE
CIKM 2013 Tutorial: Real-time Bidding: A New Frontier of Computational Advertising Research
1. Real-Time Bidding
A New Frontier of Computational Advertising Research
Jun Wang and Shuai Yuan, University College London
Xuehua Shen, iPinyou
Samuel Seljan, AppNexus
2. About us
• Dr Jun Wang and Shuai Yuan from University
College London
– Media future research group
– Computational advertising (big data analytics
and web economics)
• Dr Jun Wang is a Senior Lecturer (Associate
Professor) of Department of Computer Science,
UCL
• Shuai Yuan is Jun’s PhD student in CRS
(completing research status)
2
3. About us contd.
• Dr Xuehua Shen is CTO and co-founder of
iPinYou
– He received his PhD of Computer Science at
University of Illinois at Urbana-Champaign,
USA
iPinYou is the largest Demand Side Platform (DSP) and the leader of audience
targeting and real-time advertising in China. It makes intelligent decision for more
than 3 billion ads impressions each day. In the past two years, iPinYou is the pioneer
of programmatic buying of display media in China and organizes the annual RTB
Summit. It has more than 150 employees and is headquartered in Beijing and has
offices in Shanghai, Guangzhou, and Silicon Valley.
3
4. About us contd.
• Dr Samuel Seljan is a Quantitative Analyst at AppNexus
– Supply-side optimization
• to improve the allocation of impressions across RTB and non-RTB markets
• reserve price optimization
– He obtained a PhD in Political Science from the University of California, San
Diego
• AppNexus is one of the largest real-time advertising platforms (exchanges)
– Offers one of the most powerful, open and customizable advertising technology
platforms
– Serves Microsoft Advertising Exchange, Interactive Media (Deutsche Telekom),
and Collective Exchange
4
5. •
The background of RTB (25min)
Outline
– history, glossaries, fundamental challenges, players and their objectives
•
An empirical study of RTB auctions (15min)
– periodic features, bids’ distribution, daily pacing, frequency & recency control
•
Demand side optimisation (40min)
– bidding algorithms, conversion attribution
•
The iPinyou global bidding algorithm competition (30min, break 30min)
– results, prizes, how to participate
•
Supply side optimisation (40min)
– ad density, reserve prices, revenue channel selection, bid landscape
forecasting, pricing guaranteed delivery, data leakage
•
Financial methods in computational advertising (15min)
– game theory basic, ad options
•
Panel discussion (15min)
5
6. Part 1: The background of RTB
• Egyptians used papyrus to make sales
messages and wall posters (4000 BCE)
• In the 18th century, ads started to appear
in weekly newspapers in England
• Thomas J. Barratt has been called "the
father of modern advertising"
1806
1900
courtesy of Wikipedia
1890
1952
6
7. Ads can be not annoying
courtesy of lostateminor.com
7
8. Glossaries
• Real-Time Bidding is an important aspect of Programmatic buying, which is
getting more and more popular in Display (related) advertising. Another
major part of Online advertising is Sponsored search
• An Impression is an ad display opportunity which generates when a User
visits a webpage containing ad Placements
• The Publisher sends a bid request of this impression to an Ad network, or an
Ad exchange via his Supply side platform (SSP), then to Demand side
platforms (DSP) to reach Advertisers
• Usually, DSPs contact Data management platform (DMP) to check the
Segments of the current user, i.e., his intents or interests. Then a bid will be
computed for the Campaign
• The payment among these entities is usually in Cost per mille (CPM), but
sometimes could be Cost per click (CPC) or Cost per acquisition (CPA)
• If the advertiser wins the impression, his Creative will be displayed to the
user
8
9. The fundamental challenges
• To find the best match between a given user in a given context and a
suitable advertisement?
• To achieve the best campaign performance (e.g., ROI) within the budget
constraint?
• To generate the most revenue given the traffic and demand?
• To maintain a healthy environment so that users get less annoyed (both
quality and quantity)?
Computational advertising, AZ Border, 2008
Dynamics of bid optimization in online advertisement auctions, C Borges et al. 2007
Dynamic revenue management for online display advertising, G Roels and K Fridgeirsdottir, 2009
Advertising in a pervasive computing environment, A Ranganathan and RH Campbell, 2002
9
10. The simplified history of online (display) advertising
Real-time Bidding for Online Advertising: Measurement and Analysis, S Yuan et al., 2013
10
11. Direct sales
• Advertisers and publishers
talk to (4A) agencies
• Still popular in today’s
marketplace
27th Oct 1994, AT & T on HotWired.com
(78% CTR)
courtesy of Ad Age
11
12. Trading in ad networks
Why?
• After direct sales, some impressions
will remain unsold (remnants)
• Small publishers cannot find buyers
directly
Ad networks are first-level aggregators
of (long-tail) demand and supply.
courtesy of Admeld
12
13. Introducing the ad exchange
single ad network is easy
courtesy of www.liesdamnedlies.com
a few ad networks are manageable
hundreds of ad networks are nightmare
13
14. A video
courtesy of Internet advertising bureau, src: http://www.youtube.com/v/1C0n_9DOlwE
14
15. Introducing the ad exchange contd.
• Ad exchanges are
marketplaces
• Advertisers and
publishers have to rely
on tools to connect
• Real-Time Bidding
promotes user-oriented
bidding
Ad exchanges are second-level aggregators of demand and supply
15
17. A new picture in 2013
Demand side
courtesy of Rare Crowd
Aggregators
Supply side
17
18. Boundaries are getting blurry
Google is introducing display ads to search result pages
courtesy of Google
18
19. Introducing the Demand Side Platform (DSP)
• To connect to ad exchanges and SSPs
• To buy user-data from DMPs
• To provide campaign management
functions
• To bid by targeting rules and
optimisation algorithms
• To report and analyse the performance
A demand side platform (DSP), also called buy side optimizer and buy side
platform is a technology platform that provides centralized and aggregated
media buying from multiple sources including ad exchanges, ad networks and
sell side platforms, often leveraging real time bidding capabilities of these
sources.
IAB Wiki
courtesy of LUMAscape 2011
19
21. Introducing the Supply Side Platform (SSP)
• To upload advertisements and rich media
• To traffick ads according to differing business
rules
• To target ads to different users, or content
• To tune and optimise
• To report impressions, clicks, post-click &
post-impression activities, and interaction
metrics
A sell side platform (SSP), also called sell side optimizer,
inventory aggregator, and yield optimizer is a technology
platform that provides outsourced media selling and ad network
management services for publishers.
IAB Wiki
courtesy of LUMAscape 2011
21
23. Introducing the Data Management Platform (DMP)
• To collect users’ online behaviour data across
websites
(Mainly via 3rd party cookies)
• To predict users’ segments (intents/interests)
bases on online behaviour data
• To answer the query of users’ segments
• To provide audience profiling and expansion
services
A Data Management Platform (DMP) is a system that allows
the collection of audience intelligence by advertisers and
ad agencies, thereby allowing better ad targeting in
subsequent campaigns.
IAB Wiki
courtesy of LUMAscape 2011
23
26. Part 2: An empirical study of RTB
• To understand the bidding behaviours in RTB auctions
• To present some research challenges
• To help to get familiar with RTB in the real-world
• The data is from production DSP & SSP based in UK
– 52m impressions, 72k clicks, and 37k conversions from Feb to May 2013
• Started from convs/clicks and back-traced to imps
– 12m auctions from 50 placements from Dec 2012 to May 2013
• 16 websites of different categories
26
27. Periodic patterns
The numbers of imp (left) and click (right) both show
strong daily and weak weekly patterns,
corresponding to the normal human activity
27
28. Periodic patterns contd.
Daily periodic patterns for conv (left) and cvr (right) show that
people are less likely to convert during late night
28
29. Frequency distribution
The frequency against CVR plot from two different campaigns
Campaign 1 sets a frequency cap of 2-5 -> poor performance
Campaign 2 sets a frequency cap of 6-10 -> waste of budget
29
30. Recency distribution and conversion window
The recency factor affects the CVR (left)
Campaign 1 sets a long recency cap -> waste of budget
Campaign 2 sets a short recency cap -> poor performance
The wide conversion window (right)
challenges attribution models
30
32. Change of winner
Level of competition
(number of bidders)
The more bidders, the higher chance of winner change,
which makes it harder to detect a dynamic reserve price
32
33. Bids’ distribution
Accepted (p>0.05)
Rejected
AD test per auction
0.343
0.657
AD test per placement
0.000
1.000
CQ test per auction
0.068
0.932
The commonly adopted assumption of Uniform distribution or Log-normal distribution
were mostly rejected
• Anderson-Darling test for Normality
• Chi-squared test for Uniformity
Finding the best fit of bids’ distribution is important:
• Optimal reserve price
• Bid landscape forecasting
• etc.
And what’s the granularity? (placement, geographical location, time & weekday, etc.)
33
34. Budgeting and daily pacing
Real Time Bid Optimization with Smooth Budget Delivery in Online Advertising, KC Lee et al., 2013
34
36. A mixture of 1st and 2nd price auctions
• A high soft floor price can
make it 1st price auction
(In RTB, floor prices are not always
disclosed before auctions)
• In our dataset, 45% 1st price
auctions consumed 55%
budgets
• The complicated setting puts
advertisers in an unfavourable
position and could damage the
ad eco-system
36
37. Overview: references
•
•
•
•
•
•
The History of Advertising: How Consumers Won the War for Their Attention, HubSpot, 2013
blog.hubspot.com/the-history-of-advertising-war-for-consumer-attention-slideshare
How Cluttered Is the Advertising Landscape, Really? HubSpot, 2013
blog.hubspot.com/how-cluttered-is-advertising-landscape-timeline
Navigating Planet Ad Tech, MIT Technology Review, 2013
www.technologyreview.com/view/518551/the-evolution-of-ad-tech/
Internet Advertising: An Interplay among Advertisers, Online Publishers, Ad Exchanges and Web Users, S Yuan
et al., 2013
arxiv.org/abs/1206.1754
Ad exchanges: research issues, S Muthukrishnan, 2009
sites.google.com/site/algoresearch/start2.pdf
Behind the banner (A visualization of the adtech ecosystem), Adobe, 2013
cmsummit.com/behindthebanner/
37
38. Part 3: Demand side optimisation
• Bid optimisation
• Conversion attribution
38
39. Bid optimisation
• Input
– logs for auctions, impressions and
events
– targeting rules
– budgets and pacing preference
– internal/external user data
• The Decision Engine
– Gradient Boosting Regression
decision Tree, etc.
– fast & scalable
• Output
– to bid or not
– how much
39
40. Bid optimisation
• Baseline (constant or random, for exploration)
• Simple rule based (to bid high if the return is high)
– bid = base * pred_CTR / avg_CTR
– bid = conv_value * CVR * ROI
• Regression for estimation
– Generalised linear regression models (logistic, Bayesian probit, FTRL-Proximal,
etc.)
– Tree based models (random forest, gradient boosting regression tree, etc.)
– Neural networks and deep learning
40
41. Bid optimisation
• Looks good, but…
– metrics for evaluation?
– exploration vs.. exploitation (esp. for cold-start campaigns)
– risks (variance or confidence intervals from estimation)
– practical constraints (branding, overspending risks, inconsistent billing units, etc.)
• E-E problem
– Interactive collaborative filtering
– dimension deduction, correlation, etc.
• Risks
– Defining the Utility as the objective
– Portfolio theory
41
42. Metrics
• Top funnel metrics (to gain brand awareness)
– brand recall (awareness uplift)
– branded search
– direct website traffic
• Mid funnel metrics (to educate and engage the prospects)
– cost per new website visitor
– page view & form uplift
• Bottom funnel metrics (to generate value both online and offline)
– total conversion
– cost per conversion
– opportunity contribution (interested but not converted yet)
– revenue
courtesy of Adexchange
42
43. Transfer learning
• The Problem
– CTR is no good metrics but CVR is
too low
• Task
– To train on site visits
• Challenge
– Which site visits, and weight?
– Data availability
• Solution
– Similarity (contextual as a priori,
Bayesian)
Evaluating and Optimizing Online Advertising: Forget the Click, But There are Good Proxies, B Dalessandro, 2012
43
45. AppNexus
• AppNexus – Open and customizable advertising technology platform
•
Process over 50 Billion ad requests per day
•
Allow buyers to buy from over 90% of the web, including Facebook
•
Clients are advertisers, ad agencies, content providers, and ad networks
•
Major clients include:
• Microsoft
• Netflix
• eBay
• Zynga
• Interactive Media
• Orange - European telecom/media conglomerate
•
WPP - world’s second largest ad agency
45
46. Why advertise?
“Done well, advertising sends a whisper to your
impulses – a primal wind at the back of your neck,
suggesting where to go and what to do.”
-Don Draper
46
47. Fundamental question of advertising
• But, does advertising work?
•
Which ads work with what audience?
• Do the benefits of advertising outweigh it’s costs?
– Incremental revenue > Marginal cost?
47
48. Fundamental question of advertising
• The promise of digital advertising: precise measurement of users’ responses
to ads
• 1990s: Click tracking
• Compare CTRs across many, many dimensions:
• Campaign, image, time of day, region, location on page, gender, etc.
• Limitations?
48
49. Fundamental question of advertising
• The promise of digital advertising: precise measurement of users’ responses
to ads
• 1990s: Click tracking
• Compare CTRs across many, many dimensions:
• Campaign, image, time of day, region, location on page, etc
• Limitations?
• Late 2000s: Conversion tracking
• Conversion: when a user sees and ad and then takes an action, e.g. buys a pair or shoes
• Cost Per Action (CPA) payment: advertisers only pay when a conversion occurs
• Traders (agents of buyers) or sellers take on all the risk
• Does this answer the fundamental question of advertising?
49
50. Problems with CPA advertising
1.
Users often see many ads for the same brand on many sites
2.
Only includes online actions
3.
Causal inference – what does the association between seeing an ad and
converting mean?
4.
Ration of conversions per ads is often very small
–
Between 1-5 conversions per 100,000 impressions is common
–
Thus, takes many ads to learn “true” conversion rate.
•
For a one month campaign with a $5,000 budget, possible to learn conversion rate with
5% error on roughly 10 different web sites!
–
Difference between 1 and 5 conversions per 100,000 impression is difference
between a profitable and unprofitable campaign
50
51. Problems with CPA advertising
1.
Users often see many ads for the same brand on many sites
•
Which ad “is responsible” for the conversion?
•
Industry standard is “last touch” attribution
–
–
•
Previous graph is misleading!
Doesn’t show how many times a user saw ad
Connection to frequency optimization: creates a bias towards higher frequency ads
–
Last touch is the most recent ad, but we don’t know for sure that user even saw the
most recent ad – we just have a record of it
–
If effect of seeing ad is cumulative, this under weights importance of first view
51
52. Problems with CPA advertising
1.
–
2.
–
•
Users often see many ads for same brand on many sides
Which ad and site gets credit for the conversion?
No tracking of offline purchases
Technological and privacy challenges
3. The fundamental problem of causal inference
(Ad tech version)
Incremental revenue from ad j for user i =
Lifetime revenuei | i sees j– Lifetime revenuei | i does not see j
– But, each user either sees or does not see an ad so this cannot be calculated, even with
limitless data!
– Moreover, advertisers target users that are more likely to buy! (retargeting), thus lifetime
revenuei | i sees j is very likely to overestimate incremental revenue.
•
Most CPA optimization creates selection bias
53. Problems with CPA advertising
• Alternative framing: most CPA buying algorithms are predicting who is most
likely to buy and then focusing (targeting) buying on these users
• Thus, advertisers may see a lot of conversions associated with ads on some sites
on some users, but do not know how much revenue they would have had without
those ads
• Example: amazon targets ads at people who have recently searched for an item
on their website.
• These people are more likely to buy on Amazon than those who have not recently
searched
• But, they are also more likely to buy without seeing an ad
54. Problems with CPA advertising
•
Problem is more important for some brands than others
• For internet advertising, the potential scale of the problem can be considered by
thinking about:
Lifetime revenuei | i does not see digital ad
• For what types of campaigns is this likely to be a big problem? A small problem?
•
Principal agent problem: the people that need to understand this are the
brands themselves
• Agents buying for brands, do not have a short term incentive to solve problem
– they get paid per conversion!
• Selling more rigorous CPA optimization to brands is challenging
54
56. Problem 1: Many ads per conversion
• AppNexus Solution 1: custom conversion attribution
• We track conversions, but allow clients to divide conversion among ads
using their own “secret sauce”
• AppNexus Solution 2: Conversion Funnel
• Use events higher up the funnel to predict final sale
56
57. Problem 2: Offline tracking
• AppNexus Solution 1: allow for integrations with offline data providers and
the insertion of external data into optimization
• Other solutions?
• A privacy quid pro quo? Data for savings?
57
58. Problem 3: Fundament Problem of Causal Inf.
• AppNexus Solution: Random assignment
• Randomly assign users to group A (sees ad) or B (doesn’t see ad)
• Estimate incremental revenue from ad as:
conversion value * p(Conversion |A ) – p(Conversion | B)
– limitations: scalability, does not include off-line revenue or model social returns
• This is a problem for both group A and group B for larger brands
58
59. Problem 3: Fundament Problem of Causal Inf.
• Gold standard:
• Fowler et al. (2012): Facebook “I voted” and turnout
– RA to three groups:
» 1. Top panel
» 2. Second panel
» 3. No message
• Used federal data to compare turnout rates in groups & friends of those in each gorup
– .4% higher turnout of second gorup
• Combines solutions 2 and 3 (randomization addresses problems 2 and 3!)
59
60. Problem 3: Fundament Problem of Causal Inf.
• Remaining questions for RA: how to combine RA with other elements of a
CPA optimization algorithm…
– Over how many groups should one randomize?
» E.g. it could solve the frequency – conversion attribution problem, but that’s
a lot of groups
– What percent of impressions should be in test and control group?
60
61. Demand side optimisation: references
•
Optimal bidding on keyword auctions, B Kitts and B Leblanc, 2004
•
Stochastic gradient boosted distributed decision trees, J Ye et al., 2009
•
Web-scale bayesian click-through rate prediction for sponsored search advertising in Microsoft's Bing search
engine, T Graepel et al., 2010
•
Web-search ranking with initialized gradient boosted regression trees, A Mohan et al., 2011
•
A Gentle introduction to random forests, ensembles, and performance metrics in a commercial system, D
Benyamin, 2012
citizennet.com/blog/2012/11/10/random-forests-ensembles-and-performance-metrics/
•
A 61-million-person experiment in social influence and political mobilization, RM Bond et al., 2012
•
Deep metworks for predicting ad click through rates, G Corrado, 2012
•
Click modeling for display advertising, O Chapelle, 2012
•
Causal reasoning and learning systems, L Bottou and E Portugaly, 2012
•
Ad click prediction: a view from the trenches, HB McMahan et al., 2013
•
Deep learning, yesterday, today, and tomorrow, K Yu et al., 2013
•
Deep learning of representations: looking forward, Y Bengio, 2013
61
99. Season 3 vs. Season 1/2
Even bigger data set
Focus on Online Stage
Mobile Campaign
100.
101.
102. Any question sent to
dsp-competition@ipinyou.com
320076711
全球RTB算法大赛
iPinYou Global RTB Bidding
Algorithm Competition
103. Part 4: Supply side optimisation
• Typical revenue models
• Ad density optimal control
• Reserve price optimisation
• Ad channel selection
• Connecting the supply side markets
• Data leakage protection and pricing
103
104. Typical revenue models for the supply side
• Subscription access to content (FT.com)
• Pay Per View access to document (Downloading a paper outside the campus)
• CPM display advertising on site
• CPC advertising on site (Google AdSense)
• Sponsorship of site sections or content types (typically fixed fee for a period)
• Affiliate revenue (Compare shopping, CPA/CPC)
• Subscriber data access for marketing (VISA & MasterCard)
• User contributed data for marketing (Surveys)
Publishers will seek to use the best combination of these techniques
104
105. Ad density
The task:
•
To find the optimal advertising
density (number of ad placements)
for a given website
The challenges:
•
Users’ preference model
•
Expected CPM
•
Competition
The assumption:
• Using real-time bidding only
105
106. No ads
Some websites do not rely on
ads to compensate the
maintenance cost
• Government
• Education
• Most of .org
courtesy of www.gov.uk
106
107. All ads
• Created by Alex Tew in
2005
• Selling 100k 100-pixels
at $100 each
• Sold out in 4 months
• Almost 0% CTR
courtesy of www.milliondollarhomepage.com
107
110. Ad density: an optimal control problem
• Assumptions
– Ad density and impressions determine revenue
Ad density
CPM
Impressions (page views) Maintenance cost factor
– Ad density determines impressions
Content attraction Ad repellence
Natural growth
Question: what’s the optimal densities of multiple publishers under competition?
Management and valuation of advertisement-supported web sites, Dewan et al. 2005
110
111. Reference: ad density, layout & pricing
•
Management and valuation of advertisement-supported web sites, RM Dewan, 2003
•
Optimal pricing and advertising policies for web services, S Kumar et al., 2004
•
Is revamping your web site worthwhile? EY Huang, 2005
•
An economic analysis of ad-supported software, BJ Jiang, 2007
•
Dynamic pricing and advertising for web content providers, S Kumar and SP Sethi, 2009
•
Pricing display ads and contextual ads: Competition, acquisition, and investment, YM Li and JH Jhang-Li, 2009
•
Dynamic ad layout revenue optimization for display advertising, H Cheng et al., 2012
•
Automatic ad format selection via contextual bandits, L Tang et al., 2013
111
112. Reserve price optimisation
The task:
•
To find the optimal reserve
prices
The challenge:
•
Practical constraints v.s
common assumptions (bids’
distribution, bidding private
values, etc.)
The assumptions:
Even in the 2nd price auction,
the winner does not always
pay the 2nd highest bid (or
minimal + $0.01)
•
2nd price auction
•
With only hard floor price
112
113. Reserve price: flowchart
Suppose it is 2nd price auction
• Normal case: b2 > a
• Preferable case: b1 > a > b2
• Undesirable case: a > b1
113
114. Reserve price: example
• 2 bidders, Uniform[0, 1]
• Without a reserve price
• With the optimal auction theory
Reserve prices in internet advertising auctions: A field experiment, Ostrovsky and Schwarz, 2011
114
115. Reserve price: the optimal auction theory
• In the 2nd price auctions, advertisers bid their private values [𝑏1 , … , 𝑏 𝐾 ]
• Values are independently distributed and drawn from certain distributions
– Uniform
𝐹 𝒃 = 𝐹1 𝑏1 × ⋯ × 𝐹 𝐾 (𝑏 𝐾 )
– Log-normal
• The publisher also has a private value 𝑉𝑝
• The optimal reserve price is given by
𝛼−
1− 𝐹 𝒃
− 𝑉𝑝 = 0
′ 𝒃
𝐹
Questions:
• Are advertisers bidding their private values?
• Does Uniform/Log-normal fit well?
Optimal Reservation Prices in Auctions, Levin and Smith, 1996
115
116. Reserve price: field experiment results
Reserve prices in internet advertising auctions: A field experiment, Ostrovsky and Schwarz, 2011
116
117. Reserve price: field experiment results
Reserve prices in internet advertising auctions: A field experiment, Ostrovsky and Schwarz, 2011
117
118. Reserve price: detection and reaction
• A dynamic and repeated game between the winner (w) and the publisher (p)
• Extension form representation
– Information nodes:
• 𝐼1 : the winning bid 𝑏1 is higher
• 𝐼2 : the reserve price 𝛼 is higher
– Actions:
•
•
•
•
𝑎 𝑤1 : to increase
𝑎 𝑤2 : to increase
𝑎 𝑤3 : to decrease
𝑎 𝑤4 : to decrease
𝑏1 so that
𝑏1 so that
𝑏1 so that
𝑏1 so that
𝑏1 ≥
𝑏1 <
𝑏1 ≥
𝑏1 <
𝛼
𝛼
𝛼
𝛼
• 𝑎 𝑝1 : to increase 𝛼 so that 𝛼 ≥ 𝑏1
• 𝑎 𝑝2 : to increase 𝛼 so that 𝛼 < 𝑏1
• 𝑎 𝑝3 : to decrease 𝛼 so that 𝛼 ≥ 𝑏1
• 𝑎 𝑝4 : to decrease 𝛼 so that 𝛼 < 𝑏1
118
119. Reserve price: detection and reaction
payoff of the winner and the publisher
The dynamic and static game tree for the auction with reserve price
119
120. Reserve price: detection and reaction
• Consider playing the game repeatedly
– If the reserve price was higher, should the publisher lower it?
– The optimal auction theory
• Advertisers want to learn the publisher’s private value distribution, too
– What is the absolute minimal price that can be accepted?
• Questions
– What are the best response functions for both players?
– What are the dominant strategies and equilibrium?
120
121. Reserve price: references
•
On optimal reservation prices in auctions, Engelbrecht-Wiggans, 1987
•
Optimal reservation prices in auctions, Levin and Smith, 1996
•
Auction theory: a guide to the literature, Klemperer, 1999
•
Reserve prices in internet advertising auctions: a field experiment, Ostrovsky and Schwarz, 2009
•
Auction theory 2nd edition, Krishna, 2009
•
Optimal reserve price for the generalized second-price auction in sponsored search advertising, Xiao et al.,
2009
•
Optimal auction design and equilibrium selection in sponsored search auctions, Edelman and Schwarz, 2010
•
Optimal auction design in two-sided markets, R Gomes, 2011
121
122. Ad channel selection
• The task:
– There are multiple ad
channels giving
different payoffs over
time. Which one to
use?
• The challenge:
– Too many possible
candidates
– The payoffs change
over time
Sequential Selection of Correlated Ads by POMDPs, Yuan and Wang, 2012
122
123. Ad channel selection contd.
• A sequential selection problem
• Value iteration exact solution (high computational complexity)
• Multi-armed bandit approximation
Sequential Selection of Correlated Ads by POMDPs, Yuan and Wang, 2012
123
124. Sequential selection: references
•
Dynamic programming, RE Bellman, 1957
•
Multi-armed bandits and the Gittins index, P Whittle, 1980
•
A survey of POMDP applications, AR Cassandra, 1998
•
A survey of POMDP solution techniques, KP Murphy, 2000
•
Finite-time analysis of the multi-armed bandit problem, P Auer, P. et al.,2002
•
Multi-armed bandit algorithms and empirical evaluation, J Vermorel and M Mohri, 2005
124
125. Impression allocation between GD and NGD
• Sometimes, ad channels could have different attributes:
– GD: Guaranteed Delivery (contracts)
– NGD: Non-Guaranteed Delivery (auctions)
• Given a time window, the publisher decides
– To accept or reject a contract proposal
– To allocate impressions among multiple contracts and auctions
(NGD can be modelled as an already accepted contract with ∞ required
impressions and 0 under-delivery penalty)
Dynamic Revenue Management for Online Display Advertising, Roels and Fridgeirsdottir, 2008
125
126. Connecting GD and NGD
Four (4) major types of inventories in Programmatic selling
Now they are priced differently and separately
Separated markets implies inefficiency and arbitrage
courtesy of Internet advertising bureau
126
127. Automated (programmatic) guaranteed delivery
• AOL Upfront
– Will take effect January 1, 2014
– Two brands and five agencies have committed, around $10m for each agency
(undisclosed)
– AOL’s ad placements, e.g. The Huffington Post, TechCrunch and StyleList
– A private marketplace in the beginning
• Question
– Selling mechanisms (auction, queue, etc.)
– Inventory allocation and reserve prices
courtesy of AOL
127
128. Optimal pricing of a guaranteed delivery contract
• Suppose the publisher wants to sell 𝑆 impressions from time step 𝑇 + 1
– In advance: guaranteed delivery contracts
– On spot: RTB auctions (non-guaranteed, 2nd price auction)
• Consider the total demand as 𝑄
• The private value distribution of an advertiser is 𝐹 ⋅
• The utility of an advertiser is 𝑈 𝑎 ⋅ = 𝐹 ⋅ + 𝑔 𝑡
– Willing to pay higher if could buy in advance
What is the optimal price
at 0 < 𝑡 < 𝑇 + 1 ?
• The utility of the publisher is 𝑈 𝑝 ⋅ = 𝑅 ⋅ + ℎ(𝑡)
– Willing to charge lower if could sell in advance
128
129. Guaranteed delivery: references
•
Optimal dynamic auctions for display advertising, YJ Chen, 2009
•
Pricing guaranteed contracts in online display advertising, V Bharadwaj, 2010
•
Risk-aware revenue maximization in display advertising, A Radovanovic and WD Heavlin, 2012
•
Optimal allocation for display advertising, H Rui et al., 2012
•
A unified optimization framework for auction and guaranteed delivery in online advertising, K Salomatin,
2012
•
Maximally representative allocations for guaranteed delivery advertising campaigns, RP McAfee, 2013
129
130. Bid landscape forecasting
The tool usually exists as a service provided to advertisers
courtesy of Google AdWords
130
131. Bid landscape forecasting
• The task:
Given a campaign (a set of
targeting rules), what is the
bid-impression distribution
for a given venue (domain,
placement, etc.)?
• The challenge:
Forecasting for new &
changed campaigns
Forecasting the win rate for
unseen bids
Bid landscape forecasting in online ad exchange marketplace, Y Cui et al,. 2011
131
132. Bid landscape forecasting
A Generative Model based ad Impression Forecasting method
Ad impression forecasting for sponsored search, A Nath et al., 2013
132
133. Bid landscape forecasting: references
•
Bid landscape forecasting in online ad exchange marketplace, Y Cui et al., 2011
•
Handling forecast errors while bidding for display advertising, KJ Lang et al., 2012
•
Ad impression forecasting for sponsored search, A Nath et al., 2013
•
Forecasting user visits for online display advertising, S Cetintas et al., 2013
•
Predicting advertiser bidding behaviors in sponsored search by rationality modeling, H Xu et al., 2013
•
Optimizing volume and frequency forecasts for an online video advertiser, J Talbot et al., 2013
133
134. Data leakage protection
• Every player in the ad eco-system realises the value of audience data
– who owns it?
– what’s its value?
• The data leakage problem
– buyers collect user data from premium website
– then retarget these users on cheap inventories
• The task
– To learn who is collecting user data on the webpage (piggybacks)
– To stop the unauthorized collection and ask for payment
• The challenge
– The optimal price: limiting the user data access will hurt the CPM
134
135. Part 5: Financial methods in Computational Advertising
• Game theory basics
• Ad options
135
136. Examples
Auction for sharing compute resources
Financial methods and game theory have a long history in CS research
A futures market in computer time, IE Sutherland, 1968
136
137. Examples contd.
Auction and futures in cloud computing
• Amazon EC2 is a web service that provides resizable compute capacity in the cloud
• In late 2009, Amazon announce its spot instances pricing system
Deconstructing Amazon EC2 Spot Instance Pricing, OA Ben-Yehuda, 2011
137
138. Examples contd.
In the age of the Internet, fixed prices are a thing of the past
Professor Oren Etzioni
courtesy of Wall Street Journal
138
139. Game theory basics
• A game is “a competitive activity … in which players contend with each other
according to a set of rules”
• A strategic game (with ordinal preferences) consists of
– a set of players
– for each player, a set of actions
– for each player, preferences over the set of action profiles
• The best response
• The Nash equilibrium of static games
An introduction to game theory, MJ Osborne, 2003
139
140. Example: the prisoner's dilemma
• A typical example of noncooperative game
• The strictly dominant strategy
for both players is to confess,
which also forms the Nash
equilibrium of the game
• When played iteratively, the
winning deterministic strategy is
tit-for-tat
(first cooperate, then subsequently
replicate an opponent's previous
action)
The mysterious benedict society and the prisoner’s dilemma, TL Stewart, 2009
courtesy of Encyclopaedia Britannica, 2006
140
141. Example: The game of chicken
• While each player prefers not to
yield to the other, the worst possible
outcome occurs when both players
do not yield.
– The nuclear crisis
Dare
Chicken
– The promise on advertising effects
0, 0
7, 2
– The bidding on similar audiences
Chicken 2, 7
6, 6
Dare
courtesy of Rebel Without a Cause
simultaneously
141
142. Example: The Cournot competition
• Assumptions
– There is more than one firm and all firms produce a
homogeneous product, i.e. there is no product
differentiation;
– Firms do not cooperate, i.e. there is no collusion;
– Firms have market power, i.e. each firm's output decision
affects the good's price;
– The number of firms is fixed;
– Firms compete in quantities, and choose quantities
simultaneously;
– The firms are economically rational and act strategically,
usually seeking to maximize profit given their competitors'
decisions.
courtesy of palimpsestes.fr
142
143. Example: The Cournot competition
•
𝑝1 , 𝑝2 : prices
•
𝑞1 , 𝑞2 : quantities
• Firm 1’s profit:
Π1 = 𝑞1 𝑃 𝑞1 + 𝑞2 − 𝑐
• Firm 1’s best response function:
𝜕Π1
𝑅(𝑞2 ) =
𝜕𝑞2
• To obtain the equilibrium:
𝑅 𝑞2 = 𝑅 𝑞1
(the intersection)
courtesy of Wikipedia
143
144. Example: The Bertrand competition
• Assumptions
– Firms compete by setting prices simultaneously and
consumers want to buy everything from a firm with a
lower price
– If two firms charge the same price, consumers
demand is split evenly between them
courtesy of Wikipedia
144
145. Example: The Bertrand competition
• If both firms set equal prices
above marginal cost, firms
would get half the market at a
higher than MC price
• By lowering prices just slightly,
both firms are
pricing at
marginal cost
a firm could gain the whole
market
• Both firms are tempted to
lower prices as much as they
can
• It would be irrational to price
below marginal cost, because
courtesy of Wikipedia
the firm would make a loss
145
146. Application of the duopoly competition
• If capacity and output can be easily changed -> Bertrand model
• if output and capacity are difficult to adjust -> Cournot model
• Analogy
– Two publishers serving similar content
– Premium and long-tail publishers seeing similar users
– etc.
146
147. References: financial methods and game theory basics
•
A policy framework for trading configurable goods and services in open electronic markets, S Lamparter, 2006
•
Planning and pricing of service mashups, B Blau et al., 2008
•
Web service derivatives, T Meinl and B Blau, 2009
•
How to coordinate value generation in service networks, B Blau et al., 2009
•
Enabling cloud service reservation with derivatives and yield management, T Meinl et al., 2010
•
Web services advanced reservation contracts, C Weinhardt et al., 2011
•
Finite automata play the repeated prisioners dilemma, A Rubinstein, 1986
•
A course in game theory, MJ Osborne and A Rubinstein, 1994
•
An introduction to game theory, MJ Osborne, 2004
•
Sponsored search auctions: an overview of research with emphasis on game theoretic aspects, P Mailléet al.,
2012
•
Repeated keyword auctions played by finite automata, W Ding et al, 2013
147
148. Ad options
• The task:
– To sell impressions in advance
(a natural extension to the programmatic
guarantee)
– Both party can choose to exercise or not
• The (pricing) challenge:
– Impression prices are volatile
– Non-storability: impressions cannot be
bought and kept
– Not just about the price movements: the
uncertainty of traffic volume, CTR and etc.
courtesy of Webscope from Yahoo! Labs
148
149. Ad options contd.
Advertisers
secure impressions delivery
reduce uncertainty in auctions
cap cost
Publishers
sell the inventory in advance
have a more stable and predictable
revenue over a long-term period
increase advertisers’ loyalty
Benefits
149
150. Ad options contd.
t=T
Pays £ upfront option
5
price to obtain the
option.
t=0
Sells a list of ad keywords via a multikeyword multi-click option
multi-keyword multi-click option (3 month
term)
upfront
fee
(m = 100)
Submits a request of
guaranteed ad delivery
for the keywords ‘MSc
Web Science’, ‘MSc Big
Data Analytics’ and ‘Data
Mining’ for the future 3
month term [0, T], where
T = 0.25.
fixed CPCs
‘MSc Web Science’
£
1.80
‘MSc Big Data
Analytics’
£
6.25
‘Data Mining’
£
5
keywords list
£
8.67
Timeline
online advertiser
search engine
Multi-Keyword Multi-Click Advertisement Option Contract for Sponsored Search, B Chen et al., 2013
150
151. Exercising the option
Pays £
1.80 to the search
engine for each click until
the requested 100 clicks
are fully clicked by
Internet users.
Exercises 100 clicks of
‘MSc Web Science’ via
option.
t=T
t = t1c
Reserves an ad slot of the keyword ‘MSc
Web Science’ for the advertiser for 100
clicks until all the 100 clicks are fully
clicked by Internet users..
t = t1
t=0
Timeline
online advertiser
search engine
Multi-Keyword Multi-Click Advertisement Option Contract for Sponsored Search, B Chen et al., 2013
151
152. Not exercising the option
Pays the GSP-based CPC
for each click if winning
the bid.
If the advertiser thinks
the fixed CPC £
8.67 of the
keyword ‘Data Mining’ is
expensive, he/she can
attend keyword auctions
to bid for the keyword as
other bidders, say £
8.
t=T
Selects the winning bidder for the
keyword ‘Data Mining’ according to the
GSP-based auction model.
t =…
t=0
Timeline
online advertiser
search engine
Multi-Keyword Multi-Click Advertisement Option Contract for Sponsored Search, B Chen et al., 2013
152
153. Ad options pricing
• Building blocks
– No-arbitrage [F Black and M Scholes1973; H Varian1994]
– Stochastic underlying keyword CPC [P Samuelson1965]
– Terminal value formulation
• Formula
– n=1, Black-Scholes-Merton European call
– n=2, Peter Zhang dual strike European call
– n>=3, Monte Carlo method
Multi-Keyword Multi-Click Advertisement Option Contract for Sponsored Search, B Chen et al., 2013
153
154. Ad options: references
•
Option pricing: a simplified approach, J Cox et al., 1979
•
Online ad slotting with cancellations, F Constantin, 2008
•
A truthful mechanism for offline ad slot scheduling, J Feldman et al., 2008
•
Selling ad campaigns: online algorithms with cancellations, M Babaioff et al., 2009
•
Options, futures and other derivative securities (7th edition), J Hull, 2009
•
Online advertisement service pricing and an option contract, Y Moon and CY Kwon, 2010
•
Selling futures online advertising slots via option contracts, J Wang and B Chen, 2012
•
Multi-keyword multi-click advertisement option contract for sponsored search, B Chen et al., 2013
154
155. Additional references
•
Internet advertising and the generalized second price auction: selling billions of dollars worth of keywords, B
Edelman, 2005
•
Price cycles in online advertising auctions, X Zhang and J Feng, 2005
•
Budget optimization in search-based advertising auctions, J Feldman et al., 2007
•
The economics of the online advertising industry, DS Evans, 2008
•
Expressive banner ad auctions and model-based online optimization for clearing, C Boutilier et al., 2008
•
Computational advertising, AZ Broder, 2008
•
Algorithmic methods for sponsored search advertising, J Feldman and S Muthukrishnan, 2008
•
Internet ad auctions: Insights and directions, S Muthukrishnan, 2008
•
The online advertising industry: economics, evolution, and privacy, , DS Evans, 2009
•
Ad exchanges: Research issues, S Muthukrishnan, 2009
•
Adaptive bidding for display advertising, A Ghosh et al., 2009
•
The arrival of real-time bidding, Google, 2011
•
Algorithms and strategies for web advertising, P Papadimitriou, 2011
•
OpenRTB API specification, IAB, 2012
155
156. Additional references
•
Targeted, not tracked: client-side profiles and privacy-friendly behavioral advertising, M Bilenko and M
Richardson, 2012
•
Computational advertising in social networks, A Bhasin, 2012
•
Size, labels, and privacy in targeted display advertising, C Perlich, 2012
•
Estimating conversion rate in display advertising from past erformance data, K Lee et al,. 2012
•
Handling forecast errors while bidding for display advertising, KJ Lang et al., 2012
•
Marketing campaign evaluation in targeted display advertising, J Barajas et al., 2012
•
Ad exchange-proposal for a new trading agent competition game, M Schain and Y Mansour, 2012
•
Auctions for online display advertising exchanges: approximations and design, S Balseiro et al., 2012
•
Real-time bidding for online advertising: measurement and analysis, S Yuan et al., 2013
•
Impression fraud in on-line advertising via pay-per-view networks, K Springborn and P Barford, 2013
•
An overview of computational challenges in online advertising, RE Chatwin, 2013
•
Competition and yield optimization in ad exchanges, SR Balseiro, 2013
•
Internet advertising revenue report, IAB and PwC
156