1. Microsoft IT Showcase
Using the Cortana Intelligence Suite for
sales and marketing predictive analytics
As Microsoft continues to evolve into a productivity and platform company, the amount of data to analyze grows
exponentially. This has created unique opportunities for data scientists to add significant value at Microsoft. Data
science teams are becoming increasingly common across the enterprise.
As the business value of data science grows at Microsoft, Microsoft IT needs more centralized and standardized ways
to manage predictive models. We used Microsoft Azure and the Cortana Intelligence Suite together to create a
platform that generates and manages predictive analytics in a new, different way.
We developed this platform approach to predictive analytics to improve discoverability and shared access to
predictive modeling assets at Microsoft. Some of the benefits that this approach has provided include:
 Reduced the computing time require for data feature refreshes by 93 percent.
 Reduced the resource hours required to develop predictive models by 20 to 80 percent.
 Published scores to a single catalog, improving discovery by the sales and marketing team.
 Reduced the resource hours required to refresh the model portfolio every month by 89 percent.
Business challenge
With the increasing use of data science and scientists, comes the risk of redundant and disconnected efforts across
such a large organization. Traditionally, data scientists worked independently to build their own predictive models.
This method wasn’t efficient; often, data scientists would create a new predictive model where one already existed for
a very similar business problem. And the lack of standardization sometimes led to inconsistent results between
predictive models. The following figure illustrates these business challenges.
Figure 1. The business challenges
When we started this process, we were spending more than 80 percent of the time required to create a predictive
model on identifying datasets and generating features. We had over 100 on-premises predictive models in our
portfolio, but we weren’t managing them effectively.

2. Page 2 | Using the Cortana Intelligence Suite for sales and marketing predictive analytics
microsoft.com/itshowcase July 2016
Three data scientists managed the portfolio, each owning their own set of over 30 models, for which they had to
maintain full end-to-end operations including data management, model retraining, model scoring, score publishing,
and reporting. Using data scientists to manage data operations tasks is costly and is not a great use of their unique
expertise. Many of the operational tasks are repetitive in nature, but without a standardized process or platform, they
were difficult to automate.
Another challenge was that the models were entirely dependent on the expertise and availability of the data scientist
who owned them. For example, if someone had questions about a model, only the data scientist who built the model
could effectively answer them.
Defining our solution objectives
Our goal was to provide best-in-class, enterprise-scalable, cloud-based capabilities and services that would simplify
development, deployment, and operationalization of predictive insights in support of modern marketing and sales
transformation.
We started by looking at how we could use Microsoft Azure and the Cortana Intelligence Suite to optimize the
management of our predictive model portfolio, including data management, model development, model
maintenance (ongoing refresh), model publishing, and model operationalization. The platform approach was
developed to allow data scientists to achieve more using process standardization and predictive insights
democratization. We wanted the platform to support not just the predictive models that our team builds, but also to
support models that other teams build.
To meet our goal, we defined our objectives for the predictive analytics platform. They included:
 Facilitating predictive modeling against large, sometimes unstructured, datasets in a variety of sources and
formats.
 Using Azure services for cloud storage, computing, and orchestration.
 Providing a platform to centralize management and deployment of security and privacy policies for data sources.
 Facilitating collaboration on new data science problems and democratizing the creation and consumption of
modeling insights.
 Enabling rapid model development to meet the accelerating pace of business data needs. – fostering
collaboration by featuring unlimited extensibility (R, Python, etc.)
 Making predictive models easy to deploy as web services that support request-response services and batch
execution services.
 Automating modeling operations using training/retraining pipelines.
 Governing model performance using a standard model evaluation framework.
 Curating a central catalog of predictive models featuring both owned experiments and third-party assets.
 Delivering predictive insights for key marketing and sales scenarios.
Creating a predictive analytics platform
The predictive analytics platform we designed is an enterprise-scale data mining and reporting platform built for
developing and publishing predictive models. Intended for shared use across the company, the platform also
facilitates platform-enabled collaboration scenarios. The platform was designed for three primary stages of model
development: feature preparation, model development, and score publishing. In addition, model operationalization is
a foundational element of the architecture.
To build the platform, we first created a common library of all existing features (variables and predictors) that had
proved to be predictive in our past modeling efforts. We standardized and rationalized across these features, where
necessary, to create a feature variable mart that supports the ongoing maintenance of existing models and serves as a
feature catalog for new modeling efforts.

3. Page 3 | Using the Cortana Intelligence Suite for sales and marketing predictive analytics
microsoft.com/itshowcase July 2016
Next, we converted our portfolio of on-premises models to Azure Machine Learning (AML) experiments and created a
catalog on SharePoint Online to assist marketers and sales professionals with asset discoverability. The platform
provides a single place to publish and inventory predictive model results (scores). Sales and marketing professionals
from across the globe have one catalog where they can discover, visualize, and consume relevant scores.
Finally, we introduced model operationalization that automates the data processes required to refresh features,
rescore (and periodically retrain) models, and republish scores for all the models on the platform.
The Cortana Intelligence Suite provides key capabilities
Using key capabilities in Microsoft Azure and the Cortana Intelligence Suite, the predictive platform was built to
address all of our defined objectives, as shown in the following figure.
Figure 2. Predictive analytics platform capabilities
Feature preparation using Azure Storage
Feature preparation involves all the activities necessary to assemble data for modeling—including data selection,
derivation, enrichment, cleansing, and formatting. Our Azure-based feature variable mart is a curated collection of
pre-calculated variables (or features) spanning different business perspectives (e.g., organizations, partners,
individuals) that have proven predictive value based on models already published on the platform. As new models are
published to the platform, each model’s variables are added to the feature variable mart. These features enable
ongoing operationalization of that model and serve as reference features for other data scientists that may find them
useful in future modeling scenarios.
The feature variable mart enables rapid data model prototyping
The variable mart and the centralized repository of existing experiments on AML support rapid model prototyping
and more agile predictive analytics. The ability to use existing features and model experiments as a template for
extending and developing new models is much more efficient than designing and creating experiments from scratch.
The collaborative features of Azure ML Studio enable data scientists to easily share their workspace with others, who
can then collaborate on the same experiments, or even clone an experiment, to use as a starting point for a
completely new model.

4. Page 4 | Using the Cortana Intelligence Suite for sales and marketing predictive analytics
microsoft.com/itshowcase July 2016
Model Development via Azure Machine Learning
Model development involves training and validating multiple models (experiments) to inform selection of the model
that best fits the business need. AML has powerful cloud-based predictive analytics capability that streamlines the
model development experience for all data scientist skill levels. By focusing on AML-based experiments, the predictive
platform offers modeling workspaces with best-in-class algorithms, an intuitive drag-and-drop interface, support for
R and Python custom code, cloud-based collaboration, and the ability to easily deploy models as ready-to-consume
web services. It supports both batch execution services and request-response services.
The AML experiment catalog also reduces duplication of effort that occurs when data scientists can’t see what models
their peers have already developed, which can be common in a company of our size.
Score Publishing via Azure API services
Publishing scores involves delivering model outputs (scores) to business. All models managed on the predictive
platform publish scores to a standardized Microsoft Azure/Microsoft SQL Server analytics publisher. In addition, a
parallel model onboarding service permits non-platform (or third-party) model scoring outputs to be cataloged in our
analytics publisher. As a single, centralized catalog of model scores, AP enables:
 Improved discoverability of predictive assets.
 Increased utilization of predicted scores in business workflows.
 A standard web services (API) layer to support direct integration of predictive assets within modern applications.
 Standard model evaluation frameworks for both usage and impact.
The single layer makes it easier for application developers as well; they only need to point their applications to one
source of predictive output. Without this platform, anybody that builds a model and wants to integrate these
capabilities would have to work directly with engineering teams that maintain the specific predictive models.
Model operationalization via Azure Data Factory
Model operationalization involves refreshing features, rescoring (and periodically retraining) models, and republishing
scores. The predictive platform automates these activities using Azure Data Factory (ADF). For each model managed
on the platform, ADF:
 Schedules refreshes of relevant features in the variable mart.
 Orchestrates the batch execution of the variable mart features against AML endpoints for batch scoring and
periodic retraining.
 Provides standard model evaluation frameworks for both usage and impact.
 Configures model outputs for alignment to our analytics publisher schema to ensure compatibility with web
services and reporting.
Operationalizing predictive analytics
From an operationalization standpoint, the refresh/retrain/republish process is now automated and requires minimal
human intervention. The automated refresh is agile and adaptable to model inclusions or exclusions. The process also
has automated pipeline and model performance monitoring built in, so that system admins and model publishers are
notified of model refresh and publish status. The statistical performance of all the models is monitored and then
archived during every refresh cycle.

5. Page 5 | Using the Cortana Intelligence Suite for sales and marketing predictive analytics
microsoft.com/itshowcase July 2016
Benefits of the new platform are illustrated in the following figure:
Figure 3. Benefits of the predictive analytics platform
 Because we migrated computing and processing workloads to the Azure Data Lake, we saw a 93 percent
reduction in computing time required to refresh features compared to on-premises servers.
 The ability to create new predictive models based on cataloged experiments has reduced the time required to
build model, and reduced the need for data scientists to create custom solutions. We are more agile and have
seen a 20 to 80 percent reduction in the hours required to develop predictive models by using features in the
variable mart. The range of savings varies based on how many features have to be built or can be based on
existing feature variables in the mart to create a new scenario. Predictive models built using only the features that
are available in the variable mart can see as much as an 80 percent time saving.
 Azure Data Factory automated and operationalized the model portfolio, which reduced the hours required to
refresh the model portfolio every month by 89 percent.
Predictive Analytics is becoming a vital solution in our sales and marketing efforts at Microsoft. The predictive
analytics platform has made it easier for other sales and marketing groups to onboard their experiments and take
advantage of its capabilities.
Marketing professional Philip Jordan says, “Western Europe used Azure Machine Learning, available in the Cortana
Intelligence Suite, to analyze and segment accounts based on their 10-year Microsoft sales transaction history. That
analysis generated some interesting findings: the top 13 percent of major accounts spent 18 times more than the bottom
2 percent of major accounts over a 10-year period—concluding that not all major accounts are equal. The basis of this
early analysis is now being applied to the lead scoring models that score marketing leads generated through the system.
This new insight will help ensure that sales professionals receive high-quality, prioritized leads based on customer action
and insights.”

6. Page 6 | Using the Cortana Intelligence Suite for sales and marketing predictive analytics
microsoft.com/itshowcase July 2016
Conclusion
We developed this predictive analytics platform to enable discoverability and access to predictive modeling assets
across Microsoft. As the platform continues to grow and the capabilities continue to evolve, it is our goal to empower
more data professionals to engage, customize, and deploy their own predictive models.
For more information
Predictive analytics improves the accuracy of forecasted sales revenue
Using predictive analytics to improve financial forecasting
Cortana Intelligence Suite
Microsoft Azure - Cortana Intelligence Suite
Microsoft IT
microsoft.com/itshowcase
© 2016 Microsoft Corporation. All rights reserved. Microsoft and Windows are either registered trademarks or trademarks of Microsoft
Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein may be the
trademarks of their respective owners. This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES,
EXPRESS OR IMPLIED, IN THIS SUMMARY.