Recommended
Recommended
More Related Content
Similar to Price optimization for high-mix, low-volume environments | Using R and Tableau at Worthington Industries
Similar to Price optimization for high-mix, low-volume environments | Using R and Tableau at Worthington Industries (20)
Recently uploaded
Recently uploaded (20)
Price optimization for high-mix, low-volume environments | Using R and Tableau at Worthington Industries
- 2. Welcome
- 3. Using R and Tableau at Worthington Industries: Price optimization for high-mix, low-volume environments Steve Bartos Manager, Predictive Analytics Worthington Industries Wil Davis Analyst, Predictive Analytics Worthington Industries D.R.E.A.M.
- 4. Agenda Why pricing? A Brief History of Analytics at Worthington Industries A Machine Learning Approach Misfire. Power to the People! Let them choose with Tableau and R The Complete Stack. Rserve, CI/CD and Version Control
- 5. A Brief History of Analytics at Worthington Industries D.R.E.A.M.
- 6. Worthington Industries Founded in 1955 and headquartered in Columbus, OH Publicly traded on the NYSE under the ticker WOR 10,000 employees & 5,000 customers; 80 facilities in 11 countries Employee, customer, supplier and investor- centered philosophy Leader in safety management and injury prevention – company wide goal of zero accidents and injuries Named one of “America’s Safest Companies” by Occupational Hazards magazine, 2008 Named to Fortune’s “100 Best Companies to Work For” list five times
- 7. 7 In the beginning: one analyst, one idea
- 8. Opportunities for another crown jewel of analytics?
- 9. Key Questions Addressed by Analytics 9 D.R.E.A.M. Source: Davenport et al. Analytics at Work: Smarter Decisions, Better Results Past Present Future What happened? Reporting What is happening? Alerts What will happen? Extrapolation How and why did it happen? Modeling What’s the best action? Optimization What’s the best/worst that can happen? Prediction InsightInformation
- 10. The Price Elasticity Project Business Problem While we are tracking the won, lost, and WIP quotes at a macro level, we do not currently have a system to identify potential opportunities to demand a premium to the market based on market, region, or product (item).
- 11. SCOPE tools: from historical pricing data to our won/lost quote data 11 “We have to find a way of making the important measurable, instead of making the measurable important.” Robert McNamara, U.S. Sec. of Defense
- 12. SCOPE tools: from historical pricing data to our won/lost quote data 12
- 13. Key Questions Addressed by Analytics 13 D.R.E.A.M. Source: Davenport et al. Analytics at Work: Smarter Decisions, Better Results Past Present Future What happened? Reporting What is happening? Alerts What will happen? Extrapolation How and why did it happen? Modeling What’s the best action? Optimization What’s the best/worst that can happen? Prediction InsightInformation
- 14. 14 D.R.E.A.M. Source: Davenport et al. Analytics at Work: Smarter Decisions, Better Results Past Present Future What happened? Reporting What is happening? Alerts What will happen? Extrapolation How and why did it happen? Modeling What’s the best action? Optimization What’s the best/worst that can happen? Prediction InsightInformation Key Questions Addressed by Analytics
- 15. A Machine Learning Approach Misfire D.R.E.A.M.
- 16. Price Elasticity and Optimization 16 A traditional Price Elasticity Curve… Price Elasticity at Worthington Steel…
- 17. Model Design – What is steel? 17 • Alloy (carbon content) • Thickness • Width • Other elements (N, Si, Mb, etc) • Shape (coil, sheet, blank)
- 18. Model Design 18 • ~8,000 observations • 150 original variables (50 useable) • >500 engineered variables
- 19. Overfitting 19 ProbabilityofWinning Normalized Price Cold Rolled? CR Strip? Price < $50 Win yes Price < $60 WinLose Confounding variables!
- 20. How should features be chosen to maximize accuracy and minimize over-fitting? 20
- 21. Let the User Choose with Tableau (and R)! 21
- 22. 22
- 23. 23 User input for real-time market pricing Support for sensitivity and what-if analysis of different price points Ability to analyze variances in absolute and percentage terms
- 24. 24 Users select which training observations to include/exclude. This improves business relevance and reduces the likelihood of overfitting.
- 25. 25 Historical wins and losses by price are provided for context Tooltip provides individual quote data Chart type and layout chosen to imply logistic regression context
- 26. 26 Expected value as a function of price Model identifies price that maximizes expected value Model predicts win probability given simulated price Expected value defined as: 𝐸𝑉 = 𝑃𝑟𝑖𝑐𝑒 × 𝑃(𝑊𝑖𝑛𝑛𝑖𝑛𝑔) Leakage from sub-optimal pricing
- 27. 27 Traffic light indicates model quality Price ProbabilityofWinning Invalid Market Valid Market Tooltip provides additional context and statistical measures
- 28. What makes a successful journey? • Strong engagement from the business • Alignment across functional areas (sales, analytics, IT) – are we solving a real business problem? • Introduce complexity in pieces to reduce learning curve • Allow users to see their data (in addition to the model)
- 29. Tableau and R Technology Stack 29 D.R.E.A.M.
- 30. Tableau Server RserveVCS Data Scientist CRAN Business Analyst 1. Develop and push model as an R package 2. On merge to master, build to an internal CRAN mirror 3. Install/update package on Rserve server from CRAN 4. Call package and function from R script in Tableau 1 2 3 4
- 31. 31 Tableau Server Rserve Call package and function from R script in Tableau
- 32. Why build and maintain this infrastructure? • Efficiency of continuous integration and continuous deployment • Stability and reproducibility of version control • Automated testing in R (testthat package) • Performance gains from distributed and concurrent processing – R and Tableau are running on separate servers • Security of a CRAN repository that is behind the firewall
- 33. Thank you! #TC18 Wil Davis github.com/wkdavis William.Davis@worthingtonindustries.com Steve Bartos @OldDirtyBarGraph Steve.Bartos@worthingtonindustries.com D.R.E.A.M.
Editor's Notes
- Excitement Varying Skill Level Support taking away from Core Team resources ROI concept is a tough one – difficult to put this in the analytics bucket…focus on the how it is embedded in T2.0, optimizing the business… Relying on our business partners to act on this information. ~Cardinal Health
- Improving people These will be used by SCOPE, Product Managers, Price Risk, and potentially and Regional Managers to get some competitive info for their territories. The market information will help identify pricing trends and win rates based on which direction the market is moving, and the competitor info might shed some light on if certain competitors price more aggressively than others on certain products, and if we need to adjust our pricing based on who we are quoting against.
- Improving people These will be used by SCOPE, Product Managers, Price Risk, and potentially and Regional Managers to get some competitive info for their territories. The market information will help identify pricing trends and win rates based on which direction the market is moving, and the competitor info might shed some light on if certain competitors price more aggressively than others on certain products, and if we need to adjust our pricing based on who we are quoting against.
- Chart on the shows a typical demand curve for a functioning competitive market Chart on the right shows the demand curve we observed in our data Transforming the data (log) did not yield improved results
- Physical attributes of steel are numerous Each combo is a different product Potentially a different market Results in very high dimensional data
- machine learning approach due to the hi-dimensional nature of the data 150 original variables are mostly product attributes for quoted products (hence the “high mix environment” Knew that we ultimately needed a linear model in order to represent a functional market I’ll get into why this is the case a little later While not naturally linear, traditional demand curve could be made linear through a transformation
- Is it cold rolled material? Cold rolled means rolling mill used to reduce the thickness creates a thinner, flatter product different hardness and strength characteristics One of our more specialized products Is it Cold rolled strip A More specialized product Is the price less than $50? If yes, we win If no, we then have to ask ourselves if the price was under $60. If no, we lose If yes, we win There must be some variables we are missing that differentiate the >$60 quotes clearly in a different market customers’ willingness to pay for it is not similar to their willingness to pay for the product in the other 2 quotes that are under $60 machine learning model able to pick up on this and accurately classify the quotes but behavior of the quotes was not in line with economic theory chance of winning should decrease as price increases, not the other way around The relatively small sample size (given the dimensionality) resulted in inaccurate linear models and highly non-linear accurate modes
- Need to eliminate features to reduce over-fitting Need to retain enough features that the model can identify points of product/market differentiation
- Overview of dashboard Stoplight Filters and parameters 3 main visual displays of data Design: A lot of white space similar feel to their traditional design tool, no sensory overload White space gives some idea of scale – blowing up thickness of the line or zooming in would distort the gradient of the curve Limited use of color highlight important areas Stoplight win/loss performance predictions Colors match intuitive interpretation green = good red = bad
- Parameters Enter info about current state of the market control normalization of historical data Functions similar to a seasonal adjustment observe the deviation from some baseline rather than the baseline measurement itself The price a customer is willing to pay at least partially depend on the prevailing market indices there is not a perfect correlation, but you would not be willing to pay $5/gallon for gas if oil was $50/barrel Last parameter supports what-if analysis enter a price that draws a vertical reference line on the charts our outside sales team or our customers will give us what they feel is the “correct” price this allows the analysts to put that price in the context of the model in a visual way you’ll see the visualization later on
- Market filters User selects training data that best represents the market for the product that they are quoting Select similar product attributes, geographies, customer attributes, and end use applications With a dataset this small and exhibiting severe data quality issues, human-based feature selection is proving to be more effective at this time
- Historical performance Users want to see the actual (historic) quotes that are being used as training data visualization helps them to understand our win-loss performance in this market Tooltips provide them with additional info about the quote use the quote number in the tooltip to retrieve additional information about the quote from our ERP system structure of the plot helps the user to understand the way the model views and treats the data team understands the general concept of drawing a regression line through quantitative values winning and losing is (on the surface) binary… win-loss logistic regression model - map binary outcomes to real values wins = win probability of 100%, losses = win probability of 0% The plot displays the data in this way in order for the users to understand the underlying representation of the data in the model also see how a regression “line” could be drawn through the data Helping the user visualize the connection between the data they are familiar with (the historical training data) and the curve drawn by the model helps to build user trust the model is not a “black box” anymore (or at least it is something of a gray box)
- Model predict win probability across vector of hypothetical price points (more on the R piece later) curve is a proxy for a demand curve the probability of winning a given bid decreasing as a function of price price vs probability curve is great, but it begs a greater question We don’t want to win 100% of quotes, because that means are prices are too low We won’t want to lose 100% of our quotes… prices are too high Goldilocks – what win rate is just right? Where should we price to fall on the optimal point along this curve? objective function = maximizing the Expected Value of the quote In a hypothetical world suppose a quote is bid on an infinite number of times If we expect to win that quote 75% of the time at $100 the expected (average) value of that quote over time will be $75. I’m not going to get into more detail than that, but if you’re curious you can read up on the concept of Expected Value. Ideal world - calculation would use profit rather than price to calculate expected value profit may change as a function of probability (or volume) due to economies of scale or discounts we do not have this data readily available, so we focused exclusively on sales/revenue expected value is calculated for each price-probability combination simulated by the model Model identifies the maximum expected value from the quote and selects the corresponding price as the optimal price size of the data is relatively small - this is done with a basic search through the vector of expected values looking for the maximum price more complex datasets - use of calculus (in R) may be a more efficient means of identifying the optimal price As mentioned earlier, the user has the ability to enter a price of their own proposing that draws a vertical reference line in the visualization That line appears here can give the user a visual indication of the margin leakage that would occur if they were to charge their chosen price rather than the price indicated by the model
- Model status Perhaps the most important visualization on the dashboard binary signal of the statistical and economic validity of the model Tooltip provides more detailed quantitative measures of the model. economic theory – demand goes down as price goes up In our case – probability of win goes down as price goes up not always what the model returns incomplete data inaccurate data Sometimes model shows probability go up as price goes up calculus on the probability curve to determine if win probability goes down as price goes up may seem to be overkill in this situation analysts would look at an invalid curve and immediately identify that something is not quite right without additional context they may stop believing the model or think it’s broken borrows on the lean principles of signals and visual management green means the model is okay to rely on, and red means it is not tooltip explains to the user why the model is or is not valid we can build user trust that the model can (somewhat) police itself as data scientists we are acknowledging the fact that the world is a dangerous place outside the confines of academic theories, and we can build tools that protect us when our world does not fall in line with those theories.
- Engagement Have a champion Are you building something essential to success? If your tool breaks, will someone call you telling you they can’t do their job? Alignment Support company-wide goal Do your suppliers (IT) know what you need? Do you know what your customers (the business) need? Can we deliver throughout the value chain? Pieces Don’t wait until it’s perfect Iterate and improve (CRISP-DM) Don’t overwhelm your users – delivering in pieces allows you to incorporate feedback into new releases faster See their data In organizations new to analytics people haven’t seen their data They don’t need, or don’t know if they need, multi-layer deep learning network, because they don’t know what’s happening now A model without the underlying context causes 2 problems People won’t trust it (and won’t use it) People will blindly trust it, we won’t get feedback and won’t be able to catch problems or build enhancements
- CI/CD and Version control new to data science takes more effort to establish them up front, they will create a more robust sytem Data scientists develops code/models and pushes them to version control software model used for this dashboard was built as an actual R package When code is merged into the master branch, the R package is built and deployed to an internal CRAN mirror including automated testing A mirror is kept inside our firewall for security purposes separate linux server running Rserve checks for internal package updates on a nightly basis Running Rserve separate from Tableau Server improved performance of both applications improved concurrency Tableau server is configured to connect to the external Rserve server execute the R code in the calculated field Business users access the dashboard and model via Tableau Server
- Building the model as an R package has a number of advantages automated testing easier via packages such as testthat built-in framework for documentation Simplicity in Tableau only call a single function in our Tableau script, pe_lm custom function we designed to run our model take the 2 Tableau fields (price and status) as inputs return the prediction results formatted exactly as we would like to see them returned goal to minimize the amount of R code that exists in the Tableau calculated field Why? Ease of making changes to model and dashboard multiple output values from the model (i.e. the predicted values, the p-value, and the b0 and b1 coefficients in our case) requires 4 different calculated fields in Tableau one for each value we with to return All 4 of these calculated fields require access to the model code the model code must be written 4 different times in Tableau If change to the code and the whole model is written in the Tableau calc would have to change 4 different calculated fields creating this custom function to use in Tableau make the change 1 time in the R package, in the code for this custom function CI/CD pipeline automatically deploys the updated function and package to Rserve the next time Tableau calls that custom function, the new code is evaluated in that function Deployment is automatic, changes are tracked (in VCS)