Recommended
More Related Content
Similar to Syst 542 paper presentation erickson final
Similar to Syst 542 paper presentation erickson final (20)
Recently uploaded
Recently uploaded (20)
Syst 542 paper presentation erickson final
- 1. SYST 542 Paper Presentation: John Erickson Fall 2016 From Magnum Opus to Mea Culpa: A Cautionary Tale of Lessons Learned from a Failed Decision Support System
- 2. Outline • Introduction – Critical Success Factors: class reading, David Arnott – Literature on DSS • Synopsis of Case Study / Paper – DSS Justification – DSS Goals – DSS Development – DSS Design – Model Results – Upgrade Results • What Went Wrong – The Catalyst of the Cataclysm – The “Postmortem” – The Mistakes/ Lessons Learned • Conclusion
- 3. Introduction Why study failures and lessons learned? 1. High Failure Rate: “It is well known in the DSS literature that all types of DSS projects are high-risk and prone to failure…Some studies have even reported failure rates as high as 80%.” (Arnott class paper) 2. Lack of Literature: Gorman claims there are very few articles on DSS failure: “If one were to review the OR/MS literature on DSS implementations, that reviewer would be hard pressed to find published examples of failure.” Handbook of decision support systems, vol. 1: Basic Themes: Chapter 34, David Arnott
- 4. What makes a DSS successful? *Handbook of decision support systems, vol. 1: Basic Themes: Chapter 34, David Arnott Critical Success Factor Model (CSF) Method10 Critical Success Factors 1. Committed and informed executive sponsor 2. Widespread management support 3. Appropriate team skills 4. Appropriate technology 5. Adequate resources 6. Effective data management 7. Clear link with business 8. Well-defined information and systems requirements 9. Evolutionary development 10. Management of project scope “The implication of the table is that if a reasonable number of CSF’s are not attained or achieved, a project is likely to fail.”*
- 5. Synopsis of Paper • Michael Gorman, PhD is a management- sciences university professor and consultant in the freight / railway industry • He was hired to build a DSS for “ACME”, a growing freight transportation company that had volume growth in the early 2000s (ACME is an alias for confidentiality). • It was initially successful, then the DSS failed due to the Wall Street crisis in 2009 • His paper outlines the conditions for the need of the DSS, it’s early success, downfall, and then lessons learned.
- 6. ACME’s System Justification • Why did ACME Build the DSS? – Previous methods for load acceptance and dispatching failed to achieve desired economic results – Frontline decision makers lacked the data, analytical ability, and global perspective to manage a network that was growing in complexity. – Thus, ACME had a strong incentive to change its approach. “Because of its growing volume and asset base, in 2005, a senior executive at the company realized that ACME had the scale to justify a DSS that would support decision makers in the face of changing fundamental economics resulting from increased asset intensity. ..”
- 7. DSS Goals • DSS Goals: 1. Support real-time decision making at a company with annual revenues of $3 billion by cutting across both marketing and operations 2. Replace suboptimal rules for decision making (“outdated and myopic” rules)
- 8. System Development • Senior management: – provided input on system philosophy , design, and marketing • Mid Level Management: – operations managers and frontline staff documented the business decision making process side • IT professionals: – established system integration and other technical requirements. In short, the DSS had to be integrated in such a way as to minimize legacy system development costs and disruption to ongoing operations.
- 9. System Design Orders (These historical orders were used for demand forecasting) Economic Data (Historical cost, price, and margin data were used for predicting profitability) Service-Time Data (service times of transportation providers affect future supply Input Output Supply-and-demand forecast (supply forecast predicted equipment capacity ; demand forecast predicted daily customer load tenders) Load-accept optimization Dispatch optimization (load to- equipment assignment) -The DSS produced then a ranking of best ways to load/dispatch, based on the lowest overall cost, including both current conditions at the origin and future conditions at the destination and the network effects of other assignments to be made. -It ran without interruption each day for more than two years for forecasting and load acceptance, and every 15 minutes for load dispatching.
- 10. Model Results & Upgrade Early success • Model recommendations were generally well received by frontline decision makers. • The model recommendations were presented in a similar way as before, as ranked recommendations, but with more sophisticated model-based rankings. Motivated by early success, the model was improved: • Support modules • More detailed forecasting and dispatching • These improvements came at a cost—increased model complexity -The shadow price of violating model-based recommendations was given, showing the opportunity cost of not following recommendations. -Global optimization results were presented to dispatchers making sequential decisions -Any change in supply, demand, or non-model recommended assignments would be reflected in subsequent model runs - Current best was always available to dispatchers, given prior events. What Model Presented in Real-Time:
- 11. Upgrade Results (post-upgrade) • Preliminary results indicated higher revenues, higher service levels to the best-paying customers, lower costs, and better asset utilization (i.e., lower excess inventory). • First year results: formal internal audit after the model’s first year in production, the results indicated a number of performance improvements: – higher asset velocity (more loads per time period), – a decrease in low-margin loads – an increase in average margins – reduction in cost per load. In the model’s first year, the audit estimated savings of more than 20 times the system development cost.
- 12. Then Things Went South… • The Perfect Storm: – Financial credit crisis on Wall Street in 2009-2010 – ACME’s cutting discretionary spending on consulting in 2010 • DSS was shut down and revert back to the simpler but safer rules of load acceptance and dispatch. • Management became skeptical of DSS • Gorman felt the need to catalog the lessons learned and why his “magnum opus” ultimately failed Bottom Line: The beginning of the end was this: the Wall Street credit crisis caused a perturbation that the DSS couldn’t handle
- 13. The Central Failure in the DSS • The biggest mistake was an implicit assumption surrounding the structural stability of the demand forecast. • Although recent low-demand history helped to pull forecasts downward, previous years’ demand levels served to inflate forecasts. The underlying probability distribution used for the optimization model turned out to not be an accurate distribution when the credit crisis hit
- 14. “Postmortem” 1. Implicit Assumption: Demand Forecast Structural Stability 2. Implicit Assumption: Cost Structures 3. Too many decision makers 4. Too many parameters 5. Too much data 6. Too Little Knowledge: Insufficient Skills Transfer 7. Too Little Time Allocated: Failed Vigilance in Ongoing DSS Operation 1. More careful and proactive tracking: analysis of recent forecast errors would have reduced prolonged use of a consistently high forecast after the downturn. 2. Be leery of using historical data in the model 3. Limit scope creep: Trying to be “all things to all people” 4. Limit model parameters: by striving for perfection, model usefulness and robustness can often be diminished 5. Robust support staff: Have sufficient support staff to the data infrastructure model. 6. Ongoing training, problem solving (live case studies),and troubleshooting between developer and client 7. Project Manager: a DSS owner, who is responsible for DSS performance, responding to customer problems, and uncovering previously undiscovered anomalies, is necessary. Mistakes Lessons Learned
- 15. Current Situation – No DSS “The fear and mistrust of optimization techniques subsequent to this project, coupled with lack of tolerance for an association with the model failure, leave little appetite for advanced analytics to support decision making at ACME.” -Gorman
- 16. Author’s Final Conclusions • Beware of implicit assumptions: Models inherently assume historical data patterns as a guide to future decisions. • Minimize complexity wherever possible. • Socialize, train, and experiment with DSS models. • Maintain constant vigilance of model performance. “Modeling is never done. When a client asks when the project will be done, the best reply is “never”; follow this with an explanation that tells the client that the first version will begin delivering recommendations by a specific date, at which time the team will want to continue to assess and improve on the recommendations through more accurate data and modeling.”
- 17. Takeaways from Paper Takeway 1: Think about your DSS: A lot of thought should be put into the scope of the DSS project based on the Critical Success Factors. A bigger project may not be worth it unless the willpower, resources, and executives are all on-board. Takeaway 2: Even if the DSS meets all/most of the CSF, it takes constant vigilance to keep/upgrade/test/validate the system. This is a sobering thought for tech/internet startups!! Critique: I would have preferred that the author (Gorman) used these 10 CSF (or a similar metric) throughout his paper so that the reader can get a measurable sense of what went wrong according to the CSF. Also, use the CSF’s and compare over time (6-month increments) where the DSS stands against the CSF.
- 18. Questions ?
Editor's Notes
- Greeting, Name, Title
- Outline of My Presentation: introduction from arnott, breakdown of dss, what went wrong , and finally conclusion.
- Why would choose a paper on what went wrong? Both Arnott and Gorman have something to say here.
- This is from Arnott’s case study (required reading): The first 2 are about management support; without support from the top, the DSS can’t survive very long. The next group is skills/resources to enable/support the DSS; next is business requirements; last 2 are project management and scope. Of note—management involvement is something that seems to crop up again and again. The picture on the right is Arnott’ way Of showing that if a DSS has most/all the 10 CSF’s, and if the user is satisifed, and if the system makes a positive impact, then the DSS is successful.
- ACME is a growing freight transportation company in an evolving industry that had volume growth in the early 2000s As a result of this growth and expanded investment in centrally controlled assets, previous methods for load acceptance and dispatching failed to achieve the desired economic results. Myopic rules of thumb for load acceptance and dispatching failed to achieve economically advantageous results for both asset management and customer service. The new focus on asset management, positioning, and yield rendered previous decision-making methods ineffective.
- -better reflect strategic managerial perspectives in frontline decision making -Initial projections of improved performance were supported by positive proof-of-concept results -Because of its growing volume and asset base, in 2005, a senior executive at the company realized that ACME had the scale to justify a DSS that would support decision makers in the face of changing fundamental economics resulting from increased asset intensity The potential gains more than justified the projected system costs (the total internal and external project cost was in the hundreds of thousands of dollars), indicating a large potential return on investment for a completely developed DSS. -A formal organizational audit of the production system one year after its implementation found the anticipated savings to be largely realized, and the DSS was considered a success!! --Shortly thereafter, however, its recommendations began to be questioned, confidence in the model’s recommendations waned, and the system was quickly abandoned. --After the system’s demise, multiple meetings with the client were held to diagnose the causes for the failure and potentially reinstitute the DSS. The system was not recovered, but the information gleaned during this time was valuable nonetheless.
- Real-time decision making to increase revenues; replace old/out-dated rules
- A short-term demand forecast predicted daily customer load tenders by geographic origin, destination, date, and other attributes (e.g., specific equipment needed) over a two-week period. A supply forecast predicted equipment capacity based on expected termination of previously accepted business To maximize the potential revenue of short-term capacity, a load-accept optimization module determined which tendered orders to accept and which to reject based on anticipated capacity shortfalls, if any Dispatch optimization: The output of the dispatching system was a load to- equipment assignment. It was presented to dispatchers as a list of assignments ranked from best to worst—as they had seen prior to the model’s implementation.
- -There were many weaknesses in the system that were not exposed until the financial credit crisis on Wall Street in 2009-2010 -The downturn in 2009 led to revised 2010 corporate forecasts, budgets, and strategies across the U.S. economy. Many shipper contracts were canceled and investments deferred. This affected a corresponding freight demand for ACME and the fundamentals on which the DSS rested. -The dramatic shift in business exposed the instabilities of a well-performing and stable DSS that had been in production for more than two years. The vulnerability of the DSS was compounded by the size, speed, and disruptiveness of the shift in the market, leaving very little time to diagnose new and impactful problems in the model. -These problems were compounded by ACME’s rapidly falling discretionary funds, which caused ACME to cut discretionary spending on consulting in 2010. -I worked with ACME during the last days of the model’s operation to diagnose and rectify problems. With better timing, effort, and luck, the DSS may have been able to overcome the business environment shift in 2009–2010. In an effort to simplify operations and reduce confusion, management decided to shut down the DSS and revert to the simpler but safer rules of load acceptance and dispatch.
- Mistakes are primarily related to 1) assumptions; 2) too many or too little of various things… 3)then the time to maintain the DSS on a very vigilant scale. Regarding assumptions, this is really related to the historical data used for the underlying probability distribution. The decision makers aspect was that it tried to be all-satisfying in it’s scope—both frontline users and senior management. This is evident from numbers 3-5. Number 6 and 7 are speaking to the training and maintenance, and constant upgrade and constant vigilance of the DSS. Fine-tuning it continously.
- Having been burned by a rapidly changing market and a DSS that did not adjust to such changes, ACME has returned to a much simpler direct-cost, margin-based system. Although the approach is less likely to make a clearly poor recommendation, it regularly omits key considerations that the DSS was designed to include. By throwing away DSS-based opportunities for improvement on every transaction, ACME avoids the risk of overly ambitious recommendations,which can occur when a model ceases to reflect reality.
- In the end, Gorman’s summaryizing final conclusions are: beware of implicit assumptions, minimize complexity, socialize train and experiment; and maintain constant vigilance. His quote I think accurately summarizes his point, where he says “ modeling is never done…”