Real Time Data Warehousing Mastering Business Objects June 11


Published on

This is a copy of a presentation I gave at the Mastering Business Objects conference in Sydney, June 2011. It explains the move Star Track Express is making towards Active Data Warehousing to support both Analytical and Operational needs from a single platform.

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • On any given day this happens for 300,000 items05:40 – Driver looking for freight09:00 – Customer calling call centre10:00 – Customer using Web
  • Large data – rapid movement to
  • Delivery – anywhere in Australia, messages over wireless IP into OLTP serverSortation – in our facilities, file transfer into mainframe.Traditional Batching Not Appropriate from a timing perspective. Have to be careful about impact on source systems. Different technologies.
  • Web/EMS – tens of htousands of interactions per day – dast response required – SERVICEDashboard – not many interactionsReports – run/schedule, etc
  • Two columns – one for each need. Show implication (extra DB server, hardware, DBA and then ETL)Highlight need for same dataNew WebsiteOld scenario was TEAM basedWanted an ODS (not another transactional system)New EDWSQL2008 but wasn’t going to cut it for the growthWanted an EDWData same as in ODSSeparate or One ?Ideally 1 platform – but can you have your cake and eat it to ??
  • Cards
  • Users – Queues - TASM
  • Tibco ESBAttunity StreamData Integrator
  • Real Time Data Warehousing Mastering Business Objects June 11

    1. 1. Active Data Warehousing<br />Can you have your cake and eat it too ?<br />Jeff Monico<br />General Manager Information Systems<br />
    2. 2. Agenda<br /><ul><li>INTRODUCING STAR TRACK EXPRESS
    6. 6. THE RESULTS
    8. 8. QUESTIONS</li></li></ul><li>Company Overview<br /><ul><li>Australian owned and operated Express Freight Company.
    9. 9. Founded in 1974 by Greg Poche. Greg’s influence and core guiding principles are very much part of the business ethos today.
    10. 10. December 2003 saw Australia Post and Qantas form a joint venture to acquire Star Track Express from its founder.
    11. 11. On the 18th May 2011 the retail division of Australian Air Express was merged into Star Track Express to create Australia’s largest door-to-door express service provider.
    12. 12. 4,500+ Employees, 3,750+ Vehicles (Vans, Trucks, Prime Movers, Trailers, Forklifts).
    13. 13. Each day over 400,000 items of freight are picked up and delivered, with an industry leading ~99% success rate.</li></li></ul><li>Core Services<br />
    14. 14. How we collect and use data<br />Business activity has a very short life cycle, with data being collected and used at many points during that lifecycle……<br />18:30<br />18:40<br />19:00<br />16:45<br />DATA COLLECTION (SCANS)<br />TIME EVENT DESCRIPTION<br />16:45 Freight picked up<br />18:30 Freight unloaded at Sortation Depot<br />18:40 Freight sorted at Sortation Depot into Lane 21<br />19:00 Freight loaded onto interstate trailer M200<br />05:00 Freight unloaded at Sortation Depot<br />05:35 Freight sorted at Sortation Depot onto Centre Wishbone<br />07:00 Freight loaded onto delivery truck ABC123<br />09:30 Freight successfully delivered<br />09:30 Consignment Note signed by Bob.<br />09:30<br />DATA USAGE<br />05:00<br />05:35<br />07:00<br />
    15. 15. Problems we face<br />Large Volumes of Data<br />Producing millions or records per day which are needed for operational and analytical queries.<br />Fast Response Times<br />Users demand sub second response times on operational queries.<br />Near Real Time Loads<br />Real time business needs real time data from a variety of sources. Latency of more then a minute or two is an issue.<br />
    16. 16. Large Data Sets<br />Extending the data being collected from financial, to analytical, to transactional created exponentially larger data sets……..<br />
    17. 17. Low Latency<br />Data comes from multiple sources, from inside and outside of local networks and needs to be available with very low latency………<br />OLTP SQL Server<br />Wireless IP<br />Delivery Event<br />?<br />?<br />File Transfer<br />Mainframe (RMS)<br />Sortation Event<br />EDW<br />
    18. 18. Varied Query Response Needs<br /><1s<br /><1s<br /><10s<br /><1m<br />
    19. 19. Data captured<br />Intelligence delivered<br />Action <br />taken<br />Low latency, fast response times and effective BI together helps accelerate decisions and create business value…..<br />Business event<br />Value<br />Time<br />Missed Opportunity<br />Situation<br />Gained Opportunity<br /><ul><li>Two Trucks are dispatched to same location for delivery
    20. 20. Customer is curious and a bit frustrated as to why 2 trucks are needed…
    21. 21. 1 connote, 5 cartons and 1 pallet arrive at destination depot, pallet goes to bulk warehouse
    22. 22. Cartons and bulk are loaded ontoonetruck for delivery
    23. 23. Customer is pleased to receive the consolidated delivery</li></ul>TDWI The Business Case for Real-Time BI<br />Based on concept developed by Richard Hackathorn, Bolder Technology<br />Accelerating Decisions <br />
    24. 24. The trigger for change<br />By 2010 we had identified two needs in the business……. <br />New Operational Data Store to act as central data repository for SOA based architecture.<br />New Enterprise Data Warehouse to meet data growth as we extended the subject areas captured.<br />ODS<br />EDW<br />ADW<br />Single platform to provide both an Enterprise Data Warehouse and an Operational Data Store – an Active Data Warehouse<br />
    25. 25. What is an Active Data Warehouse ?<br />Activating the Data Warehouse means more then just a very powerful fast database server……<br />
    26. 26. MPP Data Warehouse Server<br />Large volumes, active load and active access needs a different type of database technology – Massive Parallel Processing……<br />EXAMPLE: 100 random cards, return all 7 of Hearts.<br />1 ‘CPU’<br />100 cards.<br />100 ‘CPU’ Cycles<br />100 Seconds<br />4 ‘CPU’s’<br />25 cards each<br />Each ‘CPU’ holds a suit.<br />25 ‘CPU’ Cycles from 1 CPU<br />(3 ‘CPU’s’ Idle)<br />25 Seconds<br />4 x Improvement in Time<br />4 x Improvement in CPU<br />2 ‘CPU’s’<br />50 cards each.<br />100 ‘CPU’ Cycles (50 cycles each)<br />50 Seconds<br />2x Improvement in Time<br />NO Improvement in CPU<br />
    27. 27. Active Workload Management<br />Active management of workload is needed to ensure information is delivered at the ‘right time’ and at the lowest total cost of ownership……<br />
    28. 28. Streaming Data<br />Traditional batch processing will not deliver the near real-time data loads an Active Data Warehouse demands……<br />ENTERPRISE SERVICE BUS<br />RMS<br />OLTP<br />Change Data Capture<br />OLTP<br />EDW<br />
    29. 29. Information Delivery<br />Traditional Business Intelligence tools (reporting) will not deliver information in a manner that enables fast decision making……<br />Well designed Dashboards deliver information which can be consumed very rapidly using good visual design;<br />Visual based ad-hoc analysis tools (Explorer) provide users capability to rapidly get to the information they need from vast data sets;<br />Mobile delivery provides information to people when and where they need it.<br />
    30. 30. Architecture<br />ADW<br />Change Data Capture<br />ENTERPRISE SERVICE BUS<br />Data Services<br />WebI<br />Dashboards<br />
    31. 31. The results<br />August 2010 – ‘Lift and Drop’ of existing SQL Data Warehouse onto Teradata ADW. Approximately 500GB migrated, and approximately 1,000 complex BI queries per day are run against this.<br />November 2010 –ODS and new Web Site goes Live. Supporting hundreds of thousands of tactical queries per day on the ADW platform.<br />Now ADW platform is supporting this mix of BI and ODS loads and queries, and has maintained 100% up-time and 100% of queries meeting SLA’s.<br />1 DBA + 1 Platform = Low Total Cost of Ownership.<br />EDW<br />ODS<br />
    32. 32. Operational Intelligence<br />ACTIVATINGMAKE it happen!<br />OPERATIONALIZING<br />WHAT IShappening?<br />PREDICTING<br />WHAT WILL happen?<br />Event-based triggering takes hold<br />ANALYZING<br />WHY did it happen?<br />Value<br />Continuous update and time-sensitive queries become important<br />REPORTING<br />WHAT happened?<br />Batch <br />Ad Hoc <br />Analytics<br />Analytical modeling grows<br />Increase in ad hoc analysis <br />Continuous Update/Short Queries <br />Event-Based Triggering<br />Primarily batch and some ad hoc reports<br />Complexity<br />
    33. 33. Questions, Comments and Observations<br />