World 2010 - Migration Process


Published on

This presentation was given at Micro

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • EDW Developers test in Development, Developers and Business Users test in Test. Any issues or changes that need to be made are always made in Development and then re-migrated forward.
  • A lot of our processes were dictated by issues we ran into in the past, such as migrating components before dependents. This included migrating attributes and facts before metrics and filters and those before reports. These often could be traced back to errors in Object Manager that had long since been patched.
  • Closing Slide, contact information, EDW/Ryder logos
  • Provide SQL code that mines the metadata for the object prompt dependencies. Listed here as an appendix for anyone who wants to try to implement these processes for themselves.
  • World 2010 - Migration Process

    1. 1. Bryan Brandow MicroStrategy Architect Ryder System Inc. January 27th, 2010Improving Object Migration Management with Easepowered byRyder’s Enterprise Data Warehouse
    2. 2. Ryder System, Inc. is a FORTUNE® provider of leading-edge transportation, logistics and supply chain management solutions worldwide. Ryders product offerings include: Fleet Management Solutions (FMS), which provides leasing, rental and programmed maintenance of trucks, tractors and trailers to commercial customers; Supply Chain Solutions (SCS), which manages the movement of materials and related information from the acquisition of raw materials to the delivery of finished products to end-users; and Dedicated Contract Carriage (DCC), which provides a turn-key transportation service that includes vehicles, drivers, routing and scheduling. Ryder serves customer needs throughout North America and in Latin America, Europe and Asia.For 75 years, Ryder has earned high marks for serving customers in a timely, cost-effective and reliable manner. That reputation has helped Ryder to continually rank among the industrys leading companies in many publications, including the FORTUNE® Most Admired Companies, the InformationWeek 500 of leading business users of information technology and InternetWeeks top 100 U.S. companies for effectiveness in using the Internet to achieve tangible business benefits. Ryders stock (NYSE: R) is a component of the Dow Jones Transportation Average and the Standard & Poors 500 Index. 12/29/2011 Proprietary and Confidential 2
    3. 3. EDW Platform StatisticsUsage 8 – 14,000 reports run each day Over 9 million reports generated in last 8 yearsReports > 250 templates with over 35,000 user saved reportsUsers 14,000 (7,000 Internal, 7,000 External)Applications 20 Subject Area/Applications leveraging the MicroStrategy SuiteDatabase 3 database platforms Nearly 3,000 staging, development, and production tables Over 4.0 Terabytes of dataAwards 2008 MicroStrategy World Winner Best Practices in BI Individual Excellence Operations and Maintenance, Customer Fleet Care applications considered “World Class” – Hackett Group 2007 12/29/2011 Proprietary and Confidential 3
    4. 4. Importance of Data Governance 12/29/2011 4
    5. 5. Importance of Data Governance • Single Version of the Truth ► Ensure unified business rules throughout the environment. • Reduced Support ► Standardized objects promote reusability and faster development. • Gatekeepers not Decision Makers ► EDW ensures a unified and consistent environment based on Business Rules. “How we decide how to decide” 12/29/2011 5
    6. 6. Ryder‟s Change Management Strategy• Development occurs in a dedicated environment, and after passing Developer Unit Testing, is promoted to a Test environment which mirrors Production.• Business Users utilize the Test environment to approve projects and data. Further testing from the EDW team is performed as well as a check to make sure the migration details are in order.• Once signed off, the application is promoted to Production. 12/29/2011 Proprietary and Confidential 6
    7. 7. Object Manager Overview • Object Manager provides a single graphical drag and drop interface for migrating objects between environments such as Development, Test and Production. 12/29/2011 Proprietary and Confidential 7
    8. 8. Object Manager Overview• Dependencies are determined at the time of the migration and decisions on objects with different versions between environments must be made. 12/29/2011 Proprietary and Confidential 8
    9. 9. Ryder’s Original Migration Process How we got here. What worked well. What didn‟t work so well. 12/29/2011 Proprietary and Confidential 9
    10. 10. Original Migration Process• Basic documentation of migration actions ► This Excel file is passed around between developers and administrators, each appending to the log. 12/29/2011 Proprietary and Confidential 10
    11. 11. Original Migration Process• Change Tickets for future tracking 12/29/2011 Proprietary and Confidential 11
    12. 12. Original Migration Process• Separate migrations for each folder► Unrelated objects that were in different folders had to be migrated individually. This would extremely lengthen the amount of time required for each migration depending on how many objects were involved.► In our larger projects, the processing of a single migration takes 10-15 minutes. A large migration involving several folders could easily take an hour or two of an administrator‟s time, and the environment is unavailable during this process. 12/29/2011 Proprietary and Confidential 12
    13. 13. Original Migration Process• Migrations performed in 2-Tier► Due to past experiences, all of our migrations are performed in 2-Tier. This eliminates potential object caching issues between the active IServer and the Metadata.► After a 2-Tier migration is complete and the schema is updated, the project must be reloaded. For Production projects, we reload as soon as there are no users in the project. But for our large projects that have long reload times or have constant user traffic, we have scheduled tasks that reload the projects at night. 12/29/2011 Proprietary and Confidential 13
    14. 14. Original Migration Process• Did it work? ► We had no way of knowing if a migration was successful unless we tested the reports manually or a user called us with an issue. ► Since we migrate in 2-Tier and reloaded the production projects at night, it was common that we would not catch an issue until users saw the reports the following morning. By this time, fixes that required an additional migration would now have to wait another day due to the project reload requirement. 12/29/2011 Proprietary and Confidential 14
    15. 15. Original Migration Process – Goals to Improve• Each Object location was migrated individually, lengthening the migration process ► Find a way to consolidate objects so that migrations were more efficient.• Logs were manual and time consuming ► Find a way to assist or automate the creation of logs for Developers.• Results of the migration were unknown until further testing or often discovered by the end users ► Find a way to validate migrations to ensure that they were completed correctly. 12/29/2011 Proprietary and Confidential 15
    16. 16. New Process Ideas• Each Object location was migrated individually, lengthening the migration process ► Find a way to consolidate objects so that migrations were more efficient.► Use Object Prompts to serve as containers for Migrations. 12/29/2011 Proprietary and Confidential 16
    17. 17. New Process Ideas• Logs were manual and time consuming ► Find a way to assist or automate the creation of logs for Developers. ► Query the Metadata to check the Target project BEFORE the migrations to determine if the Migration Package contains all of the necessary elements ► Create the Migration Log automatically. 12/29/2011 Proprietary and Confidential 17
    18. 18. New Process Ideas• Results of the migration were unknown until further testing or often discovered by the end users ► Find a way to validate migrations to ensure that they were completed correctly. ► Parse the log files and compare the results to the Metadata to immediately determine the accuracy of the migration. ► Determine if the migration was successful IMMEDIATELY and without ambiguity. 12/29/2011 Proprietary and Confidential 18
    19. 19. New Process Ideas• Additional enhancements ► Automatically check the Security of Reports and Documents that were involved in the migration to ensure that their security is properly set compared to the parent folder. ► Export results to HTML to serve as a receipt to attach to Change Control tickets. 12/29/2011 Proprietary and Confidential 19
    20. 20. Demonstration12/29/2011 Proprietary and Confidential 20
    21. 21. Migration PackagesTo expedite the migration process, developers can now create a MigrationPackage by adding all of their desired objects into a single Object Prompt. Step1: Step2: Step3: Create Object Prompt Add all desired objects Save with your name & date in for migration Public ObjectsMigrations 12/29/2011 Proprietary and Confidential 21
    22. 22. Migration Package BuilderSelect yourSource/TargetProjects andMigration Packagefrom the dropdown lists.The Source Projects are a list of all Dev and Int projects from our Development Project Source. The Target Projects are a list of all Int projects andProduction projects from both our Development and Production Project Sources. The Object Prompt list is a list of all Prompts (of type Objectselection) located in the Public ObjectsMigrations folder. 12/29/2011 Proprietary and Confidential 22
    23. 23. Migration Package Builder The tool will tell you if there are dependencies missing from your Migration Package but different in the Target. Review the list, and modify your Migration Package as needed.The tool will scan the contents of the Migration Package and compare to a recursive list of dependencies from the metadata. Any dependencies thatare not included in the Migration Package and do not exist in the Target Project are marked as required. Dependencies that exist with a differentversion ID are noted and optional to include. Checking the box and clicking Next uses the SDK to insert that object into your Object Prompt. 12/29/2011 Proprietary and Confidential 23
    24. 24. Migration Package Builder You will then see a list of items in your Migration Package that are conflicts with the target. Review this list and check the box for any objects that need to be marked as Replace.This list is comprised of objects that are included in the Migration Package but exist differently in the Target. The developer is required to check allboxes, but this screen serves as a sign off that they acknowledge everything that they want to replace. 12/29/2011 Proprietary and Confidential 24
    25. 25. Migration Package BuilderIf there are anyadditional specialhandlinginstructions, pleaselist them in thespace provided. 12/29/2011 Proprietary and Confidential 25
    26. 26. Migration Package BuilderThe final output textshould be copiedand pasted into theRemedy ticket.These instructionswill help expeditethe migrationprocess.The tool generates an overview of the migration and a list of all of the objects for the administrator that need to be marked as replace (including a hintat the object type). The number of new objects is noted, but since individual decisions aren‟t made on these (at least in 8), the count is FYI. 12/29/2011 Proprietary and Confidential 26
    27. 27. Migration Process – Object Audit Tool After the migration is complete, we will run an audit on the results of the migration against the Migration Package. In the event of a discrepancy, we will reconcile the difference with the developer. These could include objects that were moved that were not part of the Migration Package, or objects that were part of the package but not moved.The OMLog is parsed and compared to the metadata to check for errors. Checks include objects moved that weren‟t in the prompt (red), objects thatwere in the prompt but didn‟t get moved (purple) objects that moved as expected (green) and the Migration Package itself (blue). 12/29/2011 Proprietary and Confidential 27
    28. 28. Attach “Receipt” to Remedy Ticket After the migration is verified, we will export a receipt of everything that occurred and attach this report to the Remedy ticket. In addition to this transactional log in the ticket, the server copy will retain all history of all migrations in a single location. The OM Tool also has built in backup functionality so that we can periodically back up migration logs ourselves. Any logs can be independently loaded and viewed in the OM Tool.The receipt contains an html output of the log analysis and is attached in our Change Control system for a complete record of every operation thatoccurred during the migration. Any items that “sneak” over unintended are deleted before the migration is marked as complete. Any errors areimmediately resolved. 12/29/2011 Proprietary and Confidential 28
    29. 29. Migration Process – Object Security AuditOnce complete to production, a Security Audit will be performed to verify thepermissions set on each object. By default, they will inherit the security of theparent folder. The Migration Team will raise a question in the event that there areobvious errors and will reconcile to any special instructions included with theRemedy Ticket.This option checks the security of any Reports in the Migration and compares to their parent folder. This gives the administrator an easy view at whohas access to the reports. Differences are highlighted in Red, but the primary decisions are human based. (Note: After the above migration wasoriginally completed, the group “Finance – Vehicle “ did not have access to the report, so this highlighted that issue and we resolved before anyusers reported not being able to see it.) 12/29/2011 Proprietary and Confidential 29
    30. 30. Overall BenefitHistorically, Ryder averages 3 migrations per week between Dev->Int andInt->ProdOriginal Method Dev->Int Int->ProdDevelopers creating Migration Logs manually 60 15Admin average 7 migration steps (Tables, Facts,Attributes, Metrics, Filters Prompts, Reports) 56 56 8 eachAdmins redo 1/3 of migrations due to error 19 19QA Testing required to best of our ability in 2-Tier 30 30Total (hrs) 11 6 17New Method Dev->Int Int->ProdDevelopers create Migration Logs with MPB 1 1 15 hrs saved per weekAdmin moves Migration Package 8 8 8 eachErrors are eliminated from the process 0 0QA Testing is shorter due to higher confidence 10 10Total (hrs) 1 1 2
    31. 31. Future Plans with 9: Automated Package CreationExtract Change Journaling data to automatically build Migration Logs, furtherreducing the time it takes for a developer to create a migration package. 12/29/2011 Proprietary and Confidential 31
    32. 32. Future Plans with 9: Off Hours Scheduling with Update Packages The final step of the MPB could create an Update Package that can be placed in a monitored folder. A separate process would check this folder nightly and apply the migrations automatically, only alerting administrators in the event of an error. This would greatly minimize administrator involvement and effectively reduce the administrator’s role to approvals instead of tasks. This would also allow backup administrators to easily manage the environment with minimal technical involvement. 12/29/2011 Proprietary and Confidential 32
    33. 33. Future Plans with 9: Automated Integrity Manager TestingAfter the migration iscomplete, IntegrityManager could beexecuted to verify thatthe SQL and datamatch between Testand Prod.IM can be called fromthe command line andthe output stored inHTML and parsed forinstant off hoursalerts. 12/29/2011 Proprietary and Confidential 33
    34. 34. Thank You Questions? Bryan Brandow MicroStrategy Architect Ryder System Inc. January 27th, 201012/29/2011 Proprietary and Confidential 34
    35. 35. Build Your Own Migration Package Toolset• The following slides will map out how to build your own set of tools.• What you’ll need ► Some basic SQL skills (joins, group by, stored procedures) ► Some basic programming skills (read the contents of a file, connect to SQL, basic Web and application GUI) ► No MicroStrategy SDK is required for any portion of these tools as shown in this presentation. 12/29/2011 Proprietary and Confidential 35
    36. 36. Migration Package Builder Select * from dssmdobjinfo where object_type = 32 Select * From dssmdobjinfo where object_type = 10 and subtype = 2566 and parent_id = „GUID of Migrations Folder‟ 10 = prompt 2566 = object prompt 12/29/2011 Proprietary and Confidential 36
    37. 37. Migration Package Builder SQL Code on next slide. 12/29/2011 Proprietary and Confidential 37
    38. 38. SQL CodeThis SQL code shows how to get the contents of an object prompt and determineif there are missing dependencies. Continued on next slide … 12/29/2011 Proprietary and Confidential 38
    39. 39. SQL Code Continued 12/29/2011 Proprietary and Confidential 39
    40. 40. Object Manager Log Tool1. Parse the log file (OMLog.log in 8.x, Diagnostics Logging Tool for 9x).2. Compare object results to the contents of the Migration Package (SQL Code in previous step). 12/29/2011 Proprietary and Confidential 40
    41. 41. Report Security Query from DSSMDOBJSECU table, joining on the object id‟s of the Reports and Documents. 12/29/2011 Proprietary and Confidential 41