Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
ETL Stuck WorkFlow Monitor




Presentation Title
Presentation Date            Page
ETL Stuck WorkFlow Monitor
                     ● Premis:




Presentation Title
Presentation Date                Page
ETL Stuck WorkFlow Monitor
                     ● Premis:
                       Every ETL process has distinct PID,
    ...
ETL Stuck WorkFlow Monitor
                     ● Premis:
                       Every ETL process has distinct PID,
    ...
ETL Stuck WorkFlow Monitor
                     ● Premis:
                       Every ETL process has distinct PID,
    ...
ETL Stuck WorkFlow Monitor
                     ● Premis:
                       Every ETL process has distinct PID,
    ...
ETL Stuck WorkFlow Monitor
                     ● Premis:
                       Every ETL process has distinct PID,
    ...
ETL Stuck WorkFlow Monitor
● Context Diagram




Presentation Title
Presentation Date            Page
ETL Stuck WorkFlow Monitor
● Diagram 0




Presentation Title
Presentation Date            Page
ETL Stuck WorkFlow Monitor
● Flow Chart




Presentation Title
Presentation Date            Page
ETL Stuck WorkFlow Monitor




Presentation Title
Presentation Date            Page
ETL Stuck WorkFlow Monitor
● Pseudocode




Presentation Title
Presentation Date            Page
ETL Stuck WorkFlow Monitor
● Pseudocode
             Fetch process info for current ETL processes:




Presentation Title
...
ETL Stuck WorkFlow Monitor
● Pseudocode
             Fetch process info for current ETL processes:
               Output P...
ETL Stuck WorkFlow Monitor
● Pseudocode
             Fetch process info for current ETL processes:
               Output P...
ETL Stuck WorkFlow Monitor
● Pseudocode
             Fetch process info for current ETL processes:
               Output P...
ETL Stuck WorkFlow Monitor
● Pseudocode
             Fetch process info for current ETL processes:
               Output P...
ETL Stuck WorkFlow Monitor
● Pseudocode
             Fetch process info for current ETL processes:
               Output P...
ETL Stuck WorkFlow Monitor
● Pseudocode
             Fetch process info for current ETL processes:
               Output P...
ETL Stuck WorkFlow Monitor
● Pseudocode
             Fetch process info for current ETL processes:
               Output P...
ETL Stuck WorkFlow Monitor
● Pseudocode
             Fetch process info for current ETL processes:
               Output P...
ETL Stuck WorkFlow Monitor
● Pseudocode
             Fetch process info for current ETL processes:
               Output P...
ETL Stuck WorkFlow Monitor
● Pseudocode
             Fetch process info for current ETL processes:
               Output P...
ETL Stuck WorkFlow Monitor
● Pseudocode
             Fetch process info for current ETL processes:
               Output P...
ETL Stuck WorkFlow Monitor
● Pseudocode
             Fetch process info for current ETL processes:
               Output P...
ETL Stuck WorkFlow Monitor
● Pseudocode
             Fetch process info for current ETL processes:
               Output P...
ETL Stuck WorkFlow Monitor
● Pseudocode
             Fetch process info for current ETL processes:
               Output P...
ETL Stuck WorkFlow Monitor
● Pseudocode
             Fetch process info for current ETL processes:
               Output P...
ETL Stuck WorkFlow Monitor
● Pseudocode
             Fetch process info for current ETL processes:
               Output P...
ETL Stuck WorkFlow Monitor
● Pseudocode
             Fetch process info for current ETL processes:
               Output P...
ETL Stuck WorkFlow Monitor
● Pseudocode
             Fetch process info for current ETL processes:
               Output P...
ETL Stuck WorkFlow Monitor
● Pseudocode
             Fetch process info for current ETL processes:
               Output P...
ETL Stuck WorkFlow Monitor
● Pseudocode
             Fetch process info for current ETL processes:
               Output P...
ETL Stuck WorkFlow Monitor
● Example outputs
             Previous -
             14739 15:20:01 1:21 wf_Delete_Dim.s_m_De...
ETL Stuck WorkFlow Monitor


      Next steps
                                                                           ...
Upcoming SlideShare
Loading in …5
×

Generic Etl Monitor Review

828 views

Published on

  • Be the first to comment

  • Be the first to like this

Generic Etl Monitor Review

  1. 1. ETL Stuck WorkFlow Monitor Presentation Title Presentation Date Page
  2. 2. ETL Stuck WorkFlow Monitor ● Premis: Presentation Title Presentation Date Page
  3. 3. ETL Stuck WorkFlow Monitor ● Premis: Every ETL process has distinct PID, description, & info Presentation Title Presentation Date Page
  4. 4. ETL Stuck WorkFlow Monitor ● Premis: Every ETL process has distinct PID, description, & info Info includes process start time & cumulative CPU time Presentation Title Presentation Date Page
  5. 5. ETL Stuck WorkFlow Monitor ● Premis: Every ETL process has distinct PID, description, & info Info includes process start time & cumulative CPU time Snapshots can be taken at regular intervals and saved Presentation Title Presentation Date Page
  6. 6. ETL Stuck WorkFlow Monitor ● Premis: Every ETL process has distinct PID, description, & info Info includes process start time & cumulative CPU time Snapshots can be taken at regular intervals and saved Compare “Current” v. “Previous” Presentation Title Presentation Date Page
  7. 7. ETL Stuck WorkFlow Monitor ● Premis: Every ETL process has distinct PID, description, & info Info includes process start time & cumulative CPU time Snapshots can be taken at regular intervals and saved Compare “Current” v. “Previous” Use simple Unix comm function Presentation Title Presentation Date Page
  8. 8. ETL Stuck WorkFlow Monitor ● Context Diagram Presentation Title Presentation Date Page
  9. 9. ETL Stuck WorkFlow Monitor ● Diagram 0 Presentation Title Presentation Date Page
  10. 10. ETL Stuck WorkFlow Monitor ● Flow Chart Presentation Title Presentation Date Page
  11. 11. ETL Stuck WorkFlow Monitor Presentation Title Presentation Date Page
  12. 12. ETL Stuck WorkFlow Monitor ● Pseudocode Presentation Title Presentation Date Page
  13. 13. ETL Stuck WorkFlow Monitor ● Pseudocode Fetch process info for current ETL processes: Presentation Title Presentation Date Page
  14. 14. ETL Stuck WorkFlow Monitor ● Pseudocode Fetch process info for current ETL processes: Output PID, start time, CPU time, sub-string(process name, 13) Presentation Title Presentation Date Page
  15. 15. ETL Stuck WorkFlow Monitor ● Pseudocode Fetch process info for current ETL processes: Output PID, start time, CPU time, sub-string(process name, 13) Sort the results Presentation Title Presentation Date Page
  16. 16. ETL Stuck WorkFlow Monitor ● Pseudocode Fetch process info for current ETL processes: Output PID, start time, CPU time, sub-string(process name, 13) Sort the results Place results into current_wf_processes.txt Presentation Title Presentation Date Page
  17. 17. ETL Stuck WorkFlow Monitor ● Pseudocode Fetch process info for current ETL processes: Output PID, start time, CPU time, sub-string(process name, 13) Sort the results Place results into current_wf_processes.txt Presentation Title Presentation Date Page
  18. 18. ETL Stuck WorkFlow Monitor ● Pseudocode Fetch process info for current ETL processes: Output PID, start time, CPU time, sub-string(process name, 13) Sort the results Place results into current_wf_processes.txt If previous_wf_processes.txt exists Presentation Title Presentation Date Page
  19. 19. ETL Stuck WorkFlow Monitor ● Pseudocode Fetch process info for current ETL processes: Output PID, start time, CPU time, sub-string(process name, 13) Sort the results Place results into current_wf_processes.txt If previous_wf_processes.txt exists Then Presentation Title Presentation Date Page
  20. 20. ETL Stuck WorkFlow Monitor ● Pseudocode Fetch process info for current ETL processes: Output PID, start time, CPU time, sub-string(process name, 13) Sort the results Place results into current_wf_processes.txt If previous_wf_processes.txt exists Then Compare current… with previous… files Presentation Title Presentation Date Page
  21. 21. ETL Stuck WorkFlow Monitor ● Pseudocode Fetch process info for current ETL processes: Output PID, start time, CPU time, sub-string(process name, 13) Sort the results Place results into current_wf_processes.txt If previous_wf_processes.txt exists Then Compare current… with previous… files Place results in wf_process_compare.txt Presentation Title Presentation Date Page
  22. 22. ETL Stuck WorkFlow Monitor ● Pseudocode Fetch process info for current ETL processes: Output PID, start time, CPU time, sub-string(process name, 13) Sort the results Place results into current_wf_processes.txt If previous_wf_processes.txt exists Then Compare current… with previous… files Place results in wf_process_compare.txt If …compare file is NULL Presentation Title Presentation Date Page
  23. 23. ETL Stuck WorkFlow Monitor ● Pseudocode Fetch process info for current ETL processes: Output PID, start time, CPU time, sub-string(process name, 13) Sort the results Place results into current_wf_processes.txt If previous_wf_processes.txt exists Then Compare current… with previous… files Place results in wf_process_compare.txt If …compare file is NULL Then Presentation Title Presentation Date Page
  24. 24. ETL Stuck WorkFlow Monitor ● Pseudocode Fetch process info for current ETL processes: Output PID, start time, CPU time, sub-string(process name, 13) Sort the results Place results into current_wf_processes.txt If previous_wf_processes.txt exists Then Compare current… with previous… files Place results in wf_process_compare.txt If …compare file is NULL Then If previous… and current… are BOTH NULL Presentation Title Presentation Date Page
  25. 25. ETL Stuck WorkFlow Monitor ● Pseudocode Fetch process info for current ETL processes: Output PID, start time, CPU time, sub-string(process name, 13) Sort the results Place results into current_wf_processes.txt If previous_wf_processes.txt exists Then Compare current… with previous… files Place results in wf_process_compare.txt If …compare file is NULL Then If previous… and current… are BOTH NULL Then Presentation Title Presentation Date Page
  26. 26. ETL Stuck WorkFlow Monitor ● Pseudocode Fetch process info for current ETL processes: Output PID, start time, CPU time, sub-string(process name, 13) Sort the results Place results into current_wf_processes.txt If previous_wf_processes.txt exists Then Compare current… with previous… files Place results in wf_process_compare.txt If …compare file is NULL Then If previous… and current… are BOTH NULL Then Send alert “ETL May be down” to WorkFlow support Presentation Title Presentation Date Page
  27. 27. ETL Stuck WorkFlow Monitor ● Pseudocode Fetch process info for current ETL processes: Output PID, start time, CPU time, sub-string(process name, 13) Sort the results Place results into current_wf_processes.txt If previous_wf_processes.txt exists Then Compare current… with previous… files Place results in wf_process_compare.txt If …compare file is NULL Then If previous… and current… are BOTH NULL Then Send alert “ETL May be down” to WorkFlow support Exit Presentation Title Presentation Date Page
  28. 28. ETL Stuck WorkFlow Monitor ● Pseudocode Fetch process info for current ETL processes: Output PID, start time, CPU time, sub-string(process name, 13) Sort the results Place results into current_wf_processes.txt If previous_wf_processes.txt exists Then Compare current… with previous… files Place results in wf_process_compare.txt If …compare file is NULL Then If previous… and current… are BOTH NULL Then Send alert “ETL May be down” to WorkFlow support Exit Format alerts messages Presentation Title Presentation Date Page
  29. 29. ETL Stuck WorkFlow Monitor ● Pseudocode Fetch process info for current ETL processes: Output PID, start time, CPU time, sub-string(process name, 13) Sort the results Place results into current_wf_processes.txt If previous_wf_processes.txt exists Then Compare current… with previous… files Place results in wf_process_compare.txt If …compare file is NULL Then If previous… and current… are BOTH NULL Then Send alert “ETL May be down” to WorkFlow support Exit Format alerts messages Send pages and emails to WorkFlow support Presentation Title Presentation Date Page
  30. 30. ETL Stuck WorkFlow Monitor ● Pseudocode Fetch process info for current ETL processes: Output PID, start time, CPU time, sub-string(process name, 13) Sort the results Place results into current_wf_processes.txt If previous_wf_processes.txt exists Then Compare current… with previous… files Place results in wf_process_compare.txt If …compare file is NULL Then If previous… and current… are BOTH NULL Then Send alert “ETL May be down” to WorkFlow support Exit Format alerts messages Send pages and emails to WorkFlow support Format log info Presentation Title Presentation Date Page
  31. 31. ETL Stuck WorkFlow Monitor ● Pseudocode Fetch process info for current ETL processes: Output PID, start time, CPU time, sub-string(process name, 13) Sort the results Place results into current_wf_processes.txt If previous_wf_processes.txt exists Then Compare current… with previous… files Place results in wf_process_compare.txt If …compare file is NULL Then If previous… and current… are BOTH NULL Then Send alert “ETL May be down” to WorkFlow support Exit Format alerts messages Send pages and emails to WorkFlow support Format log info Write info to log Presentation Title Presentation Date Page
  32. 32. ETL Stuck WorkFlow Monitor ● Pseudocode Fetch process info for current ETL processes: Output PID, start time, CPU time, sub-string(process name, 13) Sort the results Place results into current_wf_processes.txt If previous_wf_processes.txt exists Then Compare current… with previous… files Place results in wf_process_compare.txt If …compare file is NULL Then If previous… and current… are BOTH NULL Then Send alert “ETL May be down” to WorkFlow support Exit Format alerts messages Send pages and emails to WorkFlow support Format log info Write info to log Copy current… into previous… Presentation Title Presentation Date Page
  33. 33. ETL Stuck WorkFlow Monitor ● Pseudocode Fetch process info for current ETL processes: Output PID, start time, CPU time, sub-string(process name, 13) Sort the results Place results into current_wf_processes.txt If previous_wf_processes.txt exists Then Compare current… with previous… files Place results in wf_process_compare.txt If …compare file is NULL Then If previous… and current… are BOTH NULL Then Send alert “ETL May be down” to WorkFlow support Exit Format alerts messages Send pages and emails to WorkFlow support Format log info Write info to log Copy current… into previous… Exit Presentation Title Presentation Date Page
  34. 34. ETL Stuck WorkFlow Monitor ● Example outputs Previous - 14739 15:20:01 1:21 wf_Delete_Dim.s_m_Delete_dim 16944 15:20:13 0:23 wf_COM_PRE.s_m_COM_PRE 17296 15:20:23 3:14 wf_ATTR_STG.s_m_ATTR_STG 18070 15:15:26 3:19 wf_SLA_STG.s_m_SLA_STG 19105 15:11:19 2:20 wf_DR_ATTR_DIM.s_m_HDA_ATTR_STG1 23059 15:21:24 0:03 wf_COM_HA.s_m_COM_HA 29936 15:23:52 0:16 wf_AB_SL_FACT.s_m_AB_SL_FACT 3159 15:18:13 2:50 wf_H_FACT.s_m_H_FACT Current - 10457 15:25:15 0:29 wf_COM_SUM.s_m_COM_SUM 11622 15:25:51 0:01 wf_WIN_SUM.s_m_WIN_SUM 14739 15:20:01 2:08 wf_Delete_Dim.s_m_Delete_dim 17296 15:20:23 3:49 wf_ATTR_STG.s_m_ATTR_STG 18070 15:15:26 3:19 wf_SLA_STG.s_m_SLA_STG 19105 15:11:19 2:50 wf_DR_ATTR_DIM.s_m_HDA_ATTR_STG1 23059 15:21:24 1:03 wf_COM_HA.s_m_COM_HA 29936 15:23:52 0:29 wf_AB_SL_FACT.s_m_AB_SL_FACT 3159 15:18:13 2:50 wf_H_FACT.s_m_H_FACT 7034 15:25:00 0:03 wf_FL_BR_STG.s_m_FL_BR_STG comm -12 current_wf_processes.txt previous_wf_processes.txt Result – 18070 15:15:26 3:19 wf_SLA_STG.s_m_SLA_STG 3159 15:18:13 2:50 wf_H_FACT.s_m_H_FACT Presentation Title Presentation Date Page
  35. 35. ETL Stuck WorkFlow Monitor Next steps 1  Coding ~ 5 hours 1  Testing ~ 9 hours 2  Documentation ~ 20 hours 1 - Rule of thumb, 4 x more that you would think it would take + one 2 - Rule of thumb, 2 x more that you would think it would take + 50% Presentation Title Presentation Date Page

×