Bilir's Business Intelligence Portfolio SSIS Project

1,464 views
1,380 views

Published on

Business Intelligence Portfolio SSIS Project

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,464
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Bilir's Business Intelligence Portfolio SSIS Project

  1. 1. 3/9/2010<br />Figen Bilir ©<br />1<br />
  2. 2. Project Overview: AllWorks<br />The SSIS project was for a fictitious construction company called AllWorks. Project was designed and build a SQL Server 2005 database to track employee and customer information, timesheet and labor rates data, as well as job order information, job materials, and customer invoices. In the client project scenario, AllWorks currently stores this information in Excel Spreadsheets, XML files, and CSV files.<br />Extract data from diverse files (*.csv, *.xls, etc.)<br />Transform data as required by business and SQL database requirements<br />Load data into a local SQL AllWorksDBStudent database<br />3/9/2010<br />Figen Bilir ©<br />2<br />
  3. 3. Database Diagram<br />3/9/2010<br />Figen Bilir ©<br />3<br />
  4. 4. Source Data Review<br />3/9/2010<br />Figen Bilir ©<br />4<br />
  5. 5. Employee Master Package<br />This package loads the employee data from an Excel Spreadsheet, Employees.XLS. Data conversion takes place to verify that the data is in a useable format. A log is created to quantify the number of updates and inserts which are sent to a specific destination. <br />The records are sent to an email recipient who will tell them if the package was a success along with the package name, user who performed action and record results. In the event of a failure of the package during execution an error would be sent to specific address alerting them of the failure.<br />3/9/2010<br />Figen Bilir ©<br />5<br />
  6. 6. Sendmail Setup & Outcome<br />3/9/2010<br />Figen Bilir ©<br />6<br />
  7. 7. Employee Rate Package<br />This package loads the employee rate data from an Excel Spreadsheet, Employees.XLS. Data conversion takes place to verify that the data is in a useable format. All employee records are validated in Lookup task, invalid ones are logged to a CSV file. A log is created to quantify the number of updates, inserts and invalid records which are sent to a specific destination. <br />The records are sent to an email recipient who will tell them if the package was a success along with the package name, user who performed action and record results. In the event of a failure of the package during execution an error would be sent to specific address alerting them of the failure.<br />3/9/2010<br />Figen Bilir ©<br />7<br />
  8. 8. Client Master Package<br />Due to the constraint between dbo.Client and dbo.County tables based on CountyPK as a FK on Clients table, first County table has been populated to the database. That’s why Data Flow for County Data was generated first, and then Data Flow Client Data has been achieved.<br />From the source file for the current package, County Definitions Worksheet within the same file has been utilized in order to populate dbo.County table in the database.<br />3/9/2010<br />Figen Bilir ©<br />8<br />
  9. 9. Client Master Package Cont’d<br />Invalidated CountyIDs has been set up to be written to the Log File above based on check from the County table since it already runs prior to Data Flow Client Data.<br />3/9/2010<br />Figen Bilir ©<br />9<br />
  10. 10. Client Groupings Master Package<br />The Clientgeographies.XLS spreadsheet has been in normalized form, after the data conversion, the incoming data has been aggregated based on groupingno and groupingname.<br />3/9/2010<br />Figen Bilir ©<br />10<br />
  11. 11. Client Groupings to Client Xref Table Package<br />There are 3 lookups to validate the incoming data. First, ClientID is checked to validate the AccountKey, then, GroupingNo is validated in the ClientsGrouping and lastly GroupingID and ClientID is validated against ClientGroupingsXClients table in order to handle to insert in the next step. <br />3/9/2010<br />Figen Bilir ©<br />11<br />
  12. 12. Project Job Master Package<br />Due to the FK relationship between Clients and JobMaster Table, ClientPK has to be validated with a lookup. If there are any invalid Clients, they are written to the CSV Log file specified in Flat File Connection Manager.<br />3/9/2010<br />Figen Bilir ©<br />12<br />
  13. 13. Project Job Time Sheets (Labor) Package<br />3/9/2010<br />Figen Bilir ©<br />13<br />This package loads data from several CSV JobTimeSheet files into SQL Server 2005 database. The content of its Data Flow task is shown on the next two slides.<br />
  14. 14. Project Job Time Sheets (Labor) Package Cont’d<br />In the data flow, data is read from the CSV files each time and EmployeeID and JobMasterID is validated with the Lookups from Employee and JobMaster Tables respectively after data conversion take places.<br />3/9/2010<br />Figen Bilir ©<br />14<br />
  15. 15. Project Job Time Sheets (Labor) Package Cont’d<br />Package reads the TimeSheet data from several CSV files and inserts new rows or update existing row if data is different.<br />Data Flow for Load Job Time Sheet control flow has a Foreach Loop Container that loops through each file and processes the job time sheet files. In order to accumulate the row count a script has been written to for the total row counts and file counts. <br />3/9/2010<br />Figen Bilir ©<br />15<br />
  16. 16. Project Job Time Sheets (Labor) Package Cont’d<br />The script is written to accumulate the total count for given variables in ReadOnly and ReadWrite Variables are used in the definition of the mail message.<br />3/9/2010<br />Figen Bilir ©<br />16<br />
  17. 17. Master Package<br />This is the main package that launches the execution of all ETL packages from SQL Server 2005.<br />After the successful execution of all ETL packages, it launches the database maintenance tasks starting with the database shrinking, indexes building, statistics update and database backup task.<br />Upon completion the whole package a notification of the successful email is sent. If any maintenance task fails, “Unsuccessful Email” is sent from each task.<br />3/9/2010<br />Figen Bilir ©<br />17<br />
  18. 18. SQL Server Agent Job<br />All packages were deployed to the (local) SQL Server and a job -Execute SSIS Student Project- in SQL Server Agent was set up to run this Master Package nightly at 12:00AM.<br />3/9/2010<br />Figen Bilir ©<br />18<br />

×