The document discusses the need for institutions to have a data management strategy. It notes the challenges of integrating data across different systems used by institutions for student information, learning, housing, and other functions. The document recommends adopting an enterprise data integration platform to standardize how data is accessed and moved between systems. It provides an example of how the Talend platform can be used to integrate weather data from the NOAA API with data in a database, demonstrating the platform's functionality and ease of use. The document concludes that data integration is key to a successful data management strategy.

1. Why Your Institution Needs a Data
Management Strategy
Brad Bronsch
Data Architect
Eastern Washington University
Building Bridges
April 13, 2016
Spokane, WA

2. ...or How I Learned to Stop Worrying
and Love Data Integration...

3. Introduction

4. Introduction
About Our Group
❖ Belong to the Business Intelligence group in IT; one manager, 3 report
developers, 3 for data architecture, integration, maintenance
About Me
❖ From spreadsheets to data warehouses
❖ Background in Financial, Retail, Utility & Education sectors with exposure
to a wide variety of data models & how to get data in and out
❖ Role as “Data Evangelist” - a Data Architect’s job is fifty percent technology
and fifty percent communication.

5. Introduction
❖ Premise: An institution’s success is either hampered or realized by the the
lack of or availability of accurate, timely information. Without a
comprehensive, enterprise approach to data management, it’s difficult to
meet this need.
❖ Discussion: A synopsis of the challenges of our educational environment
will be presented, where educational services and technology are heading
and how we’re dealing with it at EWU from a data management & data
integration perspective.
❖ Disclaimer....

6. State of the Union

7. State Of The Union
There’s a reason they call it…
...IT...
...Information Technology…

8. State Of The Union
At a high level there are two types of information IT has to contend with:
❖ Operational Data - Application Data
❖ Strategic Data - Business Intelligence

9. State Of The Union
The Student Information System (SIS)
❖ The SIS is the Enterprise Resource Planning (ERP) system for Education
❖ Ellucian - Banner & Colleague
❖ Peoplesoft - Campus Solutions
❖ THE systems of record….or are they?
❖ How many institutions get all they need from their Student Information
System?

10. State Of The Union
Ancillary Systems at EWU
❖ Learning Management System (LMS) - Canvas
❖ Customer Relationship Management (CRM) - Hobson’s Radius
❖ Degree Audit & Academic Planning - u.achieve/u.direct
❖ Content Management System (CMS) - Word Press, Ingenuix, SharePoint(?)
❖ Student Housing - StarRez
❖ Facilities Management - AiM
❖ And the list goes on…
❖ Each may be the system of record of information...

11. State Of The Union
External & Internal Data Feeds
❖ Federal & State Reporting
❖ Other third parties
❖ Transmission via internal file share, Secure File Transfer Protocol (sFTP)
or manual upload via website.
Business Intelligence
❖ Reporting Platforms - Jaspersoft, Oracle Discoverer, SQL Server Reporting
Services.
❖ Primary Data Source - Banner Operational Data Store (ODS).

12. State Of The Union
Support Challenges
❖ We have about 10 tech analysts
❖ We’re often one deep with a single tech analyst supporting a
department and one or more ancillary system.
❖ Each of these tech analysts wear several hats; business analyst,
application support, application administration…. and system (data)
integration.
❖ Question for HGTV addicts...would your framer, plumber and electrician
typically all be the same person?
❖ We have the DIY model versus a general contractor model.

13. State Of The Union
You’re a data architect, why aren’t you
off building a data warehouse, instead
of pestering me about data integration
and my system…?

14. State Of The Union
Data Warehouse Challenges
❖ Access to data in those ancillary systems (data islands).
❖ Understanding of how data is structured in those systems.
❖ Lack of standardization in data access (data integration).
❖ Each system may or may not have some sort of API (application
programming interface) for data integration provided by the vendor.
❖ Direct database access? The database implementation - Oracle, SQL
Server, MySQL?
❖ Web services?

15. State Of The Union
Case in Point
Canvas
❖ Requires mission critical data feed from Banner, our SIS.
❖ Data feed produced and transmitted using Oracle PL/SQL, a Linux Bash
shell script to execute the SQL script & then make a command line CURL
call to a Canvas web service to push the the files to Canvas.
❖ Author of the process is no longer with us which makes supporting it
challenging.
Just one example...many others.

16. Data Integration

17. State Of The Union
Enterprise data integration is the backbone of a good data management
strategy. A data integration platform...
❖ Provides standardization - a single approach to across the organization
and all systems instead of multiple services and languages cobbled
together.
❖ Is maintainable, extensible, scalable and most importantly - supportable.
❖ Can be monitored for success, failure and provides job statistics.
❖ Provides built in notifications for communication of success or failure.

18. Data Integration
Choosing a Platform
❖ Consider a platform that is database independent and avoid vendor
specific platforms. Microsoft, Oracle, IBM all have their own data
integration tools, but vary in how well they integrate with others.
❖ Avoid platforms that are specific to a single system or business sector. In
other words, don’t pick a platform that is specific to Education.

19. Data Integration
Why we chose Talend
❖ High functionality - integrates well with any data source or target; any
flavor of database, any file type, web service, FTP, LDAP (Active Directory),
etc.
❖ Cost - highest amount of functionality for the dollar.
❖ Open Source Based - in addition to excellent vendor documentation, there
is a wealth of information available in user forums, developer websites.

20. Data Integration
Case Study - Retention & Student Success
❖ Potentially requires integration of information from multiple platforms.
❖ Student Data - Banner (a given).
❖ Housing Data - StarRez (how does a student’s living situation affect
success?)
❖ Admissions Data - Hobson’s Radius (was there something about the
admissions, enrollment, registration process that adversely affected the
student’s experience?)

21. Data Integration
How About Weather Data?
❖ Does the average daily temperature affect student success?
❖ Probably not, but it allows me to demonstrate data integration without
violating FERPA or HIPAA restrictions…
❖ Also, demonstrates the trend of systems towards de-centralization
(challenging) and the good news (cloud-based).

22. Data Integration
Weather Data from NOAA
❖ National Oceanic & Atmospheric Association
❖ Cloud-based, API is a REST-based web service & very well documented
❖ The existence of an api & documentation are two things you should
seriously consider when choosing 3rd party applications. Often vendors
don’t consider you might actually want to get your data out of their
systems, and if they do consider it, they like to charge you for it.
❖ Using Community (Open Source) versions of Talend and other tools for
this demo.

23. Data Integration
NOAA Data: https://www.ncdc.noaa.gov/cdo-web/webservices/v2#gettingStarted

24. Data Integration
NOAA Data - Method: Browser Plug-In (personal token redacted)

25. Data Integration
NOAA Data (Method 1): JSON unformatted data in a text editor (Notepad++)

26. Data Integration
NOAA Data - Method 1: JSON formatted data

27. Data Integration
NOAA Data - Method 2: Using Excel with Power Query

28. Data Integration
NOAA Data - Method 2: Using Excel with Power Query
Power Query How-To: http://blog.crossjoin.co.uk/2014/03/26/working-with-web-services-in-power-query/

29. Data Integration
NOAA Data - Method 3: Using the Talend Platform, Job Design & Execution

30. Data Integration
NOAA Data - Method 3: Using the Talend Platform, Data Mapping

31. Data Integration
NOAA Data - Method 3: Using the Talend Platform, Results in Database

32. Data Integration
NOAA Data - Method 3: Using the Talend Platform, Administration

33. Conclusion

34. Conclusion
Session Summary
❖ Regardless of the systems you buy or build, you need a strategy for
efficiently moving information between those systems. Enterprise Data
Integration is the core of a good Data Management Strategy.

35. Conclusion
Brad Bronsch
Data Architect
Business Intelligence
Email: bbronsch@ewu.edu
Phone: (509) 359-6163