CERN Summer Student 2015 project for automated incremental data transfer from Oracle to Hadoop using Sqoop

2. HLoader
Data Ingestion
from Oracle Databases
to Hadoop Clusters
Automatically
On-Demand
8/13/2015 HLoader – A. Bose, D. Stein 2
HL

3. Problem
– Control and monitor data transfer
using Sqoop, a CLI tool for bulk data transfer
– Two in one
two distinct Summer Student task proposals for basically
the same job
8/13/2015 HLoader – A. Bose, D. Stein 3

4. Problem
– Frequent requests
different users with different but similar use cases
ATLAS Job Monitoring, CMS Job Monitoring, CMS data popularity, ACCLOG
– Manually executed job
that can be partially automated
8/13/2015 HLoader – A. Bose, D. Stein 4

5. Requirements
– Run jobs…
… incrementally
… communicate with
the end user
– Handle failures
retry, notify, prevent
– Be secure, stay safe
authorize, authenticate the users
without exchanging passwords
– Use what’s provided
Run on the CERN-provided
infrastructure
8/13/2015 HLoader – A. Bose, D. Stein 5

6. Client
Meta DB
REST API Agent
Oracle Databases
FIM
Hadoop Clusters
…
Solution overview
8/13/2015 HLoader – A. Bose, D. Stein 6
1. Provided infrastructure
Oracle Databases and Hadoop
Clusters
2. Transfer Data
the user wants to transfer
data, so they create a new job:
what, when, where to transfer
3. Execute the transfer on
behalf of the user
schedule and execute the job
at the requested time (also
inform the user of the status)
4. Update if needed
if the user requested
incremental updates, schedule
it after the given interval

7. Client
Meta DB
REST API Agent
Oracle Databases
FIM
Hadoop Clusters
…
Solution security
8/13/2015 HLoader – A. Bose, D. Stein 7
1. CERN SSO
authentication
no password exchange
2. Authorization
only available (ownership) and
enabled (configured) Oracle
servers could be used
3. Kerberos SSH
tunneling
separate user to log in to the
clusters, without password
4. Secure password input
other users can not see the
password as plaintext
anywhere

8. Client
Meta DB
REST API Agent
Oracle Databases
FIM
Hadoop Clusters
…
Solution security
8/13/2015 HLoader – A. Bose, D. Stein 7
1. CERN SSO
authentication
no password exchange
2. Authorization
only available (ownership) and
enabled (configured) Oracle
servers could be used
3. Kerberos SSH
tunneling
separate user to log in to the
clusters, without password
4. Secure password input
other users can not see the
password as plaintext
anywhere
1

9. Client
Meta DB
REST API Agent
Oracle Databases
FIM
Hadoop Clusters
…
Solution security
8/13/2015 HLoader – A. Bose, D. Stein 7
1. CERN SSO
authentication
no password exchange
2. Authorization
only available (ownership) and
enabled (configured) Oracle
servers could be used
3. Kerberos SSH
tunneling
separate user to log in to the
clusters, without password
4. Secure password input
other users can not see the
password as plaintext
anywhere
2

10. Client
Meta DB
REST API Agent
Oracle Databases
FIM
Hadoop Clusters
…
Solution security
8/13/2015 HLoader – A. Bose, D. Stein 7
1. CERN SSO
authentication
no password exchange
2. Authorization
only available (ownership) and
enabled (configured) Oracle
servers could be used
3. Kerberos SSH
tunneling
separate user to log in to the
clusters, without password
4. Secure password input
other users can not see the
password as plaintext
anywhere
3

11. Client
Meta DB
REST API Agent
Oracle Databases
FIM
Hadoop Clusters
…
Solution security
8/13/2015 HLoader – A. Bose, D. Stein 7
1. CERN SSO
authentication
no password exchange
2. Authorization
only available (ownership) and
enabled (configured) Oracle
servers could be used
3. Kerberos SSH
tunneling
separate user to log in to the
clusters, without password
4. Secure password input
other users can not see the
password as plaintext
anywhere
4
4

12. Client
Meta DB
REST API Agent
Oracle Databases
FIM
Hadoop Clusters
…
Solution security
8/13/2015 HLoader – A. Bose, D. Stein 7
1. CERN SSO
authentication
no password exchange
2. Authorization
only available (ownership) and
enabled (configured) Oracle
servers could be used
3. Kerberos SSH
tunneling
separate user to log in to the
clusters, without password
4. Secure password input
other users can not see the
password as plaintext
anywhere
1
2 3
4
4

13. Client
Meta DB
REST API Agent
Oracle Databases
FIM
Hadoop Clusters
…
Solution modularity
8/13/2015 HLoader – A. Bose, D. Stein 8
1. DB connector agnostic
SQLAlchemy supports several
dialects, also other connectors
can be integrated
2. Interchangeable
scheduler
based on the servers and the
needed schedule complexity
3. Flexible communication
with Hadoop
besides commands through SSH,
Oozie could also be used
4. Client communicating
using REST API
5. Changeable Sqoop JDBC
driver
normal or fast connectors if
possible

14. Client
Meta DB
REST API Agent
Oracle Databases
FIM
Hadoop Clusters
…
Solution modularity
8/13/2015 HLoader – A. Bose, D. Stein 8
1. DB connector agnostic
SQLAlchemy supports several
dialects, also other connectors
can be integrated
2. Interchangeable
scheduler
based on the servers and the
needed schedule complexity
3. Flexible communication
with Hadoop
besides commands through SSH,
Oozie could also be used
4. Client communicating
using REST API
5. Changeable Sqoop JDBC
driver
normal or fast connectors if
possible
1

15. Client
Meta DB
REST API Agent
Oracle Databases
FIM
Hadoop Clusters
…
Solution modularity
8/13/2015 HLoader – A. Bose, D. Stein 8
1. DB connector agnostic
SQLAlchemy supports several
dialects, also other connectors
can be integrated
2. Interchangeable
scheduler
based on the servers and the
needed schedule complexity
3. Flexible communication
with Hadoop
besides commands through SSH,
Oozie could also be used
4. Client communicating
using REST API
5. Changeable Sqoop JDBC
driver
normal or fast connectors if
possible
2

16. Client
Meta DB
REST API Agent
Oracle Databases
FIM
Hadoop Clusters
…
Solution modularity
8/13/2015 HLoader – A. Bose, D. Stein 8
1. DB connector agnostic
SQLAlchemy supports several
dialects, also other connectors
can be integrated
2. Interchangeable
scheduler
based on the servers and the
needed schedule complexity
3. Flexible communication
with Hadoop
besides commands through SSH,
Oozie could also be used
4. Client communicating
using REST API
5. Changeable Sqoop JDBC
driver
normal or fast connectors if
possible
3

17. Client
Meta DB
REST API Agent
Oracle Databases
FIM
Hadoop Clusters
…
Solution modularity
8/13/2015 HLoader – A. Bose, D. Stein 8
1. DB connector agnostic
SQLAlchemy supports several
dialects, also other connectors
can be integrated
2. Interchangeable
scheduler
based on the servers and the
needed schedule complexity
3. Flexible communication
with Hadoop
besides commands through SSH,
Oozie could also be used
4. Client communicating
using REST API
5. Changeable Sqoop JDBC
driver
normal or fast connectors if
possible
4

18. Client
Meta DB
REST API Agent
Oracle Databases
FIM
Hadoop Clusters
…
Solution modularity
8/13/2015 HLoader – A. Bose, D. Stein 8
1. DB connector agnostic
SQLAlchemy supports several
dialects, also other connectors
can be integrated
2. Interchangeable
scheduler
based on the servers and the
needed schedule complexity
3. Flexible communication
with Hadoop
besides commands through SSH,
Oozie could also be used
4. Client communicating
using REST API
5. Changeable Sqoop JDBC
driver
normal or fast connectors if
possible
5

19. Client
Meta DB
REST API Agent
Oracle Databases
FIM
Hadoop Clusters
…
Solution modularity
8/13/2015 HLoader – A. Bose, D. Stein 8
1. DB connector agnostic
SQLAlchemy supports several
dialects, also other connectors
can be integrated
2. Interchangeable
scheduler
based on the servers and the
needed schedule complexity
3. Flexible communication
with Hadoop
besides commands through SSH,
Oozie could also be used
4. Client communicating
using REST API
5. Changeable Sqoop JDBC
driver
normal or fast connectors if
possible
1
2
3
4
5

20. Client
Meta DB
REST API Agent
Oracle Databases
FIM
Hadoop Clusters
…
Solution infrastructure
8/13/2015 HLoader – A. Bose, D. Stein 9
1. PostgreSQL On-Demand
with Postgre and SQLAlchemy
connector
2. Central WebServices
DFS | Windows > IIS 8.5 >
FastCGI > Python 2.7 > Flask
3. Agent running separated
on DB locally managed server,
OpenStack or WebServices (TBD)
4. Client hosted with the
REST API
for easy usage and update, could
be separate

21. Client
Meta DB
REST API Agent
Oracle Databases
FIM
Hadoop Clusters
…
Solution infrastructure
8/13/2015 HLoader – A. Bose, D. Stein 9
1. PostgreSQL On-Demand
with Postgre and SQLAlchemy
connector
2. Central WebServices
DFS | Windows > IIS 8.5 >
FastCGI > Python 2.7 > Flask
3. Agent running separated
on DB locally managed server,
OpenStack or WebServices (TBD)
4. Client hosted with the
REST API
for easy usage and update, could
be separate
1

22. Client
Meta DB
REST API Agent
Oracle Databases
FIM
Hadoop Clusters
…
Solution infrastructure
8/13/2015 HLoader – A. Bose, D. Stein 9
1. PostgreSQL On-Demand
with Postgre and SQLAlchemy
connector
2. Central WebServices
DFS | Windows > IIS 8.5 >
FastCGI > Python 2.7 > Flask
3. Agent running separated
on DB locally managed server,
OpenStack or WebServices (TBD)
4. Client hosted with the
REST API
for easy usage and update, could
be separate
2

23. Client
Meta DB
REST API Agent
Oracle Databases
FIM
Hadoop Clusters
…
Solution infrastructure
8/13/2015 HLoader – A. Bose, D. Stein 9
1. PostgreSQL On-Demand
with Postgre and SQLAlchemy
connector
2. Central WebServices
DFS | Windows > IIS 8.5 >
FastCGI > Python 2.7 > Flask
3. Agent running separated
on DB locally managed server,
OpenStack or WebServices (TBD)
4. Client hosted with the
REST API
for easy usage and update, could
be separate
3

24. Client
Meta DB
REST API Agent
Oracle Databases
FIM
Hadoop Clusters
…
Solution infrastructure
8/13/2015 HLoader – A. Bose, D. Stein 9
1. PostgreSQL On-Demand
with Postgre and SQLAlchemy
connector
2. Central WebServices
DFS | Windows > IIS 8.5 >
FastCGI > Python 2.7 > Flask
3. Agent running separated
on DB locally managed server,
OpenStack or WebServices (TBD)
4. Client hosted with the
REST API
for easy usage and update, could
be separate
4

25. Client
Meta DB
REST API Agent
Oracle Databases
FIM
Hadoop Clusters
…
Solution infrastructure
8/13/2015 HLoader – A. Bose, D. Stein 9
1. PostgreSQL On-Demand
with Postgre and SQLAlchemy
connector
2. Central WebServices
DFS | Windows > IIS 8.5 >
FastCGI > Python 2.7 > Flask
3. Agent running separated
on DB locally managed server,
OpenStack or WebServices (TBD)
4. Client hosted with the
REST API
for easy usage and update, could
be separate
1
2 3
4

26. Solution meta DB
8/13/2015 HLoader – A. Bose, D. Stein 10
HL_SERVERS
HL_CLUSTERS
HL_JOBS
HL_TRANSFERS
HL_LOGS
server_idPK
server_address
server_name
cluster_idPK
cluster_address
cluster_name
job_idPK
source_server_idFK
source_schema_name
source_object_name
destination_cluster_idFK
destination_path
owner_username
sqoop_nmap
sqoop_splitting_column
sqoop_incremental_meth
od
sqoop_direct
start_time
interval
job_last_update
transfer_idPK
scheduler_transfer_id
job_idFK log_idPK
transfer_idFK
log_source
transfer_status
transfer_start
transfer_last_update
last_modified_value
log_path
log_content

27. Solution restrictions
8/13/2015 HLoader – A. Bose, D. Stein 11
– Only allow tables and views to be imported
the DB is responsible for evaluating and checking the queries
– Selected (preconfigured) source databases
gradual introduction for new users
– Preset destination folder structure
with restricted access rights, avoiding collision, unauthorized access
– Basic Sqoop command logic (for now)
eg., with primary key, only one PK attribute

28. Client
Meta DB
REST API Agent
Oracle Databases
FIM
Hadoop Clusters
…
Solution current state
8/13/2015 HLoader – A. Bose, D. Stein 12
1. Client Kate
in progress, meanwhile the
REST interface can be used
2. REST API Daniel
almost ready, missing the new
job processing interface
3. Agent Scheduling Ani
basically ready, can schedule
jobs and update itself after job
description modifications
4. Agent Runners Daniel
working for initial imports,
soon to be able to execute
incremental updates
partially working SSH and
REST monitorig

29. Solution current state
8/13/2015 HLoader – A. Bose, D. Stein 12

30. Solution future work
8/13/2015 HLoader – A. Bose, D. Stein 13
– Support more database connectors SQLA/NoSQL
– Support alternative runners like Oozie
– Prepare for Sqoop 2
– Integrate with Hive
– Resolve restrictions
– Release on GitHub with an Open Source license

31. Summary
– Easily expandable framework and service
for transferring data from Oracle to Hadoop
– Designed with automation in mind
minimal administrator intervention needed
– Service built for easy usage
easy to use for the routine jobs
8/13/2015 HLoader – A. Bose, D. Stein 14

32. Workflow tools
– GitLab
– JIRA
– Slack
– Jenkins CI
8/13/2015 HLoader – A. Bose, D. Stein 32

33. Contributors
– Anirudha Bose
– Dániel Stein
– Antonio Romero Marin
– Domenico Giordano
– Kacper Surdy
– Katarzyna Maria Dziedziniewicz-Wójcik
– Manuel Martín Márquez
– Zbigniew Baranowski
8/13/2015 HLoader – A. Bose, D. Stein 15

34. Client
Meta DB
REST API Agent
Oracle Databases
FIM
Hadoop Clusters
…
HLoader
8/13/2015 HLoader – A. Bose, D. Stein 16
HL