SpringOne Platform 2016 Speakers: Joseph Rodriguez; Engineering Manager, Pivotal. Peter Clowes; Software Developer, Savoya. Are you taking on a large project that seems insurmountable? Perhaps a full rewrite, migrating to the cloud, or a large ETL? Extreme Programming can help! Following a full rewrite of our legacy application, we were tasked with transferring data from Oracle to Postgres- with an entirely different schema. This session covers basic principles of XP and how they helped get us to production in under 2 months.

1. Tackling unwieldy projects with
XP
Peter Clowes & Joe Rodriguez
@peterclowes @joerodriguez

3. What we’ll cover
• Our unwieldy project
• Why & how we applied XP
• Interesting SQL
3

4. • Safe and secure ground transportation worldwide
• 24/7/365
• 70/100 Fortune 100 CEOs
• Founded in 2000
• Based in Dallas
• 10+ year legacy software
4

5. • Ground up re-write
• Enable new team
• Extreme Programming
5

6. The Challenge
Migrate systems in under 5 hours
6

7. First false start
Tried it ourselves
• Lack of context
7

8. First false start
Tried it ourselves
• Lack of context
• Wrong approach
8

9. First false start
Tried it ourselves
• Lack of context
• Wrong approach
• Slow development
9

10. Second false start
3rd Party ETL team
• Team not co-located
10

11. Second false start
3rd Party ETL team
• Team not co-located
• Big problems stayed big
11

12. Second false start
3rd Party ETL team
• Team not co-located
• Big problems stayed big
• Feedback anti-patterns
12
* Note: vehicles get much heavier when filled with water

13. Extreme Programming
• Refactoring
• Continuous integration
• 40-hour workweek
• On-site customer
• Metaphor
• Coding standard
13
• Test Driven Development
• Pair programming
• Rapid iteration
• User stories
• Collective ownership
• Fix It
• Simple design

14. Test Driven Development
• High risk one-off
• Prevent regressions
• Clarity
• Easy to deal with changing requirements
14

15. Messy Data
15
d=# select xmlparse(document note.value);
ERROR: syntax error
update note set value = ‘<?xml version="1.0" ?>…’
^

16. Messy Data
16
id value
290384 update note set value = ‘<notes>
<note>
<value>Please have a micro USB cable</value>
<author>Peter C</author>
<created>1201932175</created>
</note>
…
</note>’;
note

17. What does 20973889 mean?
17
id ops_status
313 20973889
reservation
We asked for the legacy source code, but we got a .war instead.
20973889 >> 4 & 128 > 0 == The reservation was confirmed!

18. 18
reservation_id name
313 created
313 confirmed
313 in_transit
reservation_events (destination)
id ops_status
313 20973889
reservation (source)

19. 19
public void testEventsAreCreated_ForReservationInTransit() {
// arrange
exec("insert into etl.reservation (id, ops_status) values (313,
20973889)");
// act
new ReservationEventsEtlService.run();
// assert
assertEvents(313, ”created", ”confirmed", ”in_transit");
}
Unit Testing ETL Code

20. Rapid iteration
• Real, up-to-date data
• Multiple pushes a day
• Dedicated environment for testers
20

21. Fake data for complex relationships
21
reservation
coordinator driver
update reservation set driver = 1;

22. One Script
Step 1: Download DB dump
Step 2: Restore dump into VM
Step 3: Copy Oracle DB to Postgres
Step 4: Run Transformation
Step 5: Create Postgres dump
Step 6: Upload dump and restart app
22

23. Co-located pair-programming
• Knowledge transfer
• Reduce defects
• Safety net
23

24. Why Postgres? Why SQL?
• Destination data-store
• As close to the data as possible
• Tooling + built-ins
• Speed
• Readability
24

25. 25
id passengers
290384 <passengers>
<passenger>
<firstName>Johnny</firstName>
<lastName>Appleseed</lastName>
<email>johnny@aol.com, johnny@juno.com</email>
<workPhone>010 0112716677, +1 999-123-5435</workPhone>
</passenger>
…
</passengers>
ms_reservation (source)
reservation_id first_name last_name email phone_number
290384 Johnny Appleseed johnny@aol.com 0100112716677
reservation_passenger (destination)

26. Basic data mappings
1. Select from source table
2. Transform data
3. Insert into destination
26
WITH passenger_objects AS (
SELECT id,
unnest(xpath(‘//passenger’, passengers::xml))
p
FROM ms_reservation
)
INSERT INTO reservation_passengers (
reservation_id, first_name, …
)
SELECT id,
xpath(‘/first_name/text()’, p) first_name, …
FROM passenger_objects;
2
1
3
2

27. A more complex mapping
27
reservation reservation_passenger passenger

28. 28
id passengers
290384 <passengers>
<passenger>
<firstName>Johnny</firstName>
<lastName>Appleseed</lastName>
<email>johnny@aol.com, johnny@juno.com</email>
<workPhone>010 0112716677, +1 999-123-5435</workPhone>
</passenger>
…
</passengers>
ms_reservation (source)

29. 29
reservation_id passenger_id
290384 321
passenger (destination)
id first_name last_name email phone_number
321 Johnny Appleseed johnny@aol.com 0100112716677
reservation_passenger (destination)

30. 30
WITH passengers_with_reservation_id AS (
SELECT reservation_id, (…) AS name , (…) AS email from etl.reservation
), new_passengers AS (
INSERT INTO passenger (name, email)
SELECT name, email FROM passengers_with_reservation_id
RETURNING *
)
INSERT INTO reservation_passenger
(reservation_id, passenger_id, name, email)
SELECT reservation_id, id, name, email
FROM passengers_with_reservation_id
JOIN new_passengers ON (…);

31. Launch Day
• ETL took 2 hours
• Handled issues over the next 24
hours
• Flew to HQ to support team
• Fully stable in 2 weeks
• Deep trust between teams
• Agile mindset
31

32. Resources.
• extremeprogramming.org
• engineering.pivotal.io/post/etl-journey-from-oracle-to-postgres
• ora2pg.darold.net
• laurenz.github.io/oracle_fdw
@springcentral
spring.io/blog
@pivotal
pivotal.io/blog
@pivotalcf
http://engineering.pivotal.io

33. Questions
Peter Clowes & Joe Rodriguez
@peterclowes @joerodriguez