Oscar Engineering part 1


Published on

In this first post, we’ll be taking an exclusive look at the backend stack for data processing here at Oscar and how it continues to evolve. This is a lengthy discussion, so it will come in three parts over the next few weeks.

Building an engineering program at a new tech company is an exciting and often tricky business. There are so many amazing new technologies out there, so many battle-tested and reliable services, and so many tools that could be right for the job if you just make a choice and tweak it until it works for you. It’s fun to try new things and experiment, and it’s rewarding to build rock-solid services. At Oscar, we’re all about maintaining a high degree of curiosity and technical exploration, while relying on proven methods and technologies. Does that sound consistent to you? It’s not!

So, how do you reconcile conflicting engineering impulses while still having a good time? By making sure you’re addressing the stuff that matters first and foremost. Let’s establish some foundational premises from which arguments can proceed. Those premises will be the properties our systems must achieve by whatever means we come up with. We’ll get back to the conflict in a later post.

Data Acquisition
First, what are the properties we need? Remember, we’re talking about backend data processes alone here. Let’s start with one of the first things we had to do in engineering - connecting partner feeds. The insurance business is a complicated space, and we’re currently handling multiple data feeds from about ten different partner companies. As of this writing, we’re receiving about 60 individual data feeds and that number is likely to explode over time. Some of the feeds are trivial, requiring us to simply copy new data into our storage systems and leave it there, but that’s the exception. One of our feeds is an ASCII database transaction log representing hundreds of tables from a remote database written in Cobol. Many are fixed-width text files or CSV that must be parsed according to a schema of byte offsets and data types. For good measure, we also see EDI and HL7.

Whatever the format, the mission remains the same: to create a unified and up-to-date view of the data for users - both external and internal. And of course, don’t let a single bit get misplaced! This data really matters. It affects the lives of our customers and if we mishandle it, we can create inconvenience or even add significant stress at a time when focusing on medical treatment should be top priority. Basically, we shouldn’t ever screw up our data, and when it gets screwed up (yes, it will happen), we must be able to recover fast. Now we have our properties - whatever our choices, our feed systems must:
● Respect order.
● Break on failure
● Be idempotent
● Be low-latency
● Maintain privacy

That’s already a fair amount of properties. If we have several competing approaches, they all have to maintain the same general behavior and that’s more work.

1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Oscar Engineering part 1

  1. 1. Our objective seemed simple at first: to create a unified and up-to-date view of incoming data for all users, without misplacing a single bit. ! It started small, just a few vendors to connect. Maybe 5 important feeds. ! 5 became 10, then grew to 50. There were data streams In the beginning Formats became more varied, as did the properties of the feeds themselves. ! Achieving the goal was going to require thought, but first, let’s look at the data. 2
  2. 2. ASCII blobs - schemas define character ranges and data types. ! Example: Transaction log from a Cobol DB. 254 tables in single feed. ! Aggressive parsing required! Often, data types and NULL constraints are the only obvious clues indicating data integrity. ASCII byte ranges Fixed-Width 3 00226345user.ix1 00000074 2013102112110400CHANGED 00000074IT8208IV*Z IT8208 IT8208 INFO| 100000.00 100000.00 100000.00 A20050101INITLD20120720IT8208 215 91515 91414 11114 7 0 9 814 7 1 0111414 0 010 7 0 51514 01415 41515 01415 0 714 0 01015 11114 51514 41515 0151414 0 1 3 1 3 1 3 1 3 1 3 1 3 1 3 1 3 1 3 1 3SASSSASSSASSNORM 0225DEMO AHAUSSLER@SS-HEALTHCARE.COM 18991231 NORMNORM A00819870000271400000002C00025 89 C00272560000050000000087
  3. 3. Since 1989 (v2), XML as of 2005 (v3) ! Targets clinical and admin data interchange among hospitals ! Core reference model has been called an incoherent standard (Smith & Ceusters, 2006) Health Level Seven HL7 4 <POLB_IN224200 ITSVersion="XML_1.0" xmlns="urn:hl7-org:v3" xmlns:xsi="http://www.w3.org/2001/ XMLSchema-instance"> <id root="2.16.840.1.113883.19.1122.7" extension="CNTRL-3456"/> <creationTime value="200202150930-0400"/> <versionCode code="2006-05"/> <interactionId root="2.16.840.1.113883.1.6" extension="POLB_IN224200"/> <processingCode code="P"/> <processingModeCode nullFlavor="OTH"/> <acceptAckCode code="ER"/> <receiver typeCode="RCV"> <device classCode="DEV" determinerCode="INSTANCE"> <id extension="GHH LAB" root="2.16.840.1.113883.19.1122.1"/> <asLocatedEntity classCode="LOCE"> … </POLB_IN224200>
  4. 4. Since 1971 ! Elements, segments, loops, and hierarchy - like XML, schema is up to the designers. Unlike XML, structure must be derived at parse time. ! Context-sensitive grammars! Need a parse stack to understand all EDI document types. Electronic Data Interchange EDI 5 ISA*00* *00* *12*ABCCOM *01*99999999*101127*1719*U*004 00*000003438*0*P*~GS*PO*440519 7800*999999999*20101127*1719*1 421*X*004010VICS~ST*834*0179~B GN*00*1*20050315*110650****~RE F*38*SAMPLE_POLICY_NUMBER~DTP* 303*D8*20080321~N1*P5*COMPAN_N AME*FI*000000000~INS*Y*18*030* 20*A REF*0F*SUBSCRIBER_NUMBER~NM1*I L*1*JOHNDOE*R***34*1*0000000~P ER*IP**HP*2138051111~N3*123 SAMPLE RD~N4*CITY*ST*12345~DMG*D8*196 90101*F~HD*030~DTP*348*D8*2008 0101~REF*1L*INDIV_POLICY_NO~SE *16*0179~GE*1*1421~IEA*1*00000 3438
  5. 5. Get a feed, write a parser, model it, store it, try to do it right, then deploy it. 50 times! In 50 ways? Sure, as long as we stay organized Easy, right? 6
  6. 6. Every feed has its own characteristics. ! Every project seems to want its own solution. ! It’s so easy at first just to implement things. Then comes The Mess. Insurance is hard. Let’s simplify it. Complexity is the Enemy! Can we identify common properties needed by our systems, and guarantee that those properties are satisfied in reaching a fundamental goal? 7
  7. 7. Control latency. Integrate data as quickly as possible. ! And last but far from least: ! Maintain privacy. Our jobs are often handling user data, and they must handle private data with great care. To keep our data correct, we need to ensure some fundamental system properties 8 Respect order. You process things out of order, you corrupt your data. ! Break on failure. Never write bad data or continue in an abnormal state. ! Be idempotent. Mid-process or mid- transaction errors are to be expected.
  8. 8. Judgment is required here, in abundance. These goals are more nebulous but of critical importance to a growing engineering team. Let’s not forget some important higher-order properties 9 Implement consistently. Avoid The Mess. Consolidate best practices. ! Deploy consistently. Every new deploy style carries a constant increase in operational complexity.
  9. 9. Parse and model each feed as a custom step, but let the framework handle common properties. ! oscaretl: a framework for transactional safety 10 Data streams in which data is not independent (often the case), will benefit from being handled as transaction logs. ! Monotonically increasing transaction IDs are very useful. Try to derive them if they don’t exist naturally in the data.
  10. 10. Parsers and schemas are custom, but data formatting, safe writes, and safe execution can be factored out. ! Common schema types can be re-used. ! Factoring out the common elements 11
  11. 11. Good start, but we need more: at this point, a good runtime model and a job scheduler. What have we achieved? 12 Strict ordering. Processing will halt on missing data. ! Idempotence. Careful state binding allows us to resume where we left off. ! Break on failure. Processing will halt on error.