Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Hold My Beer, I'm going to do DevOps with DICOM, HIPAA, and Hospitals, or at least try - Gil Bahat - DevOpsDays Tel Aviv 2017


Published on

DevOpsDays Tel Aviv 2017

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Hold My Beer, I'm going to do DevOps with DICOM, HIPAA, and Hospitals, or at least try - Gil Bahat - DevOpsDays Tel Aviv 2017

  1. 1. Hold my beer! I’m going to do DevOps with DICOM, HIPAA and Hospitals. Or at least try.
  2. 2. A little bit about me Independent DevOps consultant Like to solve hard problems (Hold my beer…) Social links:
  3. 3. Disclaimer This lecture bastardized DevOps terminology, unapologetically mixing ideals and real world concerns and generally sidesteps the whole “What is DevOps” debate by flipping a finger. We’re here to work and make this beast function somehow.
  4. 4. Two part lecture In part 1 We’re going to talk about DICOM, one of the two major healthcare protocols. In part 2 We’re going to talk about healthcare regulations and how they change our workflow and thinking. Even if you don’t care about DICOM, consider it an exercise in onboarding non- standard protocols. Even if you don’t care about healthcare regulations, consider it an exercise in handling any regulatory environment
  5. 5. DICOM
  6. 6. Motivation Learn how to handle a non-HTTP protocol: Get familiar with the protocol Scope its usage in our system Understand key features we care about (performance, security, …) See how we can realize them.
  7. 7. Get to know DICOM DICOM stands for “Digital Imaging and COmmunication In Medicine”. It’s both a wire protocol and extensible serialization format. Its designed by NEMA (electronics manufacturers) - makers of modalities (the medical term for xray/CT/MRI/… machines) Its design philosophy is “Be conservative with what you send and liberal with what you accept”. DICOM is a highly specialized protocol. Very few implementations, much less open source.
  8. 8. Scoping DICOM for our system Few Implementations means trouble. With DevOps you really want to avoid re-inventing the wheel. Some SDKs, much less integrated ‘plug and play’ packages. Auxillary components are generally missing: load balancing, monitoring, utility parsers etc. For the exact same reasons.
  9. 9. Scoping DICOM for our system We expect to use DICOM to transfer data from the hospital to our processing domain. Internally we will avoid the wire protocol as soon as possible. As for serialization, YMMV. Risks and benefits either way.
  10. 10. DICOM Technical features The protocol is binary, overlays a full OSI model over TCP/IP. Strictly regulated enumerators and hard-coded behaviours. Let’s see how easy or hard it is to retrofit stuff we need for scalability and security onto DICOM.
  11. 11. Analysis of DICOM features Commands are sent from SCU (“client”) to SCP (“server”). Only SCU is allowed to cleanly release a connection. Commands are split to notification and operation. Async operations supported, optional and negotiated.
  12. 12. Analysis of DICOM features Commands are either A (association), C (composite entities) or N (normalized entities). We’ll be looking at A and C lingo. We see the following vocabulary: A- ASSOCIATE, A-RELEASE, A-ABORT, C-STORE, C-FIND, C-MOVE, C-GET and C- ECHO. P-DATA (payload contents basically) may be fragmented. Messages have unique IDs and are all acknowledged.
  13. 13. Incoming DICOM We can pipeline when multiplexing, if negotiated during the connection. Since we’re dealing with studies, we’re looking at a lot of short transfers in same bulk. If we keep long-term established connections, we have to plan our load balancing a lot more carefully.
  14. 14. Incoming DICOM Observation: Connection establishment involves 3-way TCP and 3-way ACSE (TLS not included - yet). We can keep connections alive with C-ECHO. We cannot multiplex different ACSE names (either client or server) though, each will result in a new connection.
  15. 15. Load balancing DICOM What is it that we want from a load balancer, anyway? Balancing load between instances, to allow for good resource utilization, good responsiveness, avoid overloaded instances. Detect instance failure, remove failing instances from duty to avoid data loss / outage. Reflect a safe error condition towards the caller if the system is unresponsive Modify/tweak client behaviour to better suit the server needs (optionally) Handle/offload encryption and authentication
  16. 16. Securing Incoming DICOM DICOM Security: TLS (part of the standard). Some people do it with IPSEC VPNs, we’ll dedicate a slide for that option. DICOM authentication: User/Password, Kerberos, SAML Digital signatures supported.
  17. 17. Securing Incoming DICOM TLS is part of the DICOM standard, since mid-2000’s. Albit specified in a highly braindamaged constraining fashion, to 3 profiles: TLS 1.0 RSA with NULL encryption TLS 1.0 RSA with 3DES encryption TLS 1.0 RSA with AES encryption Entity authentication is optional.
  18. 18. Load balancing DICOM Load balancing: Selected algorithm: leastconn Detecting failed instances: C-ECHO pings, dummy C-STORE actions Static 504 replacement: A-ASSOCIATE-Rejector TLS termination + TLS certificates Without a full-fledged DICOM implementation, this is pretty much what we can get.
  19. 19. Securing Incoming DICOM Overall, things are looking great for us right now. It looks like we can achieve our stated goals, without investing too much effort into it. A bit of haproxy lua wizardry around ACSE primitives and presto! A full-fledged DICOM load balancer is probably a major undertaking.
  20. 20. DICOM in the wild (west) Strike #1: No auth methods are supported by modalities. Today’s magic sentence is: “This product does not conform to any defined DICOM Security Profiles. It is assumed that the product is used within a secured environment.”
  21. 21. DICOM in the wild (west) Strike #2: No TLS transport supported by modalities. Today’s magic sentence is still: “This product does not conform to any defined DICOM Security Profiles. It is assumed that the product is used within a secured environment.”
  22. 22. DICOM in the wild (west) Offside: why IPSEC-VPN is a bad idea. IPSEC is a cumbersome protocol with lots and lots of rope to hang yourself with. Expect long debugging sessions. Staff your support accordingly. It’s also technologically inferior to TLS with regards to ease of tunnel setup, for failover scenarios.
  23. 23. DICOM in the wild (west) Strike #3: Commonality of lost images. There is no strong guarantee that all images are being sent, and a missing image makes the entire study untrustworthy. Protocol-level guarantees are no good since they do not account for host/policy override and/or are not supported. Remember C-FIND? It’s now our job to C-FIND the missing parts of a study. What joy!
  24. 24. DICOM in the wild (west) Strike #4: C-FIND means we have to initiate a connection from our side to fetch the missing part. This would have messed up our entire architecture, but this option is impossible due to security considerations. So now we have to design and maintain our own in-premise equipment.
  25. 25. DICOM in the wild (west) Strike #5: Our favorite Open-source DICOM router does not multiplex. It does support multiple connections. But you have to tweak them really fine: too many and you lose images. Too little and your performance suffers.
  26. 26. DICOM in the wild (west) Strike #6: We’re barred from using client side certificates. This would have been our preferred way of authenticating, given the DICOM auth mechanism selection and the fact we don’t want to rely on DICOM authentication for security, and aiming for a TLS deployment The reason we’re barred from doing so is because DLP gateways will MITM any traffic which doesn’t pass a designated IPSEC tunnel.
  27. 27. DICOM in the wild (west) HTTP would have done somewhat little to avoid our woes here: we’d have to have on-site presence due to policy and access controls. We’d have to go with VPN tunnels due to DLP policies. We’ll have to suffice with trying to shift as much workload / smarts to where we can control them.
  28. 28. DICOM in the wild (west) Actually, why did we prefer a zero-footprint installation, intuitively? Because of the release cycle and control. This is a weak spot when deploying in medical environments and may very well still be so in the next few years. This means that our design still needs to make our in-hospital presence as minimal and well-tested as possible. Almost as if it’s an embedded/IoT medical device (in a sense, it is).
  29. 29. DICOM in the wild (west) Can we apply some “DevOps for embedded/IoT” best practices? Probably, but DICOM and Hospitals are going to make life hard for us here as well. These environments are not easily reproducible or simulated due to exotic protocol, exotic equipment, complex integration environment, security filters...
  30. 30. DICOM in the wild (west) This turns our DevOps transformation into a people’s problem, more than a technical one. The right way to do DevOps in regulated environments is to understand the on-site requirements and matching the deployment. This is extremely hard to genericize, so you have to assume the worst. Keep baseline exceptions to the minimum. This means a non-typical customer engagement profile. Do it. Full-Circle holistic design.
  31. 31. Regulations (HIPAA)
  32. 32. Intro to HIPAA HIPAA: HIPAA is a regulatory framework governing privacy and accountability of entities processing US healthcare data. (there are other frameworks governing EU, IL, etc). Terminology: PHI - Protected Health information. BA/BAA - Business Associate / Business associate agreement. Governs sharing your data and responsibilities with third parties.
  33. 33. Intro to HIPAA Get familiar with the regulations Scope its footprint in our system Understand key requirements/constraints we care about (security, auditing…) See how we can realize them.
  34. 34. Intro to HIPAA From a DevOps perspective, we are looking at any of the following being assigned / coordinated / supported / designed: Encryption, Vulnerability scans, Asset management, Disaster recovery, Configuration management, Incident investigation, Auditing, Logging, Coding standards, Operating procedures.
  35. 35. Intro to HIPAA Learning our lesson from before, there’s regulations and there’s unwritten regulations. Before you architect your solution, be sure to collect “unwritten” regulations from potential customers. Stick to product management / sales / marketing if needed. Do not relent. (also, lest they promise something which is impossible from this aspect)
  36. 36. Intro to HIPAA Accountability is a key line of thought for regulations. Get ready to procedurize(™) yourself to hell and back. This can align nicely with DevOps principles. Accountability implies auditing framework. This encourages an RCA culture, if actively pursued. Give heavy weight to logging solutions and their ability to handle PHI.
  37. 37. Intro to HIPAA Security updates can be fast-tracked. Don’t miss this opportunity. Encryption is your friend: at rest and at transit. But also in data store too: The encryption keys can serve as a basis for authorization - having the key implies a transferrable authorization context.
  38. 38. Intro to HIPAA DevSecRegOps(™)! Keeping your environment compliant and provably compliant. So, for all the topics we’ve mentioned before (encryption etc), we’re now looking at a three-way assessment of their impact / requirements and desired processes / culture. It’s important that they all talk the same language, and definitely doable.
  39. 39. Regulated DevOps culture Understand the boundary between patient-impacting and non-patient impacting, as well as the PHI borderline. It’s hard to keep two distinct cultures. YMMV if you want to do a cultural split personality thing or not. Embedded development with its release cycle and methodologies is already different enough.
  40. 40. Regulated DevOps culture Healthcare open source kits are good enough for scripting tests you may need. Keeping on the cultural aspects just discussed, embrace a test culture throughout. You know you wanted to anyway, now regulations are pushing you to do the right thing(™). Then again, we may need to consider some facets of testing in regulated environments...
  41. 41. Regulated DevOps culture Pro tip for testing in live - always roll back if patients start dying. An open architectural problem - roll back not yet implemented for patients. Until that feature is available (being worked on, no ETA unfortunately) - No testing in live, particularly because it’s alive and we want to keep it this way. But we can test in preprod with live data - but only after anonymization. Get a design partner.
  42. 42. Regulated DevOps culture Replicating the hospital test environments is pretty much impossible - including oddball devices, policies (some may even be hidden from you) and even datasets and integrations. However, in an online/cloud architecture we can treat it as a black box and just emulate the feed coming off of it after anonymizing it. So, split environments/cultures again...
  43. 43. Regulated DevOps culture Tenancy models: getting this stuff right. Single-tenant - eases the regulatory requirements, at the cost of resource waste Multi-tenant - more complex regulations, allows better resource sharing. Hybrid - twice the development/testing Shared compute, separate data - with enough data access layer sophistication. Accounting for hotspots, etc. A fun challenge. (geo) Data retention laws make this decision less straightforwards than other SaaS deployments.
  44. 44. Regulated DevOps culture Tenancy models affect security and regulatory architecture: Single tenant - emphasizes things like network ACLs, IAM permissions Multi tenant - emphasizes things like application ACLs, multiple encryption keys, token services.
  45. 45. Regulated DevOps culture Regulations will inevitably be a political driver for many company decisions around the development processes (and hence DevOps) Potentially it’s a double-edged sword, e.g. with external services not offering BAA or refusing PHI. DevOps can be an enabler for conceptual regulatory requirements (e.g. contingency, software acquisition, backups). Software acquisition example - mirroring/proxying sourced software
  46. 46. Regulated DevOps culture Shared responsibility model - allowing some form of (needed!) relief. What you don’t have to do - don’t do it. Physical security, disposal of physical media, all that stuff. Happily delegated. 3rd parties are your friends - only if they offer HIPAA whitepapers and design docs.
  47. 47. Questions? Now, will you hold my beer already?