Ocassionally connected devices spark final
•Download as PPTX, PDF•
2 likes•397 views
Spark Conference 2015
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Ocassionally connected devices spark final
Similar to Ocassionally connected devices spark final (20)
Recently uploaded
Recently uploaded (20)
Ocassionally connected devices spark final
- 2. What we’ll cover • Synchronization - Offline First! • CAP Theorem • Mobile Security • Responsive Design & Engineering • Case Studies • Testing / Tools
- 3. Photo by Chris Ballance
- 4. Places where I’ve built things…
- 5. Test Man Test Man, 2002, reproduced under fair use
- 6. Common Fallacies • Networks are reliable • Data must always be 100% consistent between nodes • CRUD can handle that (not very well)
- 7. Back to the Future, reproduced under fair use
- 8. Mobile First + Offline First
- 9. Designing Offline First Web Apps Alex Feyerke, 2013 http://alistapart.com/article/offline-first
- 10. Offline is not a feature
- 11. Design for Offline Add Sync as a feature
- 12. Your users will thank you…
- 13. Development Considerations • Idempotence • Immutability • Event Sourcing • Local Caching • Queueing
- 14. Immutability
- 15. Idempotency
- 16. Event Sourcing • Append only • All actions on nodes are stored • Actions can be replayed to any point in history • Current state is original + sum of all changes • Addresses inherent limitations of CRUD • Soft deletes only
- 17. CQRS • Command Query Responsibility Segregation • Able to scale read and write separately • Separation between read / write code • Safety - Queries are guaranteed to be free of side effects • Better allows for specialization on the read side using caching models such as CDNs
- 18. Project Delivery Triangle Good Fast Cheap
- 19. Project Delivery Triangle Good Fast Cheap
- 20. CAP Theorem Consistent Available Partition Tolerant
- 21. CAP Theorem Consistent Available Partition Tolerant
- 25. Consistent Available Partition Tolerant Partition Tolerance is a myth!
- 26. We can guarantee Consistency, but not Availability. We may time out or fail to return anything Consistent Available Partition Tolerant
- 27. We can guarantee Availability, but our data may not be Consistent with other nodes Consistent Available Partition Tolerant
- 28. Our business needs will drive whether we choose Consistency or Availability Available Partition Tolerant Consistent
- 29. Implementations Consistent Available Partition TolerantAP Cassandra, CouchDB CP MongoDB, Big Table (GFS) CA RDBMS SQL Server
- 30. A contrived example for Consistency A B
- 31. User writes S1 to Node B A B S0 S0 S1
- 32. User queries Node A Nodes B & A have not sync’d A B S1S0
- 33. Query is blocked until B syncs with A A B S0 S1 ?
- 34. Once B syncs with A, the query on A is unblocked and returns S1 as expected A B S1 S1S1 S1 :-) * The query could potentially time out
- 35. A contrived example for Availability A B
- 36. User writes S1 to Node B A B S1 S0 S0
- 37. User queries Node A for S1 A B S1S0
- 38. Query returns current state of A, but is not consistent with B A B S1 S0 S0 S1 ?
- 39. A later query of A yields S1 previously written to B. Eventual consistency has been achieved. A B S1 S1 S1 S1 :-)
- 40. • Which of these are Idempotent? • Console.WriteLine(person.FirstName); • person.FirstName = “Chris” • person.OrderBeer(); • person.TakeOffShoes();
- 41. Effective & Unobtrusive Get Smart, 1970, reproduced under fair use
- 42. Security • Device security • Data at rest • Data in transit • Data in memory • Remote wipe
- 43. Data in Transit
- 44. Data at Rest
- 51. Device sends public key to server
- 66. 404! WTFBBQ?
- 76. Testing Considerations • Network Latency • Data - loss in transit • Data - order of arrival • Device simulation
- 77. Testing Tools • DummyNet (Windows, etc) • Network Link Conditioner (OSX) [demo] • SoftPerfect (free for 30 second sessions) • Charles (Windows) • Android Simulator
- 78. Be Responsive! • Responsive UX • Responsive Engineering
- 82. Case Studies
- 85. A Gift Card System https://www.flickr.com/photos/68751915@N05/6710868859
- 86. Gift Card System • The business values availability over consistency to a certain tolerance ($100) • Eventual consistency is preferred to timeouts or failures • Low chance of double-spend if this is a brick & mortar store
- 90. Social Media for Astronauts • Network Latency to Low Earth Orbit is high • May have network outages depending on position in orbit • Dropped Packets • P2P for app use when disconnected from Earth Photo by NASA
- 92. Considerations for Financial Markets • Consistency is more important than Availability • Multiple markets (node partitions) • Ultra-low Network latency is crucial • Front-running orders*
- 94. RunKeeper
- 95. Occasionally Connected Devices are different… • Must work when disconnected • Device and its data are vulnerable at rest • Often connect over unsecured networks • WIFI and cellular networks can be unreliable • Other differences?
- 96. Further Reading • Designing Offline-First web apps http://alistapart.com/article/offline-first • CQRS - Martin Fowler Blog Post • Amazon Dynamo - http://www.eurecom.fr/~michiard/teaching/slides/clouds /cap-dynamo.pdf • Facebook Lite - http://www.wired.com/2015/06/facebook-launches- facebook-lite-super-slow-connections/
- 97. What we covered • Synchronization - Offline First! • CAP Theorem • Mobile Security • Responsive Design & Engineering • Case Studies • Testing / Tools
- 98. Questions?
Editor's Notes
- Thank Mark and the other co-organizer for putting on this event. Thank the Enterprise Developer’s guild for putting on such an awesome free event every year. I’m Chris Ballance, and I’m excited to have the opportunity to hang out with all the smart people here and to speak to you guys today. My company was nice enough to overnight me a shirt with our new logo for this event, so mention that Diligent is hiring. We’re a .NET, iOS, and Node.js shop and are looking for the Spolsky ideal of “smart and gets things done,” so catch me later if you’re looking.
- This is me at my desk writing code on a typical day. Segway: I know a lot of folks end up in a Development role from a wide variety of other disciplines. I’ve worked with sculptors, attorneys, classically trained musicians, and various others who excel at writing software. If you’re new to software, I encourage you to stick with it, you’ll never get bored. Oddly enough, I knew I wanted to write software since I was about Five years old. This video can explain…
- Volvo - Common web platform. Started as a junior dev and left 4 years later as a tech lead on a product called Dealer SiteBuilder which syndicated content globally to Volvo dealer sites and allowed them to customize these. Microsoft - Joined the team in Charlotte working on a product called Bemis that was the authoring backend for the Knowledge Base articles you see on support.microsoft.com for another 4 years. When I was burned out of BigCorp, I left Microsoft to join a startup called Triumph Research Intelligence based outside of London. I lead development of a Risk Based Monitoring product on a visualization platform we also built called VisualOPRA. It was fantastic right up until the moment we shipped v1.0 and ran out of money :-) <pause> Back in December I joined a new company to Charlotte called Diligent and we’ve been heads-down developing a collaboration product called TEAMS that we’re shipping in Q3.
- So, I like to make analogies to the real world when I talk about software - sometimes to a fault. For Occasionally Connected Devices, I thought this was fitting. Anyone remember this guy?
- Doc Brown the right attitude for developing software for Occasionally Connected Devices
- Open and read this site (have backup on desktop)
- Relate “have you thought about offline”
- This slide needs moving
- Need to at least get this slide to living beside the Immutability bullet list slide. Synchronization of mutable objects is hard Refine consistency using immutable objects to achieve eventual consistency
- If an operation is applied multiple times, the result is the same as if it was applied only once Spilled milk cannot be “more” spilled. (find a better example)
- Before I get into CAP theorem, I’d like to start off with another very similar triangle you’re already familiar with…
- CAP Theorem is similar. We have three competing interests
- Choose two
- Consistency - Always reads the most recent write - One version of the truth - But, if nodes are partitioned, we cannot guarantee consistency. - We can wait until we’re consistent, but in the meantime we may timeout and cannot continue.
- Every (unfailed) node always returns query results Eventually consistent No errors or timeouts, but data may be stale If partitioned, a node can return its current state, but it may be an old version of the truth
- Are all nodes always consistent? What is the lag time between node updates? One version of the truth Transactions are atomic
- -Be dramatic… PARTITION TOLERANCE IS A MYTH!!1! -This is just the nature of networks. Nothing is guaranteed. -So really, we just have two competing interests, Consistency, and Availability. -Choosing which to favor cannot be an entirely technical decision, but something to discuss with your stakeholders and it depends on the type of app you’re building.
- Once choice would be to favor Consistency. While this isn’t as popular with most workflows, it has its place.
- -Another choice would be to favor Availability. You’ll find this is very common in social media, and media in general.
- Here’s an example where we’re favoring consistency. We have a stick guy with data to write, and two nodes to write to and read from.
- S1 is going to be our payload, and we send it to node B.
- After we’ve sent our write (S1) to node B, we query Node A for the value of S. We might not even know that we’re being routed to node A for the read when we just wrote to node B. This can be even more confusing to even savvy users if this redirection is fully transparent behind a load balancer (and it typically is).
- Stick man is not a patient fellow and he’s tapping his foot, waiting for his query to return. He’s probably just going to assume the system is broken after a few seconds…
- Finally, Node A is able to synchronize with Node B to get S1 and return it to Stick guy. There’s a real risk of starving Stuck guy, and he’s already looking mighty thin.
- Now let’s look at a different approach. This time, we will favor Availability over Consistency.
- Our buddy, Stick guy, again sends S1 to Node B.
- Now he queries Node A.
- He immediately gets a reply, but it is stale data, since Nodes A & B have not yet synchronized. We don’t have to wait for the query to return, but we do get stale data that is older than what we have already sent out.
- On the later query, stick guy gets back the value S1 that he is expecting, consistent with what he previously sent to Node B.
- This needs a header slide. Not sure where it went. Now on to security…
- Things to point out about this hokey security - Security is ineffective - Causes more trouble than value add - In the end, is not used and abandoned
- TODO: change to “Physical device security”
- SSL / TLS, but this is not enough. -Securing the channel is important, but not sufficient. - Man in the middle attacks are a vulnerability (We could talk about Bob, Alice, and the other person in the crypto gang of three, but I’ll leave that for a different talk) - 128-bit symmetric keys are the SSL standard. How long do you think it takes to brute-force AES 128-bit? Less than half as long as it did last year… In a few years my refrigerator will be capable of breaking AES.
- If the data isn’t moving, encrypt it DPAPI, Asymmetric, etc.
- We’ll start with a generic App that needs to send data. In this case an iPhone 5. I wanted to use the iPhone 6 Plus, but I couldn’t get it to fit on the slide.
- We need to give the user an indicator that we have stored the data, and are prepared to sync it at the next opportunity. It should not slow down at all if we don’t have a network connection at the moment.
- Server can either inspect the payload or not. There may be routing, a service bus, or some processing that needs to occur to the payload. In the case of a social media app, its content may be syndicated to other places.
- Enter Android, wanting to synchronize.
- I’m skipping how the Android phone set up its security contract. Assume it is already in place from a previous synchronization.
- We’ve read the message / data / photo and store it encrypted locally. The next time we send a synchronization request, we’ll include some sequence information or date we last synchronized to prevent re-sending data we’ve already synchronized which hasn’t changed since we retrieved it last.
- Need a slide showing what responsive design looks like.
- This is scary, and it is significantly more scary than last year. There are 2 billion devices coming online in the next few years in developing nations. They’re probably not going to be iPhones at their current price point… Need a slide on the rise of users in developing nations with limited funds to purchase equipment and very limited connectivity.
- Add notes from this article http://www.wired.com/2015/06/facebook-launches-facebook-lite-super-slow-connections/
- I had planned to build some live demos to demonstrate, however….
- It has been a busy year… segway to case studies
- Some aspects of a financial system, but less risk adverse.
- Starbucks example —> race condition So I got an idea to buy 3 Starbucks cards $5 each.
- Completely decentralized Security based on strong cryptography, not on passwords Fits the “Occasionally Connected” model natively Partition Tolerant