Testament - a blueprint for hyper-personalized services at web scale

373 views
322 views

Published on

This is an architectural design pattern I put together to address a real world, but fairly generic, set of requirements that seem to come up in discussion time and again.

It is by no means meant to suggest a complete panacea formula...just a few concept sketches really.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
373
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Testament - a blueprint for hyper-personalized services at web scale

  1. 1. Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0 introducing the Testament Architecture ● An architecture for hyper-personalised services in an internet of things that you can use. Robert Gibbon – be.linkedin.com/in/robertgibbon The content of these slides is my own original thinking; or derived from information freely available in the public domain. Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
  2. 2. Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0 who am I? ● I am an independent information technology services consultant ● 15 years in the business ● 5 years doing stuff with big data ● And some more years on web-scale site management duty Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
  3. 3. Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0 why did I make this deck? I made this deck because ● I had a collection of ideas that were buzzing around in my head and I wanted to get them out onto paper. ● Neither the concepts and technologies, nor the way they are combined and presented in this architectural paper are related to any of my past or present clients. ● But...if you try to implement a Testament architecture and it doesn't work out for you, don't blame me! Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
  4. 4. Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0 contents This slide set goes over ● The operational context of the Testament architecture ● The challenges to service delivery that the Testament approach prioritises ● A flythrough of the major features and technologies in a Testament implementation ● Some closing thoughts Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
  5. 5. Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0 who stands to benefit from Testament? Testament is a high level architecture blueprint. It is oriented to multifaceted consumer services that require ● global reach ● millions of users ● multiple devices per user ● always available ● long data retention times ● processing very big datasets Testament proposes a set of building blocks on which to found certain aspects of such a service. Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
  6. 6. Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0 what challenges does Testament try to address? Testament prioritises these aspects of successful hyper-personalized service delivery at web scale. ● serving hyper-personalised data to millions+ concurrent users ● serving and reconciling data with split-brained services and offline-often devices ● serving “single record of truth” datums that must always be kept up to the minute ● cost-critical long term data storage and access for very big data Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
  7. 7. Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0 front end: conceptually Content Distribution Network Global Service Center 1 User X Database Branches User Y Database Branches User X Managed Devices CHICAGO AMSTERDAM TOKYO Global Service Center 2 Global Service Center n Communal Edge Service Nodes User Y Managed Devices Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
  8. 8. Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0 front end: logically Distributed Filesystem VCS, DDB, CEP Offline-often Devices CHICAGO AMSTERDAM TOKYO VCS, DDB, CEP VCS, DDB, CEP Hosted Application Servers Offline-often Devices Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0 Standalone dbs Standalone dbs
  9. 9. Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0 front end: exemplar physical model XtreemFS Git, Hbase, Storm Sunglasses, Phones, Wristbands CHICAGO AMSTERDAM TOKYO Git, Hbase, Storm Git, Hbase, Storm Games, eShops, Content on-demand, Social TVs, Hi-Fis, Fridges Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0 SQLite files SQLite files
  10. 10. Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0 front end: summary of the bits Testimonies ● Are a user, session, and device bounded collection of events and facts ● eg. “A testimony of latlongs, browsing events, DNS lookups and app usage activities from Robert's phone for last Friday” ● These testimonies are built to help better understand the user and consequently better serve the user, over time Global Service Centers ● Handle asynchronously submitted testimonies ● Pull new testimonies from the CDN ● Automerge and tag them, eg. “Merge this testimony onto the <Robert's phone> branch and tag it <last Friday>” ● Precompute ahead of pull requests from Edge Service Nodes for updated User Database Branches ● Run “single-version-of-the-truth” database services for those datums that must always be up to the minute, such as authentication credentials, avatars, screen names etc. Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
  11. 11. Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0 front end: summary of the bits Content Distribution Network ● Common service backplane ● Write-locally, read-globally ● Ship User Database Branches between Global Service Centers and Edge Service Nodes ● Ship Testimonies between Edge Service Nodes and Global Service Centers User Database Branches ● Provide a self-contained individual user database to enable hyper-personalized service delivery ● eg. attitude/trend/interest/demographic classifiers; motivators, intents and triggers; recencies; rising and decaying trends ● Form a container for Testimonies to be backshipped via the Content Distribution Network Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
  12. 12. Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0 front end: summary of the bits Communal Edge Service Nodes and Managed Devices ● Deliver actual application services to the user ● eg. Games, grocery shopping, grocery storage, streaming radio, TV on demand, household automation, remote power/HVAC management, social etc. ● Are two-way, in that they build and backship Testimonies, to help better understand the user and consequently better serve the user, over time Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
  13. 13. Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0 back end: conceptually Content Distribution Network Global Service Center 1 CHICAGO TOKYO AMSTERDAM VANCOUVER HEBRIDES ATLANTA Global Service Center 2 Global Service Center n Nearline Vault 1 Nearline Vault 2 Nearline Vault n Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
  14. 14. Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0 back end: logically Distributed Filesystem Compute Grid 1 CHICAGO TOKYO AMSTERDAM VANCOUVER HEBRIDES ATLANTA Compute Grid 2 Compute Grid n Massive Array of Idle Devices Massive Array of Idle Devices Massive Array of Idle Devices Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
  15. 15. Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0 back end: exemplar physical model XtreemFS Spark on Mesos CHICAGO TOKYO AMSTERDAM VANCOUVER HEBRIDES ATLANTA Spark on Mesos Spark on Mesos Depreciated Commodity Systems Depreciated Commodity Systems Depreciated Commodity Systems Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
  16. 16. Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0 backend: summary of the bits Global Service Centers ● Deliver the usual background batch processing services to build, refine and update apt and timely user insights from available Testimonies, so that front-end applications can better serve the user ● Identify Testimonies that the Global Service Centers cannot use at this moment and that are therefore eligible to be archived to the Nearline Vault ● Eg. Analysis technology is not yet sufficiently mature, and/or Testimony content has decayed in relevance Content Distribution Network ● Acts as a publish locally, read remotely conduit from Global Service Centers to Nearline Vaults Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
  17. 17. Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0 backend: summary of the bits Nearline Vaults ● Load old Testimonies onto a storage device on a Depreciated Server, spin down the storage device, and power off the Depreciated Server for long term storage ● Power up the Depreciated Server and spin up the required storage device on demand, for retrieval of the old Testimonies in the future. Depreciated Servers ● Old Servers that have depreciated, configured with high storage device density, often racked and packed in self-contained DC freight containers. Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
  18. 18. Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0 closing thoughts Testament only considers a small subset of architectural facets; its really only a teaser. There is a spectrum of architectural facets to desigining a successful hyper-personalised, web-scale service that shall be considered, and that are not considered by Testament. For example, User Database Branches and Testimonies must be protected and able to defend against unauthorised access, such as from competing service providers or from malicious users. Also, the personally identifiable link between Testimonies and the user to which they relate and the legal and ethical rights of users to sever that link, is another facet that these slides have not considered. Importantly, “no man is an island”, and handling Testimonies of the relationships between users has not been discussed in this slide deck. Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0
  19. 19. Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0 slide deck license Copyright 2014 Robert Gibbon Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Copyright 2014 Robert Gibbon. This content is licensed under the terms of the Apache Software License, version 2.0

×