There are no existing cloud federations today while research on the topic has been active for several years. This talk discusses theory and implementation of a closely related pair of technologies that make large-scale federations possible and feasible in practice. First, a Cloud Visitation Platform renders push/pull (container or VM) migrations feasible across heterogeneous hosts including end users (fog computing). Secondly, an OAuth-shared ecoin with a new micropayment design supports open/distributed/scalable resource accounting/metering/billing, thus facilitating a flat resource economy without centralized management. Both components are presented as research, software/APIs, and demos.

2. .
Visitation Platform + Micropayments
.
Problem is...
..
....that cloud federations are rare ... because no one knows how to build them
• most federations have rigid structure (can't grow)
• heterogeneous environments are a major problem
• there are no examples of resource economies in federations
.
Visitation 01 ...
..
.... is when VMs can visit with any cloud environment
.
Micropayments 02 ...
..
....is a technology for resource accounting in federations
01 myself+0 "A Cloud Visitation Platform for Federated Services at Network Edge" CISSE (2014)
02 myself+0 "Coins in Cloud Drives Can Use OAuth for Micropayments and Resource Metering Alike" CFI (2014)
M.Zhanikeev -- maratishe@gmail.com -- VM Visitation and Resource Micropayments ... towards Cloud Federations -- http://bit.do/marat150227 2/21
...
2/21

3. .
The 3 Know-Hows
1. the visitation platform itself, compatible with Xen platforms 01
◦ migration, APIs, interfaces, automation
2. binary diff technology for efficient migration within a federation 03
3. cloud coin for resource metering in an open federation 02
01 myself+0 "A Cloud Visitation Platform for Federated Services at Network Edge" CISSE (2014)
03 myself+0 "「Cloudy会」DIFF法：テキスト・ファイルシステム・バイナリ差分" http://goo.gl/v8kLpQ (2015)
02 myself+0 "Coins in Cloud Drives Can Use OAuth for Micropayments and Resource Metering Alike" CFI (2014)
M.Zhanikeev -- maratishe@gmail.com -- VM Visitation and Resource Micropayments ... towards Cloud Federations -- http://bit.do/marat150227 3/21
...
3/21

4. .
Visitation Platform Basics
Storage Sensors
…
Cloud Visitation PlatformContainer
Cloud Visitation Platform
App
App
App
…
APIs
…
Multitenancy
Physical Device
…
Interfaces Billing,
Fairness
Generics,
Resilience
Application
Specifics
• today: VMs are running in
blackboxes, zero
awareness of local hardware
• visitation: the opposite
approach -- VMs/containers
are fully aware of local
resources 01
• implemented as APIs/
Interfaces
• not hard -- just yum install
and add your software on
top of XCP
01 myself+0 "A Cloud Visitation Platform for Federated Services at Network Edge" CISSE (2014)
M.Zhanikeev -- maratishe@gmail.com -- VM Visitation and Resource Micropayments ... towards Cloud Federations -- http://bit.do/marat150227 4/21
...
4/21

5. .
Cloud Coin Basics
CRITICAL-ness
Distributed
Resource APIs
• micropayment: payment of
a very small amount (sub-yen)
• problem: a fixed
transaction cost per pay
• solution: open coin,
service can accumulate
payments (cost per batch)
• cloud coin: an ecoin stored
in a cloud drive 02
02 myself+0 "Coins in Cloud Drives Can Use OAuth for Micropayments and Resource Metering Alike" CFI (2014)
M.Zhanikeev -- maratishe@gmail.com -- VM Visitation and Resource Micropayments ... towards Cloud Federations -- http://bit.do/marat150227 5/21
...
5/21

6. .
Cloud Coin = OAuth-shared JSON file
• can be a simple JSON file stored in a cloud drive (Dropbox, ...) 02
• client - service provider: check/deduct the amount, store/change
the certificate (md5)
• service - coin provider: payments are cleared based on certificates
◦ note: there is a probability of collision, the coin is lossy, loss ratio is controlled by
service statistically
Client
ID
Coin Structure (JSON)
Coin
ID
Coin
Amount
Latest
Certificate
Other
Info
02 myself+0 "Coins in Cloud Drives Can Use OAuth for Micropayments and Resource Metering Alike" CFI (2014)
M.Zhanikeev -- maratishe@gmail.com -- VM Visitation and Resource Micropayments ... towards Cloud Federations -- http://bit.do/marat150227 6/21
...
6/21

7. .
From Talk to Action
M.Zhanikeev -- maratishe@gmail.com -- VM Visitation and Resource Micropayments ... towards Cloud Federations -- http://bit.do/marat150227 7/21
...
7/21

8. .
Talk → Action : A Practical Visitation
• say, I have (1) Hadoop and (2) a BigData replay platform 05
◦ bigdata replay on multicore, data streaming sketches, etc.
• I want to let other people/clouds to schedule jobs on my platform
• I can let VMs visit with my platform to:
1. run jobs locally
2. process data and run more jobs
3. ...
• I need cloud coin as a quota meter
◦ each VM has to let me know where its coin is before it visits
05 myself+0 "Replayable BigData for Multicore Processing and Statistically Rigid Sketching" IC2014 (2014)
M.Zhanikeev -- maratishe@gmail.com -- VM Visitation and Resource Micropayments ... towards Cloud Federations -- http://bit.do/marat150227 8/21
...
8/21

9. .
2 Demos: BigData Replay, Cloud Coin
M.Zhanikeev -- maratishe@gmail.com -- VM Visitation and Resource Micropayments ... towards Cloud Federations -- http://bit.do/marat150227 9/21
...
9/21

10. .
Step 1: VM Migration Automation
• the problem is to receive and run VM image as
.xva in any network environment
• PV_args is the easiest way, visitation
tools in VM parse network environment in /
proc/cmdline
1. xe... vm-import filename=?.xva
2. xe ... PV_args
3. xe ... vm-start
• when done, check if VM is on the network and
notify the remote client
M.Zhanikeev -- maratishe@gmail.com -- VM Visitation and Resource Micropayments ... towards Cloud Federations -- http://bit.do/marat150227 10/21
...
10/21

11. .
Step 2: .xva diffs
• idea: keep a library of
reference .xva files
and send only diffs for
migration
• 90-95% reduction in
volume
• diff is definitely underrated
and underused in clouds
today 03
03 myself+0 "「Cloudy会」DIFF法：テキスト・ファイルシステム・バイナリ差分" http://goo.gl/v8kLpQ (2015)
M.Zhanikeev -- maratishe@gmail.com -- VM Visitation and Resource Micropayments ... towards Cloud Federations -- http://bit.do/marat150227 11/21
...
11/21

12. .
Step 3: the DiffHub technology
Cloud
Drive
User A
User B
New
Old
Old
Diff Diff
Old version
New version
Binary diff
DiffHub
• work in progress
• a DiffHub for federations
• a file sync
technology where only
diffs are transferred
• biggest merit: master
storage is for diffs only
M.Zhanikeev -- maratishe@gmail.com -- VM Visitation and Resource Micropayments ... towards Cloud Federations -- http://bit.do/marat150227 12/21
...
12/21

13. .
Step 3b: Docker wants diffs?
• Docker's main feature is filesystem diffs
◦ .. which is a redundancy avoidance technology
• binary diffs can be more efficient than Docker filesystem layers
M.Zhanikeev -- maratishe@gmail.com -- VM Visitation and Resource Micropayments ... towards Cloud Federations -- http://bit.do/marat150227 13/21
...
13/21

14. .
Practical Visitation: Hadoop
• problem: very small public
IP space (security, rules, etc.)
• solution: run the VMs in
private space with Internet
access
• demerit: outside cannot
push into your VM, but your
VM can pull anything
M.Zhanikeev -- maratishe@gmail.com -- VM Visitation and Resource Micropayments ... towards Cloud Federations -- http://bit.do/marat150227 14/21
...
14/21

15. .
Practical Visitation: Hadoop (2)
• VM installs visitation tools,
which know where to find the
API for its PM
• the API is generic --
provides any functionality
• storage, sensors, indexing,
hadoop jobs, etc.
M.Zhanikeev -- maratishe@gmail.com -- VM Visitation and Resource Micropayments ... towards Cloud Federations -- http://bit.do/marat150227 15/21
...
15/21

16. .
Public vs Private Cloud (looks like this)
• you need at least 2 PMs, 1 public + 1 private
• note: hide your PM! (non-public IP on the
same subnet)
M.Zhanikeev -- maratishe@gmail.com -- VM Visitation and Resource Micropayments ... towards Cloud Federations -- http://bit.do/marat150227 16/21
...
16/21

17. .
Practical Visitation : Cloud Coins
M.Zhanikeev -- maratishe@gmail.com -- VM Visitation and Resource Micropayments ... towards Cloud Federations -- http://bit.do/marat150227 17/21
...
17/21

18. .
Push vs Pull
M.Zhanikeev -- maratishe@gmail.com -- VM Visitation and Resource Micropayments ... towards Cloud Federations -- http://bit.do/marat150227 18/21
...
18/21

19. .
Visitation Platform: PULL Type
Cloud Visitation Platform
App App App …
Pull-Based
Installation
(of Apps)
End
User
Device
Provider3rd Party
Service
Local
services
One
Cloud
• PULL: heterogeneous,
large-scale cloud
federations, fog computing
• active probing network, sensor
cloud, vehicular cloud, etc.
M.Zhanikeev -- maratishe@gmail.com -- VM Visitation and Resource Micropayments ... towards Cloud Federations -- http://bit.do/marat150227 19/21
...
19/21

20. .
Visitation Platform: PUSH Type
Cloud Visitation Platform
App App App …
Cloud/device
Owner
3rd Party
Service Federated
Clouds
Local
services
Federated
Cloud
Manager
• PUSH: trusted cloud
federation between
large providers
• DC-DC cross-provider
clouds
M.Zhanikeev -- maratishe@gmail.com -- VM Visitation and Resource Micropayments ... towards Cloud Federations -- http://bit.do/marat150227 20/21
...
20/21

21. .
That’s all, thank you ...
M.Zhanikeev -- maratishe@gmail.com -- VM Visitation and Resource Micropayments ... towards Cloud Federations -- http://bit.do/marat150227 21/21
...
21/21