3-Way Scripts as a Base Unit for Flexible Scale-Out Code

2015/12/15
Marat Zhanikeev
maratishe@gmail.com
SE研@博多シティ
PDF: bit.do/151215
3-Way Scripts as a Base Unit
for Flexible Scale-Out Code

Things You Might've Noticed
• most coding today is done in scripting languages -- php, python, ruby, ....
• DiY deploy is becoming more common than platforms -- Docker deploy via filesystem diffs
• smartphone-centric coding is already a reality and reaches further into
IoT, IoV, etc.
• .... a demo?
M.Zhanikeev -- maratishe@gmail.com -- 3-Way Scripts as a Base Unit for Flexible Scale-Out Code -- bit.do/151215 2/20
2/20

3-Way Scripts
function callme (…)
…
Class NAME
Other code,
Manager
CLI mode
$a = new NAME()
$a ->callme ();
HTTP mode
$a = new NAME()
echo toJSON ( $a ->callme ());
OperatorRemote code
Use in
Object mode
01 M.Zhanikeev "Github Public Repository for the 3-Way Scripting Project" https://github.com/maratishe/3wayscripting (current)
3/20

The Need (also for Speed)
• 2 ways wanted 06 詳細 : high-thru local vs low-thru remote calls to A
A
B
C
Many Many
HTTP REQ
HTTP REP (small data)
new,
call
CLI
High
volume
Big Data
Manager
06 M.Zhanikeev "Streaming Algorithms for Big Data Processing on Multicore" CRC (2015)
4/20

Objectives
1. heterogeneity
◦ in clouds known as federated and fog clouds 03
◦ in coding, ambient 18 and p2p networking between instances/agents 15
◦ ultimately VNE 詳細 for distributed software agents
2. volatility
◦ again, fog clouds 03 ambient programming 18
◦ hierarchical structures -- like Fractal 14
3. autonomy
◦ not much in programming but more actively discussed in clouds, network
management, etc. 09
◦ distant goal is Truly Autonomous 詳細
03 M.Zhanikeev "A Cloud Visitation Platform to Facilitate Cloud Federation and Fog Computing" IEEE Computer (2015)
18 J.Dedecker+4 "Ambient-Oriented Programming in AmbientTalk" 20th ECOOP (2006)
15 M.Albano+2 "Hierarchical P2P Overlays for DVE: An Additively Weighted Voronoi Based Approach" ICUMT (2009)
14 F.Baude+2 "From Distributed Objects to Hierarchical Grid Components" CoopIS, DOA, and ODBASE, Springer LNCS (2003)
09 M.Zhanikeev "Population Management in Clouds is a Do-It-Yourself Technology" IETF94/NMRG (2015)
5/20

Theory of Distributed Computing
A
Platform Platform
Bc
C
Run
A. CORBA model
A B
B. Current (traditional) Model
Platform
C. Distributed Code (proposed)
A
Platform Platform
B
C
Run
3-way code
Distribution borders
(network, etc.)
Code (objects, scripts)
Data, messages
Runtime
Invocation routes
XYZ
• CORBA used to
stringify and
send objects
• modern platforms
mostly send data
• 3-way code is
unique
◦ platformless
◦ autonomous
◦ volatility OK =
adaptible
◦ ...
6/20

Class/Function Call-by-Variable
• most scripting languages (PHP) allow for some nifty referencing techniques
• key to using the same code in 3+ ways
$a = new NAME();
$b = ‘callme’;
$a->$b( $param, $defaultparam = 10);
7/20

Basic Security for Web API
AB
Many Many
REQ( md5p , …)
Make
secure
Operator,
Manager
mp5p
mp5 (ﬁle)
Attacker
Manager
8/20

More Specific on MD5 Keys
BA
Each Each
REQ( key , …)
keyﬁle
Operator/
automation
(once)c
web.php make()
verify
same dir.
Attacker
Manager
(key exchange)
9/20

web.php key management
• see source code at 01
• key made by make() is mapped to tag, then to place
• load() is internal, used for de-mapping from place/tag
make( $libdir,$stuff,$length=10){ // returns $key
tag( $key,$tag,$cldir=‘.’){ // outgoing
place( $name,$iport,$tag,$cldir=‘.’){ // outgoing
private load( $place,$tag,$cldir=‘.’){ // outgoing
01 M.Zhanikeev "Github Public Repository for the 3-Way Scripting Project" https://github.com/maratishe/3wayscripting (current)
10/20

web.php mid-way interface
• remote calls can be reduced to remote procedure and RESTful calls
• rrun() : communicate to remote web.php and have it run something on local
commandline
• rcall() : send a RESTful REQ directly to place and return results as JSON
server( $port) { // use own IP
rrun( $place,...) { // $place maps to $tag, then $key
rcall( $place,...) {
11/20

web.php mid-way interface (2)
A
Each Each
REQ( key , …)
C keyfile
Operator/
automation
b
make()
Attacker
D keyfile
E keyfile
keyfile
(global)
12/20

Wrapup : Feature Comparison
• note the learning curve -- very important for modern programming and edge
boxes 詳細
• traffic efficiency 詳細 is not mentioned, but is part of current/future work 02
• destined for wireless fog boxes which need platformless agents 詳細
(Web)
Sockets
RESTful
calls
Hierarchical
structure
Heterogeneous
environments
Learning
curve
(simplicity)
Needs a
platform?
Traffic
encoding
optional?
Traditional
(Corba, ESB, MPI,…) YES NO NO NO NO YES NO
Advanced/modern
(SOC, Ibis, …) YES YES YES YES NO YES NO
Distributed code
(proposed) YES YES YES YES YES NO YES
02 M.Zhanikeev "DiffHub: An Efficient Cloud Sync Technology based on Binary Diffs" IPSJ JIP (2015)
13/20

That’s all, thank you ...
14/20

Application: Hadoop Interface
Hadoop Agent
... is software that runs locally as an agent of the
remote client
Storage Node
(shard)
Time-Aware
Sub-Store(s)
Manager
Client Machine
Client
Your
Sketcher
You
Start Use
Schedule
Multicore
Replay
Replay Node
many
• Hadoop interface is very
intensive, cannot be done
remotely
• there are also alternatives
to Hadoop 06
• one of applications for 3-way
scripts 戻る
06 M.Zhanikeev "Streaming Algorithms for Big Data Processing on Multicore" CRC (2015)
15/20

VNE: Virtual Network Embedding
VNE
... is the best way to formulate optimization of
multiple overlayed virtual (service) networks
Physical
layer
Virtual
Layer1
VN
Request
2
Make VN
3 VN
Reply
4 Virtual
Virtual
Physical
• still active area of research,
good heuristics wanted 07
• best for service networks in
clouds 戻る
07 M.Zhanikeev "A New VNE Method for More Responsive Networking in Many-to-Many Groups" 7th ICUFN (2015)
16/20

Truly Autonomous Agents
True Autonomy
... is alternative to VNE in that optimization is
done autonomously at service side
Network
Provider s
(NOC)
Agent
Service Traditional agents
True Autonomy
• part of current standards
discussion on future
network management 09
• traditional: NOC + agents
but NOC is interface =
bottleneck
• true autonomy: agents are
open for direct
coordination with remove
services 戻る
09
09 M.Zhanikeev "Population Management in Clouds is a Do-It-Yourself Technology" IETF94/NMRG (2015)
17/20

Wireless Network Edge
Reachable Devices
... are boxes that talk to your smartphone using
P2P WiFi (WiFi Direct is best today)
AP
User
Internet
User
Internet
WiFi
Congestion
User
Internet
3G
Smartphone
Device
Device
Device
AP Design
WLAN Design
Reachable
Devices Design
AP
DeviceWiFi User
Internet
AP Design (2)
Congestion
WiFi
P2P WiFi
Cannot use
at the same time
Reachable Platform
• modern boxes are mostly based on
the AP Design -- connectivity problem
• even if conenctivity problem is
resolved, you have congestion
problem
• with Reachable Technology,
smartphone is the realtime mid-way
interface between clouds and
devices
• link to code: software agent is
running at (1) smartphone and (2)
the box 戻る
18/20

Example Edge Box for 3-Way Scripts
• currently trying to build this box based on conventional Xen
• LHAP is Local Hardware Awareness Platform 03
• software agents are running at the box, those are Beacons, Wireless Data
Hubs, etc. 戻る
WiFi
Wireless users
Wireless
AP
Physical Device
LHAP
VM
VM
Con.Con.Con.
Storage
Sensors
… Beacon
WiFi AP
WiFi Client
P2P WiFi
03 M.Zhanikeev "A Cloud Visitation Platform to Facilitate Cloud Federation and Fog Computing" IEEE Computer (2015)
19/20

Binary Diffs and DiffHub Idea
Binary Diffs
... are the best way to achieve traffic
efficiency in a datatype-indiferent way
Cloud
Drive
User A
User B
New
Old
Old
Diff Diff
Old version
New version
Binary diff
DiffHub
• exampe: Docker uses filesystem
diffs
• recent study: binary diffs are better
than filesystem diffs
• connection: efficient deploy/
maintain of scale-out code 戻る
20/20

3-Way Scripts as a Base Unit for Flexible Scale-Out Code

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (7)

Similar to 3-Way Scripts as a Base Unit for Flexible Scale-Out Code

Similar to 3-Way Scripts as a Base Unit for Flexible Scale-Out Code (20)

More from Tokyo University of Science

More from Tokyo University of Science (20)

Recently uploaded

Recently uploaded (20)

3-Way Scripts as a Base Unit for Flexible Scale-Out Code