Presentation by Cees de Laat at SURFacademy Autumn School ' Tools for Researchers' 3 November 2009, at Leiden University Library

1. Optical Networks, LightPaths"
and Grid Computing
Cees de Laat
!

2. 100000 flops/byte!

3. The SCARIe project
SCARIe: a research project to create a Software Correlator for e-VLBI.
VLBI Correlation: signal processing technique to get high precision image from
spatially distributed radio-telescope.
Telescopes ..... To equal the hardware
correlator we need:
16 streams of 1Gbps
Input nodes
16 * 1Gbps of data
Correlator nodes 2 Tflops CPU power
2 TFlop / 16 Gbps =
Output node 1000 flops/byte!
THIS IS A DATA FLOW
PROBLEM !!!

4. LOFAR as a Sensor Network
!
20 flops/byte!
– LOFAR is a large distributed research
infrastructure:!
• Astronomy:!
– >100 phased array stations!
– Combined in aperture synthesis array!
– 13,000 small “LF” antennas!
– 13,000 small “HF” tiles!
• Geophysics:!
– 18 vibration sensors per station!
– Infrasound detector per station!
• >20 Tbit/s generated digitally!
• >40 Tflop/s supercomputer!
• innovative software systems!
– new calibration approaches!
– full distributed control!
– VO and Grid integration!
– datamining and visualisation!
Slide courtesy of Marco de Vos (LOFAR)

5. !
US and International OptIPortal Sites
NCSA &
SIO NCMIR USGS EDC TRECC
KISTI AIST RINCON & Nortel
SARA
TAMU UCI UIC CALIT2
Real time, multiple 10 Gb/s!

6. The “Dead Cat” demo! 1 Mflops/byte!
SC2004,
Pittsburgh,
Nov. 6 to 12, 2004
iGrid2005,
San Diego,
sept. 2005
Many thanks to:
AMC
SARA
GigaPort
UvA/AIR
Silicon Graphics,
Inc.
Zoölogisch Museum
M. Scarpa, R.G. Belleman, P.M.A. Sloot and C.T.A.M. de Laat, "Highly Interactive Distributed Visualization",
iGrid2005 special issue, Future Generation Computer Systems, volume 22 issue 8, pp. 896-900 (2006).!

7. IJKDIJK!
300000 * 60 kb/s * 2 sensors (microphones) to cover all Dutch dikes!

8. Sensor grid: instrument the dikes!
First controlled breach occurred on sept 27th ‘08:!
Many small flows -> 36 Gb/s!

9. CosmoGrid Observed
! Motivation:
previous simulations
Simulated
found >100 times more
substructure than is
observed!
! Simulate large structure formation in the Universe
- Dark Energy (cosmological constant)!
- Dark Matter (particles)!
! Method: Cosmological N-body code
! Computation: Intercontinental SuperComputer Grid

10. 10 Mflops/byte!
The hardware setup
! 2 supercomputers :
- 1 in Amsterdam (60Tflops Power6 @ SARA)!
- 1 in Tokyo (30Tflops Cray XD0-4 @ CFCA)!
! Both computers are connected via an
intercontinental optical 10 Gbit/s network
270 ms RTT!

11. 7.6 Gb/s!
CineGrid @ Holland Festival 2007

12. Why is more resolution is better?
1. More Resolution Allows Closer Viewing of Larger Image
2. Closer Viewing of Larger Image Increases Viewing Angle
3.Visual acuity=1.0=20/20"
Increased Viewing Angle Produces Stronger Emotional Response
Standard viewing distance" 1920 "
1080"
HDTV (2K 30º"
UHDTV(8K)" 24 Gb/s!
7680 " 3.0 × Picture Height"
SHD (4K)"
4320"
3840 "
2160"
7.6 Gb/s!
0.75 × Picture Height" 100º" 60º"
Yutaka TANAKA
SHARP CORPORATION
Advanced Image Research Laboratories
1.5 × Picture Height"

13. 100 Tbyte!
CineGrid portal! Cache & Store & Forward!

14. #! A. Lightweight users, browsing, mailing, home use!
u! Need full Internet routing, one to all!
s!
e! B. Business/grid applications, multicast, streaming, VO’s, mostly LAN!
r!
s! Need VPN services and full Internet routing, several to several + uplink to all!
C. E-Science applications, distributed data processing, all sorts of grids!
Need very fat pipes, limited multiple Virtual Organizations, P2P, few to few!
For the Netherlands 2007
ΣA = ΣB = ΣC " 250 Gb/s!
However:!
• A -> all connects!
A!
• B -> on several!
• C -> just a few (SP, LHC, LOFAR)!
B! C!
ADSL (12 Mbit/s)! GigE!
BW!
Ref: Cees de Laat, Erik Radius, Steven Wallace, "The Rationale of the Current Optical Networking Initiatives”!
iGrid2002 special issue, Future Generation Computer Systems, volume 19 issue 6 (2003)!

15. Towards Hybrid Networking!
!
• Costs of photonic equipment 10% of switching 10 % of full routing!
– for same throughput!!
– Photonic vs Optical (optical used for SONET, etc, 10-50 k$/port)!
– DWDM lasers for long reach expensive, 10-50 k$!
• Bottom line: look for a hybrid architecture which serves all classes in a cost
effective way !
– map A -> L3 , B -> L2 , C -> L1 and L2!
• Give each packet in the network the service it needs, but no more !!
L2 " 5-8 k$/port! L3 " 75+ k$/port!
L1 " 2-3 k$/port!

16. How low can you go?
!
Application! Local! MEMS! 15454! Application!
Endpoint A! Ethernet! 6500! Endpoint B!
Regional! HDXc!
POS! dark! Trans-Oceanic!
fiber!
Router
!
Ethernet
!
SONET!
DWDM
GLIF!
Fiber
!

17. Visualization courtesy of Bob Patterson, NCSA
GLIF 2008! Data collection by Maxine Brown.

19. DataExploration
RemoteControl Management
Backup
TV
Mining Media
Gaming
Medical
Web2.0 Visualisation
CineGrid
Security
Conference Meta
NetherLight!
Workflow
Clouds
Distributed
Simulations
EventProcessing
StreamProcessing
Predictions

20. SURFnet
!
In The Netherlands SURFnet
connects between 180:
- universities;
- academic hospitals;
- most polytechnics;
- research centers.
x!
x!
x! with an indirect ~750K user
base
x!
x!
2 ms! ~ 8860 km!
3 ms!
scale
comparable
to railway
system!

21. Network Description Language
• From semantic Web / Resource Description Framework.
• The RDF uses XML as an interchange syntax.
• Data is described by triplets:
Predicate
Subject Object
Object
Subject Object
Subject Subject
Object
Object
Subject
Subject
Location! Device! Interface! Link!
name! description! locatedAt! hasInterface!
connectedTo! capacity! encodingType! encodingLabel!

22. Network Description Language
Article: F. Dijkstra, B. Andree, K.
Choice of RDF instead of flat XML descriptions Koymans, J. van der Ham, P. Grosso,
C. de Laat, "A Multi-Layer Network
Grounded modeling based on G0805 description: Model Based on ITU-T G.805"

23. Multi-layer descriptions in NDL!
IP layer!
Ethernet layer!
STS layer!
OC#$%& layer!
UTP! layer!
"ber! layer!
SONET switch SONET switch
Ethernet &!
End with! SONET with! End
SONET switch!
host! Ethernet intf.! switch! Ethernet intf.! host!
Université NetherLight! Universiteit !
CA Net! StarLight! MAN LAN! van !
du Quebec! Canada! Chicago! New York! Amsterdam! Amsterdam!

24. The Problem!
I want HC and AB!
Success depends on the order!
Wouldn’t it be nice if I could request [HC, AB, ...]!
A B
H ?! C
G
D
F
E

25. Another one "!
I want AB and CD!
Success does not even depend on the order!!!!
A B
I J
E F G H
K L
C D

26. NDL + PROLOG!
Research Questions:!
• order of requests!
• complex requests!
• usable leftovers!
• Reason about
graphs!
• Find sub-graphs
that comply with
rules !

27. RDF describing Infrastructure
“I want”
Application: find video containing x,
then trans-code to it view on Tiled Display
RDF/CG! RDF/VIZ!
RDF/CG!
RDF/ST!
RDF/NDL! RDF/CPU!
RDF/NDL!
content
content
COCE

28. Linking content
to networks!

29. TeraThinking!
• What constitutes a Tb/s network?!
• CALIT2 has 8000 Gigabit drops ?->? Terabit Lan?!
• look at 80 core Intel processor!
– cut it in two, left and right communicate 8 TB/s!
• think back to teraflop computing!!
– MPI turns a room full of pc’s in a teraflop machine!
• massive parallel channels in hosts, NIC’s!
• TeraApps programming model supported by!
– TFlops ! !-> !MPI / Globus!
– TBytes ! !-> !OGSA/DAIS!
– TPixels ! !-> !SAGE!
– TSensors !-> !LOFAR, LHC, LOOKING, CineGrid, ...!
– Tbit/s ! !-> !?!
ref Larry Smarr & CdL!

30. User Programmable Virtualized Networks allows the results of
decades of computer science to handle the complexities of
!
application specific networking.
• The network is virtualized as a collection of
resources!
• UPVNs enable network resources to be
programmed as part of the application!
• Mathematica, a powerful mathematical software
system, can interact with real networks using
UPVNs!
application application
nc nc nc
nc nc nc nc
ac ac ac ac
network network network network
element element element element

31. Mathematica enables advanced graph queries, visualizations and real-
time network manipulations on UPVNs#
Topology matters can be dealt with algorithmically
Results can be persisted using a transaction service built in UPVN
Initialization and BFS discovery of NEs 139.63139.63139.63
.145.2.145.17.145.16 192.168.1.3
139.63.145.32 139.63 .145.15
192.168.1.1
Needs["WebServices`"] 139.63.145.34 139.63.145.18 139.63
139.63.145.31 .145.0
<<DiscreteMath`Combinatorica` 139.63.145.38 139.63 .145.1
<<DiscreteMath`GraphPlot`
InitNetworkTopologyService["edge.ict.tno.nl"]
139.63.145.52
139.63 139.63
.145.33 .145.3
139.63.145.86 Visualisation.0.3
192.168
192.168.1.4
139.63.145.51 139.63.145.82 192.168.1.2
139.63.145.63 139.63 .145.79
Available methods: 192.168.0.4
139.63.145.40 139.63.145.87
139.63 .145.50
{DiscoverNetworkElements,GetLinkBandwidth,GetAllIpLinks,Remote, 139.63.145.64 139.63 .145.46
139.63.145.94
139.63.145.83
192.168.0.1 192.168.0.2
NetworkTokenTransaction}
139.63.145.49 139.63.145.88
192.168.0.6
139.63.145.65 139.63.145.45 .145.75
139.63 192.168.2.2
Global`upvnverbose = True; 139.63.145.84 192.168.2.4
139.63.145.41 192.168.0.5
139.63.145.66 139.63.145.81
AbsoluteTiming[nes = BFSDiscover["139.63.145.94"];][[1]] 139.63.145.85
139.63.145.44
139.63.145.42
139.63.145.43 139.63.145.73
AbsoluteTiming[result = BFSDiscoverLinks["139.63.145.94", nes];][[1]] 139.63.145.74
192.168.2.1
139.63 .145.68 139.63.145.72
139.63 .145.69 192.168.2.3
139.63139.63.145.71
.145.70
Getting neigbours of: 139.63.145.94
Internal links: {192.168.0.1, 139.63.145.94}
(...) 192.168.1.3
Getting neigbours of:192.168.2.3
192.168.1.1
Internal links: {192.168.2.3}
99.8
Network flows using real-time
bandwidth measurements
100.
Transaction on shortest path with tokens 192.168.1.4
100.
192.168.1.2
192.168.0.3
nodePath = ConvertIndicesToNodes[ 100. 192.168.0.4
99.9 94.5
ShortestPath[ g, 99. 95.8
Node2Index[nids,"192.168.3.4"], 100. 98.9 192.168.0.2 99.9 95.9 192.168.0.1 139.63.145.94
Node2Index[nids,"139.63.77.49"]],
nids]; 100. 96.
192.168.0.6
Print["Path: ", nodePath]; 192.168.0.5
192.168.2.2
If[NetworkTokenTransaction[nodePath, "green"]==True, 192.168.2.4
Print["Committed"], Print["Transaction failed"]]; 100.
101.
Path: 100.
192.168.2.1
{192.168.3.4,192.168.3.1,139.63.77.30,139.63.77.49} 100.
192.168.2.3
Committed
ref: Robert J. Meijer, Rudolf J. Strijkers, Leon Gommans, Cees de Laat, User Programmable Virtualiized Networks,
accepted for publication to the IEEE e-Science 2006 conference Amsterdam.

32. Interactive programmable networks

33. e-Biobanking"
e-BioScience"
e-Ecology"
e-COAST"
sciences "
intensive
e-Data-
e-Food"
.....
"
Virtual Laboratory"
generic e-Science services"
High Performance & Distributed Computing"
Web & Grid services"

34. Questions ?!
A Declarative Approach to Multi-Layer Path Finding Based on Semantic Network Descriptions.!
http://delaat.net:/~delaat/papers/declarative_path_finding.pdf!
Thanks: Paola Grosso & Jeroen vd Ham & Freek
Dijkstra & team for several of the slides.!