This document discusses the evolution of distributed systems over time. It describes how advances in networking, computing power, storage, and protocols have enabled new capabilities. Specifically, it notes that networking speeds increased 348,000x from 1973 to 2005, storage capacity increased 30,000x and became 131,000x cheaper from 1977 to 2008, and computing power became smaller, cheaper, and faster over time. This allowed distributed systems to become more feasible and widespread.

1. Page 1Page 1
Distributed Systems

2. Page 2Page 2
What can we do now
that we could not do before?

3. Page 3
Technology advances
Processors
Memory
Networking
Storage
Protocols

4. Page 4
Networking: Ethernet - 1973, 1976
June 1976: Robert Metcalfe presents the concept of
Ethernet at the National Computer Conference
1980: Ethernet introduced as de facto standard (DEC,
Intel, Xerox)

5. Page 5
Network architecture
LAN speeds
• Original Ethernet: 2.94 Mbps
• 1985: thick Ethernet: 10 Mbps
1 Mbps with twisted pair networking
• 1991: 10BaseT - twisted pair: 10 Mbps
Switched networking: scalable bandwidth
• 1995: 100 Mbps Ethernet
• 1998: 1 Gbps (Gigabit) Ethernet
• 1999: 802.11b (wireless Ethernet) standardized
• 2001: 10 Gbps introduced
• 2005: 100 Gbps (over optical link)
348->35,000x
faster
348->35,000x
faster

6. Page 6
Network ConnectivityThen:
• large companies and universities on Internet
• gateways between other networks
• dial-up bulletin boards
• 1985: 1,961 hosts on the Internet
Now:
• One Internet (mostly)
• 2006: 439,286,364 hosts on the Internet
• widespread connectivity
High-speed WAN connectivity: 1– >50 Mbps
• Switched LANs
• wireless networking
439 million
more hosts
439 million
more hosts

7. Page 7
Computing power
Computers got…
•Smaller
•Cheaper
•Power efficient
•Faster
Microprocessors became technology leaders

8. Page 8
year $/MB typical
1977 $32,000 16K
1987 $250 640K-2MB
1997 $2 64MB-256MB
2007 $0.06 512MB-2GB+
9,000x cheaper
4,000x more capacity
9,000x cheaper
4,000x more capacity

9. Page 9
Storage: disk
131,000x cheaper in 20 years
30,000x more capacity
131,000x cheaper in 20 years
30,000x more capacity
Recording density increased over
60,000,000 times over 50 years
1977: 310KB floppy drive –
$1480 1987: 40 MB drive for –
$679
2008: 750 GB drive for – $99
$0.13 / GB

10. Page 10
Music Collection4,207 Billboard hits
• 18 GB
• Average song size: 4.4 MB
Today
• Download time per song @12.9 Mbps: 3.5 sec
• Storage cost: $5.00
20 years ago (1987)
• Download time per song, V90 modem @44 Kbps:
15 minutes
• Storage cost: $76,560

11. Page 11
Protocols
Faster CPU →
more time for protocol processing
• ECC, checksums, parsing (e.g. XML)
• Image, audio compression feasible
Faster network →
→ bigger (and bloated) protocols
• e.g., SOAP/XML, H.323

12. Page 12
Why do we want to network?• Performance ratio
• Scaling multiprocessors may not be possible or cost effective
• Distributing applications may make sense
• ATMs, graphics, remote monitoring
• Interactive communication & entertainment
• work and play together:
email, gaming, telephony, instant messaging
• Remote content
• web browsing, music & video downloads, IPTV, file servers
• Mobility
• Increased reliability
• Incremental growth

13. Page 13
Problems
Designing distributed software can be difficult
• Operating systems handling distribution
• Programming languages?
• Efficiency?
• Reliability?
• Administration?
Network
• disconnect, loss of data, latency
Security
• want easy and convenient access

14. Page 14Page 14
Building and classifying
distributed systems

15. Page 15
Flynn’s Taxonomy (1972)
SISD
• traditional uniprocessor system
SIMD
• array (vector) processor
• Examples:
• APU (attached processor unit in Cell processor)
• SSE3: Intel’s Streaming SIMD Extensions
• PowerPC AltiVec (Velocity Engine)
MISD
• Generally not used and doesn’t make sense
• Sometimes applied to classifying redundant systems
MIMD
• multiple computers, each with:
• program counter, program (instructions), data
• parallel and distributed systems
number of instruction streams
and number of data streams

16. Page 16
Subclassifying MIMD
memory
• shared memory systems: multiprocessors
• no shared memory: networks of computers, multicomputers
interconnect
• bus
• switch
delay/bandwidth
• tightly coupled systems
• loosely coupled systems

17. Page 17
BusBus
Bus-based multiprocessors
CPU ACPU A
SMP: Symmetric Multi-Processing
All CPUs connected to one bus (backplane)
Memory and peripherals are accessed via shared bus.
System looks the same from any processor.
CPU BCPU B
memorymemory
Device
I/O
Device
I/O

18. Page 18
Bus-based multiprocessors
Dealing with bus overload
- add local memory
CPU does I/O to cache memory
- access main memory on cache miss
BusBus
memorymemory
Device
I/O
Device
I/O
CPU ACPU A
cachecache
CPU BCPU B
cachecache

19. Page 19
Working with a cache
CPU A reads location 12345 from memory
12345:712345:7
Device
I/O
Device
I/O
CPU ACPU A
12345: 712345: 7
CPU BCPU B
Bus

20. Page 20
Working with a cache
CPU A modifies location 12345
BusBus
12345:712345:7
Device
I/O
Device
I/O
CPU ACPU A
12345: 7
CPU BCPU B
12345: 312345: 3

21. Page 21
Working with a cache
CPU B reads location 12345 from memory
12345:712345:7
Device
I/O
Device
I/O
CPU ACPU A
12345: 312345: 3
CPU BCPU B
12345: 712345: 7
Gets old value
Memory not coherent!
Bus

22. Page 22
Write-through cache
Fix coherency problem by writing all values
through bus to main memory
12345:7
Device
I/O
Device
I/O
CPU ACPU A
12345: 7
CPU BCPU B
CPU A modifies location 12345 – write-through
main memory is now coherent
12345: 312345: 3
12345:312345:3
Bus

23. Page 23
Write-through cache … continued
CPU B reads location 12345 from memory
- loads into cache
12345:312345:3
Device
I/O
Device
I/O
CPU ACPU A
12345: 312345: 3
CPU BCPU B
12345: 312345: 3
Bus

24. Page 24
Write-through cache
CPU A modifies location 12345
- write-through
12345:3
Device
I/O
Device
I/O
CPU ACPU A
12345: 3
CPU BCPU B
12345: 312345: 3
Cache on CPU B not updated
Memory not coherent!
12345:012345:0
12345: 012345: 0
Bus

25. Page 25
Snoopy cache
Add logic to each cache controller:
monitor the bus
12345: 3
Device
I/O
Device
I/O
CPU ACPU A
12345: 3
CPU BCPU B
12345: 3
write [12345]← 0
12345: 3
Virtually all bus-based architectures
use a snoopy cache
Bus
12345: 012345: 0
12345: 012345: 0
12345: 012345: 0

26. Page 26
Switched multiprocessors
•Bus-based architecture does not scale to a
large number of CPUs (8+)

27. Page 27
Switched multiprocessorsDivide memory into groups and connect chunks
of memory to the processors with a crossbar
switch
n2
crosspoint switches – expensive switching fabric
CPUCPU
CPUCPU
CPUCPU
CPUCPU
memmem memmem memmem memmem

28. Page 28
Crossbar alternative: omega network
Reduce crosspoint switches by adding
more switching stages
CPUCPU
CPUCPU
CPUCPU
CPUCPU
memmem
memmem
memmem
memmem

29. Page 29
Crossbar alternative: omega network
with n CPUs and n memory modules:
need log2n switching stages,
each with n/2 switches
Total: (nlog2n)/2 switches.
• Better than n2
but still a quite expensive
• delay increases:
1024 CPU and memory chunks
overhead of 10 switching stages to memory and 10 back.
CPUCPU
CPUCPU
CPUCPU
CPUCPU
memmem
memmem
memmem
memmem

30. Page 30
NUMA
• Hierarchical Memory System
• Each CPU has local memory
• Other CPU’s memory is in its own address space
• slower access
• Better average access time than omega network if most accesses are
local
• Placement of code and data becomes difficult

31. Page 31
NUMA
• SGI Origin’s ccNUMA
• AMD64 Opteron
• Each CPU gets a bank of DDR memory
• Inter-processor communications are sent over a HyperTransport link
• Linux 2.5 kernel
• Multiple run queues
• Structures for determining layout of memory and processors

32. Page 32
Bus-based multicomputers• No shared memory
• Communication mechanism needed on bus
• Traffic much lower than memory access
• Need not use physical system bus
• Can use LAN (local area network) instead

33. Page 33
Bus-based multicomputers
Collection of workstations on a LAN
InterconnectInterconnect
CPUCPU
memorymemory
LAN
connector
LAN
connector
CPUCPU
memorymemory
LAN
connector
LAN
connector
CPUCPU
memorymemory
LAN
connector
LAN
connector
CPUCPU
memorymemory
LAN
connector
LAN
connector

34. Page 34
Switched multicomputers
Collection of workstations on a LAN
CPUCPU
memorymemory
LAN
connector
LAN
connector
CPUCPU
memorymemory
LAN
connector
LAN
connector
CPUCPU
memorymemory
LAN
connector
LAN
connector
CPUCPU
memorymemory
LAN
connector
LAN
connector
n-port switchn-port switch

35. Page 35
Software
Single System Image
Collection of independent computers that appears as a single system to the
user(s)
• Independent: autonomous
• Single system: user not aware of distribution
Distributed systems software
Responsible for maintaining single system image

36. Page 36
You know you have a distributed
system when the crash of a
computer you’ve never heard of
stops you from getting any work
done.
– Leslie Lamport

37. Page 37
Coupling
Tightly versus loosely coupled software
Tightly versus loosely coupled hardware

38. Page 38
Design issues: Transparency
High level: hide distribution from users
Low level: hide distribution from software
• Location transparency:
users don’t care where resources are
• Migration transparency:
resources move at will
• Replication transparency:
users cannot tell whether there are copies of resources
• Concurrency transparency:
users share resources transparently
• Parallelism transparency:
operations take place in parallel without user’s knowledge

39. Page 39
Design issuesReliability
• Availability: fraction of time system is usable
• Achieve with redundancy
• Reliability: data must not get lost
• Includes security
Performance
• Communication network may be slow and/or unreliable
Scalability
• Distributable vs. centralized algorithms
• Can we take advantage of having lots of computers?

40. Page 40Page 40
Service Models

41. Page 41
Centralized model
• No networking
• Traditional time-sharing
system
• Direct connection of user terminals to system
• One or several CPUs
• Not easily scalable
• Limiting factor: number of CPUs in system
• Contention for same resources

42. Page 42
Client-server model
Environment consists of clients and servers
Service: task machine can perform
Server: machine that performs the task
Client: machine that is requesting the service
Directory server Print server File server
clientclient clientclient
Workstation model
assume client is used by one user at a time

43. Page 43
Peer to peer model
• Each machine on network has (mostly) equivalent capabilities
• No machines are dedicated to serving others
• E.g., collection of PCs:
• Access other people’s files
• Send/receive email (without server)
• Gnutella-style content sharing
• SETI@home computation

44. Page 44
Processor pool model
What about idle workstations
(computing resources)?
• Let them sit idle
• Run jobs on them
Alternatively…
• Collection of CPUs that can be assigned processes on demand
• Users won’t need heavy duty workstations
• GUI on local machine
• Computation model of Plan 9

45. Page 45
Grid computing
Provide users with seamless access to:
• Storage capacity
• Processing
• Network bandwidth
Heterogeneous and geographically distributed systems

46. Page 46Page 46
Multi-tier client-server
architectures

47. Page 47
Two-tier architecture
Common from mid 1980’s-early 1990’s
• UI on user’s desktop
• Application services on server

48. Page 48
Three-tier architecture
client
middle
tier
middle
tier
back-endback-end
- queueing/scheduling of
user requests
-
transaction processor
(TP)
- Connection mgmt
- Format converision
- queueing/scheduling of
user requests
-
transaction processor
(TP)
- Connection mgmt
- Format converision
- Database
- Legacy
application
processing
- Database
- Legacy
application
processing
- User
interface
- some data
validation/
formatting
- User
interface
- some data
validation/
formatting

49. Page 49
Beyond three tiersMost architectures are multi-tiered
client
web
server
Java
application
server
load
balancer
firewall
firewall database
Object
Store

50. Page 50Page 50
The end.