Gray_Compass99.ppt

Jim Gray, Research and Microsoft SQL Server
Microsoft Research http://research.Microsoft.com/~gray/talks/
3 October 1999
Chicago, Ill.
1
Microsoft SQL Server,
Scalability, &
Database Research
Jim Gray
Researcher
Microsoft Corporation

3 October 1999
Chicago, Ill.
2
Outline
Summary of what you heard. (10 min)
The database scene in general. (10 min)
Scaleability: Farms, Clones,Parts & Packs (15 min)
Microsoft DB research focus. (15 min)
• TerraServer (design and ops).
• RAGS.
• Data Mining
Q&A (10 min)

3 October 1999
Chicago, Ill.
3
Organizations Are Going Online
Building a digital nervous system.
Inexpensive hardware means huge
databases are possible.
But, we are drowning in data.
Databases help organize information.
Microsoft’s goal:
• Information at your fingertips.
• Make it easy to capture,
manage, and
analyze information.

3 October 1999
Chicago, Ill.
4
Microsoft SQL Server 7 Goals
Easy
 Dynamic self management
 Multi-site management
 Operation Scripting
 Job scheduling and execution
 Alert/response management
 Scriptable Install+upgrade
 DBA profiling/tuning tools
 Unicode
 English Language Query
 Integrated with NT Security
 Integrated with NT files
Scalability
Data Warehousing

3 October 1999
Chicago, Ill.
11
Scalability
 Win9x/NTW version
 Dynamic row-level locking
 Improved query optimizer
 Intra-query parallelism
 VLDB improvements
 Replication improvements
 Distributed query
 High Availability Clusters
Easy
Scalability
Data Warehousing

3 October 1999
Chicago, Ill.
12
Scale Down to Windows 95-98
Full function (same as NTW)
Integration with Access 97
MSDE in Office2000 and MSDN
WinCE version demonstrated

3 October 1999
Chicago, Ill.
13
Replication
Transactional and Merge
Remote update
ODBC and OLE DB subscribers
Wizards
Performance
2PC,
RPC
Subscriber
DB2
CICS Subscriber
Subscriber
VSAM
OS 390
DB2
Publisher
Updating Subscriber
(immediate updates)
Distributor
Subscriber

3 October 1999
Chicago, Ill.
14
Query Processor Enhancements
Parallelism
Improved scan, fetch, & sort
Smart hash & merge join
Large joins & grouping
Better query optimization
Multi-index operations
Automatic statistics maintenance
Distributed Query
Heterogeneous Query
Focus on Complex Queries

3 October 1999
Chicago, Ill.
15
•# of emp. per group
•total inc. per group
Local Agg.
4 x 50 rows
+ + + +
Disks
50,000 rows
Global Agg. Result 50 rows
+
Parallel Query
SMP & Disk Parallelism
Plus Distributed
Plus Hash Join (fanciest on the planet)
Plus Optimized Partitioned views

3 October 1999
Chicago, Ill.
16
Distributed Heterogeneous Queries
Data Fusion / Integration
Join spread sheets,
databases,
directories,
Text DBs
etc.
Any source that
exposes OLE DB
interfaces
SQL Server as
gateway,
even on the
desktop
Database
(DB2, VSAM, Oracle, …)
Spreadsheet
Photos
Mail
Maps
Documents
and the Web
Directory
Service
SQL 7.0
Query
Processor

3 October 1999
Chicago, Ill.
17
Utilities
The Key to LARGE Databases
 Backup
• Fuzzy
• Parallel
• Incremental
• Restartable
Recovery
• Fast
• File granularity
Reorganize
• shrinks file
• reclusters file
Auto-Repair
Index creation
~2x faster than 6.5
DBCC
• not required,
• a good practice
• 5x - 100x faster
0
10
20
30
40
50
60
1 2 3 4
SQL Server 6.5
SQL Server 7.0
# of indices
Recovery
time
(
secs
)

3 October 1999
Chicago, Ill.
20
Data Warehousing
 Warehousing Framework
 Visual data modeler
 Microsoft repository
 Data transformation services
(DTS)
 Plato & Dcube - Multi
Dimensional Data Cubes
 English query 2.0
Easy
Scalability
Data Warehousing

3 October 1999
Chicago, Ill.
22
Data Warehouse / Data Analysis
Data Transformation Services
to get data into the warehouse
CUBE (OLE/DB OLAP)
to analyze data
Operational
Data Extact
& Load
Data Warehouse
Storage
OLAP

3 October 1999
Chicago, Ill.
25
Source table
Partition 1
ROLAP
Partition 2
Partition 3
ROLAP
Europe
USA
Asia
MD SQL
SQL
Plato and Data Cube
and HOLAP
“Plato”
server
“Plato”
Designer
Dcube
Client
app
User 1
Dcube
Client
app
User 2
RED
WHITE
BLUE
By Color
By Make & Year
By Color & Year
By Make
By Year
Sum

3 October 1999
Chicago, Ill.
26
English Query

3 October 1999
Chicago, Ill.
27
Easy
Scalable
Data
Warehousing

3 October 1999
Chicago, Ill.
“Shiloh” The Next SQL Server
Shiloh (H1’00) - Strengthen Position
• Data Warehousing leadership
 Materialized Views
 Cascading Referential Integrity
(#1 requested user-group feature)
 XML support
• Scalability
 WinCE support
 W2K VLM (36 and 64 bit)
 Multi-instance support
Yukon – Next Big Step
• Scalability (Clusters, Partitions)
• Programmability
• Ease of Use (Self Tuning, Auto Config)

3 October 1999
Chicago, Ill.
29
Outline
• RAGS.
• Data Mining
Q&A (10 min)

3 October 1999
Chicago, Ill.
30
Info Capture
You can record everything
you see or hear or read.
What will you do with it?
How will you organize &
analyze it?
Most data will never be seen
Analysis an summarization
are key technologies
Video 8 PB per lifetime (10GBph)
Audio 30 TB (10KBps)
Read or write: 8 GB (words)
See: http://www.lesk.com/mlesk/ksg97/ksg.html
Yotta
Zetta
Exa
Peta
Tera
Giga
Mega
Kilo A Book
.Movi
e
All books
(words)
All Books
MultiMedia
Everything
!
Recorded
A Photo

3 October 1999
Chicago, Ill.
31
Data Tidal Wave
 Seagate 47GB drive @ 783$ (= 1.7 ¢/mb)
• 100 GB penny per MB drive coming in 2000
 10 $/GB = 10 k$/ Terabyte!
• “Everyone” can afford one
 What’s a terror bite?
• If you sell ten billion items a year (e.g Wal-Mart)
• And you record 100 bytes on each one
• Then you get a TeraByte/year
 Where will the terror bytes come from?
• Multimedia (like the TerraServer) and...

3 October 1999
Chicago, Ill.
32
Reducing Data’s Cost-of-Ownership
Self-Managing data
Cost of ownership:
One admin/TB (100K$ vs 10K$)
Admin cost exceeds storage cost.
SQL 7:
Suggests indices
Migrates data away from end of file
Truncates file
Someday:
Automatic move files to balance disks
Online defragmentation & restructuring
Online physical redesign

3 October 1999
Chicago, Ill.
33
OBJECT RELATIONAL
The Next Great DBMS Wave
All DB vendors have added objects to DB
Microsoft is adding DBs to Objects
Integration with COM+
Gives user-defined types and objects
Plug-ins will be Billion dollar industry
• Blades for SQL Server razor

3 October 1999
Chicago, Ill.
34
Why Is XML Important?
Self-describing data
“ABC47-Z”, “100”, “STL”, “C”, “3”, “28”
Data stream in a typical interface…
<INVENTORY>
<PART_NUM>ABC47-Z</PART_NUM>
<QUANTITY>100</QUANTITY>
<WAREHOUSE>STL</WAREHOUSE>
<ZONE>C</ZONE>
<AISLE>3</AISLE>
<BIN>28</BIN>
</INVENTORY>
Same data stream in XML…

3 October 1999
Chicago, Ill.
35
table.xsl
bar.xsl
art.xsl

3 October 1999
Chicago, Ill.
36
XML Applications
Exposing Software as a “Service”
• Websites without UI’s
• Exposed services with common scheme
• Integration points at the enterprise, value-
chain, workgroup, desktop and intelligent
gizmo “levels”
B2B value chains
• Uses XML to transmit wide range of date to a
broad set of stakeholders (regulatory
agencies, suppliers, customers, etc.).
• Leverage for prior efforts like EDI
• BizTalk a key industry effort in this regard

3 October 1999
Chicago, Ill.
37
XML Message
XML
Document
XML Message
XML: BizTalk Framework
Order Processing
MVS CICS
SAP R/3
XML
XML
Service
Interface
Browser
Client Apps
New Form Factors
XML
Document
Another Service
JD Edwards
XML
XML
Library
www.biztalk.org
XML
XML
schema

3 October 1999
Chicago, Ill.
38
Outline
• RAGS.
• Data Mining
Q&A (15 min)

3 October 1999
Chicago, Ill.
39
Terminology for scaleability
Farms of servers:
• Clones: identical
 Scaleability + availability
• Partitions:
 Scaleability
• Packs
 Partition availability via fail-over
Farm
Clone Partition
Pack

3 October 1999
Chicago, Ill.
40
Unpredictable Growth
The TerraServer Story:
• We expected 5 M hits per day
• We got 50 M hits on day 1
• We peak at 15-20 M hpd on a “hot” day
• Average 5 M hpd after 1 year
Most of us cannot predict demand
• Must be able to deal with NO demand
• Must be able to deal with HUGE demand

3 October 1999
Chicago, Ill.
41
An Architecture for Internet Services?
Need to be able to add capacity
• New processing
• New storage
• New networking
Need continuous service
• Online change of all components (hardware and software)
• Multiple service sites
• Multiple network providers
Need great development tools
• Change the application several times per year.
• Add new services several times per year.

3 October 1999
Chicago, Ill.
42
Premise: Each Site is a
Farm
Buy computing by the slice (brick):
• Rack of servers + disks.
Grow by adding slices
• Spread data and
computation
to new slices
Two growth styles:
• Clones: anonymous servers
• Parts+Packs: Partitions fail over within a pack
In both cases,
remote farm for disaster recovery
Switched
Ethernet
Switched
Ethernet
www.microsoft.com
(3)
search.microsoft.com
(1)
premium.microsoft.com
(1)
European Data Center
FTP
Download Server
(1)
SQL SERVERS
(2)
Router
msid.msn.com
(1)
MOSWest
Admin LAN
SQLNet
Feeder LAN
FDDI Ring
(MIS4)
Router
www.microsoft.com
(5)
Building 11
Live SQL Server
Router
home.microsoft.com
(5)
FDDI Ring
(MIS2)
www.microsoft.com
(4)
activex.microsoft.com
(2)
(3)
register.microsoft.com
(2)
msid.msn.com
(1)
FDDI Ring
(MIS3)
www.microsoft.com
(3)
(1)
msid.msn.com
(1)
FDDI Ring
(MIS1)
www.microsoft.com
(4)
(2)
(2)
msid.msn.com
(1) Primary
Gigaswitch
Secondary
Gigaswitch
Staging Servers
(7)
(3)
support.microsoft.com
(2)
register.msn.com
(2)
1997 Microsoft.Com Farm
MOSWest
DMZ Staging Servers
Live SQL Servers
SQL Consolidators
Japan Data Center
www.microsoft.com
(3)
(1)
HTTP
Download Servers
(2) Router
(2)
SQL SERVERS
(2)
msid.msn.com
(1)
FTP
Download Server
(1)
Router
Router
Router
Router
Router
Router
Router
Router
Internal WWW
SQL Reporting
home.microsoft.com
(4)
home.microsoft.com
(3)
home.microsoft.com
(2)
(1)
(1)
Internet
13
DS3
(45 Mb/Sec Each)
2
OC3
(45Mb/Sec Each)
2
Ethernet
(100 Mb/Sec Each)
cdm.microsoft.com
(1)
FTP Servers
Download
Replication
FTP.microsoft.com
(3)
0
Switched
Ethernet
Switched
Ethernet
www.microsoft.com
(3)
Switched
Ethernet
Switched
Ethernet
www.microsoft.com
(3)
(1)
(1)
European Data Center
FTP
Download Server
(1)
SQL SERVERS
(2)
Router
msid.msn.com
(1)
MOSWest
Admin LAN
SQLNet
Feeder LAN
FDDI Ring
(MIS4)
Router
www.microsoft.com
(5)
Building 11
Live SQL Server
Router
home.microsoft.com
(5)
FDDI Ring
(MIS2)
www.microsoft.com
(4)
activex.microsoft.com
(2)
(3)
(2)
msid.msn.com
(1)
(3)
(2)
msid.msn.com
(1)
FDDI Ring
(MIS3)
www.microsoft.com
(3)
(1)
msid.msn.com
(1)
FDDI Ring
(MIS1)
www.microsoft.com
(4)
(2)
FDDI Ring
(MIS3)
www.microsoft.com
(3)
(1)
msid.msn.com
(1)
FDDI Ring
(MIS1)
www.microsoft.com
(4)
(2)
(2)
msid.msn.com
(1) Primary
Gigaswitch
Secondary
Gigaswitch
(2)
msid.msn.com
(1) Primary
Gigaswitch
Secondary
Gigaswitch
Staging Servers
(7)
(3)
(2)
register.msn.com
(2)
1997 Microsoft.Com Farm
MOSWest
DMZ Staging Servers
Live SQL Servers
SQL Consolidators
Live SQL Servers
SQL Consolidators
Japan Data Center
www.microsoft.com
(3)
(1)
HTTP
Download Servers
(2) Router
(2)
SQL SERVERS
(2)
msid.msn.com
(1)
(1)
HTTP
Download Servers
(2) Router
(2)
SQL SERVERS
(2)
msid.msn.com
(1)
FTP
Download Server
(1)
Router
Router
Router
Router
Router
Router
Router
Router
FTP
Download Server
(1)
Router
Router
Router
Router
Router
Router
Router
Router
Internal WWW
SQL Reporting
Internal WWW
SQL Reporting
home.microsoft.com
(4)
home.microsoft.com
(3)
home.microsoft.com
(2)
(1)
(1)
home.microsoft.com
(4)
home.microsoft.com
(3)
home.microsoft.com
(2)
(1)
(1)
Internet
13
DS3
(45 Mb/Sec Each)
2
OC3
(45Mb/Sec Each)
2
Ethernet
(100 Mb/Sec Each)
cdm.microsoft.com
(1)
FTP Servers
Download
Replication
FTP.microsoft.com
(3)
0

3 October 1999
Chicago, Ill.
43
Scaleable Systems
Scale UP and Scale OUT
Everyone does both.
Choice is
• Size of a brick
• Clones or partitions
• Size of a pack

3 October 1999
Chicago, Ill.
44
Everyone scales out
What’s the Brick?
1M$/slice
• IBM S390?
• Sun E 10,000?
 100 K$/slice
• Wintel 8X
 10 K$/slice
• Wintel 4x
1 K$/slice
• Wintel 1x

3 October 1999
Chicago, Ill.
45
Clones: Availability+Scalability
Some applications are
• Read-mostly
• Low consistency requirements
• Modest storage requirement (less than 1TB)
Examples:
• HTTP web servers (IP sprayer/sieve + replication)
• LDAP servers (replication via gossip)
• App/compute servers or firewalls
Replicate app at all nodes (clones)
Spray requests across nodes.
Grow by adding clones
Fault tolerance: stop sending to dead
clone.
Growth: add a clone.

3 October 1999
Chicago, Ill.
46
Facilities Clones Need
Automatic replication
• Applications (and system software)
• Data
Automatic request routing
• Spray or sieve
Management:
• Who is up?
• Update management & propagation
• Application monitoring.
Clones are very easy to manage:
• Rule of thumb: 100’s of clones per admin

3 October 1999
Chicago, Ill.
47
Partitions for Scalability
Clones are not appropriate for some apps.
• Statefull apps do not replicate well
• high update rates do not replicate well
• Huge DBs (disk to expensive to clone)
Examples
• Email / chat / …
• Databases
Partition state among servers
Scalability (online):
• Partition split/merge
• Partitioning must be transparent to client.

3 October 1999
Chicago, Ill.
48
Partitioned (aka. Clustered) Apps
Mail servers
• Perfectly partitionable
 Business Object Servers
• Partition by set of objects.
Parallel Databases
Transparent access to partitioned tables
Parallel Query

3 October 1999
Chicago, Ill.
49
Packs for Availability
Each partition may fail (independent of others)
Partitions migrate to new node via fail-over
• Fail-over in seconds
Pack: the nodes supporting a partition
• VMS Cluster
• Tandem Process Pair
• SP2 HACMP
• Sysplex™
• WinNT MSCS (wolfpack)
Cluster In A Box
now commodity
Partitions grow in packs.

3 October 1999
Chicago, Ill.
50
What Parts+Packs Need
Automatic partitioning (in dbms, mail, files,…)
• Location transparent
• Partition split/merge
• Grow without limits (100x10TB)
Simple failover model
• Partition migration is transparent
• MSCS-like model for services
Application-centric request routing
Management:
• Who is up?
• Automatic partition management (split/merge)
• Application monitoring.

3 October 1999
Chicago, Ill.
51
Services on Clones & Partitions
Application provides a set of services
If cloned:
• Services are on subset of clones
If partitioned:
• Services run at each partition
System load balancing routes request to
• Any clone
• Correct partition.
• Routes around failures.

3 October 1999
Chicago, Ill.
52
Farm pairs: Always Up
 Two farms
 Changes from one
sent to other
 When one farm fails
other provides service
 Masks
• Hardware/Software faults
• Operations tasks (reorganize, upgrade move
• Environmental faults (power fail)

3 October 1999
Chicago, Ill.
53
Availabilty for a simple web site
Web Clients
Front End
Web File Store SQL Temp State
SQL Database
Packs for availability
Clones for availability
Load Balance

3 October 1999
Chicago, Ill.
54
Cloned Packed file servers
Packed Partitions: Database Transparency
Farm Scale Out Scenarios
SQL Temp State
Web File StoreA
Cloned
Front Ends
(firewall, sprayer,
web server)
SQL Partition 3
The FARM: Clones and Packs of Partitions
Web
Clients
Web File StoreB
replication
SQL Database
SQL Partition 2 SQL Partition1
Load Balance

3 October 1999
Chicago, Ill.
55
Clients
IIS Web Server
or other IP based services
Data Servers
SQL, Exchange, File
Network Load
Balancing
Clones
Cluster Service
Pack
COM+ Components
Component Load
Balancing (COM+)
Clones
Application Servers
Reliable, Scalable, Modular
1
2
32
3
…
1
2
8
…
4 3

3 October 1999
Chicago, Ill.
56
Talk 2 (if there is time)
Terminology for scaleability
Farms of servers:
• Clones: identical
 Scaleability + availability
• Partitions:
 Scaleability
• Packs
 Partition availability via fail-over
Farm
Clone Partition
Pack

3 October 1999
Chicago, Ill.
57
0
50
100
150
200
250
300
350
400
450
NT4
InProc
W2K
B1
inProc
W2K
RC1
InProc
W2K
RC2
InProc
1P 2P 4P 8P
0
50
100
150
200
250
300
350
400
450
NT4
InProc
W2K
B1
inProc
W2K
RC1
InProc
W2K
RC2
InProc
NT4
OOP
W2K
B1
OOP
W2K
RC1
OOP
W2K
RC2
OOP
1P 2P 4P 8P
Scalability: COM+ progress
serving 1,000-statement ASP’s (servelets)
Poor SMP Scaleability on IIS4 NT4
Big
improvements
from standard
Transaction
Processing
tricks
Out of Proc
(safe execution)
now much faster
than In Proc
was on IIS4
SPS: servelets per second
(ASPs served per second by IIS,
1,000 statement VBscript)
Shift from 4x200
Mhz to 8 450 Mhz

3 October 1999
Chicago, Ill.
58
Scaleability:
So, What about the death of NT/Alpha?
Two simultaneous Compaq TPC-C numbers
 Intel Profusion
 NT/SQL/COM+
 550 Mhz
 8 Processors
 4 GB memory
 40,368 TPM-C @
18.46$/tpmC
 745 K$ 5-year cost
 Avail: 12/31/99
 Alpha
 Unix/Sybase/Tuexdo
 700 Mhz
 8 Processors
 16 GB memory
 42,437 TPM-C @
55.45 $/tpmC
 $2.35 M$ 5-year cost
 Avail: 10/18/99
200% more expense for 5% more performance?

3 October 1999
Chicago, Ill.
59
Outline
• RAGS.
• Data Mining
Q&A (10 min)

3 October 1999
Chicago, Ill.
60
The TerraServer
http://www.terraserver.microsoft.com/

3 October 1999
Chicago, Ill.
61
 Coverage: Range from 70ºN to 70ºS
today: 35% U.S., 1% outside U.S.
 Source Imagery:
• 4 TB 1sq meter/pixel Aerial (USGS - 60,000
46Mb B&W- 151Mb Color IR files)
• 1 TB 1.56 meter/pixel Satellite
(Spin-2 - 2400 300 Mb B&W)
 Display Imagery: 200x200 pixel images,
subsample to build image pyramid
 Store 5x compressed data
 Nav Tools:
• 1.5 m place names
• “Click-on” Coverage map
• Expedia & Virtual Globe
• Pick of the week
1.6x 1.6 km “city view”
.8 x .8 km 8m thumbnail
.4 x.4 km browse
200x200 m tile
Concept: User
navigates an ‘almost
seamless’ image of
earth
Database & application UI

3 October 1999
Chicago, Ill.
62
browser
HTML
Java
Viewer
The Internet
Web Client
Image Delivery
Application
SQL Server
7
Microsoft
Site Server EE
Internet Information
Server 4.0
Image Provider Site(s)
TerraServer DB
Terra-Server
Stored Procedures
Internet
Information
Server 5.0
Image Server
Active Server Pages
(ADO)
MTS
TerraServer Web Site
Software:
Classic 3 Tier Design
SQL Server 7
24
20 (8/12)
46
46
Fire wall

3 October 1999
Chicago, Ill.
63
SourceMeta
Image
Imagery
Image
Type
Search
Image
Search
Logical Schema
Scale
Job
Load
Job
Load
Mgmt
External
Link
External
Group
External
Geo
Famous
Category
Famous
Place
TerraServer
Terra
Database
TerraAdmin
Admin
Gazetteer
Country
Name
State
Name
Place
Name
Place
Type
Small
Place Name
Pyramid

3 October 1999
Chicago, Ill.
64
TerraServer File Group Layout
Convert 324 disks to 28 RAID5 sets
plus 28 spare drives
Make 4 NT volumes (RAID 50)
595 GB per volume
Build 30 20GB files on each volume
DB is File Group of 120 files
E: F: G: H:

3 October 1999
Chicago, Ill.
65
Hardware
100 Mbps
Ethernet Switch
DS3
Site
Servers
Internet
Map
Server
SPIN-2
Web Servers
2.9 TB Database Server
AlphaServer 8400 8x400.
10 GB RAM
324 StorageWorks disks
10 drive tape library
(STC Timber Wolf DLT7000 )

3 October 1999
Chicago, Ill.
66
Backup and Recovery
• STK 9710 Tape robot
• SQL Server Backup &
Restore +
• Legato Networker
• Fast, incremental,
differential, online
• Clocked at 80 MBps (peak)
(~ 200 GB/hr)
Restore
• Fast, incremental (file
oriented), not online.
Load & Backup&Recovery
Performance
Data Bytes Backed Up 1.2 TB
Total Time 7.25 Hours
Number of Tapes Consumed 27
Total Tape Drives 10
Data ThroughPut 168 GB/Ho
Average ThroughPut Per Device 16.8 GB/Hour
Average Throughput Per Device 4.97 MB/Sec
NTFS Logical Volumes 2

3 October 1999
Chicago, Ill.
67
BAD OLD Load
DLT
Tape
“tar”
Drop’N’ DoJob
Wait 4
Load
LoadMgr
DB
100mbit
EtherSwitch
108
9.1 GB
Drives
Enterprise Storage Array
Alpha
Server
8400
108
9.1 GB
Drives
108
9.1 GB
Drives
STC
DLT
Tape
Library
60
4.3 GB
Drives
Alpha
Server
4100
ESA
Alpha
Server
4100
LoadMgr
DLT
Tape
NT
Backup
ImgCutter
Drop’N’
Images
10: ImgCutter
20: Partition
30: ThumbImg
40: BrowseImg
45: JumpImg
50: TileImg
55: Meta Data
60: Tile Meta
70: Img Meta
80: Update Place
...
LoadMgr

3 October 1999
Chicago, Ill.
68
New Image Load and Update
ODBC Tx
TerraLoader
ODBC TX
TerraServer
SQL
DBMS
DLT
Tape
“tar”
Metadata
Load DB
Active Server Pages
Cut & Load
Scheduling
System
Image
Cutter
Merge
ODBC Tx
Dither
Image Pyramid
From base

3 October 1999
Chicago, Ill.
70
TerraServer Daily Traffic
Jun 22, 1998 thru June 22, 1999
0
10M
20M
30M
Count
Sessions
Hit
Page View
DB Query
Image
After a Year:
1 TB of data
750 M records
2.3 billion Hits
2.0 billion DB Queries
1.7 billion Images sent
368 million Page Views
99.93% DB Availability
3rd design now Online
Built and operated by
team of 4 people
TotalTime (Hours)
0
720
1440
2160
2880
3600
4320
5040
5760
6480
7200
7920
8640
Up
Down Time
(Hours:minutes)
0:00
0:30
1:00
1:30
2:00
2:30
3:00
3:30
4:00
4:30
5:00
5:30
6:00
Scheduled
HW+Software
Operations

3 October 1999
Chicago, Ill.
74
What TerraServer Shows
Can serve huge databases on Internet
for about a penny a page view
mostly phone bill (!).
Advertising pays more than a penny a page.
Commodity tools do scale fairly far.
A few people (3 developers, 1 operator)
using power tools
can build an impressive web site

3 October 1999
Chicago, Ill.
75
Outline
Scaleability: Packs & Mobs (10 min)
• RAGS.
• Data Mining
Q&A (15 min)

3 October 1999
Chicago, Ill.
76
Automatic Testing
60% of Microsoft R&D is testing.
What can research do to help?
• beyond joining the 500,000 Win2K beta testers
Test generation robot:
• Make up SQL queries
• Send them to SQL Server,
Oracle, DB2, Informix,…
• If answer is the same, great,
if not there is a problem
Also good for stress tests
Found MANY bugs in our products (all fixed).
Found MANY bugs in other’s products.
Very valuable tool.
 MSR-TR-98-21 Massive Stochastic Testing of SQL, Slutz, Don
http://research.microsoft.com/scripts/pubDB/pubsasp.asp?RecordID=175
W X Y Z
1672 1672 1672 1672
232 234 241 31
1 1 1 1
31 15 12 28
1 12 5 116
0 29 32 4
18 18 19 25
45 19 18 113
Error
All four
agree 84%
Problem with
intermediate
table.
Case
W,X, and
agree 95%

3 October 1999
Chicago, Ill.
77
Some Tera-Byte Databases
Kilo
Mega
Giga
Tera
Peta
Exa
Zetta
Yotta
The Web: 1 TB of HTML
TerraServer 1 TB of images
Several other 1 TB (file) servers
Hotmail: 20 TB of email
Sloan Digital Sky Survey:
40 TB raw, 2 TB cooked
EOS/DIS (picture of planet each week)
• 15 PB by 2007
Federal Clearing house: images of checks
• 15 PB by 2006 (7 year history)
Nuclear Stockpile Stewardship Program
• 10 Exabytes (???!!)

3 October 1999
Chicago, Ill.
78
Data Mining
Find interesting structure (patterns, relationships) in data
• Prediction
• Segmentation (clustering)
• Dependency modeling (find distribution)
• Summarization
• Trend and change detection and modeling
Allow user to state the query in terms of the business
logic
• User does not speak statistics or SQL
Use data to build predictors
• regression, classification, segmentation etc.
Generate summaries and reports for insight
• find “easy to describe” segments in data automatically
• find segments not known to analyst

3 October 1999
Chicago, Ill.
79
Data Mining:
Microsoft SiteServer Commerce 3.0
Intelligent Cross-sell
Based on:
• Historical sales baskets in
stores
• Contents of current shopper
basket
• Browsing behavior of
shopper
Predict: ranking of products
in store likely to be most
interesting to shopper.
Http://www.holtoutlet.com/outlet4

3 October 1999
Chicago, Ill.
80
Mail to 25% and capture 40%
400% improved response!
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0.1%
0.2%
0.3%
0.6%
1.3%
5.3%
6.7%
25.5%
34.5%
43.8%
56.9%
68.5%
94.8%
98.5%
100.0%
% mailed
%
Captured
of
true
targets
Real data drawn from a Microsoft marketing example

3 October 1999
Chicago, Ill.
81
How do people use
www.microsoft.com?
Data Mining
(Clustering)
Engine
User
browsing
data
100M hits per day
14M users/week X segments
Cluster
Visualizer
Wizard

3 October 1999
Chicago, Ill.
82

3 October 1999
Chicago, Ill.
83
Outline
• RAGS.
• Data Mining
Q&A (10 min)

Gray_Compass99.ppt

Recommended

Recommended

More Related Content

Similar to Gray_Compass99.ppt

Similar to Gray_Compass99.ppt (20)

Recently uploaded

Recently uploaded (20)

Gray_Compass99.ppt

Editor's Notes