This document discusses SQL Server 2008 as a platform for managing petabyte-scale data. It defines what a very large database (VLDB) is and notes the challenges in managing massive amounts of data insertion, queries and high availability. The key design philosophy is to partition large databases into smaller, more manageable components rather than having a single large database. Methods for partitioning include by server, instance, database and tables. Partitioning can be by data, such as by month or state, or by function, such as sales vs manufacturing data. Table partitioning commonly uses a "sliding window" approach. Scalability is achieved through clustering while cost is reduced via compression, smaller partitions and moving historical data to cheaper storage.

1. SQL Server 2008 כפלטפורמה לניהול נתונים בעולם ה-Peta byte

2. What is a VLDB?Used to be a set size:>100mb in the 1980’s

3. >100gb in the early 1990’s

4. >1tb in the early 2000’s

5. >10tb in the mid 2000’s

6. Now we are dealing with hundreds of TeraBytesNow, a more flexible definition for VLDB prevails:It’s a VLDB when you can no longer use “common and standard” techniques for management and performanceVLDB project ChallengesPlenty of data to handleMassive data insertionMassive query loadsNeed for High AvailabilityNeed for ScalabilityCost reduction

7. Design Philosophy VLDBDon’t think of it as Large complex database, think of it as smaller manageable componentsPutting all the data in one single database doesn’t make you a hero.Partitioning data into smaller manageable sizesMethods for data partitioning:Server Partitioning

8. Instance Partitioning

9. Database Partitioning

10. Tables PartitioningData separation: Data Partitioning vs. Function PartitioningExamples of data partitioning by dataData broken up by months in a year: Jan, Feb, Mar……Data broken up by States in the US: Alabama, Arkansas, Alaska…..Examples of data partitioning by functionData separated by time: sales data, consulting data, competitive data….Data separated by business function: manufacturing data, design data, inventory data…….

11. Table Partitioning: Common Scenario “Sliding Window”New data is added continually (every minute, hourly, Daily, Weekly, Monthly)Data must be scrubbed, indexed, etc.As data ages, data is modified less frequently and then - not at all.Eventually, it may be discarded Sliding WindowTimeObsoleteHotRead-MostlyRead-Only

12. Data Feeder AggrFirst tier Databases AggrTiered ManagerETLQueriesTiered ManagerManager ServerWhere can I find my Data?QueriesSecond tier Databases AggrApplication Server

13. FileGroup 2FileGroup 1FileGroup 1Key (partitioning column)LocatingARowPartition SchemePartitionFunction

14. HA and scalabilityN+1 ClusteringSmart management enables fast scalability by adding servers, databases, instances, partitions etc.

15. Cost reductionPage Level Data compression – saves up to 50% of disk storage (and reduces IO)Smaller partitions may reduce indexes size by 30% (!) – critical in huge environmentsMoving historical to cheaper storage/media No need for Mega servers - Standard servers are doing the job

16. Valinor’s people in this projectShahar BarMeir DudaiTzahi Hakikat…. And more….

17. Thank you!