1. File StreamMicrosoft SQL Server 2008 R2 Kobi Cohen System DBA Manager Spiral Solutions
2. Objectives  Familiarization with FILESTREAM  Overview  Storage  Security & Access  Implementation  Application Considerations  Combined with other SQL Server features  Cluster Configuration  RBS (Remote BLOB store)©Spiral Solutions, Ltd. All rights reserved.
3. File Stream Overview  For unstructured data  docs, images, videos.  not new data type  Storage attribute.  Varbinary (max) only  Fully transactional.  BLOBs saved on storage (NTFS file system)  Needs coordination between DBA and System Administrator.  Can be accessed by T-SQL (DMLs) or Win32 API (stream access).  Use NT system cache:  Much faster  Avoid double writing (Buffer cache + Disk).  Reduce load from DB engine.  SQL Server buffer pool isn’t used  more free memory.  BLOB size limited only by the volume size (not limited to 2GB) even in Express Edition.©Spiral Solutions, Ltd. All rights reserved.
4. File Stream Overview  Store BLOBs in table? On separate files? or in FILESTREAM?  Data size   Larger than 1 MB  NTFS FILESTREAM.  Smaller than 256 KB  inside the database.  Between 256 KB and 1MB  depend on read vs. write ratio + overwrite rate.  Performance  Fast read access is important NTFS FILESTREAM.  Transactional Consistency  BLOB + structured data  FILESTREAM.  Security  sensitive data  FILESTREAM  SQL security over ACLs.  Manageability  integrated solution  less complex  FILESTREAM.  Data fragmentation  frequent updates  easier to handle on NTFS.  Client access  Using DB access protocols (ODBC, OLEDB)  NTFS FILESTREAM.©Spiral Solutions, Ltd. All rights reserved.
5. File Stream Overview BLOB Storage Methods Comparison Storage solution Comparison point SQL Server (using File server / file system FILESTREAM varbinary(max)) Maximum BLOB size NTFS volume size 2 GB – 1 bytes NTFS volume size Streaming performance of large Excellent Poor Excellent BLOBs Security Manual ACLs Integrated Integrated + automatic ACLs Cost per GB Low High Low Manageability Difficult Integrated Integrated Integration with structured data Difficult Data-level consistency Data-level consistency Application development and More complex More simple More simple deployment Recovery from data fragmentation Excellent Poor Excellent Performance of frequent small Excellent Moderate Poor updates©Spiral Solutions, Ltd. All rights reserved.
6. File Stream Overview Read Performance Comparison (Cold buffer pool)©Spiral Solutions, Ltd. All rights reserved.
7. File Stream Overview Read Performance Comparison (Warm buffer pool)©Spiral Solutions, Ltd. All rights reserved.
8. File Stream Overview Insert Performance Comparison (Cold buffer pool)©Spiral Solutions, Ltd. All rights reserved.
9. File Stream Overview Update Performance Comparison (Cold buffer pool)©Spiral Solutions, Ltd. All rights reserved.
10. File Stream Storage  Preparing for FILESTREAM:  Set the proper Disk/RAID level  resilience, performance, cost, load.  Use FSUTIL :  Turn off short (8.3) file names.  Disable last access time.  Format volume cluster size  Use 64-KB NTFS clusters  4-KB for compressed volumes.  Compress volume  extra CPU.  Regularly defrag FILESTREAM computer systems.  Set antivirus scan of FILESTREAM volumes to avoid delete infected files  quarantine if it’s a must.©Spiral Solutions, Ltd. All rights reserved.
11. File Stream Storage  FILESTREAM stored in special file groups:  File system filter per each volume  one for SQL Server version  Integrated security is a must.  File system directories (data containers) instead of data files (mdf, ndf).  FILESTREAM file group = FILESTREAM data container.  Adding file to FILESTREAM file group = define data container:  Each logical file name is the root directory name.  Filestream.hdr  FILESTREAM metadata describing data container.  $FSLOG  FILESTREM equivalent to transaction log.  FILESTREAM operation  mapped to LSN.  FILESTREAM garbage collection thread  triggered by CHECKPOINT.  DBCC CHECKDB  for validates link-level consistency.©Spiral Solutions, Ltd. All rights reserved.
12. File Stream Storage  File stream storage considerations:  Separate FILESTREAM data containers from:  OS volumes and paging file volumes.  Tempdb volumes.  Databases data & log files volumes.  Each data container on separate volume  granular control.  Data containers cannot be nested.  All FILSTREAM columns data in a table must be stored in same FILESTREAM file group  FILESTREAM_ON better performance.  Each row must have non-null unique row id (uniqueidentifier ROWGUID) and a unique/primary key index (clustered only for non-random data to avoid index reordering on every insert/delete).  Can use partitioning on uniqueidentifier column for load balancing.©Spiral Solutions, Ltd. All rights reserved.
13. File Stream Security & Access  FILESTREAM is secured like other data:  GRANT permission on DB, schema, table or column.  NTFS permissions (ACLs) are ignored.  Only SQL Server service account is granted NTFS access to data containers to avoid outside data editing.  Requires integrated security (Windows authentication)  Windows token  pass client identity in file I/O operation.  Support snapshot isolation level (only in 2008 R2).  Remote access By Server Message Block (SMB) protocol.  Firewall considerations:  Enable both client & server to resolve DNS names.  Open file-sharing TCP ports: 139, 445 (for remote access).©Spiral Solutions, Ltd. All rights reserved.
14. File Stream Security & Access  Dual programing model to access BLOBs:  T-SQL for DMLs:  Insert  can be used with null value, empty (‘’) value or short inline data.  Update  if FILESTREAM field is set to null, the related BLOB is deleted. Partial field update can’t be done with T-SQL (use UPDATE.Write()) since update = create new copy of FILESTREAM data file + delete old file. Partial update only by: FSCTL_SQL_FILESTREAM_FETCH_OLD_CONTENT  Delete or Truncate  underlined BLOB data is deleted in file system.  File system streaming access (for read & write):  Statement model  models T-SQL statements by using open & close methods  File handle is live in that scope  open handle = open transaction.  Storage namespace  DB engine controls BLOB physical file  new internal function column_name.PathName(@option) used by API to provide the logical UNC path of the BLOB to get the Win32 handle.©Spiral Solutions, Ltd. All rights reserved.
15. File Stream Security & Access  Transacted file system access  New function provides token for current session’s transaction GET_FILESTREAM_TRANSACTION_CONTEXT ()  binds file streaming operations with a started transaction. Return NULL if there is no explicitly started transaction.  In C++/ODBC  SQL Native Client 10  OpenSqlFilestream  Win32 handle.  In C#  .NET Framework 3.5 SP1  SqlFileStream Data Type  path name + transaction token  handle can be passed to Win32 File API.  Write-through from remote client  No write operation are cached on client side  better to make fewer writes with larger data size.©Spiral Solutions, Ltd. All rights reserved.
16. File Stream Demo Demo©Spiral Solutions, Ltd. All rights reserved.
17. File Stream Application Considerations  Use NULL instead of 0x for non-initialized FILESTREAM column to avoid file creation.  Avoid insert & update in tables with NOT NULL FILESTREAM columns  decrease application performance over time.  In replication  use NEWSEQUENTIALID() instead of NEWID()  for better performance.  Avoid T-SQL for update, append, prepend data to FILESTREAM BLOB  will be spooled into tempdb.  Avoid appending small BLOB updates  Causes FILESTREAM files to be copied  Write BLOB into varbinary(max) parameter and do single write operation.  Avoid getting lots of BLOB files length in application use DATALENGTH() in T-SQL  does not open the BLOB.©Spiral Solutions, Ltd. All rights reserved.
18. File Stream Combined with other features  Snapshots  Only by excluding FILESTREAM file groups  FILESTREAM file groups will be marked as offline.  Replication Supported by both transactional & merge with many considerations  SQL 2008 or later on all subscribers.  Log Shipping  supports FILESTREAM.  Mirroring  Does not support FILESTREAM.  Clustering  fully supported. Must enable FILESTREAM on all nodes  Backup & Restore  Fully supported on all recovery models.  Encryption  is not supported even when TDE is enabled.  Indexes  cannot be a part of: index key / include column / statistics.  Constraints (PK,FK,UQ) cannot be created on FILESTREAM column.  Temp tables & TVP  cannot be used in local/global temp tables & TVP.©Spiral Solutions, Ltd. All rights reserved.
19. File Stream Cluster Configuration  Data containers must be on shared disk resource.  Separate LUN for FILESTREAM is recommended.  Steps for enabling FILESTREAM:  Enable file stream on service level  on all nodes.  Windows Share Name must be the same on all nodes.  Allow remote clients access  create a file-share cluster resource.  Enable file stream on instance level (facets)  on all nodes.  Add dependency of FILESTREAM LUN to SQL Service.  Take SQL Service off & Bring SQL Service + dependencies on.  NetApp Snap Manager requirements:  Separate LUNs to data files, Log files & FILESTREAM files.©Spiral Solutions, Ltd. All rights reserved.
20. File Stream RBS  Complementary approach  store BLOBs on remote storage.  Provided with SQL Server 2008 R2 Feature pack.  library API set  moving BLOBs from MS SQL to external storage.  Set of SPs and SDK  RBS provider for File stream.  Client-side DLL linked into a user application + stored procedures on SQL Server.©Spiral Solutions, Ltd. All rights reserved.
21. File Stream Useful Links  FILESTREAM Storage in SQL Server 2008: http://msdn.microsoft.com/en-us/library/cc949109(v=sql.100).aspx  To BLOB or Not To BLOB - Large Object Storage in a Database or a File system: http://research.microsoft.com/apps/pubs/default.aspx?id=64525  How to enable FILESTREAM from the command line: http://sqlsrvengine.codeplex.com/wikipage?title=FileStreamEnable&referri ngTitle=Home&ProjectName=sqlsrvengine  FSUTIL syntax: http://technet.microsoft.com/en-us/library/cc785435(v=ws.10).aspx  SQL Server 2008 R2 Remote Blob Store (RBS) Download : http://www.microsoft.com/download/en/confirmation.aspx?id=16978©Spiral Solutions, Ltd. All rights reserved.
22. File Stream End Thanks©Spiral Solutions, Ltd. All rights reserved.