Information Architecture Determine the business goals What will your site structure and taxonomy look like? Standardize branding with templates and master pages Other considerations Accountability of published content using workflows or approvals Managing search scopes, security trimming, federation Isolate intranet content from extranets Testing for consistency and performance Training your site/content owners and end users http://technet.microsoft.com/en-us/library/cc262873.aspx#section2 Source: Governance Resource Centre on Microsoft TechNet
Storage in a Content Repository Increase in % of inactive data over time Data in SQL Time in years
Planning for SharePoint Storage Recycle bin Versioning Search and index information Growth Good rule of thumb for initial planning is: 3.5 x file system
Basic Storage Management Methods Set site quotas and alerts! 10 GB quota, 8 GB alert is my favorite Monitor growth trends Sites: Slow over time or large jump in size? Overall content DB size Split Content DBs if they get “too big”
Optimal Content DB Sizing Keep them small… Backup & Recovery Operations(<20-50 GB) Performance (<100 GB – nervous at 80 GB) # of objects Size of objects Hardware (servers and storage) Storage Cost (SQL Storage is EXPENSIVE!)
BLOBs - What’s the Issue? BLOBs = Binary Large OBjects SharePoint Content DB = BLOB + Metadata SQL DB storage needs high IOPS / low latency Typically RAID 10, 15k RPM Fibre Channel SAN or Direct Attached. High IOPS + low latency storage = $$$$ BLOBs do not participate in query operations – Why do BLOBS need to be in SQL? DB full of BLOBs = wasted $$$
Default SharePoint Storage SharePoint WFE SharePoint Object Model BLOBs & Metadata SQL Server Content DB Config DB
Issues with BLOBs Get much worse over time… Increase in % of inactive data over time Data in SQL Time in years Inactive sites, documents, list, libraries take up SQL storage, hindering performance
Limited Storage Optimization Options: Move the BLOBs out of the database Archive content
Extending SQL Storage:Moving BLOBS Out of the Database
Planning for Data Use & Growth What does SharePoint 2010 offer OOTB? No native archiving tools EBS extended to include RBS Available only in SQL Server 2008 SP2 Only accessible via API BCS (BDC in 2007) extended to allow for easier connectivity with legacy data systems
Available APIs for Extending SharePoint External BLOB Service(EBS) SQL Remote BLOB Service (RBS)
EBS/RBS Overview Blob Services to change BLOB storage locations EBS = External BLOB Service SharePoint 2007 SP1+ API RBS = Remote BLOB Service SQL Server 2008R2 Feature Pack API, with SharePoint 2010 support Both are interface specifications Need a provider to actually work Cannot have both providers
EBS SharePoint WFE SharePoint Object Model BLOB BLOB Store SQL Server Content DB Config DB Metadata API available in SharePoint 2007 SP1+ EBS provider can take ownership of the BLOB Provider generates a SharePoint token or stub SharePoint knows how to retrieve the object Transparent to end-users EBS Provider
EBS Implemented by SharePoint Only 1 EBS Provider per SharePoint farm Orphaned BLOBs- no direct method to compare BLOB store and Content DB Compliance- what if I don’t want to allow SharePoint to delete the object?
RBS SharePoint WFE BLOB & Metadata SharePoint Object Model SQL Server API available in SQL 2008R2 Feature Pack Not unique to SharePoint, available to any application Provider Library can be associated with each database Relational Access BLOB Provider Library X Metadata Content DB X Content DB Y RBS Client Library Provider Library Y BLOB Store BLOB Store
RBS Implemented by SQL Only 1 RBS Provider per Content DB Orphaned BLOBs much less of an issue Can lock down operations, from a unified storage perspective Can be managed via PowerShell
RBS: SQL Server 2008 Feature Pack API Handled natively by database Default Provider: FILESTREAM Enable FILESTREAM provider on SQL Provision data store and set storage location Install RBS on all SP Web and App servers Enable RBS
RBS vs. SQL Filestream Filestream storage must be file system locally attached to the SQL server RBS is an API set that allows storage on external stores - physically separate machines that may be running custom storage code, for instance EMC Centera
EBS Tighter integration with application, allows for more rules and settings EBS versus RBS, which is better?
RBS EBS Simpler, allows unified storage architecture across applications Tighter integration with application, allows for more rules and settings http://www.codeplex.com/sqlrbs EBS versus RBS, which is better?
And the Winner Is… SharePoint 2007 SharePoint 2010 Future SharePoint Release (SPS 5?) SharePoint External BLOB Storage (EBS) SQL Remote BLOB Storage (RBS) SQL Server 2005 Future SQL Releases SQL Server 2008 Microsoft will provide a PowerShell solution to migrate from EBS to RBS
2 Approaches, Same Result Backup & Recovery Operations
Databases are 60-80% smaller
Need a method to backup BLOBs synchronously
Performance improvement increase as the file/BLOB size increases. Microsoft research indicates:
<256 KB, SQL better
256 KB to 1 MB, SQL and file system comparable
>1 MB, file system better
BLOBs on lower cost storage
Archiving still needed for true savings
Extending SharePoint Storage Tier 1 Tier 2 Web Services Extend Content out of SQL Tier 3 Search / App Services Cloud Storage SQL Database
Cloud Storage Use Case SharePoint “Overdraft Protection” DB alert set at 80 GB, limit at 100 GB Cloud Storage
Could be any storage
Cloud is ideal “insurance”--cheap to setup, more expensive to use
100 80 0 Alert sent to admin } No action taken
Connecting:Not creating the BLOB Problem in the first place!
Connecting Legacy Data
Where is it in it’s lifecycle?
Do you want to expose it in SharePoint?
BCS is intended for connecting LOB’s (Databases, Windows Communication Foundation (WCF) or Web services, .NET connectivity assemblies, Custom data sources) into SharePoint, without migrating the data
No OOTB solutions for getting content out of users desktops, file shares, or other ECM systems
SharePoint 2010 Support
Should you migrate data? Business case Value add of legacy system Maintenance costs Hardware Licensing and support Migration costs Migration process Tools / Training
Options for Exposing Legacy Data 36 (File Shares, Notes, Exchange Public Folders, eRoom Documentum, LiveLink… etc?) Migrate Manually download/upload, losing author, time, security, history, other metadata 3rd Party Tool Connect BCS Mechanisms Most major ECM Vendors AvePoint’s DocAve Connector EBS/RBS API’s preferred
Connect to SharePoint: BCS Mechanisms 37 .NET Assembly Connector Provided with Microsoft Business Connectivity Services (BCS) Each .NET connectivity assembly is specific to an external content type Provides no Administration interface integration Custom Connector Connect to external systems not directly supported by Business Connectivity Services Agnostic of external content types that connect to a kind of external system (all databases or all Web services) Provides an Administration UI integration http://msdn.microsoft.com/en-us/library/ee554911.aspx
Which BCS Mechanism Should I Use? 38 The .NET Assembly Connector approach is recommended if the external system is static. Otherwise, for every change in the back end, you must make changes to the .NET connectivity assembly DLL. This, in turn, requires recompilation and redeployment of the assembly and the models. Custom connector approach is recommended if the back-end interfaces frequently change. By using this approach, only changes to the model are required. http://msdn.microsoft.com/en-us/library/ee554911.aspx
Data is left in source (legacy) system
Data is available through SharePoint
Give legacy system second life by increasing its value
Burden of storage is on legacy system
Changes propagate to source
Connect and forget
Data is moved into SharePoint
Data is available in SharePoint
SharePoint replaced legacy system
Burden of storage is on SharePoint
Changes saved in SharePoint
Migrate and decommission
Migrating vs. Connecting
Migration Planning Considerations Questions to ask yourselves… How much content needs to be migrated? Can you engage other members to assist in the process and arrange for proper training? How long will this take? How much downtime can you tolerate? What minimal requirements do you have for this migration? How much customization do you have? Can you properly map non-SharePoint related assets into SharePoint? Is this a “big bang” migration or can you migrate in a scaled/phased approach? Can you accept loss of metadata and securities? etc…
Archiving:Adding Data Lifecycle Management
42 Lifecycle of a Typical Item Initial content creation High Moderate content retrieval Access / SLA Requirements Low Time
Data Lifecycle Management Records Center Another SharePoint site (SQL Storage = $$$$) Higher % inactive content Typically 1:1 – Library/List for each Content Type Consider separate Content DB, with an RBS provider implemented for this DB Archiving Backup and delete Partner solutions
Archiving Tier 1 Analyze data Use business rules Maximize storage Web Services Tier 2 Search / App Services Tier 3 Cloud Storage SQL Database
3rd Party Archiving Tools What rules are available? Last modified time Author Versions What scope can I apply rules to? (farm to item) Does it use RBS/EBS APIs? Does it integrate with other infrastructure management tools? (backup, replication, etc.)
Summary Think carefully about organization and storage Consider where content will be stored and how it will grow over time Leverage BLOB Services APIs to Optimize SharePoint Storage EBS/RBS API’s can be leveraged to store BLOBs outside of SQL with little impact on end-users, to save $$ and optimize storage Content access is key Develop strategies to handle access to legacy data and content access from remote locations Archive content Plan for long term growth and optimal system performance
What Does AvePoint Offer? DocAve Report Center Storage growth and trending Server performance and monitoring DocAve Administrator Manage site quotas and alerts Move sites between Content DBs DocAve Content Manager Comprehensive Restructuring
What Does AvePoint Offer? Connecting DocAve Connectors Leverage EBS/RBS APIs to expose File Share Content as fully functional SharePoint object Content works with Office Applications, alerts, workflows, 3rd party application, etc… Migrating DocAve Migrators for SharePoint From previous versions of SharePoint, File Shares, Exchange Public Folders, Lotus Notes, Documentum eRoom, EMC Documentum, Livelink, Oracle/Stellent, Vignette Offers granular selection of content, full graphical user/domain/properties mapping Replication DocAve Replicator Fully mapped, live or scheduled replication of all SharePoint contents
Thank You! Q&A
Resources - www.AvePoint.com Visit us: http://www.AvePoint.com Email us:firstname.lastname@example.org Follow us: @AvePoint_Inc @rtacy101 Download a FREE, fully-enabled 30 Day trial of DocAve at www.avepoint.com/download 51
Additional Resources Storage Optimization for SharePoint Whitepaper : http://www.avepoint.com/assets/pdf/sharepoint_whitepapers/Storage_Optimization_Technical_Advisor.pdf Configure Content Database for RBS: http://technet.microsoft.com/en-us/library/ee748641(office.14).aspx FILESTREAM RBS:http://blogs.msdn.com/opal/archive/2009/12/07/sharepoint-2010-beta-with-filestream-rbs-provider.aspx Whitepaper about FILESTREAM:http://msdn.microsoft.com/en-us/library/cc949109.aspx
Connecting to SharePoint: .NET Assembly 53 Write code as Microsoft .NET Framework classes and compile the classes into a primary DLL and multiple dependent DLLs. Publish the DLLs into the Business Data Connectivity (BDC) service database. Use Microsoft SharePoint Designer to discover the .NET Connectivity Assembly and create a model. Map each entity to a class in the DLL, and map each BDC operation in that entity to a method inside that "Class".
At run time, when a user executes a BDC operation, the corresponding method in the primary DLL is executed.
Connecting to SharePoint: Custom 54 Implement ISystemUtility, IConnectionManager, and ITypeReflector interfaces. Implementing IAdministrableSystem provides Administration UI property management support and implementing ISystemPropertyValidator provides import time validation of LobSystem properties (not on the Microsoft Office client). Compile the code into a DLL and place it in the global assembly cache (GAC) on the server and clients. Author the model XML for the custom data source (SharePoint Designer 2010 does not support a model authoring experience for custom connectors).
At run time when a user executes a BDC operation, this invokes the Execute method in the ISystemUtility class. The responsibility of executing the back-end method is given to the Execute method.