• Like

Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Storage Optimization including RBS vs EBS and Applicable DocAve ...

  • 2,432 views
Uploaded on

 

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • Well this way is also very useful for optimization but I was going through the internet when i found that rbs filestream provider has some limitations please refer to this link http://www.alachisoft.com/blogs/storageedge/limitation-of-rbs-filestream-provider/ can this create any hurdle while in the process of optimization?
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
2,432
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
39
Comments
1
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • Now we come to Connecting…
  • Information architecture is critical:Considerations:determining the business goals that you have for SharePointIf deploying SharePoint to deploy it, destined to fail.What are are you going to accomplish with it that you weren’t getting before? Structure and taxonomystandardize branding with templates and master pages.Ensure accountability of your published content using workflows and approvals.
  • MLM – This diagram maps out data growth over a period of time in a collaboration environment:Electronic data will continue to grow year over yearInactive data growth outpaces growth of active, operational dataHowever, most of the data is actually inactive or stale data, which is the area in between the two linesIn SharePointConsider your work on a design documentThrough drafts and edits, multiple versions are being created. Ex. Up to 32 versions of a document, if each document is 2.5 MB, full version history takes up 80MB.When document is approved, initial 30 versions no longer needed and can be archived awayConsider project sitesProject sites bring groups of people together to work on related documents, task lists, discussions, etc.Within each org, there could be hundreds of projects that get completed each year. But entire project sites still reside on SharePointAs inactive data continue to grow, resources required for current active data is saturated by inactive data. Users experience diminishing service levels (i.e. performance degradation)Additional hardware, servers, processing power may be needed, As databases continue to grow, this would also impact the current SLA’s for backup and recovery windows that are currently in place.
  • There are several things you need to take into consideration when planning for how much storage you will need to allocate to SharePoint content.
  • There are a few basic ways to manage storage growth in SharePoint. First things first, set site quotas and alerts, we always recommend a 10 GB limit and 8 GB alert. This is going to let you stay on top of which sites are growing more quickly so you can plan future structure accordingly. Next is to monitor growth trends. Pay attention to how quickly your sites are growing, and then don’t forget to monitor the overall Content DB size. Finally, depending on growth, there’s a very good chance you’ll need to split content DB’s if they get too big. Now, what is “too big”? We’ll get to that in a minute, as there are several recommendations based on your concerns… First though, we’ll look at a couple of ways to control growth of content DBs…
  • Now, to the question we had early- how big is “too big” for content DBs? As previously mentioned, this depends on your concerns. If you’re most concerned about backup and recovery operations and you’re using a database-based backup solution, you really shouldn’t go over 100 GBs, even then you’re pushing it, as you need to make sure that you can recover your database in time to meet various SLAs. However, there are other tools out there, like the one AvePoint offers, that can leverage snapshot technology to greatly enhance the speed of backups and restores. This might also depend on your hardware. If performance is your main concern, then you might can let the size get a decent amount larger. The thing with performance, though, is this is not just going to depend on the database size, but also the # and size of objects, how heavily nested your site structure is, and once again, hardware. Obviously the goal for storage cost is to keep it as small as possible while allowing for proper backup and recovery windows and optimal performance. These are really considerations, but still we have the question, ‘what is “too big”? Which will have to come from your own organization’s policies and plans. Next we’re going to take a look at what’s actually being stored in Content DBs and evaluate ways to further optimize storage besides the few native tools already mentioned… Which brings us to the subject of BLOBs.
  • Now we come to Connecting…
  • Why are we concerned with BLOBs? If the size of content DBs affects critical aspects of SharePoint- backup and recovery windows, performance, cost, then lets evaluate what the content in the databases consists of and what’s necessary and what’s not.
  • Here’s a look at a very basic SharePoint architecture. Here we have the front end with the object model, and the SharePoint content (blobs and metadata) are stored in SQL.
  • Already we’ve talked about how the size of the Databases affect these key areas…
  • This diagram maps out data growth over a period of time in a collaboration environment:Electronic data will continue to grow year over yearInactive data growth outpaces growth of active, operational dataHowever, most of the data is actually inactive or stale data, which is the area in between the two linesIn SharePointConsider your work on a design documentThrough drafts and edits, multiple versions are being created. Ex. Up to 32 versions of a document, if each document is 2.5 MB, full version history takes up 80MB.When document is approved, initial 30 versions no longer needed and can be archived awayConsider project sitesProject sites bring groups of people together to work on related documents, task lists, discussions, etc.Within each org, there could be hundreds of projects that get completed each year. But entire project sites still reside on SharePointAs inactive data continue to grow, resources required for current active data is saturated by inactive data. Users experience diminishing service levels (i.e. performance degradation)Additional hardware, servers, processing power may be needed, As databases continue to grow, this would also impact the current SLA’s for backup and recovery windows that are currently in place.
  • Now we come to Connecting…
  • JohnAside from EBS API, no way to move BLOB contents off of SQL – providers not offered by SharePoint, requires 3rd party tools
  • There are 2 available APIs for extending content out of SQL. 1 is SharePoint specific, it’s the EBS API. The second, is SQL specific, it’s the RBS API. So which one do we use?
  • Here’s a quick overview of the EBS and RBS APIs
  • However, there are some things to note about EBS. As it is implemented by SharePoint, there’s only 1 provider allowed per SharePoint farm. There’s a chance that you could run into orphaned BLOBs, and then there are also compliance concerns.
  • As with EBS, there are some things to note with RBS… one of the main benefits is the ability to mange RBS via PowerShell, which MSFT is highly encouraging the use of over STSADM, as I believe they’re eventually doing away with STSADM.
  • Natively with SharePoint 2010, MSFT offers a RBS provider, FILESTREAM. However, it does not recommend using this with very large databases in production. To leverage this feature, you’d have to 1, 2, 3, and then 4, so you would need admin privileges on SQL and Windows server. STORAGE LOCATION IS FILE SYSTEM ONLY!!
  • So again, looking that the two blob services available, which is better, with EBS we have tighter application integration, allowing for more rules and settings to determine which BLOBs are offloaded, and then you have RBS…
  • Which is simpler, and allows for a more unified storage architecture across applications, it’s not SharePoint specified….
  • At this point, looks like RBS is the way to go… As EBS is SharePoint specific, there’s been no guarantee that in future releases they will continue to support such an API. With RBS, we know that MSFT has bought in to this API as it’s telling customers that it will provide a PowerShell solution to migrate from EBS to RBS, hence, any solution leveraging these APIs should account for this.
  • So once we leverage Blob Services APIs to offload BLOB out of SQL, these are the impacts that we’ll see relative to our previous concerns regarding Backup and Recovery, Performance, and Storage.
  • We like the concept of incorporating cloud storage as a sort of ‘overdraft protection’. Typically, with Cloud storage, it’s cheap to setup but more expensive to use. Compare this with overdraft protection for a checking account… typically it’s free even to set up, but if you use it, if you overdraw and your bank covers your charges so it goes thru, they’ll slap you with a fee, that’s often times not tiny. So think of this scenario… I’ve got my content DB limit set at 100 GB. I get an alert at 80 GBs, so I start planning how to rearrange site structure, split the DB so I won’t hit the limit. But what if in the meantime, I hit my limit? Well nothing happens at 100 GB, but as I go over, if I still haven’t had time to split my DB, cloud storage will be leveraged, allowing all operations to continue, nothing will fail, and this allows you to continue to split up your DB as per usual. OK, so that’s a scenario of leveraging BLOB services APIs to extend content… lets look at some other options for storing BLOBs out of SQL.
  • Now we come to Connecting…
  • Now we come to Connecting…
  • John
  • Now we come to Connecting…
  • If I had to take away 4 main points from today…. These would be them.
  • I’d also like to briefly introduce some applicable AvePoint tools to help you better monitor and manage growth of your Content DBs.
  • Thank you for participating to this and don’t forget to fill in the evaluation;Now we will open the floor for Q&A;
  • Here are additional resources… whitepapers on how to set up Filestream, a white paper on storage optimization sponsored by AvePoint, etc
  • Implement ISystemUtility, IConnectionManager, and ITypeReflector interfaces (only ISystemUtility is mandatory). Potentially, one can also override default connection manager and EntityInstance interfaces. In addition, implementing IAdministrableSystem provides Administration UI property management support and implementing ISystemPropertyValidator provides import time validation of LobSystem properties (not on the Microsoft Office client).Compile the code in step 1 into a DLL and place it in the global assembly cache on the server and clients.Author the model XML for the custom data source (SharePoint Designer 2010 does not support a model authoring experience for custom connectors).At run time when a user executes a BDC operation, this invokes the Execute method in the ISystemUtility class. Thus, the responsibility of executing the actual back-end method is given to the Execute method (as implemented by the custom connector).

Transcript

  • 1. SharePoint Content Lifecycle Management
    Ryan Tacy
    AvePoint
  • 2. Content Lifecycle Management
    Design
    Produce
    Consume
    Sites
    Work-flow
    Office
    Doc Library
    Version
    Publishing Site
    Record Center
    SharePoint
    2
  • 3. Agenda
    Content Organization & Storage
    Storage Optimization
    Content Access
    Archiving
  • 4. Content Organization & Storage
  • 5. Information Architecture
    Determine the business goals
    What will your site structure and taxonomy look like?
    Standardize branding with templates and master pages
    Other considerations
    Accountability of published content using workflows or approvals
    Managing search scopes, security trimming, federation
    Isolate intranet content from extranets
    Testing for consistency and performance
    Training your site/content owners and end users
    http://technet.microsoft.com/en-us/library/cc262873.aspx#section2
    Source: Governance Resource Centre on Microsoft TechNet
  • 6. Storage in a Content Repository
    Increase in % of inactive data over time
    Data in SQL
    Time in years
  • 7. Planning for SharePoint Storage
    Recycle bin
    Versioning
    Search and index information
    Growth
    Good rule of thumb for initial planning is: 3.5 x file system
  • 8. Basic Storage Management Methods
    Set site quotas and alerts!
    10 GB quota, 8 GB alert is my favorite
    Monitor growth trends
    Sites: Slow over time or large jump in size?
    Overall content DB size
    Split Content DBs if they get “too big”
  • 9. Optimal Content DB Sizing
    Keep them small…
    Backup & Recovery Operations(<20-50 GB)
    Performance (<100 GB – nervous at 80 GB)
    # of objects
    Size of objects
    Hardware (servers and storage)
    Storage Cost (SQL Storage is EXPENSIVE!)
  • 10. Storage Optimization
  • 11. BLOBs - What’s the Issue?
    BLOBs = Binary Large OBjects
    SharePoint Content DB = BLOB + Metadata
    SQL DB storage needs high IOPS / low latency
    Typically RAID 10, 15k RPM Fibre Channel SAN or Direct Attached.
    High IOPS + low latency storage = $$$$
    BLOBs do not participate in query operations – Why do BLOBS need to be in SQL?
    DB full of BLOBs = wasted $$$ 
  • 12. Default SharePoint Storage
    SharePoint WFE
    SharePoint Object Model
    BLOBs &
    Metadata
    SQL Server
    Content DB
    Config DB
  • 13. Database Size Implications
    BLOBs increase DB size, creating issues with:
    Backup & Recovery operations
    Performance
    Storage Costs
  • 14. Issues with BLOBs Get much worse over time…
    Increase in % of inactive data over time
    Data in SQL
    Time in years
    Inactive sites, documents, list, libraries take up SQL storage, hindering performance
  • 15. Limited Storage Optimization Options:
    Move the BLOBs out of the database
    Archive content
  • 16. Extending SQL Storage:Moving BLOBS Out of the Database
  • 17. Planning for Data Use & Growth
    What does SharePoint 2010 offer OOTB?
    No native archiving tools
    EBS extended to include RBS
    Available only in SQL Server 2008 SP2
    Only accessible via API
    BCS (BDC in 2007) extended to allow for easier connectivity with legacy data systems
  • 18. Available APIs for Extending
    SharePoint External BLOB
    Service(EBS)
    SQL Remote BLOB Service (RBS)
  • 19. EBS/RBS Overview
    Blob Services to change BLOB storage locations
    EBS = External BLOB Service
    SharePoint 2007 SP1+ API
    RBS = Remote BLOB Service
    SQL Server 2008R2 Feature Pack API, with SharePoint 2010 support
    Both are interface specifications
    Need a provider to actually work
    Cannot have both providers
  • 20. EBS
    SharePoint WFE
    SharePoint Object Model
    BLOB
    BLOB Store
    SQL Server
    Content DB
    Config DB
    Metadata
    API available in SharePoint 2007 SP1+
    EBS provider can take ownership of the BLOB
    Provider generates a SharePoint token or stub
    SharePoint knows how to retrieve the object 
    Transparent to end-users 
    EBS Provider
  • 21. EBS
    Implemented by SharePoint
    Only 1 EBS Provider per SharePoint farm
    Orphaned BLOBs- no direct method to compare BLOB store and Content DB
    Compliance- what if I don’t want to allow SharePoint to delete the object?
  • 22. RBS
    SharePoint WFE
    BLOB &
    Metadata
    SharePoint Object Model
    SQL Server
    API available in SQL 2008R2 Feature Pack
    Not unique to SharePoint, available to any application
    Provider Library can be associated with each database
    Relational Access
    BLOB
    Provider Library X
    Metadata
    Content DB
    X
    Content DB
    Y
    RBS Client Library
    Provider Library Y
    BLOB Store
    BLOB Store
  • 23. RBS
    Implemented by SQL
    Only 1 RBS Provider per Content DB
    Orphaned BLOBs much less of an issue
    Can lock down operations, from a unified storage perspective
    Can be managed via PowerShell
  • 24. RBS: SQL Server 2008 Feature Pack API
    Handled natively by database
    Default Provider: FILESTREAM
    Enable FILESTREAM provider on SQL
    Provision data store and set storage location
    Install RBS on all SP Web and App servers
    Enable RBS
  • 25. RBS vs. SQL Filestream
    Filestream storage must be file system locally attached to the SQL server
    RBS is an API set that allows storage on external stores - physically separate machines that may be running custom storage code, for instance EMC Centera
  • 26. EBS
    Tighter integration with application, allows for more rules and settings
    EBS versus RBS, which is better?
  • 27. RBS
    EBS
    Simpler, allows unified storage architecture across applications
    Tighter integration with application, allows for more rules and settings
    http://www.codeplex.com/sqlrbs
    EBS versus RBS, which is better?
  • 28. And the Winner Is…
    SharePoint 2007
    SharePoint 2010
    Future SharePoint Release (SPS 5?)
    SharePoint External BLOB
    Storage (EBS)
    SQL Remote BLOB Storage (RBS)
    SQL Server 2005
    Future SQL
    Releases
    SQL Server 2008
    Microsoft will provide a PowerShell
    solution to migrate from EBS to RBS
  • 29. 2 Approaches, Same Result
    Backup & Recovery Operations
    • Databases are 60-80% smaller
    • 30. Need a method to backup BLOBs synchronously
    Performance
    • Performance improvement increase as the file/BLOB size increases. Microsoft research indicates:
    • 31. <256 KB, SQL better
    • 32. 256 KB to 1 MB, SQL and file system comparable
    • 33. >1 MB, file system better
    Storage Cost
    • BLOBs on lower cost storage
    • 34. Archiving still needed for true savings
  • Extending SharePoint Storage
    Tier 1
    Tier 2
    Web Services
    Extend Content out of SQL
    Tier 3
    Search / App Services
    Cloud Storage
    SQL Database
  • 35. Cloud Storage Use Case
    SharePoint “Overdraft Protection”
    DB alert set at 80 GB, limit at 100 GB
    Cloud
    Storage
    • Could be any storage
    • 36. Cloud is ideal “insurance”--cheap to setup, more expensive to use
    100
    80
    0
    Alert sent to admin
    }
    No action taken
  • 37. Content Access
  • 38. Connecting:Not creating the BLOB Problem in the first place!
  • 39. Connecting Legacy Data
    • Where is it in it’s lifecycle?
    • 40. Do you want to expose it in SharePoint?
    • 41. BCS is intended for connecting LOB’s (Databases, Windows Communication Foundation (WCF) or Web services, .NET connectivity assemblies, Custom data sources) into SharePoint, without migrating the data
    • 42. No OOTB solutions for getting content out of users desktops, file shares, or other ECM systems
    SharePoint 2010 Support
  • 43. Should you migrate data?
    Business case
    Value add of legacy system
    Maintenance costs
    Hardware
    Licensing and support
    Migration costs
    Migration process
    Tools / Training
  • 44. Options for Exposing Legacy Data
    36
    (File Shares, Notes, Exchange Public Folders, eRoom Documentum, LiveLink… etc?)
    Migrate
    Manually download/upload, losing author, time, security, history, other metadata
    3rd Party Tool
    Connect
    BCS Mechanisms
    Most major ECM Vendors
    AvePoint’s DocAve Connector
    EBS/RBS API’s preferred
  • 45. Connect to SharePoint: BCS Mechanisms
    37
    .NET Assembly Connector
    Provided with Microsoft Business Connectivity Services (BCS)
    Each .NET connectivity assembly is specific to an external content type
    Provides no Administration interface integration
    Custom Connector
    Connect to external systems not directly supported by Business Connectivity Services
    Agnostic of external content types that connect to a kind of external system (all databases or all Web services)
    Provides an Administration UI integration
    http://msdn.microsoft.com/en-us/library/ee554911.aspx
  • 46. Which BCS Mechanism Should I Use?
    38
    The .NET Assembly Connector approach is recommended if the external system is static. Otherwise, for every change in the back end, you must make changes to the .NET connectivity assembly DLL. This, in turn, requires recompilation and redeployment of the assembly and the models.
    Custom connector approach is recommended if the back-end interfaces frequently change. By using this approach, only changes to the model are required.
    http://msdn.microsoft.com/en-us/library/ee554911.aspx
  • 47. Migrate
    Connect
    • Data is left in source (legacy) system
    • 48. Data is available through SharePoint
    • 49. Give legacy system second life by increasing its value
    • 50. Burden of storage is on legacy system
    • 51. Changes propagate to source
    • 52. Connect and forget
    • 53. Data is moved into SharePoint
    • 54. Data is available in SharePoint
    • 55. SharePoint replaced legacy system
    • 56. Burden of storage is on SharePoint
    • 57. Changes saved in SharePoint
    • 58. Migrate and decommission
    Migrating vs. Connecting
  • 59. Migration Planning Considerations
    Questions to ask yourselves…
    How much content needs to be migrated?
    Can you engage other members to assist in the process and arrange for proper training?
    How long will this take? How much downtime can you tolerate?
    What minimal requirements do you have for this migration?
    How much customization do you have?
    Can you properly map non-SharePoint related assets into SharePoint?
    Is this a “big bang” migration or can you migrate in a scaled/phased approach?
    Can you accept loss of metadata and securities?
    etc…
  • 60. Archiving:Adding Data Lifecycle Management
  • 61. 42
    Lifecycle of a Typical Item
    Initial content creation
    High
    Moderate content retrieval
    Access / SLA Requirements
    Low
    Time
  • 62. Data Lifecycle Management
    Records Center
    Another SharePoint site (SQL Storage = $$$$)
    Higher % inactive content
    Typically 1:1 – Library/List for each Content Type
    Consider separate Content DB, with an RBS provider implemented for this DB
    Archiving
    Backup and delete
    Partner solutions
  • 63. Archiving
    Tier 1
    Analyze data
    Use business rules
    Maximize storage
    Web Services
    Tier 2
    Search / App Services
    Tier 3
    Cloud Storage
    SQL Database
  • 64. 3rd Party Archiving Tools
    What rules are available?
    Last modified time
    Author
    Versions
    What scope can I apply rules to? (farm to item)
    Does it use RBS/EBS APIs?
    Does it integrate with other infrastructure management tools? (backup, replication, etc.)
  • 65. Summary
    Think carefully about organization and storage
    Consider where content will be stored and how it will grow over time
    Leverage BLOB Services APIs to Optimize SharePoint Storage
    EBS/RBS API’s can be leveraged to store BLOBs outside of SQL with little impact on end-users, to save $$ and optimize storage
    Content access is key
    Develop strategies to handle access to legacy data and content access from remote locations
    Archive content
    Plan for long term growth and optimal system performance
  • 66. What Does AvePoint Offer?
    DocAve Report Center
    Storage growth and trending
    Server performance and monitoring
    DocAve Administrator
    Manage site quotas and alerts
    Move sites between Content DBs
    DocAve Content Manager
    Comprehensive Restructuring
  • 67. What Does AvePoint Offer?
    Connecting
    DocAve Connectors
    Leverage EBS/RBS APIs to expose File Share Content as fully functional SharePoint object
    Content works with Office Applications, alerts, workflows, 3rd party application, etc…
    Migrating
    DocAve Migrators for SharePoint
    From previous versions of SharePoint, File Shares, Exchange Public Folders, Lotus Notes, Documentum eRoom, EMC Documentum, Livelink, Oracle/Stellent, Vignette
    Offers granular selection of content, full graphical user/domain/properties mapping
    Replication
    DocAve Replicator
    Fully mapped, live or scheduled replication of all SharePoint contents
  • 68.
  • 69. Thank You!
    Q&A
  • 70. Resources - www.AvePoint.com
    Visit us: http://www.AvePoint.com
    Email us:ryan.tacy@avepoint.com
    Follow us: @AvePoint_Inc
    @rtacy101
    Download a FREE, fully-enabled 30 Day trial of DocAve at www.avepoint.com/download
    51
  • 71. Additional Resources
    Storage Optimization for SharePoint Whitepaper :
    http://www.avepoint.com/assets/pdf/sharepoint_whitepapers/Storage_Optimization_Technical_Advisor.pdf
    Configure Content Database for RBS: http://technet.microsoft.com/en-us/library/ee748641(office.14).aspx
    FILESTREAM RBS:http://blogs.msdn.com/opal/archive/2009/12/07/sharepoint-2010-beta-with-filestream-rbs-provider.aspx
    Whitepaper about FILESTREAM:http://msdn.microsoft.com/en-us/library/cc949109.aspx
  • 72. Connecting to SharePoint: .NET Assembly
    53
    Write code as Microsoft .NET Framework classes and compile the classes into a primary DLL and multiple dependent DLLs.
    Publish the DLLs into the Business Data Connectivity (BDC) service database.
    Use Microsoft SharePoint Designer to discover the .NET Connectivity Assembly and create a model.
    Map each entity to a class in the DLL, and map each BDC operation in that entity to a method inside that "Class".
    • At run time, when a user executes a BDC operation, the corresponding method in the primary DLL is executed.
    http://msdn.microsoft.com/en-us/library/ee554911.aspx
  • 73. Connecting to SharePoint: Custom
    54
    Implement ISystemUtility, IConnectionManager, and ITypeReflector interfaces.
    Implementing IAdministrableSystem provides Administration UI property management support and implementing ISystemPropertyValidator provides import time validation of LobSystem properties (not on the Microsoft Office client).
    Compile the code into a DLL and place it in the global assembly cache (GAC) on the server and clients.
    Author the model XML for the custom data source (SharePoint Designer 2010 does not support a model authoring experience for custom connectors).
    • At run time when a user executes a BDC operation, this invokes the Execute method in the ISystemUtility class. The responsibility of executing the back-end method is given to the Execute method.
    http://msdn.microsoft.com/en-us/library/ee554911.aspx