The PST File Rodeo


Published on

By Paul Robichaux
Outlook personal folder files, better known simply as PST files, have been around for a very long time: they
were first introduced with the Exchange client that shipped with Exchange 4.0. PST files allow users to
manage their own mail storage and archiving, and for that reason they are widely used within organizations
of all sizes. Users love having a convenient, portable, personal message store. On the other hand, companies
want to be able to manage, monitor, and control information on their networks, especially as compliance and
electronic discovery have grown in importance since the time when Exchange 4.0 shipped. Striking the right
balance between these two opposing sets of demands is tricky, although Microsoft has tried. This paper will
discuss the broad history and functionality of PST files, point out some specific dangers and benefits of their
use, and describe methods you can use to control (or prevent) the use of PST files in your organization.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

The PST File Rodeo

  1. 1. white paperOutlook personal folder files, better known simply asPST files, have been around for a very long time: theywere first introduced with the Exchange client thatshipped with Exchange 4.0. PST files allow users tomanage their own mail storage and archiving, and forthat reason they are widely used within organizationsof all sizes. Users love having a convenient, portable,personal message store. On the other hand, companieswant to be able to manage, monitor, and control infor-mation on their networks, especially as compliance andelectronic discovery have grown in importance sincethe time when Exchange 4.0 shipped. Striking the rightbalance between these two opposing sets of demandsis tricky, although Microsoft has tried. This paper willdiscuss the broad history and functionality of PST files,point out some specific dangers and benefits of theiruse, and describe methods you can use to control (orprevent) the use of PST files in your organization.ThePSTFileRodeoBy Paul Robichaux
  2. 2. The PST File Rodeowhite papersponsored by The PST File Rodeo 2Understanding PST FilesOne of the key concepts email users have had to absorb is thattheir mail can be stored on a central server, not on a local com-puter. In the days when for-pay services like CompuServe and AOLdominated the email landscape, and company-provided emailwas relatively rare, it was natural to think of email as somethingyou downloaded to your computer and stored locally. Most us-ers work with email on a single computer, and as a result manypeople still have the idea that email is stored and managed locally.(Ask around your office or dinner table if you do not believe thatthis is true!)PST files were originally introduced to give users a means of locallystoring messages that could tie in with Exchange Server-basedstorage. As originally envisioned, PST files were meant to:• allow users to retain access to their email even while disconnected(an especially important feature given that dial-up access was thenorm when this feature was designed!)• provide a simple way to move mail content from place to place• give users a way to manage their own message storage, freeingup resources on the serverDon’t be too hard on Microsoft for the faults of PST files. Theconstraints that Microsoft had to meet with the initial design forPST files were pretty daunting—an email server might have a fewhundred megabytes of space shared among all mailboxes on theserver, while individual clients might have more. Multi-gigabytemailboxes, or mailboxes with hundreds of thousands of items,were pretty much beyond imagining at the time, which is why theoriginal PST file format couldn’t handle them.UnderstandingPST Filespage 2A Brief History ofPST Filespage 3PST vs. OST Filespage 3PSTs:Pros and Conspage 3PSTs andExchange 2010’sArchivingpage 4Managing PSTspage 5contents
  3. 3. The PST File Rodeowhite papersponsored by The PST File Rodeo 3A Brief History of PST FilesThe objects we think of as“PST files”have undergone a number ofmajor changes since their original introduction. The Outlook PSTformat we know and love was first introduced with Outlook 97,where it was officially known as the“Microsoft Outlook PersonalFolders file.”It allowed users to store up to 2GB of mail data ina single file, with a limit of 65,000 items per folder. This formatsuffered from a number of limitations, though. First was the 2GBlimit; that seemed like a very high limit back in 1997, but as diskcapacities steadily grew, users started to run into the limit. Abigger problem was that the PST structure itself was brittle andeasily corrupted. Transient problems with disk hardware, flaws inOutlook and Windows, or plain old random bad luck could quicklyrender an entire PST file unreadable—but so could adding datathat pushed the file past its 2GB limit! These problems were com-pounded by the fact that most people did not bother to back uptheir local PST files, so once a PST file was damaged its contentsmight be unrecoverable. In addition, the original PST file formatcouldn’t handle content using the Unicode system of contentencoding.To address these problems, Microsoft introduced a new file formatwith Outlook 2003. To help keep things straight, the new formatgot a new name: the Microsoft Office Outlook Personal Foldersfile1. This new version removed the 2GB size limitation, as well asthe 65,000-item cap, and it supported the Unicode character en-coding system. By default, Outlook 2003/2007/2010 Unicode PSTfiles can grow up to 20GB, although the actual limit can be config-ured by setting a registry key. As part of the new format, Microsoftmade internal format changes to help make PST files more robust.These changes are invisible to users, except that some errors thatwould have rendered old-format PSTs unreadable are now fixable.Outlook 2010 did not make any changes in the PST file format (atleast not that Microsoft is admitting). Some of the much-heraldedperformance improvements in Outlook 2007 SP2 were actuallyfixes from Outlook 2010 that were backported to the earlier ver-sion, but there do not seem to be any major changes to the PSTfile format itself.1 If you didn’t notice much difference between the two names, you’re not alone. MostOutlook experts call the Outlook 2003-and-later versions“Unicode PSTs”and the older versions“ANSI PSTs.”PST vs. OST FilesOne point of confusion that still lingers is the difference betweenPST files and the similar-sounding offline folder file (OST) format.While the PST format is designed to serve as a more or less staticcontainer for mail, the OST file is best thought of as a cache thatholds a copy of all the data in a user’s mailbox. When Outlook isrunning in cached Exchange mode, updates from the server areapplied to the OST file, as are changes the user makes locally.When Outlook is not connected to the Exchange server, the user’schanges are still applied to the OST file; when Outlook reconnects,any changes made to the Exchange mailbox or the OST file aresynchronized automatically.Contrast this with a PST, which does not have a server-sidecounterpart. If your Exchange profile is set to deliver mail to a PST,when Outlook connects to the server, the mail is automaticallydownloaded and stored in the PST file, then removed from theserver. Whatever changes you make to the local PST file, remainin the file; they are never replicated back to the server. Depend-ing on why you are using PST files, this may or may not be a goodthing.There is an additional wrinkle, too. Each OST file is linked to a spe-cific Outlook profile on an individual computer by the presenceof an encrypted token. If you delete the profile, or move the OSTto another computer, you can no longer open it. By contrast, PSTfiles are designed to be freely moved between machines.There are a number of commercial tools that can convert OST filesinto PST files. However, the simplest way to do this is to open theOST file, create a new PST, and manually move content to it. Thatdoes, however, require you to be able to open the OST file; so ifyou really need to recover content from an OST file, you may endup using a commercial tool.PSTs: Pros and ConsBecause PST files have been around for so long, many Exchangeadministrators think of them as a natural part of the Exchangelandscape. Like other flora and fauna though, some PSTs arenoxious, unwanted, and even dangerous (like, say, black widowspiders or rattlesnakes), while others see them as useful, likemosquito-eating bats. A thorough examination of the pros and
  4. 4. The PST File Rodeowhite papersponsored by The PST File Rodeo 4cons of PST files will help you better understand how they mightfit into your landscape.The Dangers of PST FilesOne of the first things I learned as a neophyte Exchange adminis-trator was a simple rule: PST = BAD. There are ample justificationsfor that rule, including these:• Because PSTs are stored on individual workstations, they frequent-ly do not get backed up. That means critical mail data may existin only one place. As an additional insult, users who move theirPSTs from place to place (or who keep multiple copies) can becomeconfused at the lack of integration between their PST file(s) andtheir online mailbox.• Neither Windows nor Exchange provides any tools to inventory ortrack the contents of PST files; so there is no built-in way to find outwhat PST files exist on your network or what is in them.• Because users control the contents of PSTs, they may freely violateyour organization’s message retention and management poli-cies—and they can probably do so with impunity, given the lack ofbuilt-in PST management tools.• PSTs are normally tied to a single computer, and they cannot beviewed within Outlook Web Access or on mobile devices. Users whoroutinely switch between OWA, mobile devices, and Outlook willonly have partial access to their mail.• Microsoft has improved the robustness of PST files since theirintroduction, but they are still relatively easy to damage or corrupt(compared to, say, a Word document or a TIFF image).• PST files can be password protected, but the algorithm used isrelatively weak. This can sometimes mislead users into believingthat their data is more secure than it really is.• PST files are difficult, even impossible, to use on other platforms.There are few reliable tools for opening and working with PST fileson Mac OS X or Linux computers.• Versions of Outlook prior to Outlook 2007 SP2 are slow to readand write PST files, particularly if the PST files are stored on networkshares. In fact, Microsoft’s official stance is that they do not supportthe use of PST files stored on network shares because doing so cancause the file server to exhaust its non-paged pool resources.The Benefits of Using PST FilesAfter that long list of disadvantages you may wonder if PST fileshave any redeeming features. (In fact, for a good laugh, plug“PSTadvantages”into your favorite search engine and examine theresults—none of them actually talk about PST advantages!) Thetruth is that the biggest benefit of PST files is their familiarity tousers; once users become accustomed to storing their own maildata, using their own preferred filing system that is totally undertheir control, it can be difficult to win them back. PST files do offersome other advantages though:• They are intentionally designed to be portable. You can put a PSTfile on a thumb drive or another storage device and open it fromany machine that has Outlook installed, which is often easier thancreating a new Outlook profile or otherwise connecting to youractual Exchange server.• They allow users to store mail in a way that does not countagainst the user’s storage quota. This can be useful for users whohave a genuine business need to keep more mail around than willfit under their actual quota.• The PST file format is a standardized way to import and exportmail content. Microsoft’s own tools (including the Exchange2007/2010 import-mailbox and export-mailbox cmdlets andthe recovery storage group tools included in Exchange 2003) canproduce and consume PST files, as can many third-party archivingand management tools.• They give users a location to store personal or sensitive contentthat they do not want read by nosy administrators.• They provide a storage mechanism for user mail that Outlookretrieves from external IMAP or POP accounts.• They provide an inexpensive, albeit limited, way to archive mail.Although it is certainly not advisable to rely on PSTs as a fullarchiving solution.PSTs and Exchange 2010’s ArchivingSpeaking of archiving, it is worth mentioning the new archivingfeatures in Exchange 2010. They are designed to provide reten-tion and control features (including litigation holds and a much-improved cross-mailbox search experience) that were not partof Exchange 2007, as well as the ability for users to create andmanage their own personal archives without using PST files.The archiving functions in Exchange 2010 are well-described inthe product documentation, but there are a few highlights youshould know about:• The archive mailbox for a user has to remain in the same mailbox
  5. 5. The PST File Rodeowhite papersponsored by The PST File Rodeo 5database as the primary mailbox. This means that you cannotseparate archive mailboxes into their own databases for disasterrecovery, backup, or high availability reasons.• The archive mailbox is only available in Outlook 2010 and OWA2010. There is no current way to access it from other clients.• The archive mailbox is only available when the user is online;there is no way to work with an offline copy.• Users can drag and drop items between their primary mailbox,their archive mailbox, and individual PST files, but there is no exist-ing way to import PST files directly to users’ archive mailboxes.Managing PSTsThe bad news: you may not be able to eliminate PST files entirely.The good news: there are some things you can do to help controlthem. In addition, there are a few management tasks (like fixingcorrupted PST files) that you will probably need to know how todo until you can eliminate them altogether.Repairing PST FilesMicrosoft has made available a repair tool that can sometimesrescue the contents of a corrupt PST. According to its documenta-tion, the Inbox Repair Tool (better known as“scanpst.exe”) doesfour things:• First, the Inbox Repair Tool makes a backup copy of whatever fileyou provide it.• The Inbox Repair Tool will try to turn any file you give it into a PSTfile. For example, if you rename an executable file to junk.pst, thetool changes the file to a mountable .pst file. The results of thisconversion may range from laughable to useful to dangerous.• The Inbox Repair Tool analyzes the PST’s directory structure anditem headers to try to recover all folders and items.• If the Inbox Repair Tool reports that it recovered the PST file, itmeans that the repair tool found problems, and repaired what itcould. In this case, you may still have lost data.Running the tool is simple. Here is what you need to do:1. Quit Outlook.2. Launch the ScanPST.Exe tool. Depending on which version of 3.Windows you are running, it can be located in a number of differ-ent locations, so your best bet is to search for it.3. Click the Browse button, then navigate to the PST file you wantto repair.4. Click the Start button. The Inbox Repair Tool will start its work.When it is done, you will see a summary dialog like the one shownin Figure 1. You can click the Details button to see a summary of er-rors found during the scan, although normally the summary will besomething meaningless like “Internal errors were found in this file.”Figure 1: After running ScanPST you can repair the file and optionallysee a list of errors5. Click the Repair button.6. When ScanPST has finished its repair, it will display a confirma-tion dialog telling you so. Click OK.7. Launch Outlook and create a new PST file. This is where you willcopy information recovered from the repaired PST file.8. Open the repaired PST file. You will notice that it contains a foldernamed “Lost and Found.” This is where all your recovered items nowlive. Be aware that depending on how successful the Inbox RepairTool was, this folder may not contain all of the items that you hadhoped to recover.9. Drag the recovered items from the Lost and Found folder to yournew PST file.If there are particular items that you were hoping to recover thatthe tool did not catch, you can open the backup file that the
  6. 6. The PST File Rodeowhite papersponsored by The PST File Rodeo 6Inbox Repair Tool creates; it will have the same path and name asthe original file, but with a“.bak”extension. Copy the file, give thecopy a“.pst”extension, and then use Outlook’s Import and ExportWizard to try to import it.Setting a Maximum PST File SizeYou can restrict the maximum size that a PST can grow to, by add-ing registry keys to the appropriate version-specific path (HKEY_CURRENT_USERSoftwarePoliciesMicrosoftOffice12.0OutlookPST for Outlook 2007 and HKEY_CURRENT_USERSoftwarePoli-ciesMicrosoftOffice11.0OutlookPST for Outlook 2003). Thereare four separate keys, as shown in the table below:As you can see, there are separate keys for ANSI and Unicode PSTs,with different size limits for each. The Unicode format can theo-retically handle up to 33TB, but the maximum size you can specifyis 20GB for Unicode PSTs and 2GB for ANSI PSTs.Interestingly, the behavior of these keys may be different fromwhat you expect. The WarnLargeFileSize and WarnFileSize keysdo not actually generate a warning. Instead, they prevent addingitems to the PST file once it reaches the specified limit. However,Outlook itself can cause the file to grow by reindexing or changingitems—so the PST file’s physical size on disk can still exceed thelimit. When you use the MaxFileSize and MaxLargeFileSize keys,they set a hard limit on the file size, which Outlook will not exceed.ControllingWhere PST Files GoBy default, Outlook stores users’PST files inside their profile’s localapplication data folder, the name of which varies between differ-ent versions of Windows. However, you can force Outlook to placePSTs at a path you specify.The first route is to bundle the desired setting as part of the Of-fice installation itself. For Office 2003, you would use the CustomInstallation Wizard (CIW); for Office 2007, you will need to use theOffice Customization Tool (OCT). In either case, you can specifythe path where PST files will be stored. This will have no effect onexisting PST files, but users will only be able to create new PST filesat the location you specify. Unfortunately, this approach worksbest when you use it before installing Office, because the configu-ration you specify is applied as part of the installation.The second method allows you to enforce a specific path at anytime, and as a bonus it can be applied to individual users. In fact,it must be applied to individual users, so keep that in mind if youwant to apply a restriction globally. To use this method, you willneed to add a new string value to the HKEY_CURRENT_USERSoft-warePoliciesMicrosoftOffice12.0Outlook (for Outlook 2007) orHKEY_CURRENT_USERSoftwarePoliciesMicrosoftOffice11.0Outlook (for Outlook 2003) keys. The new value should be namedForcePSTPath, and it should contain the path you want to requireusers to use. Remember that when you put a string value in theregistry that you must use“”to encode a single backslash, so thecorrect value for a desired path of z:email would be“z:email.”Preventing the Use of PST FilesYou can actually prevent Outlook users from accessing PSTs inseveral different ways.First, you can prevent users from creating PST files at all with theDisablePST registry key (HKEY_CURRENT_USERSoftwarePoliciesMicrosoftOffice12.0OutlookDisablePST), a DWORD value thatforces Outlook to remove the File | New | Outlook Data File andFile | Archive commands from the Outlook menu bar. This keepsusers from making new PSTs, as well as blocking their ability toimport and export items with existing PST files.You can also remove users’ability to use Outlook’s AutoArchivefeatures. Even though you block PST files, users can still useAutoArchive unless you proactively disable it. To do so, you willneed to ensure that the ArchiveDelete, ArchiveMount, ArchiveOld,DeleteExpired, DoAging, and PromptForAging registry values (allName Applies to Valid range DefaultMaxLargeFile-SizeUnicode PSTs Up to 20GB(0x00005000)20 GBWarnLarge-FileSizeUnicode PSTs Up to 20GB(0x00005000)19GBMaxFileSize ANSI PSTs 0x001F4400 –0x7C004400(2 GB)1.933 GBWarnFileSize ANSI PSTs 0x00042400 –0x7C004400(2GB)1.816 GB
  7. 7. The PST File Rodeowhite papersponsored by The PST File Rodeo 7DWORDs) are set to 0. These values live under the HKEY_CUR-RENT_USERSoftwarePoliciesMicrosoftOffice12OutlookPrefer-ences key. Add them manually or using your favorite policy distri-bution tool and users will no longer be able to use AutoArchive.Auditing and Inventorying PST FilesOne of the most important tasks in PST management is findingout where PSTs exist on the network. Sadly, Microsoft still does notoffer even a rudimentary tool for doing this. Third-party vendorssuch as Sherpa Software offer inventory and management toolsthat can scan network shares and individual workstations for PSTfiles; you will need to be able to do such scans in order to bringyour PST files under control.PST files certainly have their drawbacks, and my long-standingrecommendation is to avoid their use whenever possible. For manyorganizations though, the flexibility and usefulness of PST files forusers outweigh their faults; so knowing how to effectively managethem can greatly ease your life as an Exchange administrator.Paul Robichaux is a founding partner at 3Sharp LLC, an MCSE, and anExchange MVP. He is the author of several books, including The ExchangeServer Cookbook (Published by O’Reilly and Associates); he is also an activeblogger at PST FilesAccess to Outlook Personal Folders (PST files) is necessary for legal discovery, regulatory compliance, and implementation of corporate policy. Thedecentralized nature of PST files combined with their proliferation throughout the enterprise on desktops, file servers, and laptop machines oftenmake the task of finding, accessing and managing these information stores extremely difficult for administrators. To alleviate the headaches oftencaused by PST files, Sherpa Software offers a variety of PST management tools specifically designed to locate, search, manage, migrate, and deletecontent in desktop and network-based PST files.Software Highlights:Locate, Manage, and Search PST Files—PST content is not easily accessible. When data is stored locally on desktop or laptop computers, orscattered around your network, finding PST files and collecting information can be a difficult and time-consuming task. In addition, enforcing archiving, re-tention and deletion policies are much more difficult when data stores are not centrally accessible to IT staff. Sherpa’s products provide content discovery,migration and PST management for local and network PST files by allowing administrators to create flexible rules that monitor the email infrastructure andautomatically archive and enforce corporate policies. Archive and Delete PST File Content—Transferring PST content into an archive, between e-mail stores and giving users of those PST filesaccess to the archived messages through Outlook, Sherpa’s solutions allow you to delete unwanted PST files. Since messages from the PST files are nowavailable to be searched and restored from the archives, the use of PST files and the creation of new PST files are no longer necessary. Backup and Synchronize PST Files—The practice of storing PST files on the network is not recommended or supported by Microsoft. However,data stored locally on desktop or laptop computers is difficult to manage and makes archiving, backup, and deletion policies challenging to enforce. PSTBackup Attender is a lightweight, PST file backup utility used to locate, manage, and back up PST files on user computers, eliminating the need for storingPST files on the network.Capture Statistics and Generate Reports—The ability to report on the content of all PST files across the entire organization is an importantkey step in understanding the information assets an organization possesses. With the use of Sherpa’s tools, gather comprehensive statistics and gain a totalperspective over your entire email environment, including all PST files, with information such as: total number of PSTs, total message/attachment counts,size, and age.Mail AttenderMailbox, PST File and PublicFolder ManagementLearn MoreArchive AttenderMailbox, PST File and PublicFolder ArchivingLearn MoreDiscovery AttenderElectronic Search and DiscoveryLearn MorePST BackupAttenderPST File Backup & SynchronizationLearn More