WHITEPAPER
What Installation Authors Need to Know
about COM Extraction
By Josh Stechnij
Senior Software Engineer, Flexera ...
What Installation Authors Need to Know about COM Extraction
2
What Installation Authors Need to
Know about COM Extraction
...
What Installation Authors Need to Know about COM Extraction
3
Registry Redirection
As API hooking became less reliable wit...
What Installation Authors Need to Know about COM Extraction
4
The following table describes the recognized attributes for
...
What Installation Authors Need to Know about COM Extraction
5
By changing the COM extraction method it may be
possible tha...
Flexera Software LLC
1000 East Woodfield Road,
Suite 400
Schaumburg, IL 60173 USA
Schaumburg
(Global Headquarters):
+1 800...
Upcoming SlideShare
Loading in …5
×

What Installation Authors Need to Know about COM Extraction

792 views
642 views

Published on

Is wp co_mextract_feb13

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
792
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

What Installation Authors Need to Know about COM Extraction

  1. 1. WHITEPAPER What Installation Authors Need to Know about COM Extraction By Josh Stechnij Senior Software Engineer, Flexera Software
  2. 2. What Installation Authors Need to Know about COM Extraction 2 What Installation Authors Need to Know about COM Extraction COM extraction was originally created to be a tool for populating Windows Installer COM/registry table data without having to manually determine and populate data by hand from a given COM server. Windows Installer provides the Class, Registry, Typelib (deprecated), and ProgId tables to automatically install COM server registration data in a way that allows for consistent rollback on failed installations and also provides a way for Windows Installer to hook into application usage to verify installation consistency (install on demand/self-resiliency). Over the years since COM extraction was first provided, InstallShield has updated COM extraction functionality in an attempt to provide more accurate extraction and useful functionality. This white paper aims to provide some background on how COM extraction works in InstallShield and some challenges that have come up when attempting to use COM extraction. COM Extraction Methods Employed in InstallShield In order to extract COM information from a COM server, some method needs to be used to capture COM data from the file. While static analysis of the file (type library and registration scripts) could be used, this is limited in functionality since it can only capture declarative data that is contained in obvious places in the file. More flexible methods could be used through dynamic capture of registration data as it is written to the machine while a file is being registered. Since the introduction of COM extraction in InstallShield, a few different dynamic methods have been used in an attempt to provide the most accurate extraction. API Hooking The original support for COM extraction was implemented through the use of API hooks. When COM extraction was started for a given file, APIs in Kernel32 and Advapi32 were hooked by COM extraction to monitor registry modifications on the system. API hooking provided a relatively clean approach to capturing registry changes such that the system was affected minimally by the registration process. One downside to this approach was that the registration itself was not insulated from the system. If a given file was already registered on the machine prior to running COM extraction, this would alter how most COM registration proceeded (no changes to the system would typically occur since the registration took note that its information was already present on the machine). To work around this issue, the file to be registered is first unregistered. Then, the capture through API hooking is enabled and the file registration is started. Capture of in-process COM servers such as DLLs, OCXs, etc. is a fairly simple process of hooking APIs in the COM extraction process. However, capturing registration information from out-of-process servers in the form of EXEs requires the use of a helper DLL that is injected into the process to capture. At the time this method of COM extraction was created, injecting DLLs into an arbitrary process was rather simple. Over time, with the introduction of more heavily secured operating systems such as Windows XP SP2, Windows 2003 Server, and Windows Vista, DLL injection became increasing difficult and less reliable. In addition, API hooking also became less reliable after COM servers started building with Visual Studio 2005 (likely due to changes in the registration implementations).
  3. 3. What Installation Authors Need to Know about COM Extraction 3 Registry Redirection As API hooking became less reliable with changes in OS and development tools, a new way of capturing COM information was needed. Starting with Windows 2000, the operating system allows for redirection of all registry accesses to private keys through the use of the RegOverridePredefKey API. This API is expressly provided for the purpose of software installations to allow for capturing system registry changes. Starting with InstallShield 12 (which provided mainstream support for Windows Vista), COM extraction defaulted to using registry redirection as a means of capturing COM registration data during COM extraction. This method provided a way to present a “clean” view of a system to the file being registered, eliminating the need to unregister the file prior to registering. This method resolved most outstanding issues occurring with API hooking that had cropped up over the years. In addition, this method has the advantage that the machine state prior to and after capturing registry information will not change. As with most solutions, though, some tradeoffs appeared. Since registry redirection presents a view of the registry that is available in the redirected keys (which tend to be completely empty), COM servers that depended on existing registry information would no longer capture correctly. Some workarounds to this were available, such as writing dummy ATL registry keys needed to register some components dependent on ATL. However, no generic solution to missing registry dependencies is available, and components that have such dependencies will not correctly have all COM information extracted. Kernel Mode Registry Filtering In order to resolve the issues brought about by registry redirection COM extraction while maintaining a similar level of system isolation, and to provide some additional support of .NET assemblies, InstallShield 2012 introduced a new method of extracting COM information. On Windows Vista and newer versions of Windows, InstallShield will use kernel mode registry filtering to capture registry data for COM extraction. Kernel mode registry filtering is a Windows mechanism for intercepting system-wide registry accesses through a kernel mode driver. This support was first provided on a limited basis with Windows XP. Windows Vista improved registry filtering such that it was generally usable to capture registry information from any process running on a system. Registry filtering is used by utilities like Sysinternals’ Process Monitor to able to capture registry accesses across processes on a system. Through registry filtering, the COM extraction provided in InstallShield 2012 and newer versions results in a hybrid approach that combines aspects of the API hook and registry redirection methods. A COM server being registered with register filtering COM extraction will have a view of the actual system registry like API hooking. The captured data is completely redirected by the filter driver to a private InstallShield registry key, similar to the registry redirection method. The end result is no DLL injection is required and a registering component has complete access to any registry dependencies it may have. An additional benefit of registry filtering is being able to capture registry modifications from any process, not just a process that is registering COM data. This now allows for capturing registry changes for .NET COM interop made by regasm. Since creating a registry file with regasm does not include code-level registry changes made by an assembly, the registry filtering can now be used to capture registry changes made by registering an assembly for COM interop with regasm. In addition, C++/CLI assemblies that “register” can now also be captured correctly (previous COM extraction methods resulted in failures in the assembly or CLR that caused the extraction to fail). Specifying Exclusions for COM Extraction To prevent InstallShield from extracting undesired COM data using any of the available extraction methods (either at build or design time), the Filters.xml file contains a list of registry keys that need to be excluded. Filters.xml can be customized as needed. It is available in the Support folder of an installed copy of InstallShield (typically located in C: Program FilesInstallShield<Version>). The <Exclude> element in the Filters.xml file contains a subelement for each registry key that the COM extraction process should ignore. By default, the <Exclude> element has subelements for common system registry keys that are required. Following is a sample of a registry subelement that blocks changes to an InprocServer32 registry key, all of its values, and all of its subkeys: <Registry key="HKEY_LOCAL_MACHINE SOFTWAREClassesCLSID {00000231-0000-0010-8000-00AA006D2EA4} InprocServer32"/> Following is a sample of a registry subelement that blocks changes to only the default value of an InprocServer32 registry key: <Registry key="HKEY_LOCAL_ MACHINESOFTWAREClassesCLSID {00000231-0000-0010-8000-00AA006D2EA4} InprocServer32" value=""/> Following is a sample of a registry subelement that blocks changes to only the ThreadingModel value name for an InprocServer32 registry key: <Registry key="HKEY_LOCAL_MACHINE SOFTWAREClassesCLSID {00000231-0000-0010-8000-00AA006D2EA4} InprocServer32" value="ThreadingModel"/>
  4. 4. What Installation Authors Need to Know about COM Extraction 4 The following table describes the recognized attributes for the <Registry> subelement in the Filters.xml file. Any other attributes for the <Registry> subelement are optional and are not recognized by the COM extraction process. This enables additional attributes to be used as internal comments to identify why an item is being excluded. How InstallShield Populates the Windows Installer Tables with COM Extraction Data The following table identifies how InstallShield maps the COM extraction registry entries to Windows Installer table data. The data in the Windows Installer tables allows for rollback of the COM data in the AppId, Class, MIME, ProgId, and Registry tables. Troubleshooting COM Extraction Issues With each method of COM extraction, some issues can occur with specific COM servers. The issues typically stem more from files that do not register in a normal/stock manner. The best bet in troubleshooting issues with COM extraction is to use a tool named RegSpyUI. This tool runs COM extraction outside of the InstallShield IDE and can be set to not filter any captured registry data. RegSpyUI is available in the Support folder of an installed copy of InstallShield (typically located in C:Program Files InstallShield<Version>). This tool can be used to perform one-off testing of files that do not seem to be extracting correctly in the InstallShield IDE. Another available method of troubleshooting is to change the COM extraction method that InstallShield uses. All three methods of COM extraction are still available in recent versions of InstallShield. The method used can be changed by setting the value of the UseAPIRegistryHooks registry value located in HKEY_LOCAL_MACHINESoftware InstallShieldRegSpy (on 32-bit machines) or HKEY_LOCAL_ MACHINESOFTWAREWow6432NodeInstallShield RegSpy (on 64-bit machines). The following values are available: Attribute Description key This attribute must be lowercase. The value of this attribute (for example, HKEY_LOCAL_MACHINE SOFTWAREClassesCLSID00000231-0000- 0010-8000-00AA006D2EA4}InprocServer32 in the above examples) indicates the name of the registry key that needs to be filtered. value This attribute is optional: • To block changes to an entire registry key, do not include the value attribute. • To block changes to the default value of the specified key, set the value of this attribute to a null value. • To block changes to the name of a value of the specified key, set the value of this attribute to the name of that registry value. Data Found During COM Extraction Equivalent Windows Installer Data AppID entries in HKEY_ CLASSES_ROOT AppId table CLSID entries in HKEY_ CLASSES_ROOT Class table MIME entries in HKEY_ CLASSES_ROOT MIME table, with a CLSID association ProgID entries that are referenced by CLSID entries in HKEY_CLASSES_ROOT ProgId table Miscellaneous registry data Registry table InstallShield replaces file references with a formatted file key in short name format ([!filekey]). InstallShield replaces directory paths that are tied to the component with a component key path reference ([$componentname]). UseAPIRegistryHooks value Value Description Value not present Defaults to kernel mode registry filter with InstallShield 2012 and newer on Vista and newer versions of Windows; defaults to registry redirection on older versions and platforms 0 API hooking 1 Registry redirection 2 Kernel mode registry filter with InstallShield 2012 or newer on Vista or newer versions of Windows; invalid otherwise
  5. 5. What Installation Authors Need to Know about COM Extraction 5 By changing the COM extraction method it may be possible that another method is able to successfully capture COM registration information from a server that does not work well with another method. A Word About Seemingly Hung COM Extraction Processes A few reports of COM extraction hanging during build (IsRegSpy.exe does not exit) have come from InstallShield product support services. In nearly all of these cases, registering manually with regsvr32.exe resulted in a hung regsvr32 process. Upon further debugging of either IsRegSpy.exe or regsvr32.exe processes, the offending DLL was stuck in its DllMain function due to calling into another DLL or otherwise violating the system loader lock hierarchy. When authoring any in-process code (DLL, OCX, etc.) that is loaded through LoadLibrary, it is absolutely critical that all rules regarding what is safe or not safe to do in DllMain is respected. Issues with DllMain resulting in deadlocks are a fairly well known issue. If a DLL seems to be hanging when performing COM extraction, check to see if it also hangs with regsvr32.exe on the same or a similarly configured machine. If it does, attach a debugger to see if the DLL is stuck in its DllMain. Properly authoring a DllMain will avoid this type of behavior. Summary This white paper discusses the various types of COM extraction that are available in InstallShield and explains how to switch from one method to another. This information should help installation authors make appropriate choices for COM servers as they design installations in InstallShield.
  6. 6. Flexera Software LLC 1000 East Woodfield Road, Suite 400 Schaumburg, IL 60173 USA Schaumburg (Global Headquarters): +1 800-809-5659 United Kingdom (Europe, Middle East Headquarters): +44 870-871-1111 +44 870-873-6300 Australia (Asia, Pacific Headquarters): +61 3-9895-2000 For more office locations visit: www.flexerasoftware.com Copyright © 2013 Flexera Software LLC. All other brand and product names mentioned herein may be the trademarks and registered trademarks of their respective owners. IS_WP_COMextract_Feb13

×