Configuring Microsoft Windows IP Security to Operate with HP ...
Upcoming SlideShare
Loading in...5
×
 

Configuring Microsoft Windows IP Security to Operate with HP ...

on

  • 918 views

 

Statistics

Views

Total Views
918
Views on SlideShare
918
Embed Views
0

Actions

Likes
0
Downloads
1
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Configuring Microsoft Windows IP Security to Operate with HP ... Configuring Microsoft Windows IP Security to Operate with HP ... Document Transcript

  • HP-UX IPSec Configuring Microsoft Windows IP Security to Operate with HP-UX IPSec HP Part Number: J4256-90025 Published: June 2007 Edition: 1.0
  • 2
  • Table of Contents About This Document.........................................................................................................9 Typographic Conventions......................................................................................................................9 Introduction..........................................................................................................................................11 Testing Environment.......................................................................................................................11 Known Problem with Windows 2000 SP1 and SP2...................................................................11 Protocol Implementation Differences..............................................................................................12 Windows IP Security Configuration Overview....................................................................................13 Configuring a Windows Host-to-Host Policy.......................................................................................14 Step 1: Starting the IP Security Policies Snap-in Configuration Utility...........................................15 Step 2: Creating a Policy..................................................................................................................15 Step 3: Adding a Rule......................................................................................................................16 Step 4: Creating the IP Filter List and Filters for the Rule...............................................................18 Step 5: Configuring Filter Actions for the Rule...............................................................................21 Step 6: Configuring the IKE Authentication Method and Preshared Key for the Rule..................25 Step 7: Configuring the Connection Type for the Rule...................................................................26 Step 8: Modifying IKE Parameters for the Policy............................................................................26 Step 9: Starting the IP Security Service............................................................................................29 Step 10: Assigning the IP Security Policy........................................................................................30 Step 11: Verifying the Configuration...............................................................................................31 Example...........................................................................................................................................31 Windows Configuration.............................................................................................................31 HP-UX Configuration................................................................................................................32 Additional Options...............................................................................................................32 Configuring a Windows End-to-End Tunnel Policy.............................................................................33 Outbound Tunnel Rule Requirements............................................................................................33 Inbound Tunnel Rule Requirements...............................................................................................33 Configuring a Tunnel Rule..............................................................................................................33 Example...........................................................................................................................................34 Windows Configuration.............................................................................................................34 Outbound Rule.....................................................................................................................34 Inbound Rule........................................................................................................................35 Additional Parameters..........................................................................................................36 HP-UX Configuration................................................................................................................37 Troubleshooting Tips............................................................................................................................38 Using IKE Logging on HP-UX Systems..........................................................................................38 Using IKE Logging on Windows Systems.......................................................................................38 Additional Windows Troubleshooting Tools..................................................................................39 Comparing HP-UX and Windows IPsec Configuration Parameters....................................................40 Mirrored Filters...............................................................................................................................41 Filter Selection.................................................................................................................................42 IKE Parameter Selection..................................................................................................................42 IKE SA Key (Master Key) Lifetime Values......................................................................................42 HP-UX IKE SA Lifetime Values.................................................................................................42 Windows IKE SA Lifetime Values..............................................................................................43 Maximum Quick Modes..................................................................................................................43 Perfect Forward Secrecy (PFS).........................................................................................................43 IPsec SA Key (Session Key) Lifetime Values...................................................................................43 HP-UX IPsec SA Lifetime Values...............................................................................................43 Windows IPsec SA Lifetime Values...........................................................................................44 Related Publications..............................................................................................................................45 Table of Contents 3
  • glossary.............................................................................................................................47 4 Table of Contents
  • List of Figures 1 IP Security Policy Wizard..............................................................................................................16 2 Rules Tab.......................................................................................................................................17 3 Rule Properties Dialog Box...........................................................................................................17 4 Creating an IP Filter List...............................................................................................................18 5 Address Tab for Filter Properties..................................................................................................19 6 Protocol Tab for Filter Properties..................................................................................................20 7 Selecting the Filter List for a Rule.................................................................................................21 8 Security Methods for Filter Action................................................................................................22 9 Security Method Dialog Box.........................................................................................................23 10 Custom Security Methods Settings Dialog Box............................................................................24 11 Selecting the Filter Action.............................................................................................................25 12 Configuring A Preshared Key.......................................................................................................26 13 General Policy Properties Dialog Box ..........................................................................................27 14 Key Exchange Settings Dialog Box ...............................................................................................28 15 IKE Security Algorithms Dialog Box ............................................................................................29 16 IPSEC Services Properties Dialog Box...........................................................................................30 17 Assigning the IP Security Policy...................................................................................................31 18 Outbound Rule Filter....................................................................................................................35 19 Outbound Rule Tunnel Settings....................................................................................................35 20 Inbound Rule Filter.......................................................................................................................36 21 Inbound Rule Tunnel Settings.......................................................................................................36 5
  • 6
  • List of Tables 1 IPsec Parameters on Windows and HP-UX .................................................................................40 7
  • 8
  • About This Document This document describes how to configure Microsoft Windows IP Security to operate with the HP-UX IPSec product. Typographic Conventions This document uses the following typographical conventions: %, $, or # A percent sign represents the C shell system prompt. A dollar sign represents the system prompt for the Bourne, Korn, and POSIX shells. A number sign represents the superuser prompt. audit(5) A manpage. The manpage name is audit, and it is located in Section 5. Command A command name or qualified command phrase. Computer output Text displayed by the computer. Ctrl+x A key sequence. A sequence such as Ctrl+x indicates that you must hold down the key labeled Ctrl while you press another key or mouse button. ENVIRONMENT VARIABLE The name of an environment variable, for example, PATH. [ERROR NAME] The name of an error, usually returned in the errno variable. Key The name of a keyboard key. Return and Enter both refer to the same key. Term The defined use of an important word or phrase. User input Commands and other text that you type. Variable The name of a placeholder in a command, function, or other syntax display that you replace with an actual value. [] The contents are optional in syntax. If the contents are a list separated by |, you must choose one of the items. {} The contents are required in syntax. If the contents are a list separated by |, you must choose one of the items. ... The preceding element can be repeated an arbitrary number of times. Indicates the continuation of a code example. | Separates items in a list of choices. WARNING A warning calls attention to important information that if not understood or followed will result in personal injury or nonrecoverable system problems. CAUTION A caution calls attention to important information that if not understood or followed will result in data loss, data corruption, or damage to hardware or software. IMPORTANT This alert provides essential information to explain a concept or to complete a task NOTE A note contains additional information to emphasize or supplement important points of the main text. Typographic Conventions 9
  • 10
  • Introduction This document contains the following sections: • “Windows IP Security Configuration Overview” (page 13) This section contains a brief overview of the Windows IPsec configuration parameters and the terminology used in the Windows IPsec configuration utilities. • “Configuring a Windows Host-to-Host Policy” (page 14) This section describes how to configure IP Security (IPsec) on a Windows client to secure IP packets sent to and received from an HP-UX system in a host-to-host topology. • “Configuring a Windows End-to-End Tunnel Policy” (page 33) This section describes how to configure IPsec on a Windows client to secure IP packets sent to and received from an HP-UX system in an end-to-end tunnel topology. • “Troubleshooting Tips” (page 38) This section contains troubleshooting tips. • “Comparing HP-UX and Windows IPsec Configuration Parameters” (page 40) This section compares how HP-UX and Windows systems configure and use IPsec parameters. • “Related Publications” (page 45) This section contains a list of related HP-UX and Microsoft publications. The procedures and examples in this document use preshared keys for IKE authentication. For information about using certificates for IKE authentication with Microsoft Windows, see Using Microsoft Windows Certificates with HP-UX IPSec, available at http://docs.hp.com. The intended audience for this document is an HP-UX IPSec administrator who is familiar with the HP-UX IPSec product and with the IP Security protocol suite. If you are not familiar with the HP-UX IPSec product, see the appropriate version of the HP-UX IPSec Administrator's Guide, available at http://docs.hp.com. NOTE: The IP Security protocol suite is often referred to as IPsec. The HP-UX product that implements the IP Security protocol suite is HP-UX IPSec. Testing Environment The procedures in this white paper were tested using the following environment: Component Description HP-UX IPSec Versions A.02.01 and A.02.01.01 Microsoft Windows Client Windows XP with Service Pack 2 (SP2) Known Problem with Windows 2000 SP1 and SP2 For this white paper, HP did not test with Windows 2000 systems. However, there is a known problem with Windows 2000 base systems and Windows 2000 systems with Service Pack 1 (SP1) or Service Pack 2 (SP2). The IP Security module on these systems does not properly process IPSec ESP packets that are fragmented across IP packets and drops these packets. The symptoms vary according to how the applications handle the dropped packets. This problem is caused by a defect in the Windows 2000 SP1/ SP2 software and is fixed in Windows 2000 Service Pack 3 (SP3). Introduction 11
  • The above problem typically occurs with ESP-encrypted UDP or ICMP packets that are fragmented by IP. HP-UX 11i systems minimize IP fragmentation of ESP-encrypted TCP packets. You may still experience problems with ESP-encrypted TCP packets sent from an HP-UX system to a Windows 2000 system if an intermediate IP gateway fragments the ESP packet. Protocol Implementation Differences HP-UX and Microsoft Windows both implement the IP Security protocol suite. However, there are features in the protocol suite that HP-UX implemented which Microsoft did not implement, and vice-versa. The following features are implemented by HP-UX IPSec version A.02.01 but not by Microsoft Windows XP: • Advanced Encryption Standard (AES): HP-UX IPSec supports ESP encryption using the following protocols: AES, Triple Data Encryption Standard (3DES), and Data Encryption Standard (DES). Windows XP and Windows 2000 support 3DES and DES, but do not support AES. • Aggressive Mode (AM): HP-UX supports AM exchanges to establish IKE Security Associations (SAs). AM is an optional feature and is not supported on Windows. The following features are implemented by Microsoft Windows XP, but not by HP-UX IPSec version A.02.01: • Kerberos: Windows supports Internet Key Exchange (IKE) authentication using Kerberos. RFC 2408 defines an optional Kerberos Token payload, but does not describe how to implement it. This feature is not supported on HP-UX. • Perfect Forward Secrecy (PFS) for keys only: HP-UX IPSec supports PFS for keys in conjunction with PFS for all identities, but does not support PFS for keys only. Windows supports PFS for keys only (“session key PFS”) and PFS for keys in conjuctions with PFS for all identities (“master key PFS”). See “Perfect Forward Secrecy (PFS)” (page 43) for more information. 12
  • Windows IP Security Configuration Overview On Microsoft Windows systems, all IP Security (IPsec) configuration data resides in a single IP Security policy. You can create multiple IP Security policies, but only one local policy can be active on the system. If the system is a member of a Windows Active Directory domain, you can use an IP Security policy from a Group Policy defined for the domain. A Windows IP Security policy defines the parameters used to negotiate Internet Key Exchange Security Associations (IKE SAs) and IPsec SAs. An IKE SA is a bi-directional, secure communication channel that two peers establish before negotiating IPSec SAs. One of the primary activities during the IKE SA negotiation is the authentication of each peer's identity. After two peers establish an IKE SA, they can negotiate IPsec SAs. Each IPsec SA is a uni-directional, secure communication channel. The IPsec SA operating parameters include the IPsec protocol used (Encapsulating Security Payload, ESP, or Authentication Header, AH) and the cryptographic algorithms. IPsec SAs are negotiated in pairs (one for each direction of traffic). Each Windows IP Security policy contains the following components: • Rules A policy contains one or more rules. The main purpose of a rule is to assign actions for address filters. Each rule contains the following components: — IP Filter List An IP Filter list contains one or more filters. Each filter contains the following components: ◦ Addressing The source and destination IP addresses, network masks, and a flag that indicates if the filter is mirrored (bi-directional). ◦ Protocol The upper-layer protocol, and source and destination ports, if applicable. ◦ Description The filter name and a description. — Filter Action The filter action specifies the action to take for the rule, and can be one of the following actions: ◦ allow: allow the packet to pass ◦ block: discard the packet ◦ negotiate security: negotiate IPsec Authentication Header (AH) or Encapsulating Security Payload (ESP) Security Associations (SAs) — Authentication Methods The authentication methods specify the type of Internet Key Exchange (IKE) authentication to use (preshared key or certificates with RSA signatures). If you are using preshared key authentication, the authentication methods also specify the value of the preshared key. Windows IP Security Configuration Overview 13
  • — Tunnel Settings The tunnel settings specify if the rule is a tunnel rule. If it is a tunnel rule, the settings also specify the tunnel destination endpoint. — Connection Type The connection type specifies the connection (link) types for the rule, such as LAN. • General The general parameters for a policy specify IKE SA parameters, such as the IKE encryption algorithm, IKE hash (integrity algorithm), Diffie-Hellman Group, and IKE SA key lifetimes. The parameters correspond to IKE SA proposals. You can configure multiple IKE SA proposals and specify the preference order. The proposals are used for all rules in the policy. By comparison, a minimal HP-UX IPSec configuration consists of one or more IPsec host policies, one or more IKE policies, and one or more authentication records. The IPsec host policies specify address filters, and you can configure separate IKE policies for each peer. “Comparing HP-UX and Windows IPsec Configuration Parameters” (page 40) lists IPsec configuration parameters and how they are configured in the HP-UX IPSec and the Windows IP Security configuration utilities. Configuring a Windows Host-to-Host Policy This section describes one method for configuring host-to-host policy on a Windows XP client using the IP Security Policies snap-in utility. Windows also supports command-line utilities to configure IP Security policies: ipseccmd on Windows XP systems and netsh on Windows 2003 systems. For more information about these utilities, see the Windows documentation set. To use this method, complete the following steps: 1. Start the IP Security Policies snap-in utility. See “Step 1: Starting the IP Security Policies Snap-in Configuration Utility” (page 15). 2. Create an IP Security policy. See “Step 2: Creating a Policy” (page 15). 3. Add a rule to the policy. See “Step 3: Adding a Rule” (page 16). 4. Create a Filter List for the rule and configure filters. See “Step 4: Creating the IP Filter List and Filters for the Rule” (page 18). 5. Configure filter actions for the rule. The filter actions contain IPsec transforms or other actions. See “Step 5: Configuring Filter Actions for the Rule” (page 21). 6. Configure the IKE authentication method and preshared key for the rule. See “Step 6: Configuring the IKE Authentication Method and Preshared Key for the Rule” (page 25). 7. Specify the network link (connection) types for the rule. See“Step 7: Configuring the Connection Type for the Rule” (page 26). 8. Modify the IKE SA parameters for the policy. By default, Windows clients will use IKE SA parameters that are compatible with the default HP-UX IPSec parameters. If these parameters are acceptable, you can skip this step. See “Step 8: Modifying IKE Parameters for the Policy” (page 26). 9. Start the IP Security service. The IP Security service must be running before you can assign the new IP Security policy. See “Step 9: Starting the IP Security Service” (page 29). 10. Assign (activate) the new IP Security Policy. See “Step 10: Assigning the IP Security Policy” (page 30). 11. Verify the configuration. See “Step 11: Verifying the Configuration” (page 31). Because this is a host-to-host rule, we will use the default value for the rule tunnel setting (no tunnel). For information about configuring a tunnel rule and the tunnel setting, see “Configuring a Windows End-to-End Tunnel Policy” (page 33). 14
  • Step 1: Starting the IP Security Policies Snap-in Configuration Utility Use the following procedure to start the IP Security Policies configuration utility. This utility is a snap-in module for the Microsoft Management Console (MMC). 1. Start the Microsoft Management Console (MMC). From the Microsoft Start menu, click Run and type MMC. Click OK. 2. If the IP Security Policies snap-in configuration utility is not loaded, use the following procedure to add it: a. From the MMC window, click File→Add/Remove Snap-in. b. From the Add/Remove Standalone Snap-in window, click Add. c. From the Add Standalone Snap-in window, scroll down to IP Security Policy Management and select it. Click Add. d. In the Select Computer or Domain window, select Local computer (in this procedure, we are configuring IP Security for the local computer). Click Finish. e. Close the Add Standalone Snap-in window by clicking Close. f. Close the Add/Remove Snap-in window by clicking OK. Step 2: Creating a Policy Use the following procedure to create an IP Security policy. An IP Security policy is a set of IPsec configuration parameters. Only one local IP Security policy can be active (assigned) on a system. 1. In the left navigation pane of the IP Security Policy Management snap-in, click IP Security Policies on Local Computer to display all IP Security Policies. Depending on your Windows platform, there may be IP Security Policies already configured. 2. Right click IP Security Policies on Local Computer and select Create IP Security Policy. 3. The Policy Wizard starts and displays a startup message. Click Next. 4. The Policy Wizard opens the IP Security Policy Name window. Enter a name in the Name field. This name is used only for internal identification. Click Next. 5. The Policy Wizard opens the Requests for Secure Communication window. Clear the Activate the default response rule check box, as shown in Figure 1. (The default response rule is a pre-configured rule that causes the Windows system to dynamically build a filter list based on the receipt of IKE requests. By default, the Windows system attempts to use IPsec only if it receives an IKE request from a remote system.) Click Next. Configuring a Windows Host-to-Host Policy 15
  • Figure 1 IP Security Policy Wizard 6. The Policy Wizard opens the Completing the IP Security policy wizard window. Select the Edit properties check box if it is not already selected. Click Finish. The IP Security configuration utility opens the Policy Properties dialog box. The title of the window will be name Policy, where name is the policy name. Step 3: Adding a Rule The primary purpose of a rule is to assign actions to filters. A rule also specifies IKE authentication methods. Use the following procedure to add a rule to the IP Security policy: 1. Select the Rules tab in the Policy Properties dialog box. Clear the Use Add Wizard box check box if it is selected (Figure 2). Click Add. 16
  • Figure 2 Rules Tab 2. The IP Security configuration utility opens the Rule Properties dialog box, which has a tab for each category of rule configuration data: IP Filter List, Filter Action, Authentication Methods, Tunnel Setting, and Connection Type ( Figure 3). Figure 3 Rule Properties Dialog Box Configuring a Windows Host-to-Host Policy 17
  • TIP: After you have created a rule, you can open the Rules Properties dialog box by right clicking the rule and selecting Properties. Step 4: Creating the IP Filter List and Filters for the Rule An IP filter list can contain one or more filters. IPsec uses the filters to determine which rule to apply to an IP packet. The IP Security configuration utility displays the rules for a policy in reverse alphabetical order based on the name of the IP filter list for the rule. Filter Order There is no method for specifying the search or priority order for the filters in a rule or for the order of rules in a policy. The Windows IP Security module automatically creates an internal filter list and orders the filters from most specific to least specific. Use the following procedure to configure the IP filter list and a filter: 1. Create an IP filter list. Select the IP Filter tab from the Rule Properties dialog box. The IP Filter List tab shows a list of filters already defined for IP Security policies. Each rule can have only one filter list, but the filter list can specify multiple filters. In this example, we will create a new filter list that contains one filter. Click Add at the bottom of the dialog box. The IP Security configuration utility opens the IP Filter List dialog box. In the Name field, enter a name for the filter list. This name is used only for internal identification. Optionally, add a description. In Figure 4, the administrator enters the name foo. Clear the Use Add Wizard box check box if it is selected. Click Add. Figure 4 Creating an IP Filter List 18
  • The IP Security configuration utility opens a Filter Properties dialog box. 2. Select the Addressing tab in the Filter Properties dialog box. Use the drop-down menus to specify the address types for the source and destination addresses. The selections are: • My IP Address1 • Any IP Address • A specific DNS Name • A specific IP Address • A specific IP Subnet Enter the source and destination IP addresses or DNS names for the filter. If you selected A specific IP Subnet, enter the subnet mask. WARNING! Be careful when configuring filters that affect packets required for basic network operation, such as packets exchanged with DNS servers and ICMP packets exchanged with routers. If you configure a policy that requires IP Security for these packets and the remote node does not support IP Security, your system can lose network functionality. Leave the Mirrored check box selected, which creates a bi-directional filter that applies to packets to and from the destination system. See “Mirrored Filters” (page 41) for more information about mirrored filters. In Figure 5, the administrator specifies an address filter with the Windows system address (10.1.1.1) as the source address and the HP-UX system address (10.2.2.2) as the destination address. The Mirrored check box is selected, so the address filter also matches packets from the HP-UX system. Figure 5 Address Tab for Filter Properties 3. Select the Protocol tab in the Filter Properties dialog box. By default, the filter applies to all protocol types. Select the protocol type (for example, TCP) from the drop-down box. If you select TCP or UDP, you can also specify the From (source) port and To (destination) port. Click OK to return to the Filter Properties dialog box. 1. HP-UX did not test the My IP Address selection with multihomed Windows systems. However, the Windows documentation states that in a multi-homed system, My IP Address matches every IP address on the system. Configuring a Windows Host-to-Host Policy 19
  • In Figure 6, the administrator specifies protocol information for a Windows system that will be a telnet client. The protocol type is TCP, the source port is a wildcard (any port), and the destination port is the IANA registered TCP port number for the telnet service, 23. Figure 6 Protocol Tab for Filter Properties 20
  • 4. From the IP Filter List dialog box, you can add another filter to the filter list by clicking the Add button. Click OK in the IP Filter List dialog box to return to the IP Filter List tab in the Rule Properties dialog box. 5. Add the filter list to the rule by selecting the option button for the filter list you just created. In Figure 7, the administrator added the filter list foo for the rule. Figure 7 Selecting the Filter List for a Rule Step 5: Configuring Filter Actions for the Rule The filter action specifies the action to take for the rule, such as allow (pass), block (discard), or negotiate security (negotiate IPsec AH or ESP Security Associations). If you select negotiate security, the filter action also specifies parameters for IPsec Security Association (SA) proposals: ESP or AH transforms and IPSec SA key lifetimes. A rule can have only one filter action, but the filter action can specify multiple IPsec SA proposals. You can specify the order for the IPsec SA proposals. The filter actions you configure in the Windows IP Security rule must be compatible with the value or values specified for the -action argument in the HP-UX ipsec_config add host or add tunnel command. Use the following procedure to configure filter actions: 1. Select the Filter Action tab from the Rule Properties dialog box. The Filter Action tab shows a list of filter actions already defined for IP Security. In this procedure, we will create a new filter action. Clear the Use Add Wizard check box if it is selected and click Add. 2. The IP Security configuration utility opens the Filter Action Properties dialog box with the following tabs: Configuring a Windows Host-to-Host Policy 21
  • • Security Methods • General Select the Security Methods tab, then select Negotiate security. Verify that the following check boxes are not selected:2 • Accept unsecured communication, but always respond using IPSec. • Allow unsecured communication with non-IPSec-aware computer. In addition, verify that the Session key perfect forward secrecy (PFS) check box is not selected. (HP-UX does not support session key PFS, also referred to as PFS for keys only. HP-UX supports PFS for keys only in conjunction with PFS for identities. See “Perfect Forward Secrecy (PFS)” (page 43) for more information.) For example: Figure 8 Security Methods for Filter Action Click Add. The IP Security configuration utility opens the Security Method dialog box (Figure 9): 2. HP-UX IPSec does not have options that are equivalent to these check boxes. If an HP-UX IPsec policy requires IP security, then HP-UX always requires IP security for packets that match the policy and drops any packets that match the policy but are not secured. 22
  • Figure 9 Security Method Dialog Box The Encryption and Integrity and Integrity only methods each correspond to a set of predefined parameters for an IPsec SA proposal, including an IPsec transform type (such as ESP). The transforms and additional SA parameters defined for these methods may vary according to the Windows release installed. On Windows XP systems with SP2, these methods are defined as follows: • Encryption and Integrity Authenticated ESP using 3DES encryption and SHA1 authentication (this is equivalent to the HP-UX IPSec transform ESP_3DES_HMAC_SHA1). No SA lifetimes are specified, and these settings are compatible with the HP-UX default SA lifetimes (see “IPsec SA Key (Session Key) Lifetime Values” (page 43) for more information). • Integrity only Authenticated ESP using NULL encryption and SHA1 authentication. (this is equivalent to the HP-UX IPSec transform ESP_NULL_HMAC_SHA1) No SA lifetimes are specified, and these settings are compatible with the HP-UX default SA lifetimes (see “IPsec SA Key (Session Key) Lifetime Values” (page 43) for more information). To use a predefined method, select the appropriate method from the list and click OK to return to the Security Methods tab in the Filter Actions dialog box. To create a custom method, use the following procedure: a. Select Custom in the Security Method dialog box, then click Settings to open the Custom Security Method Settings dialog box. b. In the Custom Security Method Settings dialog box (Figure 10), select the appropriate transform type, algorithms, and session key lifetimes. See “IPsec SA Key (Session Key) Lifetime Values” (page 43) for more information about IPsec session key lifetimes. Configuring a Windows Host-to-Host Policy 23
  • Figure 10 Custom Security Methods Settings Dialog Box c. Click OK to return to the Security Methods tab in the Filter Actions dialog box. If the parameters you configured for a custom method match a predefined method, the configuration utility will display an informative message and select the matching predefined method. 3. From the Security Methods tab, you can add more methods (IPsec SA proposals) by clicking the Add button. If you have multiple IPsec SA proposals, the configuration utility lists them in the preference order IKE will use when negotiating IPsec SAs. You can change the order by using the Move up and Move down buttons. You can also use the Delete button to delete IPsec SA proposals (such as proposals that use DES, which has been cracked—data encrypted using DES has been decrypted by unauthorized parties) . 4. (Optional) Configure the action name. When you create a new action (transform), the configuration utility assigns it the name New Filter Action. To change the name, select the General tab from the Action Properties dialog box. Enter a new name and click OK to return to the Filter Action tab in the Rule Properties dialog box. 5. Apply the new action to the rule. In the Filter Action tab, click on the option button for the filter action you just created to apply the action to the rule you are configuring. For example, in Figure 11, the administrator created the action my_action and selected it for the rule. Click Apply. 24
  • Figure 1 Selecting the Filter Action 1 Step 6: Configuring the IKE Authentication Method and Preshared Key for the Rule When configuring a rule to be compatible with HP-UX IPSec, the authentication method specifies the IKE authentication method (preshared key or certificates) for IPsec. The authentication method must match the value specified for the -authentication argument in the ipsec_config add ike command. Windows also allows you to configure Kerberos (Active Directory) as an authentication method for IKE (this is the default), but HP-UX does not support this authentication method. Use the following procedure to configure the IKE authentication method: 1. Select the Authentication Methods tab from the Rule Properties dialog box. 2. Click Add to open the Authentication Method dialog box. 3. To use IKE authentication with a preshared key, select Use this string. This is equivalent to specifying -authentication PSK in the ipsec_config add ike command. Enter the preshared key as ASCII text. Do not enclose the key in double quotes. The preshared key must match the preshared key on the HP-UX system, which is configured using the -preshared argument in the ipsec_config add auth command. For example: Configuring a Windows Host-to-Host Policy 25
  • Figure 12 Configuring A Preshared Key To use IKE authentication with certificates, select Use a certificate from this certification authority (CA). Click Browse. The IP Security configuration utility opens a Select Certificate box with a list of CA certificates stored on your system. Select the CA for the appropriate CA and click OK. (For additional information about configuring Microsoft Windows certificates, see Using Microsoft Windows Certificates with HP-UX IPSec, available at http://docs.hp.com. 4. After you have specified the IKE authentication method, click OK to return to the Authentication Methods tab in the Rule Properties dialog box. 5. In the Rule Properties dialog box, remove the Kerberos authentication method from the authentication methods list by highlighting it and clicking Remove. The configuration utility will display a confirmation message (Are you sure?). Click Yes Step 7: Configuring the Connection Type for the Rule The connection type specifies the types of network connection to which the rule will apply. By default, the IP Security configuration utility creates rules that apply to all network connection types. To change the connection type, use the following procedure: 1. Select the Connection Type tab from the Rule Properties dialog box. 2. The IP Security configuration utility opens the Connection Type dialog box with the following selections: • All network connections: the rule applies to all network connections • Local area network (LAN): the rule applies only to LAN connections • Remote access: the rule applies only to VPN and dial-up connections Select the appropriate connection type and click OK. If you have configured all the required parameters for a rule, the IP Security configuration utility will return to the Policy Properties dialog box. Step 8: Modifying IKE Parameters for the Policy By default, HP-UX IPSec negotiates IKE SAs using a single proposal with the following parameters: 26
  • • Encryption algorithm: 3DES • Hash algorithm: MD5 • Diffie-Hellman Group: 2 • Maximum lifetime: 28,800 seconds (8 hours) • Maximum Quick Modes: 100 You can specify alternative values for the above parameters in the ipsec_config add ike command. On Windows XP systems with SP2, IP Security policies are pre-configured with four IKE SA proposals. The second IKE proposal matches the default HP-UX IPSec IKE proposal3, and will be used by the two systems if no changes are made to the default configuration data. If these IKE parameters meet your security requirements, you do not need to modify the IKE parameters and can skip to “Step 10: Assigning the IP Security Policy” (page 30). Use the following procedure to modify the Windows IKE SA parameters: 1. From the Policy Properties dialog box, select the General tag. The IP Security configuration utility opens the General dialog box (Figure 13). Click Advanced4. (Ignore the field labeled Check for policy changes. This field is used only when the policy is stored in an Active Directory.) Figure 13 General Policy Properties Dialog Box 2. The IP Security configuration utility opens the Key Exchange Settings dialog box (Figure 14). 3. By default, the first Windows XP proposal has the following parameters: Encryption - 3DES; Hash - SHA1; Diffie-Hellman Group - 2. The third and fourth Windows proposals are weaker, and use DES encryption and Diffie-Hellman Group 1. Refer to the Windows documentation for more information. 4. On Windows 2003 servers, this button is labeled Settings. Configuring a Windows Host-to-Host Policy 27
  • Figure 14 Key Exchange Settings Dialog Box Configure the fields as follows: • Master key perfect forward secrecy (PFS) Selecting this check box sets the maximum number of IPsec or Quick Mode (QM) negotiations that IKE can perform using an IKE SA to 1. It is equivalent to specifying -maxqm 1 in the ipsec_config add ike command. PFS is computationally expensive and HP recommends that you enable it only in hostile environments. See “Maximum Quick Modes” (page 43) and “Perfect Forward Secrecy (PFS)” (page 43) for more information. • Authenticate and generate a new key after every: ____ minutes This field specifies the maximum lifetime for an IKE SA in units of time. It is equivalent to the -life argument of the ipsec_add ike command. TIP: Note that this value is specified in minutes on Windows systems and in seconds on HP-UX systems. • Authenticate and generate a new key after every: ____ sessions This field specifies the maximum QM negotiations per IKE SA. It is equivalent to the -maxqm argument of the ipsec_add ike command. See “Maximum Quick Modes” (page 43) and “Perfect Forward Secrecy (PFS)” (page 43) for more information. Modify the values as appropriate. If you do not want to modify the IKE encryption, hash, or Diffie-Hellman Group parameters, click OK to return to the General dialog box and continue to the next step. To modify IKE encryption algorithm, hash algorithm, or Diffie-Hellman Group parameters, click Methods. The IP Security configuration utility opens the Key Exchange Security Methods dialog box, which lists IKE algorithms and Diffie-Hellman Groups that correspond to IKE SA proposals in order of preference. You can change the order by using the Move up and Move down buttons. You can also delete IKE SA proposals (such as proposals that use DES, which has been cracked) using the Delete button. To add another IKE SA proposal (method), click Add. The IP Security configuration utility opens the IKE Security algorithms dialog box (Figure 15). 28
  • Figure 15 IKE Security Algorithms Dialog Box Use the drop-down menus to select the appropriate integrity algorithm, encryption algorithm, and Diffie-Hellman Group (these are equivalent to the -hash, -encryption, and -group arguments of the ipsec_config add ike command). 3. Click OK to return to the Key Exchange Security Methods dialog box. Click OK to return to the Key Exchange Settings dialog box. Click Close to close the Policy Properties dialog box. Step 9: Starting the IP Security Service Use the following procedure to start the IP Security service. The IP Security service must be running before you can assign a policy. 1. From the Microsoft Start menu, select Control Panel→Administrative Tools→Services. 2. Scroll down and select IPSEC Services. 3. The Service manager opens the IPSEC Services Properties dialog box (Figure 16). In the Startup type selection menu, select Automatic. If the Service status is not Started, click Start. Click OK to close the IPSEC Services Properties dialog box. Close the Services dialog box. Configuring a Windows Host-to-Host Policy 29
  • Figure 16 IPSEC Services Properties Dialog Box Alternatively, you can manually start the IP Security service by entering the following Windows command: net start policyagent You can also use the following sequence of commands to manually stop and restart the IP Security service. This also clears any existing IPsec SAs,: net stop policyagent net start policyagent Step 10: Assigning the IP Security Policy The IP Security subsystem will not use the new policy until you assign (activate) it. Only one IP Security policy can be assigned or active for the system. To assign the new IP Security policy, return to the MMC window. Right click the policy in the MMC window and select Assign, as shown in Figure 17. 30
  • Figure 17 Assigning the IP Security Policy Step 1 Verifying the Configuration 1: To verify your configuration, generate traffic that matches the address filter. On the HP-UX system, enter the following command to verify that the IKE SA and IPsec SAs are established: ipsec_report -sa Example In this example, IPsec secures telnet connections from the Windows system to the HP-UX system, using authenticated ESP. The Windows system's address is 10.1.1.1 The HP-UX system's address is 10.2.2.2. Windows Configuration The Windows administrator configures and assigns an IP Security policy with the following parameters: • One rule, with the following parameters: — Filter List: One filter, with the following parameters: ◦ Addressing: – Source address: the Windows system's address. – Destination address: the HP-UX system's address. – Mirrored: yes (the Mirrored box is selected). These parameters are shown in Figure 5 (page 19). ◦ Protocol: TCP; source port any, destination port 23 (telnet). – Protocol: TCP – From port: any – To port: 23 (telnet server) These parameters are shown in Figure 6 (page 20). — Filter Action: Negotiate security, using the default settings for Encryption and Integrity (authenticated ESP using 3DES and SHA1). — Authentication Method: IKE using the preshared key my_preshared_key, as shown in Figure 12 (page 26). — Tunnel Settings: No tunnel (this is the default). — Connection Type: All network connections (this is the default). • General parameters: The general parameters for the policy are set to the default values (four IKE SA proposals, including 3DES encryption, SHA1 integrity and Diffie-Hellman Group 2). Configuring a Windows Host-to-Host Policy 31
  • HP-UX Configuration On the HP-UX system, the administrator configures the following policies and records: ipsec_config add host telnet_from_foo1 -source 10.2.2.2/32/TELNET -destination 10.1.1.1 -action ESP_3DES_HMAC_SHA1 ipsec_config add ike foo1 -remote 10.1.1.1 -auth PSK ipsec_config add auth foo1 -remote 10.1.1.1 -psk my_preshared_key If the HP-UX IPSec subsystem is not already started, the administrator starts it using the ipsec_admin -start command. Additional Options For information on additional options and commands, see the HP-UX IPSec Administrator's Guide and the following manpages: ipsec_admin(1M) ipsec_config(1M) ipsec_policy(1M) ipsec_report(1M) . 32
  • Configuring a Windows End-to-End Tunnel Policy The only IPsec tunnel topology supported between an HP-UX system and a Windows system is an end-to-end tunnel.5The procedure for configuring an end-to-end tunnel policy on Windows system is the same as procedure for configuring a host policy, except that you must configure two, non-mirrored rules: one rule for outbound packets and one rule for inbound packets, as described in the sections that follow. NOTE: Do not configure any other rules in the policy with the HP-UX system address as the destination address. This prevents the Microsoft system from applying the tunnel transform over a host-to-host (transport) transform. In end-to-end tunnel topologies, HP-UX IPSec does not support transport transforms over a tunnel transform. Outbound Tunnel Rule Requirements The outbound tunnel rule must have the following parameters: • Filter List: One filter, with the following parameters: — Address: ◦ Source address: the HP-UX system's address. ◦ Destination address: this must be a specific IP address and must be the Windows system's address. ◦ Mirrored: no (the Mirrored box is cleared). — Protocol Type: none (wildcard). The Windows documentation states that the filters in tunnel rules must not specify protocols or ports to ensure that IP Security can correctly process IP fragments. • Tunnel Setting — Tunnel endpoint: the HP-UX system's address. This is the address of the tunnel endpoint closest to the destination. Since this is an end-to-end tunnel, it is the same as the destination address in the address filter. Inbound Tunnel Rule Requirements The inbound tunnel rule must have the following parameters: • Filter List: One filter, with the following parameters: — Address: ◦ Source address: the Windows system's address. ◦ Destination address: this must be a specific IP address and must be the HP-UX system's address. ◦ Mirrored: no (the Mirrored box is cleared). — Protocol Type: none (wildcard). • Tunnel Setting — Tunnel endpoint: the Windows system's address. This is the address of the tunnel endpoint closest to the destination. Since this is an end-to-end tunnel, it is the same as the destination address in the address filter Configuring a Tunnel Rule Use the following procedure to configure an outbound or inbound tunnel rule. 5. You can also configure an IPsec topology where packets exchanged between an HP-UX system and a Windows system are tunneled through an IPsec gateway device, but neither HP-UX nor Windows systems can be configured as IPsec gateways. The only topology in which an HP-UX system can act as an IPsec gateway is when the HP-UX system is a Home Agent for Mobile IPv6 clients. The HP-UX IPSec Administrator's Guide describes how to configure a host-to-gateway IPsec topology using HP-UX and a Cisco router. Configuring a Windows End-to-End Tunnel Policy 33
  • TIP: The tunnel setting is used by all packets selected using the address filters for the rule. Do not include any filters for host-to-host (non-tunneled) packets in the filter list for a rule with a tunnel. 1. Start the IP Security Policies snap-in if necessary. 2. Create an IP Security policy or modify an existing policy. To modify an existing policy, select the policy in the right navigation pane and right click the policy. Select Properties. 3. The IP Security configuration utility opens the Policy Properties dialog box. Select the Rules tab. Click Add to create a new rule or select a rule you want to modify and click Edit. 4. Configure a new rule or modify an existing rule with the appropriate address filter for the outbound tunnel rule or inbound tunnel rule, as described in “Outbound Rule” (page 34) or “Inbound Rule” (page 35). See “Step 4: Creating the IP Filter List and Filters for the Rule” (page 18) if you need additional information about configuring address filters. Record the destination address; you will need it to configure the tunnel endpoint. 5. Return to the Rule Properties dialog box. Select the Tunnel Setting tab. 6. The IP Security configuration utility opens the Tunnel Setting dialog box. Select The tunnel endpoint is specified by this IP address. Enter the IP address of the tunnel endpoint closest to the destination. Since this is an end-to-end tunnel, it is the same as the destination address in the address filter. 7. Click Close to close the Tunnel Setting dialog box. 8. If this is a new rule, complete the configuration by configuring the appropriate filter action, authentication methods, and connection type. Click Close to close the Rule Properties dialog box. Example In this example, IPsec secures all packets between the Windows system and the HP-UX system using authenticated ESP. The Windows system's address is 10.1.1.1 The HP-UX system's address is 10.2.2.2. Windows Configuration On the Windows system, you configure one rule for outbound packets and one for inbound packets. Outbound Rule The outbound rule is for packets from the Windows system (source address 10.1.1.1) to the HP-UX system (destination address 10.2.2.2). Figure 18 shows the address filter for this rule, and Figure 19 shows the corresponding tunnel settings: 34
  • Figure 18 Outbound Rule Filter Figure 19 Outbound Rule Tunnel Settings Inbound Rule The inbound rule is for packets to the Windows system (destination address 10.1.1.1) from the HP-UX system (source address 10.2.2.2). Figure 18 shows the address filter for this rule, and Figure 21 shows the corresponding tunnel settings: Configuring a Windows End-to-End Tunnel Policy 35
  • Figure 20 Inbound Rule Filter Figure 21 Inbound Rule Tunnel Settings Additional Parameters You must configure the remaining rule parameters (filter action, authentication methods, and connection type) to be compatible with the HP-UX configuration. In addition, the general parameters for the rule (the IKE SA parameters) must be compatible with the HP-UX configuration. 36
  • HP-UX Configuration On the HP-UX system, the host and tunnel policies are bi-directional (mirrored), so you configure only one host policy and only one tunnel policy. Since this is an end-to-end tunnel, the tunnel policy does not have to specify the tunnel endpoints. HP-UX IPSec will use the end source and end destination addresses as the tunnel addresses (the tsource and tdestination values default to the source and destination values). ipsec_config add host foo1 -source 10.2.2.2 -destination 10.1.1.1 -action PASS -tunnel foo1_tunnel ipsec_config add tunnel foo1_tunnel -source 10.2.2.2 -destination 10.1.1.1 -action ESP_3DES_HMAC_SHA1 You must also configure an IKE policy and an authentication record to complete the configuration: ipsec_config add ike foo1 -remote 10.1.1.1 -auth PSK ipsec_config add auth foo1 -remote 10.1.1.1 -psk my_preshared_key Configuring a Windows End-to-End Tunnel Policy 37
  • Troubleshooting Tips Most interoperability problems occur during IKE negotiations, so examining IKE log events is useful. You can use the following procedures to enable and view IKE log events: Using IKE Logging on HP-UX Systems Use the following procedure to view detailed IKE log events on HP-UX systems: 1. Enter the following command to set the HP-UX IPSec log level to debug and increase the maximum log file size: ipsec_admin -al debug -maxsize 99999 IPSec creates the log files in the /var/adm/ipsec directory. The log file names are auditdateinfo.log 2. Reproduce the problem. 3. Enter the following command to format the audit file: ipsec_report -audit /var/adm/ipsec/auditdateinfo.log 4. Use the following command to set the HP-UX IPSec log level back to warning (the default log level): ipsec_admin -al warning Using IKE Logging on Windows Systems Use the following procedure to view IKE log events on Windows systems: 1. Enable IKE logging. On Windows XP systems, use the regedit utility to enable IKE logging in the system registry. On Windows systems, IKE logging is configured using the Oakley6 key. CAUTION: Incorrectly editing the registry may severely damage the system. Before making changes to the registry, HP recommends that you back up the registry and any valued data on the computer. Refer to the article How to back up, edit, and restore the registry in Windows XP and Windows Server 2003 in the Windows Knowledge Base for more information. The Windows Knowledge Base is available at http://support.microsoft.com Set the HKEY_LOCAL_MACHINESystemCurrentControlSetServicesPolicyAgentOakleyEnableLogging REG_DWORD value to 1. On some Windows versions, you may need to create the Oakley key. On Windows 2003 systems, enter the following command to enable IKE logging: netsh ipsec dynamic set config ikelogging 1 2. Stop and restart the IP Security service. You can use the following commands at the Windows command prompt: net stop policyagent net start policyagent Refer to “Step 9: Starting the IP Security Service” (page 29) for more information. 3. Reproduce the problem. 4. View the IKE log file. Windows creates the log file in the directory systemrootDebug (by default, this is the WINDOWSDebug directory). The file name is Oakley.log. 6. The Oakley protocol is a key-agreement protocol that is incorporated in the IKE protocol. 38
  • 5. Disable IKE logging. On Windows XP systems, set the HKEY_LOCAL_MACHINESystemCurrentControlSetServicesPolicyAgentOakleyEnableLogging REG_DWORD value to 0. On Windows 2003 systems, enter the following command: netsh ipsec dynamic set config ikelogging 0 6. Stop and restart the IP Security service. Additional Windows Troubleshooting Tools Windows supports an IP Secu;rity Monitor snap-in utility for the Microsoft Management Console (MMC) that provides IPsec statistics and information about IKE and IPsec Security Associations. Refer to the Windows documentation set for more information. Troubleshooting Tips 39
  • Comparing HP-UX and Windows IPsec Configuration Parameters This section contains Table 1, which compares how HP-UX and Windows systems configure and store IPsec parameters. It also contains the following subsections, which provide additional comparative information: • “Mirrored Filters” (page 41) • “Filter Selection” (page 42) • “IKE Parameter Selection” (page 42) • “IKE SA Key (Master Key) Lifetime Values” (page 42) • “Maximum Quick Modes” (page 43) • “Perfect Forward Secrecy (PFS)” (page 43) • “IPsec SA Key (Session Key) Lifetime Values” (page 43) Table 1 IPsec Parameters on Windows and HP-UX Parameter Windows Configuration HP-UX Configuration Notes Address Filters Specify them in the Filter Specify one filter per host, Windows and HP-UX List for a rule. The Filter List tunnel, or gateway policy. support subnet masks for IP can contain multiple Use the -source and addresses and wildcards for address filters. -destination arguments IP addresses, protocols, and in the ipsec_config add port numbers. host , tunnel, or See “Mirrored Filters” gateway command. (page 41) for additional information. IPsec SA Proposals Specify them in the Filter Specify them using the HP-UX IPSec supports ESP Action for a rule. -action argument in the encryption using the ipsec_config add following protocols: gateway, host, or tunnel Advanced Encryption command. Standard (AES), Triple Data Encryption Standard (3DES), and Data Encryption Standard (DES). Windows XP and Windows 2000 support 3DES and DES, but do not support AES. Filter Priority Not applicable. Specify it using the See “Filter Selection” -priority argument in (page 42) for additional the ipsec_config add information. gateway or host command. Maximum IPsec SA Specify it in the Custom Specify it in the transform See “IPsec SA Key (Session Lifetime, measured by time Security Methods dialog specification for the Key) Lifetime Values” or by data box under the Filter Action -action argument in the (page 43) for additional for a rule. ipsec_config add host information. or tunnel command. Tunnel endpoint address Specify the destination Specify the endpoints using See “Mirrored Filters” tunnel endpoint (the the -tsource and (page 41) for additional endpoint for the -tdestination information. destination) in the Tunnel arguments of Settings for a rule. You must theipsec_config add configure two tunnel command. uni-directional (non-mirrored) rules. IKE Authentication Method Specify it in the Specify it using the -auth Authentication Methods for argument of the a rule. ipsec_config add ike command. 40
  • Table 1 IPsec Parameters on Windows and HP-UX (continued) Parameter Windows Configuration HP-UX Configuration Notes IKE Preshared Key Specify it in the Specify it using the Authentication Methods for -preshared argument of a rule. the ipsec_config add auth command. IKE Exchange Type Windows supports only Specify it using the Main Mode exchanges. -exchange argument of the ipsec_config add auth command. The default value is MM (Main Mode). Maximum IKE SA Lifetime, Specify it in the Key Specify it using the-life The Windows IP Security measured by time Exchange Settings dialog argument in the Policy snap-in utility uses box. (To navigate to the Key ipsec_config add ike minutes as the time unit. Exchange Setting dialog command. The HP-UX ipsec_config box, select the General tab command uses seconds as in the Policy Properties the time unit. See “IKE SA dialog box, then select Key (Master Key) Lifetime Advanced settings.) Values” (page 42) for additional information. Maximum Quick Mode Specify it in the Key Specify it using the-maxqm See “Maximum Quick (QM) negotiations per IKE Exchange Settings dialog argument in the Modes” (page 43) for SA box. (To navigate to the Key ipsec_config add ike additional information. Exchange Setting dialog command. box, select the General tab in the Policy Properties dialog box, then select Advanced settings.) Perfect Forward Secrecy Windows supports PFS for HP-UX does not support See “Perfect Forward (PFS) keys only (PFS for session PFS for session keys. HP-UX Secrecy (PFS)” (page 43) for keys) and supports PFS for supports only PFS for more information. keys in conjunction with master keys. PFS for all identities (PFS Specify PFS for master keys for master keys). using the-maxqm 1 Specify PFS for master keys argument in the in the Key Exchange ipsec_config add ike Settings dialog box. (To command. navigate to the Key Exchange Setting dialog box, select the General tab in the Policy Properties dialog box, then select Advanced settings.) IKE SA Proposals Specify it in the General You can specify the See “IKE Parameter parameters for a policy. You parameters for one IKE SA Selection” (page 42) for can configure multiple IKE proposal in an IKE policy, additional information. SA proposals and their using the -encryption, preference order. -hash, and -group arguments in an ipsec_config add ike command. Mirrored Filters Microsoft filters can be mirrored (bi-directional) or not mirrored (uni-directional). If the filter is mirrored, the filter will match IP packets with the source and destination addresses and ports reversed. For example, a filter has the following specifications: Source address: 10.1.1.1 Destination address: 10.2.2.2 Comparing HP-UX and Windows IPsec Configuration Parameters 41
  • The filter matches packets with the following addresses: Source address: 10.1.1.1 Destination address: 10.2.2.2 If the filter is mirrored, it also matches packets with the following addresses: Source address: 10.2.2.2 Destination address: 10.1.1.1 The mirror setting only affects Windows IP Security behavior before IPsec SAs are established. If the Windows IP Security module receives a packet via an existing SA, it does not verify that the packet address fields match the address filter used when the SA was established. By comparison, HP-UX IPSec host and tunnel policies are always mirrored. (Gateway policies are the only HP-UX IPSec policies that are not mirrored.) Filter Selection Windows does not allow you to specify the search or priority order for the filters in a rule or for the order of rules in a policy. The Windows IP Security module automatically creates an internal filter list and orders the filters from most specific to least specific. HP-UX IPSec allows you to specify a priority value for IPsec and IKE policies. HP-UX IPSec searches the policies in priority order within each type of policy. Lower priority values have higher priority (priority value 1 is the highest priority). If you do not specify a priority value when creating a policy on HP-UX, ipsec_config automatically assigns a priority value so that the new policy is the last policy searched before the default policy within its policy type. The output of the ipsec_config show command includes the priority values for configured policies. IKE Parameter Selection On HP-UX systems, only one IKE SA proposal is used for each peer. You can configure multiple IKE policies, but only one IKE policy is selected per peer, and each IKE policy specifies only one IKE SA. During IKE negotiations, IKE searches policies in priority order and selects the first policy with a matching remote address. IKE then uses the IKE SA parameters to send an IKE SA proposal, or to evaluate the IKE SA proposal(s) it receives. On Windows systems, you can configure a set of multiple IKE SA proposals, but only one set per IP Security policy, and only one IP Security policy can be in use (assigned) on the system. IKE SA Key (Master Key) Lifetime Values IKE SA key lifetimes (referred to as Master key lifetimes on Windows systems) specify the maximum lifetimes for IKE SA keys and are specified by units of time (seconds). In addition, users can specify the maximum number of IPsec SA negotiations that can be completed per IKE SA (“Maximum Quick Modes” (page 43)). HP-UX IKE SA Lifetime Values The HP-UX IPSec default preferred lifetime value for IKE SAs is 28,800 seconds (eight hours). If the HP-UX system initiates IKE SA negotiations, the HP-UX IKE daemon proposes the preferred lifetime value to the remote system. The remote system may process this value in any manner according to the IPsec protocol suite. If the remote system initiates IKE SA negotiations and sends a proposed value that is longer than (less secure than) the HP-UX preferred value, HP-UX sends an IKE NOTIFY message with its preferred value, and this value is used for the SA. If the remote system initiates IKE SA negotiations and sends a proposed lifetime that is the same or more secure (shorter than) the HP-UX preferred value, the HP-UX IKE daemon accepts the 42
  • proposed value sent by the remote system if it is within the range specified by the IPsec protocol suite. Windows IKE SA Lifetime Values By default, Windows XP systems use the following values for preferred IKE key lifetime values: 480 minutes (eight hours) 0 (infinite) IPsec SA negotiations (sessions) In testing with HP-UX IPSec, HP configured a shorter IKE SA lifetime value on the Windows system. When the Windows system was the initiator, it sent the configured lifetime value to the remote system. When the Windows system was the responder, it accepted the value sent by the HP-UX system but did not send a notification message. Maximum Quick Modes HP-UX and Windows enable you to specify the maximum number of IPsec or Quick Mode (QM) negotiations that IKE can complete per IKE SA. Each IPsec SA negotiation establishes two IPsec SAs (one in each direction). The default maximum QM values are as follows: HP-UX: 100 Windows: 0 (infinite) If the value for maximum QM is 1, Perfect Forward Secrecy (PFS) for both keys and identities is implemented. See “Perfect Forward Secrecy (PFS)” (page 43) for more information. Perfect Forward Secrecy (PFS) With Perfect Forward Secrecy, the exposure of one key permits access only to data protected by that key. RFC 2409, The Internet Key Exchange (IKE), defines two forms of PFS: • PFS for both the keys and the IKE identities. PFS is provided for keys in conjuction with PFS for identities. IKE deletes the IKE SA after the IPsec negotiation completes. Each IKE SA is used for only one IPsec negotiation. The Windows interface refers to this type of PFS as master key PFS. • PFS for IPsec keys only. The IKE peers perform a key exchange (Diffie-Hellman exchange) to create new keying material for each IPsec negotiation. The IKE SA is re-used until the IKE SA lifetime expires. The Windows interface refers to this type of PFS as session key PFS. HP-UX IPSec supports PFS for both the keys and the IKE identities but does not support PFS for IPsec keys only. To be compatible with HP-UX IPSec, do not configure session key PFS on Windows systems. Configuring PFS is computationally expensive. In most topologies, the strength of the cryptographic algorithms is sufficient protection. HP recommends that you enable PFS only in hostile environments. IPsec SA Key (Session Key) Lifetime Values IPsec SA key lifetimes (referred to as session key lifetimes on Windows systems) specify the maximum lifetimes for IPsec SA keys and are specified by units of time (seconds) and by data units transferred (kbytes). HP-UX IPsec SA Lifetime Values By default, HP-UX uses the following values for preferred lifetime values: 28,800 seconds (eight hours) 0 (infinite) data units Comparing HP-UX and Windows IPsec Configuration Parameters 43
  • If the HP-UX system initiates IPsec SA negotiations, the HP-UX IKE daemon proposes the preferred lifetime values to the remote system. The remote system may process these values in any manner according to the IPsec protocol suite. If the remote system initiates IPsec SA negotiations and sends proposed lifetime value that is as secure or more secure than the HP-UX preferred value (it is shorter than or equal to the HP-UX preferred value), the HP-UX IKE daemon accepts the lifetime value proposed by the remote system if it is within the ranges specified by the IPsec protocol suite. If the remote system initiates IPsec SA negotiations and a proposed lifetime value is less secure (shorter than) the HP-UX preferred value, HP-UX sends an IKE NOTIFY message with its preferred value. If this value is acceptable to the remote system, the SA negotiation succeeds and the value sent in the NOTIFY message is used. Windows IPsec SA Lifetime Values By default, the Windows configuration does not specify any IPsec SA lifetime values and does not propose any during IPsec SA negotiations. This is equivalent to proposing the lifetime values 28,800 seconds (eight hours) and 0 (infinite) data units. In testing with HP-UX, HP also configured specific IPsec SA lifetime values on the Windows system and observed behavior equivalent to HP-UX behavior. When the Windows system initiated the IPsec SA negotiation, it sent the configured lifetime values in the proposal. When the remote system initiated the IPsec SA negotiation, the Windows system accepted the proposed lifetime value if it was more secure than its configured value, and sent a notification message when its configured lifetime value was more secure than the value proposed by the remote system. 44
  • Related Publications The following documents are available at http://docs.hp.com: • HP-UX IPSec Administrator's Guide • Using Microsoft Windows Certificates with HP-UX IPSec • HP-UX IPSec manpages The following documents are available at http://microsoft.com: • Step-by-Step Guide to Internet Protocol Security (IPSec) • IPSec troubleshooting tools Related Publications 45
  • 46
  • glossary 3DES Triple Data Encryption Standard. A symmetric key block encryption algorithm that encrypts data three times, using a different 56-bit key each time (168 bits are used for keys). 3DES is suitable for bulk data encryption. AES Advanced Encryption Standard. Uses a symmetric key block encryption. HP-UX IPSec supports AES with a 128-bit key. AES is suitable for encrypting large amounts of data. AH The AH (Authentication Header) protocol provides data integrity, system-level authentication for IP packets. It can also provide anti-replay protection. The AH protocol is part of the IPsec protocol suite. authentication The process of verifying a user's identity or integrity of data, or the identity of the party that sent data. DES Data Encryption Standard. Uses a 56-bit key for symmetric key block encryption. It is suitable for encrypting large amounts of data. DES has been cracked (data encoded using DES has been decoded by a third party). Diffie-Hellman Method to generate a symmetric key where two parties can publicly exchange values and generate the same shared key. Start with prime p and generator g, which may be publicly known (typically these numbers are from a well-known Diffie-Hellman Group). Each party selects a private value (a and b) and generates a public value (g**a mod p) and (g**b mod p). They exchange the public values. Each party then uses its private value and the other party's public value to generate the same shared key, (g**a)**b mod p and (g**b)**a mod p, which both evaluate to g**(a*b) mod p for future communication. The Diffie-Hellman method must be combined with authentication to prevent man-in-the-middle or third party attacks (spoofing) attacks. For example, Diffie-Hellman can be used with certificate or preshared key authentication. ESP The ESP (Encapsulating Security Payload) protocol provides confidentiality (encryption), data authentication, and an anti-replay service for IP packets. When used in tunnel mode, ESP also provides limited traffic flow confidentiality. The ESP protocol is part of the IPsec protocol suite. IKE The Internet Key Exchange (IKE) protocol is used before the ESP or AH protocol exchanges to determine which encryption and/or authentication services will be used. IKE also manages the distribution and update of the symmetric (shared) encryption keys used by ESP and AH. IKE The method used by IKE peers to authenticate each party's identity. HP-UX IPSec supports two authentication IKE authentication methods: preshared keys and RSA signatures using certificates. IKE SA IKE Security Association. An IKE SA is a bi-directional, secure communication channel that IKE uses to negotiate IPsec SAs. IKE can establish IKE SAs using either Main Mode or Aggressive Mode negotiations. Also referred to as IKE Phase One SA, ISAKMP SA, ISAKMP/MM SA, Aggressive Mode SA, Main Mode SA. IPsec SA IPsec Security Association. An IPsec SA is a uni-directional, secure communication channel. The IPsec SA operating parameters include the IPsec protocol used (ESP or AH), the mode (transport or tunnel), the cryptographic algorithms (such as AES and SHA-1), the cryptographic keys, the SA lifetime, and the endpoints (IP addresses, protocol and port numbers). IKE establishes IPsec SAs using Quick Mode negotiations. Also referred to as IKE Phase Two SA, IPsec SA, Quick Mode SA. Perfect Forward With Perfect Forward Secrecy the exposure of one key permits access only to data protected Secrecy (PFS) by that key. HP-UX IPSec supports PFS for keys and all identities (the IKE daemon can be configured to create a new IKE SA for each IPsec negotiation). HP-UX IPSec does not support PFS for keys only (the IKE SA is re-used for multiple IPsec negotiations, with a new Diffie-Hellman key exchange for each IPsec negotiation). SA See Security Association. A secure communication channel and its parameters, such as encryption and authentication method, keys and lifetime.. SHA1 (Secure Hash Algorithm-1). Authentication algorithm that generates a 160-bit message digest using a 160-bit key. 47
  • transform A transform defines the IPsec action(s) to be taken on the IP data, such as passing the data in clear text, discarding the data, authenticating and encrypting the data using ESP, or authenticating the data using AH. 48 glossary