Configuring Microsoft Windows IP Security to Operate with HP ...

HP-UX IPSec
Configuring Microsoft Windows IP Security
to Operate with HP-UX IPSec

HP Part Number: J4256-90025
Published: June 2007
Edition: 1.0

Table of Contents
About This Document.........................................................................................................9
Typographic Conventions......................................................................................................................9
Introduction..........................................................................................................................................11
Testing Environment.......................................................................................................................11
Known Problem with Windows 2000 SP1 and SP2...................................................................11
Protocol Implementation Differences..............................................................................................12
Windows IP Security Configuration Overview....................................................................................13
Configuring a Windows Host-to-Host Policy.......................................................................................14
Step 1: Starting the IP Security Policies Snap-in Configuration Utility...........................................15
Step 2: Creating a Policy..................................................................................................................15
Step 3: Adding a Rule......................................................................................................................16
Step 4: Creating the IP Filter List and Filters for the Rule...............................................................18
Step 5: Configuring Filter Actions for the Rule...............................................................................21
Step 6: Configuring the IKE Authentication Method and Preshared Key for the Rule..................25
Step 7: Configuring the Connection Type for the Rule...................................................................26
Step 8: Modifying IKE Parameters for the Policy............................................................................26
Step 9: Starting the IP Security Service............................................................................................29
Step 10: Assigning the IP Security Policy........................................................................................30
Step 11: Verifying the Configuration...............................................................................................31
Example...........................................................................................................................................31
Windows Configuration.............................................................................................................31
HP-UX Configuration................................................................................................................32
Additional Options...............................................................................................................32
Configuring a Windows End-to-End Tunnel Policy.............................................................................33
Outbound Tunnel Rule Requirements............................................................................................33
Inbound Tunnel Rule Requirements...............................................................................................33
Configuring a Tunnel Rule..............................................................................................................33
Example...........................................................................................................................................34
Windows Configuration.............................................................................................................34
Outbound Rule.....................................................................................................................34
Inbound Rule........................................................................................................................35
Additional Parameters..........................................................................................................36
HP-UX Configuration................................................................................................................37
Troubleshooting Tips............................................................................................................................38
Using IKE Logging on HP-UX Systems..........................................................................................38
Using IKE Logging on Windows Systems.......................................................................................38
Additional Windows Troubleshooting Tools..................................................................................39
Comparing HP-UX and Windows IPsec Configuration Parameters....................................................40
Mirrored Filters...............................................................................................................................41
Filter Selection.................................................................................................................................42
IKE Parameter Selection..................................................................................................................42
IKE SA Key (Master Key) Lifetime Values......................................................................................42
HP-UX IKE SA Lifetime Values.................................................................................................42
Windows IKE SA Lifetime Values..............................................................................................43
Maximum Quick Modes..................................................................................................................43
Perfect Forward Secrecy (PFS).........................................................................................................43
IPsec SA Key (Session Key) Lifetime Values...................................................................................43
HP-UX IPsec SA Lifetime Values...............................................................................................43
Windows IPsec SA Lifetime Values...........................................................................................44
Related Publications..............................................................................................................................45

Table of Contents 3

glossary.............................................................................................................................47

4 Table of Contents

List of Figures
1 IP Security Policy Wizard..............................................................................................................16
2 Rules Tab.......................................................................................................................................17
3 Rule Properties Dialog Box...........................................................................................................17
4 Creating an IP Filter List...............................................................................................................18
5 Address Tab for Filter Properties..................................................................................................19
6 Protocol Tab for Filter Properties..................................................................................................20
7 Selecting the Filter List for a Rule.................................................................................................21
8 Security Methods for Filter Action................................................................................................22
9 Security Method Dialog Box.........................................................................................................23
10 Custom Security Methods Settings Dialog Box............................................................................24
11 Selecting the Filter Action.............................................................................................................25
12 Configuring A Preshared Key.......................................................................................................26
13 General Policy Properties Dialog Box ..........................................................................................27
14 Key Exchange Settings Dialog Box ...............................................................................................28
15 IKE Security Algorithms Dialog Box ............................................................................................29
16 IPSEC Services Properties Dialog Box...........................................................................................30
17 Assigning the IP Security Policy...................................................................................................31
18 Outbound Rule Filter....................................................................................................................35
19 Outbound Rule Tunnel Settings....................................................................................................35
20 Inbound Rule Filter.......................................................................................................................36
21 Inbound Rule Tunnel Settings.......................................................................................................36

5

List of Tables
1 IPsec Parameters on Windows and HP-UX .................................................................................40

7

About This Document
This document describes how to configure Microsoft Windows IP Security to operate with the
HP-UX IPSec product.

Typographic Conventions
This document uses the following typographical conventions:
%, $, or # A percent sign represents the C shell system prompt. A dollar
sign represents the system prompt for the Bourne, Korn, and
POSIX shells. A number sign represents the superuser prompt.
audit(5) A manpage. The manpage name is audit, and it is located in
Section 5.
Command A command name or qualified command phrase.
Computer output Text displayed by the computer.
Ctrl+x A key sequence. A sequence such as Ctrl+x indicates that you
must hold down the key labeled Ctrl while you press another
key or mouse button.
ENVIRONMENT VARIABLE The name of an environment variable, for example, PATH.
[ERROR NAME] The name of an error, usually returned in the errno variable.
Key The name of a keyboard key. Return and Enter both refer to the
same key.
Term The defined use of an important word or phrase.
User input Commands and other text that you type.
Variable The name of a placeholder in a command, function, or other
syntax display that you replace with an actual value.
[] The contents are optional in syntax. If the contents are a list
separated by |, you must choose one of the items.
{} The contents are required in syntax. If the contents are a list
separated by |, you must choose one of the items.
... The preceding element can be repeated an arbitrary number of
times.
Indicates the continuation of a code example.
| Separates items in a list of choices.
WARNING A warning calls attention to important information that if not
understood or followed will result in personal injury or
nonrecoverable system problems.
CAUTION A caution calls attention to important information that if not
understood or followed will result in data loss, data corruption,
or damage to hardware or software.
IMPORTANT This alert provides essential information to explain a concept or
to complete a task
NOTE A note contains additional information to emphasize or
supplement important points of the main text.

Typographic Conventions 9

Introduction
This document contains the following sections:
• “Windows IP Security Configuration Overview” (page 13)
This section contains a brief overview of the Windows IPsec configuration parameters and
the terminology used in the Windows IPsec configuration utilities.
• “Configuring a Windows Host-to-Host Policy” (page 14)
This section describes how to configure IP Security (IPsec) on a Windows client to secure
IP packets sent to and received from an HP-UX system in a host-to-host topology.
• “Configuring a Windows End-to-End Tunnel Policy” (page 33)
This section describes how to configure IPsec on a Windows client to secure IP packets sent
to and received from an HP-UX system in an end-to-end tunnel topology.
• “Troubleshooting Tips” (page 38)
This section contains troubleshooting tips.
• “Comparing HP-UX and Windows IPsec Configuration Parameters” (page 40)
This section compares how HP-UX and Windows systems configure and use IPsec parameters.
• “Related Publications” (page 45)
This section contains a list of related HP-UX and Microsoft publications.
The procedures and examples in this document use preshared keys for IKE authentication. For
information about using certificates for IKE authentication with Microsoft Windows, see Using
Microsoft Windows Certificates with HP-UX IPSec, available at http://docs.hp.com.
The intended audience for this document is an HP-UX IPSec administrator who is familiar with
the HP-UX IPSec product and with the IP Security protocol suite. If you are not familiar with
the HP-UX IPSec product, see the appropriate version of the HP-UX IPSec Administrator's Guide,
available at http://docs.hp.com.

NOTE: The IP Security protocol suite is often referred to as IPsec. The HP-UX product that
implements the IP Security protocol suite is HP-UX IPSec.

Testing Environment
The procedures in this white paper were tested using the following environment:

Component Description

HP-UX IPSec Versions A.02.01 and A.02.01.01

Microsoft Windows Client Windows XP with Service Pack 2 (SP2)

Known Problem with Windows 2000 SP1 and SP2
For this white paper, HP did not test with Windows 2000 systems. However, there is a known
problem with Windows 2000 base systems and Windows 2000 systems with Service Pack 1 (SP1)
or Service Pack 2 (SP2). The IP Security module on these systems does not properly process IPSec
ESP packets that are fragmented across IP packets and drops these packets. The symptoms vary
according to how the applications handle the dropped packets.
This problem is caused by a defect in the Windows 2000 SP1/ SP2 software and is fixed in Windows
2000 Service Pack 3 (SP3).

Introduction 11

The above problem typically occurs with ESP-encrypted UDP or ICMP packets that are fragmented
by IP. HP-UX 11i systems minimize IP fragmentation of ESP-encrypted TCP packets. You may
still experience problems with ESP-encrypted TCP packets sent from an HP-UX system to a
Windows 2000 system if an intermediate IP gateway fragments the ESP packet.

Protocol Implementation Differences
HP-UX and Microsoft Windows both implement the IP Security protocol suite. However, there
are features in the protocol suite that HP-UX implemented which Microsoft did not implement,
and vice-versa.
The following features are implemented by HP-UX IPSec version A.02.01 but not by Microsoft
Windows XP:
• Advanced Encryption Standard (AES): HP-UX IPSec supports ESP encryption using the
following protocols: AES, Triple Data Encryption Standard (3DES), and Data Encryption
Standard (DES). Windows XP and Windows 2000 support 3DES and DES, but do not support
AES.
• Aggressive Mode (AM): HP-UX supports AM exchanges to establish IKE Security
Associations (SAs). AM is an optional feature and is not supported on Windows.
The following features are implemented by Microsoft Windows XP, but not by HP-UX IPSec
version A.02.01:
• Kerberos: Windows supports Internet Key Exchange (IKE) authentication using Kerberos.
RFC 2408 defines an optional Kerberos Token payload, but does not describe how to
implement it. This feature is not supported on HP-UX.
• Perfect Forward Secrecy (PFS) for keys only: HP-UX IPSec supports PFS for keys in
conjunction with PFS for all identities, but does not support PFS for keys only. Windows
supports PFS for keys only (“session key PFS”) and PFS for keys in conjuctions with PFS for
all identities (“master key PFS”). See “Perfect Forward Secrecy (PFS)” (page 43) for more
information.

12

Windows IP Security Configuration Overview
On Microsoft Windows systems, all IP Security (IPsec) configuration data resides in a single IP
Security policy. You can create multiple IP Security policies, but only one local policy can be
active on the system. If the system is a member of a Windows Active Directory domain, you can
use an IP Security policy from a Group Policy defined for the domain.
A Windows IP Security policy defines the parameters used to negotiate Internet Key Exchange
Security Associations (IKE SAs) and IPsec SAs. An IKE SA is a bi-directional, secure
communication channel that two peers establish before negotiating IPSec SAs. One of the primary
activities during the IKE SA negotiation is the authentication of each peer's identity.
After two peers establish an IKE SA, they can negotiate IPsec SAs. Each IPsec SA is a
uni-directional, secure communication channel. The IPsec SA operating parameters include the
IPsec protocol used (Encapsulating Security Payload, ESP, or Authentication Header, AH) and
the cryptographic algorithms. IPsec SAs are negotiated in pairs (one for each direction of traffic).
Each Windows IP Security policy contains the following components:
• Rules
A policy contains one or more rules. The main purpose of a rule is to assign actions for
address filters. Each rule contains the following components:
— IP Filter List
An IP Filter list contains one or more filters. Each filter contains the following
components:
◦ Addressing
The source and destination IP addresses, network masks, and a flag that indicates
if the filter is mirrored (bi-directional).
◦ Protocol
The upper-layer protocol, and source and destination ports, if applicable.
◦ Description
The filter name and a description.

— Filter Action
The filter action specifies the action to take for the rule, and can be one of the following
actions:
◦ allow: allow the packet to pass
◦ block: discard the packet
◦ negotiate security: negotiate IPsec Authentication Header (AH) or Encapsulating
Security Payload (ESP) Security Associations (SAs)
— Authentication Methods
The authentication methods specify the type of Internet Key Exchange (IKE)
authentication to use (preshared key or certificates with RSA signatures). If you are
using preshared key authentication, the authentication methods also specify the value
of the preshared key.

Windows IP Security Configuration Overview 13

— Tunnel Settings
The tunnel settings specify if the rule is a tunnel rule. If it is a tunnel rule, the settings
also specify the tunnel destination endpoint.
— Connection Type
The connection type specifies the connection (link) types for the rule, such as LAN.

• General
The general parameters for a policy specify IKE SA parameters, such as the IKE encryption
algorithm, IKE hash (integrity algorithm), Diffie-Hellman Group, and IKE SA key lifetimes.
The parameters correspond to IKE SA proposals. You can configure multiple IKE SA
proposals and specify the preference order. The proposals are used for all rules in the policy.
By comparison, a minimal HP-UX IPSec configuration consists of one or more IPsec host policies,
one or more IKE policies, and one or more authentication records. The IPsec host policies specify
address filters, and you can configure separate IKE policies for each peer. “Comparing HP-UX
and Windows IPsec Configuration Parameters” (page 40) lists IPsec configuration parameters
and how they are configured in the HP-UX IPSec and the Windows IP Security configuration
utilities.

Configuring a Windows Host-to-Host Policy
This section describes one method for configuring host-to-host policy on a Windows XP client
using the IP Security Policies snap-in utility. Windows also supports command-line utilities to
configure IP Security policies: ipseccmd on Windows XP systems and netsh on Windows 2003
systems. For more information about these utilities, see the Windows documentation set.
To use this method, complete the following steps:
1. Start the IP Security Policies snap-in utility. See “Step 1: Starting the IP Security Policies
Snap-in Configuration Utility” (page 15).
2. Create an IP Security policy. See “Step 2: Creating a Policy” (page 15).
3. Add a rule to the policy. See “Step 3: Adding a Rule” (page 16).
4. Create a Filter List for the rule and configure filters. See “Step 4: Creating the IP Filter List
and Filters for the Rule” (page 18).
5. Configure filter actions for the rule. The filter actions contain IPsec transforms or other
actions. See “Step 5: Configuring Filter Actions for the Rule” (page 21).
6. Configure the IKE authentication method and preshared key for the rule. See “Step 6:
Configuring the IKE Authentication Method and Preshared Key for the Rule” (page 25).
7. Specify the network link (connection) types for the rule. See“Step 7: Configuring the
Connection Type for the Rule” (page 26).
8. Modify the IKE SA parameters for the policy. By default, Windows clients will use IKE SA
parameters that are compatible with the default HP-UX IPSec parameters. If these parameters
are acceptable, you can skip this step. See “Step 8: Modifying IKE Parameters for the Policy”
(page 26).
9. Start the IP Security service. The IP Security service must be running before you can assign
the new IP Security policy. See “Step 9: Starting the IP Security Service” (page 29).
10. Assign (activate) the new IP Security Policy. See “Step 10: Assigning the IP Security Policy”
(page 30).
11. Verify the configuration. See “Step 11: Verifying the Configuration” (page 31).
Because this is a host-to-host rule, we will use the default value for the rule tunnel setting (no
tunnel). For information about configuring a tunnel rule and the tunnel setting, see “Configuring
a Windows End-to-End Tunnel Policy” (page 33).

14

Step 1: Starting the IP Security Policies Snap-in Configuration Utility
Use the following procedure to start the IP Security Policies configuration utility. This utility is
a snap-in module for the Microsoft Management Console (MMC).
1. Start the Microsoft Management Console (MMC). From the Microsoft Start menu, click Run
and type MMC. Click OK.
2. If the IP Security Policies snap-in configuration utility is not loaded, use the following
procedure to add it:
a. From the MMC window, click File→Add/Remove Snap-in.
b. From the Add/Remove Standalone Snap-in window, click Add.
c. From the Add Standalone Snap-in window, scroll down to IP Security Policy
Management and select it. Click Add.
d. In the Select Computer or Domain window, select Local computer (in this procedure,
we are configuring IP Security for the local computer). Click Finish.
e. Close the Add Standalone Snap-in window by clicking Close.
f. Close the Add/Remove Snap-in window by clicking OK.

Step 2: Creating a Policy
Use the following procedure to create an IP Security policy. An IP Security policy is a set of IPsec
configuration parameters. Only one local IP Security policy can be active (assigned) on a system.
1. In the left navigation pane of the IP Security Policy Management snap-in, click IP Security
Policies on Local Computer to display all IP Security Policies. Depending on your Windows
platform, there may be IP Security Policies already configured.
2. Right click IP Security Policies on Local Computer and select Create IP Security Policy.
3. The Policy Wizard starts and displays a startup message. Click Next.
4. The Policy Wizard opens the IP Security Policy Name window. Enter a name in the Name
field. This name is used only for internal identification.
Click Next.
5. The Policy Wizard opens the Requests for Secure Communication window. Clear the Activate
the default response rule check box, as shown in Figure 1. (The default response rule is a
pre-configured rule that causes the Windows system to dynamically build a filter list based
on the receipt of IKE requests. By default, the Windows system attempts to use IPsec only
if it receives an IKE request from a remote system.)
Click Next.

Configuring a Windows Host-to-Host Policy 15

Figure 1 IP Security Policy Wizard

6. The Policy Wizard opens the Completing the IP Security policy wizard window. Select the
Edit properties check box if it is not already selected.
Click Finish.
The IP Security configuration utility opens the Policy Properties dialog box. The title of the
window will be name Policy, where name is the policy name.

Step 3: Adding a Rule
The primary purpose of a rule is to assign actions to filters. A rule also specifies IKE authentication
methods. Use the following procedure to add a rule to the IP Security policy:
1. Select the Rules tab in the Policy Properties dialog box.
Clear the Use Add Wizard box check box if it is selected (Figure 2).
Click Add.

16

Figure 2 Rules Tab

2. The IP Security configuration utility opens the Rule Properties dialog box, which has a tab
for each category of rule configuration data: IP Filter List, Filter Action, Authentication
Methods, Tunnel Setting, and Connection Type ( Figure 3).

Figure 3 Rule Properties Dialog Box


TIP: After you have created a rule, you can open the Rules Properties dialog box by right
clicking the rule and selecting Properties.

Step 4: Creating the IP Filter List and Filters for the Rule
An IP filter list can contain one or more filters. IPsec uses the filters to determine which rule to
apply to an IP packet. The IP Security configuration utility displays the rules for a policy in
reverse alphabetical order based on the name of the IP filter list for the rule.
Filter Order There is no method for specifying the search or priority order for the filters in a
rule or for the order of rules in a policy. The Windows IP Security module automatically creates
an internal filter list and orders the filters from most specific to least specific.
Use the following procedure to configure the IP filter list and a filter:
1. Create an IP filter list.
Select the IP Filter tab from the Rule Properties dialog box.
The IP Filter List tab shows a list of filters already defined for IP Security policies. Each rule
can have only one filter list, but the filter list can specify multiple filters. In this example,
we will create a new filter list that contains one filter.
Click Add at the bottom of the dialog box.
The IP Security configuration utility opens the IP Filter List dialog box. In the Name field,
enter a name for the filter list. This name is used only for internal identification. Optionally,
add a description. In Figure 4, the administrator enters the name foo.
Clear the Use Add Wizard box check box if it is selected.
Click Add.

Figure 4 Creating an IP Filter List

18

The IP Security configuration utility opens a Filter Properties dialog box.
2. Select the Addressing tab in the Filter Properties dialog box. Use the drop-down menus to
specify the address types for the source and destination addresses. The selections are:
• My IP Address1
• Any IP Address
• A specific DNS Name
• A specific IP Address
• A specific IP Subnet
Enter the source and destination IP addresses or DNS names for the filter. If you selected A
specific IP Subnet, enter the subnet mask.

WARNING! Be careful when configuring filters that affect packets required for basic
network operation, such as packets exchanged with DNS servers and ICMP packets
exchanged with routers. If you configure a policy that requires IP Security for these packets
and the remote node does not support IP Security, your system can lose network functionality.
Leave the Mirrored check box selected, which creates a bi-directional filter that applies to
packets to and from the destination system. See “Mirrored Filters” (page 41) for more
information about mirrored filters.
In Figure 5, the administrator specifies an address filter with the Windows system address
(10.1.1.1) as the source address and the HP-UX system address (10.2.2.2) as the destination
address. The Mirrored check box is selected, so the address filter also matches packets from
the HP-UX system.

Figure 5 Address Tab for Filter Properties

3. Select the Protocol tab in the Filter Properties dialog box. By default, the filter applies to all
protocol types. Select the protocol type (for example, TCP) from the drop-down box. If you
select TCP or UDP, you can also specify the From (source) port and To (destination) port.
Click OK to return to the Filter Properties dialog box.
1. HP-UX did not test the My IP Address selection with multihomed Windows systems. However, the Windows
documentation states that in a multi-homed system, My IP Address matches every IP address on the system.


In Figure 6, the administrator specifies protocol information for a Windows system that will
be a telnet client. The protocol type is TCP, the source port is a wildcard (any port), and the
destination port is the IANA registered TCP port number for the telnet service, 23.

Figure 6 Protocol Tab for Filter Properties

20

4. From the IP Filter List dialog box, you can add another filter to the filter list by clicking the
Add button.
Click OK in the IP Filter List dialog box to return to the IP Filter List tab in the Rule Properties
dialog box.
5. Add the filter list to the rule by selecting the option button for the filter list you just created.
In Figure 7, the administrator added the filter list foo for the rule.

Figure 7 Selecting the Filter List for a Rule

Step 5: Configuring Filter Actions for the Rule
The filter action specifies the action to take for the rule, such as allow (pass), block (discard), or
negotiate security (negotiate IPsec AH or ESP Security Associations). If you select negotiate
security, the filter action also specifies parameters for IPsec Security Association (SA) proposals:
ESP or AH transforms and IPSec SA key lifetimes. A rule can have only one filter action, but the
filter action can specify multiple IPsec SA proposals. You can specify the order for the IPsec SA
proposals.
The filter actions you configure in the Windows IP Security rule must be compatible with the
value or values specified for the -action argument in the HP-UX ipsec_config add host
or add tunnel command.
Use the following procedure to configure filter actions:
1. Select the Filter Action tab from the Rule Properties dialog box.
The Filter Action tab shows a list of filter actions already defined for IP Security. In this
procedure, we will create a new filter action.
Clear the Use Add Wizard check box if it is selected and click Add.
2. The IP Security configuration utility opens the Filter Action Properties dialog box with the
following tabs:


• Security Methods
• General
Select the Security Methods tab, then select Negotiate security. Verify that the following
check boxes are not selected:2
• Accept unsecured communication, but always respond using IPSec.
• Allow unsecured communication with non-IPSec-aware computer.
In addition, verify that the Session key perfect forward secrecy (PFS) check box is not selected.
(HP-UX does not support session key PFS, also referred to as PFS for keys only. HP-UX
supports PFS for keys only in conjunction with PFS for identities. See “Perfect Forward
Secrecy (PFS)” (page 43) for more information.)
For example:

Figure 8 Security Methods for Filter Action

Click Add.
The IP Security configuration utility opens the Security Method dialog box (Figure 9):

2. HP-UX IPSec does not have options that are equivalent to these check boxes. If an HP-UX IPsec policy requires IP
security, then HP-UX always requires IP security for packets that match the policy and drops any packets that match
the policy but are not secured.

22

Figure 9 Security Method Dialog Box

The Encryption and Integrity and Integrity only methods each correspond to a set of
predefined parameters for an IPsec SA proposal, including an IPsec transform type (such
as ESP). The transforms and additional SA parameters defined for these methods may vary
according to the Windows release installed. On Windows XP systems with SP2, these methods
are defined as follows:
• Encryption and Integrity
Authenticated ESP using 3DES encryption and SHA1 authentication (this is equivalent
to the HP-UX IPSec transform ESP_3DES_HMAC_SHA1). No SA lifetimes are specified,
and these settings are compatible with the HP-UX default SA lifetimes (see “IPsec SA
Key (Session Key) Lifetime Values” (page 43) for more information).
• Integrity only
Authenticated ESP using NULL encryption and SHA1 authentication. (this is equivalent
to the HP-UX IPSec transform ESP_NULL_HMAC_SHA1) No SA lifetimes are specified,
and these settings are compatible with the HP-UX default SA lifetimes (see “IPsec SA
Key (Session Key) Lifetime Values” (page 43) for more information).
To use a predefined method, select the appropriate method from the list and click OK to
return to the Security Methods tab in the Filter Actions dialog box.
To create a custom method, use the following procedure:
a. Select Custom in the Security Method dialog box, then click Settings to open the
Custom Security Method Settings dialog box.
b. In the Custom Security Method Settings dialog box (Figure 10), select the appropriate
transform type, algorithms, and session key lifetimes. See “IPsec SA Key (Session Key)
Lifetime Values” (page 43) for more information about IPsec session key lifetimes.


Figure 10 Custom Security Methods Settings Dialog Box

c. Click OK to return to the Security Methods tab in the Filter Actions dialog box.
If the parameters you configured for a custom method match a predefined method, the
configuration utility will display an informative message and select the matching
predefined method.

3. From the Security Methods tab, you can add more methods (IPsec SA proposals) by clicking
the Add button. If you have multiple IPsec SA proposals, the configuration utility lists them
in the preference order IKE will use when negotiating IPsec SAs. You can change the order
by using the Move up and Move down buttons. You can also use the Delete button to delete
IPsec SA proposals (such as proposals that use DES, which has been cracked—data encrypted
using DES has been decrypted by unauthorized parties) .
4. (Optional) Configure the action name. When you create a new action (transform), the
configuration utility assigns it the name New Filter Action. To change the name, select the
General tab from the Action Properties dialog box. Enter a new name and click OK to return
to the Filter Action tab in the Rule Properties dialog box.
5. Apply the new action to the rule. In the Filter Action tab, click on the option button for the
filter action you just created to apply the action to the rule you are configuring. For example,
in Figure 11, the administrator created the action my_action and selected it for the rule.
Click Apply.

24

Figure 1 Selecting the Filter Action
1

Step 6: Configuring the IKE Authentication Method and Preshared Key for the Rule
When configuring a rule to be compatible with HP-UX IPSec, the authentication method specifies
the IKE authentication method (preshared key or certificates) for IPsec. The authentication method
must match the value specified for the -authentication argument in the ipsec_config
add ike command.
Windows also allows you to configure Kerberos (Active Directory) as an authentication method
for IKE (this is the default), but HP-UX does not support this authentication method.
Use the following procedure to configure the IKE authentication method:
1. Select the Authentication Methods tab from the Rule Properties dialog box.
2. Click Add to open the Authentication Method dialog box.
3. To use IKE authentication with a preshared key, select Use this string. This is equivalent
to specifying -authentication PSK in the ipsec_config add ike command.
Enter the preshared key as ASCII text. Do not enclose the key in double quotes. The preshared
key must match the preshared key on the HP-UX system, which is configured using the
-preshared argument in the ipsec_config add auth command. For example:


Figure 12 Configuring A Preshared Key

To use IKE authentication with certificates, select Use a certificate from this certification
authority (CA). Click Browse. The IP Security configuration utility opens a Select Certificate
box with a list of CA certificates stored on your system. Select the CA for the appropriate
CA and click OK. (For additional information about configuring Microsoft Windows
certificates, see Using Microsoft Windows Certificates with HP-UX IPSec, available at
http://docs.hp.com.
4. After you have specified the IKE authentication method, click OK to return to the
Authentication Methods tab in the Rule Properties dialog box.
5. In the Rule Properties dialog box, remove the Kerberos authentication method from the
authentication methods list by highlighting it and clicking Remove.
The configuration utility will display a confirmation message (Are you sure?). Click Yes

Step 7: Configuring the Connection Type for the Rule
The connection type specifies the types of network connection to which the rule will apply. By
default, the IP Security configuration utility creates rules that apply to all network connection
types. To change the connection type, use the following procedure:
1. Select the Connection Type tab from the Rule Properties dialog box.
2. The IP Security configuration utility opens the Connection Type dialog box with the following
selections:
• All network connections: the rule applies to all network connections
• Local area network (LAN): the rule applies only to LAN connections
• Remote access: the rule applies only to VPN and dial-up connections
Select the appropriate connection type and click OK. If you have configured all the required
parameters for a rule, the IP Security configuration utility will return to the Policy Properties
dialog box.

Step 8: Modifying IKE Parameters for the Policy
By default, HP-UX IPSec negotiates IKE SAs using a single proposal with the following parameters:
26

• Encryption algorithm: 3DES
• Hash algorithm: MD5
• Diffie-Hellman Group: 2
• Maximum lifetime: 28,800 seconds (8 hours)
• Maximum Quick Modes: 100
You can specify alternative values for the above parameters in the ipsec_config add ike
command.
On Windows XP systems with SP2, IP Security policies are pre-configured with four IKE SA
proposals. The second IKE proposal matches the default HP-UX IPSec IKE proposal3, and will
be used by the two systems if no changes are made to the default configuration data. If these
IKE parameters meet your security requirements, you do not need to modify the IKE parameters
and can skip to “Step 10: Assigning the IP Security Policy” (page 30).
Use the following procedure to modify the Windows IKE SA parameters:
1. From the Policy Properties dialog box, select the General tag. The IP Security configuration
utility opens the General dialog box (Figure 13).
Click Advanced4. (Ignore the field labeled Check for policy changes. This field is used
only when the policy is stored in an Active Directory.)

Figure 13 General Policy Properties Dialog Box

2. The IP Security configuration utility opens the Key Exchange Settings dialog box (Figure 14).

3. By default, the first Windows XP proposal has the following parameters: Encryption - 3DES; Hash - SHA1;
Diffie-Hellman Group - 2. The third and fourth Windows proposals are weaker, and use DES encryption and
Diffie-Hellman Group 1. Refer to the Windows documentation for more information.
4. On Windows 2003 servers, this button is labeled Settings.


Figure 14 Key Exchange Settings Dialog Box

Configure the fields as follows:
• Master key perfect forward secrecy (PFS)
Selecting this check box sets the maximum number of IPsec or Quick Mode (QM)
negotiations that IKE can perform using an IKE SA to 1. It is equivalent to specifying
-maxqm 1 in the ipsec_config add ike command. PFS is computationally
expensive and HP recommends that you enable it only in hostile environments. See
“Maximum Quick Modes” (page 43) and “Perfect Forward Secrecy (PFS)” (page 43)
for more information.
• Authenticate and generate a new key after every: ____ minutes
This field specifies the maximum lifetime for an IKE SA in units of time. It is equivalent
to the -life argument of the ipsec_add ike command.

TIP: Note that this value is specified in minutes on Windows systems and in seconds
on HP-UX systems.

• Authenticate and generate a new key after every: ____ sessions
This field specifies the maximum QM negotiations per IKE SA. It is equivalent to the
-maxqm argument of the ipsec_add ike command. See “Maximum Quick Modes”
(page 43) and “Perfect Forward Secrecy (PFS)” (page 43) for more information.
Modify the values as appropriate. If you do not want to modify the IKE encryption, hash,
or Diffie-Hellman Group parameters, click OK to return to the General dialog box and
continue to the next step.
To modify IKE encryption algorithm, hash algorithm, or Diffie-Hellman Group parameters,
click Methods. The IP Security configuration utility opens the Key Exchange Security
Methods dialog box, which lists IKE algorithms and Diffie-Hellman Groups that correspond
to IKE SA proposals in order of preference. You can change the order by using the Move
up and Move down buttons. You can also delete IKE SA proposals (such as proposals that
use DES, which has been cracked) using the Delete button.
To add another IKE SA proposal (method), click Add.
The IP Security configuration utility opens the IKE Security algorithms dialog box (Figure 15).

28

Figure 15 IKE Security Algorithms Dialog Box

Use the drop-down menus to select the appropriate integrity algorithm, encryption algorithm,
and Diffie-Hellman Group (these are equivalent to the -hash, -encryption, and -group
arguments of the ipsec_config add ike command).
3. Click OK to return to the Key Exchange Security Methods dialog box.
Click OK to return to the Key Exchange Settings dialog box.
Click Close to close the Policy Properties dialog box.

Step 9: Starting the IP Security Service
Use the following procedure to start the IP Security service. The IP Security service must be
running before you can assign a policy.
1. From the Microsoft Start menu, select Control Panel→Administrative Tools→Services.
2. Scroll down and select IPSEC Services.
3. The Service manager opens the IPSEC Services Properties dialog box (Figure 16).
In the Startup type selection menu, select Automatic.
If the Service status is not Started, click Start.
Click OK to close the IPSEC Services Properties dialog box.
Close the Services dialog box.


Figure 16 IPSEC Services Properties Dialog Box

Alternatively, you can manually start the IP Security service by entering the following Windows
command:
net start policyagent
You can also use the following sequence of commands to manually stop and restart the IP Security
service. This also clears any existing IPsec SAs,:
net stop policyagent

Step 10: Assigning the IP Security Policy
The IP Security subsystem will not use the new policy until you assign (activate) it. Only one IP
Security policy can be assigned or active for the system. To assign the new IP Security policy,
return to the MMC window. Right click the policy in the MMC window and select Assign, as
shown in Figure 17.

30

Figure 17 Assigning the IP Security Policy

Step 1 Verifying the Configuration
1:
To verify your configuration, generate traffic that matches the address filter.
On the HP-UX system, enter the following command to verify that the IKE SA and IPsec SAs
are established:
ipsec_report -sa

Example
In this example, IPsec secures telnet connections from the Windows system to the HP-UX system,
using authenticated ESP.
The Windows system's address is 10.1.1.1
The HP-UX system's address is 10.2.2.2.

Windows Configuration
The Windows administrator configures and assigns an IP Security policy with the following
parameters:
• One rule, with the following parameters:
— Filter List: One filter, with the following parameters:
◦ Addressing:
– Source address: the Windows system's address.
– Destination address: the HP-UX system's address.
– Mirrored: yes (the Mirrored box is selected).
These parameters are shown in Figure 5 (page 19).
◦ Protocol: TCP; source port any, destination port 23 (telnet).
– Protocol: TCP
– From port: any
– To port: 23 (telnet server)
These parameters are shown in Figure 6 (page 20).
— Filter Action: Negotiate security, using the default settings for Encryption and Integrity
(authenticated ESP using 3DES and SHA1).
— Authentication Method: IKE using the preshared key my_preshared_key, as shown
in Figure 12 (page 26).
— Tunnel Settings: No tunnel (this is the default).
— Connection Type: All network connections (this is the default).
• General parameters: The general parameters for the policy are set to the default values (four
IKE SA proposals, including 3DES encryption, SHA1 integrity and Diffie-Hellman Group
2).


HP-UX Configuration
On the HP-UX system, the administrator configures the following policies and records:
ipsec_config add host telnet_from_foo1
-source 10.2.2.2/32/TELNET -destination 10.1.1.1
-action ESP_3DES_HMAC_SHA1

ipsec_config add ike foo1 -remote 10.1.1.1 -auth PSK

ipsec_config add auth foo1 -remote 10.1.1.1
-psk my_preshared_key
If the HP-UX IPSec subsystem is not already started, the administrator starts it using the
ipsec_admin -start command.

Additional Options
For information on additional options and commands, see the HP-UX IPSec Administrator's Guide
and the following manpages:
ipsec_admin(1M)
ipsec_config(1M)
ipsec_policy(1M)
ipsec_report(1M)
.

32

Configuring a Windows End-to-End Tunnel Policy
The only IPsec tunnel topology supported between an HP-UX system and a Windows system is
an end-to-end tunnel.5The procedure for configuring an end-to-end tunnel policy on Windows
system is the same as procedure for configuring a host policy, except that you must configure
two, non-mirrored rules: one rule for outbound packets and one rule for inbound packets, as
described in the sections that follow.

NOTE: Do not configure any other rules in the policy with the HP-UX system address as the
destination address. This prevents the Microsoft system from applying the tunnel transform over
a host-to-host (transport) transform. In end-to-end tunnel topologies, HP-UX IPSec does not
support transport transforms over a tunnel transform.

Outbound Tunnel Rule Requirements
The outbound tunnel rule must have the following parameters:
• Filter List: One filter, with the following parameters:
— Address:
◦ Source address: the HP-UX system's address.
◦ Destination address: this must be a specific IP address and must be the Windows
system's address.
◦ Mirrored: no (the Mirrored box is cleared).
— Protocol Type: none (wildcard). The Windows documentation states that the filters in
tunnel rules must not specify protocols or ports to ensure that IP Security can correctly
process IP fragments.
• Tunnel Setting
— Tunnel endpoint: the HP-UX system's address. This is the address of the tunnel endpoint
closest to the destination. Since this is an end-to-end tunnel, it is the same as the
destination address in the address filter.

Inbound Tunnel Rule Requirements
The inbound tunnel rule must have the following parameters:
• Filter List: One filter, with the following parameters:
— Address:
◦ Source address: the Windows system's address.
◦ Destination address: this must be a specific IP address and must be the HP-UX
system's address.
◦ Mirrored: no (the Mirrored box is cleared).
— Protocol Type: none (wildcard).
• Tunnel Setting
— Tunnel endpoint: the Windows system's address. This is the address of the tunnel
endpoint closest to the destination. Since this is an end-to-end tunnel, it is the same as
the destination address in the address filter

Configuring a Tunnel Rule
Use the following procedure to configure an outbound or inbound tunnel rule.

5. You can also configure an IPsec topology where packets exchanged between an HP-UX system and a Windows
system are tunneled through an IPsec gateway device, but neither HP-UX nor Windows systems can be configured
as IPsec gateways. The only topology in which an HP-UX system can act as an IPsec gateway is when the HP-UX
system is a Home Agent for Mobile IPv6 clients. The HP-UX IPSec Administrator's Guide describes how to configure
a host-to-gateway IPsec topology using HP-UX and a Cisco router.

Configuring a Windows End-to-End Tunnel Policy 33

TIP: The tunnel setting is used by all packets selected using the address filters for the rule. Do
not include any filters for host-to-host (non-tunneled) packets in the filter list for a rule with a
tunnel.
1. Start the IP Security Policies snap-in if necessary.
2. Create an IP Security policy or modify an existing policy. To modify an existing policy, select
the policy in the right navigation pane and right click the policy. Select Properties.
3. The IP Security configuration utility opens the Policy Properties dialog box. Select the Rules
tab. Click Add to create a new rule or select a rule you want to modify and click Edit.
4. Configure a new rule or modify an existing rule with the appropriate address filter for the
outbound tunnel rule or inbound tunnel rule, as described in “Outbound Rule” (page 34)
or “Inbound Rule” (page 35). See “Step 4: Creating the IP Filter List and Filters for the Rule”
(page 18) if you need additional information about configuring address filters.
Record the destination address; you will need it to configure the tunnel endpoint.
5. Return to the Rule Properties dialog box. Select the Tunnel Setting tab.
6. The IP Security configuration utility opens the Tunnel Setting dialog box.
Select The tunnel endpoint is specified by this IP address.
Enter the IP address of the tunnel endpoint closest to the destination. Since this is an
end-to-end tunnel, it is the same as the destination address in the address filter.
7. Click Close to close the Tunnel Setting dialog box.
8. If this is a new rule, complete the configuration by configuring the appropriate filter action,
authentication methods, and connection type.
Click Close to close the Rule Properties dialog box.

Example
In this example, IPsec secures all packets between the Windows system and the HP-UX system
using authenticated ESP.
The Windows system's address is 10.1.1.1
The HP-UX system's address is 10.2.2.2.

Windows Configuration
On the Windows system, you configure one rule for outbound packets and one for inbound
packets.

Outbound Rule
The outbound rule is for packets from the Windows system (source address 10.1.1.1) to the
HP-UX system (destination address 10.2.2.2). Figure 18 shows the address filter for this rule, and
Figure 19 shows the corresponding tunnel settings:

34

Figure 18 Outbound Rule Filter

Figure 19 Outbound Rule Tunnel Settings

Inbound Rule
The inbound rule is for packets to the Windows system (destination address 10.1.1.1) from the
HP-UX system (source address 10.2.2.2). Figure 18 shows the address filter for this rule, and
Figure 21 shows the corresponding tunnel settings:


Figure 20 Inbound Rule Filter

Figure 21 Inbound Rule Tunnel Settings

Additional Parameters
You must configure the remaining rule parameters (filter action, authentication methods, and
connection type) to be compatible with the HP-UX configuration. In addition, the general
parameters for the rule (the IKE SA parameters) must be compatible with the HP-UX configuration.

36

HP-UX Configuration
On the HP-UX system, the host and tunnel policies are bi-directional (mirrored), so you configure
only one host policy and only one tunnel policy. Since this is an end-to-end tunnel, the tunnel
policy does not have to specify the tunnel endpoints. HP-UX IPSec will use the end source and
end destination addresses as the tunnel addresses (the tsource and tdestination values
default to the source and destination values).
ipsec_config add host foo1 -source 10.2.2.2
-destination 10.1.1.1 -action PASS -tunnel foo1_tunnel

ipsec_config add tunnel foo1_tunnel -source 10.2.2.2
-destination 10.1.1.1 -action ESP_3DES_HMAC_SHA1
You must also configure an IKE policy and an authentication record to complete the configuration:
ipsec_config add ike foo1 -remote 10.1.1.1 -auth PSK

ipsec_config add auth foo1 -remote 10.1.1.1
-psk my_preshared_key


Troubleshooting Tips
Most interoperability problems occur during IKE negotiations, so examining IKE log events is
useful. You can use the following procedures to enable and view IKE log events:

Using IKE Logging on HP-UX Systems
Use the following procedure to view detailed IKE log events on HP-UX systems:
1. Enter the following command to set the HP-UX IPSec log level to debug and increase the
maximum log file size:
ipsec_admin -al debug -maxsize 99999
IPSec creates the log files in the /var/adm/ipsec directory. The log file names are
auditdateinfo.log
2. Reproduce the problem.
3. Enter the following command to format the audit file:
ipsec_report -audit /var/adm/ipsec/auditdateinfo.log
4. Use the following command to set the HP-UX IPSec log level back to warning (the default
log level):
ipsec_admin -al warning

Using IKE Logging on Windows Systems
Use the following procedure to view IKE log events on Windows systems:
1. Enable IKE logging. On Windows XP systems, use the regedit utility to enable IKE logging
in the system registry. On Windows systems, IKE logging is configured using the Oakley6
key.

CAUTION: Incorrectly editing the registry may severely damage the system. Before making
changes to the registry, HP recommends that you back up the registry and any valued data
on the computer. Refer to the article How to back up, edit, and restore the registry in Windows
XP and Windows Server 2003 in the Windows Knowledge Base for more information. The
Windows Knowledge Base is available at http://support.microsoft.com
Set the
HKEY_LOCAL_MACHINESystemCurrentControlSetServicesPolicyAgentOakleyEnableLogging
REG_DWORD value to 1. On some Windows versions, you may need to create the Oakley
key.
On Windows 2003 systems, enter the following command to enable IKE logging:
netsh ipsec dynamic set config ikelogging 1
2. Stop and restart the IP Security service. You can use the following commands at the Windows
command prompt:
net stop policyagent
Refer to “Step 9: Starting the IP Security Service” (page 29) for more information.
3. Reproduce the problem.
4. View the IKE log file. Windows creates the log file in the directory systemrootDebug (by
default, this is the WINDOWSDebug directory). The file name is Oakley.log.

6. The Oakley protocol is a key-agreement protocol that is incorporated in the IKE protocol.

38

5. Disable IKE logging. On Windows XP systems, set the
HKEY_LOCAL_MACHINESystemCurrentControlSetServicesPolicyAgentOakleyEnableLogging
REG_DWORD value to 0.
On Windows 2003 systems, enter the following command:
netsh ipsec dynamic set config ikelogging 0
6. Stop and restart the IP Security service.

Additional Windows Troubleshooting Tools
Windows supports an IP Secu;rity Monitor snap-in utility for the Microsoft Management Console
(MMC) that provides IPsec statistics and information about IKE and IPsec Security Associations.
Refer to the Windows documentation set for more information.

Troubleshooting Tips 39

Comparing HP-UX and Windows IPsec Configuration Parameters
This section contains Table 1, which compares how HP-UX and Windows systems configure and
store IPsec parameters. It also contains the following subsections, which provide additional
comparative information:
• “Mirrored Filters” (page 41)
• “Filter Selection” (page 42)
• “IKE Parameter Selection” (page 42)
• “IKE SA Key (Master Key) Lifetime Values” (page 42)
• “Maximum Quick Modes” (page 43)
• “Perfect Forward Secrecy (PFS)” (page 43)
• “IPsec SA Key (Session Key) Lifetime Values” (page 43)
Table 1 IPsec Parameters on Windows and HP-UX
Parameter Windows Configuration HP-UX Configuration Notes

Address Filters Specify them in the Filter Specify one filter per host, Windows and HP-UX
List for a rule. The Filter List tunnel, or gateway policy. support subnet masks for IP
can contain multiple Use the -source and addresses and wildcards for
address filters. -destination arguments IP addresses, protocols, and
in the ipsec_config add port numbers.
host , tunnel, or See “Mirrored Filters”
gateway command. (page 41) for additional
information.

IPsec SA Proposals Specify them in the Filter Specify them using the HP-UX IPSec supports ESP
Action for a rule. -action argument in the encryption using the
ipsec_config add following protocols:
gateway, host, or tunnel Advanced Encryption
command. Standard (AES), Triple Data
Encryption Standard
(3DES), and Data
Encryption Standard (DES).
Windows XP and Windows
2000 support 3DES and
DES, but do not support
AES.

Filter Priority Not applicable. Specify it using the See “Filter Selection”
-priority argument in (page 42) for additional
the ipsec_config add information.
gateway or host
command.

Maximum IPsec SA Specify it in the Custom Specify it in the transform See “IPsec SA Key (Session
Lifetime, measured by time Security Methods dialog specification for the Key) Lifetime Values”
or by data box under the Filter Action -action argument in the (page 43) for additional
for a rule. ipsec_config add host information.
or tunnel command.

Tunnel endpoint address Specify the destination Specify the endpoints using See “Mirrored Filters”
tunnel endpoint (the the -tsource and (page 41) for additional
endpoint for the -tdestination information.
destination) in the Tunnel arguments of
Settings for a rule. You must theipsec_config add
configure two tunnel command.
uni-directional
(non-mirrored) rules.

IKE Authentication Method Specify it in the Specify it using the -auth
Authentication Methods for argument of the
a rule. ipsec_config add ike
command.

40

Table 1 IPsec Parameters on Windows and HP-UX (continued)
Parameter Windows Configuration HP-UX Configuration Notes

IKE Preshared Key Specify it in the Specify it using the
Authentication Methods for -preshared argument of
a rule. the ipsec_config add
auth command.

IKE Exchange Type Windows supports only Specify it using the
Main Mode exchanges. -exchange argument of
the ipsec_config add
auth command. The
default value is MM (Main
Mode).

Maximum IKE SA Lifetime, Specify it in the Key Specify it using the-life The Windows IP Security
measured by time Exchange Settings dialog argument in the Policy snap-in utility uses
box. (To navigate to the Key ipsec_config add ike minutes as the time unit.
Exchange Setting dialog command. The HP-UX ipsec_config
box, select the General tab command uses seconds as
in the Policy Properties the time unit. See “IKE SA
dialog box, then select Key (Master Key) Lifetime
Advanced settings.) Values” (page 42) for
additional information.

Maximum Quick Mode Specify it in the Key Specify it using the-maxqm See “Maximum Quick
(QM) negotiations per IKE Exchange Settings dialog argument in the Modes” (page 43) for
SA box. (To navigate to the Key ipsec_config add ike additional information.
Exchange Setting dialog command.
box, select the General tab
in the Policy Properties
dialog box, then select
Advanced settings.)

Perfect Forward Secrecy Windows supports PFS for HP-UX does not support See “Perfect Forward
(PFS) keys only (PFS for session PFS for session keys. HP-UX Secrecy (PFS)” (page 43) for
keys) and supports PFS for supports only PFS for more information.
keys in conjunction with master keys.
PFS for all identities (PFS Specify PFS for master keys
for master keys). using the-maxqm 1
Specify PFS for master keys argument in the
in the Key Exchange ipsec_config add ike
Settings dialog box. (To command.
navigate to the Key
Exchange Setting dialog
box, select the General tab
in the Policy Properties
dialog box, then select
Advanced settings.)

IKE SA Proposals Specify it in the General You can specify the See “IKE Parameter
parameters for a policy. You parameters for one IKE SA Selection” (page 42) for
can configure multiple IKE proposal in an IKE policy, additional information.
SA proposals and their using the -encryption,
preference order. -hash, and -group
arguments in an
ipsec_config add ike
command.

Mirrored Filters
Microsoft filters can be mirrored (bi-directional) or not mirrored (uni-directional). If the filter is
mirrored, the filter will match IP packets with the source and destination addresses and ports
reversed. For example, a filter has the following specifications:
Source address: 10.1.1.1
Destination address: 10.2.2.2

Comparing HP-UX and Windows IPsec Configuration Parameters 41

The filter matches packets with the following addresses:
If the filter is mirrored, it also matches packets with the following addresses:
The mirror setting only affects Windows IP Security behavior before IPsec SAs are established.
If the Windows IP Security module receives a packet via an existing SA, it does not verify that
the packet address fields match the address filter used when the SA was established.
By comparison, HP-UX IPSec host and tunnel policies are always mirrored. (Gateway policies
are the only HP-UX IPSec policies that are not mirrored.)

Filter Selection
Windows does not allow you to specify the search or priority order for the filters in a rule or for
the order of rules in a policy. The Windows IP Security module automatically creates an internal
filter list and orders the filters from most specific to least specific.
HP-UX IPSec allows you to specify a priority value for IPsec and IKE policies. HP-UX IPSec
searches the policies in priority order within each type of policy. Lower priority values have
higher priority (priority value 1 is the highest priority).
If you do not specify a priority value when creating a policy on HP-UX, ipsec_config
automatically assigns a priority value so that the new policy is the last policy searched before
the default policy within its policy type. The output of the ipsec_config show command
includes the priority values for configured policies.

IKE Parameter Selection
On HP-UX systems, only one IKE SA proposal is used for each peer. You can configure multiple
IKE policies, but only one IKE policy is selected per peer, and each IKE policy specifies only one
IKE SA. During IKE negotiations, IKE searches policies in priority order and selects the first
policy with a matching remote address. IKE then uses the IKE SA parameters to send an IKE SA
proposal, or to evaluate the IKE SA proposal(s) it receives.
On Windows systems, you can configure a set of multiple IKE SA proposals, but only one set
per IP Security policy, and only one IP Security policy can be in use (assigned) on the system.

IKE SA Key (Master Key) Lifetime Values
IKE SA key lifetimes (referred to as Master key lifetimes on Windows systems) specify the
maximum lifetimes for IKE SA keys and are specified by units of time (seconds). In addition,
users can specify the maximum number of IPsec SA negotiations that can be completed per IKE
SA (“Maximum Quick Modes” (page 43)).

HP-UX IKE SA Lifetime Values
The HP-UX IPSec default preferred lifetime value for IKE SAs is 28,800 seconds (eight hours).
If the HP-UX system initiates IKE SA negotiations, the HP-UX IKE daemon proposes the preferred
lifetime value to the remote system. The remote system may process this value in any manner
according to the IPsec protocol suite.
If the remote system initiates IKE SA negotiations and sends a proposed value that is longer than
(less secure than) the HP-UX preferred value, HP-UX sends an IKE NOTIFY message with its
preferred value, and this value is used for the SA.
If the remote system initiates IKE SA negotiations and sends a proposed lifetime that is the same
or more secure (shorter than) the HP-UX preferred value, the HP-UX IKE daemon accepts the

42

proposed value sent by the remote system if it is within the range specified by the IPsec protocol
suite.

Windows IKE SA Lifetime Values
By default, Windows XP systems use the following values for preferred IKE key lifetime values:
480 minutes (eight hours)
0 (infinite) IPsec SA negotiations (sessions)
In testing with HP-UX IPSec, HP configured a shorter IKE SA lifetime value on the Windows
system. When the Windows system was the initiator, it sent the configured lifetime value to the
remote system. When the Windows system was the responder, it accepted the value sent by the
HP-UX system but did not send a notification message.

Maximum Quick Modes
HP-UX and Windows enable you to specify the maximum number of IPsec or Quick Mode (QM)
negotiations that IKE can complete per IKE SA. Each IPsec SA negotiation establishes two IPsec
SAs (one in each direction).
The default maximum QM values are as follows:
HP-UX: 100
Windows: 0 (infinite)
If the value for maximum QM is 1, Perfect Forward Secrecy (PFS) for both keys and identities is
implemented. See “Perfect Forward Secrecy (PFS)” (page 43) for more information.

Perfect Forward Secrecy (PFS)
With Perfect Forward Secrecy, the exposure of one key permits access only to data protected by
that key. RFC 2409, The Internet Key Exchange (IKE), defines two forms of PFS:
• PFS for both the keys and the IKE identities. PFS is provided for keys in conjuction with PFS
for identities. IKE deletes the IKE SA after the IPsec negotiation completes. Each IKE SA is
used for only one IPsec negotiation.
The Windows interface refers to this type of PFS as master key PFS.
• PFS for IPsec keys only. The IKE peers perform a key exchange (Diffie-Hellman exchange)
to create new keying material for each IPsec negotiation. The IKE SA is re-used until the
IKE SA lifetime expires.
The Windows interface refers to this type of PFS as session key PFS.
HP-UX IPSec supports PFS for both the keys and the IKE identities but does not support PFS for
IPsec keys only. To be compatible with HP-UX IPSec, do not configure session key PFS on
Windows systems.
Configuring PFS is computationally expensive. In most topologies, the strength of the
cryptographic algorithms is sufficient protection. HP recommends that you enable PFS only in
hostile environments.

IPsec SA Key (Session Key) Lifetime Values
IPsec SA key lifetimes (referred to as session key lifetimes on Windows systems) specify the
maximum lifetimes for IPsec SA keys and are specified by units of time (seconds) and by data
units transferred (kbytes).

HP-UX IPsec SA Lifetime Values
By default, HP-UX uses the following values for preferred lifetime values:
28,800 seconds (eight hours)
0 (infinite) data units

Comparing HP-UX and Windows IPsec Configuration Parameters 43

If the HP-UX system initiates IPsec SA negotiations, the HP-UX IKE daemon proposes the
preferred lifetime values to the remote system. The remote system may process these values in
any manner according to the IPsec protocol suite.
If the remote system initiates IPsec SA negotiations and sends proposed lifetime value that is as
secure or more secure than the HP-UX preferred value (it is shorter than or equal to the HP-UX
preferred value), the HP-UX IKE daemon accepts the lifetime value proposed by the remote
system if it is within the ranges specified by the IPsec protocol suite.
If the remote system initiates IPsec SA negotiations and a proposed lifetime value is less secure
(shorter than) the HP-UX preferred value, HP-UX sends an IKE NOTIFY message with its
preferred value. If this value is acceptable to the remote system, the SA negotiation succeeds and
the value sent in the NOTIFY message is used.

Windows IPsec SA Lifetime Values
By default, the Windows configuration does not specify any IPsec SA lifetime values and does
not propose any during IPsec SA negotiations. This is equivalent to proposing the lifetime values
28,800 seconds (eight hours) and 0 (infinite) data units.
In testing with HP-UX, HP also configured specific IPsec SA lifetime values on the Windows
system and observed behavior equivalent to HP-UX behavior. When the Windows system
initiated the IPsec SA negotiation, it sent the configured lifetime values in the proposal. When
the remote system initiated the IPsec SA negotiation, the Windows system accepted the proposed
lifetime value if it was more secure than its configured value, and sent a notification message
when its configured lifetime value was more secure than the value proposed by the remote
system.

44

Related Publications
The following documents are available at http://docs.hp.com:
• HP-UX IPSec Administrator's Guide
• Using Microsoft Windows Certificates with HP-UX IPSec
• HP-UX IPSec manpages
The following documents are available at http://microsoft.com:
• Step-by-Step Guide to Internet Protocol Security (IPSec)
• IPSec troubleshooting tools

Related Publications 45

glossary
3DES Triple Data Encryption Standard. A symmetric key block encryption algorithm that encrypts
data three times, using a different 56-bit key each time (168 bits are used for keys). 3DES is
suitable for bulk data encryption.
AES Advanced Encryption Standard. Uses a symmetric key block encryption. HP-UX IPSec supports
AES with a 128-bit key. AES is suitable for encrypting large amounts of data.
AH The AH (Authentication Header) protocol provides data integrity, system-level authentication
for IP packets. It can also provide anti-replay protection. The AH protocol is part of the IPsec
protocol suite.
authentication The process of verifying a user's identity or integrity of data, or the identity of the party that
sent data.
DES Data Encryption Standard. Uses a 56-bit key for symmetric key block encryption. It is suitable
for encrypting large amounts of data.
DES has been cracked (data encoded using DES has been decoded by a third party).
Diffie-Hellman Method to generate a symmetric key where two parties can publicly exchange values and
generate the same shared key. Start with prime p and generator g, which may be publicly
known (typically these numbers are from a well-known Diffie-Hellman Group). Each party
selects a private value (a and b) and generates a public value (g**a mod p) and (g**b mod p).
They exchange the public values. Each party then uses its private value and the other party's
public value to generate the same shared key, (g**a)**b mod p and (g**b)**a mod p, which both
evaluate to g**(a*b) mod p for future communication.
The Diffie-Hellman method must be combined with authentication to prevent man-in-the-middle
or third party attacks (spoofing) attacks. For example, Diffie-Hellman can be used with certificate
or preshared key authentication.
ESP The ESP (Encapsulating Security Payload) protocol provides confidentiality (encryption), data
authentication, and an anti-replay service for IP packets. When used in tunnel mode, ESP also
provides limited traffic flow confidentiality. The ESP protocol is part of the IPsec protocol suite.
IKE The Internet Key Exchange (IKE) protocol is used before the ESP or AH protocol exchanges to
determine which encryption and/or authentication services will be used. IKE also manages the
distribution and update of the symmetric (shared) encryption keys used by ESP and AH.
IKE The method used by IKE peers to authenticate each party's identity. HP-UX IPSec supports two
authentication IKE authentication methods: preshared keys and RSA signatures using certificates.
IKE SA IKE Security Association. An IKE SA is a bi-directional, secure communication channel that
IKE uses to negotiate IPsec SAs. IKE can establish IKE SAs using either Main Mode or Aggressive
Mode negotiations. Also referred to as IKE Phase One SA, ISAKMP SA, ISAKMP/MM SA,
Aggressive Mode SA, Main Mode SA.
IPsec SA IPsec Security Association. An IPsec SA is a uni-directional, secure communication channel.
The IPsec SA operating parameters include the IPsec protocol used (ESP or AH), the mode
(transport or tunnel), the cryptographic algorithms (such as AES and SHA-1), the cryptographic
keys, the SA lifetime, and the endpoints (IP addresses, protocol and port numbers). IKE
establishes IPsec SAs using Quick Mode negotiations. Also referred to as IKE Phase Two SA,
IPsec SA, Quick Mode SA.
Perfect Forward With Perfect Forward Secrecy the exposure of one key permits access only to data protected
Secrecy (PFS) by that key. HP-UX IPSec supports PFS for keys and all identities (the IKE daemon can be
configured to create a new IKE SA for each IPsec negotiation). HP-UX IPSec does not support
PFS for keys only (the IKE SA is re-used for multiple IPsec negotiations, with a new
Diffie-Hellman key exchange for each IPsec negotiation).
SA See Security Association. A secure communication channel and its parameters, such as encryption
and authentication method, keys and lifetime..
SHA1 (Secure Hash Algorithm-1). Authentication algorithm that generates a 160-bit message digest
using a 160-bit key.

47

transform A transform defines the IPsec action(s) to be taken on the IP data, such as passing the data in
clear text, discarding the data, authenticating and encrypting the data using ESP, or
authenticating the data using AH.

48 glossary

Configuring Microsoft Windows IP Security to Operate with HP ...

More Related Content

What's hot

Similar to Configuring Microsoft Windows IP Security to Operate with HP ...

More from webhostingguy

Configuring Microsoft Windows IP Security to Operate with HP ...