Chapter 6: Internet Protocol Security

In Brief

Internet Protocol Security (IPSec) is a framework of open standards for ensuring secure private communications over IP networks. Based on standards developed by the Internet Engineering Task Force (IETF), IPSec ensures confidentiality, integrity, and authenticity of data communications across a public IP network. IPSec provides a necessary component for a standards−based, flexible solution for deploying a networkwide security policy.

The IPSec initiative proposed to offer a standard way of establishing authentication and encryption services between end points. This means not only standard algorithms and transforms, but also standard key negotiation and management mechanisms to promote interoperability between devices by allowing for the negotiation of services between them.

IPSec provides Network layer encryption, and the standards provide several new packet formats. Authentication Header (AH) provides data integrity, and Encapsulating Security Payload (ESP) provides data integrity and confidentiality. The Diffie−Hellman protocol is used to create a shared secret key between two IPSec peers, and Internet Key Exchange (IKE), based on Internet Security Association Key Management Protocol (ISAKMP)/Oakley, is the protocol used to manage the generation and handling of keys. It is also the protocol by which potential peer devices form security associations.

A security association (SA) is a negotiated policy or agreed−upon way of handling the data that will be exchanged between two peer devices; the transform used to encrypt data is an example of a policy item. The active SA parameters are stored in the Security Association Database (SAD).

SAs for both IKE and IPSec are negotiated by IKE over various phases and modes. During Phase 1, IKE negotiates IPSec security associations. Two modes can be used for Phase 1:

∙Main mode, which is used the majority of the time.

∙Aggressive mode, which is used under rare circumstances.

The user cannot control which mode is chosen because the router automatically chooses a mode. The mode chosen depends on the configuration parameters used between each peer.

During Phase 2, IKE negotiates IPSec security associations. The only Phase 2 exchange is quick mode:

∙Phase 1—During Phase 1, IKE negotiates IPSec security associations. Two modes can be used for Phase 1:

1.Main mode, which is used the majority of the time.

2.Aggressive mode, which is used under rare circumstances.

The user cannot control which mode is chosen, because the router automatically chooses a mode. The mode chosen depends on the configuration parameters used between each peer:

∙Phase 2—During Phase 2, IKE negotiates IPSec security associations and the only Phase 2 exchange is quick mode.

IPSec SAs terminate through deletion or by timing out. When the SAs terminate, the keys are also

189

discarded. When subsequent IPSec SAs are needed for a flow, IKE performs a new Phase 2 and, if necessary, a new Phase 1 negotiation. A successful negotiation results in new SAs and new keys. New SAs can be established before the existing SAs expire so that a given flow can continue uninterrupted.

Security associations are unidirectional, meaning that for each pair of communicating systems there are at least two security connections. The security association is uniquely identified by a randomly chosen unique number called the security parameter index (SPI) and the destination IP address of the IPSec peer. When a system sends a packet that requires IPSec protection, it looks up the security association in its database, applies the specified processing, and then inserts the SPI from the security association into the IPSec header. When the IPSec peer receives the packet, it looks up the security association in its database by destination address and SPI and then processes the packet as required. In summary, the security association is simply a statement of the negotiated security policy between two devices.

IPSec Packet Types

IPSec defines a set of headers that are added to IP packets. These new headers are placed after the IP header and before the Layer 4 protocol. They provide information for securing the payload of the IP packet. The security services are provided by the Authentication Header (AH) and the Encapsulating Security Payload (ESP) protocols. AH and ESP can be used independently or together, although for most applications, just one of them is sufficient. For both of these protocols, IPSec does not define the specific security algorithms to use; instead it provides an open framework for implementing industry−standard algorithms.

Authentication Header

Authentication Header (AH), described in RFC 2402, ensures the integrity and authenticity of the data, including the invariant fields in the outer IP header. It does not provide confidentiality protection, meaning it does not provide encryption. When an AH mode header is added to an IP packet, it provides authentication for as much of the IP header as possible, as well as for all the upper−layer protocols of an IP packet. However, some of the IP header fields may change in transit, and the value of these fields, when the packet arrives at the receiver, may not be predictable by the sender. These fields are known as mutable fields, and their values cannot be protected by AH. Predictable fields are also known as immutable fields.

AH provides authentication and integrity to packets by performing an integrity check, which is a keyed hash using a shared secret value that creates a message digest, when used together they are known as the integrity check value (ICV). The ICV is computed on the following:

∙IP header fields that are either immutable in transit or predictable in value upon arrival at the end point for the Authentication Header security association.

∙The Authenticated Header, payload length, reserved fields, SPI, sequence number, authentication data, and padding bytes.

∙The upper−layer protocol data.

The result of the integrity check value helps the remote peer to determine if the packet has changed in transit. Upon receipt of the packet, the remote peer performs the same integrity check on the packet and compares the integrity check value that it has computed against the value of the integrity check value that the sender provided. The following are immutable fields and are included in the Authentication Header integrity check value computation:

190

∙Version

∙Internet header length

∙Total length

∙Identification

∙Protocol

∙Source address

∙Destination address

AH cannot include in the integrity check any mutable field of the packet that might be modified by other routers within the transmission path of the packet from the source to the destination. The following are mutable fields and are not included in the computation of the AH ICV:

∙Type of Service (ToS)

∙Flags

∙Fragment offset

Encapsulating Security Payload

Encapsulating Security Payload (ESP), described in RFC 2406, ensures the confidentiality, integrity, and optional authenticity of the data, yet ESP does not use the invariant fields in the IP header to validate data integrity. ESP has an optional field used for authentication. It contains an ICV that is computed over the remaining part of the ESP, minus the authentication field. The length of the optional field varies depending on the authentication algorithm that is chosen. If authentication is not chosen, the ICV is omitted. Authentication is always calculated after the encryption is done.

ESP performs encryption at the IP packet layer. It supports a variety of symmetric encryption algorithms. The default algorithm for IPSec is the 56−bit Data Encryption Standard using the cipher block chaining mode transform (DES−CBC). This cipher must be implemented to guarantee interoperability with other IPSec products.

The services that are provided by ESP depend on the options that are configured during the IPSec implementation and after an IPSec SA is established.

IPSec Modes of Operation

The format of the AH and ESP headers and the values contained within each packet vary according to the mode in which each is used. IPSec operates in either tunnel or transport mode.

Transport Mode

Transport mode is used when both peers are hosts. It may also be used when one peer is a host and the other is a gateway if that gateway is acting as a host. Transport mode has an advantage of adding only a few bytes to the header of each packet. When transport mode is used, the original header is not protected. This setup allows the true source and destination addresses to be viewed by intermediate devices implemented based on the contents of the IP header. One advantage of not changing the original header is that Quality of Service (QoS), can be processed from the information in the IP header. One disadvantage is that it is possible to do traffic analysis on the packets. Transport mode can be used only if the two end devices are the ones providing IPSec protection. Transport mode cannot be used if an intermediate device, such as a router or firewall, is providing the IPSec protection. Figure 6.1 displays an example of a normal IP packet.

191

Figure 6.1: IP packet.

When Authentication Header (AH) is used in transport mode, the AH services protect the external IP header along with the data payload. It protects all the fields in the header that do not change in transport. The AH header goes after the IP header and, if present before the ESP header, if present, and other higher−layer protocols, like TCP. Figure 6.2 shows an example of using AH in transport mode.

Figure 6.2: AH in transport mode.

When ESP is used in transport mode, the IP payload is encrypted; however, the original headders are not encrypted. The ESP header is inserted after the IP header and before the upper−layer protocol header, like TCP. The upper−layer protocols are encrypted and authenticated along with the ESP header. ESP does not authenticate the IP header or the higher−layer information, such as TCP port numbers in the Layer 4 header. Figure 6.3 shows an example of using ESP in transport mode.

Figure 6.3: ESP in transport mode.

Tunnel Mode

Tunnel mode is used between two gateway devices, such as two PIX firewalls, or between a host and a gateway. In tunnel mode, the entire original IP packet is encrypted and becomes the payload of a new IP packet. The new IP header has the destination address of its IPSec peer. This allows for tunneling of IP packets from a protected host through a router or firewall usually to another router or firewall, which can both be acting as security gateways. One of the advantages of tunnel mode is that intermediate devices, such as routers, can do encryption without any modification to the end system. All the information from the original packet, including the headers, is protected. Tunnel mode protects against traffic analysis because, although the IPSec tunnel end points can be determined, the true source and destination end points cannot be determined because the information in the original IP header has been encrypted.

When Authentication Header (AH) is used in tunnel mode, the original header is authenticated and the new IP header is protected in exactly the same manner it was protected when used in transport mode. Figure 6.4 shows an example of using AH in tunnel mode.

192