Interoperating with Other Vendors

If your connection fails, and the Openswan logs give no further hints, check the logs on the Contivity at Status | Event Log, where you will find the System Log and the Security Log.

Until very recently at least, the Nortel Contivity did not implement Path MTU Discovery for IPsec, instead using a hardcoded MTU of 1500. This can cause Openswan to send an ICMP error back to the Contivity when it tries to build packets that would have a size of 1500 plus the IPsec header. Setting the LAN or the Contivity's MTU to 1400 might avoid this issue.

Checkpoint

Checkpoint has a few VPN appliances in its portfolio. The best known is the VPN-1; other products are the NG VPN and the FP3.

Some checkpoint firewalls do not support PFS. You might need to add pfs=no to your connection.

One issue with a subnet-to-subnet tunnel between Checkpoint and Openswan is that Checkpoint seems to negotiate separate /32 tunnels for each machine in the local LAN. For instance if you have a subnet-to-subnet connection for 10.0.1.0/24 to 10.0.2.0/24, you will get messages like:

Cannot respond to IPsec SA request because no connection is known for 10.0.1.15/32===a.b.c.d[S-C]...e.f.g.h===10.0.2.101/32

Checkpoint also has a mode called Hybrid Mode, which is a precursor to what became XAUTH. It is supported by the Checkpoint SecureClient. This mode is used by default when PSK is used and when using SecureID cards. Openswan does not support Hybrid Mode.

Checkpoint has more issues stemming from its breaking of the RFCs by using ID_USER_FQDN without data, and by stripping the leading @ symbol of the user ID outside of XAUTH. Patches are floating around that send a null ID of type ID_USER_FQDN to mimic the SecureClient behavior, but as far as we know, no one has successfully interoperated Openswan in Hybrid Mode with Checkpoint.

The most problematic feature of Checkpoint is that it will claim to accept any IPsec SA if the authentication matches, even if the requested policies in Phase 2 are wrong. It will then silently drop the packets. This is rather confusing (and violate the IETF RFC standard) because the Openswan end will receive "IPsec SA Established", while, in fact, the Checkpoint appliance has not established the proper IPsec SA.

WatchGuard Firebox

The WatchGuard Firebox is a commonly used VPN gateway. We have seen problems where a WatchGuard appliance used an incorrect implementation of MD5 and would accept an IKE proposal with MD5 but would then send mangled packets that are dropped by Openswan. As a workaround you can tell Openswan to not allow MD5 by adding the following lines to the connection definition:

ike=3des-sha1 esp=3des-sha1

The WatchGuard Firebox SOHO 6tc and BF4S16E6 with a firmware release of 7.2 or earlier seem to have a typo in their code for handling NAT-Traversal. They incorrectly use a value of 61433 instead of 61443 as the code for the Encapsulation Mode Transform attribute signifying ESP over UDP encapsulation.

216