Site-to-Site VPN tunnel goes down when the Phase 2 IPSec Outbound SA lifetime threshold is reached (ASA 8.4 bug)

If you have recently upgraded to ASA 8.4 or above, you might have come across a VPN behavior where the outbound IPSec SA reaches it’s data lifetime threshold and you have to manually bounce the tunnel to bring it back up.

This happens because of a bug found in the code 8.4(2.240) and 8.6. The bug is such that the IPSec outbound SA in Phase 2 fails to rekey when the ‘data lifetime’ reaches it’s threshold limit (default 4608000). CSCtq57752 is the bug ID which you can lookup in the bug tool kit (requires a CCO login).

Continue reading

Using packet-tracer for validating ICMP traffic

TCP and UDP being statefully inspected by default, you just have to run the packet-tracer on the source interface and you can be sure the return traffic will be allowed through.  With ICMP, it’s a different story.

Because the ASA does not statefully inspect ICMP  packets (by default) you have to vouch for the return packets as well. So you’ll be running two packet-tracer commands to verify that ICMP packets go through and come back. Continue reading

Upgrading to ASA 8.4 – ‘No ACL was changed as part of Real-ip migration’

If you have tried to upgrade your ASA to 8.4, you may or may not have come across the most common error, i.e. –

No ACL was changed as part of Real-ip migration

The reason this happens is because of a conflict between the NAT O statement and Static NATs.

Here’s the message I got when the ASA was upgraded to 8.4:-

INFO: MIGRATION - Saving the startup configuration to file
INFO: MIGRATION - Startup configuration saved to file 'flash:8_2_5_13_startup_cfg.sav'
*** Output from config line 4, "ASA Version 8.2(5)13 "
WARNING: 
MIGRATION: NAT Exempt command is encountered in config.
Static NATs which overlap with NAT Exempt source are not migrated.
Please check migrated ACLs for accuracy.
*** Output from config line 1000, "access-group OUTSIDE_ACL..."
WARNING: 
MIGRATION: NAT Exempt command is encountered in config.
Static NATs which overlap with NAT Exempt source are not migrated.
Please check migrated ACLs for accuracy.
*** Output from config line 1001, "access-group INSIDE_ACL ..."
NAT migration logs:
INFO: NAT migration completed.
Real IP migration logs:
 No ACL was changed as part of Real-ip migration

The reason this failed was because it clearly states in the message above that – Continue reading

ASA Failover/High Availabillity – An overview (terms and concepts)

Failover/High Availability – Two ASAs are paired to operate together and provide redundancy in case of a device failure.

Primary Unit – Determines which of the two ASAs will be referred as the ‘Primary Unit’. Initially, all the configurations are done on the primary unit. It should not be confused with the term ‘active’. The primary unit can either be active or standby.

Secondary Unit – Determines which of the ASAs will be referred as the ‘Secondary Unit’. It should not be confused with the term ‘standby’. The secondary unit can either be active or standby.

Active – The device with the active role will handle all the security functions.

Standby – The unit with the standby role will monitor the active unit for failure and take the active role when a failure occurs.

Failover link – A dedicated link between the two ASAs to propagate communication regarding the failover state, configuration/commands synchronization. Although you can use the already existing data interfaces for the failover related communication, it is not recommended.

Stateful Failover link – A dedicated link between the two ASAs to propagate communication regarding the active TCP/UDP sessions so that they can be replicated on to the standby unit for a stateful failover. Always a separate interface must be used because of the large number of connections that are established/turned down, as the same information is cascaded to the standby ASA in realtime.

Types of failover:

  1. Active/Standby Failover – One ASA take the active role and handles all the normal security functions. The other ASA stays in standby mode and is ready to take over the active role in the event of a failure. This type of failover provides device redundancy.
  2. Active/Active Failover – This can be achieved only when using ASAs in multiple context mode. The contexts can be organized into groups and each ASA is active for one group and standby for the other group. This type of failover provides device redundancy as well as load balancing across contexts.

Stateless failover – The TCP/UDP connections are NOT replicated on the standby unit, this causes the active connections to drop when a failover occurs. Both the above failover types support stateless failover.

Stateful failover – The TCP/UDP connections are replicated on the standby unit, so at the time of failover the standby unit has the TCP/UDP state information and the connections are not dropped. Both the above failover types support stateful failover.

State information replicated in Stateful failover- NAT, ARP, MAC, TCP/UDP connections, H.323 and SIP, MGCP and HTTP (if explicitly enabled).

State information not replicated in Stateful failover- Dynamic routing table entries, uauth cut-through proxy, DHCP leases, Phone proxy and SSM Activity.

Prerequisites:

  1. Both the units must be of the same model.
  2. Both the units should have Identical sets of interfaces.
  3. Both the units must have the same software configuration and proper license.
  4. Both the units must have the same SSM modules, if any.
  5. Both the units must be in the same mode (transparent/routed, single/multiple context)

The Election Procedure:

  1. While booting, if both ASAs are equally healthy, primary > active and secondary > standby.
  2. While booting, if one is healthier than the other, the healthier one takes the active role.
  3. While booting, if the active role is already assigned, the other ASA takes the standby role.
  4. While booting, if no peer is detected, the booting ASA will become active.

Monitored Interfaces – By default, every ASA interface will be monitored to detect a failure that might trigger a failover. Interfaces can be excluded from monitoring if required.

Monitoring peer’s health:

  1. If hellos are received over the LAN failover interface, the peer is alive and no failover occurs.
  2. If hellos are not received over the LAN failover interface but are received on other monitored interfaces, the peer is alive and no failover occurs. However, the LAN failover interface is declared as “failed” and should be replaced ASAP.
  3. If no hellos are received over any interface, the peer is declared to be “failed” and failover occurs.

Hello packets – Over the LAN failover interface > 1 second
Over other monitored interface > 5 seconds

Hold timer – Over the LAN failover interface > 15 seconds
Over other monitored interface > 5 times the hello timer (25 seconds)

The Interface testing mode – If hello packets are not seen on a monitored interface within ‘half’ of the hold time (25 seconds), that interface is moved into the ‘testing’ mode to determine if a failure has occurred.

  1. Interface Status – If the link is ‘down’, the interface is failed.
  2. Network activity – If no packets are received for 5 seconds, the next testing phase begins. If packets are received, the interface can still be used.
  3. ARP – The interface stimulates received traffic by sending ARP requests . If no traffic is received in 5 seconds, the next testing phase begins.
  4. Broadcast ping – Traffic is stimulated by sending an ICMP echo request to the broadcast address on the interface. If no replies are received in 5 seconds, the interface is marked as “failed”.

The truth about security-level

When you start studying Cisco ASA, the first thing that will interest you the most is the security-level of an interface. How traffic from a higher security-level interface is permitted to a lower security-level interface and how traffic from a lower security-level interface to a higher security-level interface is denied ‘by default’. This could be termed or referred as default access policies that are implicitly present on every interface on the ASA when configured with a security-level.

The truth is – in the real world you are never going to find an ASA that depends on these so called default access policies to filter traffic on the interface. The security-levels of the interface are not going to decide the fate of your network. Period. And I’ll tell you why…

Right out of the box, the security-level configured on an interface works just fine as it is intended to unless you configure an Access Control List on an interface. Once you apply an ACL on an interface, ASA does not consider the security-level of that interface while making permit/deny decisions, instead it checks the configured ACL on that interface for appropriate actions to be taken when there is a match. Any other traffic that doesn’t have a match is denied by the hidden implicit deny statement at the end of the ACL.

And here’s a hidden fact – Security-level does play an important role when making NAT Exemption decisions. More on this later…