FortiGate HA: From Heartbeat Cables to a Clean Failover

Introduction

Most firewall pairs get built once and trusted forever. The cluster comes up, the dashboard shows two green units, and nobody touches it again until the night one of them dies. That is the night you find out whether you built a cluster or just racked two firewalls together.

This is for anyone who looks after FortiGate pairs, on the first-line desk or on the escalation. HA is short for high availability: two firewalls set up so that if one stops, the other carries the traffic. The protocol behind it on FortiOS is FGCP, the FortiGate Clustering Protocol.

Core concepts

Primary and secondary

A cluster is two identical FortiGates that behave as one firewall on the network. One is the primary: it forwards the traffic, holds the routing table, runs the security profiles, and copies its configuration across. The other is the secondary, and it waits. That arrangement is active-passive, written a-p in the CLI. In active-active, a-a, the primary still owns every connection and makes every decision but hands some inspection work to the other unit.

Two units form a cluster only if the group ID, group name, password and mode match, on the same model and the same firmware.

The heartbeat

The units keep in touch over heartbeat interfaces: ports cabled directly between them that carry a short "I am still here" message every 200 ms. A heartbeat is a raw Ethernet frame, not an IP packet, so it never crosses a router and never shows up in a routing table.

Virtual MAC addresses

Here is what makes a failover clean. Every device is reached on the wire by a MAC address, the hardware address a switch learns against a port. A cluster does not use the real address burned into each box. It uses a virtual MAC that belongs to the cluster, so when the secondary takes over nothing has to move: the IPs and the MACs are already its own. It sends a gratuitous ARP, a broadcast telling every switch and router on the segment to refresh where that address lives, and traffic follows it over. FortiOS also lets you set those addresses by hand per interface, rather than only by group ID.

Architecture and workflow

Three things travel between the units. Config sync copies the primary's settings down and keeps a checksum, a fingerprint of the config, so both sides can prove they match. Session sync copies the session table, the list of live connections, and only when you turn it on. Heartbeat carries both and decides who leads.

User traffic only touches the primary. The secondary's data ports are up and carry nothing, which is why a dead link there can sit unnoticed for months unless something watches it.

Components and configuration

The whole cluster lives in one CLI block. The comments say what each line does:

A few of those lines need more than a comment.

Heartbeat links and backup-hbdev

The second heartbeat link is not there to share load. It is the one that saves you when someone unplugs the first while tracing another problem.

backup-hbdev is new in 7.6 and has one job. A secondary that hears nothing on any heartbeat port assumes the primary is dead and promotes itself. If the primary was alive all along, both units now answer to the same addresses on one network. That is split brain. The backup heartbeat gives the secondary one more chance to hear the primary, and carries liveness only.

Interface monitoring and session pickup

monitor lists the ports whose failure should move the primary role, so monitoring a link that already has a backup path underneath buys an outage you did not need. session-pickup is off by default, and even switched on it ignores UDP and ICMP unless you add session-pickup-connectionless.

Priority and override

priority and override decide who holds the role. Priority defaults to 128, runs from 0 to 255, and the higher number wins. Override decides where priority sits in that decision. Disabled, the unit that has been up longest keeps the role. Enabled, priority is read first, so your preferred unit takes the role back as soon as it returns. Set override-wait-time too, so a rebooting unit cannot grab the role every few seconds.

Neither setting is synchronized, and that surprises people. Priority, override, the hostname and the reserved management port stay on the unit where you typed them, and licences are per unit. That is why priority is set separately on each box, and why a config restored from the wrong member can hand the role to the wrong one.

Practical examples

Two FortiGate 101Fs at a distribution site: ha1 and ha2 cabled back to back, port1 to the ISP handoff, port5 to the core, both monitored, override disabled.

The optic in port1 on the primary fails at 09:14. A monitored port is down, so that unit's HA uptime drops to zero and the secondary takes the role. Copied sessions survive, and the site-to-site tunnels stay up because the keys that encrypt them, the IPsec security associations, are synced between the members. SSL VPN is the exception: those sessions are not carried across, so remote users reconnect.

The useful part comes at 14:00, when the optic is replaced. The repaired unit rejoins as a secondary and stays there, because override is disabled. With override enabled and a higher priority on that unit, it would have taken the role back on the spot, and the site would have taken a second hit for no gain. Failing back is a decision, not a default.

Common issues and troubleshooting

Failover timing

Slow failover is normally not a fault, it is the default timing. You can get under a second by lowering either number, and every millisecond shaved raises the chance of writing off a unit that was only busy.

Sessions that drop anyway

Sessions that drop despite session pickup were usually never eligible: UDP or ICMP without session-pickup-connectionless, SSL VPN tunnels, or anything in proxy-based inspection.

Commands worth keeping in reach

• get system ha status shows health, mode, uptime and the reason behind every change of primary. Run it first when a pair swapped overnight and nobody knows why.
• diagnose sys ha status shows low-level cluster state, including heartbeat ports and virtual cluster membership. Use it when the dashboard and reality disagree.
• diagnose sys ha checksum cluster compares the config fingerprint on each member. Run it when a unit reports out of sync, which usually means something was changed on the secondary directly.
• diagnose sys ha checksum show breaks that fingerprint down by global and per-VDOM scope. Use it to find the part of the config that actually differs.
• diagnose sys ha dump-by group lists the members with their ages and roles. Use it to see the uptime difference that decided the last election.
• execute ha manage <id> <admin> opens a CLI session on the other member from the primary. Use it when the secondary has no management address of its own.
• execute ha synchronize start forces a manual config sync. Use it when a member sits out of sync after a maintenance window.
• diagnose sys ha reset-uptime sets HA uptime on the current primary to zero and hands the role over. Use it to test a failover when override is disabled, inside a window.
• diagnose debug application hatalk -1 and hasync -1 print heartbeat and sync activity live. Use them when a cluster forms and then falls apart minutes later.

Best practices

• Two heartbeat links with different numbers, cabled directly between the units where they share a rack.
• A backup heartbeat on a data port, so a cut heartbeat cable cannot cause split brain.
• Monitor only the ports whose loss stops the unit doing its job. Never monitor the heartbeat ports.
• Leave override disabled unless one specific unit has to hold the role.
• Turn on session pickup where connections are long lived, with connectionless if voice or DNS matters.
• Give each unit a management port of its own, so monitoring, FortiManager and SSH reach the secondary.
• Leave uninterruptible-upgrade enabled, so firmware loads on the secondary first and costs one controlled failover.
• Test by pulling the link you actually expect to fail, in a window.

Frequently asked questions

Can I cluster two different FortiGate models? No. Both units must be the same model on the same firmware. Mixed models will not form a cluster.

Does the secondary need its own licences? Yes. Licences are per unit and are not synced, so both boxes need their own subscriptions registered.

Will my IP addresses change after a failover? No. The cluster keeps both its IP and its MAC addresses, which is why switches follow the new primary after one gratuitous ARP.

Can I manage the secondary directly? Yes, if you give it a reserved management port. Without one, reach it from the primary with execute ha manage.

Do I have to enable session pickup? Not always. It costs memory and heartbeat bandwidth, so it earns its place where connections are long lived. Short web sessions will reconnect on their own either way.

Why did my pair fail over when nothing was wrong? Usually a monitored port flapped, or the units were too busy to answer heartbeats in time. get system ha status names the reason for every change of primary.

Should I enable override? Only when one specific unit has to hold the role, for example when the two boxes are not equally capable. Otherwise it buys you a second outage when the repaired unit comes back.

Summary

• FGCP joins identical FortiGates into one firewall that keeps its IP and MAC addresses through a failover.
• Heartbeat runs every 200 ms, 20 missed beats writes off a peer, so detection takes about four seconds.
• The role goes to the unit with monitored ports up, then longest HA uptime, then higher priority, then lower serial. Override moves priority ahead of uptime.
• Priority, override, hostname and the reserved management port stay on the unit you set them on.
• Session pickup is off until you turn it on. IPsec tunnels survive a failover, SSL VPN tunnels do not.
• The backup heartbeat is the 7.6 answer to split brain.
• get system ha status explains what happened. diagnose sys ha checksum cluster explains what disagrees.

References

Fortinet, HA heartbeat interface, FortiOS 7.6.0 Administration Guide.