Site to Site VPN between FortiGate and Check Point – malformed payload

Just thought I’d drop a post on a compatability issue which caused a lot of hassle for us.

Problems establishing site to site VPN between FortiGate 1500D and Check Point 1430 appliance with Gaia embedded.

Main Mode Sent Notification to Peer: payload malformed – possibly a mismatch in pre-shared keys

We were trying to set up a site to site VPN between FortiGate and Check Point and spent a considerable amount of time debugging and troubleshooting this issue.
All googling, sniffing and diagnostic pointed to two things:
– Check Pre-shared key
– Check encryption settings and key life time.

But – all settings were identical.
We tried stripping down the configuration to a bare minimum, but did not get anywhere.

Encryption suite used originally:
Phase 1: AES256 / SHA1 group 19
Phase 2: AES128 / SHA1

We tried a number of things, from more complex settings to the simplest encryption suite available, such as the following:
Phase 1: DES / MD5 – group 1 (768 bits)
Phase 2: DES / MD5

The solution:
For some odd reason, the groups we tested (group 1 and 19) were not compatible between the Check Point and FortiGate.
We ended up with group 14 (2048 bits), as shown below. We were also able to use group 2 (1024 bits)

Phase 1: AES256 / SHA1 group 14
Phase 2: AES128 / SHA1

Hopefully someone stomped in a similar scenario may reach a speedier resolution than we did. ^_^

5.00 avg. rating (98% score) - 1 vote

Leave a Reply

Your email address will not be published. Required fields are marked *