Just thought I’d drop a post on a compatability issue which caused a lot of hassle for us.
Problem:
Problems establishing site to site VPN between FortiGate 1500D and Check Point 1430 appliance with Gaia embedded.
Error:
Main Mode Sent Notification to Peer: payload malformed – possibly a mismatch in pre-shared keys
Background:
We were trying to set up a site to site VPN between FortiGate and Check Point and spent a considerable amount of time debugging and troubleshooting this issue.
All googling, sniffing and diagnostic pointed to two things:
– Check Pre-shared key
– Check encryption settings and key life time.
But – all settings were identical.
We tried stripping down the configuration to a bare minimum, but did not get anywhere.
Encryption suite used originally:
Phase 1: AES256 / SHA1 group 19
Phase 2: AES128 / SHA1
We tried a number of things, from more complex settings to the simplest encryption suite available, such as the following:
Phase 1: DES / MD5 – group 1 (768 bits)
Phase 2: DES / MD5
The solution:
For some odd reason, the groups we tested (group 1 and 19) were not compatible between the Check Point and FortiGate.
We ended up with group 14 (2048 bits), as shown below. We were also able to use group 2 (1024 bits)
Phase 1: AES256 / SHA1 group 14
Phase 2: AES128 / SHA1
Hopefully someone stomped in a similar scenario may reach a speedier resolution than we did. ^_^
Thanks, this was good to know.