Found out that you need to create your profiles first on the master (Max 4 server profiles)
(config) #mgmt-server profile server-amp1
(config) #mgmt-server profile server-amp2
(config) #mgmt-server profile server-amp3
(config) #mgmt-server profile server-amp4
Then go through each profile and give tiem the correct IP address
(config) #mgmt-server type amp primary-server <IP> profile server-amp1
(config) #mgmt-server type amp primary-server <IP> profile server-amp2
(config) #mgmt-server type amp primary-server <IP> profile server-amp3
(config) #mgmt-server type amp primary-server <IP> profile server-amp4