Can not connect Virtual Mobility Master with Mobility Controler

[Audrius]

Hello,

I have a problem with Virtual Mobility Master and Mobility controller connectivity.

We have a client, which has installed VMM and 7008 MC

I received demo licences from out local HPE representitive.

I configured both VMM and MC at first time and everything worked fine,

but after 1 month licences expired and I received new licences.

After new  licences where installed second time to VMM, MC and VMM  do not make a new IPSec tunnel.

I am adding log from both sides

1) VMM

(MM) [mynode] #show  log all 20

Oct 11 12:35:56  fpapps[4292]: <399838> <4295> <WARN> |fpapps|  Received TUN_UP from IKE for default-local-master-ipsecmap-20:4c:03:0a:ad:d0 mapid 0x0, vlanid 0, flags = 0x0 uplink_priority 0
Oct 11 12:35:58  fpapps[4292]: <399838> <4295> <WARN> |fpapps|  Received TUN_DOWN from IKE for default-local-master-ipsecmap-20:4c:03:0a:ad:d0
Oct 11 12:35:58  fpapps[4292]: <399838> <4295> <WARN> |fpapps|  Received TUN_UP from IKE for default-local-master-ipsecmap-20:4c:03:0a:ad:d0 mapid 0x0, vlanid 0, flags = 0x0 uplink_priority 0
Oct 11 12:35:59  fpapps[4292]: <399838> <4295> <WARN> |fpapps|  Received TUN_DOWN from IKE for default-local-master-ipsecmap-20:4c:03:0a:ad:d0
Oct 11 12:35:59  fpapps[4292]: <399838> <4295> <WARN> |fpapps|  Received TUN_UP from IKE for default-local-master-ipsecmap-20:4c:03:0a:ad:d0 mapid 0x0, vlanid 0, flags = 0x0 uplink_priority 0
Oct 11 12:36:00  fpapps[4292]: <399838> <4295> <WARN> |fpapps|  Received TUN_DOWN from IKE for default-local-master-ipsecmap-20:4c:03:0a:ad:d0
Oct 11 12:36:00  fpapps[4292]: <399838> <4295> <WARN> |fpapps|  Received TUN_UP from IKE for default-local-master-ipsecmap-20:4c:03:0a:ad:d0 mapid 0x0, vlanid 0, flags = 0x0 uplink_priority 0
Oct 11 12:36:02  fpapps[4292]: <399838> <4295> <WARN> |fpapps|  Received TUN_DOWN from IKE for default-local-master-ipsecmap-20:4c:03:0a:ad:d0
Oct 11 12:36:02  fpapps[4292]: <399838> <4295> <WARN> |fpapps|  Received TUN_UP from IKE for default-local-master-ipsecmap-20:4c:03:0a:ad:d0 mapid 0x0, vlanid 0, flags = 0x0 uplink_priority 0
Oct 11 12:36:04  fpapps[4292]: <399838> <4295> <WARN> |fpapps|  Received TUN_DOWN from IKE for default-local-master-ipsecmap-20:4c:03:0a:ad:d0
Oct 11 12:36:04  fpapps[4292]: <399838> <4295> <WARN> |fpapps|  Received TUN_UP from IKE for default-local-master-ipsecmap-20:4c:03:0a:ad:d0 mapid 0x0, vlanid 0, flags = 0x0 uplink_priority 0
Oct 11 12:36:10  aaa[4284]: <125022> <4284> <WARN> |aaa|  Authentication failed for User testuser, Logged in from 192.168.160.1 port 27145, Connecting to 192.168.160.99 port 22 connection type SSH
Oct 11 12:36:12  fpapps[4292]: <399838> <4295> <WARN> |fpapps|  Received TUN_DOWN from IKE for default-local-master-ipsecmap-20:4c:03:0a:ad:d0
Oct 11 12:36:12  fpapps[4292]: <399838> <4295> <WARN> |fpapps|  Received TUN_UP from IKE for default-local-master-ipsecmap-20:4c:03:0a:ad:d0 mapid 0x0, vlanid 0, flags = 0x0 uplink_priority 0
Oct 11 12:36:13  -cli: PAPI_open_udp_socket: Service 15301 is taken. Looking for next ID
Oct 11 12:36:13  cli[59144]: USER: admin has logged in from 192.168.160.1.
Oct 11 12:36:13  fpapps[4292]: <399838> <4295> <WARN> |fpapps|  Received TUN_DOWN from IKE for default-local-master-ipsecmap-20:4c:03:0a:ad:d0
Oct 11 12:36:13  fpapps[4292]: <399838> <4295> <WARN> |fpapps|  Received TUN_UP from IKE for default-local-master-ipsecmap-20:4c:03:0a:ad:d0 mapid 0x0, vlanid 0, flags = 0x0 uplink_priority 0
Oct 11 12:36:14  fpapps[4292]: <399838> <4295> <WARN> |fpapps|  Received TUN_DOWN from IKE for default-local-master-ipsecmap-20:4c:03:0a:ad:d0
Oct 11 12:36:14  fpapps[4292]: <399838> <4295> <WARN> |fpapps|  Received TUN_UP from IKE for default-local-master-ipsecmap-20:4c:03:0a:ad:d0 mapid 0x0, vlanid 0, flags = 0x0 uplink_priority 0

2) MC

(MC) #show log all 20

Oct 11 12:34:07  cfgm[3279]: <399816> <3279> <ERRS> |cfgm|  handle_read: State(READY:LAST SNAPSHOT:CFGID-32:PEND-0:INITCFGID:0) FD=27:Failure receiving heartbeat response header information Result=-1 Err=Connection timed out
Oct 11 12:34:07  cfgm[3279]: <399838> <3279> <WARN> |cfgm|  LmsHeartBeatResultAction: State(READY:LAST SNAPSHOT:CFGID-32:PEND-0:INITCFGID:0) FD=27:Cannot heartbeat with the master.
Oct 11 12:34:16  cfgm[3279]: <399838> <3279> <WARN> |cfgm|  LmsHeartBeatResultAction: State(CONNECTINPROGRESS:LAST SNAPSHOT:CFGID-32:PEND-0:INITCFGID:0) FD=27:Cannot heartbeat with the master.
Oct 11 12:34:37  cfgm[3279]: <399816> <3279> <ERRS> |cfgm|  handle_read: State(READY:LAST SNAPSHOT:CFGID-32:PEND-0:INITCFGID:0) FD=27:Failure receiving heartbeat response header information Result=-1 Err=Connection timed out
Oct 11 12:34:37  cfgm[3279]: <399838> <3279> <WARN> |cfgm|  LmsHeartBeatResultAction: State(READY:LAST SNAPSHOT:CFGID-32:PEND-0:INITCFGID:0) FD=27:Cannot heartbeat with the master.
Oct 11 12:34:46  cfgm[3279]: <399838> <3279> <WARN> |cfgm|  LmsHeartBeatResultAction: State(CONNECTINPROGRESS:LAST SNAPSHOT:CFGID-32:PEND-0:INITCFGID:0) FD=27:Cannot heartbeat with the master.
Oct 11 12:34:49  ofa: <310202> <3725> <ERRS> |ofa|  ARP/ICMPv6 flows are not installed
Oct 11 12:34:56  cfgm[3279]: <399838> <3279> <WARN> |cfgm|  LmsHeartBeatResultAction: State(CONNECTINPROGRESS:LAST SNAPSHOT:CFGID-32:PEND-0:INITCFGID:0) FD=27:Cannot heartbeat with the master.
Oct 11 12:34:59  ofa: <310202> <3725> <ERRS> |ofa|  ARP/ICMPv6 flows are not installed
Oct 11 12:35:06  cfgm[3279]: <399838> <3279> <WARN> |cfgm|  LmsHeartBeatResultAction: State(CONNECTINPROGRESS:LAST SNAPSHOT:CFGID-32:PEND-0:INITCFGID:0) FD=27:Cannot heartbeat with the master.
Oct 11 12:35:07  cfgm[3279]: <399816> <3279> <ERRS> |cfgm|  handle_read: State(READY:LAST SNAPSHOT:CFGID-32:PEND-0:INITCFGID:0) FD=27:Failure receiving heartbeat response header information Result=-1 Err=Connection timed out
Oct 11 12:35:07  cfgm[3279]: <399838> <3279> <WARN> |cfgm|  LmsHeartBeatResultAction: State(READY:LAST SNAPSHOT:CFGID-32:PEND-0:INITCFGID:0) FD=27:Cannot heartbeat with the master.
Oct 11 12:35:07  ofa: <310202> <3725> <ERRS> |ofa|  ARP/ICMPv6 flows are not installed
Oct 11 12:35:16  cfgm[3279]: <399838> <3279> <WARN> |cfgm|  LmsHeartBeatResultAction: State(CONNECTINPROGRESS:LAST SNAPSHOT:CFGID-32:PEND-0:INITCFGID:0) FD=27:Cannot heartbeat with the master.
Oct 11 12:35:17  ofa: <310202> <3725> <ERRS> |ofa|  ARP/ICMPv6 flows are not installed
Oct 11 12:35:26  cfgm[3279]: <399838> <3279> <WARN> |cfgm|  LmsHeartBeatResultAction: State(CONNECTINPROGRESS:LAST SNAPSHOT:CFGID-32:PEND-0:INITCFGID:0) FD=27:Cannot heartbeat with the master.
Oct 11 12:35:37  cfgm[3279]: <399816> <3279> <ERRS> |cfgm|  handle_read: State(READY:LAST SNAPSHOT:CFGID-32:PEND-0:INITCFGID:0) FD=27:Failure receiving heartbeat response header information Result=-1 Err=Connection timed out
Oct 11 12:35:37  cfgm[3279]: <399838> <3279> <WARN> |cfgm|  LmsHeartBeatResultAction: State(READY:LAST SNAPSHOT:CFGID-32:PEND-0:INITCFGID:0) FD=27:Cannot heartbeat with the master.
Oct 11 12:35:46  cfgm[3279]: <399838> <3279> <WARN> |cfgm|  LmsHeartBeatResultAction: State(CONNECTINPROGRESS:LAST SNAPSHOT:CFGID-32:PEND-0:INITCFGID:0) FD=27:Cannot heartbeat with the master.

VMM and MC are on the same VLAN(subnet)

Maybe someone had the same problem?

I  also add logical schema part with VMC and MC, and licencing info

[dojjan]

Hello,

Can you see the IPSEC trying to build with "show datapath session table" on both sides? You should see entries using UDP 4500.

Also is "show crypto ipsec sa" empty?

I actually had this problem today (not able to establish IPSEC), i re-wrote the PSK for the IPSEC in the MM and then it worked. Longshot for you though.

[Audrius]

I will add more debug info from VMM and MC, it seem like something is wrong with licences, but I do not understand what, because VMM is saying

Oct 11 19:24:12  licensemgr[4380]: <300199> <4380> <ERRS> |licensemgr|  __license_expire: Invalid license key

I also tried to change IPSec key from VMM side, but did not help me,  I would also change IPSec key on MC side but I can not access it physicaly, because it is at client datacenter.

Is  there are any other possibilities to change MC IPSec key remotely?

Still waiting for possible answers from you all :)

[Victor Fabian]

Your issue is not the IPSec key rather it looks like your license expired and the MC still pointed to use the expired license instead of the new one.

Go to the MM > System > Licensing and adjust the MM license assignment to reflect the new license

Sent from Mail for Windows 10

[Audrius]

I check VMM licences  and found strange thing, that VMM GUI is showing only enabled licences and CLI show all licences expired+enabled

Do I need to delete old licences?

Because I tried to reinstall new licences and it did not help me?

One more strange thing that expired licence installation time is 1970's.

Why old licences are still there? How to point VMM to new licences?