Wired Intelligent Edge

Hi all,

I noticed a strange ephemeral (<300 ms) link state change on a VSX LAG's member interface and this event was apparently not involving (was not generated by) a change on interface status on partner side (a Server from which there is no evidence of such a link state transition from the standpoint of its LAN involved interfaces).

See, as example:

2019-04-28:21:37:52.991547|intfd|404|LOG_INFO|AMM|-|Link status for interface 1/1/19 is down
2019-04-28:21:37:52.999348|hpe-vsxd|7012|LOG_INFO|AMM|-|VSX 10 state local down, remote up
2019-04-28:21:37:53.005495|lacpd|1321|LOG_INFO|AMM|-|LAG 10 State change for interface 1/1/19: Actor state: ALFO, Partner state ALFNCD
2019-04-28:21:37:53.271224|lldpd|110|LOG_INFO|AMM|-|Configured LLDP reinit-delay to 2
2019-04-28:21:37:53.287067|intfd|403|LOG_INFO|AMM|-|Link status for interface 1/1/19 is up
2019-04-28:21:37:53.303273|lacpd|1313|LOG_INFO|AMM|-|LAG 10 set as VSX
2019-04-28:21:37:53.303774|lacpd|1321|LOG_INFO|AMM|-|LAG 10 State change for interface 1/1/19: Actor state: ALIO, Partner state PLIO
2019-04-28:21:37:53.307751|lacpd|1321|LOG_INFO|AMM|-|LAG 10 State change for interface 1/1/19: Actor state: ALFO, Partner state PLIO
2019-04-28:21:37:53.310138|lacpd|1321|LOG_INFO|AMM|-|LAG 10 State change for interface 1/1/19: Actor state: ALFO, Partner state PLFO
2019-04-28:21:37:53.312916|lacpd|1309|LOG_INFO|AMM|-|Partner is detected for interface 1/1/19 LAG 10 : 65534,00:00:00:00:00:00. Actor state: ALFOE, partner state PLFO
2019-04-28:21:37:53.312962|lacpd|1321|LOG_INFO|AMM|-|LAG 10 State change for interface 1/1/19: Actor state: ALFOE, Partner state PLFO
2019-04-28:21:37:53.315002|lacpd|1321|LOG_INFO|AMM|-|LAG 10 State change for interface 1/1/19: Actor state: ALFOE, Partner state PSFO
2019-04-28:21:37:53.317183|lacpd|1321|LOG_INFO|AMM|-|LAG 10 State change for interface 1/1/19: Actor state: ALFOX, Partner state PSFO
2019-04-28:21:37:53.319350|lacpd|1310|LOG_WARN|AMM|-|Partner is out of sync for interface 1/1/19 LAG sport: 11. Actor state: ALFOX, partner state PSFO
2019-04-28:21:37:54.313367|lacpd|1309|LOG_INFO|AMM|-|Partner is detected for interface 1/1/19 LAG 10 : 32768,00:14:5e:99:54:8c. Actor state: ALFOX, partner state ALFNC
2019-04-28:21:37:54.313411|lacpd|1321|LOG_INFO|AMM|-|LAG 10 State change for interface 1/1/19: Actor state: ALFOX, Partner state ALFNC
2019-04-28:21:37:54.319045|lacpd|1321|LOG_INFO|AMM|-|LAG 10 State change for interface 1/1/19: Actor state: ALFO, Partner state ALFNC
2019-04-28:21:37:55.344468|lacpd|1321|LOG_INFO|AMM|-|LAG 10 State change for interface 1/1/19: Actor state: ALFNCD, Partner state ALFNC
2019-04-28:21:37:55.349362|hpe-vsxd|7013|LOG_INFO|AMM|-|VSX 10 state local up, remote up
2019-04-28:21:37:55.353634|hpe-vsxd|7013|LOG_INFO|AMM|-|VSX 10 state local up, remote up
2019-04-28:21:37:56.311533|lacpd|1321|LOG_INFO|AMM|-|LAG 10 State change for interface 1/1/19: Actor state: ALFNCD, Partner state ALFNCD

As you can see:

2019-04-28:21:37:52.991547|intfd|404|LOG_INFO|AMM|-|Link status for interface 1/1/19 is down
2019-04-28:21:37:52.999348|hpe-vsxd|7012|LOG_INFO|AMM|-|VSX 10 state local down, remote up
2019-04-28:21:37:53.287067|intfd|403|LOG_INFO|AMM|-|Link status for interface 1/1/19 is up
2019-04-28:21:37:55.349362|hpe-vsxd|7013|LOG_INFO|AMM|-|VSX 10 state local up, remote up
2019-04-28:21:37:55.353634|hpe-vsxd|7013|LOG_INFO|AMM|-|VSX 10 state local up, remote up

The link down/up duration is below 300 ms and the VSX LAG 10 (lag 10) local down - remote up -> local up - remote up transition lasts about 2300 ms...

I'm a little bit puzzled. DOM and Transceiver statuses look good (within normal operating thresholds). Any idea?

Which version please.

Sorry Vincent, VSX is (still) actually running on ArubaOS-CX 10.02.0010.

Now that you asked me, looking at latest ArubaOS-CX 10.02.0031...what happened seems similar (at least on the "briefly drops connectivity" part) to the VSX related CR 49260 bug recently solved:

VSX CR_49260

Symptom: A VSX pair briefly drops connectivity on a multichassis LAG link.

Scenario: The VSX switch may briefly experience some packet drops after a downstream multichassis link to the primary VSX switch is recovered.

The only condition I don't recognize in my case is the "...after a downstream multichassis link to the primary VSX switch is recovered." part...I'm not sure it's our case.

Is there a posibility to upgrade to .0031?

(vsx update-sotware)

Yes, I'm going to schedule it for the end of this week (the delay is due to the fact that, every time the VSX is upgraded, there are necessarily various alarm acknowledgement and error logs cleaning actions to perform on all of our connected peer systems...actions needed to just reset to "green" our monitoring system, it's a time consuming manual operation although trivial...OTOH I try to stage software updates at best and not to apply each release every time it is publicly available).

I upgraded our VSX to ArubaOS-CX 10.02.0031 last Friday afternoon.

I'm still seeing events like the one initially described.

See this one (on VSX Secondary):

2019-05-06:07:50:27.285278|intfd|404|LOG_INFO|AMM|-|Link status for interface 1/1/26 is down
2019-05-06:07:50:27.299506|lacpd|1321|LOG_INFO|AMM|-|LAG 15 State change for interface 1/1/26: Actor state: ALFO, Partner state ALFNCD
2019-05-06:07:50:27.574298|lldpd|110|LOG_INFO|AMM|-|Configured LLDP reinit-delay to 2
2019-05-06:07:50:27.590662|intfd|403|LOG_INFO|AMM|-|Link status for interface 1/1/26 is up
2019-05-06:07:50:27.610540|lacpd|1313|LOG_INFO|AMM|-|LAG 15 set as VSX 
2019-05-06:07:50:27.611397|lacpd|1321|LOG_INFO|AMM|-|LAG 15 State change for interface 1/1/26: Actor state: ALIO, Partner state PLIO
2019-05-06:07:50:27.613323|lacpd|1321|LOG_INFO|AMM|-|LAG 15 State change for interface 1/1/26: Actor state: ALFO, Partner state PLIO
2019-05-06:07:50:27.618955|lacpd|1321|LOG_INFO|AMM|-|LAG 15 State change for interface 1/1/26: Actor state: ALFO, Partner state PLFO
2019-05-06:07:50:27.626911|lacpd|1309|LOG_INFO|AMM|-|Partner is detected for interface 1/1/26 LAG 15 : 65534,00:00:00:00:00:00. Actor state: ALFOE, partner state PLFO 
2019-05-06:07:50:27.626993|lacpd|1321|LOG_INFO|AMM|-|LAG 15 State change for interface 1/1/26: Actor state: ALFOE, Partner state PLFO
2019-05-06:07:50:27.630100|lacpd|1321|LOG_INFO|AMM|-|LAG 15 State change for interface 1/1/26: Actor state: ALFOE, Partner state PSFO
2019-05-06:07:50:27.633237|lacpd|1321|LOG_INFO|AMM|-|LAG 15 State change for interface 1/1/26: Actor state: ALFOX, Partner state PSFO
2019-05-06:07:50:27.636732|lacpd|1310|LOG_WARN|AMM|-|Partner is out of sync for interface 1/1/26 LAG sport: 13. Actor state: ALFOX, partner state PSFO 
2019-05-06:07:50:28.408864|lacpd|1309|LOG_INFO|AMM|-|Partner is detected for interface 1/1/26 LAG 15 : 32768,98:be:94:78:2a:e0. Actor state: ALFOX, partner state ALFNC 
2019-05-06:07:50:28.408949|lacpd|1321|LOG_INFO|AMM|-|LAG 15 State change for interface 1/1/26: Actor state: ALFOX, Partner state ALFNC
2019-05-06:07:50:28.414475|lacpd|1321|LOG_INFO|AMM|-|LAG 15 State change for interface 1/1/26: Actor state: ALFO, Partner state ALFNC
2019-05-06:07:50:30.349777|lacpd|1321|LOG_INFO|AMM|-|LAG 15 State change for interface 1/1/26: Actor state: ALFNCD, Partner state ALFNC
2019-05-06:07:50:30.406915|lacpd|1321|LOG_INFO|AMM|-|LAG 15 State change for interface 1/1/26: Actor state: ALFNCD, Partner state ALFNCD

Note how fast was the interface 1/1/26 - Down->Up - link status transition: slightly more than 300 ms.

No errors were logged on physical system connected to that port.

Hi Davide,

I already reported your issue to test engineering, but an official case to support would help. Would you mind opening it ?

Regards,

Vincent

Yes, sure. Will update with the case number here once opened. Thank you.

Edit: case 5338383679 opened.

The issue was on interface 1/1/19 and now it is on 1/1/26.

Can you share:

show int 1/1/26 transceiver detail

Did you try changing fiber, transceiver ?

It seems L1 issue as port is getting down.

On the server side you might not see such brief transition state.

---

@vincent.giles wrote:

The issue was on interface 1/1/19 and now it is on 1/1/26.

Exactly, it happens randomly: few days ago it involved interface 1/1/19 (when we still were at ArubaOS-CX 10.02.0010) then it involved interface 1/1/26 (after the update to ArubaOS-CX 10.02.0031).

---

@vincent.giles wrote: Can you share:

show int 1/1/26 transceiver detail

Yes, here it is (I'm currently retrieving data requested by Aruba Support too) with some more info:

Aruba-8320-2# show interface 1/1/26 dom
----------------------------------------------------------------------------------------------------------------
Port      Type        Channel#  Temperature     Voltage         Tx Bias         Rx Power        Tx Power        
                                (Celsius)       (Volts)         (mA)            (mW/dBm)        (mW/dBm)        
----------------------------------------------------------------------------------------------------------------
1/1/26    SFP+SR                32.61           3.39            8.84            0.53, -2.76     0.55, -2.60     
              
Aruba-8320-2# show interface 1/1/26 transceiver        
------------------------------------------------------------
Port      Type        Product   Serial          Part        
                      Number    Number          Number      
------------------------------------------------------------
1/1/26    SFP+SR      J9150D    MY84xxxxxx      1990-4175   

Aruba-8320-2# show interface 1/1/26 transceiver detail 
Transceiver in 1/1/26
 Interface Name      : 1/1/26
 Type                : SFP+SR      
 Connector Type      : LC
 Wavelength          : 850nm
 Transfer Distance   : 0.00km (SMF), 30m (OM1), 80m (OM2), 300m (OM3)
 Diagnostic Support  : DOM
 Product Number      : J9150D 
 Serial Number       : MY84xxxxxx (S/N sanitized, as above)      
 Part Number         : 1990-4175

 Status
  Temperature : 32.61C
  Voltage     : 3.39V
  Tx Bias     : 8.84mA
  Rx Power    : 0.53mW, -2.76dBm
  Tx Power    : 0.55mW, -2.60dBm

 Recent Alarms:

 Recent Errors:
 
Aruba-8320-2# show interface 1/1/26 transceiver threshold-violations          
-------------------------------------------------------------
Port      Type        Channel#  Types of Recent              
                                Threshold Violations         
-------------------------------------------------------------
1/1/26    SFP+SR                none

---

@vincent.giles wrote: Did you try changing fiber, transceiver ?

It seems L1 issue as port is getting down.

On the server side you might not see such brief transition state.

---

No, I didn't change anything physical (10 meters long OM3 quality Fiber Optic cable and/or Aruba J9150A SFP+ Transceiver) since that layer looks normal to me (Transceviers are all below thresholds and no errors can be found at a first sight).

Server side I'm quite sure I can say our IBM Power 9 servers are able to recognize and log every type of Ethernet adapter link down event (also one which could be so short [*])...no matter who caused it.

[*] side note: Am I wrong or 300/350 ms is an interface downtime duration comparable with downtime typically seen during VSX Update phases?

Hello, it looks like Aruba Development was able to reproduce what was reported in the case HPE 5338383679 into their Lab and it was able to recognize its root cause. There isn't a definitive date yet for a fix but, probably, that fix will be available for July. Thanks you all for support provided.

Follow Up:

Forgot to say that, at time, Aruba suggested to update to Aruba 10.03 and since the VSX update to ArubaOS-CX 10.03.0031 our VSX didn't suffered from this issue anymore...so the normal operations lasted months...but...

Bad news:

AFTER the update ArubaOS-CX 10.03.0031 to 10.03.0080 performed Friday, 7th February...our VSX started to show this behaviour (sub-second link down/up changes) again:

vsx-1:

2020-02-09T10:45:32.134838+01:00 vsx-1 intfd[1603]: Event|404|LOG_INFO|AMM|-|Link status for interface 1/1/13 is down
2020-02-09T10:45:32.486311+01:00 vsx-1 intfd[1603]: Event|403|LOG_INFO|AMM|-|Link status for interface 1/1/13 is up

2020-02-11T09:31:51.864853+01:00 vsx-1 intfd[1603]: Event|404|LOG_INFO|AMM|-|Link status for interface 1/1/25 is down
2020-02-11T09:31:52.219858+01:00 vsx-1 intfd[1603]: Event|403|LOG_INFO|AMM|-|Link status for interface 1/1/25 is up

vsx-2:

2020-02-11T21:10:22.504161+01:00 vsx-2 intfd[1593]: Event|404|LOG_INFO|AMM|-|Link status for interface 1/1/7 is down
2020-02-11T21:10:22.865073+01:00 vsx-2 intfd[1593]: Event|403|LOG_INFO|AMM|-|Link status for interface 1/1/7 is up

2020-02-11T22:09:19.151340+01:00 vsx-2 intfd[1593]: Event|404|LOG_INFO|AMM|-|Link status for interface 1/1/18 is down
2020-02-11T22:09:19.520512+01:00 vsx-2 intfd[1593]: Event|403|LOG_INFO|AMM|-|Link status for interface 1/1/18 is up

It looks like a regression...

Opened new Case 5345100256.

I'm little confused because, if initially link status was software controlled and thus we understood that the issue was software related (that happened on AOS-CX 10.02), then we solved by moving to a link status mechanism controlled by Hardware (thus the upgrade to AOS-CX 10.03) as Aruba Support explained us at time.

Now the question: why updating AOS-CX from 10.03.0031 to 10.03.0080 (within AOS-CX 10.03) caused such issue to appear again if link-status in 10.03 is not software controlled anymore?

That's incredible...I'm seeing ArubaOS-CX 10.03.0081 right now (it's today) and look at one of two issues solved:

"Physical Port

CR_71226

Symptom: The switch fails to notify a link flap event to the peer device.Scenario: In certain cases, the switch may fail to notify the peer device when a very brief interface link flap up/down occurs."

It seems really like mine...am I going to update?

Hello,
I currently encounter the same type of problem on 8320 in 10.03.0080 connected to 2930.

The bug CR_71226 relates to peer notification problems but does not explain the loss of connectivity between the 8320 and the remote equipment (the connection break is noted in the logs of our 2930).
No Ethernet error is recorded at the ports used for the interconnection between the 8320 and 2930.

Did you updated in 10.03.0081?

Regards

Nicolas.

Yes, I did the update to ArubaOS-CX 10.03.0081...but...it didn't fix the link status flapping as I initially thought.

Indeed initially, until at least the whole Friday 21 (so about 48 hours after the update done on Wednesday 19th), the link status flapping disappeared completely...but now you forced me to have a look again and guess what?

I found other Link Status flaps during the W/E (Saturday and Sunday)...on both nodes:

VSX-1 (Saturday):

2020-02-22T08:10:48.604219+01:00 vsx-1 intfd[1585]: Event|404|LOG_INFO|AMM|-|Link status for interface 1/1/3 is down
2020-02-22T08:10:48.929103+01:00 vsx-1 intfd[1585]: Event|403|LOG_INFO|AMM|-|Link status for interface 1/1/3 is up
2020-02-22T16:04:40.066768+01:00 vsx-1 intfd[1585]: Event|404|LOG_INFO|AMM|-|Link status for interface 1/1/3 is down
2020-02-22T16:04:40.394476+01:00 vsx-1 intfd[1585]: Event|403|LOG_INFO|AMM|-|Link status for interface 1/1/3 is up
2020-02-22T17:02:43.672763+01:00 vsx-1 intfd[1585]: Event|404|LOG_INFO|AMM|-|Link status for interface 1/1/12 is down
2020-02-22T17:02:43.985910+01:00 vsx-1 intfd[1585]: Event|403|LOG_INFO|AMM|-|Link status for interface 1/1/12 is up

VSX-2 (Sunday):

2020-02-23T19:30:55.140462+01:00 vsx-2 intfd[1575]: Event|404|LOG_INFO|AMM|-|Link status for interface 1/1/8 is down
2020-02-23T19:30:55.469621+01:00 vsx-2 intfd[1575]: Event|403|LOG_INFO|AMM|-|Link status for interface 1/1/8 is up

I'm now in touch with Aruba ERT who followed my first Case. Let me see what they will suggest.

Edit: A new case was opened (5345424795).

Hello,

  Thanks for the feedback, I will contact Aruba support to open a case too and will keep in touch.

Regards,

Nicolas.

Hello,

I made a conf-call with the support and they confirmed that our two problems were identical. They are working to deliver a fix in the next firmware version. My case number is 5345557816.

Regards,

Nicolas.

Thus we're all getting wet together, I suppose ;-). Luckily it's nothing really disrupting. Let we see the fix.

Hi Nicolas, Aruba just released ArubaOS-CX 10.03.0083 but, contrary to what I expected for a software release post build 0081, there is no reference of a fix for our issue. Do you have any news about that?

Hello Parnassus,

No news for the moment. I had a new remote session with Aruba support last wednesday. We checked once again some logs on the two 8320 and the remotely attached 2930F stacks via some LAG. We saw the 2 seconds link up/down flapping logs on a member of LAG on several links. We checked the transceivers and stats on the interfaces concerned : everything is OK. Now the problem is transfered to the Aruba Engineering Resolution Team (ERT) and I will have a new remote session in the beginning of next week.

By resuming the reading of your first messages, our respective problems are a little different:
- you do not see a trace of these state transitions on the remote equipment while I have the trace
- your state transitions last only 300ms maximum while those which concern me last 2 seconds.

I'll keep in touch,

Regards,
Nicolas.

Hello Nicolas, thanks for the update. My issue thus is slightly different than your one (Always, constantly, random link status flapping lasts about 350 milliseconds on VSX LAGs member interfaces and, yes, connected peers - hosts since I haven't any dual homed peer switches yet - are - luckily - not aware of the link status flapping).

Hi! a tiny follow up: it looks like Aruba is going to bake a new ArubaOS-CX 10.03 software release to (also) fix this issue. I'll keep you updated. 

Hi Parnassus! Thanks, we stay tuned.

Nicolas.

Follow-up: the recent ArubaOS-CX 10.03.0090 software release definitively solved the link status flapping issue our VSX was experiencing. Case closed.

Thank you for your patience, and your follow-up.

Hi,

After a week running 10.03.90, the 2 seconds LACP flapping problem did not occured so it can be concluded that this new release solved the problem as expected.

Thanks to Aruba support to have provided that patch.

Regards,

Nicolas

Glad it fixed your issue too!