Wired Intelligent Edge

We had a very strange and concerning problem several weeks ago and HP support (level 2) is still stumped.  During the middle of the day we lost connection to several IDF closets.  When I logged into the switch and started looking around I noticed that most of the trunks in the running config had been modified.  I looked at the startup config and it had been modified as well.  I will attach that portion of the configs to show what I'm talking about as well as an exerpt from the Show tech all during this time.

The tech at 2nd level support told me that since I rebooted the switch all history was gone and he could not help find an answer... I got my case esclated to a second tech that has been looking into this for 3 weeks now.  I noticed that the show tech all had plenty of pre reboot entries that led up to this event.  

Anyway... I was in this switch creating a new trunk for a new facility when all of this happened.  I added a port (B3) to the new trunk group (static no protocol) when boom all of a sudden everything started crashing... Here is a copy of the config before the change and after the change.  I am the only one that has access to this switch and we have PCM policies disabled (no switch updating)  Does anyone have any idea why this change might have suddenly happened.  I could understand if I accidently removed one port from a trunk by a typo but I refuse to accept that I made this mistake 15 or 16 times then saved the config to startup... 

I have attached an exerpt fromt he show tech all that shows all of these ports reporting "off-line" then they come back online but no longer members of the trunk to which they were previoiusly assigned?????

Config BEFORE:

_________________________________________________________________________________

trunk D21,J21 Trk6 Trunk
trunk B1-B2,H1-H2 Trk10 Trunk
trunk D9-D10,J9-J10 Trk11 Trunk
trunk D11-D12,J11-J12 Trk12 Trunk
trunk A5,G5 Trk15 Trunk
trunk A6,G6 Trk16 Trunk
trunk A8,G8 Trk18 Trunk
trunk A9,G9 Trk19 Trunk
trunk A10,G10 Trk20 Trunk
trunk A11,G11 Trk21 Trunk
trunk A12,G12 Trk22 Trunk
trunk A13,G13 Trk23 Trunk
trunk A14,G14 Trk24 Trunk
trunk A15,G15 Trk25 Trunk
trunk F1,L1 Trk31 Trunk
trunk F2,L2 Trk32 Trunk
trunk F3,L3 Trk33 Trunk
trunk F4,L4 Trk34 Trunk
trunk F5,L5 Trk35 Trunk
trunk F6,L6 Trk36 Trunk
trunk F7,L7 Trk37 Trunk
trunk F8,L8 Trk38 Trunk
trunk B3,H3 Trk39 Trunk
trunk D23-D24,J24 Trk100 LACP

CONFIG AFTER:

__________________________________________________________________________________

trunk J21 Trk6 Trunk
trunk B1-B2 Trk10 Trunk
trunk J9-J10 Trk11 Trunk
trunk J11-J12 Trk12 Trunk
trunk A5 Trk15 Trunk
trunk A6 Trk16 Trunk
trunk A8 Trk18 Trunk
trunk A9 Trk19 Trunk
trunk A10 Trk20 Trunk
trunk A11 Trk21 Trunk
trunk A12 Trk22 Trunk
trunk A13 Trk23 Trunk
trunk A14 Trk24 Trunk
trunk A15 Trk25 Trunk
trunk L1 Trk31 Trunk
trunk L2 Trk32 Trunk
trunk L3 Trk33 Trunk
trunk L4 Trk34 Trunk
trunk L5 Trk35 Trunk
trunk L6 Trk36 Trunk
trunk L7 Trk37 Trunk
trunk L8 Trk38 Trunk
trunk J24 Trk100 LACP

Has anything similar ever happened to anyone?  I am new to HP Switches coming from Cisco and am acustom to Cisco support.  HP support thus far has not been very helpful...

Attachment(s)

After - 8212.txt   703 B 1 version

show tech all exerpt.txt   703 B 1 version

Very odd indeed.  Not that it should matter, but were you making these changes via command line or another way?  I've had E series switches completely flake out on my and lose part of the config, but it was repeatable after comitting ACL updates. It was addressed by a code update.  

Yeah I was making the changes vial command line.  I am thinking that this event had something to do with the fabric flaking out because all of the ports that were removed from the trunk groups were on the 2nd fabric.

I have since updated all of my 8212 and 5412 switches to 15_9_0012.  Which had wierd effects all it's own.  We had several modules that did not come back online after the reboot and there were no log events to notify us.  There was a big red alarm on the switch but no log event saying that "Module H" is down???  We had to walk to each IDF and physically look at each switch.  This happend on 4 different switches???

You should be able to see the logs from before the reboot if there was no power failure. Use show log -a (I believe, check online help) to see the message from before the reload.

Did you experience a power failure ? How many power supplies do you have ? Since the 12 slot requires 2 ps to come online, if 1 ps would fail, to last 6 modules are going down ... (so min 3 ps for redundancy!).

Thanks Peter

I was able to see everything that happened up to the failure.  I attached an excerpt from my log file during the time that ports started dropping.  From the log file you can see that ports just start dropping .. then when they came back online any ports from modules "D" "F" and "G" "H"were no longer members of their assigned trunk groups... then of course all chaos broke out from the loops that were created.  These ports came back online not configured for their respective trunks.  All I know is that I can't type that fast... all of this happened in less than a minute... Then SNMP saved the running config to startup... why I still don't know.  HP support has been looking through the PCM+ logs as well.  We have all policies turned off as I do not allow PCM+ write to the switches... (or so I thought)

We have 4 PS for this switch and  we did not experience a power failure.  I was in the switch creating a new trunk group for a new facility... notice the log entries where I was adding "B3" to trk39.. next thing I know the phone started ringing off the hook :)

Somehow, someway all ports from "D" "F" and "G", "H" got removed from their trunks...??? HP support has been researching this since Mar. 14th.... the search continues....

Hi,

I cannot give any additional input regarding the issue, however, to limit the impact of such a problem, it would be recommended to use LACP on the trunks (trunk a1,b1 trk1 lacp) on both sides of the link.

When one of the port is not properly configured on 1 side, the remote side would not get the correct LACP information anymore, and would apply a software block on the port, which would stop the broadcast storm at least ...

Hope this helps, Peter.