Wired Intelligent Edge

 View Only
last person joined: 15 hours ago 

Bring performance and reliability to your network with the HPE Aruba Networking Core, Aggregation, and Access layer switches. Discuss the latest features and functionality of your switching devices, and find ways to improve security across your network to bring together a mobile-first solution
Back to discussions
Expand all | Collapse all

hpe-snmpd crashed on Aruba 6100 48G with ARUBAOS-CX 10.07.0010

This thread has been viewed 50 times

  • 1.  hpe-snmpd crashed on Aruba 6100 48G with ARUBAOS-CX 10.07.0010

    Posted Jul 29, 2021 11:46 AM
    Hi All,

    I've been dealing with setting up the SNMP service for our monitoring system but having this error message:

    2021-07-27T09:42:20.880617+0200 systemd-coredump[6262] <CRIT> Event|1201|LOG_CRIT|AMM|-|hpe-snmpd crashed due to signal:11

    2021-07-27T09:43:19.789453+0200 systemd-coredump[6332] <CRIT> Event|1201|LOG_CRIT|AMM|-|hpe-snmpd crashed due to signal:11

    2021-07-27T09:44:19.796674+0200 systemd-coredump[6405] <CRIT> Event|1201|LOG_CRIT|AMM|-|hpe-snmpd crashed due to signal:11

    2021-07-27T09:45:19.657429+0200 systemd-coredump[6476] <CRIT> Event|1201|LOG_CRIT|AMM|-|hpe-snmpd crashed due to signal:11


    Our monitoring system managed to reach the switch, however, connection is lost in ~1min after sync. So, it had worked for 1 min until the crash info did not pop up.  I tried to reboot the sw but I am still seeing this problem.

    For me, the log message says that the snmp daemon stopped working for some reason. I have checked the release note of ver 10.07.0010 but such a problem like this is not mentioned in the document.

    sw# show core-dump
    ==================================================================================================
    Daemon Name | Instance ID | Present | Timestamp | Build ID
    ==================================================================================================
    hpe-snmpd 6339 Yes 2021-07-27 09:44:19 c075dcd
    hpe-snmpd 6266 Yes 2021-07-27 09:43:19 c075dcd
    hpe-snmpd 6233 Yes 2021-07-27 09:42:20 c075dcd
    hpe-snmpd 5970 Yes 2021-07-27 09:39:20 c075dcd
    hpe-snmpd 5922 Yes 2021-07-27 09:38:48 c075dcd
    hpe-snmpd 4005 Yes 2021-07-27 09:09:40 c075dcd
    hpe-snmpd 3952 Yes 2021-07-27 09:09:05 c075dcd
    hpe-snmpd 3833 Yes 2021-07-27 09:08:25 c075dcd
    hpe-snmpd 3763 Yes 2021-07-27 09:07:26 c075dcd
    hpe-snmpd 3706 Yes 2021-07-27 09:06:26 c075dcd
    ==================================================================================================
    Total number of core dumps : 10
    ==================================================================================================


    Is it a bug or I missed something? Even if I missed something, I think it is not normal that snmpd does crash after reboot.

    Have you guys seen such a problem like this ?
    Thanks a million for your help and support on this issue.

    Additional info:

    -There is no ACL applied under snmp community.
    -Community public has been deleted.
    -SNMP v2c is being used

    sw# sh version
    -----------------------------------------------------------------------------
    ArubaOS-CX
    (c) Copyright 2017-2021 Hewlett Packard Enterprise Development LP
    -----------------------------------------------------------------------------
    Version : PL.10.07.0010
    Build Date : 2021-06-10 00:38:14 UTC
    Build ID : ArubaOS-CX:PL.10.07.0010:c075dcdbb1f5:202106100007




    ------------------------------
    Gábor Fejér
    ------------------------------


  • 2.  RE: hpe-snmpd crashed on Aruba 6100 48G with ARUBAOS-CX 10.07.0010

    EMPLOYEE
    Posted Jul 29, 2021 12:24 PM
    Best route is to flag a TAC case on this.

    ------------------------------
    Kamal Takodra
    If my post was useful accept solution and/or give kudos
    ------------------------------

    Original Message


  • 3.  RE: hpe-snmpd crashed on Aruba 6100 48G with ARUBAOS-CX 10.07.0010

    Posted Aug 05, 2021 03:23 AM
    Thank you for the advice.
    Yes, I raised a ticket yesterday. 
    Will share some info as soos as I get feedback from the support team.


    ​

    ------------------------------
    Gábor Fejér
    ------------------------------

    Original Message


  • 4.  RE: hpe-snmpd crashed on Aruba 6100 48G with ARUBAOS-CX 10.07.0010

    MVP GURU
    Posted Aug 06, 2021 03:18 PM
    Hello Gabor, have a look at ArubaOS-CX 10.07.0020 released today, it seems there something related to hpe-snmpd crash that was fixed.

    ------------------------------
    Davide Poletto
    ------------------------------

    Original Message


  • 5.  RE: hpe-snmpd crashed on Aruba 6100 48G with ARUBAOS-CX 10.07.0010

    Posted Aug 09, 2021 03:16 AM
    Hello Davide!

    Thank you for the heads-up on this issue. Yes, I've upgraded the firmware to .0020 this morning, so far, so good. It seems the problem has been fixed, however, I will be monitoring the device for a couple of days.

    ------------------------------
    Gábor Fejér
    ------------------------------

    Original Message


  • 6.  RE: hpe-snmpd crashed on Aruba 6100 48G with ARUBAOS-CX 10.07.0010

    Posted Jan 24, 2022 10:42 AM

    Hi,

    does anyone knows how to restart the hpe-snmpd only?

    Because now on one of my 8325 Core Switches the hpe-snmpd seems to have crashed:

    2022-01-24T15:59:31.192111+01:00 csw-rz-r08 systemd-coredump[1772919]: Event|1201|LOG_CRIT|AMM|-|hpe-snmpd crashed due to signal:11
    2022-01-24T15:59:28.402821+01:00 csw-rz-r08 systemd-coredump[1772898]: Event|1201|LOG_CRIT|AMM|-|hpe-snmpd crashed due to signal:11
    2022-01-24T15:59:25.610190+01:00 csw-rz-r08 systemd-coredump[1772880]: Event|1201|LOG_CRIT|AMM|-|hpe-snmpd crashed due to signal:11
    2022-01-24T15:59:22.823105+01:00 csw-rz-r08 systemd-coredump[1772862]: Event|1201|LOG_CRIT|AMM|-|hpe-snmpd crashed due to signal:11
    2022-01-24T15:59:20.032118+01:00 csw-rz-r08 systemd-coredump[1772851]: Event|1201|LOG_CRIT|AMM|-|hpe-snmpd crashed due to signal:11
    2022-01-24T15:59:17.210880+01:00 csw-rz-r08 systemd-coredump[1772832]: Event|1201|LOG_CRIT|AMM|-|hpe-snmpd crashed due to signal:11

    And our SNMP Monitoring Tool (PRTG) does not receive snmp "replies", so snmp really does not work...

    System is a 8325 and Software Version is > 10.07.0020

    csw-rz-r08# sh system
    Vendor : Aruba
    Product Name : JL635A Aruba 8325-48Y8C 48p 25G 8p 100G Swch
    ArubaOS-CX Version : GL.10.07.0041

    SNMP Deamon crashed wehen I have started a inventory and topology scan from our docusnap server.

    Thanks and rind regards

    Robert



    ------------------------------
    Robert Großmann
    ------------------------------

    Original Message


  • 7.  RE: hpe-snmpd crashed on Aruba 6100 48G with ARUBAOS-CX 10.07.0010

    Posted Jan 24, 2022 01:29 PM
    You may be able to restart the service from the shell.
    switch@ start-shell
switch:~$ sytemctl | grep snmp
snmp-alarmd.service      loaded active runnning Snmp Alarm Daemon
snmpd-wrapper-in-c.service loaded active running snmpd-wrapper

​
    Not sure if one of those would be the needed service, would recommend trying it on a test switch first. 

    I ran the following on my switch and didn't have any issues with crashing:

    sudo systemctl restart snmp-alarmd.service
    sudo systemctl restart snmpd-wrapper-in-c.service

    ------------------------------
    Kyle Higgins
    ------------------------------

    Original Message


  • 8.  RE: hpe-snmpd crashed on Aruba 6100 48G with ARUBAOS-CX 10.07.0010

    MVP GURU
    Posted Jan 24, 2022 02:30 PM
    Hi Robert, why not to plan a move from AOS-CX 10.07.0020 (August 2021) to AOS-CX 10.07.0050 (December 2021) which is the latest build released for AOS-CX 10.07 software line and then - eventually - try another jump to the latest build of ArubaOS-CX 10.09 software line (AOS-CX 10.07 to 10.09 is a permitted step without necessarily passing through a AOS-CX 10.08 build).

    I haven't checked yet but, maybe, it is a software bug already fixed on newer 10.07 (newer than build 0020) or on newer AOS-CX 10.08/10.09 software lines (latest builds).

    Have you checked on relevant Release Notes?

    ------------------------------
    Davide Poletto
    ------------------------------

    Original Message


  • 9.  RE: hpe-snmpd crashed on Aruba 6100 48G with ARUBAOS-CX 10.07.0010

    Posted Jan 25, 2022 02:52 AM

    @Kyle: The command systemctl | grep does not work for me.
    What I have done yesterday are these commands:

    csw-rz-r08# start-shell

csw-rz-r08:~$ sytemctl | grep snmp
bash: sytemctl: command not found
csw-rz-r08:~$

csw-rz-r08:~$ ps -aux | grep snmp
root        1381  0.0  0.0  50452 10620 ?        Ss    2021  10:30 /usr/bin/snmp-alarmd --detach --pidfile
root        8519  0.0  0.1 496408 15940 ?        Ssl   2021   6:07 /usr/bin/snmpd_wrapper --pidfile -vSYSLOG:INFO
root     1778291  0.1  0.2 157508 43124 ?        Sl   Jan24   1:41 /usr/bin/hpe-snmpd -x /var/agentx/master_VRF_1 -p /var/run/openvswitch/hpe-snmpd-VRF_1.pid
root     1778301  0.0  0.0  26080  6168 ?        S    Jan24   0:02 /usr/sbin/snmpd -Lo0-6d -f -x /var/agentx/master_VRF_1 -p /var/run/openvswitch/snmpd-VRF_1.pid
root     1778310  0.0  0.0  25476  5712 ?        S    Jan24   0:00 /usr/sbin/snmptrapd -f -C -c /var/net-snmp/snmptrapd-VRF_1.conf -Lf /var/log/snmptrapd-VRF_1.log -Dsnmptrapd -p/var/run/op
root     1778318  0.2  0.2 157520 46268 ?        Sl   Jan24   2:15 /usr/bin/hpe-snmpd -x /var/agentx/master_VRF_3 -p /var/run/openvswitch/hpe-snmpd-VRF_3.pid
root     1778328  0.0  0.0  26264  6680 ?        S    Jan24   0:38 /usr/sbin/snmpd -Lo0-6d -f -x /var/agentx/master_VRF_3 -p /var/run/openvswitch/snmpd-VRF_3.pid
root     1778346  0.0  0.0  25476  5752 ?        S    Jan24   0:01 /usr/sbin/snmptrapd -f -C -c /var/net-snmp/snmptrapd-VRF_3.conf -Lf /var/log/snmptrapd-VRF_3.log -Dsnmptrapd -p/var/run/op
root     1778977  0.0  0.0  25684  5852 ?        Ss   Jan24   0:02 /usr/sbin/snmpd -LS0-6d -f
manager  1904472  0.0  0.0   3068   824 pts/1    S+   08:33   0:00 grep snmp
​
csw-rz-r08:~$ service restart snmpd
bash: service: command not found

csw-rz-r08:~$ sudo systemctl restart snmpd

csw-rz-r08:~$ sudo systemctl restart hpe-snmpd
Failed to restart hpe-snmpd.service: Unit hpe-snmpd.service not found.

csw-rz-r08:~$


    Davide:​​​
    All the switches still are on software version 10.07.0041. They are frozen in this state, as we have other problems with the aruba cx 8325 switches: Aruba-CX VSX ISL Link: native VLAN 1 is not tagged, is it valid to use another VLAN as native? | Wired Intelligent Edge (arubanetworks.com)

    ​We are waiting on feedback/solution from Aruba in TAC Case 5361150231 and 5361329337

    But the general question is, how can I (re)start switch services without booting the system? Has is to be done in shell or can it be done in the diagnostic mode?

    For my special case with snmp my solution was as following:

    csw-rz-r08# conf t
csw-rz-r08(config)# snmp-server agent-port 1161
csw-rz-r08(config)# snmp-server agent-port 161
csw-rz-r08(config)# end
csw-rz-r08# write mem


    ------------------------------
    Robert Großmann
    ------------------------------

    Original Message


  • 10.  RE: hpe-snmpd crashed on Aruba 6100 48G with ARUBAOS-CX 10.07.0010

    MVP GURU
    Posted Jan 25, 2022 07:08 AM
    On a 8320 running on AOS-CX 10.07.0050 these are the only services - both enabled, loaded, active and running - related to SNMP I was able to find:

    snmp-alarmd.service (Snmp Alarm Daemon)
    snmpd-wrapper-in-c.service (snmpd-wrapper)

    exactly as reported above by Kyle Higgins.

    I'm not sure if restarting the latter service (with sudo systemctl restart snmpd-wrapper-in-c.service) would really be of help to solve your issue but it is easy to understand that it is tied to hpe-snmpd binary (check with sudo systemctl status snmpd-wrapper-in-c.service).

    OTOH, generally speaking, user should not be forced to go that deep (I mean, into AOS-CX shell).




    ------------------------------
    Davide Poletto
    ------------------------------

    Original Message