Security

 View Only

  • 1.  Clearpass MABs occurring far too frequently

    Posted Aug 26, 2021 03:59 PM

    I've been seeing some issues with machines Machine Authenticating extremely frequently all over our site. We use Cisco switches (2960X/9200). I'm not seeing any failures in authentication, but in the Accounting records, I do see session terminating due to 'Lost-Carrier'. In checking the Cisco logs, the ports do seem to be going up and down, but cable diagnostics/replacement seem to eliminate that as a cause. It's also happening far too frequently for far too many systems for me to think that it's simply a bad cable.
    The question is - would a switch having trouble reaching the Clearpass cluster cause it to shut the port? I currently have a TAC case open with Cisco, but has anyone else encountered anything along these lines?

    Here's a snippet of the log that I'm seeing:

    Aug 25 10:49:52 10.126.58.41 EDT: dot1x-ev:[Gi1/0/14] Interface state changed to DOWN
    Aug 25 10:49:52 10.126.58.41 EDT: dot1x-ev:[Gi1/0/14] No DOT1X subblock found for port down
    Aug 25 10:49:53 10.126.58.41 EDT: AUTH-EVENT: [Gi1/0/14] Link DOWN
    Aug 25 10:49:53 10.126.58.41 EDT: AUTH-EVENT: [Gi1/0/14] Deleting clients - Link DOWN
    Aug 25 10:49:53 10.126.58.41 EDT: AUTH-EVENT: [Gi1/0/14] free all contexts
    Aug 25 10:49:54 10.126.58.41 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/14, changed state to down
    Aug 25 10:49:55 10.126.58.41 EDT: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/14, changed state to down
    Aug 25 10:49:56 10.126.58.41 EDT: AUTH-EVENT: [Gi1/0/14] Link UP
    Aug 25 10:49:56 10.126.58.41 EDT: dot1x-ev:[Gi1/0/14] Interface state changed to UP
    Aug 25 10:49:56 10.126.58.41 EDT: dot1x-ev:DOT1X Supplicant not enabled on GigabitEthernet1/0/14
    Aug 25 10:49:56 10.126.58.41 EDT: AUTH-EVENT: [Gi1/0/14] Link UP
    Aug 25 10:49:56 10.126.58.41 EDT: AUTH-EVENT: [Gi1/0/14] Link already UP - ignoring
    Aug 25 10:49:58 10.126.58.41 EDT: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/14, changed state to up
    Aug 25 10:49:58 10.126.58.41 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/14, changed state to up

    It really feels like dot1x is causing the port to be shut, which in turn causes the interface link to go down. 

    Here's the accounting record:



    ------------------------------
    Chris Zeigler
    ------------------------------


  • 2.  RE: Clearpass MABs occurring far too frequently

    Posted Aug 27, 2021 07:46 AM
    From the log, it appears the link between your device and switch is lost (lost-carrier). The authentication that happens when the port comes up again is just a result of that.

    Cabling would be the first for me to look at, but if you mention it is too many devices, having a look at the connected device and see if it is maybe all the same, or if they have logging that gives an indication if it is the device or the switch (but may be hard to decide). Are those devices PoE devices? I don't see PoE logs, so probably not.

    Simple elimination would be to bring one of the devices causing issues near your switch connected to a short patch-cable, and see if the issue is still there.

    Do you have CoA enabled? In that case you would see a CoA tab in access tracker. That also is the only way (that I can think of) that ClearPass would be actively disconnecting devices. But in that case I would also expect other log messages than 'Lost-Carrier'.

    ------------------------------
    Herman Robers
    ------------------------
    If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

    In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.
    ------------------------------

    Original Message


  • 3.  RE: Clearpass MABs occurring far too frequently

    Posted Aug 27, 2021 09:39 AM
    Hey Herman, 
    I had definitely thought cabling as well initially - I've replaced cables, run tdr tests, and had our Telecom group validate the cable. But again, since this is so many machines all over campus, I don't believe there's cabling issues. None of these devices are PoE - they've ranged from PC/Mac to printers, to card readers, so there doesn't seem to be any consistency for what is connected.

    We do have CoA enabled, but I'm not seeing a CoA tab in the access tracker. But it definitely does feel like that would be the reason things are being disconnected, I guess it's just the question of why that would be occurring.

    ------------------------------
    Chris Zeigler
    ------------------------------

    Original Message


  • 4.  RE: Clearpass MABs occurring far too frequently

    Posted Aug 27, 2021 10:13 AM
    Then chances are really small that ClearPass is sending out CoA, and not reporting it in Access Tracker. To be sure, you could enable logging on the switch, or use the 'Collect Logs' in ClearPass to run a packet capture for a few minutes, and check in there if you see CoA traffic going out, which is udp port 3799.

    If there was a reason, like CoA or SNMP, I would expect another message in the switch logs than 'Lost Carrier'. You may also trigger a CoA manually from ClearPass and check if the messages in the switch logs are the same as what you see. This issue seems unrelated to ClearPass given the information you shared.

    ------------------------------
    Herman Robers
    ------------------------
    If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

    In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.
    ------------------------------

    Original Message


  • 5.  RE: Clearpass MABs occurring far too frequently

    Posted Aug 27, 2021 10:46 AM
    That makes sense - I'll try and kick off a packet capture or two on the some of the Clearpass servers in the cluster to see what I can see, but I agree - it definitely seems like the issue does not lie in the Clearpass side. I just figured I'd try to eliminate all possibilities and gather as much information as possible to present to Cisco. Thanks again for the help Herman

    ------------------------------
    Chris Zeigler
    ------------------------------

    Original Message


Related Content