Developer

 View Only
last person joined: 2 hours ago 

Back to discussions
Expand all | Collapse all

ANSIBLE: Connection reset by remote peer (switches) in a "random" manner

This thread has been viewed 5 times

  • 1.  ANSIBLE: Connection reset by remote peer (switches) in a "random" manner

    Posted Oct 30, 2023 08:28 AM

    Summary

    When I run a playbook that sends commands on remote switches, sometimes some of the tasks fails on some switches, returning an error message "[Errno 104] Connection reset by peer".
    Some important facts:

    • Im using AWX
    • The playbook is scheduled to run every 10min
    • The same switch that fails on the previous playbook run can succeed on the next playbook run
    • the inventory has aprox. 30 switches, all located in the same subnet
    • sometimes, all the switches succeeds, sometimes, a couple switches fails with this message.
    • no specific switches failing, could be any them
    • Switches: J9773A 2530-24G-PoEP (YA.16.10.0015)

    Issue Type

    Bug Report

    Component Name

    arubanetworks.aos_switch.arubaoss_command

    Ansible Version

    AWX 21.13.0
ansible-core: 2.9
    <clipboard-copy aria-label="Copy" class="ClipboardButton btn js-clipboard-copy m-2 p-0 tooltipped-no-delay" data-copy-feedback="Copied!" data-tooltip-direction="w" value="AWX 21.13.0 ansible-core: 2.9" role="button" tabindex="0"></clipboard-copy>

    Configuration

    I'm using the default ansible configuration. No parameters were updated.

    OS / Environment

    RHEL 8

    Steps to Reproduce

    I'm putting one of the tasks from the playbook as an example, but it fails in different tasks in a random way, not on a specific task.

    - name: GET SHOW INT BRIEF OUTPUT TO PREVENT FALSE-POSITIVE
  arubanetworks.aos_switch.arubaoss_command:
    commands:
      - "show interface brief"
    wait_for: result[0] contains Status
  register: int_brief_output_prevent

- name: Print show interface brief output
  debug:
    msg: "{{ int_brief_output_prevent }}"
    <clipboard-copy aria-label="Copy" class="ClipboardButton btn js-clipboard-copy m-2 p-0 tooltipped-no-delay" data-copy-feedback="Copied!" data-tooltip-direction="w" value="- name: GET SHOW INT BRIEF OUTPUT TO PREVENT FALSE-POSITIVE arubanetworks.aos_switch.arubaoss_command: commands: - "show interface brief" wait_for: result[0] contains Status register: int_brief_output_prevent - name: Print show interface brief output debug: msg: "{{ int_brief_output_prevent }}"" role="button" tabindex="0"></clipboard-copy>

    Expected Results

    {
  "msg": {
    "changed": false,
    "stdout": [
      "Status and Counters - Port Status\n\n                   | Intrusion                           MDI  Flow\n  Port  Type       | Alert     Enabled Status Mode       Mode Ctrl\n  ----- ---------- + --------- ------- ------ ---------- ---- ----\n  1     100/1000T  | No        Yes     Up     1000FDx    MDI  off \n  2     100/1000T  | No        Yes     Up     10HDx      MDI  off \n  3     100/1000T  | Yes       Yes     Up     10HDx      MDI  off \n  4     100/1000T  | No        Yes     Down   1000FDx    MDI  off \n  5     100/1000T  | No        Yes     Down   1000FDx    MDIX off \n  6     100/1000T  | No        Yes     Down   1000FDx    MDI  off \n  7     100/1000T  | No        Yes     Down   1000FDx    MDI  off \n  8     100/1000T  | No        Yes     Down   1000FDx    MDI  off \n  9     100/1000T  | No        Yes     Up     10HDx      MDI  off \n  10    100/1000T  | No        Yes     Up     10HDx      MDI  off \n  11    100/1000T  | No        Yes     Down   1000FDx    MDI  off \n  12    100/1000T  | No        Yes     Down   1000FDx    MDI  off \n  13    100/1000T  | No        Yes     Up     100FDx     MDIX off \n  14    100/1000T  | No        Yes     Down   1000FDx    MDIX off \n  15    100/1000T  | No        Yes     Down   1000FDx    MDI  off \n  16    100/1000T  | No        Yes     Up     10HDx      MDIX off \n  17    100/1000T  | No        Yes     Down   1000FDx    MDI  off \n  18    100/1000T  | No        Yes     Down   1000FDx    MDI  off \n  19    100/1000T  | No        Yes     Down   1000FDx    MDIX off \n  20    100/1000T  | No        Yes     Down   1000FDx    MDI  off \n  21    100/1000T  | No        Yes     Down   1000FDx    MDI  off \n  22    100/1000T  | No        Yes     Down   1000FDx    MDIX off \n  23    100/1000T  | No        Yes     Down   1000FDx    MDI  off \n  24    100/1000T  | No        Yes     Down   1000FDx    MDIX off \n  25    1000LX     | No        Yes     Up     1000FDx    NA   off \n  26               | No        Yes     Down   .               off \n  27               | No        Yes     Down   .               off \n  28               | No        Yes     Down   .               off"
    ],
    "stdout_lines": [
      [
        "Status and Counters - Port Status",
        "",
        "                   | Intrusion                           MDI  Flow",
        "  Port  Type       | Alert     Enabled Status Mode       Mode Ctrl",
        "  ----- ---------- + --------- ------- ------ ---------- ---- ----",
        "  1     100/1000T  | No        Yes     Up     1000FDx    MDI  off ",
        "  2     100/1000T  | No        Yes     Up     10HDx      MDI  off ",
        "  3     100/1000T  | Yes       Yes     Up     10HDx      MDI  off ",
        "  4     100/1000T  | No        Yes     Down   1000FDx    MDI  off ",
        "  5     100/1000T  | No        Yes     Down   1000FDx    MDIX off ",
        "  6     100/1000T  | No        Yes     Down   1000FDx    MDI  off ",
        "  7     100/1000T  | No        Yes     Down   1000FDx    MDI  off ",
        "  8     100/1000T  | No        Yes     Down   1000FDx    MDI  off ",
        "  9     100/1000T  | No        Yes     Up     10HDx      MDI  off ",
        "  10    100/1000T  | No        Yes     Up     10HDx      MDI  off ",
        "  11    100/1000T  | No        Yes     Down   1000FDx    MDI  off ",
        "  12    100/1000T  | No        Yes     Down   1000FDx    MDI  off ",
        "  13    100/1000T  | No        Yes     Up     100FDx     MDIX off ",
        "  14    100/1000T  | No        Yes     Down   1000FDx    MDIX off ",
        "  15    100/1000T  | No        Yes     Down   1000FDx    MDI  off ",
        "  16    100/1000T  | No        Yes     Up     10HDx      MDIX off ",
        "  17    100/1000T  | No        Yes     Down   1000FDx    MDI  off ",
        "  18    100/1000T  | No        Yes     Down   1000FDx    MDI  off ",
        "  19    100/1000T  | No        Yes     Down   1000FDx    MDIX off ",
        "  20    100/1000T  | No        Yes     Down   1000FDx    MDI  off ",
        "  21    100/1000T  | No        Yes     Down   1000FDx    MDI  off ",
        "  22    100/1000T  | No        Yes     Down   1000FDx    MDIX off ",
        "  23    100/1000T  | No        Yes     Down   1000FDx    MDI  off ",
        "  24    100/1000T  | No        Yes     Down   1000FDx    MDIX off ",
        "  25    1000LX     | No        Yes     Up     1000FDx    NA   off ",
        "  26               | No        Yes     Down   .               off ",
        "  27               | No        Yes     Down   .               off ",
        "  28               | No        Yes     Down   .               off"
      ]
    ],
    "ansible_facts": {
      "discovered_interpreter_python": "/usr/bin/python"
    },
    "deprecations": [
      {
        "msg": "Distribution rhel 8.7 on host SW-MAG040-ASW04-R02-ADM should use /usr/libexec/platform-python, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information",
        "version": "2.12"
      }
    ],
    "failed": false
  },
  "_ansible_verbose_always": true,
  "_ansible_no_log": false,
  "changed": false
}
    <clipboard-copy aria-label="Copy" class="ClipboardButton btn js-clipboard-copy m-2 p-0 tooltipped-no-delay" data-copy-feedback="Copied!" data-tooltip-direction="w" value="{ "msg": { "changed": false, "stdout": [ "Status and Counters - Port Status\n\n | Intrusion MDI Flow\n Port Type | Alert Enabled Status Mode Mode Ctrl\n ----- ---------- + --------- ------- ------ ---------- ---- ----\n 1 100/1000T | No Yes Up 1000FDx MDI off \n 2 100/1000T | No Yes Up 10HDx MDI off \n 3 100/1000T | Yes Yes Up 10HDx MDI off \n 4 100/1000T | No Yes Down 1000FDx MDI off \n 5 100/1000T | No Yes Down 1000FDx MDIX off \n 6 100/1000T | No Yes Down 1000FDx MDI off \n 7 100/1000T | No Yes Down 1000FDx MDI off \n 8 100/1000T | No Yes Down 1000FDx MDI off \n 9 100/1000T | No Yes Up 10HDx MDI off \n 10 100/1000T | No Yes Up 10HDx MDI off \n 11 100/1000T | No Yes Down 1000FDx MDI off \n 12 100/1000T | No Yes Down 1000FDx MDI off \n 13 100/1000T | No Yes Up 100FDx MDIX off \n 14 100/1000T | No Yes Down 1000FDx MDIX off \n 15 100/1000T | No Yes Down 1000FDx MDI off \n 16 100/1000T | No Yes Up 10HDx MDIX off \n 17 100/1000T | No Yes Down 1000FDx MDI off \n 18 100/1000T | No Yes Down 1000FDx MDI off \n 19 100/1000T | No Yes Down 1000FDx MDIX off \n 20 100/1000T | No Yes Down 1000FDx MDI off \n 21 100/1000T | No Yes Down 1000FDx MDI off \n 22 100/1000T | No Yes Down 1000FDx MDIX off \n 23 100/1000T | No Yes Down 1000FDx MDI off \n 24 100/1000T | No Yes Down 1000FDx MDIX off \n 25 1000LX | No Yes Up 1000FDx NA off \n 26 | No Yes Down . off \n 27 | No Yes Down . off \n 28 | No Yes Down . off" ], "stdout_lines": [ [ "Status and Counters - Port Status", "", " | Intrusion MDI Flow", " Port Type | Alert Enabled Status Mode Mode Ctrl", " ----- ---------- + --------- ------- ------ ---------- ---- ----", " 1 100/1000T | No Yes Up 1000FDx MDI off ", " 2 100/1000T | No Yes Up 10HDx MDI off ", " 3 100/1000T | Yes Yes Up 10HDx MDI off ", " 4 100/1000T | No Yes Down 1000FDx MDI off ", " 5 100/1000T | No Yes Down 1000FDx MDIX off ", " 6 100/1000T | No Yes Down 1000FDx MDI off ", " 7 100/1000T | No Yes Down 1000FDx MDI off ", " 8 100/1000T | No Yes Down 1000FDx MDI off ", " 9 100/1000T | No Yes Up 10HDx MDI off ", " 10 100/1000T | No Yes Up 10HDx MDI off ", " 11 100/1000T | No Yes Down 1000FDx MDI off ", " 12 100/1000T | No Yes Down 1000FDx MDI off ", " 13 100/1000T | No Yes Up 100FDx MDIX off ", " 14 100/1000T | No Yes Down 1000FDx MDIX off ", " 15 100/1000T | No Yes Down 1000FDx MDI off ", " 16 100/1000T | No Yes Up 10HDx MDIX off ", " 17 100/1000T | No Yes Down 1000FDx MDI off ", " 18 100/1000T | No Yes Down 1000FDx MDI off ", " 19 100/1000T | No Yes Down 1000FDx MDIX off ", " 20 100/1000T | No Yes Down 1000FDx MDI off ", " 21 100/1000T | No Yes Down 1000FDx MDI off ", " 22 100/1000T | No Yes Down 1000FDx MDIX off ", " 23 100/1000T | No Yes Down 1000FDx MDI off ", " 24 100/1000T | No Yes Down 1000FDx MDIX off ", " 25 1000LX | No Yes Up 1000FDx NA off ", " 26 | No Yes Down . off ", " 27 | No Yes Down . off ", " 28 | No Yes Down . off" ] ], "ansible_facts": { "discovered_interpreter_python": "/usr/bin/python" }, "deprecations": [ { "msg": "Distribution rhel 8.7 on host SW-MAG040-ASW04-R02-ADM should use /usr/libexec/platform-python, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information", "version": "2.12" } ], "failed": false }, "_ansible_verbose_always": true, "_ansible_no_log": false, "changed": false }" role="button" tabindex="0"></clipboard-copy>

    Actual Results

    {
  "msg": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.9/site-packages/ansible/module_utils/connection.py\", line 200, in send\n    response = recv_data(sf)\n  File \"/usr/local/lib/python3.9/site-packages/ansible/module_utils/connection.py\", line 76, in recv_data\n    d = s.recv(header_len - len(data))\nConnectionResetError: [Errno 104] Connection reset by peer\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/local/bin/ansible-connection\", line 310, in main\n    conn.set_options(var_options=variables)\n  File \"/usr/local/lib/python3.9/site-packages/ansible/module_utils/connection.py\", line 184, in __rpc__\n    response = self._exec_jsonrpc(name, *args, **kwargs)\n  File \"/usr/local/lib/python3.9/site-packages/ansible/module_utils/connection.py\", line 150, in _exec_jsonrpc\n    out = self.send(data)\n  File \"/usr/local/lib/python3.9/site-packages/ansible/module_utils/connection.py\", line 204, in send\n    raise ConnectionError('unable to connect to socket', err=to_text(e, errors='surrogate_then_replace'), exception=traceback.format_exc())\nansible.module_utils.connection.ConnectionError: unable to connect to socket\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/local/bin/ansible-connection\", line 350, in <module>\n    main()\n  File \"/usr/local/bin/ansible-connection\", line 313, in main\n    raise ConnectionError('Unable to decode JSON from response set_options. See the debug log for more information.')\nansible.module_utils.connection.ConnectionError: Unable to decode JSON from response set_options. See the debug log for more information.\n",
  "_ansible_no_log": false
}


  • 2.  RE: ANSIBLE: Connection reset by remote peer (switches) in a "random" manner

    Posted Nov 01, 2023 12:51 PM

    Hi @lhcunha!

    Since this is happening inconsistently it's a little difficult to troubleshoot but we can attempt a few things to try and improve the behavior:

    1) What version of the ansible.netcommon collection do you have installed? This is a dependency for the aos-switch collection and updating it could improve the behavior.

    2) Would you be willing to upgrade your ansible-core version? I believe 2.13 is the most recent

    3) Could you include an ansible.cfg file within your playbook directory that defines host_key_checking = false? This is described here: https://github.com/aruba/aos-switch-ansible-collection#sshcli-modules



    ------------------------------
    Ti Chiapuzio-Wong (they/them)
    HPE Aruba Networking
    ------------------------------

    Original Message


  • 3.  RE: ANSIBLE: Connection reset by remote peer (switches) in a "random" manner

    Posted Nov 07, 2023 09:32 AM

    Hi Tiffany, thanks for the answer.

    1) Its ansible.netcommon latest version

    2) I'm not able to update ansible core at this moment

    3) I will do this and update this thread later with the result


    Original Message