pythonpython-3.xnetmiko

Netmiko: Error message 'NoneType' object has no attribute 'recv_ready' when attempting to reconnect to a device after a reload


I want to run precheck commands, reload the device, wait for it to come back up, then reconnect and rerun the same commands. However, after the reload, I get the following error message when it's time to run the same commands again.

Notes: input.txt: contains the IPs of the devices to reboot. reload_cmds.txt: contains the list of commands sectioned per layer.

<...commands output cut for brevity...>

Would you like to proceed with reload?[Type yes or no]:  yes

========= Saving configuration =========

write mem

Building configuration...
Compressed configuration from 3524 bytes to 1809 bytes[OK]
SW1#

========= Reloading the switch =========


Reload scheduled in 1 minute by admin on vty1 (192.168.1.51)
Proceed with reload? [confirm]
Disconnected
Waiting 60 seconds

========= Pinging 10.10.50.121 every 60 secs =========
10.10.50.121 is unreachable

========= Pinging 10.10.50.121 every 60 secs =========
10.10.50.121 is unreachable

========= Pinging 10.10.50.121 every 60 secs =========
10.10.50.121 is unreachable

========= Pinging 10.10.50.121 every 60 secs =========
10.10.50.121 is unreachable

========= Pinging 10.10.50.121 every 60 secs =========
10.10.50.121 is unreachable

========= Pinging 10.10.50.121 every 60 secs =========
10.10.50.121 is unreachable

========= Pinging 10.10.50.121 every 60 secs =========
10.10.50.121 is unreachable

========= Pinging 10.10.50.121 every 60 secs =========
10.10.50.121 is unreachable

========= Pinging 10.10.50.121 every 60 secs =========

10.10.50.121 is back online


############### Post-Verification ###############

Reconnected

 {'device_type': 'cisco_ios', 'ip': '10.10.50.121', 'username': 'admin', 'password': 'admin'}

 <netmiko.cisco.cisco_ios.CiscoIosSSH object at 0x1095fb400>

#################### Layer 1 ###################
Traceback (most recent call last):
  File "/myscripts/reload.py", line 177, in <module>
    ping()
  File "/myscripts/reload.py", line 123, in ping
    ping()
  File "/myscripts/reload.py", line 123, in ping
    ping()
  File "/myscripts/reload.py", line 123, in ping
    ping()
  [Previous line repeated 5 more times]
  File "/myscripts/reload.py", line 133, in ping
    layer_1_check()
  File "/myscripts/reload.py", line 32, in layer_1_check
    layer1_output = ssh_connect.send_command(command)
  File "/usr/local/lib/python3.9/site-packages/netmiko/utilities.py", line 500, in wrapper_decorator
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/netmiko/base_connection.py", line 1475, in send_command
    prompt = self.find_prompt(delay_factor=delay_factor)
  File "/usr/local/lib/python3.9/site-packages/netmiko/base_connection.py", line 1178, in find_prompt
    self.clear_buffer()
  File "/usr/local/lib/python3.9/site-packages/netmiko/base_connection.py", line 1216, in clear_buffer
    data = self.read_channel()
  File "/usr/local/lib/python3.9/site-packages/netmiko/base_connection.py", line 526, in read_channel
    output = self._read_channel()
  File "/usr/local/lib/python3.9/site-packages/netmiko/base_connection.py", line 500, in _read_channel
    if self.remote_conn.recv_ready():
AttributeError: 'NoneType' object has no attribute 'recv_ready'

My code:

#!/bin/python3

# Modules and functions to import
from getpass import getpass #to hide password 
from selectors import BaseSelector
from time import sleep
from netmiko import ConnectHandler, SSHDetect, redispatch #for ssh
import os #for ping

# Enable netmiko logging
import logging
logging.basicConfig(filename='test.log', level=logging.DEBUG)
logger = logging.getLogger("netmiko")

# Get login credentials
user = input("Username: ")
password = getpass("Password: ")

# Define list of commands
list_cmds = open('reload_cmds.txt').read().split('&')
layer_1 = list_cmds[1]
layer_2 = list_cmds[3]
layer_3_ospf = list_cmds[5]
layer_3_ospfv3 = list_cmds[6]
layer_3_bgp = list_cmds[7]

# Defining check for different layers as function
## Run LAYER 1 precheck commands
def layer_1_check():
    print('\n#################### Layer 1 ###################')
    for command in layer_1.splitlines():
        layer1_output = ssh_connect.send_command(command)
        if 'Invalid' in layer1_output:
            print(f'========= {command} =========')
            print('Invalid command\n')
        else:
            print(f'========= {command} =========')
            print(layer1_output)

## Run LAYER 2 precheck commands
def layer_2_check():
    print('\n#################### Layer 2 ###################')
    for command in layer_2.splitlines():
        layer2_output = ssh_connect.send_command(command)
        if 'Invalid' in layer2_output:
            print(f'========= {command} =========')
            print('Invalid command\n')
        else:
            print(f'========= {command} =========')
            print(layer2_output)

## Run LAYER 3 precheck commands
def layer_3_check():
    print('\n#################### Layer 3 ###################')
    layer3_show_ip_pro = ssh_connect.send_command("show ip protocols")
    print('========= show ip protocols =========')
    print(layer3_show_ip_pro)
# run commands based on routing protocols
# ospf
    if 'ospf ' in layer3_show_ip_pro:
        for command in layer_3_ospfv3.splitlines():
            layer3_output = ssh_connect.send_command(command)
            if 'Invalid' in layer3_output:
                print(f'========= {command} =========')
                print('Invalid command\n')
            else:
                print(f'========= {command} =========')
                print(layer3_output)

# ospfv3
    if 'ospfv3' in layer3_show_ip_pro:
        for command in layer_3_ospfv3.splitlines():
            layer3_output = ssh_connect.send_command(command)
            if 'Invalid' in layer3_output:
                print(f'========= {command} =========')
                print('Invalid command\n')
            else:
                print(f'========= {command} =========')
                print(layer3_output)
# bgp
    if 'bgp' in layer3_show_ip_pro:
        for command in layer_3_bgp.splitlines():
            layer3_output = ssh_connect.send_command(command)
            if 'Invalid' in layer3_output:
                print(f'========= {command} =========')
                print('Invalid command\n')
            else:
                print(f'========= {command} =========')
                print(layer3_output)

# Define reload function
def reload():
    proceed = input('\nWould you like to proceed with reload?[Type yes or no]:  ')
    if proceed == 'yes':
        print('\n========= Saving configuration =========\n')
        reload_output = ssh_connect.save_config()
        print(reload_output)
        print('\n========= Reloading the switch =========\n')
        reload_output = ssh_connect.send_command(
            command_string='reload in 001',
            expect_string=r'confirm'
        )
        print(reload_output)
        # reload_output += ssh_connect.send_command('\n') # Troubleshoot
        # print(reload_output) # Troubleshoot
        ssh_connect.disconnect()
        print('Disconnected')

    else:
        ssh_connect.disconnect()

# Define ping function to find out when the device comes back online
def ping():
    print(f'\n========= Pinging {ipv4} every 60 secs =========')
    # sleep(5) # Troubleshoot
# remove the '.0' in '100.0%' for linux
    ping_cmd = os.popen(f'ping -c 1 -W 1 {ipv4} | grep -q "100.0% packet loss" && echo "0" || echo "1"').read().splitlines()
    ping_response = ping_cmd[0]
    # print(ping_response)
    if ping_response == "0":
        print(f'{ipv4} is unreachable')
        sleep(10)
        ping()
    
    elif ping_response == "1":
        print(f'\n{ipv4} is back online\n')
        print('\n############### Post-Verification ###############\n')
        # sleep(10) 
        ssh_connect = ConnectHandler(**host)
        print('Reconnected') # Troubleshoot
        print(f'\n {host}') # Troubleshoot
        print(f'\n {ssh_connect}') # Troubleshoot
        layer_1_check()
        layer_2_check()
        layer_3_check()
        ssh_connect.disconnect()
        # print('End of ping() ') # Troubleshoot
    
    else:
        pass



# Define list of devices
ipv4_list = open('input.txt')
list_ips = ipv4_list.read().splitlines()
# print(list_ips)


for ipv4 in list_ips:
# Define host attributes   
    host = {
        'device_type': 'autodetect',
        'ip': ipv4 ,
        # 'port': '7663',
        'username': user,
        'password': password
    }
    # print(ipv4)
# Auto-detect device type and update    
    guesser = SSHDetect(**host)
    best_match = guesser.autodetect()
    if best_match == 'cisco_ios':
        print('################################################')
        print(f'########## Connecting to {ipv4} ##########')
        print('################################################')
        host['device_type'] = best_match
        ssh_connect = ConnectHandler(**host)
        print('\n############### Pre-Verification ###############\n')
        layer_1_check()
        layer_2_check()
        layer_3_check()
        reload()
# Wait for the switch to die !!! You have 60 secs to change your mind. 
        print('Waiting 60 seconds') # Troubleshoot
        sleep(60)
        ping()
        

    else:
        print(f'\nThe platform for the device with ip {ipv4} is {best_match} which is not supported by the script.\n')
        pass

Solution

  • You have a function scoping issue--you create your original ssh_connect variable at the global scope (inside best_match == "cisco_ios" conditional). You then kill that SSH connection after the reload (the Netmiko .disconnect() call will set this ssh_connect variable to None).

    You then create a new ssh_connect variable, but you do it inside of your ping function (and do NOT pass it as an argument to layer_1_check(), layer_2_check(), layer_3_check().

    Or worded differently, you have two ssh_connect variables--one that is only known inside of the ping_function and one that is known in global scope (main scope). The one in the global scope is the one being used by layer_[123]_check and that one is dead and set to None.

    I pretty much always pass variables into functions (except for CONSTANTS) or alternatively make a class/objects.