RANCID for Cisco Nexus 5K - Be careful with CSCts72635. [EXPLAINED]

Today I’ve added a couple of Nexus 5K switches to RANCID. Doing so I remembered potential issue with older Nexus software version and decided to make a quick note.

It’s related to Cisco bug CSCts72635 that causes Nexus 5K with software older than 5.1(3)N1(1) to reboot after issuing “show version” command after 10240 times. Remember, RANCID executes this command every time to grab software version.

Here are bug details:

Conditions:If callhome is configured and sending alerts (like temperature alarms), the bios_daemon can be triggered in this process. The problem is that bios_daemon opens a file descriptor and it does not close.

“show version” is another way to trigger this.

The problem will occur after 10240 callhome alerts or instances of running “show version”

Known Fixed Releases: 5.1(3)N1(1)

Original post about it is here.

Good luck!

Can’t connect to MikroTik via SSH - Corrupt host’s key, regenerating it! Reboot required! [SOLVED]

Well, I already had an issue with SecureCRT and SSH on MikroTik and put a note about it some time before. Now I get another one. I tried to log in to RouterOS 6.42.1 via SSH from Linux machine and here is what I saw on Linux side:

linux# ssh -l username 1.1.1.1
Received disconnect from 1.1.1.1 port 22:3:
Disconnected from 1.1.1.1 port 22
linux#

I checked debug message of SSH client, but it didn’t help much:

linux# ssh -l username 1.1.1.1 -v
...
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha1 compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha1 compression: none
debug1: kex: diffie-hellman-group-exchange-sha256 need=20 dh_need=20
debug1: kex: diffie-hellman-group-exchange-sha256 need=20 dh_need=20
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<7680<8192) sent
debug1: got SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
Received disconnect from 1.1.1.1 port 22:3:
Disconnected from 1.1.1.1 port 22
linux#

RouterOS generated the following log messages for every attempt:

ssh,error Corrupt host's key, regenerating it! Reboot required!

I used the same thing to fix it:

[admin@MikroTik] > /ip ssh regenerate-host-key
This will regenerate current SSH host keys, yes? [y/N]:
y
22:15:58 echo: ssh,critical SSH host key regenerated!
[admin@MikroTik] >

Good luck!

Debian - cannot execute binary file: Exec format error - How to reinstall a package. [SOLVED]

This morning I tried to capture the traffic on Debian server and got an error:

debian# tcpdump -i eth0
-su: /usr/sbin/tcpdump: cannot execute binary file: Exec format error
debian#

Apparently this server experienced a problem with local disks and file got corrupted so I had to reinstall the package:

debian# dpkg --purge tcpdump
debian# apt-get install --reinstall tcpdump

Good luck!

Cambium cnMaestro on XenServer - Welcome to emergency mode! [SOLVED]

I had to work on deploying cnMaestro On-Premises v2.1.0 on XenServer.

I was given OVA file cnmaestro-on-premises_2.1.0-r22_amd64.ova. Please don’t be confused by amd64 suffix in the file name, it simply means that it is 64bit (amd architecture based but not limited to amd processors).

After deploying OVA file on XenServer the VM started, but Ubuntu 16.04.5 (OS the product is built on) got stuck and went to “emergency mode”:

Welcome to emergency mode! After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" or ^D to
ty again to boot into default mode.
Press Enter for maintenance
(or press Control-D to continue):

After pressing Enter (it gives you root CLI) and executing “journalctl -xb” I figured that system was not able to mount sdb1 partition:

... systemd[1]: dev-sdb1.device: Job dev-sdb1.device/start timed out.
... systemd[1]: Timed out waiting for device dev-sdb1.device.

It’s worth to mention that according to “cnMaestro On-Premises Quick Start Guide” OVA supports VMware and Oracle VirtualBox, but NOT XenServer. As you might know, block device assignment is different between VMware and XenServer. In VMware environment you would see /dev/sdX for SCSI disks, in XenServer you would expect /dev/xvdX. You can see devices by executing “fdisk -l” command.

Checking /etc/fstab file I noticed an entry for /dev/sdb1. One of the easiest and straightforward workarounds is to modify /etc/fstab to replace /dev/sdb1 with /dev/xvdb1 which I did and it fixed the issue.

Keep in mind after VM started you should wait for few more seconds until UI if fully ready. Otherwise you will see “Service Temporarily Unavailable” page.

Good luck!

VMware - The OVF package is invalid and cannot be deployed. [SOLVED]

Quick note on the error I got when I tried to deploy OVA file to VMware ESXi 5.5.

Error
The OVF package is invalid and cannot be deployed.
The following manifest file entry (line 1) is invalid: SHA245(example.ovf)=…

VMware KB 2151537 explains it:

This issue occurs because the vSphere Client does not support the SHA256 hashing algorithm.
You can convert the OVA from the Cryptographic Hash Algorithm SHA256 to SHA1. To do this, you can use OVFTool which is available on all OS at: https://www.vmware.com/support/developer/ovf/.

To do the conversion, run this command:

ovftool.exe --shaAlgorithm=SHA1 /path/to/the/original/ova_file.ova /path/to/the/new/ova/file-SHA1.ova

This is exactly what happened. I tried to deploy OVA file designed for VMware ESXi 6.0 on older ESXi 5.5. After downloading and installing VMware OVF Tool 4.3.0 I converted OVA file to use SHA1 and imported it to ESXi 5.5 successfully.

Good luck!

Admin area