Cisco ASR1K - 100M link issue. [SOLVED]

I was labbing today and hit one strange issue - a link between Cisco ASR1001-X (GLC-T transceiver) and Catalyst 3750 switch (100Mbps ports) was up at both sides, but CDP didn’t work, I didn’t see MAC addresses learned on the switch side. To fix an issue I had to disable auto-negotiation and hardcode 100M speed on ASR side.

interface GigabitEthernet0/0/0
 no negotiation auto
 speed 100

Even more, on Catalyst side I had to hardcode “duplex full” to get rid of duplex mismatch issue. Looks like it’s an issue with ASR or a transceivers. Anyways, it’s strange and annoying to see the link UP on both ends, but without actual connectivity:

Good luck!

Cisco ASA - crypto ipsec df-bit clear-df. [TESTED]

When you use Cisco ASA to build a VPN you better make sure that you’re passing packets with max size with DF-bit enabled. Here is the way to do it:

crypto ipsec df-bit clear-df outside

Before:

Router#ping vrf TEST 1.1.1.1 size 1500 df-bit
Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
Packet sent with the DF bit set
M.M.M
Success rate is 0 percent (0/5)
Router#

“M” means “Could not fragment.”

After:

Router#ping vrf TEST 1.1.1.1 size 1500 df-bit
Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
Packet sent with the DF bit set
.!!!!
Router#

Good luck!

Cisco Catalyst - %PM-4-ERR_DISABLE: link-flap error detected. [SOLVED]

If you have a device connected to a Cisco Catalyst switch that sometimes behaves weirdly and flapping interfaces (during reboot process, for example) mostlikely Catalyst switch will shut this interface with the following log message:

May  6 09:26:34.805 PDT: %PM-4-ERR_DISABLE: link-flap error detected on Gi0/49, putting Gi0/49 in err-disable state

As you know, link-flap error detection is enabled by default with the following parameters:

Switch#show errdisable flap-values
ErrDisable Reason    Flaps    Time (sec)
-----------------    ------   ----------
pagp-flap              3       30
dtp-flap               3       30
link-flap              5       10
Switch#

Switfch#show errdisable detect
ErrDisable Reason            Detection        Mode
-----------------            ---------        ----
arp-inspection               Enabled          port
bpduguard                    Enabled          port
channel-misconfig (STP)      Enabled          port
community-limit              Enabled          port
dhcp-rate-limit              Enabled          port
dtp-flap                     Enabled          port
gbic-invalid                 Enabled          port
iif-reg-failure              Enabled          port
inline-power                 Enabled          port
invalid-policy               Enabled          port
l2ptguard                    Enabled          port
link-flap                    Enabled          port
loopback                     Enabled          port
lsgroup                      Enabled          port
mac-limit                    Enabled          port
pagp-flap                    Enabled          port
port-mode-failure            Enabled          port
pppoe-ia-rate-limit          Enabled          port
psecure-violation            Enabled          port/vlan
security-violation           Enabled          port
sfp-config-mismatch          Enabled          port
sgacl_limitation             Enabled          port
small-frame                  Enabled          port
storm-control                Enabled          port
udld                         Enabled          port
vmps                         Enabled          port
psp                          Enabled          port
Switch#

We have two ways: Configure show errdisable recovery mechanism or disable detection based on port-flapping. Here is how you can disable it:

conf t
 no errdisable detect cause link-flap
 end

Checking:

Switch#show errdisable detect
ErrDisable Reason            Detection        Mode
-----------------            ---------        ----
arp-inspection               Enabled          port
bpduguard                    Enabled          port
channel-misconfig (STP)      Enabled          port
community-limit              Enabled          port
dhcp-rate-limit              Enabled          port
dtp-flap                     Enabled          port
gbic-invalid                 Enabled          port
iif-reg-failure              Enabled          port
inline-power                 Enabled          port
invalid-policy               Enabled          port
l2ptguard                    Enabled          port
link-flap                    Disabled
loopback                     Enabled          port
lsgroup                      Enabled          port
mac-limit                    Enabled          port
pagp-flap                    Enabled          port
port-mode-failure            Enabled          port
pppoe-ia-rate-limit          Enabled          port
psecure-violation            Enabled          port/vlan
security-violation           Enabled          port
sfp-config-mismatch          Enabled          port
sgacl_limitation             Enabled          port
small-frame                  Enabled          port
storm-control                Enabled          port
udld                         Enabled          port
vmps                         Enabled          port
psp                          Enabled          port
Switch#

Good luck!

Cisco Nexus 5010 - Secondary power supply.

Quick note on secondary power supplies in Cisco Nexus 5010.

Today I did some maintenance with Nexus 5010 - installed secondary power supplies and rerouted power cables. It was pretty straightforward. Here is the status without secondary power supply:

n5k-1# show environment power 

Power Supply:
Voltage: 12 Volts
-----------------------------------------------------------
PS  Model                Input Power       Power     Status
                         Type  (Watts)     (Amp)
-----------------------------------------------------------
1   N5K-PAC-550W         AC     544.56     45.38     ok
2   --                   --         --        --     absent              

Mod Model                   Power     Power       Power     Power       Status
                            Requested Requested   Allocated Allocated
                            (Watts)   (Amp)       (Watts)   (Amp)
--- ----------------------  -------   ----------  --------- ----------  ----------
1    N5K-C5010P-BF-SUP      349.20    29.10       349.20    29.10       powered-up

Power Usage Summary:
--------------------
Power Supply redundancy mode:                 Redundant
Power Supply redundancy operational mode:     Non-redundant

Total Power Capacity                              544.56 W

Power reserved for Supervisor(s)                  349.20 W
Power currently used by Modules                     0.00 W

                                                -------------
Total Power Available                             195.36 W
                                                -------------
n5k-1#

I followed Cisco Nexus 5000 Series Hardware Installation Guide > Replacing or Installing Power Supplies. After you install secondary power supply you will see the following syslog messages. If power cable is not plugged in yet you will see “FAIL” yellow led on PS.

2019 Apr 29 08:41:30 n5k1 %PFMA-5-PS_FOUND: Power supply 2 found (Serial number DTM142700X1)
2019 Apr 29 08:41:30 n5k1 %NOHMS-2-NOHMS_DIAG_ERR_PS_FAIL: System minor alarm on power supply 2: failed
2019 Apr 29 08:41:30 n5k1 %PFMA-2-PS_FAIL: Power supply 2 failed or shutdown (Serial number DTM142700X1)

Status with two PSUs installed, but cable is not yet connected to the 2nd PSU:

n5k1# show environment power 

Power Supply:
Voltage: 12 Volts
-----------------------------------------------------------
PS  Model                Input Power       Power     Status
                         Type  (Watts)     (Amp)
-----------------------------------------------------------
1   N5K-PAC-550W         AC     544.56     45.38     ok
2   --                   --         --        --     fail/shutdown       

Mod Model                   Power     Power       Power     Power       Status
                            Requested Requested   Allocated Allocated
                            (Watts)   (Amp)       (Watts)   (Amp)
--- ----------------------  -------   ----------  --------- ----------  ----------
1    N5K-C5010P-BF-SUP      349.20    29.10       349.20    29.10       powered-up

Power Usage Summary:
--------------------
Power Supply redundancy mode:                 Redundant
Power Supply redundancy operational mode:     Non-redundant

Total Power Capacity                              544.56 W

Power reserved for Supervisor(s)                  349.20 W
Power currently used by Modules                     0.00 W

                                                -------------
Total Power Available                             195.36 W
                                                -------------
n5k1#

After I plugged in the power cable I had to wait ~15 seconds to get the following syslog messages:

2019 Apr 29 08:45:42 n5k-1 %NOHMS-2-NOHMS_DIAG_ERR_PS_RECOVERED: Recovered: System minor alarm on power supply 2: failed

Here is the status with two PSUs and both cables plugged in:

n5k-1# show environment power 

Power Supply:
Voltage: 12 Volts
-----------------------------------------------------------
PS  Model                Input Power       Power     Status
                         Type  (Watts)     (Amp)
-----------------------------------------------------------
1   N5K-PAC-550W         AC     544.56     45.38     ok
2   N5K-PAC-550W         AC     544.56     45.38     ok                  

Mod Model                   Power     Power       Power     Power       Status
                            Requested Requested   Allocated Allocated
                            (Watts)   (Amp)       (Watts)   (Amp)
--- ----------------------  -------   ----------  --------- ----------  ----------
1    N5K-C5010P-BF-SUP      349.20    29.10       349.20    29.10       powered-up

Power Usage Summary:
--------------------
Power Supply redundancy mode:                 Redundant
Power Supply redundancy operational mode:     Redundant

Total Power Capacity                             1089.12 W

Power reserved for Supervisor(s)                  349.20 W
Power currently used by Modules                     0.00 W

                                                -------------
Total Power Available                             739.92 W
                                                -------------
n5k-1#

Then to reroute power cables I disconnected 1st power supply for ~5 seconds and haven’t got any syslog messages. So, there’s some delay and you have to be aware that brief power cable disconnection will not be registered in the logs.

Good luck!

Cisco Catalyst - Unsupported transceiver - GBIC_SECURITY_CRYPT-4-VN_DATA_CRC_ERROR

For documenting purposes I will go ahead and put step-by-step procedure how to enable NON-Cisco SFP transceivers in Catalyst switches. In this test we will be using C3560G with 15.0(2)SE11 and generic Finisar MMF transceiver.

When you install NON-Cisco SFP you would see the following in logs:

*Mar  1 00:07:19.932: %GBIC_SECURITY_CRYPT-4-VN_DATA_CRC_ERROR: GBIC in port Gi0/49 has bad crc
*Mar  1 00:07:19.932: %PM-4-ERR_DISABLE: gbic-invalid error detected on Gi0/49, putting Gi0/49 in err-disable state

Here is how show interface output would look like. Interface will be in “err-disabed” state.

Switch#show int gi0/49
GigabitEthernet0/49 is down, line protocol is down (err-disabled)
  Hardware is Gigabit Ethernet, address is 0023.ab7d.76c1 (bia 0023.ab7d.76c1)
  Internet address is 1.1.1.1/24
  MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Auto-duplex, Auto-speed, link type is auto, media type is unknown
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output never, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     0 packets output, 0 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
Switch#

Transceiver will NOT be shown in inventory:

Switch#show inv
NAME: "1", DESCR: "WS-C3560G-48TS"
PID: WS-C3560G-48TS-E  , VID: V03  , SN: FOC1243W1GS

Switch#

Now let’s bounce the port to see if any difference:

conf t
 int gi0/49
  shutdown
  no sh

As the result, “err-disabled” state changed to down (notconnected)”:

Switch#show int gi0/49
GigabitEthernet0/49 is down, line protocol is down (notconnect)
  Hardware is Gigabit Ethernet, address is 0023.ab7d.76c1 (bia 0023.ab7d.76c1)
  Internet address is 1.1.1.1/24
  MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Auto-duplex, Auto-speed, link type is auto, media type is unknown
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output never, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     0 packets output, 0 bytes, 0 underruns
     0 output errors, 0 collisions, 2 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
Switch#

But link is not coming up. When you try to see signal level you get the following:

Switch#show int gi0/49 transceiver
Diagnostic Monitoring is not implemented.

Switch#

Next logical step is to apply two magic commands to allow NON-Cisco transceivers:

conf t
 service unsupported-transceiver
 no errdisable detect cause gbic-invalid

Port bounce will not change anything so you have a choice:
• Save the config and reboot the router.
• Physically pull the transceiver out of the chassis and plug it back in.

If you go the 2nd ways here is what you would see:

*Mar  1 00:12:24.748: %GBIC_SECURITY_CRYPT-4-VN_DATA_CRC_ERROR: GBIC in port Gi0/49 has bad crc
*Mar  1 00:12:24.748: %PHY-4-UNSUPPORTED_TRANSCEIVER: Unsupported transceiver found in Gi0/49
*Mar  1 00:12:30.268: %LINK-3-UPDOWN: Interface GigabitEthernet0/49, changed state to up
*Mar  1 00:12:31.275: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/49, changed state to up

Switch#show int gi0/49
GigabitEthernet0/49 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 0023.ab7d.76c1 (bia 0023.ab7d.76c1)
  Internet address is 1.1.1.1/24
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 1000Mb/s, link type is auto, media type is unsupported
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:08, output 00:00:19, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     11 packets input, 4068 bytes, 0 no buffer
     Received 11 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 10 multicast, 0 pause input
     0 input packets with dribble condition detected
     1 packets output, 64 bytes, 0 underruns
     0 output errors, 0 collisions, 3 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
Switch#

Interface came up, but still showing “media type is unsupported”, but it’s working and we can check optical signal level:

Switch#show int gi0/49 transceiver
ITU Channel not available (Wavelength not available),
Transceiver is internally calibrated.
If device is externally calibrated, only calibrated values are printed.
++ : high alarm, +  : high warning, -  : low warning, -- : low alarm.
NA or N/A: not applicable, Tx: transmit, Rx: receive.
mA: milliamperes, dBm: decibels (milliwatts).

                                 Optical   Optical
           Temperature  Voltage  Tx Power  Rx Power
Port       (Celsius)    (Volts)  (dBm)     (dBm)
---------  -----------  -------  --------  --------
Gi0/49       25.0       3.32      -4.9      -3.9   

Switch#

Moreover, unsupported transceiver showed up in “show inventory” output:

Switch#show inventory
NAME: "1", DESCR: "WS-C3560G-48TS"
PID: WS-C3560G-48TS-E  , VID: V03  , SN: FOC1243W1GS

NAME: "GigabitEthernet0/49", DESCR: "unsupported"
PID: Unspecified       , VID:      , SN: NSH1U19         

Switch#

After disconnecting fiber optic from the transceiver we noticed that it was showing -27.2. Cisco transceivers usually show -40.0. It probably depends on Transceiver model/vendor/firmware.

Switch#show int gi0/49 transceiver
ITU Channel not available (Wavelength not available),
Transceiver is internally calibrated.
If device is externally calibrated, only calibrated values are printed.
++ : high alarm, +  : high warning, -  : low warning, -- : low alarm.
NA or N/A: not applicable, Tx: transmit, Rx: receive.
mA: milliamperes, dBm: decibels (milliwatts).

                                 Optical   Optical
           Temperature  Voltage  Tx Power  Rx Power
Port       (Celsius)    (Volts)  (dBm)     (dBm)
---------  -----------  -------  --------  --------
Gi0/49       25.5       3.32      -4.9     -27.2   

Switch#

Good luck!

Admin area