The various flavors of Cisco IOS

Cisco IOS comes in several different flavors including Cisco IOS, Cisco IOS XE and Cisco IOS XR.

Cisco IOS (Classic IOS)

The classic Cisco IOS network operating system uses a monolithic kernel that runs all of the necessary modules in the same memory space. Any issues within IOS can cause the entire operating system to stop responding or crash.

Cisco IOS XE

IOS XE leverages an underlying Linux operating system and runs IOS as a daemon called ‘IOSd’ and as such the underlying Linux operating system is able to exert monitoring and control over IOSd. Some of the possibilities of this type of virtualization include software based high availability by having two copies of IOSd running at the same time without the need for a completely duplicated route process or hardware layer. Another exciting possibility is to leverage in service software upgrade or ISSU which allows for much easier software upgrades without the need to reload and stop packet processing to change software versions.

Cisco IOS XR

IOS XR is a completely new network operating system running directly within a Linux operating system. This type of rewrite led to a truly modular operating system where different IOS XR features run as isolated and distinct processes which can individually be stopped and restarted.

Cisco NX-OS

Cisco NX-OS started life as SAN-OS which was used on Cisco’s line of storage switches. The Nexus line of switches use a modified version of SAN-OS which was re-written to include more data networking focused while still including the SAN features needed in a converged data center networking product. Cisco NX-OS is modular and in many ways similar to IOS XR from a very high level.

Useful Links

Cisco.com IOS XE site – a generic overview of IOS XE

White Paper: Cisco IOS and NX-OS Software Reference Guide – an excellent overview of which flavor of IOS runs on what platform

Cisco IOS Login Enhancements

Since 2005 Cisco IOS has offered several security enhancements to help mitigate the risks of dictionary attacks used against login sessions. Now you are probably wondering why I’m blogging about something more than 7 years after it was released. I rarely see people implement any of these features and I’m hoping to raise the visibility of these simple configurations.

If you’ve ever left SSH open to the internet I’m confident you’ve seen how quickly  a dictionary attack will commence. There are two basic ways to combat this issue. The first, and highly recommended, way is to simply block SSH access from the internet. The second option involves slowing down dictionary attacks to the point where usernames and passwords can’t be tested quickly enough to guess a valid combination thereof.

The Cisco IOS login enhancements include two features:

  1. Introducing a delay between successive login attempts
  2. Blocking all login attempts once a failure threshold is reached

By introducing even a small delay the time taken for a remote attacker to apply a dictionary attack is greatly increased.

Cisco refers to the second feature as “quiet mode” and also includes an option to specify an access-list which is exempted during the block period.

A quick example should tie all these pieces together. First we will create an access-list that permits specific network(s) from which logins will never be denied.

ip access-list extended SSH_IN
 remark Local Network
 permit ip 10.1.0.0 0.0.0.255 any

Next we will configure the login blocking which requires three parameters. These three parameters are the length of time to block for, the number of failed attempts after which to take action and the period of time during which to monitor failed logins. Here’s the completed configuration to block logins for 60 seconds after 3 failed attempts within a 60 second period. The second line of configuration will reference the access-list created above to never block the specified networks.

login block-for 60 attempts 3 within 60
login quiet-mode access-class SSH_IN

Previously I mentioned that a simple delay between login attempts would greatly slow down dictionary attacks. This delay can be accomplished using the following configuration. The delay duration is specified in seconds.

login delay 2

Cisco Documentation Link

Cisco IOS SIP dial-peer status notifications

This configuration will create a syslog message as well as an SNMP trap either of which your monitoring/alerting systems should be able to use as a trigger to take action.

1. Copy the linked TCL file to the root of the flash filesystem on the router that connects to the SIP trunk(s). Download TCL Script
2. Add the command to “voice-class sip options-keepalive” to one of the dial-peers pointing to the service provider and make a note of the dial-peer number you are adding the command to

Example:

dial-peer voice 100 voip
voice-class sip options-keepalive

3. Add the command track 100 stub-object to the global config.
4. Add the command snmp-server enable traps event-manager to the global config (this is needed for the SNMP trap functionality).

4. Finally the following configuration template can be applied to the router. Notice the number 100 below is a reference to the dial-peer and tracking number from above and must match for this to work.

event manager environment dial_peer_number 100
event manager environment check_interval 30
event manager directory user policy "flash:/"
event manager applet siptrunk_down
event track 100 state down
action 10 snmp-trap strdata "SIP Trunk Down"
action 20 syslog msg "Alert - SIP Trunk Down"
event manager policy check_dial_peer_status.tcl

This was documented at the following Cisco page as well: Link

General Cisco IOS Debugging Reference

Many engineers have been scared away from running debugs in production network due to bad experiences with high CPU utilization requiring drastic action like powering down the device and letting it reload. The CPU impact of debugging can be greatly decreased by changing the configuration of IOS. The snippet below shows a basic template that both reduces the performance impact of debugging and also helps improve the accuracy of logged debugging information.

Router(config)# service timestamps debug datetime localtime msec
Router(config)# service sequence-numbers
Router(config)# logging buffered 1000000 debug
Router(config)# no logging console
Router(config)# no logging monitor

I love config templates as much as the next guy, but a better understanding of what each of these commands does is important too!

Let’s break it down line by line:

service timestamps debug datetime localtime msec Ensures that debug logging entries include millisecond time stamps based on the router’s clock. Millisecond level precision is desired as debugging can generate a lot of messages very quickly and it’s important to know what order things happened in

service sequence-numbers Assigns numerically increasing values to the beginning of log entries to quickly identify the order in which messages occurred.

logging buffered 1000000 debug Increases the size of the onboard log buffer to 1 MB and enables logging of debug level messages. Be aware that issuing this command will erase the current log entries.

no logging console Disables logging output to the console port. By default all logging output is sent to the console port whether or not anything is even connected to it. Not configuring this is one of the biggest contributors to the high CPU utilization and sluggish response. Simply put Cisco IOS schedules the output of text to the console port ahead of many other tasks. The default baud rate on the console interface is 9600 bps, which further amplifies the problem. This means that when lots of things are being sent to the console other tasks must wait before they are allowed to execute.

no logging monitor Disables logging output to other terminal interfaces (telnet or SSH). This prevents us from being able to use the “terminal monitor” command which can be a useful way to see the output of a debug while connected to a router over telnet or SSH. The downside of examining debugs in this way is that depending upon how quickly the messages are being generated there is a good chance some won’t ever show up on your screen. It’s best to simply log them to the internal buffer and view them using “show log”.

Both of the previous commands could be altered to prevent just debug logs from being displayed. Simply add the word “debugging” after one or both and you’ll still be able to see everything that isn’t a debug message.

As wise network professionals always say be sure to test and validate these configurations in a lab environment before implementing anything in production. Even with the optimizations above be sure to check CPU and memory utilization before enabling any debugs.

Cisco IOS dial peer basics

To find all dial-peers configured use “show run | include dial-peer”
To see the configuration of all the dial-peers use “show run | section dial-peer”
If you want to see the entire configuration use “show run”, but I’d start to get comfortable with using the | syntax to help reduce paging through lengthy configurations. Before you make any changes it is a good idea to save a copy of the configuration as a reference. This can be done in a few ways, but the easiest is usually to log your terminal session (turn on logging in PuTTY) and then type “show run” and page through the entire configuration to ensure that it’s been logged. By doing this you can easily revert back to a previous configuration without scrambling to find a backup copy somewhere on the network.When you’ve found a dial-peer that is a good template for what you want to do simply copy and paste that single dial-peer to notepad. A single dial-peer will start with the “dial-peer voice xxx” and ends with the last indented line. When Cisco IOS parses the configuration the indents are automatically added to help make it easier to see what sub-commands are related to a parent command. In the examples below the “dial-peer voice xxx” commands are parent commands and everything indented below them are commands related to these parent commands. You’ll see that same syntax many places within the configuration for things other than dial-peers as well.Note: Make sure when you configure a new dial-peer that you choose a unique number to identify the dial-peer otherwise you’ll be overwriting the existing dial-peer with your new configuration. In the example below you’d be safe creating a dial-peer “102”. The router will not prompt you or prevent you from overwriting existing configuration it will simply assume that you want to change what it already there. You can think of when you press the “insert” key in Word and whenever you type with the cursor in front of existing text you simply overwrite what’s there. You should also know that any configuration changes take effect immediately.Example:

Router#show run | include dial-peer
dial-peer voice 6000 voip
dial-peer voice 100 voip
dial-peer voice 101 voipRouter#show run | section dial-peerdial-peer voice 6000 voip
destination-pattern 60[01].
session protocol sipv2
session target ipv4:192.168.1.100
voice-class codec 2
dtmf-relay rtp-nte
no vad
dial-peer voice 100 voip
translation-profile outgoing OUTBOUND
destination-pattern 9[2-9]..[2-9]......
progress_ind setup enable 3
session protocol sipv2
session target ipv4:192.168.1.100
voice-class sip early-offer forced
dtmf-relay rtp-nte
codec g711ulaw
no vad
dial-peer voice 101 voip
translation-profile outgoing OUTBOUND
destination-pattern 91[2-9]..[2-9]......
progress_ind setup enable 3
session protocol sipv2
session target ipv4:192.168.1.100
voice-class sip early-offer forced
dtmf-relay rtp-nte
codec g711ulaw
no bad

To add a new dial-peer you first need to ensure that you’re in configuration mode (take a look at the router prompt and if you see the word “config” after the router hostname you’re good to go). If you’re in enable mode (you’ll see the hash or octothorpe after the router hostname) type “config t” to enter config mode.

Example:

Router#Now we enter configuration mode:Router#config tThe prompt changes to:Router(config)#From the configuration prompt you can type or paste your new dial-peer.

Router(config)#dial-peer voice 102 voip
translation-profile outgoing OUTBOUND
destination-pattern 9011T
progress_ind setup enable 3
session protocol sipv2
session target ipv4:192.168.1.100
voice-class sip early-offer forced
dtmf-relay rtp-nte
codec g711ulaw
no vad

To return to enable mode type "exit"

Router(config)#exit

The prompt will return to the enable mode prompt:

Router#

To save your configuration type "write memory". If you don't do this and someone restarts the router or power is lost your configuration changes will be lost.

Router#write memory

You'll see two messages displayed and then you'll be returned to the enable mode prompt:

Router#wr mem
Building configuration...

[OK]
Router#

To logout of the router and close your session type "exit":

Router#exit

Strange crypto map commands logged at boot up

When configuration archiving and logging is configured on an IOS router with a crypto capable image you may some rather odd commands every time the router starts up.

Here’s an example of what’s shown when you run the “show archive log config all”

1 1 console@console |access-list 199 permit icmp host 10.10.10.10 host 20.20.20.20
2 1 console@console |crypto map NiStTeSt1 10 ipsec-manual
3 1 console@console |match address 199
4 1 console@console |set peer 20.20.20.20
5 1 console@console |exit
6 1 console@console |no access-list 199
7 1 console@console |no crypto map NiStTeSt1

At first glance this appears to something nefarious, but it’s actually a test routine built in to IOS in order to meet FIPS requirements. It’s completely normal and does not impact normal operations.

Cisco IOS DHCP inheritance

Until recently I was completely unaware of the fact that DHCP server attributes on Cisco IOS support inheritance which not only shortens configurations, but decreases the chances of making typographical errors. The feature works quite simply. When Cisco IOS receives a DHCPDISCOVER message it matches the request against all matching DHCP pools based on the IP and subnet assigned to the interface that the DHCPDISCOVER message was received on. For example a router with an Ethernet interface configured as follows would look for DHCP scopes matching 192.168.1.0/24:

interface FastEthernet0/1
   ip address 192.168.1.1 255.255.255.0

Basic DHCP pool configuration:

ip dhcp pool DHCP-POOL
   network 192.168.1.0 /24
   default-router 192.168.1.1
   domain-name MyDomain.local
   dns-server 8.8.8.8 4.4.4.2

I would also recommend excluding a portion of any DHCP scope for things that are going to be statically assigned. I will be excluding the following address range for this example 192.168.1.1 -> 192.168.1.10. This can be done using the following syntax:

ip dhcp pool excluded-address 192.168.1.1 192.168.1.10

If I want to extend this configuration and hand out a static DHCP reservation to my laptop should I duplicate all the settings I’ve already defined for the “DHCP-POOL” scope or just the ones that are unique? The answer is you can do either! I would recommend shortening your configuration and specifying only what you need. Here’s an example:

ip dhcp pool MYLAPTOP
   hardware-address 0012.3456.1111
   host 192.168.1.100 mask 255.255.255.0

Notice that the only thing I’m specifying is the hardware-address which matches the MAC address of my laptop and the host attribute which is the static reservation I want to always give to this DHCP client.

Cisco IOS DHCP Static Reservations

I recently wanted to configure an HP Jetdirect print server to receive an IP address via DHCP with the server configured to hand out a static reservation. I had used the IOS DHCP pool sub-command of “client-identifier” in the past to create static reservations for Cisco IP phones and various Windows and MAC OS desktops. When I user the familiar syntax of a client-identifier formatted as 01xx.xxxx.xxxx.xx where the leading 01 indicates the Ethernet media type.

This technique has worked well for me in the past for most network devices, but it refused to assign the address I wanted to this HP Jetdirect. I did some research and found that another way to assign DHCP addresses based on MAC addresses is to use the DHCP pool sub-configuration parameter of “hardware-address xxxx.xxxx.xxxx” where the xxxx.xxxx.xxxx is the full MAC address without any prefix like we used above.

The working configuration looked like this:

ip dhcp pool HPJETDIRECT
   host 10.1.0.15 255.255.255.0
   hardware-address 0011.0aed.62b6
   default-router 10.1.0.1

The working configuration for my laptop looks like this (notice the leading 01 on the client-identifier followed by my MAC address):

ip dhcp pool NATEMACBOOKPRO
   host 10.1.0.20 255.255.255.0
   client-identifier 0134.159e.90de.3b
   default-router 10.1.0.1

Catalyst 3750 Cable Test

I was recently faced with a remote troubleshooting situation caused by a newly installed IP phone that could receive PoE power, but was unable to acquire a DHCP lease. I then remembered a useful feature of the 3750 that allows for a cable test to be executed on any copper interface. To perform this test simply run the following command:

Catalyst-3750# test cable-diagnostics tdr interface gig 2/0/2
TDR test started on interface Gi1/0/2
A TDR test can take a few seconds to run on an interface
Use ‘show cable-diagnostics tdr’ to read the TDR results.
Catalyst-3750#show cable-diagnostics tdr interface gig 2/0/2
TDR test last run on: December 30 08:31:09

Interface Speed Local pair Pair length Remote pair Pair status
——— —– ———- —————— ———– ——————–
Gi2/0/2 auto Pair A 40 +/- 4 meters N/A Open
Pair B 12 +/- 4 meters N/A Open
Pair C 40 +/- 4 meters N/A Open
Pair D 40 +/- 4 meters N/A Open

From this output it was easy to see that Pair B was not connected end-to-end and needed to be checked by our cabling vendor.

The technology in use here is TDR (time-domain reflectometry) and more information can be found at the following Wikipedia page here

More information from Cisco can be found here

DSL Interface Statistics

I found this link while troubleshooting a possible DSL issue. Cisco routers support the following command to show DSL interface statistics “show dsl interface atm x/y” where x and y are the port and slot identifiers.

The output of this command will be similar to the following:

Alcatel 20150 chipset information
ATU-R (DS) ATU-C (US)
Modem Status: Showtime (DMTDSL_SHOWTIME)
DSL Mode: ITU G.992.1 (G.DMT)
ITU STD NUM: 0x01 0x1
Vendor ID: ‘ALCB’ ‘ALCB’
Vendor Specific: 0x0000 0x0000
Vendor Country: 0x00 0x0F
Capacity Used: 100% 98%
Noise Margin: 10.0 dB 10.0 dB
Output Power: 20.0 dBm 12.0 dBm
Attenuation: 37.5 dB 23.0 dB
Defect Status: None None
Last Fail Code: None
Selftest Result: 0x00
Subfunction: 0x15
Interrupts: 48337 (0 spurious)
PHY Access Err: 0
Activations: 4
Init FW: embedded
Operation FW: embedded
SW Version: 3.8131
FW Version: 0x1A04

Interleave Fast Interleave Fast
Speed (kbps): 3648 0 832 0
Reed-Solomon EC: 41799 0 11786 0
CRC Errors: 0 0 0 0
Header Errors: 0 0 0 0
Bit Errors: 0 0
BER Valid sec: 0 0
BER Invalid sec: 0 0

The information of most interest is the noise margin, output power, and attenuation as they are the metrics by which line quality is judged. You can also see the current trained DSL speed, downstream is shown first followed by upstream. In this example the downstream speed is 3648 kbps and the upstream is 832 kbps.

The following link covers some good target values for noise margin, output power, and attenuation.

Link