Cisco IOS comes in several different flavors including Cisco IOS, Cisco IOS XE and Cisco IOS XR.
Cisco IOS (Classic IOS)
The classic Cisco IOS network operating system uses a monolithic kernel that runs all of the necessary modules in the same memory space. Any issues within IOS can cause the entire operating system to stop responding or crash.
Cisco IOS XE
IOS XE leverages an underlying Linux operating system and runs IOS as a daemon called ‘IOSd’ and as such the underlying Linux operating system is able to exert monitoring and control over IOSd. Some of the possibilities of this type of virtualization include software based high availability by having two copies of IOSd running at the same time without the need for a completely duplicated route process or hardware layer. Another exciting possibility is to leverage in service software upgrade or ISSU which allows for much easier software upgrades without the need to reload and stop packet processing to change software versions.
Cisco IOS XR
IOS XR is a completely new network operating system running directly within a Linux operating system. This type of rewrite led to a truly modular operating system where different IOS XR features run as isolated and distinct processes which can individually be stopped and restarted.
Cisco NX-OS started life as SAN-OS which was used on Cisco’s line of storage switches. The Nexus line of switches use a modified version of SAN-OS which was re-written to include more data networking focused while still including the SAN features needed in a converged data center networking product. Cisco NX-OS is modular and in many ways similar to IOS XR from a very high level.
Cisco.com IOS XE site – a generic overview of IOS XE
White Paper: Cisco IOS and NX-OS Software Reference Guide – an excellent overview of which flavor of IOS runs on what platform
Since 2005 Cisco IOS has offered several security enhancements to help mitigate the risks of dictionary attacks used against login sessions. Now you are probably wondering why I’m blogging about something more than 7 years after it was released. I rarely see people implement any of these features and I’m hoping to raise the visibility of these simple configurations.
If you’ve ever left SSH open to the internet I’m confident you’ve seen how quickly a dictionary attack will commence. There are two basic ways to combat this issue. The first, and highly recommended, way is to simply block SSH access from the internet. The second option involves slowing down dictionary attacks to the point where usernames and passwords can’t be tested quickly enough to guess a valid combination thereof.
The Cisco IOS login enhancements include two features:
- Introducing a delay between successive login attempts
- Blocking all login attempts once a failure threshold is reached
By introducing even a small delay the time taken for a remote attacker to apply a dictionary attack is greatly increased.
Cisco refers to the second feature as “quiet mode” and also includes an option to specify an access-list which is exempted during the block period.
A quick example should tie all these pieces together. First we will create an access-list that permits specific network(s) from which logins will never be denied.
ip access-list extended SSH_IN
remark Local Network
permit ip 10.1.0.0 0.0.0.255 any
Next we will configure the login blocking which requires three parameters. These three parameters are the length of time to block for, the number of failed attempts after which to take action and the period of time during which to monitor failed logins. Here’s the completed configuration to block logins for 60 seconds after 3 failed attempts within a 60 second period. The second line of configuration will reference the access-list created above to never block the specified networks.
login block-for 60 attempts 3 within 60
login quiet-mode access-class SSH_IN
Previously I mentioned that a simple delay between login attempts would greatly slow down dictionary attacks. This delay can be accomplished using the following configuration. The delay duration is specified in seconds.
login delay 2
Cisco Documentation Link
This configuration will create a syslog message as well as an SNMP trap either of which your monitoring/alerting systems should be able to use as a trigger to take action.
1. Copy the linked TCL file to the root of the flash filesystem on the router that connects to the SIP trunk(s). Download TCL Script
2. Add the command to “voice-class sip options-keepalive” to one of the dial-peers pointing to the service provider and make a note of the dial-peer number you are adding the command to
dial-peer voice 100 voip
voice-class sip options-keepalive
3. Add the command
track 100 stub-object to the global config.
4. Add the command
snmp-server enable traps event-manager to the global config (this is needed for the SNMP trap functionality).
4. Finally the following configuration template can be applied to the router. Notice the number 100 below is a reference to the dial-peer and tracking number from above and must match for this to work.
event manager environment dial_peer_number 100
event manager environment check_interval 30
event manager directory user policy "flash:/"
event manager applet siptrunk_down
event track 100 state down
action 10 snmp-trap strdata "SIP Trunk Down"
action 20 syslog msg "Alert - SIP Trunk Down"
event manager policy check_dial_peer_status.tcl
This was documented at the following Cisco page as well: Link
Many engineers have been scared away from running debugs in production network due to bad experiences with high CPU utilization requiring drastic action like powering down the device and letting it reload. The CPU impact of debugging can be greatly decreased by changing the configuration of IOS. The snippet below shows a basic template that both reduces the performance impact of debugging and also helps improve the accuracy of logged debugging information.
Router(config)# service timestamps debug datetime localtime msec
Router(config)# service sequence-numbers
Router(config)# logging buffered 1000000 debug
Router(config)# no logging console
Router(config)# no logging monitor
I love config templates as much as the next guy, but a better understanding of what each of these commands does is important too!
Let’s break it down line by line:
service timestamps debug datetime localtime msec Ensures that debug logging entries include millisecond time stamps based on the router’s clock. Millisecond level precision is desired as debugging can generate a lot of messages very quickly and it’s important to know what order things happened in
service sequence-numbers Assigns numerically increasing values to the beginning of log entries to quickly identify the order in which messages occurred.
logging buffered 1000000 debug Increases the size of the onboard log buffer to 1 MB and enables logging of debug level messages. Be aware that issuing this command will erase the current log entries.
no logging console Disables logging output to the console port. By default all logging output is sent to the console port whether or not anything is even connected to it. Not configuring this is one of the biggest contributors to the high CPU utilization and sluggish response. Simply put Cisco IOS schedules the output of text to the console port ahead of many other tasks. The default baud rate on the console interface is 9600 bps, which further amplifies the problem. This means that when lots of things are being sent to the console other tasks must wait before they are allowed to execute.
no logging monitor Disables logging output to other terminal interfaces (telnet or SSH). This prevents us from being able to use the “terminal monitor” command which can be a useful way to see the output of a debug while connected to a router over telnet or SSH. The downside of examining debugs in this way is that depending upon how quickly the messages are being generated there is a good chance some won’t ever show up on your screen. It’s best to simply log them to the internal buffer and view them using “show log”.
Both of the previous commands could be altered to prevent just debug logs from being displayed. Simply add the word “debugging” after one or both and you’ll still be able to see everything that isn’t a debug message.
As wise network professionals always say be sure to test and validate these configurations in a lab environment before implementing anything in production. Even with the optimizations above be sure to check CPU and memory utilization before enabling any debugs.
When configuration archiving and logging is configured on an IOS router with a crypto capable image you may some rather odd commands every time the router starts up.
Here’s an example of what’s shown when you run the “show archive log config all”
1 1 console@console |access-list 199 permit icmp host 10.10.10.10 host 220.127.116.11
2 1 console@console |crypto map NiStTeSt1 10 ipsec-manual
3 1 console@console |match address 199
4 1 console@console |set peer 18.104.22.168
5 1 console@console |exit
6 1 console@console |no access-list 199
7 1 console@console |no crypto map NiStTeSt1
At first glance this appears to something nefarious, but it’s actually a test routine built in to IOS in order to meet FIPS requirements. It’s completely normal and does not impact normal operations.
Until recently I was completely unaware of the fact that DHCP server attributes on Cisco IOS support inheritance which not only shortens configurations, but decreases the chances of making typographical errors. The feature works quite simply. When Cisco IOS receives a DHCPDISCOVER message it matches the request against all matching DHCP pools based on the IP and subnet assigned to the interface that the DHCPDISCOVER message was received on. For example a router with an Ethernet interface configured as follows would look for DHCP scopes matching 192.168.1.0/24:
ip address 192.168.1.1 255.255.255.0
Basic DHCP pool configuration:
ip dhcp pool DHCP-POOL
network 192.168.1.0 /24
dns-server 22.214.171.124 126.96.36.199
I would also recommend excluding a portion of any DHCP scope for things that are going to be statically assigned. I will be excluding the following address range for this example 192.168.1.1 -> 192.168.1.10. This can be done using the following syntax:
ip dhcp pool excluded-address 192.168.1.1 192.168.1.10
If I want to extend this configuration and hand out a static DHCP reservation to my laptop should I duplicate all the settings I’ve already defined for the “DHCP-POOL” scope or just the ones that are unique? The answer is you can do either! I would recommend shortening your configuration and specifying only what you need. Here’s an example:
ip dhcp pool MYLAPTOP
host 192.168.1.100 mask 255.255.255.0
Notice that the only thing I’m specifying is the hardware-address which matches the MAC address of my laptop and the host attribute which is the static reservation I want to always give to this DHCP client.
I recently wanted to configure an HP Jetdirect print server to receive an IP address via DHCP with the server configured to hand out a static reservation. I had used the IOS DHCP pool sub-command of “client-identifier” in the past to create static reservations for Cisco IP phones and various Windows and MAC OS desktops. When I user the familiar syntax of a client-identifier formatted as 01xx.xxxx.xxxx.xx where the leading 01 indicates the Ethernet media type.
This technique has worked well for me in the past for most network devices, but it refused to assign the address I wanted to this HP Jetdirect. I did some research and found that another way to assign DHCP addresses based on MAC addresses is to use the DHCP pool sub-configuration parameter of “hardware-address xxxx.xxxx.xxxx” where the xxxx.xxxx.xxxx is the full MAC address without any prefix like we used above.
The working configuration looked like this:
ip dhcp pool HPJETDIRECT
host 10.1.0.15 255.255.255.0
The working configuration for my laptop looks like this (notice the leading 01 on the client-identifier followed by my MAC address):
ip dhcp pool NATEMACBOOKPRO
host 10.1.0.20 255.255.255.0
I was recently faced with a remote troubleshooting situation caused by a newly installed IP phone that could receive PoE power, but was unable to acquire a DHCP lease. I then remembered a useful feature of the 3750 that allows for a cable test to be executed on any copper interface. To perform this test simply run the following command:
Catalyst-3750# test cable-diagnostics tdr interface gig 2/0/2
TDR test started on interface Gi1/0/2
A TDR test can take a few seconds to run on an interface
Use ‘show cable-diagnostics tdr’ to read the TDR results.
Catalyst-3750#show cable-diagnostics tdr interface gig 2/0/2
TDR test last run on: December 30 08:31:09
Interface Speed Local pair Pair length Remote pair Pair status
——— —– ———- —————— ———– ——————–
Gi2/0/2 auto Pair A 40 +/- 4 meters N/A Open
Pair B 12 +/- 4 meters N/A Open
Pair C 40 +/- 4 meters N/A Open
Pair D 40 +/- 4 meters N/A Open
From this output it was easy to see that Pair B was not connected end-to-end and needed to be checked by our cabling vendor.
The technology in use here is TDR (time-domain reflectometry) and more information can be found at the following Wikipedia page here
More information from Cisco can be found here
I found this link while troubleshooting a possible DSL issue. Cisco routers support the following command to show DSL interface statistics “show dsl interface atm x/y” where x and y are the port and slot identifiers.
The output of this command will be similar to the following:
Alcatel 20150 chipset information
ATU-R (DS) ATU-C (US)
Modem Status: Showtime (DMTDSL_SHOWTIME)
DSL Mode: ITU G.992.1 (G.DMT)
ITU STD NUM: 0x01 0x1
Vendor ID: ‘ALCB’ ‘ALCB’
Vendor Specific: 0x0000 0x0000
Vendor Country: 0x00 0x0F
Capacity Used: 100% 98%
Noise Margin: 10.0 dB 10.0 dB
Output Power: 20.0 dBm 12.0 dBm
Attenuation: 37.5 dB 23.0 dB
Defect Status: None None
Last Fail Code: None
Selftest Result: 0x00
Interrupts: 48337 (0 spurious)
PHY Access Err: 0
Init FW: embedded
Operation FW: embedded
SW Version: 3.8131
FW Version: 0x1A04
Interleave Fast Interleave Fast
Speed (kbps): 3648 0 832 0
Reed-Solomon EC: 41799 0 11786 0
CRC Errors: 0 0 0 0
Header Errors: 0 0 0 0
Bit Errors: 0 0
BER Valid sec: 0 0
BER Invalid sec: 0 0
The information of most interest is the noise margin, output power, and attenuation as they are the metrics by which line quality is judged. You can also see the current trained DSL speed, downstream is shown first followed by upstream. In this example the downstream speed is 3648 kbps and the upstream is 832 kbps.
The following link covers some good target values for noise margin, output power, and attenuation.