Cisco IronPort Email Security Appliance Reporting Data Corruption

I recently experienced an unexpected power down of my IronPort Email Security Appliance which led to the appliance generating an hourly email containing the following:


The Critical message is:

An application fault occurred: ('aggregator/master_aggregator.py _process_export_files|605', "<class 'reporting.aggregator.master_aggregator.ExportProcessError'>", '', '[aggregator/master_aggregator.py main|272] [aggregator/master_aggregator.py watch_incoming_queue|401] [aggregator/master_aggregator.py _process_export_files|605]')
After doing some investigating I noticed that I also wasn’t seeing any of the normal reporting data in the web interface on the appliance. The resolution ended up being quite simple in my case. Here are the steps to delete the reporting database (all your reporting data will be deleted, you’ve been warned).
  1. SSH to the IronPort appliance
  2. Enter diagnostic mode by typing “diagnostic”
  3. Then enter reporting mode by typing “reporting”
  4. Finally type “deletedb”
  5. You’ll be prompted to confirm that you do indeed wish to delete the reporting database

This process took a few minutes on my lab system and then reporting started working again.

Here’s what the process looked like:


ironport.intsystek.com> diagnostic

Choose the operation you want to perform:
- RAID - Disk Verify Utility.
- DISK_USAGE - Check Disk Usage.
- NETWORK - Network Utilities.
- REPORTING - Reporting Utilities.
- TRACKING - Tracking Utilities.
- RELOAD - Reset configuration to the initial manufacturer values.
- SERVICES - Service Utilities.
[]> reporting

The reporting system is currently enabled.

Choose the operation you want to perform:
- DELETEDB - Reinitialize the reporting database.
- DISABLE - Disable the reporting system.
[]> deletedb

This command will delete all reporting data and cannot be aborted. In some instances it may take several minutes to complete. Please do not attempt a system restart until the command has
returned. Are you sure you want to continue? [N]>Y

Reseting reporting data......
The reporting system is currently enabled.

Cisco SD-WAN on IOS-XE

It’s finally here!

After many months of hard work following Cisco’s acquisition of Viptela Cisco has released IOS-XE 16.9.1 for the ISR 1000, ISR 4000 and ASR 1000 series platforms supporting Viptela SD-WAN features. Detailed documentation on the process can be found here. There are some caveats to be aware of so be sure to check the link above, but this also brings some long sought after features like T1/E1 support.

In the coming months, the team will be busily working to add existing IOS-XE capabilities to the SD-WAN version of IOS-XE.

Cisco SD-WAN Overview

With the acquisition of Viptela by Cisco in 2017 I’ve spent quite a bit of time learning about their platform and the various components that comprise their SD-WAN solution. Below is a brief overview of the solution elements and their roles in creating an SD-WAN network.

vBond – Orchestrates control and management plane. vBond provides the first point of authentication (white-list model), facilitates NAT traversal, and distributes a list of vSmarts & vManage to all vEdge routers.

vSmart – vSmart coordinates fabric discovery, distributes control plane information between vEdges, disseminates date plane data plane and application-aware routing policies to the vEdge routers, implements control plane policies (including service chaining, multi-topology, and multi-hop), dramatically reduces control plane complexity.

vEdge – vEdge is a full-featured WAN router supporting VRRP, OSPF, and BGP. vEdge provides a secure data plane between other vEdge routers and establishes secure control plane connections with the vSmart controller. Implements data plane and application-aware routing policies and exports information and statistics. Support for zero-touch provisioning. vEdge is available in both physical and virtual form factors.

vManage – vManage is the management plane for Cisco SD-WAN and acts as the user interface for initial configuring and ongoing maintenance activities. vManage supports multitenancy, centralized provisioning, policies and templates, troubleshooting, monitoring, and software upgrades. vManage provides a rich set of REST and NETCONF APIs.

Terminology

  • Overlay Management Protocol (OMP) – Control plane protocol distributing reachability, security, and policies throughout the fabric
  • Transport Locator (TLOC) – Transport attachment point and next hop route attribute
  • Color – Control plane tag used for IPSec tunnel establishment logic
  • Site ID – Unique per-site numeric identifier used in policy application
  • System IP – Unique per-device (vEdge and controllers) IPv4 notation identifier. Also used as Router ID for BGP and OSPF
  • Organization Name – Overlay identifier common to all elements of the fabric
  • VPN (VRF) – Device-level and network-level segmentation

This is a very basic introduction to the pieces and parts of the solution. I plan to follow this up with additional content on how these pieces work together to provide a flexible architecture allowing for nearly any network topology to be created.

Cisco DNA Center Supported Hardware

I get asked many times what hardware and software versions are required to integrate with Cisco DNA Center. In addition, there are a variety of capabilities of DNA Center including network provisioning, software management, network visibility, and segment so it can make it challenging to know which network components are supported with which DNA features. Oh and don’t forget there are software version requirments 😉 Here’s a link here that provides this information.

Cisco ISR 4K Product Numbering

Here’s a secret decoder ring for the part numbers of the 4000 series of the Cisco Integrated Services Routers.

First digit = the family, all are 4

Second digit = the sub-family with 4 (highest performance), 3 (middle performance), and 2 (lowest performance)

The third digit = total number of slots, the sum of NIM and SM

The fourth digit = 1, identifying the first in that series. Allows for incrementing for the subsequent platforms in the series.

Here’s a link to the ISR 4000 model platform comparison 

Video Endpoints and Cisco Spark

I’ve recently spent more time testing video endpoints with Cisco Spark (SX10, SX20, Spark Board, Spark Room Kit, DX80, etc.) with my customers and have run in to several things that I think many others probably encounter as well.

  1. Check support Spark endpoints – https://help.webex.com/docs/DOC-4205
  2. Is your endpoint running the right software version to avoid certificate validation errors when attempting to register for the first time? (To activate your room device on Cisco Spark, the device must run software version CE8.2.0 or later.) https://help.webex.com/docs/DOC-7709
    1. To upgrade the codec it’s a simple process of downloaded CE software 8.2 or higher and then logging in to the web interface of the codec to complete the manual upgrade.
  3. If you plan to use a Touch 10 with any endpoint you must pair it to the codec BEFORE you register it to Spark – https://help.webex.com/docs/DOC-11657

Before I upgraded my SX20 to CE 8.2 I was seeing errors in the logs similar to

2017-12-09T09:51:08.966-06:00 a8 appl[1796]: 762.54 Wx2Http W: HTTP(2) Error: NetworkError (Peer certificate cannot be authenticated with given CA certificates)

The basic issue is that in releases of CE software prior to 8.2 the necessary CA certificates were not installed so the certificates presented by the Cisco Spark registration system weren’t able to be validated.

Cisco CWS and OpenDNS Data Center Locations

Cisco Cloud Web Security (CWS) and OpenDNS both provide cloud based security services. CWS offers an HTTP/HTTPS proxy and OpenDNS provides security and visibility at the DNS resolution layer. I’ve been asked many times where both CWS and OpenDNS host their services as this can make a big impact in end user experience if the hosting location is far away from the user and could lead to high latency and a lousy experience.

CWS Proxy Location and Status Page: http://servicestatus.sco.cisco.com/status

OpenDNS Location and Status Page: https://www.opendns.com/data-center-locations/

Cisco Tools Link

I regularly share useful links with customers and colleagues and often find that this page is a great starting point to explore some of the web tools Cisco has available http://www.cisco.com/c/en/us/support/web/tools-catalog.html

Some of these tools include the Cisco Power Calculator, Cisco Feature Navigator, Cisco IOS to NX-OS configuration converter, and many others. Give it a click and explore some tools you likely didn’t even know existed.

Communications Manager 11.5 Deprecated Phones

With the release of Cisco Unified Communications Manager (CallManager) version 11.5 support was removed for some of the oldest IP phone models. Support was removed for these phones as they do not support the latest security features that Cisco is working to standardize.

The following models are prevented from registering in version 11.5:

  • Cisco IP Phone 12 S
  • Cisco IP Phone 12 SP
  • Cisco IP Phone 12 SP+
  • Cisco IP Phone 30 SP+
  • Cisco IP Phone 30 VIP
  • Cisco Unified IP Phone 7902G
  • Cisco Unified IP Phone 7905G
  • Cisco Unified IP Phone 7910
  • Cisco Unified IP Phone 7910G
  • Cisco Unified IP Phone 7910+SW
  • Cisco Unified IP Phone 7910G+SW
  • Cisco Unified IP Phone 7912G
  • Cisco Unified Wireless IP Phone 7920
  • Cisco Unified IP Conference Station 7935

For more background on this check out the following Cisco Field Notice

Cisco Universal APs

A brief history lesson:

Prior to the announcement from Cisco of “Universal Access Points” you had to select an access point based on which regulatory domain it would operate within. Regulatory domains are simply places in the world with certain laws and regulations pertaining to radio frequency devices (mobile phones, radios, access points, etc.). In the world of Cisco this meant that an access point for use in Germany would have a different part number than one used in the United States. This led to much frustration and confusion.

The good stuff:

Cisco has announced Universal AP’s which are a single part number per access point model (2700, 3700, etc.) rather than the myriad of part numbers aligned to each and every regulatory domain. The details on how this works can be found in the following blog post.