A vulnerability rated as a low risk this morning could turn into your worst nightmare tonight. To meet the ever-increasing speed with which exploits are written and propagated, traditional network-based vulnerability scanners have morphed into more full-scale vulnerability management products. In our latest Clear Choice Test of eight products -- assessing their accuracy in pinpointing holes in the network and their usefulness in addressing those vulnerabilities -- we found vulnerability identification success rates are still low across the board and the scans can wreak havoc on wireless access points. They also can do damage to some printers, and can suck up network bandwidth and CPU utilization on target machines.
Vulnerability remediation and tracking are the major management features added to these products since our last test, providing mechanisms to assign and alert administrators to new vulnerabilities. These additions range from providing vulnerability remediation information to offering full-blown ticketing systems that automatically verify if an issue has been fixed.
Business analysis features have been included in many products. With this functionality, assets can be given values -- in terms of cash or business-critical value. How vulnerabilities potentially could affect business and give management a more accurate picture of the company's overall security posture can be correlated. A critical vulnerability on the core, Internet-facing system that generates revenue should be treated differently than a critical vulnerability on a system inside a test network that's isolated from the rest of the company, for example.
The companies that provided products and/or services for this test are Lockdown Networks Inc., nCircle Network Security Inc., PredatorWatch Inc., Qualys Inc., StillSecure, Tenable Network Security Inc., Trace Security Inc. and Visionael Corp. EEye Digital Security, Internet Security Systems Inc., Foundstone Inc., NetIQ Corp., Bindview Development Corp. and Harris Corp. declined. We also tested Citadel Security Software Inc.'s Hercules and Sunbelt Software Distribution Inc., but because they offer no scanning module or management features, respectively, we could not directly compare them.
Qualys' QualysGuard is our Clear Choice winner based on its accuracy and strong management capabilities. NCircle's IP360 comes in second, only slightly trailing Qualys in vulnerability identification and general ease of use. Visionael Enterprise Security Protector and Lockdown's Auditor also rose to the top based on their developing management capabilities.
QualysGuard -- one of the two vulnerability assessment services we tested -- has a 1U appliance that sits on your network and lets Qualys scan your internal subnets. Setup is easy, and the quick start guide will have you scanning in no time. Because it is provided as a service, the Qualys team seamlessly adds the vulnerability checks.
Our discovery assessment focuses on how well the products find and identify systems, system software and services running on the network. Our accuracy measurement takes into account how well the product identified vulnerabilities that existed on a sample of lab systems.
Qualys scored highest in our operating system identification checks and was the only product to correctly identify the wireless access point. It performed as well as any of the other products in the vulnerability accuracy tests, but still reported some false positives and false negatives. It did perform strongest among the products in identifying Windows system vulnerabilities, though.
Scan impact was low from a network perspective, but we did need to restart a Red Hat Enterprise system that became completely unresponsive after the scan.
Overall, QualysGuard is very flexible and easy to use. IT staff and/or corporate executives can be given varying levels of access to system groups and reports. Scan and report templates provide flexibility in the types of checks that are performed and how the results are viewed.
Remediation policies can be configured to automatically assign tickets in the Qualys ticketing system to defined individuals based on scan results. Qualys could improve on remediation if it added some preemptive notification mechanism to tell IT folks they have been assigned a remediation task.
In terms of providing some business analysis capabilities, Qualys lets you rank assets in terms of how critical they are to your business. A score is then provided in the summary based on your overall exposure level that can be weighted based on how critical the vulnerable asset might be.
One of the best features of QualysGuard is its mapping functionality, which provides a graphical representation of all the devices it discovers on your network. You can drill down on the map to identify the operating systems and services running on these devices, but can't see information on identified vulnerabilities from this vantage point. In addition to the mapping, we'd also like to see some sort of overview console that provides high-level information on the state of vulnerabilities on the network.
NCircle IP360 6.2
NCircle provided a central reporting server, VnE Manager, and scanning point, Device Profiler. With this tiered approach, nCircle runs in a more distributed model than some of the other products tested.
The IP360 provides the best business impact and risk-rating features, offering unparalleled levels of detail. Users can provide asset values for each host and calculate risk scores for each system based on the asset value. This value is a quantitative number, generally dollars, of the value of the asset to the company. As a consequence of this increased functionality, it is not as easy to use as some of the other products tested.
For system discovery, nCircle uses dynamic host discovery, its technique for continuously evaluating environments for new systems on the network. After running on the network for a few minutes, the system had found all the devices in the lab.
For operating system identification, nCircle joined Qualys as the only products to correctly identify the Cisco VPN Concentrator. But it missed a few key systems that most other products identified, including the FreeBSD 5.2 server and the Quantum Snap Server.
For vulnerability identification, nCircle consistently reported the smallest number of vulnerabilities, minimizing false positives, but potentially introducing some false negatives as well.
While nCircle's scan results might appear to include false negatives, following the remediation guidelines for identified vulnerabilities will address the known vulnerabilities in the system.
NCircle accrued the lowest network and system impact, with no identified issues or spikes in network traffic or CPU utilization.
One unique feature of the IP360 is its continuous scanning mode, which provides non-intrusive, back-to-back scans of the whole network or of only select segments.
This is ideal for critical systems or networks that need to be monitored at all times. NCircle provides a classic scanning model of scheduling scans, grouping systems and providing detailed user access.
NCircle takes a different approach in providing vulnerability remediation information. For the sample of vulnerabilities we reviewed, nCircle provided links to patched versions or specific patches for a variety of operating systems. In a few instances, the vulnerability remediation information did not match the specific vulnerability identified, although following the recommended course of action would in most cases have fixed the vulnerability because one patch would fix several issues.
Visionael Enterprise Security Protector
Visionael uses Nessus as its underlying scanning engine and focuses on providing some of the best vulnerability management functionality, such as a customizable portal for viewing security trending information.
Installing Visionael on Red Hat Enterprise worked well, although we'd like to see Visionael better secure the assessment server by default rather than leaving that up to the systems administrator.
Upon initial logon, Visionael provides the best portal functionality, allowing customization for each user and quick views of identified vulnerabilities, current risk level, trending and trouble ticket status.
There were a few issues in terms of system identification for the hosts on the lab network, namely system identification was not happening as we configured it. Working with support, we enabled the detailed operating system checks and reduced the concurrent threads from 200 to 20. With these changes in place, we got operating systems identification results, but they were not as detailed as we would like to see. For example, all Windows systems, regardless of version, reported back as "Windows."
For network and system impact, Visionael is quite loud. The scan locked up the wireless access point, bluescreened a Windows XP system and consumed 30 percent of the CPU on the monitored target system.
Viewing individual scan results provides an overview of identified vulnerabilities, with a breakout summary of the SANS Top 20, which is unique to this product. We would like to be able to drill down into the report directly from the vulnerability numbers reported in this overview screen.
The reporting module provides a wizard to create custom reports. But the customization options are so abundant that they are almost overwhelming.
The ticketing system is very strong, although tickets only can be auto-assigned for SANS20 or high-level vulnerabilities, which is fine if you prefer to do more detailed analysis on the other levels of vulnerabilities before tasking them out.
Visionael can auto-remediate identified vulnerabilities, but this functionality was not enabled in the license we received for testing.
For business analysis, Visionael provides strong trending information, executive reports and business rank, based on assigning systems one of four levels depending on how critical it is to your business.
Lockdown Auditor 3.0
Auditor provides the most intuitive management features, but lags a bit with its scanning engine.
Lockdown's 1U scanning appliance is the most intrusive, utilizing 40M bytes of network bandwidth and on average 40 percent CPU on the target system. Administrators do not have any options to change scan configuration settings.
It performed fairly well in terms of operating system identification, missing only two devices and not clearly distinguishing a few Windows versions. In terms of accuracy, Lockdown Auditor hit and/or missed to the same degree as most competitors.
Scan reporting is strong, providing a summary of identified vulnerabilities. We liked the job queue functionality, which shows the percent completion of each system being analyzed in the current scan.
In terms of management, Lockdown's user interface is the best of the products tested, combining graphics and a workflow very effectively. While it does not contain a specific portal, the initial logon window defaults to the report section. This provides an online version of the Executive Summary, which contains an overview of scan results and trending information.
For business impact analysis, the system provides a rating number based on scan results. You can assign critical values to specific systems, which then will be weighted more heavily when calculating the overall rating.
Lockdown's vulnerability notification capability was excellent and lets administrators define policies that trigger alerts. You can configure a policy that sends a page or SNMP trap and opens a ticket if a specific port was opened on a system.
Other unique lockdown features are the ability to encrypt e-mail that has account information using gpg and the ability to authenticate users against a corporate Lightweight Directory Access Protocol directory.
For remediation, it provides an excellent breakdown of problem, solution and resolution for identified vulnerabilities, including Common Vulnerabilities and Exposures numbers and links to related security advisories or remediation steps.
StillSecure VAM 4.0
StillSecure's VAM was a solid performer in both scanning and management. Setting up VAM on the vendor-supplied server was simple. The software automatically installed on the system from a CD when it booted up.
StillSecure's user interface is not intuitive and is difficult to navigate. The screen is often cluttered, making it difficult to identify specific information or tasks.
However, StillSecure performed fairly well in scan tests. In terms of operating system identification, it only missed a few of the network devices, such as the NetScreen-100 and the Cisco VPN Concentrator. It incorrectly identified the wireless access point as an ATM switch.
It performed well on scan impact analysis, providing no noticeable issues.
The scan report provides a generic list of vulnerability titles that you can drill down into for more details, although report navigation is a bit cumbersome.
VAM includes a robust ticketing system for tracking vulnerabilities, but it doesn't provide any business impact analysis functionality.
Reporting functionality in StillSecure is functional and offers some trending and executive report facilities. But it doesn't provide all the flexibility in other products.
Tenable Lightning 2.5, NeVo 2.0 and Nessus 2.0
The primary author of Nessus founded Tenable, so it's no surprise that Tenable's suite of products taps deeply into the Nessus base code to yield some unique features, such as Unix authentication for local vulnerability checks.
In addition to the Nessus active scanning engine, Tenable's Lightning product is the management console, and its NeVo product is the passive vulnerability scanner.
The Lightning/Nessus combination provides a very robust vulnerability search mechanism with the ability to search databases of identified vulnerabilities on almost any criteria. However, it doesn't include any mechanisms to control trouble ticketing or remediation functions or offer any business impact analysis.
In scanning tests, Tenable performed fairly well. In terms of operating system identification, it missed some of the network devices but performed the best of the Nessus-based products on scan accuracy. The overall scan impact was minimal.
The reporting module inside the console automatically generates a few reports, such as 30-, 60- and 90-day vulnerability details. You also can create custom reports from a selection of filter criteria.
One area that could be improved is in its vulnerability plug-in descriptions, which is what Nessus uses for its vulnerability checks. When trying to view the checks Nessus scans run, the drop-down box of signatures lists them by a nondescript ID, such as "CSCdp58462." These aren't very useful when trying to figure out what check is being performed.
Overall, Tenable has a very strong foundation, and we would like its vulnerability management functionality improved.
TraceAudit is delivered as a service but also includes an ISO - a file that contains a complete image of a disk - and a hardened version of Red Hat that installs on a network system and provides TraceSecurity access to internal systems for scanning.
The results of your scans are sent over an encrypted channel to TraceSecurity's servers, with the results available from the Web-based management interface. The scan results are not encrypted when stored on TraceSecurity's servers, though.
Reports or general scan results do not include operating system information. Accuracy of scan results was in line with the rest of the products.
Scan impact was fairly low on the network, but it did cause a core dump on the HP print server.
The user interface is not intuitive from the start, but it becomes more usable once you understand the workflow. One oddity after the initial scan was that you couldn't view the results until you associate the systems to a group. If you launch a scan, the results should be automatically viewable through a default group or similar architecture.
TraceAudit doesn't have a formal ticketing system, but the company recommends customers use the system grouping functionality to manage vulnerabilities. While this might work for some organizations, it doesn't provide the full accountability and tracking we prefer in an enterprise vulnerability management product. It also doesn't include an overview portal/summary for reporting or the ability to filter out SANS 20 results. This product also doesn't offer any business impact analysis.
PredatorWatch Auditor 128 2.2
Predator Watch was a bit above average in its scanning features but really needs to bulk up its management capabilities to compete with this lot.
PredatorWatch runs on a small, square appliance that easily could fit on a
desk. The software is available on a 1U appliance.
In our scanning tests, PredatorWatch was on par with several other products in that it missed some network devices and didn't differentiate between some versions of Windows. Scans locked up the wireless access point.
One function we couldn't get to work was launching an immediate scan. We would select a scan to start, but it never began. Scans would start fine when scheduled.
The management GUI is slow and unresponsive. We had requests processing for 30 seconds or more before the page hit the screen. The GUI is also difficult to navigate and not intuitive.
PredatorWatch doesn't provide a ticketing system, business impact analysis and user management functionality. It also only provides three reports - executive, management and administration - for each scan results set. Limited trending information is included from the previous scan, but we would like to see more custom report options and trending information.
PredatorWatch offers a unique feature with its compliance reports. Based on identified vulnerabilities, administrators can run reports to help identify weaknesses in Sarbanes-Oxley, Health Insurance Portability and Accountability Act and ISO 17799 compliance.
It's good news that vulnerability assessment tools are embracing vulnerability management functionality. Ticketing systems, business impact analysis, console dashboards and custom reporting options quickly are becoming standard features.
However, the number of false positive and false negative scan results still points out that vendors need to continue to refine their scanning engines. Users will benefit from strong management tools only if vendors make sure the vulnerabilities bubbling up to the management tools are complete, accurate and do not affect target system functionality.
How we did it
We set up the test network as a flat network, with all devices working off one Cisco Catalyst 3500 switch. This network had 31 devices running on it, varying from a wireless access point, NetScreen firewall, Cisco VPN Concentrator, HP Print Server, Snap Server, FreeBSD 5.2 server, and various flavors of Windows and Red Hat Linux on various-sized servers and workstations.
We installed all products under test according to vendor specifications and on vendor-supplied hardware where applicable.
To begin testing, we configured a "Lab" group in each product. We then performed device discovery, including operating system discovery options, where appropriate, to make sure the product could find everything on the network.
After discovery, we ran a full "safe" scan from each product, enabling local authentication checks when offered. We watched any status information the product provided during the scan and reviewed the results as soon as the scan completed. We also checked all servers being scanned for functionality to see if the scan had affected performance. We monitored a Red Hat Enterprise server with SNMP and Multi Router Traffic Grapher during each scan to graph CPU utilization, network bandwidth and TCP connection loads.
We then added a new server to the network, created a new user in the product, defined a SANS 20 scan template, defined asset classification and set up automatic ticket assignment with notification. Once these tasked were completed, we launched a second scan based on the new settings and policies.
To analyze the scan results, we compiled a chart of identified operating systems for 20 devices on the lab network, including NetScreen-100 firewall; Quantum Snap Server; HP Print Server; Belkin's KVM over IP appliance; Hawking Print Server; Avaya/Orinoco Wireless Access Point; FreeBSD 5.2; Solaris 8; Cisco Catalyst 3500 switch; Cisco VPN Concentrator; and several systems running various versions of Windows and Red Hat at different patch levels.
For vulnerability identification testing, we compiled the vulnerability results across all products for four systems running default installation of the following operating systems - Windows XP Professional, Windows 2003 Enterprise Server, Red Hat Fedora Core 2 and Red Hat 6.2.
To test reporting, we attempted to generate four reports - a SANS 20 report, a critical vulnerability report, a new systems report and an executive summary. We also attempted to export a report into the available format, such as PDF, and review any summary information provided by a dashboard or console in the product.
Getting a second scanning opinion
Companies often want to run multiple scanners against their systems for completeness, but the cost of that second opinion can be prohibitive. If you are in the market for a less expensive secondary scanner, you might want to take a look at SunBelt Network Security Inspector.
SunBelt -- which acts as an OEM to the Harris vulnerability database -- sets its price at about US$1,500 per administrator, not licensed by IP address like most of its competitors. Based on this price, relative scanning speed and ease of use, SunBelt could be used as a secondary point scanner for quick scans, but based on our accuracy testing, we don't recommend that you rely on it as a primary vulnerability assessment tool.
While the scan impact was minimal, SunBelt's product did not identify all the devices in our operating system identification test and provided limited information, such as "Linux" or "Windows" for the systems it identified.
Vulnerability scan results also were out of line with the other products tested. SunBelt would sometimes identify different vulnerabilities than the other products, although the remediation tasks would result in the issue being fixed. On the Red Hat 6.2 system, though, it identified 14 vulnerabilities that were false positives and not reported by any of the other products tested.
The SunBelt GUI is very strong. The product is fast and very easy to use. Reports are provided through a Crystal Reports engine, but they are only exportable by printing to PDF.
Because this product is designed to be a point scanner, scan scheduling and options configuration are not included. Vulnerability updates are also only performed manually.
Citadel focuses on vulnerability remediation
While vulnerability assessment products are moving solidly in the direction of vulnerability management, Citadel Security Software takes that shift one step further with its focus on automatic remediation.
Citadel's Hercules is an automated vulnerability remediation tool that pulls results from vulnerability assessment scanners, applies its own detailed knowledge on remediation procedures across a variety of platforms, and then either recommends remediation steps or makes them happen.
The main focus of the product is to run remediation tasks, which could be changing file permissions, changing a password, setting an operating system configuration option or installing a patch. Administrators also can script their own fixes if they desire, and full rollback options are available.
Agents are installed on target systems that listen for information from a Citadel central server on how to handle remediation tasks. Remediation scripts also running on the target systems are responsible for fixing identified vulnerabilities.
You can use Hercules without agents, relying on Secure Shell, Windows Services or HTTP/Secure-HTTP for communications to the system from the central server.
Citadel also helps you use its remediation tool to further policy compliance. You can use Hercules to define policy groups. When scan results are imported, Hercules automatically pushes out fixes to machines that don't follow group policy if that is how the administrator has set up the script.
While auto-remediation sounds great, system administrators are wary. The remediation is based on the vulnerability assessment results, so you need to ensure those results are accurate for auto-remediation to be successful.
Andress is president of ArcSec Technologies, a security company focusing on product reviews and analysis. She can be reached at email@example.com -- Network World (US)