Full visibility into high-performance nets: Demand 100 percent packet capture

Full visibility into high-performance nets: Demand 100 percent packet capture

With the cost of network downtime measured in millions of dollars per hour, knowing what's going on inside the network isn't just a nice to have, it's critical.

This vendor-written tech primer has been edited to eliminate product promotion, but readers should note it will likely favour the submitter's approach.
A new class of packet based network monitoring and recording solutions are emerging that enable companies running high-speed and ultra-high-speed networks to address the issue of network blindness, a condition that exposes organisations to a raft of operational, legal, compliance and reputational risks. With the cost of network downtime measured in millions of dollars per hour, knowing what’s going on inside the network isn’t just a nice to have, it’s critical.

Today’s 10 Gigabit networks are so complex that there’s invariably duplicate traffic from badly configured switches and routers consuming bandwidth without being noticed, resulting in everything from videoconferencing falling over to failure of critical business applications. Installing a packet-based monitoring and recording fabric enables organisations to alleviate network blindness and gain visibility into network congestion issues.

It’s clear that ultra-high-speed networking is on the horizon in many industries. In a recent survey of 100 organisations in North America, 71 percent said they have made the transition to 10Gbps networking. The companies that participated included tier-two telcos, online service providers, retailers, manufacturing companies, health service providers and gaming companies, all with annual revenue of at least US$10 billion. In addition, 43 percent of the organisations surveyed said they have plans to adopt 40Gbps or 100Gbps networking.

While the organisations that participated in the survey were much bigger than most New Zealand companies, Kiwi businesses should take note as what occurs in global markets will ultimately flow on to New Zealand.

According to the senior networking, operations and security professionals surveyed, many of their incumbent network monitoring and security vendors are unable to reliably manage higher network speeds. In fact, 47 percent of the respondents believe they are missing potentially significant network events due to failing or under-performing systems. Another 65 percent of the organisations do not record network traffic for forensic analysis of network events, and 43 percent percent reported experiencing “significant difficulties” investigating and remediating network events.

Other findings of note:

- 33 percent of organisations reported experiencing some kind of data loss in the previous 12 months.

- 39 percent were unable to accurately identify what was lost.

- 42 percent admitted to having been the victim of a cyberattack in the past 12 months.

- 67 percent of those victimised by an attack admitted to having serious problems investigating the attack.

The North American market is more advanced when it comes to network speeds, as well as the adoption of network monitoring technology. Despite world-leading packet-capture technology being developed by Kiwis, very few New Zealand organisations currently have visibility of what's going on in their network.

There are a plethora of 10 Gbps-capable monitoring tools available, but most of them start to get a nasty case of network myopia as network speeds hit 3Gbps. What they claim to be able to do, and what they actually do, are turning out to be quite different things. The challenge they have is that they are unable to get packets off the wire fast enough to figure out what’s really going on. The interrupt rates of standard NICs overwhelm CPUs, causing packets to be dropped. Therefore, there is a need for dedicated and purpose-built packet capture hardware and this will become increasingly important to New Zealand organisations as the national Ultra-Fast Broadband network is developed.

In the past, simple visibility to Layer 4 was enough, but that’s no longer the case as so many applications are now Web-based. Without visibility into the application layer, organisations can’t distinguish between Skype and SAP or Dropbox and FarmVille. With visibility into the application layer organisations can really be able to start to see what’s happening and who’s responsible.

Packet-based monitoring is the only way to get really high levels of granular visibility into the network. When compared to sampled NetFlow-based tools or traditional SNMP polling, the difference in information resolution is an order of magnitude greater. It’s only by recording packet-level information that organisations can go back in time to perform forensic investigations into events. But remember, the output of a packet-based tool is only ever going to be as good as the quality of the input, and without every packet, almost any analysis that you do is pretty much pointless.

Tools that accurately record network traffic (as part of an integrated security solution) have proven themselves to be an essential way of providing post-security attack forensics, including the ability to understand what data may have been lost and how. Security teams and network operation teams need to isolate packets for forensic investigation and deal with rising expectations for network uptime and performance.

In order to select technologies that will scale to meet their needs as they move to 10Gbps networks and beyond, organisations need to start asking the network monitoring and network security solution vendors a different set of questions:

• How far (up the OSI) stack can you see?

• How fast can you capture and record packets before you start losing them?

• How can I access the raw packets that have been captured?

• Do the packets leave the datacentre when I access them (big compliance risk)?

To be acceptable, vendors need to be able to prove their claims around network performance (the standard metric is dropped packet counts at different network speeds).

Today, a new breed of distributed packet-based network monitoring and recording fabrics are emerging to help organisations solve the problem of network optimisation, and real-time anomaly detection and forensic examinations. They are reducing both the amount of time it takes to investigate any given network issue (meantime to resolution) and reducing the average skill set required to do so.

One-hundred percent packet capture-based network monitoring and recording platforms provide a common network management infrastructure that allows users to chop and change between tool sets quickly and easily, depending on the issues they need to address.

Packet-level recording is only going to become a bigger deal in New Zealand as network speeds increase and organisations look to prevent and manage future security issues through best practice solutions. What's best practice today is regulatory compliance tomorrow and countries like the United States are already looking to introduce legislation making network visibility tools mandatory. New Zealand will almost certainly follow suit – it's not a matter of if, but when - so what’s your plan?

Tim Nichols is the vice president of marketing at Endace, which provides open network monitoring and recording systems that form the basis for mission-critical network security, monitoring and measurement solutions. He recently moved from Endace's New Zealand headquarters to its new office in Silicon Valley as part of the company's increased focus on the US market.

Join the CIO New Zealand group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.

Join the newsletter!


Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.
Show Comments