Defining Risk

What are the scariest three words an intrusion analyst is likely to hear?

We can't reasonably manage risk if we don't know what risk is. Risk occurs in the domain of uncertainty. If there is no uncertainty, there is no risk. Jumping out of an airplane two miles up without a parachute isn't risky; it is suicide. For such an action, there is a nearly 1.0 probability you will go splat when you hit the ground, or an almost 0.0 probability you will survive. However, there is also risk to jumping out of perfectly good airplanes with parachutes, as several skydivers discover each year.

Let's apply this concept to router protection filters. In many cases, these filters are connection events—that is, they are port number based. If we see a TCP connection at port 25, we identify it as sendmail and take whatever action is prescribed. However, any service can actually run at any port. There is the uncertainty; there is a risk that we will make the wrong decision. With the ephemeral ports (above 1024), this happens often. This uncertainty, coupled with the fact that an adverse action could be exploited (a service we intended to block could penetrate our site), leads to a risk. This is one reason many security professionals think that a filtering router does not serve as a firewall.

An intrusion-detection analyst needs to know the degree of uncertainty for specific filters. As an example, SYN flood filters often have a high degree of uncertainty. If an intrusion-detection analyst continues to report these, there is the potential for an adverse action. The CIRT might begin to trivialize this analyst's reports. Therefore, a filter's degree of uncertainty can result in risk to the analyst and the organization, especially in high-profile cases. Conversely, the expert analyst knows the conditions in which a filter is likely to perform well and also the conditions that lead to failure. These analysts develop the ability to "read between the lines."

Perhaps, the simple issue of reputation doesn't grab you. The same problem, uncertainty of filters, gets more interesting if a site employs automated response techniques.

I want to briefly mention one more potential adverse result of uncertainty with intrusiondetection filters. Several commercial IDS vendors provide lists of their filters. Sometimes, they rate their filters by their probability of producing a false positive and perhaps list conditions known to cause the false positives. This is a great service to the analyst. What if a company lists some of its filters as not having any chance of a false positive—that is, there should be no uncertainty, therefore there is no risk. Then, you dig in and find several of these filters do generate false positives. That realization can undermine your confidence in the company. I know; it happened to me. In fact, I started building test cases for the filters that according to the literature had no chance of a false positive and found several other filters had flaws. Well this really bugged me. Why say it doesn't error if it does? Then, I remembered that I had been issued a brain to keep my heart in check. Why get mad at this company when they have the most complete filter documentation of any commercial IDS? So, I just updated my copy of the filter documentation and sent them traces of my test cases. What do I get for my effort? I know a lot more about which detects to be uncertain about and the conditions likely to cause the filters to error and generate a false positive.