Sheffield Hallam University
Module Code and Title: Systems Application Security
Assignment Title: |
Buffer Overflow and NIDPS |
Assessment Tutor: |
Dr David Day |
Student name: |
Evgeny Danshin |
Weighting Towards Module Grade: |
50% |
Word Limit: |
3000 Words |
Date Set: |
29-10-12 |
Hand In Date: |
6-1-2012 |
TABLE OF CONTENT
Abstract............................................................................................................................................3
1 Introduction...................................................................................................................................4
2 Buffer overflow protection techniques.........................................................................................5
2.1 Data Execution Prevention (DEP).................................................................................5
2.2 Canaries (Canary words)................................................................................................6
2.3 Address Space Layout Randomization (ASLR)............................................................8
3 Intrusion detection and prevention systems (IDS/IPS).................................................................9
3.1 Signature-based IDS/IPS................................................................................................9
3.2 Anomaly-based IDS/IPS..............................................................................................10
4 Novel approach in buffer overflow detection.............................................................................11
5 Summary.....................................................................................................................................12
ABSTRACT
A poorly designed [1] software or hardware's program source code can contain a small inappropriate placed character or multiple characters that are improperly processed and as a result it can lead to buffer overflow attacks. Buffer overflow vulnerabilities might be found in local programs, operation system applications, services, remote software such as DNS, FTP, E-mail and Web servers. These buffer overflow vulnerabilities allow for the implementation of a various range of attacks and they can the cause for some of the most serious damages in a company's network.
This work presents a few techniques to protect systems and applications against buffer overflow attacks and these techniques do not require modifications of source code. Buffer overflow protection methods and IDS/IPS demonstrate an advanced approach in prevention and avoidance of buffer overflow and worm propagation attacks.
Keywords: buffer overflow attacks, randomization, worm propagation, intrusion detection and prevention systems, malicious code, stack, heap.
1 Introduction
Historically, buffer overflow is the most common vulnerability used by malicious software to infect; compromise and destroy a system or an application. Worms which may be a part of a malicious code are the most "active users" of buffer overflow vulnerabilities than viruses. By using buffer overflow a worm can replicate and spread itself through a network thus infecting more hardware recipients. Buffer overflow vulnerabilities created by worm propagation may allow an attacker to take control over a system, crash an application, escalate privileges, create a "backdoor" and etc. Moreover, if a worm can infect one PC using the heap or buffer overflow, X thousand PCs can be infected and damaged - if they have the same features, software or security configurations [2].
In contrast to the Morris worm that had an aim to define how big the Internet is, modern worms have a various range of goals. In general, a worm's goal relies on the attacker's aims and it can have unpredictable results and effects. For example, worms can create bot-nets to spread spam or implement the dDoS attacks, steal an account's details or login credentials, gain an authorized access, take control over industrial or SCADA systems (e.g. Stuxnet, Duqu). Considering what was stated above we can clearly see that absolutely all systems and applications need to be protected against buffer overflow attacks and worm propagation. It is not surprisingly that software developers [3] are constantly introducing mechanisms which allow for avoiding and preventing buffer overflow attacks. Some advanced techniques, for instance - Program Shepherding [4], Canaries, Data Execution Prevention and Address Space Layout Randomization - can protect system's stack and prevent code injection attacks, format string execution and, additionally, attacks that use vulnerabilities in the heap structure management.
In spite of the fact that software developers patch any vulnerabilities found in their products and the use of stack overflow mechanisms, buffer overflow attacks are still used to a high degree (Figure 1) [5].
Figure 1: Buffer overflow tendency in accordance with NIST database
Moreover, today's stack overflow vulnerabilities tend to have a "zero-day" character and it becomes increasingly difficult to protect a system against such vulnerabilities, for this reason, it is necessary to apply an additional mechanism for detecting and preventing buffer overflow attacks and worm propagation. Such mechanisms might be antivirus programs, firewalls, host or network -based intrusion detection (and/or prevention) systems.
2 Buffer overflow protection techniques
Buffer overflow attacks may be used to overwrite data in memory in order to provoke direct changes in execution flow process in accordance with attacker's goals [6]. However, modern buffer protection mechanisms allow for prevention and avoidance of such attacks. DEP, ASLR and Canaries are the modern buffer protection approaches which will be considered below.
2.1 Data Execution Prevention (dep)
DEP is a security feature integrated into Microsoft Windows operation systems (e.g. - XP SP2, Vista, Windows 7 and Windows Server 2003), Linux and MAC OS. In Microsoft Windows, DEP is a default option set only to protect its OS applications and services. On the other hand, DEP is a set of hardware and software technologies that perform an additional check on the memory and prevent malicious code from executing in memory data pages. Specifically, DEP can block the spreading of malware programs trying to execute its code in regions of the system memory which are reserved for authorized programs (e.g. - legitimate system applications and services). Additionally, DEP prevents code from executing in the heap, stack and memory pools. Furthermore, DEP does not allow the execution of programs that take advantage of exception-handling mechanisms (especially in Windows OS).
By detecting that a program uses the system memory incorrectly [7] or attempts to execute some code from a protected data page, DEP initiates a memory access violation and if this exception can not to be processed - terminates the calling of a process, making an enforced closure of a program and displays an "alert" message.
Technically, execution of injected code into stack, heap or protected memory pages generates an exception. If an application needs to start executing a code from a protected memory page, the application gets an exception "STATUS_ACCESS_VIOLATION" which is generated by DEP. Initially, if an application must execute a code from non-executable memory page, the correct set of virtual memory attributes must be set up and allocated for secure code executing. Some examples of these attributes [8] are PAGE_EXECUTION, PAGE_EXECUTE_READ and PAGE_EXECUTE_WRITEREAD. More commonly, DEP is configured in accordance with the no-execute memory page protection policy's settings in the boot configuration data (system boot).
An application can obtain the policy's settings [7] by calling GetSystemDEPPolicy function and by relying on these settings the application is able to modify the current DEP attributes for the current process by calling SetProcessDEPPolicy function which possibly creates a disadvantage of DEP's approach.
As mentioned, DEP has two realization approaches: software enforced DEP and hardware enforced DEP. Software enforced DEP must be supported by operation systems and protects only user-mode processes. Hardware enforced DEP provides a set of instructions to the CPU to make some certain memory pages non-executable. Technically, hardware enforces DEP specifies a bit in the page table entry that points the system to prevent code from being run from a virtual memory page that must contain only non-executable data.