The Apache web server is vulnerable to various critical failures, some of which are beyond the control of the web server itself due to the ability to extend the functionality of the core web server via loadable modules. Regardless of the origin of a problem, certain diagnostic tasks are critical to finding the root cause of the problem. Unfortunately the ease, or even the possibility, of performing these tasks varies by platform, and often between versions of the platform.
This paper will define a standard set of diagnostic tasks necessary for efficient diagnosis of critical web server failures. Additionally, several reports included with the paper provide detailed information on performing these tasks on specific platforms. It is hoped that independent reports will follow, based on the reports provided initially.
Beyond showing how a web server configuration change or OS upgrade or platform change can enhance serviceability, the detailed mechanics shown in the report can serve as a tutorial to users of that platform.
Certain configuration decisions which improve web server performance can impede serviceability. It is important to know what these decisions are and to what extent the performance or serviceability is impeded when making a choice.
A benchmark report is not expected to be a detailed performance analysis, but a comparison of memory use and static file performance for best-serviceability vs. worst-serviceability would be very useful.
This task is critical for diagnosing a web server hang. With a traceback from all threads in all web server processes, the area of concern can often be quickly identified.
This task is useful when there is a looping or high CPU web server process. Getting tracebacks at intervals from all threads in the process can be used to identify the area of concern.
It is important to be able to display the storage for critical Apache data structures, such as the request_rec structure for a request being processed.
(hypothetical but detailed example to follow)
This task is critical for diagnosing any type of crash.
It is important to be able to display the storage for critical Apache data structures, such as the request_rec structure for the request being processed by the thread that crashed.
(hypothetical but detailed example to follow)
This task is helpful when diagnosing hangs, reproducible crashes, high-CPU problems, and many other less critical problems.
A key part of this is being able to trace the startup of child processes, such as CGI scripts.
(hypothetical but detailed example to follow)
This task is helpful when diagnosing various reproducible problems. Even if no symbolic information is built, it is important to be able to set breakpoints on functions and run to them, in order to determine which paths are taken.
(hypothetical but detailed example to follow)
Any benchmark reports will be specific to the web server version, the MPM (if a choice is possible), the compiler (if a choice is possible), and the OS version. For some platforms, kernel and libc changes during the life of a specific OS version can affect serviceability. Note that reports on web servers based on Apache can be useful as well. An idea of specificity of the reports is shown by the following examples: