High latency cause detection using multilevel dynamic analysis

The performance of applications remains a major concern to programmers. An unexpected latency can be caused by a bug or a bad program design, but it can also be caused by external factors such as resource contention or system overload. There exist tools, program profilers, that are used to detect latency. These tools, however, provide a limited view of a system's execution. For example, user space profilers can only detect slow functions but are unable to pinpoint the root causes-whether the problem comes from a slow I/O operation, interrupt, lock contention, or other problems. Kernel tracers, on the other hand, are able to collect detailed information about the operating system execution at various levels from hardware counters to system calls, disks, network I/O, etc, from which the main performance problems can be detected. In this paper, we combine user space and kernel space tracing data to understand and diagnose system performance problems and to guide users to identify the root causes. Our approach works by making a single data model by synchronizing and correlating the data gathered from different layers. We show the effectiveness of our approach by applying it to understand the latency of PHP web applications in handling web requests.