Traps and Pitfalls: Practical Problems in System Call Interposition Based Security Tools.

TLDR

System call interposition is a powerful method for regulating and monitoring application behavior, yet it carries many pitfalls that can allow tools to be easily circumvented. The authors aim to illuminate the pitfalls of system call interposition by sharing lessons learned from developing their own sandboxing tool. They identify problems such as incorrect OS semantics replication, overlooked indirect resource paths, race conditions, improper interface subsetting, and denial side effects, and then propose practical solutions and general principles to avoid these difficulties.

Abstract

System call interposition is a powerful method for regulating and monitoring application behavior. In recent years, a wide variety of security tools have been developed that use this technique. This approach brings with it a host of pitfalls for the unwary implementer that if overlooked can allow his tool to be easily circumvented. To shed light on these problems, we present the lessons we learned in the course of several design and implementation cycles with our own system call interposition-based sandboxing tool. We first present some of the problems and pitfalls we encountered, including incorrectly replicating OS semantics, overlooking indirect paths to resources, race conditions, incorrectly subsetting a complex interface, and side effects of denying system calls. We then present some practical solutions to these problems, and provide general principles for avoiding the difficulties we encountered.