A global-state-triggered fault injector for distributed system evaluation

TitleA global-state-triggered fault injector for distributed system evaluation
Publication TypeJournal Articles
Year of Publication2004
AuthorsChandra R, Lefever RM, Joshi KR, Cukier M, Sanders WH
JournalParallel and Distributed Systems, IEEE Transactions on
Pagination593 - 605
Date Published2004/07//
ISBN Number1045-9219
Keywordsdistributed processing, distributed system evaluation, fault tolerant computing, global-state-based fault injection mechanism, Loki, offline clock synchronization, performance evaluation, post-runtime analysis, Synchronisation, system recovery, user-specified performance

Validation of the dependability of distributed systems via fault injection is gaining importance because distributed systems are being increasingly used in environments with high dependability requirements. The fact that distributed systems can fail in subtle ways that depend on the state of multiple parts of the system suggests that a global-state-based fault injection mechanism should be used to validate them. However, global-state-based fault injection is challenging since it is very difficult in practice to maintain the global state of a distributed system at runtime with minimal intrusion into the system execution. We present Loki, a global-state-based fault injector, which has been designed with the goals of low intrusion, high precision, and high flexibility. Loki achieves these goals by utilizing the ideas of partial view of global state, optimistic synchronization, and offline analysis. In Loki, faults are injected based on a partial, view of the global state of the system, and a post-runtime analysis is performed to place events and injections into a single global timeline and to discard experiments with incorrect fault injections. Finally, the experiments with correct fault injections are used to estimate user-specified performance and dependability measures. A flexible measure language has been designed that facilitates the specification of a wide range of measures.