Toward A More Reliable Theory Of Software Reliability

PERSPECTIVES James A. Whittaker Florida Institute of Technology Jeffrey Voas Cigital The notions of time and the operational profile incorporated into software reliability are incomplete. Reliability should be redefined as a function of application complexity, test effectiveness, and operating environment. 36 Computer Toward a More Reliable Theory of Software Reliability S oftware development and testing is an error-prone process. Despite developers’ best attempts at checking every detail, errors made during development often cause postrelease software failures. Software reliability theory is one of industry’s seminal approaches for predicting the likelihood of software field failures. Unfortunately, the assumptions that software reliability measurement models make do not address the complexities of most software, resulting in far less adoption of theory into practice than is possible. Even though reliability models are quantitative, the industry only uses these results qualitatively. An example of this use is testing. When the mean time to failure (MTTF) falls below X according to a particular reliability model, testing stops, but developers and users cannot assume that the software will always behave with an MTTF less than X in the field. Software reliability theory appears to work accurately in telecommunications and aerospace. There are two reasons for this successful use. First, governments essentially regulate product quality in these two fields: Telephones must work and airplanes must fly. In other disciplines, such as the production of commercial, shrink-wrap software, quality has historically been an add-on, of lesser market value than feature richness or short release cycles. Second, telecom and aerospace software works in an embedded environment, and in many ways it is indistinguishable from the hardware on which it resides. Because the software assumes many characteristics of its hardware environment, it is not a huge leap to apply hardware reliability directly to embedded software. Today, accurate quality measurement can no longer be confined to particular industries; it is especially needed in shrink-wrap software. The ubiquity of network computing has brought quality concerns to the forefront. Buggy operating systems, Web browsers, and even client-side desktop applications can cause security holes. Bugs provide entry points for viruses and cause denial of service. Hackers can exploit buffer overruns to execute malicious programs. Because consumers are starting to demand it, quality is a concern for any company that develops software designed to run on networked computers. Even after a decade of practice in the confines of the hardware-dominated telecom and aerospace worlds, we wonder if the wider audience is ready for software reliability theory. Successful case studies and a thriving research community 0018-9162/00/$10.00 © 2000 IEEE Reliability coexist with skepticism and doubt among potential practitioners. For example, the US Federal Aviation Administration’s standard for developing software embedded on airplanes (RTCA DO178-B) concludes “currently available (reliability) methods do not provide results in which confidence can be placed.” Of course, skepticism should shadow new ideas, but software reliability is understood in only a small sector of the software industry. Our goal is to create a dialogue in the reliability community and to identify a base of technology that will widen interest in software reliability among pract