IEEE Std 1044.1-1995 IEEE Standard Classification for Software Anomalies.pdf

The Institute of Electrical and Electronics Engineers, Inc. 345 East 47th Street, New York, NY 10017-2394, USA Copyright © 1996 by the Institute of Electrical and Electronics Engineers, Inc. All rights reserved. Published 1996. Printed in the United States of America. ISBN 1-55937-697-X IEEE Std 1044.1-1995 IEEE Guide to Classification for Software Anomalies Sponsor Software Engineering Committee of the IEEE Power Engineering Society Approved December 12, 1995 IEEE Standards Board Approved 1 August, 1996 American National Standards Institute Abstract: This guide provides supporting information to assist users applying the IEEE Std 1044- 1993, IEEE Standard Classification for Software Anomalies, to decide whether to conform com- pletely to or just extract ideas from IEEE Std 1044.1. This guide will enable users of IEEE Std 1044- 1993 to implement and customize IEEE Std 1044-1993 for their organization in an effective and ef- ficient manner. Keywords: anomaly, category, classification, software, supporting data item IEEE Standards documents are developed within the Technical Committees of the IEEE Societies and the Standards Coordinating Committees of the IEEE Standards Board. Members of the committees serve vol- untarily and without compensation. They are not necessarily members of the Institute. The standards devel- oped within IEEE represent a consensus of the broad expertise on the subject within the Institute as well as those activities outside of IEEE that have expressed an interest in participating in the development of the standard. Use of an IEEE Standard is wholly voluntary. The existence of an IEEE Standard does not imply that there are no other ways to produce, test, measure, purchase, market, or provide other goods and services related to the scope of the IEEE Standard. Furthermore, the viewpoint expressed at the time a standard is approved and issued is subject to change brought about through developments in the state of the art and comments received from users of the standard. Every IEEE Standard is subjected to review at least every five years for revision or reaffirmation. When a document is more than five years old and has not been reaffirmed, it is reasonable to conclude that its contents, although still of some value, do not wholly reflect the present state of the art. Users are cautioned to check to determine that they have the latest edition of any IEEE Standard. Comments for revision of IEEE Standards are welcome from any interested party, regardless of member- ship affiliation with IEEE. Suggestions for changes in documents should be in the form of a proposed change of text, together with appropriate supporting comments. Interpretations: Occasionally questions may arise regarding the meaning of portions of standards as they relate to specific applications. When the need for interpretations is brought to the attention of IEEE, the Institute will initiate action to prepare appropriate responses. Since IEEE Standards represent a consensus of all concerned interests, it is important to ensure that any interpretation has also received the concurrence of a balance of interests. For this reason IEEE and the members of its technical committees are not able to provide an instant response to interpretation requests except in those cases where the matter has previously received formal consideration. Comments on standards and requests for interpretations should be addressed to: Secretary, IEEE Standards Board 445 Hoes Lane P.O. Box 1331 Piscataway, NJ 08855-1331 USA Authorization to photocopy portions of any individual standard for internal or personal use is granted by the Institute of Electrical and Electronics Engineers, Inc., provided that the appropriate fee is paid to Copyright Clearance Center. To arrange for payment of licensing fee, please contact Copyright Clearance Center, Cus- tomer Service, 222 Rosewood Drive, Danvers, MA 01923 USA; (508) 750-8400. Permission to photocopy portions of any individual standard for educational classroom use can also be obtained through the Copy- right Clearance Center. Note: Attention is called to the possibility that implementation of this standard may require use of subject matter covered by patent rights. By publication of this stan- dard, no position is taken with respect to the existence or validity of any patent rights in connection therewith. The IEEE shall not be responsible for identifying all patents for which a license may be required by an IEEE standard or for conducting inquiries into the legal validity or scope of those patents that are brought to its attention. iii Introduction (This introduction is not a part of IEEE Std 1044.1-1995, IEEE Guide to Classification for Software Anomalies.) On most projects, some formality in reporting anomalies allows you to record the facts and impressions about an anomaly when it is encountered so that it can be understood by someone at a later date. Typically, information about what happened, how it happened, and the environment in which it happened are put into the anomaly reports. IEEE Std 1044-1993 calls this information “supporting data items.” Armed with this information, the anomaly can be reproduced, a fix can be applied, and the new program can be verified not to have the same anomaly symptoms. Many anomaly tracking information systems of this type are well documented, have been used successfully for years, and are thought of as good anomaly tracking systems. However, when an organization begins to mature, the universal software quality question is asked: “How best can we get rid of the bugs in our products or prevent them from happening?” To effectively answer this question, you need more information about the anomalies that have been found in the software. Knowing the date, submitter, software version, and module an anomaly was reported against usually does not point in the direction of problem areas. The information that will facilitate the discovery of “common errors made,” “most effective activity to find anomalies,” “the place where anomalies were first introduced,” and other telling facts about your development process comes from classification analysis of each anomaly reported. This is where application of the IEEE Std 1044-1993 classification scheme will prove invaluable. Implementing IEEE Std 1044-1993 can lead to better project management decisions, increased depth of data analysis, and improved software development processes. The consistent historical data allows trend analysis through several releases of the same project, several projects, and/or organizations. This historical consistency of data makes it possible to measure the effects of any process changes you implement from release to release, project to project, or organization to organization. Additionally, using a database and change control procedure reduces the labor of tracking changes, analyzing data, and providing an audit trail. This guide provides the alternatives and methods for getting the most out of IEEE Std 1044-1993. Many benefits of using IEEE Std 1044-1993 could be obtained by using any classification scheme. However, the big advantage of IEEE Std 1044-1993 is the effort saved by not re-inventing and debating yet another bug taxonomy. Also, wide-spread use of IEEE Std 1044-1993 means that eventually industry-wide informa- tion will be available to use for comparative purposes or to benchmark the process(es) used in software development. iv At the time this guide was completed, the Classification Guide Working Group had the following membership: Cynthia Brehmer, Chair Jaya R. Carl, Co-Chair The following persons were on the balloting committee: Robert JohnsonBrian Lawrence Richard Evans Colin James Syed Ali Man K. Au John T. Bell Mordechai Ben-Menachem H. Ronald Berlack Christos Bezirtzoglou William J. Boll Fletcher Buckley David W. Burnett Michael Caldwell John P. Chihorek Antonio M. Cicu Theo Clarke Sharon Cobb Virgil Lee Cooper Geoff Cozens Gregory T. Daich Geoffrey Darnton Taz Daughtrey Bostjan K. Derganc C. Einar Dragstedt Leo G. Egan William Eventoff John W. Fendrich Julian Forster Kirby Fortenberry Eva Freund Adel N. Ghannam Hiranmay Ghosh Julio Gonzalez Sanz Lawrence M. Gunther David A. Gustafson John Harauz Rob Harker Myron Hecht Rick Hefner Gary E. Hendrzak Umesh P. Hiriyannaiah John W. Horch Charles C. Howell Fabrizio Imelio Frank P. Jay David Johnson Martha Johnson Vladan V. Jovanovic William S. Junk Chris F. Kemerer Judy Kerner Robert J. Kierzyk Motti Y. Klein Thomas M. Kurihara John B. Lane J. Dennis Lawrence Stanley H. Levinson Fang Ching Lim Mancini Loredana Austin J. Maher Robert A. Martin Tomoo Matsubara Scott D. Matthews Mike McAndrew Patrick McCray Sue McGrath Jacques Meekel Glen A. Meldrum Jerome W. Mersky James Bret Michael Celia H. Modell Charles S. Mooney James W. Moore John Musa Dennis E. Nickle Mike Ottewill Gerald L. Ourada Lalit Mohan Patnaik Donald J. Pfeiffer John G. Phippen Peter T. Poon Lawrence S. Przybylski Dennis Rilling James R. Roberts R. Waldo Roth Terence P. Rout Andrew P. Sage Helmut Sandmayr Stephen R. Schach Hans Schaefer Peter E. Schilling Gregory D. Schumacher Katsutoshi Shintani David M. Siefert Raghu P. Singh Nancy M. Smith Melford E. Smyre Jag Sodhi Alfred R. Sorkowitz Julia Stesney Fred J. Strauss Toru Takeshita Booker Thomas Leonard L. Tripp Mark-Rene Uchida Theodore J. Urbanowicz Glenn D. Venables David W. Vickers Udo Voges Dolores Wallace William M. Walsh Camille S. White-Partain Scott A. Whitmire Paul A.T. Wolfgang Paul R. Work Weider D. Yu Janusz Zalewski Geraldine Zimmerman Peter F. Zoll v When the IEEE Standards Board approved this standard on December 12, 1995, it had the following membership: E. G. “Al” Kiener, ChairDonald C. Loughry, Vice Chair Andrew G. Salem, Secretary *Member Emeritus Also included are the following nonvoting IEEE Standards Board liaisons: Satish K. Aggarwal Steve Sharkey Robert E. Hebner Chester C. Taylor Angela M. Girardi IEEE Standards Project Editor Gilles A. Baril Clyde R. Camp Joseph A. Cannatelli Stephen L. Diamond Harold E. Epstein Donald C. Fleckenstein Jay Forster* Donald N. Heirman Richard J. Holleman Jim Isaak Ben C. Johnson Sonny Kasturi Lorraine C. Kevra Ivor N. Knight Joseph L. Koepfinger* D. N. “Jim” Logothetis L. Bruce McClung Marco W. Migliaro Mary Lou Padgett John W. Pope Arthur K. Reilly Gary S. Robinson Ingo Rusch Chee Kiow Tan Leonard L. Tripp Howard L. Wolfman vi Contents CLAUSE PAGE 1.Overview 1 1.1 Purpose. 1 1.2 Scope 1 1.3 Goals 2 1.4 Audiences.2 2.References 2 3.Definitions . 2 4.Getting started 3 5.Categories 5 5.1 Category meanings 5 5.2 Standard compliance at the category level 7 6.Classifications 9 6.1 Establishing classification definitions . 9 6.2 Choosing classifications within a category 11 6.3 Standard compliance at the classification level. 14 7.Supporting data items 16 8.Anomaly processing 17 8.1 Generic anomaly process 18 8.2 Specific example of a real anomaly process mapped to the generic . 19 9.Methodologies .21 9.1 Using a commercial anomaly tracking product. 21 9.2 Using an internally-developed anomaly tracking product. 22 9.3 Linking with other tools, processes, and planning documents 23 9.4 Using paper versus a database for tracking anomalies 23 10.Data analysis 23 10.1 Statistical analysis 24 10.2 Project management. 27 10.3 Process improvement. 27 10.4 Product assessment 28 11.Relationships to other standards 28 11.1 Software Engineering Institute Capability Maturity Model 28 11.2 ISO 9000 29 11.3 Department of Defense and military standards 29 vii ANNEX Annex A (informative)Sample definitions of categories and classifications in IEEE Std 1044-1993 30 A.1Actual cause. 30 A.2Corrective action 32 A.3Customer Value. 33 A.4Disposition. 34 A.5Mission/safety 34 A.6Priority.35 A.7Product Status 35 A.8Project Activity 36 A.9Project Cost37 A.10 Project Phase 38 A.11 Project Quality/Reliability. 40 A.12 Project Risk40 A.13 Project Schedule 41 A.14 Repeatability 41 A.15 Resolution42 A.16 Severity 43 A.17 Societal.43 A.18 Source 44 A.19 Suspected Cause 46 A.20 Symptom 48 A.21 Type.49 Annex B(informative)Bibliography. 52 1 IEEE Guide to Classification for Software Anomalies 1. Overview 1.1 Purpose The purpose of this guide is to provide supporting information to assist users who are applying the IEEE Std 1044-19931, IEEE Standard Classification for Software Anomalies, whether to conform completely to IEEE Std 1044-1993 or just to extract ideas from it. The guide will enable users of IEEE Std 1044-1933 to imple- ment and customize IEEE Std 1044-1993 for their organization in an effective and efficient manner. 1.2 Scope This guide accomplishes the following tasks: a)Describes how to apply IEEE Std 1044-1993 b)Describes what is and is not compliant customization by 1)Providing examples of use and customization 2)Providing example definitions of classification terms and situations where different definitions might be applicable c)Provides some additional explanation of the “supporting data items” from IEEE Std 1044-1993 d)Provides examples of measures that can be derived from the classification data e)Provides examples of where those metrics are useful f)Describes methods for analyzing the classification data 1Information about references can be found in clause 2. IEEE Std 1044.1-1995IEEE GUIDE TO CLASSIFICATION 2 1.3 Goals The goals of this guide are as follows: a)To facilitate the determination of the level of conformance and amount of detail in IEEE Std 1044- 1993 that is appropriate to the organization b)To define the difference between conforming to IEEE Std 1044-1993 and using it as a reference in the organizations anomaly classification scheme 1.4 Audiences This guide serves two audiences. The primary audience is those persons working on a project required to use IEEE Std 1044-1993 and looking for information to help implement IEEE Std 1044-1993 on their project. The second audience is comprised of the project managers or organization managers wishing to start or expand an anomaly tracking system and looking for proven methodologies to support that effort. 2. References This guide shall be used in conjunction with the following publication: IEEE Std 1044-1993, IEEE Standard Classification for Software Anomalies.2 3. Definitions The IEEE Std 1044-1993 anomaly classification scheme consists of categories and classifications. The classification scheme is hierarchical in nature. Categories, the top level, represent attributes of an anomaly you might want to classify. Classifications within each category are hierarchical lists of choices to describe the attribute. 3.1 anomaly: Any condition that deviates from the expected based on requirements specifications, design documents, user documents, standards, etc. or from someones perceptions or experiences. Anomalies may be found during, but not limited to, the review, test, analysis