A Binomial Software Reliability Model Based on Coverage of Structural Testing Criteria
Resumo
A new approach to software reliability modeling is discussed where variables indirectly related with software reliability are used to provide additional information for the modeling process. Previous studies, empirical evidences, and results from experiments indicate that there is a strong relationship between software reliability and coverage of program elements required to be exercised by structural testing criteria. In this vein, a Binomial software reliability Model Based on Coverage - BMBC - is proposed and discussed. The BMBC, the first software reliability model based on coverage, was assessed with test data from a real application, making use of the following structural testing criteria: all-nodes, all-edges, and potential-uses — a data-flow based family of criteria. Results from the experiments show a clear superiority of the BMBC over the traditional models and point to a very promising research direction in software reliability.
Palavras-chave:
software reliability models, structural testing criteria, test coverage
Referências
Adams, E. N., "Minimizing Cost Impact of Software Defects", IBM Research Division, Report RC 8228(35669), 1980.
Bastani, F. B. and Ramamoorthy, C. V., "Input-domain-based Models for Estimating the Correctness of Process Control Programs", A. Serra and R. E. Barlow (eds), Reliability Theory, North Holland, Amsterdam, pp. 321-378, 1986.
Bishop, P.(ed), "Prediction and Measurement of Software Reliability", Dependability of Critical Computer Systems 3, Elsevier Science Publisher, London, 1990.
Butler, R. W. and Finelli, G. B., "The Unfeasibility of Experimental Quantification of Life-critical Software Reliability", Proceedings of the ACM SIGSOFT '91 Conference on Software for Critical Systems, ACM Press, December 1991, pp. 66-76.
Chaim, M. L., "Uma Ferramenta para Suporte ao Teste Estrutural de Programas Baseado em Análise de Fluxo de Dados", Dissertação de Mestrado, DCA/FEE/UNICAMP - Campinas, SP, Abril 1991.
Chen, M., "Tools and Techniques for Testing Based Software Reliability Estimation," Ph. D. Thesis, Purdue University, West Lafayette — Indiana, August 1994.
Chen, M.; Mathur, A. P. and Rego, V. J., "Effect of Testing Technique on Software Reliability Estimates Obtained Using A Time-Domain Model," IEEE Transactions on Reliability, vol. 44, no. 1, March 1995, pp. 97-103.
Chen, M.; Lyu, M. R. and Wong, W. Eric "An Empirical Study of the Correlation between Code Coverage and Reliability Estimation", Proceedings of the Third International Software Metrics Symposium, Berlin, Germany, March 25-26, 1996.
Crespo, A. N.; Matrella P. and Pasquini, A., "Sensitivity of Reliability Growth Models to Operational Profile Errors", Proceedings, The Seventh International Symposium on Software Reliability Engineering, White Plains, New York, November 1996, pp. 35-44.
Crespo, A. N.; Pasquini, A.; Jino, M. e Madonado, J. C., "Cobertura dos Critérios Potenciais-usos e Confibilidade de Software", Anais do XI Simpósio Brasileiro de Engenharia de Software - Fortaleza, outubro de 1997.
Crespo, A. N., "Modelos de Confiabilidade de Software Bascados em Cobertura de Critérios Estruturais de Teste", Tese de Doutado, DCA/FEEC/UNICAMP - Campinas, SP. dezembro de 1997.
Crespo, A.N.; Pasquini, A.; Jino, M. e Madonado, J. C., "Code Coverage of the Potential-Uses Criteria and Software Reliability", Proceedings of the Fourth ISSAT International Conference, Seattle, Washington, USA, August, 1998.
Frate, F. D.; Garg, P.; Mathur, A. P. and Pasquini, A., "Experiments to Investigate the Correlations Between Code Coverage and Software Reliability", SERC-TR-162-P, Software Engineering Research Center, Purdue University, West Lafayette, Indiana 47907, April 1995.
Garg, P., "Investigating Coverage - Reliability Relationship and Sensitivity of Reliability to Errors in Operational Profile", Technical Report - Department of Computer Sciences, Purdue University, West Lafayette, IN 47907, May 1994.
Goel, A. L. and Okumoto, K., "A Time Dependent Error Detection Rate Model for Software Reliability and Other Performance Measures", IEEE Transactions on Reliability, Vol. 28, 1979, pp.206-211.
Hamlet, D., "Are We testing for True Reliability", IEEE Software, vol. 9, no. 4, July 1992.
Hudson, A., "Program Errors as a Birth and Death Process", Technical Report SP - 3011, Santa Monica, Cal.: Systems Development Corporation, 1967.
— ISO/IEC 9126. Information technology - Software product evaluation — Quality characteristics and guidelines for their use, ISO/IEC — 1991.
Jelinski Z. and Moranda P. B., "Software Reliability Research", Proceedings of the Statistical Methods for the Evaluation of Computer System Performance, Academic Press, 1972, pp. 465-484.
Li N. and Malaiya, Y. K., "On Input Profile Selection For Software Testing", Computer Science Department, Colorado State University, Fort Collins, CO 80523, 1994.
Littlewood, B. and String, L., "Validation of Ultrahigh Dependability for Software-based Systems", Communication of the ACM, vol. 36, no. 1, Jan. 1993
Lyu, M. R., "Handbook of Software Reliability Engineering", McGraw-Hill, 1996.
Malaiya, Y. K.;Li, N.; Bieman, J.; Karcick, R. and Skibe, B., "The Relationship Between Test Coverage and Reliability", Proceedings of the Fifth International Symposium on Software Reliability Engineering, Monterey, CA, November 6-9, 1994, pp. 186-195.
Maldonado, J. C.; Chaim, M. L.; e Jino, M., "Bridging the Gap in the Presence of Infeasible Paths: Potential Uses Testing Criteria"; XII International Conference of the SCCC, Sociedad Chilena de Ciencia de la Computacion, Santiago, Chile, outubro de 1992, pp. 323-340.
Maldonado, J. C.; Chaim, M. L.; e Jino, M., "Using the essencial Branch Concept to Support Data-Flow Based Testing Criteria Application"; Toulouse'92, Fifth International Conference on Software Engineering and Its Applications; Toulouse, França, dezembro de 1992, pp. 613-623.
Mood, A.; Graybill, F. and Boes, D., "Introduction to the Theory of Statistics", 3d ed., McGraw-Hill, Inc., New York, 1974.
Musa, J. D.; Ianino, A. and Okumoto, K., Software Reliability - Measurement, Prediction, Application, McGraw-Hill, New York, 1987.
Nelson, F., "Estimating Software Reliability From Test Data", Microelectronics and Reliability, vol. 17, no. 1, 1978, pp. 67-73.
Pasquini, A.; Crespo, A. N. and Matrella, P., "Sensitivity of Reliability Growth Models to Operational Profile Errors vs Testing Accuracy", IEEE Transactions on Reliability, December 1996, vol. 45, no. 4, pp. 531-540.
Ramsey, J. and Basili, V. R., "Analyzing the Test Process Using Structural Coverage," Proceedings ICSE'85, pp. 306-312, 1985.
Rapps, S. and Weyuker, E. J., "Selecting Software Test Data Using Data Flow Information", IEEE Transactions on Software Engineering, April 1985, Vol. SE-11, No. 4, pp. 367-375.
Shooman, M. L., "Probabilistíc Models for Software Realiability Prediciion," Statistical Computer Performance Evaluation, W. Freiberg, Ed. New York: Academic Press, 1972, pp.485-502.
Siegel, Sidney, "Estatística Não Paramétrica", McGaw Hill, 1977.
Varadan, G. S. "Trends in Reliability and Test Strategies," IEEE Software, May 1995, pp. 10.
Veevers, A. and Marshall, A. "A Relationship Between Software Coverage Metrics and Reliability" Software Testing, Verification and Reliability, vol. 4, 1994, pp.3-8.
Xie, M., "Software Reliability Models - A Selected Annotated Bibliography", Software Testing Verification and Reliability, vol. 3, 1993, pp. 3-28.
Bastani, F. B. and Ramamoorthy, C. V., "Input-domain-based Models for Estimating the Correctness of Process Control Programs", A. Serra and R. E. Barlow (eds), Reliability Theory, North Holland, Amsterdam, pp. 321-378, 1986.
Bishop, P.(ed), "Prediction and Measurement of Software Reliability", Dependability of Critical Computer Systems 3, Elsevier Science Publisher, London, 1990.
Butler, R. W. and Finelli, G. B., "The Unfeasibility of Experimental Quantification of Life-critical Software Reliability", Proceedings of the ACM SIGSOFT '91 Conference on Software for Critical Systems, ACM Press, December 1991, pp. 66-76.
Chaim, M. L., "Uma Ferramenta para Suporte ao Teste Estrutural de Programas Baseado em Análise de Fluxo de Dados", Dissertação de Mestrado, DCA/FEE/UNICAMP - Campinas, SP, Abril 1991.
Chen, M., "Tools and Techniques for Testing Based Software Reliability Estimation," Ph. D. Thesis, Purdue University, West Lafayette — Indiana, August 1994.
Chen, M.; Mathur, A. P. and Rego, V. J., "Effect of Testing Technique on Software Reliability Estimates Obtained Using A Time-Domain Model," IEEE Transactions on Reliability, vol. 44, no. 1, March 1995, pp. 97-103.
Chen, M.; Lyu, M. R. and Wong, W. Eric "An Empirical Study of the Correlation between Code Coverage and Reliability Estimation", Proceedings of the Third International Software Metrics Symposium, Berlin, Germany, March 25-26, 1996.
Crespo, A. N.; Matrella P. and Pasquini, A., "Sensitivity of Reliability Growth Models to Operational Profile Errors", Proceedings, The Seventh International Symposium on Software Reliability Engineering, White Plains, New York, November 1996, pp. 35-44.
Crespo, A. N.; Pasquini, A.; Jino, M. e Madonado, J. C., "Cobertura dos Critérios Potenciais-usos e Confibilidade de Software", Anais do XI Simpósio Brasileiro de Engenharia de Software - Fortaleza, outubro de 1997.
Crespo, A. N., "Modelos de Confiabilidade de Software Bascados em Cobertura de Critérios Estruturais de Teste", Tese de Doutado, DCA/FEEC/UNICAMP - Campinas, SP. dezembro de 1997.
Crespo, A.N.; Pasquini, A.; Jino, M. e Madonado, J. C., "Code Coverage of the Potential-Uses Criteria and Software Reliability", Proceedings of the Fourth ISSAT International Conference, Seattle, Washington, USA, August, 1998.
Frate, F. D.; Garg, P.; Mathur, A. P. and Pasquini, A., "Experiments to Investigate the Correlations Between Code Coverage and Software Reliability", SERC-TR-162-P, Software Engineering Research Center, Purdue University, West Lafayette, Indiana 47907, April 1995.
Garg, P., "Investigating Coverage - Reliability Relationship and Sensitivity of Reliability to Errors in Operational Profile", Technical Report - Department of Computer Sciences, Purdue University, West Lafayette, IN 47907, May 1994.
Goel, A. L. and Okumoto, K., "A Time Dependent Error Detection Rate Model for Software Reliability and Other Performance Measures", IEEE Transactions on Reliability, Vol. 28, 1979, pp.206-211.
Hamlet, D., "Are We testing for True Reliability", IEEE Software, vol. 9, no. 4, July 1992.
Hudson, A., "Program Errors as a Birth and Death Process", Technical Report SP - 3011, Santa Monica, Cal.: Systems Development Corporation, 1967.
— ISO/IEC 9126. Information technology - Software product evaluation — Quality characteristics and guidelines for their use, ISO/IEC — 1991.
Jelinski Z. and Moranda P. B., "Software Reliability Research", Proceedings of the Statistical Methods for the Evaluation of Computer System Performance, Academic Press, 1972, pp. 465-484.
Li N. and Malaiya, Y. K., "On Input Profile Selection For Software Testing", Computer Science Department, Colorado State University, Fort Collins, CO 80523, 1994.
Littlewood, B. and String, L., "Validation of Ultrahigh Dependability for Software-based Systems", Communication of the ACM, vol. 36, no. 1, Jan. 1993
Lyu, M. R., "Handbook of Software Reliability Engineering", McGraw-Hill, 1996.
Malaiya, Y. K.;Li, N.; Bieman, J.; Karcick, R. and Skibe, B., "The Relationship Between Test Coverage and Reliability", Proceedings of the Fifth International Symposium on Software Reliability Engineering, Monterey, CA, November 6-9, 1994, pp. 186-195.
Maldonado, J. C.; Chaim, M. L.; e Jino, M., "Bridging the Gap in the Presence of Infeasible Paths: Potential Uses Testing Criteria"; XII International Conference of the SCCC, Sociedad Chilena de Ciencia de la Computacion, Santiago, Chile, outubro de 1992, pp. 323-340.
Maldonado, J. C.; Chaim, M. L.; e Jino, M., "Using the essencial Branch Concept to Support Data-Flow Based Testing Criteria Application"; Toulouse'92, Fifth International Conference on Software Engineering and Its Applications; Toulouse, França, dezembro de 1992, pp. 613-623.
Mood, A.; Graybill, F. and Boes, D., "Introduction to the Theory of Statistics", 3d ed., McGraw-Hill, Inc., New York, 1974.
Musa, J. D.; Ianino, A. and Okumoto, K., Software Reliability - Measurement, Prediction, Application, McGraw-Hill, New York, 1987.
Nelson, F., "Estimating Software Reliability From Test Data", Microelectronics and Reliability, vol. 17, no. 1, 1978, pp. 67-73.
Pasquini, A.; Crespo, A. N. and Matrella, P., "Sensitivity of Reliability Growth Models to Operational Profile Errors vs Testing Accuracy", IEEE Transactions on Reliability, December 1996, vol. 45, no. 4, pp. 531-540.
Ramsey, J. and Basili, V. R., "Analyzing the Test Process Using Structural Coverage," Proceedings ICSE'85, pp. 306-312, 1985.
Rapps, S. and Weyuker, E. J., "Selecting Software Test Data Using Data Flow Information", IEEE Transactions on Software Engineering, April 1985, Vol. SE-11, No. 4, pp. 367-375.
Shooman, M. L., "Probabilistíc Models for Software Realiability Prediciion," Statistical Computer Performance Evaluation, W. Freiberg, Ed. New York: Academic Press, 1972, pp.485-502.
Siegel, Sidney, "Estatística Não Paramétrica", McGaw Hill, 1977.
Varadan, G. S. "Trends in Reliability and Test Strategies," IEEE Software, May 1995, pp. 10.
Veevers, A. and Marshall, A. "A Relationship Between Software Coverage Metrics and Reliability" Software Testing, Verification and Reliability, vol. 4, 1994, pp.3-8.
Xie, M., "Software Reliability Models - A Selected Annotated Bibliography", Software Testing Verification and Reliability, vol. 3, 1993, pp. 3-28.
Publicado
04/10/2000
Como Citar
CRESPO, Adalberto Nobiato; JINO, Mario; PASQUINI, Alberto; MALDONADO, José Carlos.
A Binomial Software Reliability Model Based on Coverage of Structural Testing Criteria. In: SIMPÓSIO BRASILEIRO DE ENGENHARIA DE SOFTWARE (SBES), 14. , 2000, João Pessoa/PB.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2000
.
p. 211-226.
DOI: https://doi.org/10.5753/sbes.2000.25930.
