Mutation Analysis; Integration testing; CPS Interoperability; Cyber-Physical Systems
Abstract :
[en] Cyber-physical systems (CPSs) typically consist of a wide set of integrated, heterogeneous components; consequently, most of their critical failures relate to the interoperability of such components. Unfortunately, most CPS test automation techniques are preliminary and industry still heavily relies on manual testing. With potentially incomplete, manually-generated test suites, it is of paramount importance to assess their quality. Though mutation analysis has demonstrated to be an effective means to assess test suite quality in some specific contexts, we lack approaches for CPSs. Indeed, existing approaches do not target interoperability problems and cannot be executed in the presence of black-box or simulated components, a typical situation with CPSs. In this paper, we introduce data-driven mutation analysis, an approach that consists in assessing test suite quality by verifying if it detects interoperability faults simulated by mutating the data exchanged by software components. To this end, we describe a data-driven mutation analysis technique (DaMAT) that automatically alters the data exchanged through data buffers. Our technique is driven by fault models in tabular form where engineers specify how to mutate data items by selecting and configuring a set of mutation operators. We have evaluated DaMAT with CPSs in the space domain; specifically, the test suites for the software systems of a microsatellite and nanosatellites launched on orbit last year. Our results show that the approach effectively detects test suite shortcomings, is not affected by equivalent and redundant mutants, and entails acceptable costs.
Research center :
Interdisciplinary Centre for Security, Reliability and Trust (SnT) > Software Verification and Validation Lab (SVV Lab)
Disciplines :
Computer science
Author, co-author :
VIGANO, Enrico ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV
Cornejo, Oscar
PASTORE, Fabrizio ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV
BRIAND, Lionel ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV
External co-authors :
no
Language :
English
Title :
Data-driven Mutation Analysis for Cyber-Physical Systems
Publication date :
01 April 2023
Journal title :
IEEE Transactions on Software Engineering
ISSN :
0098-5589
eISSN :
1939-3520
Publisher :
Institute of Electrical and Electronics Engineers, New-York, United States - New York
S. K. Khaitan and J. D. McCalley, “Design techniques and applications of cyberphysical systems: A survey,” IEEE Syst. J., vol. 9, no. 2, pp. 350–365, Jun. 2015.
H. Roehm, J. Oehlerking, M. Woehrle, and M. Althoff, “Model conformance for cyber-physical systems: A survey,” ACM Trans. Cyber-Phys. Syst., vol. 3, no. 3, 2019, Art. no. 30.
Y. Jia and M. Harman, “An analysis and survey of the development of mutation testing,” IEEE Trans. Softw. Eng., vol. 37, no. 5, pp. 649–678, Sep./Oct. 2011.
M. Papadakis, M. Kintis, J. Zhang, Y. Jia, Y. Le Traon, and M. Harman, “Mutation testing advances: An analysis and survey,” in Advances in Computers. Amsterdam, The Netherlands: Elsevier, 2019, pp. 275–378.
M. Papadakis, D. Shin, S. Yoo, and D.-H. Bae, “Are mutation scores correlated with real fault detection? A large scale empirical study on the relationship between mutants and real faults,” in Proc. IEEE/ACM 40th Int. Conf. Softw. Eng., 2018, pp. 537–548.
O. Givehchi, K. Landsdorf, P. Simoens, and A. W. Colombo, “Interoperability for industrial cyber-physical systems: An approach for legacy systems,” IEEE Trans. Ind. Informat., vol. 13, no. 6, pp. 3370–3378, Dec. 2017.
V. Jirkovsky, M. Obitko, and V. Mařík, “Understanding data heterogeneity in the context of cyber-physical systems integration,” IEEE Trans. Ind. Informat., vol. 13, no. 2, pp. 660–667, Apr. 2017.
S. A. Asadollah, R. Inam, and H. Hansson, “A survey on testing for cyber physical system,” in Proc. IFIP Int. Conf. Testing Softw. Syst., 2015, pp. 194–207.
P. Ammann and J. Offutt, Introduction to Software Testing, 1st ed. New York, NY, USA: Cambridge Univ. Press, 2008.
M. E. Delamaro, J. Offutt, and P. Ammann, “Designing deletion mutation operators,” in Proc. IEEE 7th Int. Conf. Softw. Testing Verification Validation, 2014, pp. 11–20.
M. E. Delamaro, L. Deng, V. H. S. Durelli, N. Li, and J. Offutt, “Experimental evaluation of SDL and one-op mutation for C,” in Proc. IEEE 7th Int. Conf. Softw. Testing Verification Validation, 2014, pp. 203–212.
N. He, P. Rummer, € and D. Kroening, “Test-case generation for embedded Simulink via formal concept analysis,” in Proc. Des. Autom. Conf., 2011, pp. 224–229.
B. Aichernig, H. Brandl, J. Elisabeth, W. Krenn, R. Schlick, and S. Tiran, “Model-based mutation testing for UML,” in Proc. IEEE Int. Conf. Softw. Testing Verification Validation, 2015. [Online]. Available: www.momut.org
X. Devroey, G. Perrouin, M. Papadakis, A. Legay, P.-Y. Schobbens, and P. Heymans, “Featured model-based mutation analysis,” in Proc. 38th Int. Conf. Softw. Eng., 2016, pp. 655–666.
F. Belli, C. J. Budnik, A. Hollmann, T. Tuglular, and W. E. Wong, “Model-based mutation testing—Approach and case studies,” Sci. Comput. Program., vol. 120, pp. 25–48, 2016. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S0167642316000137
ESA, “European space agency,” 2020. [Online]. Available: https://www.esa.int
F. Davoli, C. Kourogiorgas, M. Marchese, A. Panagopoulos, and F. Patrone, “Small satellites and CubeSats: Survey of structures, architectures, and protocols,” Int. J. Satell. Commun. Netw., vol. 37, no. 4, pp. 343–359, 2019. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/sat.1277
E. Vigano, O. E. Cornejo Olivares, F. Pastore, and L. Briand, “Replicability package,” 2022. [Online]. Available: https://doi.org/10.6084/m9.figshare.21276093
E. Vigano, O. E. Cornejo Olivares, F. Pastore, and L. Briand, “DAMAT source code,” 2022. [Online]. Available: https://github.com/SNTSVV/DAMAT
OASIS, “OASIS web services business process execution language (WSBPEL) TC,” 2007. [Online]. Available: https://www.oasisopen.org/committees/tc_home.php?wg_abbrev¼wsbpel
R. Natella, D. Cotroneo, and H. S. Madeira, “Assessing dependability with software fault injection: A survey,” ACM Comput. Surveys, vol. 48, no. 3, 2016, Art. no. 44.
A. J. Offutt, A. Lee, G. Rothermel, R. H. Untch, and C. Zapf, “An experimental determination of sufficient mutant operators,” ACM Trans. Softw. Eng. Methodol., vol. 5, no. 2, pp. 99–118, 1996.
G. Rothermel, R. H. Untch, and C. Zapf, “An experimental determination of sufficient mutant operators A. Jefferson Offutt Ammei Lee George Mason university,” ACM Trans. Softw. Eng. Methodol., vol. 5, no. 2, pp. 99–118, 1996.
J. H. Andrews, L. C. Briand, and Y. Labiche, “Is mutation an appropriate tool for testing experiments?,” in Proc. 27th Int. Conf. Softw. Eng., 2005, pp. 402–411.
M. Kintis, M. Papadakis, Y. Jia, N. Malevris, Y. Le Traon, and M. Harman, “Detecting trivial mutant equivalences via compiler optimisations,” IEEE Trans. Softw. Eng., vol. 44, no. 4, pp. 308–333, Apr. 2018.
M. Harman, Y. Jia, and W. B. Langdon, “A manifesto for higher order mutation testing,” in Proc. IEEE 3rd Int. Conf. Softw. Testing Verification Validation Workshops, 2010, pp. 80–89.
Q. Zhu and A. Zaidman, “Mutation testing for physical computing,” in Proc. IEEE Int. Conf. Softw. Qual. Rel. Secur., 2018, pp. 289–300.
M. E. Delamaro, J. Maidonado, and A. P. Mathur, “Interface mutation: An approach for integration testing,” IEEE Trans. Softw. Eng., vol. 27, no. 3, pp. 228–247, Mar. 2001.
M. Grechanik and G. Devanla, “Mutation integration testing,” in Proc. IEEE Int. Conf. Softw. Qual. Rel. Secur., 2016, pp. 353–364.
Y. Jiang, S.-S. Hou, J.-H. Shan, L. Zhang, and B. Xie, “Contract-based mutation for testing components,” in Proc. IEEE 21st Int. Conf. Softw. Maintenance, 2005, pp. 483–492.
P. R. Mateo, M. P. Usaola, and J. Offutt, “Mutation at system and functional levels,” in Proc. IEEE 3rd Int. Conf. Softw. Testing Verification Validation Workshops, 2010, pp. 110–119.
T. K. Tsai, M.-C. Hsueh, H. Zhao, Z. Kalbarczyk, and R. K. Iyer, “Stress-based and path-based fault injection,” IEEE Trans. Comput., vol. 48, no. 11, pp. 1183–1201, Nov. 1999.
J. H. Barton, E. W. Czeck, Z. Z. Segall, and D. P. Siewiorek, “Fault injection experiments using FIAT,” IEEE Trans. Comput., vol. 39, no. 4, pp. 575–582, Apr. 1990.
S. Han, K. G. Shin, and H. A. Rosenberg, “DOCTOR: An integrated software fault injection environment for distributed real-time systems,” in Proc. IEEE Int. Comput. Perform. Dependability Symp., 1995, pp. 204–213.
S. Dawson, F. Jahanian, T. Mitton, and T.-L. Tung, “Testing of fault-tolerant and real-time distributed systems via protocol fault injection,” in Proc. IEEE Annu. Symp. Fault Tolerant Comput., 1996, pp. 404–414.
D. Di Nardo, F. Pastore, A. Arcuri, and L. Briand, “Evolutionary robustness testing of data processing systems using models and data mutation (T),” in Proc. IEEE/ACM 30th Int. Conf. Automated Softw. Eng., 2015, pp. 126–137.
D. Di Nardo, F. Pastore, and L. Briand, “Generating complex and faulty test data through model-based mutation analysis,” in Proc. IEEE 8th Int. Conf. Softw. Testing Verification Validation, 2015, pp. 1–10.
R. Matinnejad, S. Nejati, L. C. Briand, and T. Bruckmann, “Test generation and test prioritization for Simulink models with dynamic behavior,” IEEE Trans. Softw. Eng., vol. 45, no. 9, pp. 919–944, Sep. 2019.
A. K. Ghosh, M. Schmid, and V. Shah, “Testing the robustness of windows NT software,” in Proc. IEEE 9th Int. Symp. Softw. Rel. Eng., 1998, pp. 231–235.
P. Godefroid, A. Kiezun, and M. Y. Levin, “Grammar-based whitebox fuzzing,” in Proc. 29th ACM SIGPLAN Conf. Program. Lang. Des. Implementation, 2008, pp. 206–215.
P. Godefroid, M. Y. Levin, and D. Molnar, “SAGE: Whitebox fuzzing for security testing,” Commun. ACM, vol. 55, no. 3, pp. 40–44, 2012.
E. Bounimova, P. Godefroid, and D. Molnar, “Billions and billions of constraints: Whitebox fuzz testing in production,” in Proc. IEEE Int. Conf. Softw. Eng., 2013, pp. 122–131.
V.-T. Pham, M. Bohme, € and A. Roychoudhury, “Model-based whitebox fuzzing for program binaries,” in Proc. IEEE/ACM 31st Int. Conf. Automated Softw. Eng., 2016, pp. 543–553.
X. Bai, S. Lee, and Y. Chen, “Mutation-based simulation test data generation for testing complex real-time software,” in Proc. IEEE 40th Annu. Simul. Symp., 2007, pp. 73–80.
Y. Isasi et al., “The ESAIL multipurpose simulator,” in Onlin Proc. Simul. EGSE Space Programmes, 2019. [Online]. Available: https://atpi.eventsair.com/QuickEventWebsitePortal/sesp-2019/ website/Agenda
L. Zhang, M. Gligoric, D. Marinov, and S. Khurshid, “Operator-based and random mutant selection: Better together,” in Proc. IEEE/ACM 28th Int. Conf. Automated Softw. Eng., 2013, pp. 92–102.
R. Gopinath, A. Alipour, I. Ahmed, C. Jensen, and A. Groce, “How hard does mutation analysis have to be, anyway?,” in Proc. IEEE 26th Int. Symp. Softw. Rel. Eng., 2015, pp. 216–227.
J. Offutt, P. Ammann, and L. Liu, “Mutation testing implements grammar-based testing,” in Proc. IEEE 2nd Workshop Mutat. Anal., 2006, Art. no. 12.
European Space Agency, “ESA Vega launch,” 2020. [Online]. Available: https://www.esa.int/Enabling_Support/Space_Transportation/Vega/
D. Shin, S. Yoo, and D.-H. Bae, “A theoretical and empirical study of diversity-aware mutation adequacy criterion,” IEEE Trans. Softw. Eng., vol. 44, no. 10, pp. 914–931, Oct. 2018.
M. Papadakis, C. Henard, M. Harman, Y. Jia, and Y. Le Traon, “Threats to the validity of mutation-based test assessment,” in Proc. 25th Int. Symp. Softw. Testing Anal., 2016, pp. 354–365.
R. Ramler, T. Wetzlmaier, and C. Klammer, “An empirical study on the application of mutation testing for a safety-critical industrial software system,” in Proc. ACM Symp. Appl. Comput., 2017, pp. 1401–1408.
O. E. Cornejo Olivares, F. Pastore, and L. Briand, “Mutation analysis for cyber-physical systems: Scalable solutions and results in the space domain,” IEEE Trans. Softw. Eng., early access, Aug. 27, 2021, doi: 10.1109/TSE.2021.3107680.
G. Guizzo, F. Sarro, and M. Harman, “Cost measures matter for mutation testing study validity,” in Proc. 28th ACM Joint Meeting Eur. Softw. Eng. Conf. Symp. Found. Softw. Eng., 2020, pp. 1127–1139.
European Cooperation for Space Standardization, “ECSS-E-ST-40C - Software general requirements,” 2009. [Online]. Available: http://ecss.nl/standard/ecss-e-st-40c-software-generalrequirements/
European Cooperation for Space Standardization, “ECSS-Q-ST-80C Rev.1 - Software product assurance,” 2017. [Online]. Available: http://ecss.nl/standard/ecss-q-st-80c-rev-1-software-productassurance-15-february-2017/
C. Cadar et al., “KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs,” in Proc. 8th USENIX Conf. Operating Syst. Des. Implementation, 2008, pp. 209–224.
T. T. Chekam, M. Papadakis, and Y. Le Traon, “Mart: A mutant generation tool for LLVM,” in Proc. 27th ACM Joint Meeting Eur. Softw. Eng. Conf. Symp. Found. Softw. Eng., 2019, pp. 1080–1084.
A. Denisov and S. Pankevich, “Mull it over: Mutation testing based on LLVM,” in Proc. IEEE Int. Conf. Softw. Testing Verification Validation Workshops, 2018, pp. 25–31.
LLVM, “The LLVM compiler infrastructure,” 2021. [Online]. Available: https://llvm.org/
M. Beller et al., “What it would take to use mutation testing in industry: A study at Facebook,” in Proc. IEEE/ACM 43rd Int. Conf. Softw. Eng.: Softw. Eng. Pract., 2021, pp. 268–277.
G. Petrović and M. Ivanković, “State of mutation testing at Google,” in Proc. 40th Int. Conf. Softw. Eng.: Softw. Eng. Pract., 2018, pp. 163–171.
P. Delgado-Pérez, I. Habli, S. Gregory, R. Alexander, J. Clark, and I. Medina-Bulo, “Evaluation of mutation testing in a nuclear industry case study,” IEEE Trans. Rel., vol. 67, no. 4, pp. 1406–1419, Dec. 2018.
R. Baker and I. Habli, “An empirical evaluation of mutation testing for improving the test quality of safety-critical software,” IEEE Trans. Softw. Eng., vol. 39, no. 6, pp. 787–805, Jun. 2013.
