Redakcja
Introduction
Software testing is one of the fundamental elements of software development process that can ensure increased reliability and improved quality of products. Among the different purposes of testing one can find: identifying defects, building confidence in the quality of the software, delivering the information needed to make decisions or prevent the creation of defects [ISTQB, 2011, p.13]. However, is it possible that testing can prove that the software is free from defects? Do long hours verification of software, or checking all possible combinations of data inputs and initial conditions give confidence that there are no errors in it?
In this paper, I present some citations from several information sources that provide evidence of the thesis: testing cannot confirm that the software is without defects.
The thesis to substantiate: Testing cannot prove that software is free from defects.
The first argument can be found in the ISTQB Certified Tester Foundation Level syllabus. This document forms the basis of information about software testing and includes, among others things, Seven Testing Principles. The first principle says that testing shows the presence of errors in a product, but ”cannot prove that there are no defects” [ISTQB, 2011, p.14]. And moreover, ”testing reduces the probability of undiscovered defects remaining in the software but, even if no defects are found, it is not a proof of correctness” [ISTQB, 2011, p.14].
Another argument is the second principle which says that exhaustive testing is not possible. In light of the fact that ”testing everything (all combinations of inputs and preconditions) is not feasible except for trivial cases” [ISTQB, 2011, p.14], the analysis of risks and priorities should be used to define the scope of testing.
The further proofs that confirm this thesis may be found in the book Software Testing by Ron Patton. The theory and concept of testing described by Patton is easily understandable to readers. The author clearly writes that ”software testing works exactly as the exterminator does. It can show that bugs exist, but it can't show that bugs don’t exist. (…) but at no point can you guarantee that there are no longer any bugs to find” [Patton, 2000, p.41]. In addition, it stresses that the verification of a software program is very important and desirable, but such activity as: ”fully test it, find all the bugs, and assure that the software is perfect. Unfortunately, this isn’t possible, even with the simplest programs, due to four key reasons:” firstly, ”the number of possible inputs is very large”; secondly, ”the number of possible outputs is very large”; thirdly, ”the number of paths through the software is very large”; and finally, fourthly, ”the software specification is subjective” [Patton, 2000, p.38]. Moreover, he points out that ”one of the sad realities of software testing is that even after all your hard work, not every bug you find will be fixed” [Patton, 2000, p.42]. Actually, it is a common practice. Sometimes, it is safer to leave known errors to avoid the risk of creating new ones. I have encountered this situation several times. For example, when a defect is found in a rarely used feature, it is better to leave it than to risk creating new, unknown errors that may be more critical. Of course, any such decision – which errors will be fixed and which will not – should be based on a business risk analysis and be well founded. And there are numerous different reasons for leaving errors, such as: the bug has a very low impact for an end user; the bug only appears in a really specific test scenario or environment; there is a system or technology limitation; the time and effort required to fix the bug outweigh the benefits of fixing it; or it is just not worth it. I cannot fail to mention the favorite phrase of developers, 'It's not a bug, it's a feature!'
The next position supporting the postulated thesis is a Software Testing and Quality Assurance: Theory and Practice book by Kshirasagar Naik and Priyadarshi Tripathy. This book is based on the teaching and industrial experiences of authors in software testing and quality assurance. They say that, ”Complete, or exhaustive, testing means there are no undiscovered faults at the end of test phase. All problems must be known at the end of complete testing. For most of the system, complete testing is near impossible.” [Naik & Tripathy, 2010, p.13]. Additionally, they also highlight important aspects such as ”We must realize that though the outcome of complete testing, that is, discovering all faults, is highly desirable, it is a near-impossible task, and it may not be attempted.” [Naik & Tripathy, 2010, p.13]. This fact must be accepted and should be used in the software development process. The testing should focus on the most important functionalities and the most vulnerable areas of the product. The range of test activities should be specified based on analysis of risk. Personally, I agree with these statements. Let us consider, for example, a system that, among others, enables the control of another device via an application. Then, this application allows a user to create different rules for the device. There are some parameters to set for them: switching on or off, start time (hour, minute, day, month, year), and duration (minutes, hours, and days). In theory, I should check every starting time for every day with all combinations of time duration. So within one day, I have twenty-four hours to choose, and for every hour, sixty minutes to choose. Then for every starting time, I should verify all possible durations: one minute, two minutes, ..., one hour, one hour and one minute, and so on. Then, I need to check these cases for the next days, all days in the next months, years, and so forth. But what if I want to increase the number of controlled devices at the same time as well as at different times? And what if the application has to be released in six languages? Theoretically, each of these additional conditions should be tested. Moreover, editing these rules should be verified, as well. There are literally over a billion of potential combinations. It is fairly obvious that with so many available cases, it is impossible to fully test all them. For this reason, testing a single feature cannot take up the whole of the test’s time. After all, most software projects have many various features to verify. Therefore, spending too much time testing one aspect could result in too little testing of remaining ones, or not testing them at all. Moreover, if one person repeats the same scenario for the hundredth time, overlooking an error is highly likely.
Additional proof may be found in Software Reliability Methods book by Doron A. Peled. This book presents methods and techniques which can be used for dealing with software reliability problems. One of the methods of software testing he mentions is control coverage flow technique. It has various limitations, for example: ”First, coverage is hardly comprehensive: it is usually impossible to cover all paths, and thus errors may be left unfound, as an existing error can occur on a path that was nor covered”.[Peled, 2001, p.257].
Furthermore, it is worth mentioning that this topic is not something new for the software testing community. The question: 'Is software free from errors or not?' was posted many years ago. For example, such answer can be found in the book Design Assurance for Engineers and Managers by John A. Burgess: ”A final word about testing – its main purpose is to find defects, but experience shows that testing may not find all errors in a program. Therefore, just because no errors are found in the final phrase of testing, this does not assure that the program is now error-free.” [Burgess, 1984, p.140].
Additionally, a few arguments can be found in articles related to testing. Kingsley Asuamah in his article entitled 10 Software Testing Fallacies writes, ”The believe that the software application should be 100% bug free, if a tester with superb testing skills tested the application. Yes Zero Defects may sound good, but the reality is, Software Development is an adaptive system. We can't predict all the effects and outcomes even in very predictable systems” [Asuamah, 2016]. He also notes that testing activities should minimize the amount of errors in software and there is nothing like ”The Last Bug” [Asuamah, 2016]. Moreover, in his The Zero Bug Policy article Sam Hatoum states that ”Bugs will always exist” [Hatoum, 2016]. Since the defects will always be present in the software, there is no possibility to eliminate all of them by testing. Due to this, it cannot guarantee that the software is free of them. So testing can reduce the number of errors and improve the quality of products.
Summary
All above-mentioned examples and citations confirm the thesis. The testing cannot prove that the software is free from defects or is perfect. Bugs that will be found and fixed during testing can improve the quality of the software.
Additionally, I agree with all the arguments mentioned. It is generally true that finding all defects is impossible, but testing and discovering some of them can prevent many errors and product failures. The found and fixed bugs can increase confidence in the quality of the software. While there is no possibility of total verification of software, it is essential to determine a relevant scope of test activities. The main goal is to select an optimal amount of testing, not to test too much or too little. This is not a simple task, but it is important to learn how to design and choose test cases that minimize the risk and optimize the testing process and available resources.
References
- ISTQB Certified Tester Foundation Level syllabus, Released Version 2001
- Ron Patton (2000). Software Testing, 800 E. 96th St., Indianapolis, Indiana, 46240 USA, SAMS
- Kshirasagar Naik, Priyadarshi Tripathy (2010). Software Testing and Quality Assurance: Theory and Practice, John Wiley & Sons
- Doron A. Peled (2001).Software Reliability Methods, Springer
- John A. Burgess (1984). Design Assurance for Engineers and Managers, Marcel Dekker, New York, NY
- Kingsley Asuamah (2016, July 11). 10 software testing fallacies, Retrieved from https://www.utest.com/articles/10-software-testing-fallacies
- Sam Hatoum (2016, May 9). Zero-bug software development, Retrieved from https://medium.com/quality-faster/the-zero-bug-policy-b0bd987be684
Wstęp
Testowanie oprogramowania jest jednym z podstawowych elementów wytwarzania oprogramowania, który może zapewnić większą niezawodność oraz lepszą jakość dostarczanych produktów. Jak powszechnie wiadomo mogą być różne cele testowania. Może to być znajdowanie usterek, budowanie zaufania co do jakości testowanego oprogramowania, dostarczanie niezbędnych informacji do podejmowania decyzji, jak i zapobieganie powstawaniu nowych defektów. Czy można jednak powiedzieć, że testowania daje gwarancję, że oprogramowanie jest pozbawione błędów? Czy wielogodzinne weryfikowanie oprogramowania, sprawdzanie wszystkich możliwych kombinacji danych wejściowych i warunków początkowych daje pewność, że nie ma już żadnych innych — niewykrytych błędów? W poniższym opracowaniu przedstawiam kilka cytatów zaczerpniętych z piśmiennictwa, które stanowią dowody słuszności postawionej tezy: Testowanie nie może udowodnić, że oprogramowanie jest wolne od defektów.
Podsumowanie
Wspomniane przykłady i cytaty stanowią jasne argumenty potwierdzające postawioną tezę. Testowanie nie jest w stanie udowodnić, że oprogramowanie jest pozbawione defektów czy też zapewnić, że jest ono idealne. Wykrycie wszystkich defektów, uwidocznienie wszystkich błędów w oprogramowaniu jest niemożliwe. Natomiast warto podkreślić, że usterki, które zostały znalezione w fazie testowania, a następnie naprawione, mogą znacząco podnieść jakość oprogramowania lub systemu. Wobec tego ‘wyłapanie’ nawet niektórych z nich i ich naprawa może przyczynić się do zmniejszenia ryzyka wystąpienia problemów i awarii w czasie użytkowania oprogramowania. Biorąc pod uwagę fakt niemożności przetestowania całego oprogramowania czy sprawdzenia wszystkich możliwych kombinacji danych wejściowych i ścieżek należy określić odpowiedni zakres czynności testowych. Zatem kluczowym wydaje się być wyznaczenie optymalnej ilości testów i/lub scenariuszy testowych. Oczywiście, nie jest to proste zadanie. W związku z tym bardzo ważne jest, aby nauczyć się właściwie projektować i wybierać przypadki testowe tak, by zminimalizować ryzyko a proces testowy zoptymalizować w zależności od dostępnych zasobów.
Judyta Michalska
