How we test the code analyzer
Upcoming SlideShare
Loading in...5

Like this? Share it with your network


How we test the code analyzer

Uploaded on

The article describes the testing technologies used when developing PVS-Studio static code analyzer. The developers of the tool for programmers talk about the principles of testing their own......

The article describes the testing technologies used when developing PVS-Studio static code analyzer. The developers of the tool for programmers talk about the principles of testing their own program product which can be interesting for the developers of similar packages for processing text data or source code.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. How we test the code analyzerAuthor: Evgeniy Ryzhkov, Paul EremeevDate: 05.07.2011AbstractThe article describes the testing technologies used when developing PVS-Studio static code analyzer.The developers of the tool for programmers talk about the principles of testing their own programproduct which can be interesting for the developers of similar packages for processing text data orsource code.IntroductionWhen developing and promoting PVS-Studio program product, we pay much attention to issues ofsoftware quality, development processes and principles of programmers labor management. But wehave never discussed the question how we ourselves develop our own program product. Do we usethose technologies, recommendations and practices that we describe in our articles? After all, does thephrase "the shoemakers wife is worst shod" apply to us?In this article, we decided to tell you how we test our program product PVS-Studio. On one hand it isdone to convince our (potential) users in the good quality of our tool. On the other hand, we want toshare our successful experience of using these practices.Before we start speaking about the methods of testing our product we need to briefly remind ourreaders what PVS-Studio is. To learn more see the article "Common information on working with thePVS-Studio analyzer" [2].PVS-Studio is a static code analyzer intended for developers of modern resource-intensive C and C++applications. By "modern applications" we understand 64-bit and/or parallel applications. Developmentof such programs involves some difficulties different from those for traditional programs. For besidescommon and familiar errors like uninitialized pointers detected by any compiler, there are specific typesof problems.We speak about errors in programs which occur when porting 32-bit applications to 64-bit platform. Orwhen paralleling code for multi-processor or multi-core support. It is rather difficult to develop suchapplications because of lack of tools simplifying the creation of 64-bit and parallel programs. PVS-Studiois such a tool.Testing practices used when developing PVS-StudioWhen developing PVS-Studio we use six basic testing methods: 1. Static code analysis. It would be strange to develop a static analyzer without using static analysis. 2. Unit-tests at the level of classes, methods and functions. 3. Functional tests at the level of independently written files.
  • 2. 4. Functional tests at the level of separate files. 5. Functional tests at the level of separate third-party projects and solutions. 6. Functional tests for the interface of the extension plug-in integrated into Visual Studio IDE.Lets briefly describe these methods.As PVS-Studio is a static code analyzer, we use the method of static code analysis when developing it tosearch for issues in the analyzer itself. Firstly, we do not need to be reproached by the fact that we donot use our own product. And secondly, it actually helps us find errors before users find them.Unit-tests at the level of classes, methods and functions allow us to make sure that adding a newfunctionality will not break the existing code. At this level, separate program entities are tested with thehelp of simple small tests. In other words, these are the most common and usual unit-tests.Functional tests at the level of independently written files allow us to make sure that everything thatmust be diagnosed is still diagnosable. All the potential problems detected in the code must always bedetected.Functional tests at the level of separate files allow us to make sure that various separate files containingcode are tested by the analyzer without any issues. By issues here we understand the initialization ofASSERTs, unexpected warnings about errors and crashes as well. For a code analyzer is just the sameprogram as all the others and alas – crashes in it are rather frequent.Functional tests at the level of separate third-party projects and solutions allow us to make sure that theanalyzer can still test projects and the number of diagnostic warnings changes from version to versioncontrollable and it is not chaotic.Of course, besides these methods there are other common approaches like "a programmers alert eyes"or "we are up to release tomorrow, check THIS", but they proved their disadvantages and we try no touse them.User interface functional testing allows us to verify the operability of the extension plug-in used tointegrate the analyzer into Visual Studio IDE. During these tests, not only the availability and expectedfunctionality of individual UI elements are being verified, but also the expected interrelationshipsbetween these elements and the operation of the analyzer itself.Now lets speak about the methods we use in detail.Static code analysis by a static code analyzerDoes the sections title puzzle you? What else can perform static code analysis but a static codeanalyzer? However, there are always some peculiarities arising when developing a tool forprogrammers.As you perhaps know the first versions of programming languages compilers are rarely written in thesevery languages. As a rule, to develop a compiler of a new language an entirely different language is usedthat is a standard for the time. It is nowadays that all C++ compilers are written in C++, but the firstversion was written in C.In the same way, when developing the first version of the static code analyzer, we could not test it. It isdue to this that the first version of our product (it was called Viva64 then) was not 64-bit at all! But 1.10
  • 3. version that appeared January 16, 2007 (that is, 17 days after release of the first version), contained thefollowing line among other innovations: • With the help of Viva64 we prepared the 64-bit version of Viva64! But the user must not worry about choosing the right version. It will be selected automatically during installation.Thus, as soon as we got a static analyzer detecting problems of 64-bit code, we started testing our owncode with its help.Do we benefit from static analysis of our own product? Of course, yes. But still we have some peculiarpoints due to the specific character of our task. For we write articles on how to make 64-bit code, we donot make 64-bit mistakes in the new code. But we benefit from using static analysis in the way ofenhancing diagnosis. For example, we can examine what types of syntactic constructions obviously leadto a false response and exclude their diagnosis.Thus, we benefit from using static analysis.Unit-tests at the level of classes, methods and functionsUnit-tests at the level of classes, methods and functions are a set of tests for checking separate logicalelements of a program. Here are some examples of functionality we provide unit-tests for: • using various rules of diagnosing potentially unsafe constructions; • calculation of types in expressions, type conversion operations etc; • operators work (addition, subtraction and so on); • operation of the files loading and preprocessing mechanisms; • verification of registration information;Of course our unit-tests cover not only these scopes - we just gave some examples.How do we use these unit-tests? When correcting an error in the analyzer or adding a new functionality,unit-tests are launched in release-build mode. If the tests run without problems they are then launchedin debug-build mode under the debugger. It is done to make sure that the ASSERTs the code aboundswith are not initialized. Later, it will be clear why you should not launch the debug-version immediately.Although such tests do allow us to detect errors, still they are not a completely adequate solution. Thepoint is that we have too few unit-tests in PVS-Studio in comparison to the number of unit-tests neededto check "almost a compiler". Thats why it is very difficult to provide wide coverage for unit-tests in ourcase. It is a very labor-intensive task that we will hardly manage.However, unit-tests at the level of classes, methods and functions are the first "defense" layer in ourtesting system.Functional tests at the level of independently written filesWhen developing a code analyzer, it is important not to "lose" diagnosis of those potential errors whichhave been detected since the beginning. Here we use functional tests at the level of independentlywritten files. Absolutely all detected potential problems are gathered into separate files in the form ofsource code. These files are marked in a special way. The lines where the code analyzer must detecterrors contain special mark symbols. And it is stated in these marks how many errors there should be in
  • 4. a particular line: one, two and so on. When something fails to be diagnosed we can see it at once. Forexample, earlier, a warning about two errors was generated in line 17, and now it is only one error. Or,vice versa, unnecessary warnings appeared.This approach resembles in its working principle functional tests at the level of separate projects (we willspeak about them further) but differs from them in high operation speed and the fact that the filesbeing tested were written (and marked) independently.Besides, this system for sure can be used when developing new diagnostic warnings. At first, you shouldmanually write in the files the code where a new error will be diagnosed, then mark it and after that youmay settle to implementation of the errors diagnosis itself. Until there is no implementation, the testingsystem will inform that a diagnosable error must be present here but it is absent. As soon as diagnosis isimplemented the test runs correctly.Functional tests at the level of separate filesTesting of files from real projects (instead of separate classes/methods or files created manually) allowsus to provide wider coverage of the code. It is the fourth layer in our testing system. Our tests includeseparate preprocessed files from various projects: wxWidgets, fox-toolit, CImg, Lame, Boost, CxImage,FreeType etc. They also include preprocessed files built on the basis of standard system header files(CRT, MFC etc).After adding a new or correcting an existing functionality the programmer launches the release-versionof the tests at first and then the debug-version. Why not launch the debug-version immediately? Itsvery simple - because the release-version of the tests runs for one minute and the debug-version for fiveminutes.Testing at the level of separate files is a very powerful instrument. It allows you to instantly detect anerror if some functionality has "fallen off" during the analyzers development. A great amount of errorshas been avoided thanks to these tests.Functional tests at the level of separate third-party projects andsolutionsThe most powerful tier of our testing system is its fifth layer which consists of testing for separateprojects and solutions. It is thanks to this system that each new version of PVS-Studio is at least of thesame quality as the previous.This system operates as follows. There are dozens of projects and solutions of various programsavailable in the Internet. For example: Lame, Emule, Pixie, Loki etc. Each of these projects has beentested with PVS-Studio and the testing results have been saved (in the form of a PVS Studio log-file).After installing a new version (a version being developed) we launch a system we have developedspecially for this purpose, and it opens each project in turn, tests it with PVS-Studio, saves the resultsand then compares them to the reference ones. If there are differences it saves them into a separate file(an analogue of a standard diff) that you can easily view with the help of PVS-Studio and find out thereasons of these differences.For example, in the new version of PVS-Studio, a new diagnostic warning with code V118 has appeared.We launch the testing system and it must inform us that V118 warnings appeared in some projects.
  • 5. Then we manually look through all the changes in the results and decide if warning V118 has beengenerated correctly or not.But if besides warning V118 we see that some warnings V115 are missing, it means that the testsshowed a disadvantage of the current version and it is sent back for improvement. In case of admittingall the changes as correct the new log-files are considered reference and the next time the new resultswill be compared to them.This system has one more purpose. Because PVS-Studio product is intended for operation both in VisualStudio 2005 ,2008 and 2010, we always make sure that the warning messages coincide in the differentversions of Visual Studio. That is, if, for example, we have got 10 000 diagnostic warnings in all theprojects in Visual Studio 2005, we must have the same number of warnings in Visual Studio 2008.How much time does such testing take? It is a difficult question - difficult because the base of these testsis constantly increasing because of the new projects. And of course the time spent also increases. A yearago, when the analyzer operated with only one core involved, the tests were performed in about anhour. Then we provided operation in several threads and the time of operation was reduced nearlytwice for a two-core computer. With the lapse of time we had added more and more new projects intothe tests. And now the tests run a bit more than an hour on the same two-core computer. Of course, itis all for the release-version.Functional tests for the interface of the extension plug-in integrated intoVisual Studio IDE.The sixth tier contains the automated user interface testing system. Because the analyzer itself is aconsole application, it does not possess a user interface of its own. But the collection of verified filescompilation data and the creation of a large number of temporary configuration files are required for itsoperation and verification of these mentioned files, thus making the manual launch quite inconvenient.Thats why, drawing the analogy from Visual Studio compiler, all these moments are hidden form theend user and are automated using the extension plug-in for Visual Studio IDE. And it is the GUI testing ofthe extension that is performed on the sixth tier of our system, i.e. in fact the verification is done for theadditional functionality of the interface of Visual Studio IDE which is being provided by the integratedPVS-Studio itself.To cover all functional capabilities of PVS-Studio the system utilizes 3 testing sets (one for each of theVisual Studio versions supported), each containing individual testing scenarios (use cases). Each one ofthese scenarios contains the verification of 3 to 5 test cases (a set of conditions which determine thesatisfaction of the predefined condition). Each one of the testing sets contains identical use cases andidentical requirements for each of the test cases because it is intended that the PVS-Studio analyzershould operate identically in every Visual Studio version supported.The user interface testing system was implemented with the help of Visual Studio Tem Test embeddedsystem for unit testing. (the extension allowing the automated testing for user interface was added inVisual Studio 2010) It is also worth noting that the user interfaces of Visual Studio versions 2005 and2008 are identical for this system. (they possess an identical UI Mapping and are covered by the sameimplementation of testing scenarios) On the contrary, Visual Studio 2010 IDE possesses entirely new UIwhich is based mostly on WPF elements, requiring the separate implementation of these testingscenarios.
  • 6. What lies further?It is known that there is no limit for perfection. We are continuing to develop our testing system in everydirection. Of course, we are constantly enlarging the base of the tests.ConclusionIn this article you have learned how we test our static code analyzer PVS-Studio. Perhaps our experiencewill help you introduce such testing practices into your own working projects. And we hope that afterreading about the process of testing PVS-Studio you would like to have a look at our tool.References 1. PVS-Studio code analyzer. 2. Evgeniy Ryzhkov. Common information on working with the PVS-Studio analyzer.