64-bit Loki


Published on

The article is a report about testing of portability of Loki library with 64-bit systems with the help of Viva64 code analyzer performed by OOO "Program Verification Systems" Company. It contains recommendations for users of the library. The article will be also useful for the users of other libraries built on templates for it describes the peculiarities of analysis of such libraries.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

64-bit Loki

  1. 1. 64-bit LokiAuthors: Evgeniy Ryzhkov, Andrey Karpov, Andrei Alexandrescu, Peter Kummel, Rich Sposato,Date: 25.09.2009AbstractThe article is a report about testing of portability of Loki library with 64-bit systems with the help ofViva64 code analyzer performed by OOO "Program Verification Systems" Company. It containsrecommendations for users of the library. The article will be also useful for the users of other librariesbuilt on templates for it describes the peculiarities of analysis of such libraries.IntroductionLoki library has been developed by Andrei Alexandrescu as part of the highly acclaimed book "ModernC++ Design: Generic Programming and Design Patterns Applied". An abstract of the book declares: "Thisbook introduces the concept of generic components-reusable design templates that produce boilerplatecode for compiler consumption-all within C++. Generic components enable an easier and more seamlesstransition from design to application code, generate code that better expresses the original designintention, and support the reuse of design structures with minimal recoding."OOO "Program Verification Systems" Company creating Viva64 code analyzer for developing 64-bitapplications participates actively in communication with the authors of various software projects. Onceone of Loki project administrators, Rich Sposato, asked us to check the code of the library for portabilityto 64-bit systems with the help of our code analyzer Viva64. The Loki library is known for using the mostpowerful and up-to-date abilities of C++ language; so if Viva64 managed Loki then it can easily managesimpler projects.We should notice that this article is based on the Loki version of May 2009 (it is more modern than theofficial version of that time Loki 0.1.7), so the stated problems will be removed in further Loki versions.So, we downloaded the latest version of Loki from the SVN repository on SourceForge and set to work.Building of the 64-bit version of Loki in Microsoft Visual Studio 2005Viva64 code analyzer integrates into Microsoft Visual Studio development environment, thats why itwould be sensible to build a version of Loki for this particular environment. Loki package contains readyfiles of solutions for Visual Studio 2005 and Visual Studio 2008. But now, in May 2009, these solutionscontain only 32-bit configurations. Thats why we needed to create configurations for x64 platform inVisual Studio. After adding the necessary configurations we could launch compilation of the 64-bitversion.Loki library is intended for operating on many various platforms and is built with the help of many mostpopular compilers. It explains why the 64-bit Loki version has been compiled nearly at once. The Lokilibrary is actually only one of 20 projects in the most recent version of Loki. The other 19 projects aretest programs which exercise various parts of Loki to demonstrate the code compiles and runs correctly.By including so many test programs with Loki, the programmers can provide guarantees for much of
  2. 2. Lokis functionality despite the complexity of the Loki source code. Out of 20 projects in the solutiononly SafeFormat failed to build:========== Build: 19 succeeded, 1 failed, 0 up-to-date, 0 skipped==========------ Build started: Project: SafeFormat, Configuration: Debug x64 ------Compiling...main.cpp.main.cpp(255) : error C3066: there are multiple waysthat an object of this type can be called with these arguments ....includeloki/SafeFormat.h(109): could beLoki::PrintfState<Device,Char>&Loki::PrintfState<Device,Char>::operator ()(bool)... while trying to match the argument list (UInt)The text of the error is abridged a bit for it would take the whole page in a full form. Lets consider thecode causing the error:void test_dword(){ typedef signed int Int; typedef unsigned int UInt; typedef signed long Long; typedef unsigned long ULong; Int i(0); UInt ui(0); Long l(0); ULong ul(0); Printf("%d")(i); Printf("%d")(ui); // the problem is in this line Printf("%d")(l); Printf("%d")(ul);}
  3. 3. The PrintfState struct uses the LOKI_PRINTF_STATE_FORWARD macro to cast various parameter typesto unsigned long. We found one problem here where a 64 bit parameter may get sliced down to 32 bitsby this code snippet inside the file SafeFormat.h:#if (defined(_WIN32) || defined(_WIN64)) LOKI_PRINTF_STATE_FORWARD(unsigned long)#elseWe recommend using this snippet to avoid slicing 64 bit types:#if (defined(_WIN32) || defined(_WIN64))#if (defined(_WIN64)) LOKI_PRINTF_STATE_FORWARD(unsigned int)#endif LOKI_PRINTF_STATE_FORWARD(unsigned long)#elseAfter this single correction the compilation error disappears and all the 20 projects of the librarycompiled with only some diagnostic warnings. One such warning tells us about a potentially unsafeconversion of size_t type inside typicalUse() function in the file CachedFactoryTest.cpp:// Registering objectsfor(size_t i=0;i<objectKind;i++) CC.Register(i, createProductNull);The control variable (objectKind) of the loop is an unsigned type. After changing the type of the loopcounter to unsigned so it matches the type of the control variable, the warning disappears: // Registering objectsfor(unsigned i=0;i<objectKind;i++) CC.Register(i, createProductNull);After these small corrections the 64-bit library compiles successfully and does not show any diagnosticwarnings on 64-bit mode. But is the librarys code actually correct? By running the many test projectsand seeing no incorrect results or assertion failures, we can confidently say the Loki library is correct.We also used our code analyzer, Viva64, to perform static analysis of Loki.Testing of the 64-bit Loki version with the help of Viva64To make sure that Loki is compatible with 64-bit systems, lets perform analysis of the code usingViva64. Viva64 code analyzer is intended for developing new 64-bit applications and porting of existing32-bit ones on a 64-bit platform.
  4. 4. During the analysis of Loki, Viva64 detected 89 potentially unsafe syntactic constructions. It does notmean Loki projects contain 89 errors related to 64-bit code, but that the developers should review these89 places should be to understand if these are errors. Of course, we have studied these locations in thecode to ascertain if there are actual errors. Many of the 89 locations are inside test projects rather thaninside Loki itself.1 Incorrectly used constant LONG_MINLets begin with an error relating to the incorrectly used constant LONG_MIN in the following function:char* RenderWithoutSign(LOKI_SAFEFORMAT_SIGNED_LONG n, char* bufLast, unsigned int base, bool uppercase)It is located in the file SafeFormat.h. The problem is in this line:if (n != LONG_MIN) {LOKI_SAFEFORMAT_SIGNED_LONG type is defined as a type capable of storing 64-bit values in a 64-bitsystem. In Unix-systems (with LP64 data model) long type is used for this purpose. But in 64-bitWindows-systems (LLP64 data model) long type remained 32-bit. Thats whyLOKI_SAFEFORMAT_SIGNED_LONG type is defined in Loki in this way:#if defined(_WIN32) || defined(_WIN64) #define LOKI_SAFEFORMAT_SIGNED_LONG intptr_t #define LOKI_SAFEFORMAT_UNSIGNED_LONG uintptr_t#else #define LOKI_SAFEFORMAT_SIGNED_LONG signed long #define LOKI_SAFEFORMAT_UNSIGNED_LONG unsigned long#endifAs long type remained 32-bit in 64-bit Windows-systems, LONG_MIN constant defines the minimumvalue of the 32-bit variable. This means that using it is incorrect when operating 64-bit types (in this caseintptr_t). A strongly recommended solution is to use your own constant so you can guarantee its valueremains the same for all platforms, or so you can control its value per platform rather than relying onthe compiler. One possible correction is as follows:#if defined(_WIN32) || defined(_WIN64)# define LOKI_SAFEFORMAT_SIGNED_LONG intptr_t#if defined(_WIN64)# define LOKI_SAFEFORMAT_SIGNED_LONG_MIN_VALUE LLONG_MIN# define LOKI_SAFEFORMAT_SIGNED_LONG_MAX_VALUE LLONG_MAX#else# define LOKI_SAFEFORMAT_SIGNED_LONG_MIN_VALUE LONG_MIN
  5. 5. # define LOKI_SAFEFORMAT_SIGNED_LONG_MAX_VALUE LONG_MAX#endif...#else# define LOKI_SAFEFORMAT_SIGNED_LONG signed long# define LOKI_SAFEFORMAT_SIGNED_LONG_MIN_VALUE LONG_MIN# define LOKI_SAFEFORMAT_SIGNED_LONG_MAX_VALUE LONG_MAX...#endifConsequently, the lineif (n != LONG_MIN) {must be replaced withif (n != LOKI_SAFEFORMAT_SIGNED_LONG_MIN_VALUE) {Fortunately, the creators of Loki library must be praised because this is the only line needing thiscorrection. All the other comments described further may be interesting but they are not important.2 Some types of magic numbers - errors or not?Many of the problems detected by the Viva64 analyzer in Loki are because the lines seem to containmagic numbers. From the viewpoint of migration of code from a 32-bit platform on a 64-bit one, somenumbers are the most dangerous. Perhaps, a programmer wants to have a definite size of a data type insome code section and this may cause a problem. Those who look through the warnings of the codeanalyzer often complain that they are irrelevant. Really, why should the code analyzer about number 4in lines like these?::Loki::ScopeGuard guard4 = ::Loki::MakeGuard( &HasFour, 1, 2, 3, 4 );::Loki::ScopeGuard guard5 = ::Loki::MakeGuard( &HasFive, 1, 2, 3, 4, 5);Sometimes there appear such constructions that must be studied very thoroughly. For example, in thefile SafeFormatmain.cpp we see the code:case X: // TestCase(formatSpec, RandomInt(-10000, 10000)); // dont test negative values on 64bit systems, because // snprintf does not support 64 Bit values TestCase(formatSpec, RandomInt( -10000 * (sizeof(size_t)>4 ? 0 : 1) , 10000));
  6. 6. break;case e:Of course, this particular use of a magic number is not an actual problem, but detecting and diagnosingmagic numbers helps make the code more portable from one operating system to another and from 32bits to 64 bits.3 Addition of int to a pointer as a potential errorThe file flexsimplestringstorage.h contains the function:void resize(size_type newSize, E fill){ const int delta = int(newSize - size()); if (delta == 0) return; if (delta > 0) { if (newSize > capacity()) { reserve(newSize); } E* e = &*end(); flex_string_details::pod_fill(e, e + delta, fill); } pData_->pEnd_ = pData_->buffer_ + newSize;}Viva64 analyzer warns about a potential problem here:flex_string_details::pod_fill(e, e + delta, fill);The defect occurs in adding, delta, a variable of int type, to e, a pointer. This is a potential problem sincethe pod_fill function will not be able to process a data size of more than 2 GB (INT_MAX characters).This particular instance is not a no problem for there are hardly any lines that occupy more than 2 GB.Even so, it is better to change delta from an int type to a ptrdiff_t type:const ptrdiff_t delta = ptrdiff_t(newSize - size());4 Using int for indexing arrays is incorrectTo access very large data arrays - more than INT_MAX elements - we recommend using types ptrdiff_tor size_t instead of unsigned int or unsigned long. The file SmallObjSmallObjBench.cpp contains a large
  7. 7. macro LOKI_SMALLOBJ_BENCH_ARRAY in which an int is used to index an array since compilers oftenuse 32 bit variables for ints, but must use an appropriately sized data type for size_t.5 Right arguments of functionsThe file CachedFactoryCachedFactoryTest.cpp contains the following function:template< class Cache >milliSec typicalUse(Cache &CC, unsigned objectKind, unsigned maxObjectCount, unsigned maxIteration)We recommend using the size_t type for the objectKind parameter, but as this code exists only in a testproject, the code does not affect the Loki library itself.Loki library is compatible with 64-bit systems - does it mean that we cansay the same about a program using it?All the few problems of Loki library described above are easy to correct. Does that mean that if Lokidoes not contain any 64-bit problems (and this is true) than any application using this library is also safefrom the viewpoint of 64-bit code? Unfortunately, it is not!The point is that Loki library uses templates actively, so when the code analyzer examines a templatecode sometimes it cannot detect a problem. To be absolutely sure the analyzer needs to performinstancing of template classes and functions.Lets give an example that does not relate to Loki library. Among other problems Viva64 analyzer candetect non-optimal data structures in code:template <class T>struct TClass{ int m_a; T m_b; int m_c;};If T has int type here the structure is optimal. But if T has size_t type the structure will occupy 24 bytesinstead of possible 16 bytes. When there are many such objects, it is better to rewrite the code in thisway:template <class T>struct TClass{ T m_b;
  8. 8. int m_a; int m_c;};But the analyzer can check it only by instancing the template. That is, you cannot detect a problemwhen there is only definition of a class in the header.Another example, again not relating to Loki, concerns type conversion:template<typename T1, typename T2>class TemplateClass{public: void test1() { m_a.m_value = m_b.m_value; // is there an error here? }private: T1 m_a; T2 m_b;};In this code the error of type conversion can or cannot occur depending on the parameters with whichinstancing of TemplateClass template is performed. Without performing instancing, by simply analyzingthe functions code, the analyzer cannot detect an error.The two examples of template classes described above do not relate to Loki library but they areimportant for understanding the principles of code analyzers operation. The peculiarity of templatelibraries like Loki is that even if a library is completely compatible with 64-bit systems it does not meanthat the code using it is correct. This point changes completely the approach to verification ofapplications. Unlike common (non-template) libraries when complete compatibility with 64-bit systemsis enough to make sure that the whole application is correct, in case of template libraries you cannot besure about it.All this means that although Loki library does not contain problems relating to 64-bit code, a userapplication which uses it must be tested by the code analyzer additionally to see if there are suchproblems, for errors depend on the parameters with which instancing of templates is performed.ConclusionThe results of testing Loki library on compatibility with 64-bit systems performed by the workers of OOO"Program Verification Systems" Company allowed us to draw the following conclusions:
  9. 9. The library is fully compatible with 64-bit systems and does not contain potential errors. The errorsstated in this article are likely to be corrected very easily.Viva64 code analyzer intended for developing 64-bit applications and porting existing 32-bit ones turnedout to be very efficient when testing a complicated template code of the library. It proves a very goodquality of the code analyzer.Although Loki library does not contain 64-bit problems they can occur in user applications using Loki. Asthe end code depends on the parameters with which templates have been instanced, it is necessary totest user applications with the help of the code analyzer. Only then you can be sure that the userapplication is compatible with 64-bit systems.AcknowledgementsWe would like to thank the following people who assisted us with our analysis of Loki library, orreviewed our work on Loki: • We thank Program Verification Systems technical team who reviewed Loki library and carried out verification of its 64-bit compliance: Andrey Karpov and Evgeniy Ryzhkov. • We thank Loki team: Andrei Alexandrescu, Peter Kummel, and Rich Sposato for cooperation and for checking our article, editing it and providing valuable advice. • We are glad that Rich Sposato expressed the will of working together in future and we would like to cooperate in future with the team of Loki developers.References 1. Loki Library. http://www.viva64.com/go.php?url=515. 2. Viva64 Tool. http://www.viva64.com/viva64-tool.