Your SlideShare is downloading. ×

A 64-bit horse that can count

252

Published on

The article concerns the peculiarities of Visual C++ compiler's behavior when generating 64-bit code and …

The article concerns the peculiarities of Visual C++ compiler's behavior when generating 64-bit code and
possible errors relating to it.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
252
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. A 64-bit horse that can countAuthor: Andrey KarpovDate: 11.05.2009AbstractThe article concerns the peculiarities of Visual C++ compilers behavior when generating 64-bit code andpossible errors relating to it.IntroductionThe phenomenon of "The Clever Hans", Mr. von Ostens horse, was described in 1911 [1]. The CleverHans was famous because of his ability to read and solve mathematical problems by tapping with hisfront hoof. Of course, there were a lot of skeptics. Thats why a team of experts tested Hans abilitiesand proved that the horse was showing them without any help of Mr. von Osten. But how could acommon horse possess such an intellectual level - a human one?! The psychologist O. Pfungst carriedout some very thorough experiments and discovered that Hans received very faint unintentional hintsfrom those who were asking him questions. For example, when people asked Hans about anything theystarted to stare at his front hoof with the help of which the horse "answered". But as soon as Hans hadtapped the right number, they raised their eyes or head just a little waiting for him to finish his answer.And the horse, that had been trained to note and use these very subtle motions considered them assignals to stop his action. From aside it looked as if the horse had given the right answer to the question.Such a wonderful horse it was that counted and solved arithmetic problems although he was unable todo it. 64-bit programs turned out to be such digital horses of the beginning of the 21st century, many ofwhich cannot count either although are successful in pretending to do so. Lets consider thisphenomenon in detail.1. Potential errorsI am the author and co-author of some articles devoted to the problems of developing 64-bitapplications. You can see the articles on our site: http://www.viva64.com/articles/64-bit-development/.In these articles, I try to use the term "a potential error" or "a hidden error" rather than just "an error"[2, 3, 4].This is explained by that one and the same code can be viewed upon as both correct and incorrectdepending on its purpose. A simple example - using a variable of int type for indexing an arrays items. Ifwe address an array of graphics windows with the help of this variable, everything is alright. We neverneed, and moreover it is impossible, to operate billions of windows. But when we use a variable of inttype for indexing an arrays items in 64-bit mathematical programs or data bases, it can well be aproblem when the number of items excesses 0..INT_MAX range.But there is one more much subtler reason to call errors "potential". The point is that it depends notonly on the input data but on the mood of the compilers optimizer if an error occurs or not. I have beenavoiding this topic for a long time for most of such errors occur explicitly in the debug-version and onlyin release-versions they are "potential". But not every program built as debug can be debugged at large
  • 2. data sizes. There is a situation when the debug-version is tested only at very small sizes of data. Andoverload testing and testing by end users at actual data is performed only in release-versions whereerrors can be temporarily hidden. Thats why I decided to tell you what I know about it. I hope that I willmanage to persuade you that it is dangerous to rely only on the checks of the execution stage (unit-tests, dynamic analysis, manual testing) when porting a program on a different platform. You will saythat all this is meant for promoting Viva64 tool. Yes, you are right, but still read the horror stories Imgoing to tell you. I am fond of telling them.2. How it all begun- Why do you have two identical JMPs in a row in your code?- What if the first one wouldnt work?I faced the peculiarities of Visual C++ 2005 compilers optimization for the first time when developingPortSample program. This is a project included into Viva64 distribution kit and is intended fordemonstrating all the errors which Viva64 analyzer diagnoses. The examples included into this projectmust work correctly in 32-bit mode and cause errors in 64-bit one. Everything was alright in the debug-version but I faced difficulties in the release-version. The code which was to lead to a hang or crash in64-bit mode worked successfully! The cause lay in optimization. The solution consisted in additionalredundant complication of the examples code and adding "volatile" key words which you can see inPortSample project in a great number.The same relates to Visual C++ 2008. The code differs a bit but everything written in this article can beapplied both to Visual C++ 2005 and Visual C++ 2008. We wont make any difference between themfurther.If you think that it is good that some errors dont occur, refuse this thought. Code with such errorsbecomes very unstable and a smallest change of it not relating directly to an error can cause change ofthe codes behavior. To make sure, I would like to point out that this is not the fault of the compiler butof the hidden defects of the code. Further, we will show sample phantom errors which disappear andoccur in release-versions when smallest alterations of the code are introduced and which you have tohunt for a long time.3. PhantomsThe section will be long and boring, so I will begin with a funny story which is an abstract of the section:Once Heracles was walking by a lake and there he saw Hydra. He ran up to her and cut her single headoff. But instead of one head two more grew. Heracles cut them off too but 4 more appeared. He cut the 4heads off - and there were 8 ones... So passed one hour, two hours, three hours... And then Heracles cutHydras 32768 heads off and Hydra died for she was 16-bit.Like in this funny story errors lie in types overflow which can occur or fail to occur depending on thecode the compiler will generate when optimization is enabled. Lets consider the first example of thecode which works in release mode although it shouldnt be so:int index = 0;
  • 3. size_t arraySize = ...;for (size_t i = 0; i != arraySize; i++) array[index++] = BYTE(i);This code fills the whole array with values correctly even if the arrays size is much larger than INT_MAX.Theoretically it is impossible because index variable has int type. Some time later, because of theoverflow access to items by a negative index must occur. But optimization leads to generating thefollowing code:0000000140001040 mov byte ptr [rcx+rax],cl0000000140001043 add rcx,10000000140001047 cmp rcx,rbx000000014000104A jne wmain+40h (140001040h)As you can see, 64-bit registers are used and there is no overflow. But lets alter the code a bit:int index = 0;for (size_t i = 0; i != arraySize; i++){ array[index] = BYTE(index); ++index;}Lets consider that the code look more beautiful this way. I think you will agree that functionally itremains the same. But the result will be quite different - a program crash will occur. Lets examine thecode generated by the compiler:0000000140001040 movsxd rcx,r8d0000000140001043 mov byte ptr [rcx+rbx],r8b0000000140001047 add r8d,1000000014000104B sub rax,1000000014000104F jne wmain+40h (140001040h)That very overflow occurs that must occur in the previous example as well. r8d = 0x80000000 registersvalue extends into rcx as 0xffffffff80000000. The consequence is writing outside the limits of the array.Lets consider another example of optimization and see how easy it is to spoil everything:unsigned index = 0;for (size_t i = 0; i != arraySize; ++i) { array[index++] = 1;
  • 4. if (array[i] != 1) { printf("Errorn"); break; }}Assembler code:0000000140001040 mov byte ptr [rdx],10000000140001043 add rdx,10000000140001047 cmp byte ptr [rcx+rax],1000000014000104B jne wmain+58h (140001058h)000000014000104D add rcx,10000000140001051 cmp rcx,rdi0000000140001054 jne wmain+40h (140001040h)The compiler decided to use 64-bit register rdx for storing index variable. As a result the code maycorrectly process arrays with the size more than UINT_MAX.But the world is fragile. It is enough just to complicate the code a bit and it becomes incorrect:volatile unsigned volatileVar = 1;...unsigned index = 0;for (size_t i = 0; i != arraySize; ++i) { array[index] = 1; index += volatileVar; if (array[i] != 1) { printf("Errorn"); break; }}Using "index += volatileVar;" expression instead of index++ leads to participation of 32-bit registers inthe code and therefore occurrence of overflows:0000000140001040 mov ecx,r8d
  • 5. 0000000140001043 add r8d,dword ptr [volatileVar (140003020h)]000000014000104A mov byte ptr [rcx+rax],1000000014000104E cmp byte ptr [rdx+rax],10000000140001052 jne wmain+5Fh (14000105Fh)0000000140001054 add rdx,10000000140001058 cmp rdx,rdi000000014000105B jne wmain+40h (140001040h)In conclusion I will give an interesting but large example. Unfortunately, I didnt manage to abridge itbecause it was necessary to show the behavior. It is this why such errors are dangerous for you cannotforesee the consequence of a smallest alteration of the code.ptrdiff_t UnsafeCalcIndex(int x, int y, int width) { int result = x + y * width; return result;}...int domainWidth = 50000;int domainHeght = 50000;for (int x = 0; x != domainWidth; ++x) for (int y = 0; y != domainHeght; ++y) array[UnsafeCalcIndex(x, y, domainWidth)] = 1;This code cannot fill correctly the array consisting of 50000*50000 items. It is impossible because whencalculating "int result = x + y * width;" an overflow must occur.Miraculously the array is filled correctly in the release-version. UnsafeCalcIndex function integratesinside the loop and 64-bit registers are used:0000000140001052 test rsi,rsi0000000140001055 je wmain+6Ch (14000106Ch)0000000140001057 lea rcx,[r9+rax]000000014000105B mov rdx,rsi000000014000105E xchg ax,ax0000000140001060 mov byte ptr [rcx],10000000140001063 add rcx,rbx
  • 6. 0000000140001066 sub rdx,1000000014000106A jne wmain+60h (140001060h)000000014000106C add r9,10000000140001070 cmp r9,rbx0000000140001073 jne wmain+52h (140001052h)All this takes place because UnsafeCalcIndex function is simple and can be integrate easily. But once youmake it a bit more complicated or the compiler considers that it shouldnt be integrated, an error occursat large data sizes.Lets modify (complicate) UnsafeCalcIndex function a bit. Pay attention that the functions logic has notbeen changed at all:ptrdiff_t UnsafeCalcIndex(int x, int y, int width) { int result = 0; if (width != 0) result = y * width; return result + x;}The result is a program crash when the arrays limits are exceeded:0000000140001050 test esi,esi0000000140001052 je wmain+7Ah (14000107Ah)0000000140001054 mov r8d,ecx0000000140001057 mov r9d,esi000000014000105A xchg ax,ax000000014000105D xchg ax,ax0000000140001060 mov eax,ecx0000000140001062 test ebx,ebx0000000140001064 cmovne eax,r8d0000000140001068 add r8d,ebx000000014000106B cdqe000000014000106D add rax,rdx0000000140001070 sub r9,10000000140001074 mov byte ptr [rax+rdi],1
  • 7. 0000000140001078 jne wmain+60h (140001060h)000000014000107A add rdx,1000000014000107E cmp rdx,r120000000140001081 jne wmain+50h (140001050h)I think you have become bored by this moment. I am sorry. I just wanted to show you how simply anefficient 64-bit program may fail after introducing most harmless alterations into it or building it byanother version of the compiler.4. Diagnosis of potential errors A program is a sequence of processing errors. (c) An unknown authorI suppose that many already existing 64-bit applications or those which will be soon ported on 64-bitsystems, can suddenly spring more and more unpleasant surprises. A lot of defects may be found inthem when increasing the size of input data which was unavailable for processing in 32-bit systems.Hidden defects can suddenly occur during further modification of the program code or change oflibraries or a compiler.Like in the story about the horse, the first impression can be deceptive. It can only seem to you thatyour program processes large data sizes successfully. You need to perform a more thorough check tosee exactly if your 64-bit horse can actually count.To make sure that a 64-bit program is correct, the minimum thing you can do is to use not only therelease-version but the debug-version as well at all stages of testing. Keep in mind that it is a necessarybut far not sufficient condition. If your tests use data sets which, for example, dont cover a large mainmemory size, an error can fail to occur both in release- and debug-versions [5]. It is necessary to extendunit-tests and data sets for overload and manual testing. It is necessary to make algorithms process newdata combinations which are available only in 64-bit systems [6].An alternative way of diagnosing 64-bit errors lies in using static analysis tools. It is much more radicaland safe than guessing if you have added enough tests or not. It is convenient for it doesnt demandusing the debug-version for crunching gigabytes of data.The point of the method is to perform a full analysis of a project for a single time when porting theprogram and look through all the diagnostic messages on suspicious sections in the code. Many arefrightened off by the list of thousands and tens of thousands of warnings. But the total time spent atonce on analyzing them will be much less than the time spent on correcting various bug-reportsappearing literally from nowhere for many years. It will be those very phantoms described above.Besides, when you start working with the list of warnings you will soon find out that most of them canbe filtered and there will be much less work than you have expected. Further, you will have only to usestatic analysis for a new code and it doesnt take much time.Of course, when speaking about a toolkit for searching 64-bit phantoms, I offer the tool that we develop- Viva64. By the way, this tool will soon be included into PVS-Studio which will unite all our staticanalysis tools.
  • 8. To be more objective and avoid constantly being driven out from sites with this article as an advertisingone, I will mention other tools as well. We should list Gimpel PC-Lint and Parasoft C++test. Rules fortesting 64-bit errors are implemented in them too, but they possess less diagnostic abilities than a highlytailored Viva64 [7]. There is also Abraxas CodeCheck in the new version of which (14.5) functions ofdiagnosing 64-bit errors are also implemented but I dont possess more detailed information about it.ConclusionI will be glad if this article helps you master new platforms easier, for you will know what hiddenproblems can occur. Thank you for attention.References 1. Wikipedia. Clever Hans. http://www.viva64.com/go.php?url=223. 2. Andrey Karpov. 64 bits, Wp64, Visual Studio 2008, Viva64 and all the rest... http://www.viva64.com/art-1-2-621693540.html 3. Andrey Karpov, Evgeniy Ryzhkov. Static code analysis for verification of the 64-bit applications. http://www.viva64.com/art-1-2-184080346.html 4. Andrey Karpov. Seven Steps of Migrating a Program to a 64-bit System. http://www.viva64.com/art-1-2-850243650.html 5. Andrey Karpov, Evgeniy Ryzhkov. 20 issues of porting C++ code on the 64-bit platform. http://www.viva64.com/art-1-2-599168895.html 6. Andrey Karpov, Evgeniy Ryzhkov. Traps detection during migration of C and C++ code to 64-bit Windows. http://www.viva64.com/art-1-2-2140958669.html 7. Andrey Karpov. Comparison of analyzers diagnostic possibilities at checking 64-bit code. http://www.viva64.com/art-1-2-914146540.html

×