Case Study of the Unexplained


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Cuz if I didn't write it, the code obviously sucks.
  • Floating processor flags.
  • “work” Couldn’t redo the bug.
  • There were also bigger.Confronted tester and they admitted to the crime
  • Winzip why are you so bad with tars.
  • Horrible default. Why smart? Does it look at file? No. Does it have a white list? No. Even though there are far more binary file types then text types it surely must not be using a black list? It uses a black list. AAAAAAAHHH stupid stupidstupid.
  • No matter how far they have to walk, or how much extra work.
  • Best guess on assembly
  • PHP DOS attach was caused buy using 64-bit string conversion algorithm with 80-bit chip. Java also fell.
  • C/C++ make it easy to do, unless practicing modern C++.
  • Raise hands who actually know the problem
  • There is the line. Look at the type of d, it’s a pointer not a value. Here is what the compiler does.
  • Now you can see that the what the problem really.
  • Bad solution, and is C with classes style & not C++
  • Missing a *
  • Tell c++ don’t use constructor to do implicit conversion. Still C++ style but will get rid
  • C++03 rule of three if there is any complex or pointer data.
  • These are the move constructor and assignment operator. That’s another talk.
  • CLANG uses a spell check algorithm when it finds an unknown symbol
  • Automagic JNI Junk is a dead tool for generating JNI.
  • Fills memory on allocation, deletion, and add guard on end.Involved steps in the GUI for crash. Had to track down the pointer use.
  • Breakpoint automatically disabled when re-running program.
  • Breakpoint automatically disabled when re-running program.
  • Case Study of the Unexplained

    1. 1. Case Study of Programmer Nightmares Shannon’s Edition 20120624
    2. 2. What is the talk about• Inspired by Mark Russinovich’s Presentation – Case of the Unexplained – us/sysinternals/bb963887• Here are my cases – Mainly fixing programming problem – Mostly C++, some interop & cross-platform. – Most are from my bad memory. – Sorry about the boring slides.
    3. 3. Steps to Debug a problem1. There is no step 12. See Step 13. ???4. Profit
    4. 4. General Guidelines• Reproducible test case.• Learn the tools.• Make a Wild A** Guess (WAG) on source• Persistent – Grind through it.• Ask someone else to handle it. (NOT ME)
    5. 5. Case: WOMM
    6. 6. Problem: Debug vs Release Debug Optimized• Program is not drawing the circle around the cursor but is where the user clicks.• Same class does both drawings, different location• Did work previously
    7. 7. Causes Optimization Problems1. Undefined Behaviors 1. Uninitialized Memory 2. Overflows/underflows2. Thread problems.3. Code or Data is wrong.….999.Complier Bug (not likely, see #1)1000.Hardware/OS/driver bug.
    8. 8. Step I took• What’s changed. – Major merge with other branch. – Massive file and project settings changes.• Build optimized with debug symbols & debug – Could jump around a lot – Local variables will not be present or wrong* – this pointer only valid on member function entry.• Compare working/non-working objects
    9. 9. Found the Function• Formula: A* x B * y C I D*w E *h F 1• D, E, F = 0, so have this, and verified all inputs. I A* x B * y C
    10. 10. Next Trick: Binary Search• Turning on/off optimizations – Per Library – Per File – Per function – Per optimization• Found, Global Optimization “Cause” problem – Last merge turned it on. – Turned it off. Everything works 
    11. 11. Extremely Important Rule• Unless you understand why the problem is fixed, its not fixed. The problem is likely still there just hidden better.
    12. 12. Missed something important• Formula: A* x B * y C I D*w E *h F 1• D, E, F = 0, so have this, and verified all inputs A* x B * y C I 1
    13. 13. Lets talk this Out• w & h were uninitialized, but can’t be it. MAYBE• 0 time any number is 0. TRUE• w & h are number. FALSE – Double Precision IEEE 754• IEEE 754 only contains number. FALSE – Contains ±0, ±INF, … NaN
    14. 14. NaN is weird.• Any operation with NaN results in NaN – *, +, -, /, sin, etc• Most comparisons with NaN are false. – <, <=, >, ==, etc, so NaN == NaN is false• Not equals is always true. – NaN != NaN is true.• Multiple types – QNaN, SNaN
    15. 15. Case Close• Should have trusted 1st guess.• Gave up too soon with a quick wrong fix.
    16. 16. Case: Works Everywhere Else
    17. 17. Problem• 6-8 high priority bugs from FAT.• All bugs had the same pattern. – Only occurred on Window 2000 box. – Display wrong converted values. – Works on XP, and 2003.• It a cross-platform assign to Me.
    18. 18. Steps I took• Start Debug Build of Integration Branch.• Get the release, and try to reproduce bug. – Grabbed it from the build NFS share. – Didn’t “fail”• Try the test box. – It “fails”, but can’t debug. – Copy it to dev box – It fails on my box.
    19. 19. WAG time• Cosmic Rays corrupted the Executable. – No replacing them with debug build still had bug.
    20. 20. What Could It Be• Diff installed w/ what should be there. – Should be No Differences• Massive Differences.• Install CD didn’t have and Differences• I know what happened.
    21. 21. Here is What Happened• Tester skipped using the Install Win2K CD. – Didn’t want to walk to other end of hall.• TAR-ed up NFS install shared.• FTP it over.• Used WinZip to untar file.
    22. 22. Why is WinZip Bad?
    23. 23. Case Closed• The “table.dat” file was converted to windows newlines. – Doesn’t work properly like that.• All Test Follow Proper Procedures.• Don’t take Short Cuts. – Especially During FAT.
    24. 24. Case: Psychic Debugging
    25. 25. Problem: Phone Call1. Got a phone call2. Developer describe the problem and steps taken to track down the problem.3. Answer with the root cause and how to fix.Now its time for the interactive part of this talk.Pretend you me, ….
    26. 26. Real Problem• File parsing code incorrectly errors out. – Worked on following • Windows 32/64-bits debug/release, • Irix 32/64-bits debug/release, • Solaris SPARC 32/64-bit debug/release • Linux 64-bit debug/release, 32-bit debug. – Fails on Linux 32-bit x86 gcc optimize
    27. 27. What does the code do?• Read text like file – Contains repeated floating point numbers. – Lots of other data between repeated number.• Parses data into native types (int, double)• Validate Data is sane – Number are with spec. – Repeated doubles are the same with != check. • This step failed.
    28. 28. Codedouble lat1 = atof(buff1);…double lat2 = atof(buff2);…if(lat1 != lat2) return -1;
    29. 29. I’m 95% certain of problemWrite down your answer now. More info from the developer
    30. 30. Additional QA with develop• Did they check input file is valid? YES• How did the developer track down it down? – Printf debugging number same, but check failed.• Did adding/moving additional printf make the problem go away? YES – This confirmed that I guessed right 
    31. 31. Your Turn• Failed 32-bit x86 optimized linux• Deal with C++ native double types – uses != to compare them.• Adding some printfs made problem go away.Who know what happened.
    32. 32. Additional Slide If No One Knows• Root cause is 486• Specifically math co-processor• C++ doubles are 64-bits in memory• 486 math registers are 80-bits• Can’t store 80-bits in 64-bit• Round double when copied into memory.• Optimizer will speed up code – Will attempt to reduce the # of memory copies.• Wait here until some guesses.
    33. 33. Here is what happened• Function converted 1st string to 80-bit double• Compiler moved result into 64-bit on stack• Function conerted 2nd string to 80-bit double• Compiler got smart and kept it in 80-bits.• Loaded 1st 64-bit double into 80-bit register.• 2nd number has more precision so it didn’t match.
    34. 34. Optimized ASM Codecall atof ; buff1 in eaxfstor [sp+20], ST(0)……call atof ; buff2 in eaxfload ST(1), [sp+20]fcmp ST(0), ST(1) ; compare 80 w/ 64-bitsjmpe +8 ; skip over next line if ==ret ; error
    35. 35. Case Close• Changed to use strcmp instead.• Never directly compare double without a tolerance.• Round errors will cause mathematically impossible to happen.• Stupid 80-bits.
    36. 36. Case: Shoot Self in Foot
    37. 37. Problem: Crash with no reasons• New developed code• Crashed on Solaris while calling constructor• No “obvious” problem with code
    38. 38. Codeclass A { … A(A *d) { *this = d; } … A& operator=(const A &d) { … return *this; }};
    39. 39. Steps I took• Build code on Windows. – Visual Studio Debugger is 10x nicer• Got a helpful warning – warning C4717: ‘A::A’ : recursive on all control paths, function will cause runtime stack overflow
    40. 40. Code Againclass A { A(A *d) { *this = d; } A& operator=(const A &d) {…}};
    41. 41. What the Compiler Doesclass A { A(A *d) { A __tempA(d); *this->operator=(__tempA); } A& operator=(const A &d) {…}};
    42. 42. Solution #1class A { A(A *d) { *this = *d; } A& operator=(const A &d) {…}};
    43. 43. Problem With Solution #1• What does the following code do A d = NULL;• Compile does this following A d = A(NULL);• Which crashes.• “A d = 0” also crashes.
    44. 44. Solution #2class A { explicit A(A *d) { *this = *d; } A& operator=(const A &d) {…}};
    45. 45. C++ “Rule of 3” Solutionclass A { A(const A &d) {…} ~A() {…} A& operator=(const A &d) {…}};
    46. 46. C++11 “Rule of 3,4, or 5” Solutionclass A { A(const A &d) {…} A(A &&d) {…} ~A() {…} A& operator=(const A &d) {…} A& operator=(A &&d) {…}};
    47. 47. Case Close• Pay Attention to compiler warnings. – This particular warning appear in 3 other places.• Use Compiler that give better warnings. – CLANG/LLVM has the best error/warnings.
    48. 48. Case: “Random” Crashes
    49. 49. Problem: GUI randomly crashes Java Automagic JNI Junk C++
    50. 50. Steps I took• Build Debug – debug runtimes make it crash faster due to checks• Use 2 Debugger Visual Studio & JBuilder• 4 hours of persistent.
    51. 51. Track it down, but no clue• Java had valid pointer to C++ object.• Pressed button, & pointer no longer valid• Trick time.
    52. 52. Data Breakpoint.• x86 has 4 hardware data breakpoints – Program runs at full speed. – 1 is reserved by OS• Must take following form. (Old Info) – Memory address, length(must be 4). – 0x12345678,4
    53. 53. How to do it VS2010• Step 1
    54. 54. How to do it VS2010• Step 2
    55. 55. How to do it VS2010• Step 3 Done
    56. 56. How to do it VS2010• Step 4 See Results
    57. 57. BAM Data Changed• Java GC – > finalizer – > Automagic JNI junk – > delete object• Why, leaky abstraction.
    58. 58. Here is What Happened.Java C++AMJJArray ARRAY | | | | | | |AMJJThing
    59. 59. Case Close• Data Breakpoints Rule.• All Abstraction Leak – Know how before proceeding.
    60. 60. That’s all for NowQuestions, Comment, etc.