It is difficultAuthor: Andrey KarpovDate: 05.07.2008AbstractThe article concerns the problem of excess of program creation terms as a result of a prejudice thatprogramming is simple and even simpler.IntroductionFailure to finish development of software in time is a common thing in practice of program productcreation. It is bad for all the participants from the ordinary developers who have to work in hectic rushto the managers and clients who have to put up with lags and extra costs.There is a great amount of reasons for this phenomenon but in this article I would like to suggest thatthe main cause is a wide-spread belief that software development is simple. And I would like toemphasize an interesting thing here. On the whole, everybody understands that program creation israther a laborious process but as it comes to discussion or presentations everyone pursuing his owninterests starts saying that all the difficulties are left behind and it is now when everything becomessimple and easy when we can use a new technology / programming language / methodology. Thishappens again and again for many years. But on the whole there are no improvements and terms areexceeded as before. It is the causes and consequences of these phenomena that I would like to touchupon in this article.Why do I think about these questions?I will rely not on abstract speculations but on my own experience. I am involved in creation of staticcode analyzers Viva64 and VivaMP to check programs written for 64-bit systems and created with theuse of OpenMP technology. In such programs there are some error patterns which are, however, rathereasy to detect with the help of source program-code analysis. But it is not the main point. The matter isthat while promoting these tools I faced great resistance caused by a wrong idea that 64-bittechnologies and OpenMP are very simple things. The result of this confidence is the necessity toovercome immense resistance of people who consider that there are no problems and that there cannotbe any. So to say, I turned out to be the enemy of their peace. Of course, it is the same with othertechnologies but in this article I will speak upon OpenMP technologies and 64-bit programs as the mostfamiliar to me.The belief that everything is simple is not unusual. You are constantly being conceived that you caneasily make your program a 64-bit one and get the increase of performance and immense memory sizeat once. It is described how you can easily turn your code into a parallel one by using OpenMP. We aresurrounded by different articles containing theses of "it is enough just to recompile an application", "bya simple arrangement of OpenMP directions" kind. And I am walking around at this wonderful feast withposters with large titles of articles "20 issues of porting C++ code on the 64-bit platform", "32 OpenMPtraps for C++ developers". And I feel just like a mean little old man who is dissatisfied with and is
constantly grumbling at everything. It is this strange feeling and resistance to "simplicity adherents" thatmade me ponder.A thought appeared that perhaps it is me who is wrong. Together with others I must say that everythingis simple. And that with our tools it is even simpler. As simple as it just can be. But something is wrong.There is a contradiction. I am familiar with the actual state of things and I am sure that certaindifficulties exist that await programmers while mastering parallel technologies and 64-bit systems withlarge memory size. There is no simplicity. It is deceit. And deceit is harmful. It is harmful to me as aperson who is interested in promoting his tools. It is harmful to a manager who cannot forecast theterms correctly. And of course it is harmful to a programmer who will experience disappointment in newtechnologies and will face overtime work. Thats why I decided to adhere to my opinion and I will try tochange your viewpoint too.Where does the difficulty come from?One can still try to convince me that creation of 64-bit programs is not so difficult, that some programscan be recompiled without corrections for 64-bit systems and that errors can be automatically foundand corrected when different methodologies are used. Although it would be not so easy because Ipossess experience proving that these processes are complicated and that one can easily make a greatmistake forecasting the terms.But when parallel programming is viewed in the same way it is sabotage. It is a complex and too difficulttask which cannot be solved by a simple arrangement of OpenMP directions around loops and use ofsome library. But it is considered to be so. A lot of developers who havent studied the questions ofcreating parallel algorithms yet are convinced that one shouldnt think about it and the problem will besolved by OpenMP or any other technology. It seems easy and clear to parallelize any programaccording to these articles. But popularization of what the reader cannot do and check himself createsonly an illusion of understanding. Perhaps it enlarges his vocabulary and mind but it also brings thisuseless illusion of understanding where there is actually no understanding and knowledge.This doesnt mean that OpenMP or any other technology is bad. They are all perfect. But you shouldunderstand their possibilities and impossibilities. Dont hope to parallelize your existing programs easily.Most likely, you will have to create new algorithms and convert data structures and mechanismsworking with them. If you want to get sure try yourself to parallelize the algorithm of array sorting whicheverybody wrote at school and university quickly. If you fail, look at Batchers parallel sorting. Dont findit simple? Of course, it is not. This example is given to show what kind of difficulties you are to facewhen dealing even with common algorithms used in your programs. But it is not enough. Perhaps youuse iterators in loops in your C++ programs, dont you? But do you know that when giving examples offor parallelizing popular articles concerning OpenMP do not mention that iterators cannot be used in it?Indexes should be simple data types (for example, int). That is when you calculate time forreconstructing algorithms keep in mind that you may have to change data structures and methods ofworking with them as well. And this all demands additional time and effort.If everything is difficult why do they say that it is simple?There are no bad people who want to deceive you and convince you that everything is simple. But thereare different interests leading to mutual deceit. Those who promote 64-bit hardware platforms andoperating systems are surely interested in convincing you that it is very simple to port your applications
on them. It is understandable. Porting on 64-bit systems is already delayed and if one says that there aredifficulties it can prolong it for several years.Developers of parallel systems advertise different program technologies and libraries saying thateverything will become parallel nearly by itself. Those who create these libraries/technologies try toconvince programmers that their library/technology is the best because it is the simplest. And a sadconfusion occurs. The fact that a library is simple and convenient (for example, in my opinion, OpenMPis simple and convenient) doesnt necessarily mean that creation of parallel programs is simple too!Programmers who have read a lot of articles telling them that everything is wonderful in 64-bit andparallel worlds but who lack enough practice deceive their managers and themselves when it comes toterms. Managers deceive clients. However managers themselves prefer to consider everything simple asit seems to them that it will help to save on developers. As the result there are no guilty but everybodydeceives each other.ConclusionsWhile writing this text I came across a wonderful article by Dijkstra "Two views of programming" whichis very close to this article in sense. I would like to quote a short passage which sums up everything saidbefore in a very good way: "It is not so much the computer manufacturers, that want to do as if they sell an easy product; it is not somuch the managers of software projects, that would like to view the programming activity as a simpleand predictable one; it is not so much our educational institutes, that would like to train their studentswith predictable success.It is the comfortable illusion of Man as elaborate automata that, like a drug, seems to have freed itsvictims from the burden of responsibility. Accepting programming as a hard intellectual challenge wouldplace the full weight of that burden back upon their shoulders.".But theweight of this burden mustnt become the reason to go on keeping our eyes shut. It is harmful toeveryone. We must accept the problem as it is.The conclusion of this all must be a more critical view of the possibilities of new technologies. Newmeans and methods are useful but they have never been able - and perhaps never will be - to replacethe very process of software development which consists of many steps and items. First of all, it isnecessary to study and improve the development process and only after that choose new technologiesfor implementing a project.Wonderful books by Steve McConnell "How much a program project costs" and "Code Complete" areindispensable to breaking the circular deceit of simplicity. Besides, one should pay attention to a cultbook by Frederick P. Brooks "The Mythical Man-Month: Essays on Software Engineering".I wish you soon recovery, success in forecasting difficulty of program projects and their completion intime!References 1. Andrey Karpov, Evgeniy Ryzhkov. 20 issues of porting C++ code on the 64-bit platform. http://www.viva64.com/art-1-2-599168895.html
2. Alexey Kolosov, Evgeniy Ryzhkov, Andrey Karpov. 32 OpenMP traps for C++ developers. http://www.viva64.com/art-3-2-1023467288.html3. Edsger Dijkstra. Two views of programming. http://www.viva64.com/go.php?url=1454. Steve McConnell. Software Estimation: Demystifying the Black Art. Redmond, Wa.: Microsoft Press, 352 pages, 2006, ISBN: 0735605351..5. Steve McConnell. Code Complete, 2nd Edition. Redmond, Wa.: Microsoft Press, 2004. 960 pages, ISBN: 0735619670.6. Frederick P. Brooks. The Mythical Man-Month: Essays on Software Engineering. Addison- Wesley, 1995, 322 pages (ISBN 0201835959).