Introduction into the problems of developing parallel programs


Published on

As developing parallel software is rather a difficult task at present, the questions of theoretical training of specialists and investigation of methodology of projecting such systems become very urgent. Within the framework of this article we provide historical and technical information preparing a programmer for gaining knowledge in the sphere of developing parallel computer systems.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Introduction into the problems of developing parallel programs

  1. 1. Introduction into the problems ofdeveloping parallel programsAuthor: Andrey KarpovDate: 21.01.2008AbstractAs developing parallel software is rather a difficult task at present, the questions of theoretical trainingof specialists and investigation of methodology of projecting such systems become very urgent. Withinthe framework of this article we provide historical and technical information preparing a programmerfor gaining knowledge in the sphere of developing parallel computer systems.For readerThis document is part of a series of articles devoted to issues of creating quality and effective programsolutions for modern 64-bit multi-core systems. You can read other articles on the site is very difficult for programmers who only begin to use multi-processor computers to master all thepeculiarities of their usage while developing programs for applied tasks. As practice shows difficultiesbegin when effectiveness and mobility are required of parallel software being developed. It is explainedby that universal means simplifying a programmers labor and providing full access to debugginginformation are only being developed. The problem is that there are no standards in the sphere ofcreating and debugging programs for parallel systems because the field of computer science is veryyoung. Correspondingly, there are no logically complete training courses on parallel programming forbeginners at present.Development of multi-processor computers is inseparably linked with development of parallelprogramming technologies, both universal and for concrete computer architectures. By a programmingtechnology, that is by organization of work with memory, we mean usage of means of controlling aconcrete computer.It should be noted that while developing software (both controlling means and means for solvingapplied tasks) for super-computers special attention should be paid to programming technique, i.e.building of a logical program architecture. By this we mean development and addition of parallelingalgorithms increasing effectiveness of their execution on multi-processor computers.1. History of development of multi-processor complexes and parallelcomputations50 years have passed since appearance of the first computing machines - computers. During this timethe sphere of their usage has covered almost every field of human activity. Nowadays, it is impossible to
  2. 2. imagine effective work without using computers in such spheres as production scheduling and control,projecting and developing complex devices, publishing, education, in other words, in all the fields whereprocessing of large sizes of information is needed. Such tasks appeared in the middle of the previouscentury due to development of atomic energetics, aircraft building, rocket-cosmic technologies andsome other science and technique fields [1].Nowadays, the field of tasks demanding powerful computing resources for their solution has extendedeven more. This relates to fundamental changes in the very organization of scientific investigations.Because of wide introduction of computers, computational modeling and numerical experiment havedeveloped greatly [2]. Filling the gap between physical experiments and analytical approaches,computational modeling allowed us to investigate phenomena which are either too complicated to beinvestigated through analytical approaches or too expensive or dangerous to be investigatedexperimentally. Meanwhile, numerical experiment allowed us to make the process of scientific andtechnical search much cheaper. It became possible to model in real time the processes of intensivephysico-chemical and nuclear reactions, global atmospheric processes, processes of economical andindustrial development of regions etc. It is obvious that solution of such great tasks requires greatcomputational resources.Usage of computers for computational purposes has always remained the main force of the progress incomputer technologies. Thats why it is no wonder that as a main characteristic of computers we usesuch an index as performance, i.e. the value showing what quantity of arithmetic operations it canperform in a time unit. It is this index which shows the scale of progress achieved in computertechnologies. Thus, for example, performance of one of the first computers EDSAC was only about 100operations per second, while peak performance of Earth Simulator, one of the most powerful super-computers nowadays, is 40 trillion operations per second. Thus, performance has increased a 400 billiontimes! There is no other sphere of human activity where progress is so evident and so great. Of course,anyone would immediately ask: why did it become possible? Strangely enough, the answer is rathersimple: because of 1000-time increase of electronic circuits performance and maximum extension ofparalleling of data processing.The idea of parallel data processing as a powerful source for increasing performance of computers wasexpressed by Charles Babbage about a hundred years before the first electronic computer appeared. Butthe level of technological development in the middle of the 19th century didnt allow him to fulfill thisidea. With the appearance of the first electronic computers these ideas became more than once thestarting point when developing the most advanced and high-performance computer systems [3].Without exaggeration we can say that the whole history of development of high-performance computersystems is the history of fulfilling the ideas of parallel processing at a certain stage of development ofcomputer technologies, naturally, combined with increase of speed and safety of electronic circuits.Brand new decisions in increasing performance of computer systems were introduction of pipelineorganization of command execution, inclusion of vector operations into the command system allowingyou to process whole data arrays by one command; distribution of calculations among many processors.Combination of these 3 mechanisms in the architecture of the super-computer Earth Simulatorconsisting of 5120 vector-pipeline processors allowed it to gain record performance, which excessesperformance of modern personal computers by 20000 times.It is obviously that such systems are too expensive and are produced in single copies [4]. And whatabout commercial production nowadays? The wide variety of computers produced in the world today
  3. 3. can be roughly divided into four classes: Personal Computers (PC); Workstations (WS); Supercomputers(SC); cluster systems [5].This division is very approximate because of rapid progress in the sphere of development ofmicroelectronic technologies. Performance of computers of every class doubles nearly every 18 monthsat present (in accordance to the so called Moores Law). Because of this the supercomputers of thebeginning of the 90-s often yield to modern workstations in performance, and personal computersbecome successful rivals to workstations. However, lets try to classify them somehow.Personal computers. Strange enough, in this case we mean single-processor systems on Intel or AMDplatforms controlled by single-user OS (Microsoft Windows and others). They are used mostly as apersonal work place.Workstations. Most often these are computers with RISC-processors with multi-user OS relating to UNIXOS family. They contain from one to four processors, support remote control [6] and can maintain needsof a small group of users.Supercomputers. Their distinctive feature is that they are usually large and, consequently, veryexpensive multi-processor systems. In most cases supercomputers use the same commercial processorsas workstations. Thats why the difference between them is often rather in quantity than in quality. Forexample, we can speak of a 4-processor workstation by SUN company and a 64-processorsupercomputer by the same company. Most likely, the both use the same microprocessors.Cluster systems. In recent years they have been used in the whole world as a cheap alternative tosupercomputers. A system of the required performance is assembled from ready-made commercialcomputers united in their turn by some commercial DCE. Thus, multi-processor systems which havebeen early associated with supercomputers mostly, nowadays become popular in the whole range ofproduced computer systems, from personal computers to supercomputers on the basis of vector-pipeline processors. On the one hand, this circumstance increases availability of supercomputertechnologies and, on the other hand, makes mastering them urgent as you need to use specialprogramming technologies for all the types of multi-processor systems in order to allow a program tofully use the resources of a high-performance computer system [7, 8]. Usually this is implemented bydividing a program with the help of some tool into parallel branches each of which is executed on aseparate processor.2. Using multi-processor systemsSupercomputers are developed first of all to solve complex tasks demanding large quantity ofcalculations. Meanwhile, this implies that a single program can be created requiring all thesupercomputers resources for its execution. But creating such a program can be impossible orunreasonable. In fact, when you develop a parallel program for a multi-processor system, it is notenough to divide it into parallel branches. For effective usage of the resources you need to providebalanced load of all the processors what in its turn means that all the program branches should executeapproximately the same quantity of computational work. But sometimes it is impossible. For example,when solving some parametric task for different parameters values the time of searching for solutioncan vary greatly. It seems more reasonable in such cases to perform calculations for each parameterwith the help of a simple single-processor program [9]. But even in this simple case we may needresources of a supercomputer because execution of full computational work on a single-processorsystem may require too much time. Parallel execution of many programs for different parameters
  4. 4. values allows us to significantly speed up solving the task. And finally we should mention that usingsupercomputers is always more effective for maintaining needs of a large group of users than using thecorresponding number of single-processor workstations as it is easier in this case to provide balancedand more effective load of computational resources with the help of the task managing system.Unlike common multi-user systems, OS of supercomputers, as a rule, in order to get the maximum rateof program execution dont permit to share resources of one processor between different,simultaneously executed programs. Thats why there can be the following modes of using an n-processor system as two opposite variants: • all the resources are allocated for execution of one program and in this case we expect an n-fold speed-up of program execution in comparison to a single-processor system; • n common single-processor programs are executed simultaneously and the user expects that other programs wont influence the speed of execution of his program.3. Parallelism in computational modeling tasks3.1. Static and dynamic balancingWhen solving various tasks of mathematical physics on multi-processor systems with the help of meshmethods [10] two approaches to building parallel programs are widely used. The first approach is calledgeometrical parallelism method, and the second one - group decision method [11]. Ideas on which thesemethods are based are simple and smart. It wont be exaggeration to say that most tasks of gasdynamics, microelectronics, ecology and many others, which are now solved by using the finitedifference method or finite element method, are solved effectively by the geometrical parallelismmethod. The group decision method is reasonable to use when building parallel algorithms of solvingtasks by Monte-Carlo methods, when a series of single-type calculations is performed and in some othercases.We should note that the geometrical parallelism method is a method of static load balancing whichdefines a section of the mesh executed by each processor beforehand. Static balancing is effective whenpriori information is enough for preliminary distribution of the common computing load equally amongprocessor nodes. The group decision method is a method of dynamic balancing load. When using thismethod it is not known beforehand what particular mesh nodes will be processed by this or thatprocessor. The processors receive tasks dynamically as they have executed the already received, whatprovides balanced load of processor nodes when there are many independent tasks.3.2. Parallelism of "group decision" typeParallelism of "group decision" type is convenient for performing calculations dividing into more single-type tasks each of which is solved independently from the others. No data transfer occurs between suchtasks and, consequently, there is no need of their mutual synchronization.Lets consider an example of a computational mesh as a set of independent nodes in each of which weshould define some parameters on each temporal layer by solving a system of ODE with thecorresponding initial data [12]. Solution of the system in each node depends only on local values of thevariables in this node. Meanwhile computational load differs very much in different nodes. Whenbuilding a parallel program with the help of the classical "group decision" method the following strategyof computational load distribution is used.
  5. 5. One control processor is defined while all the other processors are used as processing nodes, i.e.computing nodes. Each computing processor performs primary tasks - solution of ODE system for thenext mesh point with the corresponding local parameters. The control processor distributes the primarytasks among the computing processors and collects the results.In the beginning of the next step each processor waits for a new data chunk, processes it, returns theresult and starts waiting for the next task until instead of the next task it gets a message that all themesh points are processed.As there is no need to synchronize primary tasks, different processors can get different number ofcomputational nodes as the data processing is finished. Thereby the problem of balanced load ofprocessors is solved even if the time for solving the equation system for different mesh points orprocessors performance vary greatly.In case of heterogeneous computational load when computing different points of the spatial mesh,usage of the "group decision" method potentially allows you to significantly reduce downtime andincrease effectiveness of paralleling in comparison to the geometrical parallelism method consideredfurther. The advantages of this method can be fully implemented if the data for processing are from thebeginning concentrated on one of the processors which in this case can serve as the control processor.When the source data are initially distributed among the processors at random, preliminary collection ofthe data corresponding to all the computational points on one of the processors is required to use thismethod. The necessity of the preliminary data copying from all the processors to one and the followingreturn of the results from this processor to the processors-"holders" of the points significantly reduceseffectiveness of this method and makes it of little use for solving most tasks of computational modeling.3.3. Geometrical parallelism.The source task can be split into a group of fields independent from each other at each computationalstep and crossing only at the division boundary. That is, we compute (n+1) temporal layer in each fieldand after that coordinate the boundaries and pass on to computing the next layer.But using this approach we have problems with recalculation of values at the boundaries between thesefields when we divide the computational field into non-crossing subfields, thats why we offer the nextlogical step - to divide the source field into mutually crossing subfields.There will appear two "dummy" points to the left of the first field and to the right of the last field. Thus,we get four processes independent from each other at each step. To pass on to the next iteration weneed to coordinate the boundaries as the first field should give to the second one its left boundary forthe next step, and in its turn the second field should give to the first one its right boundary and so on.This method can be generalized into most computational methods based on equations for modelingphysical processes.4. Effectiveness of a parallel program4.1. Notion of an effective parallel programUsing supercomputers imposes certain requirements on the new developed software providing safe andeconomical implementation of the algorithm when solving applied tasks. Effectiveness of usingsupercomputers becomes apparent when creating complex research complexes and expert systems.
  6. 6. It is much more difficult to write a parallel program than to write a sequential one. Creation of softwarefor parallel computers is the central problem of supercomputer calculations [13].Partially the problem of choosing the optimal number of parallel branches in correspondence with thecriterion of minimum total time costs can be solved with the help of automations of parallel programgeneration. A particular case of solving this problem for the computer systems with MIMD architectureis considered in the article by V.A. Kostenko "To the question of evaluating the optimal parallelism level"[14].Effectiveness of using multi-processor computer systems is to a large degree determined by the qualityof the applied parallel programs. A program is considered effective when all the processors defined forprocesses are loaded during its execution. But practically it is impossible.4.2. Properties of an ideal parallel programLets note that an ideal parallel program possesses the following properties: 1. Lengths of simultaneously executed branches are equal. 2. Downtimes relating to data waiting, control transfer and conflicts occurring when using common resources, are fully excluded. 3. Data transfer is fully combined with calculations.Increase of parallelisms effectiveness (decrease of time costs on the overhead costs) is reached by thefollowing means: • enlargement of paralleling units; • decrease of complexity of the algorithms of generating parallel procedures (subprograms); • preliminary preparation of the package of different source data variants; • paralleling of the algorithms of generating parallel procedures (subprograms).4.3. Adaptation of programs to the parallel computers architectureThe main stages of the process of adapting programs to the architecture of parallel computers anddescription of the tasks occurring at each of these stages are given in the article by A.S. Antonov"Effective adaptation of sequential programs to the modern vector-pipeline and array-parallelsupercomputers" [15]. We would like to pay special attention to some of the tasks which the authors ofthis analysis faced. Among these tasks are: • investigation of the common program structure; • definition of the main computational core, input-output localization; • definition of the potential parallelism of a fragment; • definition of the sequential fragments of calculations and attempt to use alternative algorithms for such fragments; • definition and minimization of data redistribution points; • conversion of the traditional loop for parallel algorithms; • minimization of the number and size of temporary arrays for optimizing cache-memory handling; • passing on from the source program working with full arrays to the program processing only a local chunk distributed for a processor: change of arrays sizes and the corresponding transformation of the program text.
  7. 7. We should note that solution of these tasks allows us to perform an effective port of a sequentialprogram on a parallel architecture.The process of developing a parallel program is very long and laborious despite that, as a rule, therealready exists an implementation of its "sequential" counterpart. A program is usually developed on acomputer with a certain architecture and its practical application is performed on another computer,more powerful and with the typology different from that of the former machine. This approach allowsyou to economize computer time on more powerful supercomputers which are much fewer thancheaper ones.When porting a parallel program on computers with a different architecture a programmer faces theproblem of invalidity of once developed parallel procedures.At present there are no universal means of adapting programs to a concrete architecture ofsupercomputers and thats why this problem has to be mostly solved manually what makes the processvery labor-intensive [15]. To save labor of a programmer RAS mathematical institutions are developinglibraries of effective procedures and algorithms for concrete architectures of supercomputers (RAS UralDepartment, Research-and-development computer center of MSU named after M.V. Lomonosov). Usingthese libraries can partially save labor of an applied programmer not only at the stage of modifying aprogram for more powerful supercomputers, but at the stage of the primary development of a parallelprogram.5. Debugging and monitoring issuesThe problem of debugging and monitoring is very urgent as there are no managers that could provide anapplied software developer by intermediate information especially urgent at the initial stage ofdesigning [16]. In the general case the task of debugging and monitoring such systems is put in thefollowing way [17, 18]. There is a mesh of nodes heterogeneous in their hardware and/or softwareplatforms, on each of which many processes (threads) are executed simultaneously [19]. There is also atotal number of users each of which would like to control and/or operate his subset of program and/orhardware components.Understanding of debugging/monitoring as controlled execution changes the position of debugging inthe systems life cycle, allows you to use architectural and protocol solutions characteristic of controllingmeans. This makes the controlling means scalable, capable of maintaining the distributedheterogeneous systems.It is important for further development of debugging/monitoring means to create a set of specificationsdefining functionality of the manager programs being developed [20].Programs are complex dynamic systems, especially parallel and interactive (operating in dialog mode)ones which include complex interactions between program processes themselves and between theprocesses and the outer world. Analysis of such programs cannot be performed in terms of relationsbetween input and output values of the program as it is usually performed for sequential programs. Thisshows that checking and proving correct work of such programs demands developing adequate meansof formal specification. In particular, it is necessary to be able to express relations between the systemsstates at those instants of time when some events occur accompanying the program systems operation.The article "Applying temporal logic to program specification" by M.K. Vasilyev [21] discusses theapproach to analysis of a parallel program based on applying mathematical logic.
  8. 8. Process control is one of the most important tasks of OS. To perform this function on supercomputerssemaphore technology [22] can be used which consists in locking and unlocking of processes.Semaphores have been traditionally used for synchronizing processes addressing shared data. Eachprocess should exclude for all the other processes the possibility of simultaneous address to its data.When solving applied tasks the size of the received information in most cases is so large that thepossibility of verification - the detailed analysis of the data received directly by a computational program- is impossible. As there are no universal graphical packages with visualization of different isometricprojections and color gammas for such situations, applied software developers are advised to startdeveloping such packages.6. Paralleling objects modelingDeveloped computer-usage approaches are based on the thesis: computer is a cognition tool with thehelp of which people get new information about an object or phenomenon being investigated [23].Consequently, a qualified user should know the modern cognition methodology, i.e. modeling. Modelingis not only designing of a cognition object, but a cognition method as well. Modeling is workmethodology whose effectiveness becomes apparent only when specialists are highly qualified andknow well the modern formalization means - logic and mathematics.Having defined the problem and stated the goal a researcher starts searching for a solution. The way hepasses becomes a method.The process of modeling presupposes both the way "from the object to the model" (reflection of realityin a paradigm) and the way "from the model to the object" (test of the models truth on its possibilities).Computer is the natural means of performing such "research" cycles.Software-development theory specialists rarely pay attention to modeling when describing the processof software creation. On the other hand, modeling specialists prove urgent necessity of wide usage oftheir methods when projecting any complex system [24]. As a software complex is a complex systemwith many levels and components and a complex structure of relations between them, it is necessary touse modeling when developing such systems.Taking into consideration that parallel software development (development of a paralleling object) isvery difficult nowadays, the problem of creating theoretical basis of its projecting is even more urgent.Besides analysis of the structure and properties of the developed programs on all the projecting stages,modeling can help describe all the peculiarities of interaction between parallel processes at the level ofa simulation model. In his work "Modeling of parallel software using PS-networks" [25] N.G. Markovsuggests using the graph-analytical approach to simulation modeling of a program project on the basisof the demands put before parallel software. The aim of this work was to work out the demands to theparallel software simulation modeling mechanism and also to create a mechanism keeping balancebetween mathematical simplicity and rigor on the one hand and practical applicability on the otherhand.Thus, we can state that the most convenient means of analyzing computational algorithms of parallelcomputations is graphs [26].The problem of creating modern packages of applied programs intended for a wide range of mechanicstasks goes out of limits of synthesizing these tasks from separate program modules. It is related to the
  9. 9. global optimization of the whole computational sequence of tasks [27]. Thats why a package ofprograms as a product used for scientific-applied purposes not only by its creators but by end users aswell, should be developed at an absolutely new programming level. When developing modern softwareit is necessary not only to take into consideration non-linear (with feedback) relations between all thelinks of a calculation chain but also implement the possibility of segmenting a program at high, middleand low paralleling levels of the computational process. Segmentation is necessary for more effectiveusage of multi-processor systems. Besides, when developing a numerical algorithm we shouldcoordinate the issues of accuracy and safety of the end software and also the issues of its effectivenessand portability on a concrete supercomputers architecture. Such parallel programming differs greatlyfrom the traditional programming, i.e. sequential programming.6.1. Levels of decomposition of paralleling objectsTo provide supercomputers users with possibilities of simultaneous performance of many scientificcalculations or multi-thread processing of requests in a database on multi-processor computers, thecorresponding software should be installed. In this case paralleling functions are performed not only byapplied software but by the OS as well.In the general case, two main interrelated problems occur when creating OS of parallel data processing:the first one is minimizing of the time of performing the given calculations size, and the second issynchronization of many simultaneously interacting parallel process [28]. To solve each of themdifferent approaches are being developed. In the mentioned work it is offered to take into considerationthat when implementing complex synchronization mechanisms overhead costs increase and thisinfluence badly the efficiency of solving tasks. The stated problem in the systems with parallelismlimited by the number of processors is solved by minimizing the total time of performing the givencalculations size.The results of implementing this approach relate, first of all, to "operational parallelism". The methodbased on building the schedule of launching and finishing each of the competing processes can be usefulin such systems. It gives you an opportunity not only to more effectively solve the processsynchronization problem but significantly reduce system costs and wasteful downtimes of theprocessors. The method of managing interaction between parallel processes is implemented with thehelp of "semaphore" technology [29].When researchers create applied software, the practical value of numerical methods they develop isdetermined not only by the results received with their help when investigating complex phenomena butby their applicability on concrete supercomputers as well. It was found that as performance of personalcomputers grows stimulating development of computational methods, there also occur qualitativechanges in supercomputers architecture focused on development of parallelism and specialization ofprocessors. And this, in its turn, stimulates search for new representations of physical phenomena thatwould permit more direct presentation on the computers architecture. Thus, for example, the cellular-automat approach appeared in gas- and hydrodynamics [30]. The article shows a new model of parallelcalculations - cellular-neural network (CNN). The article describes the essence of a cellular-automatmodel and also shows rich opportunities of CNN for representing spatio-temporal dynamics of activemediums. This model can serve as the basis for creating parallel programs intended for solvingdifferential equations in partial derivatives and also for imitation of nonlinear dynamics phenomena. It isnoted that usage of CNN calculation methods together with parallel processors will allow us to greatlyincrease the quality of solution of such tasks.
  10. 10. The aim of any work connected with parallel programming is review of interrelations between themathematical algorithms structure and a multi-processor computational systems architecture.Depending on the complexity of the stated task different types of interrelations can be implemented.These interrelations are called the levels of decomposition of the source task. They can be defined asfollows [31]:The first level - division of a task into subtasks.The second level - division of each separate subtask into a subset of quasi-single-type proceduresexecuted simultaneously at different source data. In mathematical physics this parallelism type is calledgeometrical parallelism or data parallelism as paralleling is performed in this case by distributingcalculations in different points of the computational field into different processors.The third level - paralleling of separate procedures.The fourth and the deepest paralleling level - division of arithmetical processes according to the numberof processors.It is recommended not to use the last level on supercomputers with distributed memory in which foreach processor local memory is allocated. The researchers of most applied tasks are advised to stop theprocess of their decomposition at the second level.6.2. Possibility of paralleling objects in computational modeling algorithmsAnd now lets consider what objects in the algorithms of task solution can be paralleled.The main numerical methods (the finite-element method, the finite-difference method and others)bring the source task to forming a system of linear algebraic equations (SLAE) and its further solution[32, 33]. For example, in a sequential program implementing the finite-element method, most time isspent on forming the SLAE itself (calculation of coefficients) but not on its solution. It is also importantto mention that the elements of SLAE matrix depend only on their locations in it and do not depend oneach other. In this case parallel algorithms of SLAE formation can be used effectively. And here youshould perform the following operations: 1. split the computational task into parallel branches; 2. perform calculations in these branches; 3. form and solve SLAE (by any method).The article [34] gives an example of description of a parallel algorithm of SLAE formation and alsopeculiarities of using MPI technology.The article [35] considers implementation of Gauss-method for solution of sparse systems of linearalgebraic equations on computers with parallel processes and shared memory. It is pointed out thatdivision into several command threads can be performed either according to the functional feature ordirectly by data. When the task is stated like this, only data-relating division can be implemented.Meanwhile, you should pay attention whether it is possible to single out unlinked fields from the task.The same article points that the offered parallel algorithm is bound to a concrete computer architecturebut it also states that effectiveness of the paralleling algorithm depends only on the correlation betweenthe number of processes and processors and also on the size of information processed at one loop.
  11. 11. We can propose a thesis that loops are one of the most important program constructions withaccessible parallelism. The problem of extracting fine-grained parallelism (parallelism inside loops) fromthese constructions is of great importance in view of increasing popularity of superscalar computers[36].The article [37] presents algorithms for computational procedures and also results received with theirhelp and based on the high-accuracy parallel arithmetic methodology. It is suggested that thismethodology be used for solving applied tasks of linear algebra and mathematical physics. Thementioned work is devoted to creating algorithmic and program means of supporting accurate arraycomputations based on complex usage of parallelism of MIMD-systems [38] and multibit arithmetic withdynamic operand length. Special attention is paid to influence of roundings in basic array operations onthe accuracy of matrix tasks calculation. The work includes the library of programs and text examplesdemonstrating effectiveness of the developed approach. The given results show the possibility ofperforming accurate array computations with simultaneous message transfer on parallel computers.The developed package of applied programs can be adapted for execution on parallel computers ofdifferent types.It is obvious that using multibit arithmetic is not typical of supercomputers. Its usage will inevitably leadto slowdown of application execution. But time loss in this case will be compensated not only bycalculations with maximum usage of standard data types but also by adaptation of highly effectiveparallel algorithms initially suited for execution on one-processor computers to means of high-accuracyprocessing. Multibit arithmetic should be used only in the most "heavy" algorithm sections. But even inthis case the dynamic operand length helps process only a limited number of bits. It is this way which issupposed for reaching balance between speed and accuracy of computations.The article [39] analyzes in detail the vector-pipeline architecture of supercomputers of CRAY family. Asthe result of the performed research, programming factors reducing supercomputers performancewere discovered. To them relate: • sectioning of long vector operations (increases overhead costs); • overload of commands buffers (increases overhead costs); • conflicts of memory access (in case of using shared resources); • limited capacity of data transfer channels (depends on the supercomputers architecture); • other factors.It also gives examples (in program codes) showing the way out.In some of the works mentioned above the development is singled out in which the algorithm structuredoesnt adapt to the computers structure but defines its structure itself [40]. The work is intended forcreating new modern computer technologies and methods of parallel programming meant forincreasing effectiveness of solving fundamental scientific and applied problems in the sphere ofcomputational modeling of aerodynamics and gas-dynamics tasks. Special attention is paid totheoretical issues of paralleling. The work considers different methods of decomposing a full task intosimultaneously executed subtasks. Of high strategic importance is the theoretical stage of investigatingthe problem of paralleling a program complex, that is development of principles (and concrete methodson its basis) of optimal decomposition of the whole totality of algorithms, composing the processorsystem and its operational environment.
  12. 12. The article describes three main decomposition (segmentation) types for the program complex "Thread-3" planned for development in the process of preparation for paralleling the algorithms which make itup: • physico-mathemetical; • geometrical; • technological.One of the global types of high-level structuring for the task being solved is decomposition of theinvestigated physical process into subprocesses composing it and, consequently, segmentation of thecommon algorithm of solving the full task into several algorithms of solving the subtasks composing it.A segmental algorithm of parallel calculations of physical processes is suggested and meanwhile all themodule-segments of the computational core of the program are launched simultaneously. Besides,inside each segment subsegments can simultaneously start working.When paralleling computational procedures of extreme importance is synchronization and routing ofdata improper organization of which either leads to incorrect calculation or to large overhead costs ofcomputer and astronomical time due to various delays of calculations because of data waiting and,consequently, downtime of the processors in some segments. It is supposed that the latter leads tononoptimality or even impossibility to use the configurable processor space.Geometrical decomposition (segmentation) of the full task and the following parallelization ofcalculations allows you to significantly reduce astronomical time needed for calculations. Geometricaldecomposition consists in dividing the whole integration domain into a map of subdomains(subsegments) and also in a single-step calculation of the physical process state in each subdomainfollowed by joining of solutions. The article lists requirements to mathematical definition of the taskpermitting geometrical decomposition.Technological decomposition implies segmentation of mathematically independent tasks. There can beseveral levels of technological decomposition. The most typical example of decomposition is parallelinga program into certain physico-mathemetical tasks each of which can similarly consist of algorithmicallyindependent tasks. The process of technological decomposition depends greatly on the programsstructure and numerical methods used in it.Using the latter decomposition type presupposes special attention to the parallel programseffectiveness.ConclusionDespite obvious success in using multi-processor systems there appear debates about their loweffectiveness. Increase of multi-processor systems performance is generally determined by balancebetween computational operation and data exchanges on its background. Non-fulfillment of thiscondition is one of the causes of performance loss during paralleling with increasing number ofcomputational program modules.Evaluation of programs effectiveness has been carried out since first multi-processor systems -transputers. Even then the first attempts were made to successfully solve the problem of maximumusage of calculation time. When solving a concrete task, first of all it is necessary to search forparallelism variants by dividing a separate task into several subtasks. After that, data parallelism (or
  13. 13. geometrical parallelism) can be performed, that is division of computational field. This type ofparallelism means that the computational field is divided into subfields each of which is correlated to aseparate systems processor.When developing real parallel programs, as a rule, high effectiveness demands many changes of theprogram to find the best scheme of its paralleling. Success of this search is determined by simplicity ofthe programs modification.References 1. V.N. Datsuk, A.A. Bukatov, A.I. Zhegulo. Electronic users guide on the course "Multi-processor systems and parallel programming" Part I. Introduction into programming organization and methods of multi-processor computer systems. Rostov-on-Don, 2000. 2. E.V. Neupokoev, G.A. Tarnavskiy, V.A. Vshivkov. Paralleling marching algorithms: target computational experiments. // Autometriccs, N 4, volume 38, 2002, pp. 74-87. 3. V.V. Korneev. Parallel computer systems. Moscow: "Knowledge", 1999. - 320 pp. 4. G.I. Shpakovskiy. Parallel computers architecture. - Minsk, 1989. - 136 pp. 5. A.O. Latsis. How to build and use a supercomputer. Moscow: Bestseller Publishing house, 2003. 274 pp. 6. D.U. Labutin. System of remote access to the computational cluster (access manager): high- performance parallel computations on cluster systems. Material of the second international scientific-practical seminar, Nizhny Novgorod: Nizhny Novgorod University Publishing house, 2002. pp.184-187. 7. K.E. Afanasyev. Multi-processor computer systems and parallel programming: tutorial/ K.E. Afanasyev, S.V. Stukolov, A.V. Demidov, V.V. Malishenko; Kemerovo State University. - Kemerovo: Kuzbassvuzizdat, 2003. - 182 pp. 8. S.A. Nemnyugin, O.L. Stesik. Parallel programming for multi-processor computer systems. - St. Petersburg: BHV-Petersburg, 2002. - 400 pp. 9. A.V. Demidov, K.V. Sidelnikov. Emulation of parallel data processing on a personal computer // XLI International scientific student conference "Students and Scientific-and-Technological Advance". Collection of works. Novosibirsk, 2003. pp. 110-111. 10. A.A. Samarskiy, E.S. Nikolaev. Methods of solving mesh equations. Moscow: Science, 1978. 561 pp. 11. V.V. Samofalov, A.V. Konovalov, S.V. Sharf. Dynamics and statics: searching for compromise // Works of All-Russian scientific conference "High-performance computations and their applications". Moscow, 2000. pp. 165-167. 12. S.K. Godunov, V.S. Ryabenkiy. Difference schemes. - Moscow: Science, 1973. - 400 pp. 13. V.V. Voevodin. Supercomputers: yesterday, today, tomorrow. // Collection of popular science articles "Russian science at the dawn of the new century". Under the editorship of academician V.P. Skulachov. Moscow: Scientific world, 2001. pp. 475-483. 14. V.A. Kostenko. To the question of evaluating the optimal parallelism level. // Programming. 1995, 4, pp. 24-28. 15. A.S. Antonov, V.V. Voevodin. Effective adaptation of sequential programs to the modern vector- pipeline and array-parallel supercomputers. // Programming. 1996, 4, pp. 37-51. 16. E. Sallivan. Time is money. Creating a team of software developers/Translated from English. - Moscow: Publishing house "Russkaya Redaktsiya", 2002. - 368 pp.: illustrations.
  14. 14. 17. V.A. Galatenko, K.A. Kostuhin. Debugging and monitoring of distributed heterogeneous systems. // Programming, 2002, 1. pp. 27-37.18. V.A. Krukov, R.V. Udovichenko. "Debugging of DVM programs". Programming. - 2001. N. 3.- pp.19-29.19. A.P. Sapozhnikov, T.F. Sapozhnikova. Reengineering technology of distributed computations in the local network. Works of international conference "Distributed computations and Grid- technologies in science and education" (Dubna, June 29 - July 2 2004). 11-2004-205, Dubna, JINR, 2004. pp.183-190.20. V.V. Samofalov, A.V. Konovalov. Technology of debugging programs for computers with mass parallelism // "Issues of atomic science and technique". Series Mathemetical modeling of physical processes. 1996. Issue 4. pp. 52-56.21. M.K. Valiev. Applying temporal logic to program specification. // Programming. 1998, 2, pp. 3-9.22. V.A. Krukov. OS of distributed computer systems. (tutorial).23. N.L. Zaharyeva, V.B. Hoziev, P.D. Shirkov. Modeling and education.// Mathematical modeling, 1999, volume 11, N 5, pp. 101-116.24. A.A. Shalito. Automatic program projecting. Algorithmization and programming of logical control tasks. "Izvestiya akademii nauk. Teoriya i sistemi upravleniya" magazine, issue 6. November- December 2000. pp.63-81.25. N.G. Markov, E.A. Miroshnichenko, A.V. Saraykin. Modeling of parallel software using PS- networks. // Programming. 1995, N 5, pp. 24-32.26. N.M. Ershov. Building graphs of computational algorithms by autotracing method. // Programming. 2000, N 6, pp. 58-64.27. V.P. Gergel, R.G. Strongin. Fundamentals of parallel computations for multi-processor computer systems. Tutorial. Nizhniy Novgorod: Publishing house of Nizhniy Novgorod State University named after N.I. Lobachevskiy, 2000. - 176 pp.28. V.P. Ivannikov, N.R. Kovalevskiy, V.M. Metelskiy. About minimal time of implementing distributed competing processes in synchronous modes. // Programming. 2000, N 5, pp. 44-52.29. M.K. Valiev. Applying temporal logic to program specification. // Programming. 1998, 2, pp. 3-9.30. O.L. Bandman. Cellular-neural models of spatio-temporal dynamics. // Programming. 1999, N 1, pp. 4-17.31. A.E. Duysekulov, T.G. Elizarova. Using multi-processor computer systems for implementing kinetically coordinated difference schemes of gas-dynamics. // Mathematical modeling, 1990, volume. 2, N 7, pp. 139-147.32. O.A. Dmitrieva. Parallel algorithms of numerical solution of simple differential equations. //Mathematical modeling N 5, 2000, pp. 81-86.33. N.R. Bahvalov. Numerical methods // Main editorship of physico-mathematical literature of "Science" publishing house, Moscow, 1975. - 632 pp.34. D.B. Moskvin, V.A. Pavlov. Experience of using MPI technology for solving a system of second- order Fredholm integral equations. //Mathematical modeling, 2000, N 8, pp. 3-8.35. A.N. Zavorin. Parallel solution of linear systems when modeling electric circuits. // Mathematical modeling, 1991, volume 3, N 3, pp. 91-96.36. Ki-Chang Kim. Fine-grained paralleling of incomplete loop nests. // Programming. 1997, N 2, pp. 52-66.37. V.A. Morozov, A.P. Vazhenin. Matrix arithmetic of multiple accuracy for parallel systems with message transfer. // Programming. 1999, N1, pp. 66-77.38. V.V. Voevodin. Mathematical models and methods in parallel processes. - Moscow: Science, 1986. - 296 pp.
  15. 15. 39. V.V. Voevodin. Is it easy to get the promised gigaflop? // Programming. 1995, N 4, pp. 13-23. 40. G.A. Tarnavskiy, R.I. Shpak. Decomposition of methods and paralleling of algorithms of solving aerodynamics and physical gas-dynamics tasks: computer system "Thread-3". // Programming. 2000, N 6, pp. 45-57.Additional references. 1. A.R. Antonov. Parallel programming using MPI technology: tutorial. - Moscow: MSU publishing house, 2004. - 71 pp. 2. R.B. Berozin, V.M. Paskonov. Component system of visualizing the results of computations on multi-processor computer systems // Materials of All-Russian scientific conference "High- performance computations and their applications", 2000, pp. 202-203. 3. N.V. Bocharov. Parallel programming technologies and technique. Review.// "Programming", 2003, N 1. PP. 5-23. UDC 681.3.06 4. V.V. Voevodin, Vl.V. Voevodin. Parallel computations. - St.Petersburg, 2002. - 600 pp. 5. V.P. Gergel, A.N. Svistunov. Development of an integrated environment of high-performance computations on cluster systems. Materials of the second international scientific-practical seminar, Nyzhniy Novgorod: Nyzhniy Novgorod publishing house, 2002. PP.78-82. 6. V.P. Ilyin. About paralleling strategies in mathematical modeling. // Programming. 1999, N 1, pp. 41-46. 7. A.N. Karpov. Data visualization on parallel computer complexes // 15th International conference GRAPHICON-2005. Novosibirsk, Russia, June 20-24 2005. 8. N.A. Konovalov, V.A. Krukov, A.A. Pogrebtsov, U.L. Sazanov. C-DVM - the language fordeveloping mobile parallel programs. // Programming. - 1999. N1. PP. 20-28. 9. N.A. Konovalov, V.A. Krukov, R.N. Mikhailov, L.A. Pogrebtsov. Fortran-DVM - the language for developing mobile parallel programs. // Programming. 1995, N 1. PP. 49-54. 10. R.V. Popova, R.V. Sharf. Organization of saving temporary data on MBC // Theses of the report from All-Russian conference "Urgent problems of applied mathematics and mechanics" (Yekaterinburg, February 3-7 2003), pp. 62. 11. I.V. Prangishvili, R.Ya. Vilenkin, I.L. Medvedev. Parallel computer systems with shared control. - Moscow: Energoatomizdat, 1983. - 312 pp. 12. L.B. Sokolinskiy. Parallel machines of databases. // Collection of popular science articles "Russian science at the dawn of the new century". Under the editorship of V.P. Skulachov. -Moscow: scientific world, 2001. PP. 484-494. 13. E. Tanenbaum. Distributed systems. Principles and paradigms. - St.Petersburg: Piter, 2003. - 877 pp. 14. M.V. Yakobovskiy, R.A. Sukov. Dynamic load balancing // Materials of "High-performance computations and their applications" conference, Chernogolovka, 2000, PP. 34-39. 15. A.V. Komolkin, R.A. Nemnugin. Electronic tutorial "Programming for high-performance computers".About the authorAndrey Karpov, http://www.viva64.comDevelops program solutions in the sphere of resource-intensive applications quality and performanceincrease. One of the developers of Viva64 static analyzer for verifying 64-bit software. Participates indeveloping VivaCore open library for working with C/C++ code.