A Semi-Lagrangian NWP Model For Real-Time And Research Applications Evaluation In Single- And Multi-Processor Environments

Full Terms & Conditions of access and use can be found at
https://www.tandfonline.com/action/journalInformation?journalCode=tato20
Atmosphere-Ocean
ISSN: 0705-5900 (Print) 1480-9214 (Online) Journal homepage: https://www.tandfonline.com/loi/tato20
A Semi-Lagrangian NWP Model for Real-time and
Research Applications: Evaluation in Single- and
Multi-Processor Environments
L. M. Leslie & R. J. Purser
To cite this article: L. M. Leslie & R. J. Purser (1997) A Semi-Lagrangian NWP Model for
Real-time and Research Applications: Evaluation in Single- and Multi-Processor Environments,
Atmosphere-Ocean, 35:sup1, 75-101, DOI: 10.1080/07055900.1997.9687343
To link to this article: https://doi.org/10.1080/07055900.1997.9687343
Published online: 26 Jul 2011.
Submit your article to this journal
Article views: 53
View related articles

A Semi-Lagrangian NWP Mode1for Real-time
and ResearchApplications: Evaluation in
Single- and Multi-Processor Environments
L.M. Leslie
School of Mathematics, The Univers@ of NSW
Sydney, NSW, Australia 2052
and
R.J. F%user
UCAR Visiting Scientist,
National Meteorological Center,
Washington, D. C.
[Original manuscript received 9 December 1994; in revised form 29 August 19951
ABSTRA~ A semi-Lagrangian numen’cal weather prediction (hWP) mode1 developed for
both real-time prediction and for research simulations has been evaluated. The mode1 is
second-order in time, high-order (23) in space, and employs a non-staggered grid in both
the horizontal and vertical.
A version of the mode1 which is third-order in space was compared with two Eulerian
models: the Australian Bureau of Meteorology’s current operational regional mode1 which
is quasi-second-order in rime and space, and also a new version of this mode1 with a third-
order upwind scheme that was developed for use as the Australian Bureau of Meteorology’s
next operational limited-area NWP model,
In a three month tria1 of twice-daily 48-hour forecasts it was found thut both the
third-order models were significantly more skillful than the current operational mode& as
measured by the standard performance statistics such as the Sr ski11 score (Teweles and
Wobus, 1954) and root-mean-square (RMS) errors. The specij?c implications of this greater
accuracy were examined in case studies of severe weather events.
The semi-Lagrangian mode1 also bas been adapted to global fom and run on a daily
basis out to 5 days using archived operational data over a period of almost 6 months.
Finally, the semi-Lagrangian mode1 code was parallelized on a workstation cluster and
also on a scalable parallel computer, ami it was found that the mode1 was well-suited to
parallelization on both computer platfonns.
RÉSUMÉ On a évalué un modele semi-lagrangien de prévision numérique conçu pour la
prévision en temps réel et pour la recherche en simulation. Le modéle est de second ordre
dans le temps et d’ordre élevé (23) dans l’espace, employant une grille non decalee autant
dans l’horizontale que dans la verticale.

76 / L.M. Leslie and R J. Purser
On compare une version du modele de troisième ordre dans l’espace avec deux modèles
euleriens: le modèle regional présentement en exploitation à 1’Australian Bureau of Mete-
orology (ABh4), de quasi-second ordre dans le temps et l’espace; et une nouvelle version
développee comme le prochain modele d’exploitation à regions restreintes de l’ABh4, ayant
un scht?ma de troisieme ordre à pas spatial vers l’amont.
Durant un essai de trois mois de prévisions biquotidiennes de 48 heures, on a trouvé
que, selon les normes de per$ormance statistiques comme l’indice d’habileté St (Teweles et
Wobus, 1954) et les erreurs quadratiques moyennes, les deux modeles de troisième ordre
étaient signijïcativement plus habiles que le modèle présentement en exploitation. Les im-
plications spécifiques de cette plus grande précision ont été examinées dans des études de
cas de temps violents.
Le modèle seini-lagrangien a aussi été adapté à une forme mondiale et intégré chaque
jour jusqu’à cinq jours, en utilisant des données d’archives opérationnelles couvrant une
période de plus de six mois.
Enfin, on a parallélisé le code du modèle semi-lagrangien sur un réseau de stations de
travail et sur un ordinateur parallt?le à architecture évolutive et trouvé qu’il convenait bien
à la parallélisation sur les deux plates-formes d’ordinateur.
1 Introduction
The recently formed Atmospheric Modelling Group in the School of Mathematics
at The University of New South Wales has decided to develop a numerical weather
prediction capability. The modelling effort is intended to provide both a real-time
and research forecasting capacity anywhere on the globe, but particularly over
southeastem Australia (see location map, Fig. la). This project is part of a larger
effort by the autbors to develop a high-order, mass-conserving, semi-Lagrangian
mode1 tbat cari be run over the entire range from a high-resolution short-range
limited-area mode1 through the medium and extended range to general circulation
mode.
The current operational short-range mode1 employed by the Australian Bureau of
Meteorology (BoM) has Eulerian semi-implicit numerics (McGregor et al., 1978;
Leslie et al., 1985). The numerical formulation of the mode1 relies heavily on the
work of Robert and co-workers (Robert et al., 1972) and had remained largely
unchanged for over 15 years until a new mode1 was developed by the first-named
author (LML). This new mode1 wasbased on an explicit Eulerian algorithm, namely
an odd-order (3rd or higher) upwind differencing scheme, and Will be implemented
as tbe next operational regional mode1 by the Australian BoM (Puri, private com-
munication). It also has been used extensively in research applications (see, for
example, Leslie and Skinner, 1994; Speer and Leslie, 1994). At the same time
that the new Eulerian upwind mode1 was being tested in real-time mode, a very
different research path has been taken in the development of the above-mentioned
semi-Lagrangian model, particularly for very high-resolution and very long-range
predictions, where the time step size becomes a critical factor. In a series of studies,
the present authors have developed a sequence of increasingly accurate and efficient
semi-Lagrangian models. Currently, the most advanced product of this work is a

A Semi-Lagrangian NWP Model for Research Applications / 77
NEW SOUTH
OCEAN
1500
1000
fgg 500
Fig. 1 a. Location map of southeastern Australia, showing area of interest for the case studies in
Section 4. b. The Australian region analysis/forecast domain showing the disposition of grid
points in the horizontal at 150 km resolution.

78 / L.M. Leslie and R.J. Purser
high-order (in space and time), mass-conserving, semi-implicit, semi-Lagrangian
model (Leslie and Purser, 1994) that has as many equations as possible cast in
conserving form. It also employs forward trajectories (Purser and Leslie, 1994)
instead of the standard backward trajectories, and is cast on a grid, non-staggered
in both the horizontal and the vertical. It currently has been converted to global
form on a Gaussian grid and this version will be reported on in a future paper
when it is running successfully as a general circulation model (GCM). However,
the version that is currently used most heavily in research applications is essentially
that described by Leslie and Purser (1991), which is a bi-cubic, semi-Lagrangian
model using an iterative backward trajectory scheme on a non-staggered grid. It
is the model which is used in this study. It has been run almost exclusively in
limited-area form, but a global version (running out to 5 days) has been tested
successfully on operations data over a period of almost six months.
There are three main aims in this study. Section 2 provides a brief description
of each of the three models, namely, the current semi-implicit operational model,
the new explicit (upwind) operational model, and the semi-Lagrangian model. In
particular, some remarks are made concerning the reasons for developing the par-
ticular formation of each of the models. The first aim is to compare the three
models. The comparison is presented in Section 3 where the performance of the
models over the three month period December 1992 to February 1993 is summa-
rized, using a range of standard measures of skill. A second component of Section
3 is to compare directly the accuracy of the upwind and semi-Lagrangian schemes,
which are both formally second-order accurate in time and third-order accurate in
space. From Section 4 onwards, attention is focused on the semi-Lagrangian model
alone and the second aim of this study is to demonstrate how the accuracy of
the semi-Lagrangian model is realized effectively in particular case studies. Two
severe weather events of very different kinds are chosen for this purpose in Sec-
tion 4. The third aim is to assess how well the semi-Lagrangian model can be
adapted to a multi-processor computing environment. The adaptability of a model
to emerging computer architectures is an extremely important aspect of model per-
formance. This is achieved in Section 5 by examining the performance of the model
on two quite different computing platforms, a cluster of RISC/6000 workstations
and an IBM SP2 parallel computer. Finally, Section 6 discusses the results and
makes some concluding comments.
2 The models
The three models are described briefly and in turn, below.
a The current operational model
The current operational limited-area model used by the Australian BoM is known
as RASP Begional Assimilation and Prognosis). It is described fully in the se-
quence of articles by McGregor et al. (1978), Leslie et al. (1985) and Mills and
Seaman (1990). The model is cast in flux form, has centred (leapfrog) semi-implicit

temporal differencing, and second-order energy conserving spatial differencing on
an Arakawa C-grid in the horizontal. It is important to point out that the centred
temporal formulation and the use of the Asselin filter to control the computational
mode degrade the temporal accuracy to less than second-order. The vertical dis-
position to variables is on the Lorenz grid. Apart from some modifications to the
representation of physical processes, the model has remained largely unchanged
since the formulation of Leslie et al. (1985). The model has been run with a hori-
zontal resolution of 10 km until May 1994 when it was reduced to 75 km. There
are 17 levels in the vertical. All of the comparisons in Section 3 are at 150 km
resolution.
b The new Eulerian (upwind) model
The new Eulerian model was selected on the basis of simplicity, accuracy and
computational economy, particularly the anticipated (and subsequently realized)
efficiency on emerging parallel computing platforms. The temporal differencing
is second-order, two-time-level, and is optionally the Heun scheme (Mesinger and
Arakawa, 1976) or the Miller-Pearce scheme (Miller and Pearce, 1974). The stability
of these schemes has been analyzed extensively elsewhere and there is no need to
repeat the process here. However, it should be pointed out that both schemes are
weakly unstable and have slightly positive phase errors. However, the stability is
very weak and the phase leads are quite small. The spatial differencing employed is
a third-order upwind scheme, although it can be run at higher orders as an option.
The merits of using odd-order upwind schemes have been argued for by a number
of advocates (see, for example, Leonard, 1984). In essence the attractive properties
of the odd-order upwind schemes are: excellent phase properties at all orders;
very good amplitude properties at order 3 and higher; and a lack of oscillatory
behaviour in the solution near sharp boundaries, thereby minimizing the possibility
of destructive feedback from the oscillatory behaviour near sharp boundaries present
in the even-order upwind schemes.
This version of the model is being implemented by the Australian BoM as its
next operational regional NWP model, most likely sometime in early or mid-1995
(Puri, Australian BoM, personal communication).
c The semi-Lagrangian model
The formulation of the semi-Lagrangian model used in this study is also fully-
documented elsewhere (Leslie and Purser, 1991). It is a two-time-level split scheme.
The procedure comprises a semi-Lagrangian advection step followed by a number
(usually 4 or 5) of adjustment steps. The adjustment steps use the forward-
backward technique (Mesinger, 1977). The temporal differencing therefore is for-
mally second-order. As in the upwind model a non-staggered grid is used. The
interpolation scheme is third-order accurate, although it can be used optionally at
any order up to eight. The model has been tested extensively using both archived

TABLEI. SI skill scores for each of the three models for the
period December 1992to February1993.
Model December January February
Operational 35 39 40
Upwind 31 35 35
S-Lagrangian 31 36 34
operational data and in research mode of the simulation of a range of significant
weather events.
This model also was designed to take advantage of existing vector supercom-
puters, but also in anticipation of merging parallel systems as parallel computers
are finding widespread usage, and that trend is expected to increase from the mid-
1990’s onwards.
3 Results
The three models were run on a total of approximately 180 cases, at 0000 UTC
and 1200 UTCfrom 1 December 1992 to 28 February 1993. The forecasts were all
verified at 24 hours and were run at the then operational resolution of 150 km, on the
grid shown in Fig. lb. The forecast experiments run using the archived operational
initial fields and nested in the archived operational lateral boundary conditions
(forecasts) from the BoM’s global NWP model. In that sense the comparison was
a very clean one. The time steps used for the three models were the same as for the
physics step (16 minutes), but were 16 minutes, 4 minutes and 32 minutes for the
operational, upwind and semi-Lagrangian models respectively. The model statistics
prepared were the S1 skill scores, and the RMS values of the temperature and wind
fields.
a SI skill scores
The SI skill scores are shown in Table 1, and it is immediately clear that the third-
order models have very similar errors and are far more accurate than the operational
model. In each of the three months the scoresfor the third-order models are about 4
points lower than for the operational model. It is our experience that this difference
is large enough to be obvious to the eye on a daily inspection of forecasts charts.
An example will be presented below to illustrate what a four point skill score
difference can look like. The other significant feature of Table 1 is that the skill
scores for the third-order models are almost identical, for each of the three months.
Figures 2a and 2b are plots of the SI skill scores on a daily basis for December
1992. Figure 2a is a comparison of the semi-Lagrangian model and the RASP
model, while Fig. 2b compares the semi-Lagrangian and upwind models. The semi-
Lagrangian model is markedly more accurate than RASP on almost every day but,
in contrast, is very close to the upwind model on all but a few days. The means

30. .
25 .. --
20 ::I:::::::: ::::::::::::::::::
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
lime (DAYS)
50
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 2!j 26 27 20 29 30 31
llme(DAYS)
Fig. 2 a. A comparison of the daily SI skill scores at 0000 UTC for the operational model (dashed
line) and the semi-Lagrangian model (heavy line) for December 1992. b. As in 2a except for
the upwind (dashed line) and semi-Lagrangian model (heavy line).

TABLE 2. Mean RMS errors in 24-hour wind forecasts(m/s) at 850,
500 and 200 hPa levels averaged over the period De-
cember 1992 to February 1993.
Model 850 wind 500 wind 200 wind
Operational 5.1 6.5 9.1
Upwind 4.3 5.2 7.2
S-Lagrangian 4.1 5.3 7.0
TABLE 3. RMS errors (Degrees Celsius) in the 24-hour forecast tem-
peratures at 850, 500 and 200 hPa levels for the three
models, averaged over the period Dec. 1992 to Feb. 1993.
Model 850 Temp.
Operational 2.2
Upwind 1.8
S-Lagrangian 1.8
500 Temp.
1.8
I.5
1.4
200 Temp.
2.1
1.9
1.9
for the month are 35 for the operational model and 31 for each of the upwind and
semi-Lagrangian models.
b RMS wind and height errors
As was the case for the skill scores, the RMS wind and height errors are shown in
Tables 2 and 3 as a function of the month. They are also shown asfunctions of sev-
eral selected pressure levels, the lower, middle and upper troposphere (850,500 and
200 hPa). The reductions in model error, measured relative to radiosonde values,
are once again quite large for the third-order models over the operational model.
Again they are very similar in magnitude for the upwind and semi-Lagrangian
models.
Turning now to the RMS temperature errors, the same pattern appears in Table
3 as for Table 2. The mean temperature errors are markedly more accurate in
the 24-hour forecasts from the third-order models compared with the second-order
operational model, and are very similar between the two third-order models. Again,
the errors are measured relative to radiosonde values.
c Example
It is instructive to look at a case in detail. The particular event chosen involves
a tropical cyclone which was located off the east coast of Australia at 1200 UTC
13 January 1993, and was in the process of becoming extra-tropical. During the
following 24 hours the numerical analysis shows the cyclone moving SSE and the
central pressure filling in from 990 hPa to 994 hPa (see Figs 3a and 3b). The 24-
hour forecasts from the operational, upwind and semi-Lagrangian models are shown
in Figs 3c to 3e respectively. The semi-Lagrangian and upwind models produced

a
Fig. 3 a. The initial SLP analysis at 1200 UTC 13 January 1993. b. The verifying SLP analysis at
1200 UTC 14January 1994. c. The 24-hour forecast from the operational model, valid at 1200
UTC 14 January 1994. d. As in c. except for the upwind model. e. As in c. except for the
semi-Lagrangian model.

84 / L.M. Leslie and RJ. Purser
d
Fig. 3 Continued.

e
Fig. 3 Concluded.
forecasts that are almost identical, apart from a one hPa difference in the central
pressure of the cyclone. Both models have some minor deficiencies, such as not
moving the cyclone far enough to the south. However, the forecast obtained from
RASP is markedly inferior, even allowing for the different contouring interval.
The central pressure and position of the cyclone are quite poor, as is the prediction
of the anti-cyclone to the south-west of the continent. The skill score of 38 is 5
points worse than that of the other two models, both of which had St scores of 33.
d Global integration
The semi-Lagrangian model has been converted to a global form, in anticipation
of its use for both medium and extended range forecasting. At this stage only the
mass-conserving version of the model has not been adapted to the globe. In its non
mass-conserving form, the global version of the semi-Lagrangian model has been
run out to 5 days on a regular basis since March 1994. No concerted effort has been
made to verify the model other than to record the skill scores and to monitor mean
global quantities. In the first half of 1995 it is planned to begin trials of the mass-
conserving form of the model in real-time at the National Meteorological Center
(NMC), Washington in a comparison with existing NMC models. One example of
the global model is given in Figs 4a and 4b which show, respectively, the 5-day
forecast and the verifying analysis-valid at CKKMIUTC 18 September 1994.
4 Case studies
The coastal region of southeastern Australia (see location map, Fig. la) has a
diverse range of synoptic and mesoscale events, and many of these have been well-

documented. They include the so-called “east-coast lows” (Holland et al., 1987;
Leslie et al., 1987; McInnes et al., 1993) which are synoptic scale events that in
their most severe form are responsible for the major flooding events of this part of
the country. They also include tropical cyclones that have become extra-tropical.
Flooding also occurs from mesoscale events and it is such an event that is the
subject of the first case study, namely, a line of thunderstorms that crossed the
greater Sydney metropolitan area during the early afternoon of 10 February 1990
and brought with it flash flooding. This event has been studied in considerable detail
by Speer and Leslie (1994) as part of a more general investigation of mesoscale
weather events over southeastern New South Wales (NSW), and the Sydney area
in particular.
Another very different form of severe weather but one that is equally destructive
is the strong, dry, west-to-northwest winds that produce wild-fires, or bush-fires
as they are known in Australia. The most severe fire weather days are those that
occur at the end of a long period of dry weather, which in turn has been preceded
by a wet period. January 1994 saw the most destructive fires in the history of
Sydney, with the days of January 7th and 8th the most serious. Timely indications
of wind speed and direction are critical for fire authorities when these conditions
are present. Forecasts for the period 1200 UTC7 January to 1800 UTC 8 January 1994
form the basis of the second case study. It is intended also to extend investigation
of this event to a major diagnostic and simulation case study.
a The thunderstom line of 10 February 1990
This was a mesoscale event in which a quasi-stationary line of thunderstorms
formed to the west of Sydney in a meso-low into which a strong northwards moving
coastal ridge penetrated from the south. The following ingredients for convection
therefore were present: high low level relative humidity, instability, and a triggering
mechanism in the form of colder, drier air in the ridge. A line of convection
formed almost parallel to the coastal ranges and moved very slowly eastwards
across the greater Sydney area. Radar imagery (not shown) clearly indicated the
line of thunderstorms as well as remnants of earlier less organized convection, at
0350 U-X. Over the Sydney area the one-hour rainfall totals for the period 1350 to
1450 local time (0450 to 0550 UTC) are presented in Fig. 5. There are two distinct
maxima of over 60 tmn.
For this event, the semi-Lagrangian model was run at a resolution of 15 km
and with 25 levels in the vertical, and a time step reduced linearly to 16 minutes
and 8 minutes respectively. From stability considerations, the reduction was not
necessary, but a detailed test of efficiency was not the primary consideration here.
It was decided not to have a large disparity with the physics time step size. Initial
and boundary conditions were provided by telescoping down from the operational
analysis resolution of 150 km to 50 km horizontal resolution and finally to 15 km.
The telescoping was performed in the usual manner, namely, interpolation of fields
to the higher resolutions. No additional data were included in these runs. The model

Fig. 5 The observed one-hour rainfall totals (mm) for the period 1350 to 1450 local time (approxi-
mately 0450 to 0550 UTC)on 10 February 1990. Note the double maximum.
was run for a period of 6 hours from the initial time of 0000 UTC 10 February 1990
as this period of time covered the entire life-cycle from clear skies over Sydney to
the development and passage of the line of thunderstorms.
The initial fields are shown in Figs 6a to 6d. Figure 6a is the initial SLP (sea-
level pressure), with little sign of the cooler air associated with the coastal ridging.
The cooler air shows up in the SLP fields in that the air behind the ridge is from
a cooler southern ocean air mass. However, Fig. 6b reveals a large amount of low
level moisture with a large strip of the coastal region having near-surface rela-

Fig. 6 The initial fields from the numerical analysis at 0000 UTC IO February 1994. Figures a to d
are, respectively: the SLP (in hPa); the relative humidity at 950 hPa (percent); the plan view
of relative vorticity at 850 hPa ( 10W5); and a cross-section of relative vorticity through the line
A-B.
tive humidities above 90 per cent. Figures 6c and 6d indicate the plan view at
850 hPa and a cross-section through the line A-B, respectively, of the relative
vorticity field. Note that there is an area of low-level cyclonic vorticity present. At
6 hours into the simulation the cooler, drier air from the ridging had penetrated the
region and provided the lifting mechanism. The squall-line has developed dramat-
ically in these first 6 hours. Figures 7a to 7d are valid at 0600 UTCand reveal that
the line of thunderstorms has been well simulated. Figure 7a is the precipitation
rate (mm/hr) with a maximum of over 70 mmihr. Note that there is a clear double
maximum, as was the case in the observed rainfall. Figures 7b and 7c are again
the plan view at 850 hPa and the vertical cross-section, respectively, of the relative
vorticity field, this time the cross-section being through the line C-D. Finally, Fig.

Fig. 7 The six-hour forecast from the semi-Lagrangian model for 0600 UTC 10 February 1994. Figures
a to d arc, respectively: the instantaneous rainfall rate (mmk); the plan view of relative vorticity
at 850 hPa, the cross-section of relative vorticity along the line C-D; and the cross-section of
vertical velocity (IO-’ m/s) along the line C-D.
7d is the vertical cross-section through the line C-D of the vertical velocity, with
a classical updraft/downdraft structure.
b The$re weather event of 7 to 8 January 1994
Between 27 December 1993 and 14 January 1994, eastern NSW experienced its
worst bushfire period for many decades. For the greater Sydney area, with the
overwhelming majority of the state’s population, the worst period was 7 and 8
January. The fires presented the Australian BoM with an urgent need for timely
forecasts of wind speed and direction. In particular it emphasized the important
role that could be played by an operational high-resolution limited area model.
The most severe individual day of the fires in Sydney was 8 January 1994. For
that reason, simulations are presented for the 30-hour period starting at 1200 UTC

Fig. 8 Figures a and b are the 24-hour analyses from the Australian BoM operational limited area
model valid at 9 am and 9 pm local time on 8 January 1994. Figures c and d are the operational
model forecasts valid at the same times.
7 January to 1800trrc 8 January 1994. Figures 8a and 8b show the SLP analyses
at 0000 UTC8 January and 1200 UTC8 January. The 24-hour forecastsfrom the
BoM’s operational model for the corresponding times are given in Figs 8c and
8d. The operational forecastswere performed at the then operational resolution
of 150km and provide very little detail over the critical areaof the NSW coast.
For example, the 24-hour operational forecast for 0000 UTC(9 am local time)
indicatesa broad westerly airstreamover NSW comparedwith the analysiswhich
haslighter andmorevariable winds. The forecastfor 1200UTC (9 pm local time) is
better but providesalmostnoneof the detail that is critical for determiningbushfire
movement.
In contrast, the semi-Lagrangianmodel run at the much higher resolution of
50 km providesimportant detailsnot capturedat 150km. Figures9ato 9d and 10a
to 1Odshowthe sequence
of six-hourly forecastsof SLPand wind vectors (usingthe
samedatasetsasthe operational model) for the period 1800UTC 7 January to 1200

92 / L.M. Leslie and R J. Purser
30.05
35. OS
30. OS
3s. OS
145.OE 150.M 155. OE 160.OE
Fig. 9 A sequence of four SLP forecasts at 6-hourly intervals from 3 am local time to 9 pm local time
on 8 January 1994 from the semi-Lagrangian model at 50 km resolution, and 25 levels.

A Semi-Lagraqian NWP Model for Research Applications / 93
30.0s
35.0s
---
lr15.OE 150.OE 155.OE 160.OE
lU5.M lSO.OE
-
155.OE 160.OE
Fig. 9 Concluded.

96 / L.M. Leslie and RJ. Purser
Workstation
Token A
Ring
C Ethernet
I
I I I I I I i’l L
Fig. 11 Schematic illustration of the 16 node workstation cluster and server.
UTC8 January. These early results of the fire weather simulation are sufficiently
accurate to encourage further simulations down to much higher resolutions. These
will be attempted with the parallelized version of the model, described below.
5 Implementation in a parallel computing environment
As mentioned above, the upwind and semi-Lagrangian models were implemented
not only on scalar computers and vector supercomputers, but also on parallel com-
puting platforms. The performance of the upwind model on a workstation cluster
has been discussed elsewhere (Wightwick et al., 1995). The performance of the
semi-Lagrangian model on the same workstation cluster and on a scalable parallel
computer will be described now.
a Workstation cluster
The first parallel application of the semi-Lagrangian model was on a cluster of
16 RJSC/600 workstations topologically arranged as a 4 x 4 set connected by 4
Ethernet LANs to a server workstation. The workstation cluster is shown schemat-
ically in Fig. 11. The workstations were interconnected by Token Ring LANs in
a simple mesh topology, again as shown in Fig. 11. The earliest version of the

TABLE 4. Speed-up achieved relative to one processor, for 4. 8 and
16processors in the workstation cluster.
Resolution 4 Processor 8 Processor 16 Processor
150 km 1.99 2.29 2.74
50 km 3.01 5.98 8.66
20 km 7.52 II.64
cluster used the Parallel Virtual Memory (PVM) system Version 2.8 as the mes-
sage passing language (later upgraded to Version 3.2). PVM is a public domain
software system (Sunderam and Geist, 1990) that allows the utilization of a network
of computers as a single resource. It comprises a server process, referred to as the
server daemon, and a library of routines for message passing, process control and
sychronization. The model code also was written in a way that attempted to keep
changes from the serial version of the model to a minimum. Moreover, the “parallel
code” was contained, as much as possible, in separate subroutines in order to make
the code as general as possible and changing the number of processors a relatively
simple task.
The model was implemented first on a single machine. The forecast was run
at 150 km resolution on a domain comprising 64 x 40 points in the horizontal
and 15 levels in the vertical. It was then run on 4 processors, 8 processors and 16
processors using a simple domain decomposition approach. The resolution wasthen
increased to 50 km, and finally 20 km. The speed-up achieved on the workstation
cluster is summarized in Table 4. Note that the maximum speed-up achieved was
almost 12 for the 20 km, 16 processor case. Only very low speed-up values were
obtained for the small grid sizes used in the 150 km resolution forecasts. Given
that the communication overheads were so high with the slow message passing
associated with the token rings LANs, this level of speed-up was very acceptable.
b Parallel Supercomputer
During the second half of 1994 the models have been implemented on an advanced
parallel machine, namely the IBM PowerParallel SP2. Full details of the work,
which is not fully completed, are to be published elsewhere (Wightwick et al.
1995), and only a brief description will be given of the results that have been
obtained thus far. Access was restricted at this stage to only 12 processors of the
SP2. However, as will be seen, about a Gflop of performance was obtained from
this relatively small number of nodes.
The process of implementing the NWP model on the SP2 consisted largely
of porting the PVM based version on the RISC/6000 cluster to the SP2. The
changes required will not be described. In order to take advantage of the SP2
high-performance switch, it was necessary to rewrite the message passing routines
in a manner that was consistent with the IBM Message Passing Library (MPL).
The PVM version of the model adopted a master/slave programming approach.

The MPL uses the Single Program Multiple Data (SPMD) approach and supports
a number of functions not available in PVM. Once again, speed-up was calculated
relative to the performance of the model on a single SP2 node. Figure 12 shows
the speed-up as a function of four model resolutions, using 1, 2, 4, 8 and 12
processors. Note that owing to memory restrictions it was not possible to run the
19 km resolution version of the model on 1 or 2 processors. The speed-up for the
19 km resolution forecasts are shown relative to the 4 processor model run.
The most important result was that near-linear and even greater than linear speed-
up is achieved for a number of runs. The greater than linear speed-up is made
possible by the fact that the impact of the combined caches and translation look-
aside buffers of the SP2 nodes is a function of the problem size. This effect is
explained in much greater detail by Wightwick et al. (1995).
6 Discussion and conclusions
A new semi-Lagrangian NWP model has been developed and tested with the inten-
tion of using it as a real-time and research model. Initial work has focused on high
resolution regional modelling, but there are plans to use the model extensively in
global form as a medium-range, extended term and general circulation model.
In this study the new semi-Lagrangian model was compared directly with the
BoM’s present operational semi-implicit model, and also with a new high-order
upwind Eulerian model. The semi-Lagrangian and Eulerian models were both for-
mally second-order in time and chosen to be third-order in space for the purposes
of this comparison. It was found that the operational model was markedly infe-
rior to the new models using traditional measures of model skill on a three-month
archived operational dataset. The upwind and semi-Lagrangian models were on av-
erage 4 skill score points superior to the operational model, and had RMS errors in
the 24-hour forecast height and wind fields at various levels in the atmosphere that
were about 30 per cent lower. Moreover, it was also found that there was almost
negligible difference between the Eulerian and semi-Lagrangian models in terms of
skill. However, an accurate comparison of the elapsed times of the two approaches
in terms of times required per forecast day have not been carried out in this study
and no conclusions have been drawn. Examples and case studies also were pre-
sented to illustrate the performance of the semi-Lagrangian model in both limited
area and global form. The authors intend to continue to use the semi-Lagrangian
model for most of the future research and development work as it appears to pro-
vide the widest range of options. Nearly all of the problems associated with the
semi-Lagrangian scheme have been resolved, so the advantage of being able to
use a time step many times longer than that allowed by the Eulerian formulation
appears to be decisive in most NWP applications. Remaining problems such as
lack of formal and practical conservation properties are now being addressed by a
number of researchers including the present authors.
It was anticipated that the semi-Lagrangian scheme might pose some problems
for parallelization, notably in the parts of the code responsible for the interpola-

A Semi-Lagrangian NW Model for Research Applications / 99
IZ- -c Physics __.*-
--A-. NoPhysics
. . . .
_...
._.*
to-
.___._ Li-
I .
._.'
Number of Processors
Fig. 12 Speed-up achieved on the parallel scalable computer at four different resolutions as a function
of the number of nodes.

tions. However, this proved not to be the case and a quasi-linear speed-up was
obtained. Additional work not reported on here (Wightwick and Leslie, personal
communication) for the semi-implicit semi-Lagrangian formulation also was ex-
pected to provide additional difficulties in the solving of the Helmholtz equations
associated with the semi-implicit scheme. However, the multi-grid method used
to solve the Helmholtz equations also was found to parallelize easily, and yielded
a quasi-linear speed-up. Future applications of the model are being made on ma-
chines with considerably greater numbers of nodes and this work will be reported
on at a later time.
Acknowledgements
The authors would like to thank staff members of the Australian Bureau of Meteo-
rology and the National Meteorological Center, Washington for helpful discussions.
Glenn Wightwick of The Australian Computing and Communications Institute and
of IBM Australia provided much of the expertize in the implementation of the
model code on the parallel computers. RIP was supported for this project by the
University Corporation for Atmospheric Research (UCAR) Visiting Scientist Pro-
gram, and LML was partially supported by the Office of Naval Research under
ONR Grant NOOO14-94-1-0556.
References
HOLLAND, G.J.; L.M. LESLIE and A.H. LYNCH. 1987.
Australian east-coast cyclones. Part I: Synoptic
overview and case study. Mon. Weather Rev.
115: 30243036.
LEONARD. B.P. 1984. Third-order upwinding as a
rational basis for computationaljhid dynamics.
0X-83, J. Noye and C.A.J. Fletcher (Eds),
Elsevier North-Holland, pp. 106.
LESLIE.L.M.;G.A.MILLS.L.W.LOGAN, D.J.GAUNTLETT,
G.A. KELLY, M.J. MANTON, J.L. MCGREGOR and
J.M. SARDIE. 1985. A high resolution primitive
equations model for operations and research.
Austral. Meteorol. Mag, 33: I l-35.
-; G.J. HOLLAND and A.H.LYNCH. 1987. Aus-
tralian east-coast cyclones. Part II: Numerical
Modelling Study.Mon. Weather Rev. 115: 3037-
3053.
- and R.J. PURSER. 1991. High-order nu-
merics in a three-dimensional time-split semi-
Lagrangian forecast model. Mon. Weather Rev.
119: 1612-1623.
and -. 1994. A mass-conserving
semi-Lagrangian scheme employing forward
trajectories. Proc. 10th Conf. on Numerical
Weather Pnxtion. Portland, Oregon. 18-22 July
and T.C.L. SKINNER. 1994. Numerical ex-
periments with the WestAustralian summertime
heat trough. Weather Forecas?. 9: 371-383.
MCGREGOR, J.L.; L.M. LESLIE and D.J. GAUNTLE-IT.
1978.The ANMRC limited area model: Consol-
idated formulation and operational results. Mon.
Weather Rev. 106: 427-438.
MCINNES, K.L.; L.M. LESLIE and J.L. MCBRIDE. 1993.
Numerical simulation of cut-off lows on the
Australian east coast: Sensitivity to sea-surface
temperature and implications for global warming.
ht. J. Climatol. 12: 783-795.
MESINGER, F. 1977.Forward-backward scheme and
its use in a limited-area model. Contrib. Atmos.
Phys. 50: 200-210.
-and A. ARAKAWA. 1976. Numerical methods
used in atmospheric models. GARP Publ. Ser.
No. 17, ICSLVWMO, Geneva, Switzerland,
pp. 53-56.
MILLER, M.J. and R.P. PEARCE. 1974. A three-
dimensional primitive equations model of cu-
mulus convection. Q. J. R. Meteorof. Sot. 1001
133-154.
MILLS, G.A. and R.S. SEAMAN. 1990. The BMRC re-
gional data assimilation system.Mon. Weather
1994. Am. Meteorol. Sot., Boston. pp. 82-83. Rev. 118: 1217-1237.

PURSER, RJ. and L.M. LESLIE. 1994. An efficient
semi-Lagrangian schemeusing third-order semi-
implicit time integration and forward time tra-
jectories. Mon. Weather Rev. 122: 745-756.
ROBERT.A.; H.HENDERSON and CTURNBULL. 1972.
An implicit time integration scheme for bam-
clinic modes of the atmosphere. Mon. Weather
Rev. 1001 329-335.
SPEER, M.S. and L.M. LESLIE. 1994. Predictability of
mesobeta convection using a high resolution
NWP model. Proc. 10th Conf. on Numerical
Weather Prection. Portland, Oregon. l&22 July
1994, Am. Meteorol. Sot., Boston. pp. 472-
474.
SUNDERAM, v. andG. GEIST. 1990.The PVM System.
In: Scientij% Excellence in Supercomputing,
K.R. Billingsley and E. Demhanes (Eds), Bald-
win Press,Athens, Georgia, pp. 779-804.
TEWELES, s. and H. WOBUS. 1954. Verification of
prognostic charts. Bull. Am. Meteorof. Sot. 35:
455-463.
WIGHTWICK.G.R. and LMLESLIE. 1992. Parallelim-
plementation of a Numerical Weather Ptedic-
tion model on a RJSC/6000 cluster. Proc. Fifth
Australian Supercomputer Conf., Melbourne,
Australia. pp. 135-142.
-; - and S.F. WAIL. 1995. A numeric
weather prediction model for the IBM SP2 par-
allel computer. First International Conference
on Algorithms and Architectures for Parallel
Processing. Brisbane, Australia, 1%21 April
1995, pp. 71-78.

A Semi-Lagrangian NWP Model For Real-Time And Research Applications Evaluation In Single- And Multi-Processor Environments

Recommended

Recommended

More Related Content

Similar to A Semi-Lagrangian NWP Model For Real-Time And Research Applications Evaluation In Single- And Multi-Processor Environments

Similar to A Semi-Lagrangian NWP Model For Real-Time And Research Applications Evaluation In Single- And Multi-Processor Environments (20)

More from Alicia Buske

More from Alicia Buske (20)

Recently uploaded

Recently uploaded (20)

A Semi-Lagrangian NWP Model For Real-Time And Research Applications Evaluation In Single- And Multi-Processor Environments