Data Management for High-Performance Scientific Computation
Traditionally, scientists and engineers investigated natural
phenomena through empirical experimentation and theoretical
analysis. In recent years, increases in computing power and
our understanding of how to model physical phenomena
have allowed simulations to become a third investigative
paradigm. Now computer simulations are a vital part of
applications as diverse as weather prediction, protein
folding, and aircraft design.
Because recent increases in computational power have not been
matched by increases in I/O subsystem speeds, I/O is an ever-increasing
fraction of run time for high-performance parallel simulation
codes. To make matters worse, the degree of parallelism in
I/O subsystems has actually been dropping, due to the popularity
of SMP-based parallel computers and shared file systems.
Another complication is the fact that scientists normally visualize
the results of their simulation runs on a computer located far
away from the simulation platform, and movement of the data
across the internet is not infrequently the slowest part of
an end-to-end simulation run.
The Panda project addresses these challenges by designing new
approaches for moving data to and from local and remote disks,
as well as techniques for hiding the cost of these I/O activities.
The new approaches are implemented in the context of the Panda
parallel I/O library and related libraries, and tested in the
context of real applications. The testbed applications are
usually drawn from the Center for the Simulation of Advanced
Rockets, where Panda is in daily use in CSAR's production codes.
We view techniques that prove their merit in the context of CSAR
applications as candidates for technology transfer to widely
used parallel I/O libraries, such as implementations of the MPI-IO
parallel I/O standard; transfer of active buffering from
Panda to the ROMIO implementation of MPI-IO is currently underway.