Subject: IO Requirements Summary

From: [email-masked] (Alok Choudhary)
Date: Sun, 25 Sep 94 15:33:37 EDT

% I was not clear who is collecting these (Ian or Piyush?). Please provide
% feedback and suggestions. I believe these will be sections of
% and overall requirement section? Alok

\documentstyle[11pt]{article}

% Prepared by : Alok Choudhary
% To add bibliography, if needed
% Only a brief discussion of requirements and questions these requirement
% pose are presented. No specific proposal for language features
% is presented at this time.

% SUGGESTIONS TO ADD and MODIFY THE MATERIAL are most welcome. Organizational
% suggestions are welcome as well.
% 

\begin{document}

\section{A Summary of Input-Output Requirements}

Incorporating features in HPF to allow High-Performance Input-Output
is considered very important in order to provide scalable performance
for accessing secondary storage and beyond. Several basic requirements have
been identified; namely,
\begin{itemize}
\item transparent parallel accesses to files,
\item out-of-core arrays and persistent arrays,
\item real-time output,
\item checkpointing and restart,
\item an ability to stripe files over disks, and
\item asynchronous I/O for overlapping computation with I/O.
\end{itemize}

The following summarizes each of these requirements.

\subsection{Transparent Parallel Accesses to Files}

>From a user's perspective, high-performance access to files is
important, whether the underlying mechanism involves parallel accesses or
something else. It is clear, however, that given the technological trend,
extending parallelism to I/O is the only possible way to achieving
high-performance in I/O.

Parallelism in accessing files has an implication on the language 
using which the parallel accesses will be performed.
Fortran (on which HPF is based) file I/O has sequential semantics. 
There are several questions to be resolved for implementing
parallel accesses to files from HPF. Is it necessary or important
to provide an explicit mechanism to perform parallel accesses to
a file using new constructs such as parallel read and write statements?
If yes, how do these statements interact with traditional (sequential)
I/O accesses semantics?
Given that parallel access to files is needed, is it necessary to
define new types of files for this purpose? 

\subsection{Striping/Distributing Files over Disks}

Given that parallelism is crucial to achieving high-performance in I/O,
files are and will be striped over disks. The important questions
to address are what controls does a user have in specifying
such distributions. Does the user see the individual stripes (or portions)
of files from within an HPF program, and are the Fortran/HPF semantics
preserved in such a file? These questions are obviously related.
Most parallel machines implement some sort of striped file system, and
many allow control on stripe size, data distribution, number of disks etc.

To take advantage of above features, it has been argued that future extensions
in HPF allow an option of
breaking away from traditional sequential semantics of
fortran I/O, at least for some special
types of files.
This argument is similar to the one on which distribution of data
over processor memories is based in the HPF model.
Examples of these files include scratch files which
may exist during the execution of a program to store temporary data.
Furthermore, in many applications, it is desired to write data in one
distribution and read it in another (potentially by a different number of
processors). Following the current language semantics, accesses in most
cases will have to be sequentialized.

Various proposals for distributing files over disks were made during the
HPF-I round. These proposals are summarized in the Journal of Development
of the HPF-I language specification.

\subsection{Out-of-Core and Persistent Arrays}
%Model?

Many potential applications of HPF operate on large quantities of data.
Primary data structures for these applications reside on disks.
These data structures are termed as out-of-core. In order not to
restrict application developers to problem sizes that fit in the memory
of a system,
providing a way to specify out-of-core data structures from HPF
is considered important. The mechanisms for providing this feature
includes extending the directives in HPF to declare out-of-core arrays
and their distributions on disks. 

Persistent arrays are those that persist beyond a program's execution.
For example, data produced by one program is later needed by another,
may be provided using this technique. Meta data associated with these
arrays may describe distribution and access mechanism for 
persistent arrays. Mechanisms for doing this from HPF need to be considered.

\subsection{Real-Time Output}

This requirement essentially deals with needing output data in real-time
from a program while the program is executing. The main application
of this requirement is on-line visualization and interpretation
of the data produced by a program, which could be used
to check the results or status and for tuning or navigating a program's
execution (i.e., modifying input parameters to a program to guide
its execution).

This requirement is also related to the task-parallel requirements
in HPF.
One way of incorporating real-time output would be to view
the HPF program as one data-parallel task and the visualization task
as another task. This would require, therefore, communication between
two tasks, an issue being considered by the task-parallel group as well.

\subsection{Checkpointing/Restart}

A capability to checkpoint an HPF program's state and restart the
program later
(potentially on a different number of processors) is deemed quite
important.
This is especially critical in those applications that execute for
a very long time, and thus,  in those situations
the possibilities of machine or software failures
are great. Users want a capability to restart their applications later,
but from an intermediate state to avoid recomputing from the beginning.
Specifically, the checkpointing time should be minimized, which in turns
means that efficient parallel I/O for checkpointing is required.
Another important requirement is the ability to restart the
computation from a previously saved checkpoint on a different number of
processors than the number of processors on which the program was
executing when the last checkpoint was taken.
The reason for desiring this may be the availability of less number
of processors
(due to failures or some other reasons) or more number of nodes.
This has clear implications
on the language, compilation and the runtime systems. If checkpointing
information is distribution and number of processor dependent then
restarting on a different number of processors (which will have
a different distribution of data, if not in shape, definitely in size)
may be very difficult, if not impossible.

Language features or directives can help compilers determine good places
in programs to perform checkpoints. Just like users can use their knowledge
of the application domain to provide information on data mapping and
interactions using directives, users can also provide information
on which data to checkpoint and where. This can not only reduce the
burden on the compiler but it can help reduce the amount of information
to be saved in a checkpoint.

\subsection{Asynchronous I/O and Prefetching}

Given that I/O is very slow compared to the computation and even communication,
it is obviously attractive to overlap computation with I/O as much as possible.
This requires an ability to perform asynchronous I/O and 
user level prefetching of data. There are three possible approaches to
providing prefetching and asynchronous I/O capabilities; namely, runtime
libraries, language extensions or directives. 

The idea of using runtime libraries for asynchronous I/O
is similar to the usage of libraries for irregular computations
and reductions as incorporated in the HPF library. This requires the
least number of extensions to the language. However, since
for asynchronous I/O, the control returns to the program while
I/O is going on, the mechanism for signaling completion and
synchronization is not clear, and needs to be addressed.
Furthermore, ensuring that the buffers containing data for I/O
are not polluted must be resolved within the constraints of
HPF (and thus within the constraints of the base language F90).

Using directives for asynchronous I/O is another option. However,
issues of signaling and data pollution need to be resolved here
as well. The following example illustrates the use of directives.
The basic idea is for the user to specify a region in which 
the user guarantees not to access the buffers involved in the I/O.
Thus, asynchronous I/O may be performed while computation within
the region is going on. If the I/O does not finish within this
region, then a mechanism is required to block further computation
until the I/O completes.

\begin{verbatim}

      READ (7,30) A(:,:), Q,Z
!HPF$ I PROMISE NOT TO TOUCH THE I/O VARIABLES
C Specifying that the variables involved in the preceding read operation
C will not be accessed between this directive and its corresponding
C end of region directive

      B=TRANSPOSE (MATMUL(C<D))
      CALL TEDIOUS CRUNCHING
!HPF$   NOT!
C A directive to specify end of user guarantee.
      z= sum (A) *z

\end{verbatim}

There are many open issues to be addressed if this mechanism is to work.
These include, how to handle error conditions? if the I/O
is performed in subroutines and control returns to the calling
routine, how to specify completion of I/O or how to inquire
about completion etc?

\end{document}