Subject: FORALL details and proposal

From: David Loveman <[email-masked]>
Date: Tue, 2 Jun 92 14:07:41 PDT

Following is a commentary on FORALL,  (for reference) the Fortran 8x
definition of FORALL, some notes on the Thinking Machines, MasPar, and
DECmpp definitions of FORALL, and an attempt to augment the April 2
FORALL proposal with some more detail, especially for the simple case. 
Note that the "scalarized" definition here is semantically different,
and I think more accurate, than the one given in the April 2 proposal,
and needs to be talked about.

Sorry about the LaTeX.  I intend to put it into HPF Canonical LaTeX
Form (HCLF) once it is defined.  I hope it is readable.  (It will LaTeX
as is, with no header files.)

-David

-------------------------------------------------------------------------------

%forall.tex

\documentstyle[11pt]{article}
\pagestyle{plain}
\pagenumbering{arabic}
\marginparwidth 0pt
\oddsidemargin  .25in
\evensidemargin  .25in
\marginparsep 0pt
\topmargin   -.5in
\textwidth 6in
\textheight  9.0in

\title{Analysis of FORALL Definitions and Proposal}
\author{David Loveman}
%\date{   }

\begin{document}

\maketitle

\section{Overview}

There are currently three definitions of FORALL:  Appendix F of S8.104
(Fortran 8x), MasPar (DECmpp) Fortran, and Thinking Machines Fortran. 
In addition, there is an existing HPFF proposal for FORALL that differs
in some details from the existing definitions.  This document reviews
the existing definitions of the FORALL statement as well as the current
HPFF proposal for FORALL, discusses the differences between them, and
provides a detailed revised HPFF FORALL proposal based on the S8.104
definition and the current HPFF proposal.

COMPASS originally implemented FORALL in its compiler technology based
on the definition in Appendix F of S8.104 (Fortran 8x).  This COMPASS
compiler technology is the base technology for the Thinking Machines,
MasPar, and DECmpp Fortran compilers.  As a result, the definitions of
FORALL are ``the same'' except for some wording differences, and they
mean the same since the base implementations were the same.

These compilers have been evolving on their own for a while, so there
may be some slight divergence but  the three companies all ``know what
they mean'' when they say FORALL. The major evolution efforts have been
to handle progressively more FORALL cases in parallel.  

This proposal anticipates that, assuming HPF is successful, there will
be other compilers developed that will contain FORALL.  The developers
of those compilers will base their definition on the HPF definition and
may not ``know what we mean'' when we say FORALL.  Thus it is in all
our interests to get the HPF stand-alone definition of FORALL accurate,
especially since the S8.104 definition is in a somewhat obsure place. 
In addition, there is a requirement to allow efficient implementations
on scalar machines as well as on parallel machines.

\section{Definition of FORALL from X3J3/S8 Version 104 - April 1987}

{\bf F.2.3 Element Array Assignment - FORALL.}  The element array
assignment statement is used to specify an array assignment in terms of
array elements or array sections.  The element array assignment may be
masked with a scalar logical or bit expression.  Rule R223 for {\it
action-stmt} is extended to include the {\it forall-stmt} and appears
as RF40 (F4.3.2).\\

\noindent
{\bf F2.3.1 General Form of Element Array Assignment.}

\begin{verbatim}
RF27 forall-stmt          is FORALL (forall-triplet-spec-list [ ,scalar-mask-expr ])
                                  forall-assignment

RF28 forall-triplet-spec  is subscript-name = subscript : subscript [ : stride]
\end{verbatim}

\noindent
Constraint:  {\it subscript-name} must be a {\it scalar-name} of type integer.\\

\noindent
Constraint:  A {\it subscript} or a {\it stride} in a {\it
forall-triplet-spec} must not contain a reference to any {\it
subscript-name} in the {\it forall-triplet-spec-list}.

\begin{verbatim}
RF29 forall-assignment    is array-element = expr
                          or array-section = expr
\end{verbatim}

\noindent
Constraint:  The {\it array-section} or {\it array-element} in a {\it
forall-assignment} must reference all of the {\it forall-triplet-spec
subscript-names}.

For each subscript name in the {\it forall-assignment}, the set of
permitted values is determined on entry to the statement and is
\[  m1 + (k-1) * m3, where~k = 1, 2, ..., INT((m2 - m1 + m3) / m3)  \]

\noindent
and where {\it m1}, {\it m2}, and {\it m3} are the values of the first
subscript, the second subscript, and the stride respectively in the
{\it forall-triplet-spec}.  If {\it stride} is missing, it is as if it
were present with a value of the integer 1.  The expression {\it
stride} must not have the value 0.  If for some subscript name 
$INT((m2 -m1 + m3) / m3) \leq 0$, the {\it forall-assignment} is not executed.\\

Examples of element array assignments are:

\begin{verbatim}
FORALL (I=1:N, J=1:N) H(I,J) = 1.0 / REAL(I + J - 1)

FORALL (I=1:N, J=1:N, A(I,J) .NE. 0.0) B(I,J) = 1.0 / A(I,J)
\end{verbatim} 

{\bf F.2.3.2 Interpretation of Element Array Assignments.}  Execution
of an element array assignment consists of the evaluation in any order
of the subscript and stride expressions in the {\it
forall-triplet-spec-list}, the evaluation of the scalar mask
expression, and the evaluation of the expr in the {\it
forall-assignment} for all valid combinations of subscript names for
which the scalar mask expression is true, followed by the assignment of
these values to the corresponding elements of the array being assigned
to.  If the scalar mask expresion is omitted, it is as if it were
present with the value true.  If the scalar mask expression is of type
BIT, an expression with value B'1' is treated as true and an expression
value B'0' is treated as false.

The {\it forall-assignment} must not cause any element of the array
being assigned to be assigned a value more than once.  The scope of the
subscript name is the FORALL statement itself. A function reference
appearing in any expression in the {\it forall-assignment} must not
redefine any subscript name.

\section{DECmpp Parallel Fortran Version 1.0 (= MasPar MPF Version 1.2)}

\begin{itemize}

\item An upper bound subscript in a forall-triplet-spec may be omitted,
no semantics given.

\item The scalar-mask-expr must be of type logical (i.e. not type bit,
which is also a removed extension) and can reference the subscript-names.

\item Conflict:  The text says `` A [forall-stmt] can be specified in
terms of array elements or array sections . . .'' but then goes on to
say that ``[forall-asmt] is $<$array-element = expr$>$''

\item The forall-assignment must not contain a character expression.

\item Text says that subscripts and strides are evaluated first, but
does not say ``in any order.''

\item There is no precise statement of the set of permitted values of
the subscripts as determined on entry.

\item There is no explicit statement corresponding to the last
paragraph:  ``The forall-assignment must not cause any element of the
array being assigned to be assigned a value more than once.  The scope
of the subscript name is the FORALL statement itself. A function
reference appearing in any expression in the forall-assignment must not
redefine any subscript name.''

\item There is an explicit statement describing the cases for whch
parallel code is generated:  ``Arrays indexed by FORALL subscripts must
use all the FORALL subscripts exactly once, in the order in which they
appear in the FORALL header.  These subscripts must be bare - that is,
not in expressions - and they cannot appear as section bounds or
strides.  There can be additional scalar subscripts or sections not
involving any FORALL subscript-name(s); any sections must follow the
FORALL subscript-name(s).  There cannot be any transformational
intrinsics nor any user-written function calls, only scalar intrinsics
(not involving any FORALL subscripts) or elemental intrinsics
(involving FORALL subscripts).''

\end{itemize}

\section{CM Fortran Reference Manual Version 1.0, February 1991}

\begin{itemize}

\item The text is very close to that in S8.104.

\item No bit expressions in scalar-mask-expr.

\end{itemize}

\section{Current HPFF Proposal - Synchronized (SIMD-style) FORALL statement}

\subsection{Single Assignment}

\begin{verbatim}
      FORALL (v1=l1:u1:s1, v2=l1:u2:s2, ..., vn=ln:un:sn [, mask] ) &
           a(e1,...,em) = rhs
\end{verbatim}

    The FORALL statement declares v1,v2,...,vn to be integer
    variables locally to the FORALL statement.  Associated
    with each is a subscript-triplet.  There is also an optional
    logical mask expression that may depend on the variables
    v1,v2,...,vn. The single statement controlled must be an
    assignment with an array reference on the left-hand-side.
    Each of the variables v1,v2...,vn must appear within the
    subscript expression(s) on the left-hand-side.  (This is
    a syntactic consequence of the semantic rule that no two
    execution instances of the body may assign to the same
    array element.)  The assignment statement is executed
    once for every combination of values for the variables
    for which the mask expression is true.  (If there is no
    mask expression, it is assumed always to be true.)
    The various instances of the assignment statement are
    executed in such a way that it appears that the right-hand
    sides of all instances, and also the subscript expressions
    on the left-hand-sides of all instances, and also the mask
    expression, are evaluated before any assignment is performed.
    So a FORALL statement is roughly equivalent to:

\begin{verbatim}
      templ1 = l1
      tempu1 = u1
      temps1 = s1
      DO v1=templ1,tempu1,temps1
        templ2(v1) = l2
        tempu2(v1) = u2
        temps2(v1) = s2
        DO v2=templ2(v1),tempu2(v1),temps2(v1)
          ...
            templn(v1,v2,...,v<n-1>) = ln
            tempun(v1,v2,...,v<n-1>) = un
            tempsn(v1,v2,...,v<n-1>) = sn
            DO vn=templn(v1,v2,...,v<n-1>),tempun(v1,v2,...,v<n-1>),  &
                 tempsn(v1,v2,...,v<n-1>)
            tempmask(v1,v2,...,vn) = mask
            tempe1(v1,v2,...,vn) = e1
            ...
            tempem(v1,v2,...,vn) = em
            temprhs(v1,v2,...,vn) = rhs
        END DO
	  ...
        END DO
      END DO
      DO v1=templ1,tempu1,temps1
        DO v2=templ2(v1),tempu2(v1),temps2(v10
          ...
            DO vn=templn(v1,v2,...,v<n-1>),tempun(v1,v2,...,v<n-1>),  &
                 tempsn(v1,v2,...,v<n-1>)
              IF (tempmask(v1,v2,...,vn)) THEN
                a(tempe1(v1,v2,...,vn),...,tempem(v1,v2,...,vn))   &
                      = temprhs(v1,v2,...,vn)
              END IF
            END DO
          ...
        END DO
      END DO
\end{verbatim}

Of course, there are other ways to implement it as well.

\subsection{Block FORALL}

\begin{verbatim}
    FORALL (... e1 ... e2 ...)
        s1
        s2
        ...
        sn
      END FORALL
\end{verbatim}

means exactly the same as

\begin{verbatim}
      temp1 = e1
      temp2 = e2
      FORALL (... temp1 ... temp2 ...) s1
      FORALL (... temp1 ... temp2 ...) s2
      ...
      FORALL (... temp1 ... temp2 ...) sn
\end{verbatim}

That is, a block FORALL means exactly the same as replicating
the FORALL header in front of each array assignment statement
in the block, except that any expressions in the FORALL header
are evaluated only once, rather than being re-evaluated before
each of the statements in the body.

Thus one may wish to think of a block FORALL as synchronizing twice
per contained statement: once after handling the rhs and other
expressions but before performing assignments, and once after all
assignments have been performed but before commending the next
statement.  (In practice, appropriate flow analysis often permits
the compiler to eliminate unnecessary synchronizations.)

\subsection{Allow Statements Other Than Assignments in Body}

\subsubsection{IF and WHERE}

Assume that a block IF is first reduced to single IF statements
by introducing temporary variables and replicating the IF header:

\begin{verbatim}
      IF (m) THEN
        s1
        ...
        sm
      ELSE
        t1
        ...
        tn
      END IF
\end{verbatim}

\noindent
becomes

\begin{verbatim}
      tempm = m
      IF (tempm) s1
      ...
      IF (tempm) sm
      IF (.NOT. tempm) t1
      ...
      IF (.NOT. tempm) tn
\end{verbatim}

Then we simply define

\begin{verbatim}
      FORALL (v1=l1:u1:s1,...,vn=ln:un:sn,fmask) IF (imask) s
\end{verbatim}

to mean

\begin{verbatim}
      FORALL (v1=l1:u1:s1,...,vn=ln:un:sn,fmask .AND. imask) s
\end{verbatim}

WHERE can be treated similarly, though the details are a bit
more complicated.

The motivation here is to make it easier to transform DO loops
into FORALL statements and vice versa, despite the fact that they
may contain conditional statements.

\subsubsection{FORALL}

\begin{verbatim}
      FORALL (va1=...,...,van=...,maska)  &
           FORALL (vb1=...,...,vbn=...,maskb) s
\end{verbatim}

\noindent
means

\begin{verbatim}
      FORALL (va1=...,...,van=...,vb1=...,...,vbn=...,maska.AND.maskb) s
\end{verbatim}

\noindent
assuming there is no duplication of the variable names (a compiler
should logically rename the variables if there is a duplication).

The motivation here is to make it easier to transform DO loops
with DO loops (that happen not to be closely nested) into FORALL
blocks within FORALL blocks, and vice versa.

\section{Comments on the Above}

\begin{itemize}

\item I believe that we should base the HPF FORALL on the original
definition from S8.104.

\item Fortran 90 does not have a BIT data type.  The definition of
FORALL must change accordingly.

\item Digital, MasPar, and Thinking Machines seem to agree that
character expressions should be disallowed.

\item Omissions in the Digital and MasPar descriptions appear to be
document wording problems, rather than intention or implementation
problems.  Thinking Machines documents do not have this problem because
they copied text directly from S8.104.

\item Naturally I am favorably disposed to the approach in the current
proposal of definition by means of source-to-source transformation to
simpler forms.  Unfortunately, the definition in the current proposal
is not quite correct with regard to questions such as ``How many times
does the rhs get evaluated.''  The source-to-source transformational
definitions should be correct scalarizations of the language
constructs, usable as such for a (naive) workstation implementation.

\item Our definitions should contain at least two parts:  a formal
proposal part defining, in the Fortran 90 specification style, what the
HPF language features are;  and a consequences part providing
discussion, rationale, intended usage, and (possibly) non-obvious
consequences of the formal proposal. 

\end{itemize}

\section{A Revised Proposal for FORALL}

\subsection{Element Array Assignment - FORALL}  The element array
assignment statement is used to specify an array assignment in terms of
array elements or array sections.  The element array assignment may be
masked with a scalar logical expression.  Rule R215 for {\it
executable-construct} is extended to include the {\it forall-stmt}.

\subsubsection{General Form of Element Array Assignment}

\begin{verbatim}
forall-stmt          is FORALL (forall-triplet-spec-list [ ,scalar-mask-expr ])
                          forall-assignment

forall-triplet-spec  is subscript-name = subscript : subscript [ : stride]
\end{verbatim}

\noindent
Constraint:  {\it subscript-name} must be a {\it scalar-name} of type integer.\\

\noindent
Constraint:  A {\it subscript} or a {\it stride} in a {\it
forall-triplet-spec} must not contain a reference to any {\it
subscript-name} in the {\it forall-triplet-spec-list}.

\begin{verbatim}
forall-assignment    is array-element = expr
                     or array-section = expr
\end{verbatim}

\noindent
Constraint:  The {\it array-section} or {\it array-element} in a {\it
forall-assignment} must reference all of the {\it forall-triplet-spec
subscript-names}.

For each subscript name in the {\it forall-assignment}, the set of
permitted values is determined on entry to the statement and is
\[  m1 + (k-1) * m3, where~k = 1, 2, ..., INT((m2 - m1 + m3) / m3)  \]

\noindent
and where {\it m1}, {\it m2}, and {\it m3} are the values of the first
subscript, the second subscript, and the stride respectively in the
{\it forall-triplet-spec}.  If {\it stride} is missing, it is as if it
were present with a value of the integer 1.  The expression {\it
stride} must not have the value 0.  If for some subscript name 
$INT((m2 -m1 + m3) / m3) \leq 0$, the {\it forall-assignment} is not executed.\\

Examples of element array assignments are:

\begin{verbatim}
FORALL (I=1:N, J=1:N) H(I,J) = 1.0 / REAL(I + J - 1)

FORALL (I=1:N, J=1:N, A(I,J) .NE. 0.0) B(I,J) = 1.0 / A(I,J)
\end{verbatim} 

\subsubsection{Interpretation of Element Array Assignments}  

Execution of an element array assignment consists of the evaluation in
any order of the subscript and stride expressions in the {\it
forall-triplet-spec-list}, the evaluation of the scalar mask
expression, and the evaluation of the expr in the {\it
forall-assignment} for all valid combinations of subscript names for
which the scalar mask expression is true, followed by the assignment of
these values to the corresponding elements of the array being assigned
to.  If the scalar mask expresion is omitted, it is as if it were
present with the value true.

The {\it forall-assignment} must not cause any element of the array
being assigned to be assigned a value more than once.  The scope of the
subscript name is the FORALL statement itself. A function reference
appearing in any expression in the {\it forall-assignment} must not
redefine any subscript name.

\subsubsection{Scalarization of the FORALL Statement}

A {\it forall-stmt} of the general form:

\begin{verbatim}
FORALL (v1=l1:u1:s1, v2=l1:u2:s2, ..., vn=ln:un:sn [, mask] ) &
      a(e1,...,em) = rhs
\end{verbatim}

\noindent
is equivalent to the following standard Fortran 90 code:

\begin{verbatim}
!evaluate subscript and stride expressions in any order
templ1 = l1
tempu1 = u1
temps1 = s1
templ2 = l2
tempu2 = u2
temps2 = s2
  ...
templn = ln
tempun = un
tempsn = sn

!then evaluate the scalar mask expression
DO v1=templ1,tempu1,temps1
  DO v2=templ2,tempu2,temps2
    ...
      DO vn=templn,tempun,tempsn
        tempmask(v1,v2,...,vn) = mask
      END DO
	  ...
  END DO
END DO

!then evaluate the expr in the forall-assignment 
!(and lhs subscripts) 
!for all valid combinations of subscript names 
!for which the scalar mask expression is true
DO v1=templ1,tempu1,temps1
  DO v2=templ2,tempu2,temps2
    ...
      DO vn=templn,tempun,tempsn
        IF (tempmask(v1,v2,...,vn)) THEN
          !in any order
          tempe1(v1,v2,...,vn) = e1
            ...
          tempem(v1,v2,...,vn) = em
          temprhs(v1,v2,...,vn) = rhs
        END IF
      END DO
	  ...
  END DO
END DO

!then perform the assignment of these values to 
!the corresponding elements of the array being assigned to
DO v1=templ1,tempu1,temps1
  DO v2=templ2,tempu2,temps2
    ...
      DO vn=templn,tempun,tempsn
        IF (tempmask(v1,v2,...,vn)) THEN
          a(tempe1(v1,v2,...,vn),...,tempem(v1,v2,...,vn)) = temprhs(v1,v2,...,vn)
        END IF
      END DO
	  ...
  END DO
END DO
\end{verbatim}

\subsubsection{Consequences of the Definition of the FORALL Statement}

\begin{itemize}

\item The {\it scalar-mask-expr} may depend on the {\it subscript-name}s.

\item Each of the {\it subscript-name}s must appear within the
subscript expression(s) on the left-hand-side.  (This is a syntactic
consequence of the semantic rule that no two execution instances of the
body may assign to the same array element.)

\item Subscripts on the left hand side of a {\it forall-assignment} are
evaluated only for valid combinations of subscript names for which the
scalar mask expression is true.

\item The evaluation of expressions within {\it array-element} or {\it
array-section} must neither affect nor be affected by the evaluation of {\it expr}.

\end{itemize}

\subsection{FORALL Construct}

The FORALL construct is a generalization of the element array
assignment statement allowing multiple {\it forall-assignment}s to be
controlled by a single {\it forall-triplet-spec-list}.  Rule R215 for
{\it executable-construct} is extended to include the {\it forall-construct}.\\

\subsubsection{General Form of the FORALL Construct}

\begin{verbatim}
forall-stmt          is FORALL (forall-triplet-spec-list [ ,scalar-mask-expr ])
                          forall-body-stmt-list
                        END FORALL

forall-body-stmt     is forall-assignment

forall-triplet-spec  is subscript-name = subscript : subscript [ : stride]
\end{verbatim}

\noindent
Constraint:  {\it subscript-name} must be a {\it scalar-name} of type integer.\\

\noindent
Constraint:  A {\it subscript} or a {\it stride} in a {\it
forall-triplet-spec} must not contain a reference to any {\it
subscript-name} in the {\it forall-triplet-spec-list}.

\begin{verbatim}
forall-assignment    is array-element = expr
                     or array-section = expr
\end{verbatim}

\noindent
Constraint:  The {\it array-section} or {\it array-element} in a {\it
forall-assignment} must reference all of the {\it forall-triplet-spec
subscript-names}.

For each subscript name in the {\it forall-assignment}s, the set of
permitted values is determined on entry to the construct and is
\[  m1 + (k-1) * m3, where~k = 1, 2, ..., INT((m2 - m1 + m3) / m3)  \]

\noindent
and where {\it m1}, {\it m2}, and {\it m3} are the values of the first
subscript, the second subscript, and the stride respectively in the
{\it forall-triplet-spec}.  If {\it stride} is missing, it is as if it
were present with a value of the integer 1.  The expression {\it
stride} must not have the value 0.  If for some subscript name 
$INT((m2 -m1 + m3) / m3) \leq 0$, the {\it forall-assignment}s are not executed.\\

Examples of the FORALL construct are:

\begin{verbatim}
 t o  b e  d o n e
\end{verbatim}

\subsubsection{Interpretation of the FORALL Construct}  

Execution of a FORALL construct consists of the evaluation in any order
of the subscript and stride expressions in the {\it
forall-triplet-spec-list}, the evaluation of the scalar mask
expression, and the evaluation of the {\it forall-assignment}s in
order.  If the scalar mask expresion is omitted, it is as if it were
present with the value true.

The evaluation of a {\it forall-assignment} consists of the evaluation
of the expr in the {\it forall-assignment} for all valid combinations
of subscript names for which the scalar mask expression is true,
followed by the assignment of these values to the corresponding
elements of the array being assigned to.

A {\it forall-assignment} must not cause any element of the array being
assigned to be assigned a value more than once.  The scope of the
subscript name is the FORALL statement itself. A function reference
appearing in any expression in a {\it forall-assignment} must not
redefine any subscript name.

\subsubsection{Scalarization of the FORALL Construct}

A {it forall-construct} of the general form:

\begin{verbatim}
FORALL (... e1 ... e2 ... en ...)
    s1
    s2
     ...
    sn
END FORALL
\end{verbatim}

\noindent
is equivalent to the following scalar code:

\begin{verbatim}
temp1 = e1
temp2 = e2
 ...
tempn = en
FORALL (... temp1 ... temp2 ... tempn ...) s1
FORALL (... temp1 ... temp2 ... tempn ...) s2
   ...
FORALL (... temp1 ... temp2 ... tempn ...) sn
\end{verbatim}

\subsubsection{Consequences of the Definition of the FORALL Construct}

\begin{itemize}

\item A block FORALL means exactly the same as replicating the FORALL
header in front of each array assignment statement in the block, except
that any expressions in the FORALL header are evaluated only once,
rather than being re-evaluated before each of the statements in the body.

\item One may think of a block FORALL as synchronizing twice per
contained statement: once after handling the rhs and other expressions
but before performing assignments, and once after all assignments have
been performed but before commencing the next statement.  (In practice,
appropriate dependence analysis will often permit the compiler to
eliminate unnecessary synchronizations.)

\item The {\it scalar-mask-expr} may depend on the {\it subscript-name}s.

\item Each of the {\it subscript-name}s must appear within the
subscript expression(s) on the left-hand-side.  (This is a syntactic
consequence of the semantic rule that no two execution instances of the
body may assign to the same array element.)

\item Subscripts on the left hand side of a {\it forall-assignment} are
evaluated only for valid combinations of subscript names for which the
scalar mask expression is true.

\end{itemize}

\subsection{Extending the Definition of forall-body-stmt}

The motivation here is to make it easier to transform nested DO loops
into FORALL constructs and vice versa, despite the fact that they may
contain conditional statements and non-tightly nested sub-loops.  

{\bf !!MORE WORK NEEDED HERE!!}

\subsubsection{IF Statements as forall-body-stmts}

Assume that a block IF is first reduced to single IF statements by
introducing temporary variables and replicating the IF header:

\begin{verbatim}
IF (m) THEN
  s1
    ...
  sm
ELSE
  t1
    ...
  tn
END IF
\end{verbatim}

\noindent
becomes

\begin{verbatim}
tempm = m
IF (tempm) s1
  ...
IF (tempm) sm
IF (.NOT. tempm) t1
  ...
IF (.NOT. tempm) tn
\end{verbatim}

Then we simply define

\begin{verbatim}
FORALL (v1=l1:u1:s1,...,vn=ln:un:sn,fmask) IF (imask) s
\end{verbatim}

to mean

\begin{verbatim}
FORALL (v1=l1:u1:s1,...,vn=ln:un:sn,fmask .AND. imask) s
\end{verbatim}

\subsubsection{WHERE Statements as forall-body-stmts}

WHERE can be treated similarly, though the details are a bit more complicated.

\subsubsection{FORALL Statements as forall-body-stmts}

\begin{verbatim}
FORALL (va1=...,...,van=...,maska)  &
  FORALL (vb1=...,...,vbn=...,maskb) s
\end{verbatim}

\noindent
means

\begin{verbatim}
FORALL (va1=...,...,van=...,vb1=...,...,vbn=...,maska.AND.maskb) s
\end{verbatim}

\noindent
assuming there is no duplication of the variable names (a compiler
should logically rename the variables if there is a duplication).

\end{document}