Subject: Replication, etc.

From: Rob Schreiber <[email-masked]>
Date: Tue, 14 Jul 92 17:23:54 PDT

Topics:  
1.       Distributing COMMON is not necessary.
2.       Replication belongs in distribute.
3.       No sequential variables in alignments.
4.       Wrapping twice is a bad idea.

At the last meeting, we more or less agreed with Andy Meltzer to allow
COMMON blocks to appear in directives.   Our goal is to allow
the user to distribute sequential COMMON as if it were a one-dimensional
array.   THe user might want to replicate the COMMON on all processors,
for example.

Ken Kennedy also argued successfully for a rule that breaks up a COMMON
into the sequential aggregates created by EQUIVALENCE and the non-storage-associated
variables not touched by EQUIVALENCE, allowing the latter to be distributed
freely.

Ken and I have drafted a proposal along those lines.  This effort has revealed
a few things that seem important to me.

1.   We dont need to distribute COMMON blocks.   We can get the same
effect by distributing a local that is equivalenced in such a way that it
"covers" the COMMON block.   This is an big simplification.  COMMON
blocks may not occur in mapping directives, only variables, templates, and
processor arrangements.

2.   In the current draft on directives, replication is allowed in (RE)ALIGN
but not in (RE)DISTRIBUTE directives.   This is a mistake for two reasons.

   a.   What if I want ONE COPY PER PROCESSOR?   Replicating onto a template
   is a funny way to express this.

   b.   Replicating a sequential part (or all of) a COMMON block CANNOT BE DONE
   using (RE)ALIGN because of (3) below.

3.   We are proposing to have sequential variables, and to allow them to be used
in some directives.   If I ALIGN a sequential variable to a template, the
effect will be to make the template sequential, too!   This is *not* desirable.

Thus, we should make the rule:   Sequential variables may not occur in
(RE)ALIGN(ABLE) directives.    If one-dimensional, they may occur in
(RE)DISTRIBUTE(ABLE) directives.   And we should allow replication in the
(RE)DISTRIBUTE directives by some appropriate syntax.

4.   We also squeezed wraparound in alignment into the system.  There
is already wraparound in the CYCLIC(k) distributions.   Now we can get
double wraparound mappings like this:

       real X(24)
       template T(10)
       processors P(4)

       ALIGN X with T                        (wraps twice)
       DISTRIBUTE CYCLIC(2) ONTO X :: T      (wraps again)

Yields:

     Processor: 	1	|       2	|       3	|       4
				|               |               |
				|               |               |
     T Indices:	    	1 2	|       3 4	|       5 6	|       7 8
			9 10    |               |               |
				|               |               |
				|               |               |
     X Indices:		1 2	|       3 4	|       5 6	|       7 8
			9 10	|               |               |
			11 12	|       13 14	|       15 16	|       17 18
			19 20   |               |               |
			21 22	|       23 24   |               |

****    DO WE WANT THIS?    ****

---

From: [email-masked]
Date: Fri, 17 Jul 92 16:10:53 -0400

Date: Tue, 14 Jul 92 17:23:54 PDT
  From: Rob Schreiber <[email-masked]>

<material deleted>

Well, I think it was always the intention of replication
specifications in (RE)ALIGN directives to yield only one copy per
processor.  The intuition is that you are saying in your (RE)ALIGN
directive that you want a copy of the replicated datum available
cheaply at each template point specified (presumably because you use
the replicated datum in operations involving other data aligned to
each of those template points).  However, if the distribution in use
maps all of the template points over which a datum is replicated onto
the same processor, then in fact only one copy of the "replicated"
datum will be allocated.  And in the more general case, there will
only be one copy of the replicated datum for each processor onto which
one or more of those template points is mapped.

This is the intended semantics of replicated alignments, and in my
opinion it is the only useful one -- why you would ever want more than
one replica of the same datum on the same processor is beyond me!  If
the current specifications don't make these intended semantics clear
enough, maybe they need to be revised...

Just a quick question: what is a "sequential template" and why don't I
want to have one?  In other words, what am I allowed to do with other
kinds of templates that I'm not allowed to do with sequential
templates?  I don't understand the answers to these questions, and
without understanding them, I don't understand why it's undesirable to
allow the kind of alignment that you propose to outlaw.

Well, this sounds like something that you'll have to work out in
determining the rules for using "sequential" variables.  But if you
have to introduce another concept to handle this situation, there is
probably more than one solution to the problem of "contaminating"
templates by aligning sequential variables to them.  The details will
certainly depend on the goals you are trying to achieve, but instead
of _distributing_ sequential variables directly onto the processors
(and introducing the concept of replicated distributions) it might be
just as simple to introduce the concept of an "anonymous" or
"sequential" template that has the same size as the array of
processors, to which your sequential variables are _aligned_ (making
available the existing syntax for specifying replicated alignments).

In any case, my argument is that if you introduce a concept of
replicated distributions, you will only be doing it to deal with cases
2b and 3.  I disagree that it's needed for your case 2a.

  4.   We also squeezed wraparound in alignment into the system.  There
  is already wraparound in the CYCLIC(k) distributions.   Now we can get
  double wraparound mappings like this: ...

Your example strikes me as the most cogent argument yet against
wraparound in alignment directives.  When there are truncation effects
that leave more data elements (resp. template points) on some template
points (resp. processors) than on others, because the wrap-around
didn't cover the whole space evenly, your example makes a pretty
convincing case that it's better to have the truncation effects at one
level only, rather than compounding the unevennesses created by
composing two truncations!  Is there an equally compelling case *in
favor* of allowing wraparound alignments?	    -Bert Halstead