Subject: COMMON with sequence and storage assoc

From: [email-masked] (Andy Meltzer)
Date: Thu, 30 Apr 92 17:44:20 CDT

As promised, what follows is a suggestion for an extension to the 
hpff-f90 group's proposal.  This extension deals exclusively with
distributing COMMON blocks while maintaining sequence and storage
association.

I'm working on getting some example codes which will find this
extension particularly useful.  I will probably not be able to
post them for some weeks yet, however.

#############################################

COMMON blocks with Sequence and Storage Association

Suggestion for extension to group 1 (hpff-F90) Proposal

Currently in the group 1 proposal, COMMON blocks are not allowed to 
both maintain sequence and storage association and be explicitely
distributed by the user.  This extension allows the user to specify
the distribution of a COMMON block with sequence and storage 
association.

DEFINITIONS:

    As is currently proposed:

  	"A sequential COMMON block is any COMMON block which has the
        properties of storage association and sequence association."

        "A distributable COMMON block is any COMMON block which does not have
        the properties of storage association or sequence association."

    These definitions combine two concepts into one (they are used as 
    antonyms in the proposal.)   In a program there may be the assertion 
    by the user that sequence and storage association must be maintained, 
    this is a correctness issue.   The user may also wish to guide the 
    compiler as to his/her choice of how to distribute the COMMON block;
    this is a performance issue.

    I will use the directive "MSASA" to indicate that the user wants to
    Maintain Sequence And Storage Association, and "NO MSASA" as its 
    antonym. 

    A common block may be mapped across the processors using the same
    syntax as an array which is mapped across the processors (more on
    this later.)

RATIONALE:

    There are many reasons why a user may desire to map a COMMON block
    across the memories of a machine and still maintain sequence and
    storage association.  I will enumerate a few of them here. Note that
    in all cases where I refer to COMMON blocks I am speaking of COMMON
    blocks which require sequence and storage association (MSASA COMMON).

    1. Porting old codes.

       Many user's have codes which they want to port to MPPs.  These
       codes currently make extensive use of COMMON blocks for which
       sequence and storage association are heavily relied upon.

       While it is not my opinion that a user will get optimal performance
       simply be distributing these COMMON blocks, allowing them to do
       so will give them a faster path to parallelism and on many machines
       allow them to process their distributed data in parallel. Although 
       their processing will be on predominantly non-local references, it 
       will, nontheless, be in parallel.  On many machines, being able to
       do 1000 things at a time, even with the overhead of non-local data
       accesses will be considerably faster than sequentially processing 
       everything.  

       For those COMMON blocks where this is the case the user should be
       allowed to specify the appropriate mapping.  
       
       Where an implementation does not feel that a particular layout is
       appropriate, it may be ignored.

    2. Replicated Storage.
    
       On shared memory machines some COMMON blocks may be more ideally 
       suited to being replicated across processors.  While occasionally
       a compiler may be able to determine this, without interprocedural
       analysis it is a difficult optimization.  

       If the data in COMMON block is rarely updated, the performance
       of an application may benefit greatly by having local copies to 
       refer to.  The user will generally be able to determine this and
       should be able to tell the compiler.

    3. Memory availability.

       We have a number of customers who are concerned that their codes
       will not fit within any single node's memory.  This need of the 
       users must be met.

    4. HPF user flexibility.

       We as language designers cannot know all of the ways which our 
       users will conceive of using HPF, we should therefore not try
       to force restrictions upon them.   As far as I have seen,
       sophisticated users of languages for MPPs have spent a large
       amount of effort trying to get around weaknesses and restrictions
       in languages.  To purposefully put one of these in their way 
       seem counter-productive.

    5. Ease of understanding.
    
       A language is simpler to understand if each directive embodies only 
       one concept.  

       Whether or not we decide that a COMMON block may have its distribution
       specified by the user, differentiating between the concepts of 
       maintaining sequence and storage association, and the distribution
       of a COMMON block makes the directives clearer.  It also allows a
       pathway for allowing the feature in the future.

    The main arguments against allowing users to specify a distribution for a 
    COMMON block with sequence and storage association is that it is of
    questionable use and that it will inevitable lead to poor performance.
    
    Its usefulness should be determined by the user community.  It is not hard
    to implement.  It can simply be ignored by those vendors who feel that 
    users will not take advantage of it.  It is a porting aid.
    
    While there are many cases where mapping COMMON blocks across processors 
    is a bad idea, it will not always reduce performance.  As point (2) above
    states, replication may be a good idea.  As point (1) states, there are
    codes for which it will provide speedup.

SYNTAX:

  - The syntax of distributing MSASA COMMON is identical to the syntax of 
    distributing other COMMON blocks where applicable.  Restrictions are 
    listed in the semantics section.

  - The syntax for ALIGNing an MSASA COMMON block is identical to the syntax 
    for ALIGNing an ordinary COMMON.

  - An additional keyword is added "MSASA" and its inverse "NOMSASA"
    to indicate that sequence and storage association is required.  
    NOMSASA is the default for COMMON blocks which are explicitely
    distributed or replicated. MSASA is the default for other COMMON 
    blocks.

	msasa-directive		is MSASA [::] msasa-list

	msasa-item		is /common-block-name/

        example: 
		COMMON /XXX/ A(40), B(50,60), C
		COMMON /YYY/ D(40), E(50,60)
    	CHPF$	MSASA:: /XXX/, /YYY/

  - Since one cannot specify a template for the distribution, an additional
    keyword REPLICATE is added.  It is used in place of the keyword
    DISTRIBUTE.  With REPLICATE no blocking attribute can be specified.

	replicate-directive 	is REPLICATE repl-stuff

	repl_stuff		is single-repl-stuff
				or fancy-repl-stuff

	fancy-repl-stuff	is fancy-attr-stuff :: msasa-list

	fancy-attr-stuff	is [ONTO repl-target]

	single-repl-stuff	is msasa-item [ONTO repl-target]

        repl-target		is processors-name

	example:

	CHPF$	PROCESSORS P(256)
		COMMON /XXX/ A(40), B(50,60), C
		COMMON /YYY/ D(40), E(50,60)
		COMMON /ZZZ/ F(10), G(50)
    	CHPF$	MSASA:: /XXX/, /YYY/
	CHPF$	REPLICATE ONTO P :: /XXX/, /YYY/
	CHPF$	REPLICATE /ZZZ/ ONTO P

SEMANTICS:

  There are not many rules necessary for MSASA COMMON blocks.  They are
  intended to behave much like ordinary COMMON blocks.  The semantic
  restrictions are only in place to prevent the programmer from trying
  to ask for nonsense.   While this approach may lead to some poorly 
  performing programs (in the case, for example, of a COMPLEX number 
  crossing PE boundaries,) it leaves the user with a clean simple set
  of guidelines.

  - An MSASA COMMON block cannot be ALIGNed to any template other than 
    its own (implicitely) or the template of an MSASA COMMON block of equal 
    or larger size.   
    
    While aligning MSASA COMMON with these restrictions appears to be 
    meaningless, it is not.  There may be implementations which choose not to 
    automatically map all objects to the hardware in an identical fashion.  

	example:

	CHPF$	PROCESSORS P(256)
		COMMON /XXX/ A(40), B(50,60), C
		COMMON /YYY/ D(40), E(50,60)
    	CHPF$	MSASA:: /XXX/, /YYY/
        CHPF$ 	DISTRIBUTE /XXX/ ONTO P
	CHPF$	ALIGN /YYY/ WITH /XXX/
	C The inverse alignment would be illegal

  - A BLANK COMMON cannot be specified as MSASA. HPF may decide
    that its default is MSASA, however the decision is outside the 
    scope of this proposal.

  - No data item can be aligned with a data item in an MSASA COMMON block
    nor can it be aligned with the MSASA COMMON block.
    
  - No data item from an MSASA COMMON may be separately ALIGNED or DISTRIBUTED.

  - An MSASA COMMON block may not be redistributed or realigned.

  - Correctness of implementation is all that must be maintained.  This
    directive (as is true of many others) may be considered a suggestion
    by an HPF conforming implementation.

open issue:  It makes sense to allow the user to specify SMOOTH, CYCLIC,
	     or BLOCK.  Does it make sense to allow the user to specify
	     BLOCK sizes since these may vary from machine to machine?

	     In my opinion (at this moment) this should be allowed, but 
	     I am not yet confident of this.

							Andy Meltzer

---

From: Richard Shapiro <[email-masked]>
Date: Tue, 5 May 92 16:02:51 EDT

I've been doing some thinking (well, philosophizing really) about the
larger issues in the discussions in sequence and storage order debate (as
well as the F90 debate).  I've come to the conclusion that HPF should
specify only a few things that *everyone* should have (for example, the
minimal subset of Fortran 90 + some form of FORALL), and the rest should be
of the flavor "If you have functionality X, do it this way" I think this
will make it a lot easier to agree on some things (for example, Andy's
suggestions about COMMON blocks and storage association), while at the same
time maintaining a common syntax and semantics for more architecture and
vendor specific features.

I propose that we try to make the directives have as much functionality as
we want (including, perhaps, some fairly specific functionality), while
trying to preserve a relatively easy-to-understand model. Them we simply
say to any implementor that they can implement as much or as little of the
functionality as they want.

The gist of this comment is that I don't think we should spend a lot of
time debating the merits of allowing users to distribute COMMON blocks or
pointer targets. Rather, we should specify what it means to do so and let
the vendors decide whether they want to implement a feature.

On Andy's proposal itself: I don't like the name MSASA. It's useful for
discussion, but without context it's not obvious what it means. I'd prefer
something like "LINEARMODEL" or "STORAGE_ASSOCIATE".  My feeling is that no
restrictions should be placed on distribution templates, so I should be
able to align my entire COMMON block with a template of the appropriate
size.  What the "storage unit" means should be left up to the vendor. For
example, on a Cray, the storage unit might be 8 bytes, while on a CM-2, it
might be 4 bytes.  Users already have to deal with this kind of issue in
F77 codes, and this doesn't seem any worse than what we already have.