Re: Questions involving main<->PCI bus interactions.

[kevin.normoyle@Eng.Sun.COM (Kevin Normoyle)]

randy says:

> 	*) Do accesses from an I/O processor on the PCI bus to main
> 	   memory respect the main (application) processors cache?

> 	*) If you think of a processor on the system bus and the
> 	   processor on the PCI bus in an SMP context (i.e. if you are
> 	   running programs on both of them and communicating through
> 	   memory), what is the consistency model of the interaction
> 	   between the two?  Can it be described by any of the
> 	   standard terms (sequentially consistent, processor
> 	   consistent, or weakly consistent)?  I think Appendix E
> 	   attempts to explain this, but I had a hard time translating
> 	   it into concepts that map well to an SMP programming model.

> 	*) If a processor on the PCI bus does a word (32 bit) access
> 	   to main memory, is that access atomic WRT accesses by the
> 	   applications processor?  What's the largest access that is
> 	   atomic?  (I suspect the answer to this question is either
> 	   "aligned doubleword (64 bits)" or "byte").
> 

My interpretations/opinions:

Although PCI transactions to main memory may update or invalidate caches, or
return data from caches, there is no global ordering model for ALL loads and
stores in a system, except possibly implementation-specific rules/contracts.

The PCI model only describes ordering between a single producer and a single 
consumer, with the shared address space located at a possible third party.

If two agents are both producing and consuming, there is no ordering model
for the event flows with respect to one another
(although there may be an implemenation specific one)

And there is no definition of the ordering of cacheable and noncacheable 
load/stores by a processor, with respect to processor PCI activity. 
This is processor/system specific definition.

However, the spec doesn't seem to require an global order to be seen by 
multiple readers for the events created by either a single writer or 
multiple writers.

This is a feature in that it allows bus hierachies to be created more easily 
than traditional SMP ordering models. However it's interesting in its
uniqueness, in that most memory models are not defined only for 1 or 2 
parties sharing access to an address space, but for N.

It seems to make the ability to use a single semaphore for controlled
shared access to a common resouce not possible. 

This is odd, because LOCK is supported, although deprecated in the 2.1 spec
language, with (to me) a lack of clarity on it's operation across bridges.

Most memory models don't talk about LOCK protocols, but instead talk about
atomic operations. An atomic read/write protocol is not explicity stated, other
than for LOCK support, in my understanding of the 2.1 spec.

I think the language for a multi-processor memory model that has
the strength of the language for memory models of typical SMPs is lacking.


The other weird part is that there is "Cache Support" in the protocol,
which is basically snooping support. But there are apparently no extensions
to the memory model to fully flesh out this protocol or support, and 
it isn't widely used or required anyhow.


Now of cource, the PCI protocol on a single bus, in the absence of
bus bridges, creates some defacto ordering due it's circuit switched
nature. I suspect some systems work and depend on the side-effects of
this, rather than any spec'ed behavior.

-kevin
Ùxf