[lug] q: multithreading on smp weirdness

[karheng at softhome.net]

greetings. 

i've got a test app with 2 threads processing
several chunks of data. thread 1 fetches chunk 1
& processes it. once done, it's relayed to thread 2.
then thread 1 fetches & processes chunk 2, while
thread 2 receives & processes chunk 1. this goes
on until all chunks are processed by both threads. 

my problem is that if the chunk size is made small
enough, the elapse time suddenly almost doubles...
and i need these chunks to be small eventually. 

more specifically:
(chunk size * iteration count always == 6000)
single threaded:
chunk size of 6000 unit, 1 iteration, 12+ secs elapse time
multi-threaded:
chunk size of 12 unit, 500 iterations, 6+ secs elapse time
chunk size of 10 unit, 600 iterations, 6+ secs elapse time
chunk size of 8 unit, 750 iterations, 6++ secs elapse time
chunk size of 6 unit, 1000 iterations, 6++ secs elapse time most of the time
chunk size of 5 unit, 1200 iterations, 6+++ secs elapse time most of the 
time, 11+ secs at other times.
chunk size of 4 unit, 1500 iterations, 11+ secs elapse time
chunk size of 3 unit, 2000 iterations, 11++ secs elapse time
chunk size of 2 unit, 3000 iterations, 11+++ secs elapse time
chunk size of 1 unit, 6000 iterations, 12+ secs elapse time
(note that in above, elapse times of 7, 8, 9 & 10 secs are
mostly not present, just 1 or 2 sporadic ones). 

i ran this on a 4 CPU itanium linux server. 

this problem doesn't appear when i ran a port
of this app on a windows hyperthreading machine. 

anyone might have any idea what's wrong?
thanks in advance. 

rgds, 

kh