[lug] Web crawler advice

[Nate Duehr]

George Sexton wrote:

> OTOH, threading in Java is dead easy, and this kind of app would benefit 
> from multi-threading.

Dead-easy until it blows up.  :-)

http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.html

Threading is starting to get as bad a rap as "goto" got in the 70s.

The author specifically talks about how subtle problems will crop up on 
multi-core machines, especially.

Had something similar lately.  The symptom was that Perl wouldn't start 
on a 4-processor Sun box.

Perl (for some UNHOLY reason) uses floating-point math to compare the 
main perl version number with the version numbers in any modules it 
loads at run-time.

What had happened was that the FPU in CPU #3 on the box was flaky. 
Since it was running very little else that required floating-point 
calculations, the only "symptom" was, "Perl won't run consistently, or 
dies halfway through scripts!"  (The scripts that were dying were 
loading more modules.

Frackin' ugly troubleshooting session that was... until we "caught" the 
FPU doing naughty things with Sun's hardware test tools.

I shudder to think how long that would have taken on PeeCee hardware 
where such test tools simply don't (really) exist on most hardware/OS 
combinations.

Nate