empirical engineering discussion (was: Re: [lug] Memory Leak Detection)

[Nate Duehr]

Timothy Klein wrote:

>> Heh, funny that you used the phrase "poke the machines".
>> A friend and I were chatting about "empirical engineering" just a few 
>> minutes ago, and we decided the best definition of that we could come 
>> up with was "Poke at it until you fix it and then stop poking."  LOL!
>
>
> I'm going to be a pedant and say that all engineering is empirical by 
> default.  Or am I missing some form of non-empirical engineering?

Ahh... we were specifically discussing engineers that have no connection 
(either by choice or by corporate layers) to the real world systems they 
deploy... so the empirical feedback loop isn't closed, and they forever 
release real crud.

The cliche' is:  Any engineer who's always done all his/her work on 
paper* and never deployed anything to the "real" world, or written code 
to sell that they had no intention on ever testing in enough scenarios 
to simulate the "real" world correctly... i.e. Type in code, run once, 
check-in to revision control, release.

* Today it's usually Visio diagrams.

There's plenty of engineers in telecommunications that never set foot in 
a Central Office and their only input to their development process is 
from trouble tickets escatlated slowly up through three or four layers 
of people.  They don't get much chance to be "empirical".  Maybe at a 
microscopic level to fix a particular bug report, but they have 
virtually zero idea what really happens out where the equpiment actually 
gets used until the inevitable escalations happen after a deployment.

A case in point that I can probably get away with without causing myself 
too much trouble: Whoever the hardware engineer is on the three products 
the company I work for (products released over 10 years time) has never 
worked in a Central Office installing one. 

How do I know?  The screws are smaller than a #1, they're non-Phillips 
and the only good thing they did was make them captive so they can't 
fall out.  But they should be a larger size and Phillips... and should 
have been for the past two products. 

But they keep using the tiny little captive ones, and ironically our 
customer support keeps buying and giving away tiny little screwdrivers 
to customers because they're always hunting for screwdrivers small 
enough. Pretty little screwdrivers with the company logo(s) over the 
years has been the "fix" for this problem.  Heh.

The "empirical engineers" in the group take extra screwdrivers along 
with them and a long string and TIE THEM TO THE EQUIPMENT.  Heh... not 
the "best" solution, but at least the customer has the tool at hand when 
they need it. 

If the guy who did the original design had any way to close his 
"empirical" loop, and see how much grief the tiny little screws cause, 
he'd surely redesign the faceplate and mounting mechanism in the next 
product!  :-)   (Worse, this cascades into other engineering small but 
significant screw-ups -- no pun intended! -- like I don't think I've 
ever seen a way to report a "bug" against the HARDWARE... most engineers 
building trouble ticketing systems don't think about that!  I'm sure 
it's in there somewhere, but the number of software tickets vs. number 
of hardware tickets is probably 99 to 1, and the process isn't much 
used... because it's a foreign idea... a bug in hardware?)

It's another example of a place where if someone could close their 
"empirical" loop by sending a note... ("Hey where do I put a defect in 
against the screw not being big enough?" -- again, no pun intended.  
Hah!) the next revision might be more intelligent about such things.

The fact that it's easier to "close the loop" on open-source software 
(usually), is one of the reasons I like it far more than closed 
software, where it's virtually impossible to get anything but highly 
filtered feedback back to the original engineer/hack that wrote it in 
most companies. 

Open-source developers don't LIKE the way it feels to get a flame-mail 
saying, "Man does your software SUCK!", but it is a better "closing of 
the loop" than a whitewashed trouble ticket with politically correct 
phrasing that's been shared with four managers on spreadsheets and other 
documents before the poor engineer has ever seen it!  :-)

Some of the stuff I've read about how psycho Steve Jobs has been (and 
probably still is) at Apple seems to relate to this... he bursts out 
into (some say drug-induced) rages at his engineers when something isn't 
"beautiful" or "it sucks".  It's very very rare for engineers to get 
this type of EXCELLENT feedback loop closure.  (Of course, Steve 
apparently takes this way too far, but I bring it up to elicit an 
emotion from those that have read the stories.  Actually Steve's 
probably just an ass, but hey ... you all probably get the emotional 
idea here...)

These are some of the weird things I lie awake and think about some 
nights... how in the world do large companies with 20 layers of 
protection around their engineering talent ever survive and release 
things their customers actually want...?  I honestly don't know sometimes!

Nate