[lug] software engineering

Wed Nov 15 00:17:37 MST 2006

On Mon, Nov 13, 2006 at 01:58:53PM -0700, Nate Duehr wrote:
>>Software simply doesn't go through anything close to this level of 
>>testing or anything many orders of magnitude close to it.

Sure it is, just not ALL of it.  In applications where loss of equipment or
life is at risk, it's not uncommon for the development process to involve
track bugs in the low bugs per million lines of code.  Where much
commercial software is probably more on the low hundreds or thousands of
lines of code.

Part of being a professional is selecting the appropriate level of
complexity and safety.  My grandfather sketched the plans for the house he
spent most of his life in on an 8.5x11 sheet of paper.  When he built the
church or did the renovation on the mall, he went through a much more
intensive process...

Building software is a very different thing than building a large bridge or
building...  Sure, software failures are a huge annoyance to many people...
But, do you really want to have to secure 75 million dollars in funding
before you can start work on that open source project like a word processor
or mail reader?  I pick $75m, because that's what it cost to build the
golden gate bridge back in the 1930s...

Again, part of being a professional is to select the appropriate level of
engineering.  The brackets that hold the power strips to the cabinets at
our facility weren't engineered the same way as the parts on space-craft,
even though they were both built by the same person, on the same
equipment...

>In contrast -- In the computer fields, the best sysadmins want the hell 
>away from testing if at all possible, knowing that it's likely the whole 

I'm not sure that's really fair to say.  A lot of the best sys admins I
know *LOVE* messing around with new technology.  They just don't do it in
production.  An experimental plane, flown by a test pilot, doesn't have a
bunch of "test passengers" filling the passenger seats.  Well, it does, but
they aren't Chuck Yeager, they are "Buster" from "Myth Busters"...

I have a story that I think illustrates this particularly well.

Several years back, the Northern Colorado Internet Cooperative was
switching to running multi-homed with multiple upstream Internet
connections for redundancy.  To do this, they needed to run BGP to handle
the routing across multiple lines.

We had two options, one was to get a fairly powerful Cisco router, the
other was to use a Linux machine running an open source software suite for
doing BGP routing.  BGP is kind of an odd duck, because it puts fairly
heavy RAM requirements on the router it runs on, hence needing a fairly
beefy Cisco.  Network throughput wasn't that heavy, but we needed at least
a quarter GB of RAM.

We discussed setting up a Linux machine to do it, but one of the guys kept
pushing FUD that none of our providers would support us if we weren't using
Cisco, etc...  Of course, now that I think about it, he was a Cisco
reseller.

Anyway, it boiled down to the fact that for half the price of a Cisco, we
could buy a pair of beefy PCs, with tons of RAM, and run redundant routers
or at least have a hot spare on hand, test it out, and if it didn't work
out we could use them to replace some other aging Linux machines.

I set up the Linux machine in a test environment and ran it for the better
part of a month before switching production traffic over to it.

So, sure, a lot of production sys admins may be more like an airline pilot
than a test pilot.  But there are a lot of software test pilots behind the
scenes that have gotten things to the point that the production sys admins
can conduct a safe, boring, take-off and later collision with a planet safe
and boring.

So, back to the original question here -- why aren't the "software test
pilots" superstars?  Often they are...  Our Chuck Yegar is is a Finnish
guy named Linus, and he pronounces Linux as Linux.

But more down to earth, a lot of the best system admins I know are doing
all sorts of experimentation outside of production.

Sean
-- 
 Canadian phone sex: What kind of hockey jersey are you wearing?
Sean Reifschneider, Member of Technical Staff <jafo at tummy.com>
tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability
      Back off man. I'm a scientist.   http://HackingSociety.org/