[lug] software engineering

[Nate Duehr]

Sean Reifschneider wrote:
> On Mon, Nov 13, 2006 at 01:58:53PM -0700, Nate Duehr wrote:
>>> Software simply doesn't go through anything close to this level of 
>>> testing or anything many orders of magnitude close to it.
> 
> Sure it is, just not ALL of it.  In applications where loss of equipment or
> life is at risk, it's not uncommon for the development process to involve
> track bugs in the low bugs per million lines of code.  Where much
> commercial software is probably more on the low hundreds or thousands of
> lines of code.

Okay, I like that answer... not ALL of it.  I think you probably have 
references hiding somewhere for those numbers too.  :-)

> Part of being a professional is selecting the appropriate level of
> complexity and safety.  My grandfather sketched the plans for the house he
> spent most of his life in on an 8.5x11 sheet of paper.  When he built the
> church or did the renovation on the mall, he went through a much more
> intensive process...

Okay so perhaps the problems that keep cropping up in Linux are a result 
of someone doing an "8 1/2 X 11" design on something that truly was far 
more important than they thought at first glance, and then no one going 
back later and doing the proper design work, perhaps?

> Building software is a very different thing than building a large bridge or
> building...  Sure, software failures are a huge annoyance to many people...
> But, do you really want to have to secure 75 million dollars in funding
> before you can start work on that open source project like a word processor
> or mail reader?  I pick $75m, because that's what it cost to build the
> golden gate bridge back in the 1930s...

Ahh, yeah -- you've hit something here, that leads to my 
curiousity-based question about underlying motivation.  No one's paid 
(or paid too little) to work on certain parts of Linux?

This possible solution to the dilemma falls apart a bit on the example 
from today, though... someone *is* paid to write the installer for 
Ubuntu, methinks.  Hmmm.

> Again, part of being a professional is to select the appropriate level of
> engineering.  The brackets that hold the power strips to the cabinets at
> our facility weren't engineered the same way as the parts on space-craft,
> even though they were both built by the same person, on the same
> equipment...

Okay, yeah - I get it.  Here's the rub... if the brackets kept falling 
off the cabinet you'd know NOT to use those brackets and there would be 
other options.  Folks on the other thread got emotional when I stated 
that it'd be easier to just try another distro and dump the baby out 
with the bathwater when the installer barfs.  But... if the installer's 
bad, there's a darn good possibility there are other serious bugs 
lurking, right?  Our installers are "our" first and best chance to put a 
good foot forward, and if the darn stuff won't even install -- that 
bodes very badly for Linux in general.  I truly haven't found an 
installer I really think works well on all hardware, etc... Ubuntu 
sounds (from the descriptions of others) like it's pretty nice... if you 
don't hang at the boot screen and can't get to a boot prompt.  :-)

>> In contrast -- In the computer fields, the best sysadmins want the hell 
>> away from testing if at all possible, knowing that it's likely the whole 
> 
> I'm not sure that's really fair to say.  A lot of the best sys admins I
> know *LOVE* messing around with new technology.  They just don't do it in
> production.  An experimental plane, flown by a test pilot, doesn't have a
> bunch of "test passengers" filling the passenger seats.  Well, it does, but
> they aren't Chuck Yeager, they are "Buster" from "Myth Busters"...

Okay, I'll admit it.  I love messing around with new technology.  I 
*hate* running into the same *problems* in that new technology as the 
last version of "new" technology.  I get this impression Linux is really 
bad at learning from its earlier mistakes?

> I have a story that I think illustrates this particularly well.
> 
> Several years back, the Northern Colorado Internet Cooperative was
> switching to running multi-homed with multiple upstream Internet
> connections for redundancy.  To do this, they needed to run BGP to handle
> the routing across multiple lines.

<snipped BGP router story>

I like it.  Nicely done!  I wish more companies would take "minor risks" 
like that one and gain from the technology instead of buying brand 
names.  I fall off the ends of two hands counting the times I've heard, 
"We're a Cisco shop.", like that's the only router in the world.  :-)

> So, sure, a lot of production sys admins may be more like an airline pilot
> than a test pilot.  But there are a lot of software test pilots behind the
> scenes that have gotten things to the point that the production sys admins
> can conduct a safe, boring, take-off and later collision with a planet safe
> and boring.

You think there are that many?  I think that's the mystery I'm trying to 
solve there... just how much testing REALLY goes into things?  I know 
I'll never get that answer from a commercial shop -- or only a 
half-assed one.  But I'm surprised that some Linux project somewhere 
hasn't attempted to document testing done to Linux's various components. 
  Would be interesting.  Lots of "consortiums" out there to come up with 
standards like LFS, etc... between organizations that pay people to work 
on Linux, but no standard testing benchmarks for the core "stuff". 
Layer 7, yes... benchmarks for databases, network I/O & throughput, 
etc... kinda a "proof is in the results" approach, but no good way to 
tell if the last 100 patches broke something major in an installer?  Do 
you kinda see where I'm headed here?

> So, back to the original question here -- why aren't the "software test
> pilots" superstars?  Often they are...  Our Chuck Yegar is is a Finnish
> guy named Linus, and he pronounces Linux as Linux.

Heh.  Nice.  But only one guy, and no army of testers wanting to take 
his spot?  There's still a motivation problem underlying all of this 
here, somewhere.  Mostly probably that software testing sucks and 
doesn't include much to inspire.  Flight testing on the other hand, is 
monotonous and boring too, but you get to see a sunrise over the Mohave 
once in a while.  :-)

> But more down to earth, a lot of the best system admins I know are doing
> all sorts of experimentation outside of production.

Okay, but is experimentation and goofing off really disciplined formal 
testing?  I get it... I do, but I'm asking where the next level is. 
Doesn't Linux *have* to go there, someday?  Or is it a lost cause, 
meaning there's not much hope for stupid installer bugs, ever?  Does the 
current situation define the next steps in growing up for Linux, or is 
it the end game?

Nate