[lug] software engineering

[David Morris]

On Sat, Dec 02, 2006 at 02:12:08AM -0700, Nate Duehr wrote:
> David Morris wrote:
> >On Mon, Nov 13, 2006 at 05:32:43PM -0700, Bear Giles wrote:
> >>The levels where it starts getting difficult?... they're
> >>management, not engineering. Don't get me wrong, it's
> >>usually still people on the engineering side of the house,
> >>but when you're trying to make sure 14 teams are all
> >>working on the same problem that's a management issue, not
> >>something you solve with your grandfather's slide rule.
> >
> >At the risk of going back to prehistory I'm so far behind in
> >reading my email........
> >
> >The concept you're striving for here is Systems Engineering,
> >a field newer than even software engineering.  At least as
> >an official engineering discipline, people have *done* it
> >for quite some time with varying degrees of success.
> 
> You think it's really that new?  Or was the role filled
> (even if they didn't have the title) in traditional
> "Engineering" organizations by Principal and Head
> Engineers with a lot of passion for the end-result?

Oh, the concept is not new at all....a document famous in
the Aerospace field written by NASA even mentions it was
probably Skyscrapers that first had a need for this type of
role.  I think all companies *do* systems engineering to
some extent, even if they don't realize it.  What is
relatively new is acceptance as a separate (and needed) role
on the project and how it _should_ fit into the project
organization to be most effective.  By formalizing the role,
consistent results can be achieved.

> >In short, the traditional engineer is focused primarily
> >on how something should be done, while a systems engineer
> >focuses more on the question of why.
> >
> >In the field of Aerospace (where I work), this is a vital
> >role because of the need to get everything right the
> >first time in an extremely complex system.  Aerospace
> >companies are still working out how systems engineering
> >works into the mix and just finding out that its a good
> >role to have around on small scale projects as well as
> >large.  I've worked on projects with good systems
> >engineering support and others with next to none and the
> >difference it makes is astronomical (sometimes literally,
> >in this industry! ;)
> 
> Who made those decisions before they had an officially
> titled "Systems Engineer"?  I'm thinking specifically
> about Aerospace here, since you mention it's your field...
> I've read a lot of things about Kelly Johnson's Lockheed
> Skunk-works, for example.  Was it handled by someone like
> Kelly himself back then because his "name was on the
> door"?

I think you hit on the answer yourself:  dedicated lead
engineers.  Their success in the role depended on the
quality of the rest of the team and their ability to
distance themselves from both details and management
concerns, combined with managements ability to realize the
effectiveness of what they are trying to do.  There are some
phenomenal success stories out there on this happening, but
not with any sort of repeatability.

> In software, we don't seem to have our act together yet --
> a "successful" software project typically only means the
> thing did some percentage of the desired result and
> manages to stay running to say, six-nines... that magical
> number that got a lot of marketing traction in the last
> few years.  Or it can mean something completely different.
> 
> Do you think it's possible to state in black and white a
> list of requirements that EVERY "successful" piece of
> software has to have?  I don't, and I think that's one of
> the dilemmas of software engineers. What's "successful"?
> If the software does everything that was required of it,
> but is a nightmare to maintain for the operations people
> -- is that successful?  Is it successful if a customer
> later complains that it's too hard to maintain?  Is the
> answer ALWAYS, "the next version is coming, welcome to our
> software treadmill?"

I think there absolutely can be a "black and white" list of
requirements a successful piece of software has to have.
Computer games are a perfect example of this, if you think
about it.

Most companies in the gaming industry focus on the "typical"
software model of release now, patch later.  A few, on the
other hand, release products that are so polished they might
never need a single patch.  The "Prince of Persia" (PoP)
line of games comes to mind (at least the recent three, I
don't know about the early ones).

I don't know how the PoP development team at Ubisoft is
structured, but of the three games they released, one
required no patches and the other two had only a single
patch which effected a tiny fraction of the users.  Patches
related, incidentally, to software/products which were
released very close the relevant PoP game release.

Three times is more than just coincidence.  Someone there
knows how to develop software the right way.  Unfortunately
it isn't general company policy as other games are not of
the same quality.

A bit of open source software which is universally rock
solid is VIM (VI-Improved).  It does have bugs, but I
certainly have never encountered one and I use it more than
any other piece of software installed on my computers at
home and work.

There clearly is a way to develop software and have it come
out right the first time, the process for *how* to do it
just isn't well known and/or accepted.  A company which can
do this reliably on large-scale software systems will 

> I think customers tend to hate the whole, "That will be
> fixed in the next version (which will break three other
> things and make your life complex because you need to
> figure out how to take the downtime necessary to upgrade"
> syndrome, but I don't see any way to get software
> engineering as we know it today beyond that stage.

That is a revolution which, I think, needs to happen from
the top down.  The software engineers *know* about the
problem and hate it.  Some even try and fix the problem. But
that clearly is not working.  Customers *know* about the
problem, but don't realize there is another alternative so
they aren't vocal enough to change company policies.

Most likely what will happen is some company will manage to
implement a system which provides bug-free releases.  They
will then get so much business that other companies will be
forced to follow their lead in order to compete.

Something I find absolutely fascinating is that Aerospace
has found a portion of the solution to this problem.

Satellite systems have bugs, software and hardware.  Not
only that, parts fail during the mission and must be worked
around.  The problem is, of course, that you can't walk up
to the satellite for maintenance, everything must be done
remotely.

The way the Aerospace industry has solved this problem is
*not* by eliminating bugs or building more reliable parts,
but by ensuring the fault recovery process is flawless.  A
huge amount of effort goes into this task.  The end result
is that a portion of the software *is* actually near perfect
the first time:  the fault management system.

Now, this focus can provide for a successful mission, but it
wastes a huge amount of money because it is more expensive
by at least two orders of magnitude to solve a fault (bug or
hardware failure) later than to avoid it in the first place.
I'm not certain how to scale up that success to the whole
system (I've been trying for years), but it must be
possible.

> I keep meaning to see what (if any) good reading there is
> on the team that built MS Excel.  As much as our little
> Linux list might not like MS, you have to admit -- after
> about the second or third version of Excel, the thing
> generally "did" what its end-users wanted, relatively
> bug-free.

Heh, excel has its share of bugs.  Anyone in finance using
Excel can tell you that.  The difference in Excel is that
90% of the users use 10% of the features.  That 10% either
went through the normal release/patch cycle long ago or is
tested through other applications in the office suite (e.g.
interface changes).

--David