[lug] Running a mixed Python environment

[Rob Nagler]

On Tue, Sep 15, 2020 at 10:23 AM Davide Del Vento wrote:
> It's painful, but why should they do what you and I, the unpaying
customers, are asking?

I'd be happy to pay maintainers to not break interfaces. I don't think
money is the issue. It's more about expectations and assumptions.

> But I've lived on the other side (even paid), and I can tell you, it's
not pretty and I think for most project is not going to happen. The example
you provide of the opposite I think are (and will remain) the rare
exceptions.

The list is quite long of compatible interfaces. You just don't think about
them, because well, they are compatible. Consider that 'ls -l' has worked
for decades. They could have changed it to do what 'ls -lh' does, but they
didn't. Many scripts depend on that format and that the output is in bytes,
not human readable GBs, TBs, etc. Maddenlingly, 'rm -r' crosses file
systems, when 'rm -r --one-file-system' should be the default. It's fine
that they didn't fix this. I could go on about this, but a tremendous
amount of software never ever breaks APIs.

The big difference between Guido and Larry was that Guido was a professor
and Larry was a sysadmin. Perl has a culture of backwards compatibility.
Python has a culture of breakage. The famous line from Ken Thompson "I'd
spell creat with an e" is an example of the culture of backwards
compatibility. Unix has a terrible naming culture, but it is an admirable
culture of backwards compatibility. This statement sums up both.

The thing that's happened is that people can upload to pypi, npmjs, etc.
willy nilly, and they do with a second thought. There are many GitHub
Issues where I scratch my head about why they broke things, randomly. One
time pip (the program) hid all of its interfaces, overnight, because they
had not been hidden before. The purpose of hiding interfaces is so you
don't have to break them when you change them. Since they were public, they
were, well, public, and by hiding them, they broke them, everywhere, all at
once. Brilliant.

> I like chronological versioning, but that assumes there will never be the
need for a "fork" for bugfixes.

That's true, and the point is that bug releases can be made at any time
without breaking anything, because you have a culture of backwards
compatibility. If someone is depending on the buggy behavior (re:
javascript) that will be a problem for them, of course, but that's
different from changing interfaces.

> Unfortunately the plethora of hardware and software we use (even paid
one) is so complicated.... I have no idea on how to make happen in practice
what you say for a large project, let alone for different projects with
competing needs!

You can make your own software backwards compatible. It's not hard. I've
been doing it for a few decades now. :)

> Case example: two applications utilizing the same underlying
communication library -- one application wanting bit-for-bit
reproducibility first, the other wanting highest performance first. How to
update the library with these two competing needs? Major-minor numbers
allow that, chronological versioning doesn't. As you know this is NCAR's
bread'n butter

I would say you have a configuration parameter that sets the speed, just
like in a Tesla. This happens all the time in Python with libraries that
are written in C and pure Python. Sometimes they make choices implicitly,
or you can be explicit. I don't see how this is relevant to versioning or
backwards compatibility.

I use MPI, and it's had the same horrible interface for decades. In fact, I
was using it in the 80s. It's designed to work on supercomputers with
direct access to the network and on virtual machines running Linux on my
Mac, and guess what, it does. Now CUDA is version-nightmare especially
coupled to Tensorflow.

Back in the day, I worked extensively on operating systems and networking
code. The one rule was "don't break it". Linus understands this, because he
read code from people like Andy Tannebaum, who understood this. I am
confident that one can design communication libraries today that don't
break compatibility while providing for different users' needs.

NCAR has a large FORTRAN culture, which is renowned for its backward
compatibility. Why has this not permeated the Python or communications
library world at NCAR?

Rob
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lug.boulder.co.us/pipermail/lug/attachments/20200915/c4bdd739/attachment.html>