[xplc-general] Forward/backward compatibility

[Avery Pennarun]

In http://www.townhall.com/columnists/alanreynolds/ar20040325.shtml,

But there is something very annoying with this.
If you add a function, like I previously said, this is not forward
compatible, so you have to rename the library.

This is not actually true, and since a lot of your later argument relies on
this, it will get you into trouble. There are reasons you need XPLC, but
this isn't it.

Major sonames are for breakage in backward compatibility only; things that
are backward but not forward compatible do *not* generally change the
soname, unless the package maintainer is an idiot (and some are, of course).
Every library also has a "minor" soname (which is not actually an ELF
feature; it's just the filename) which changes with every change,
incompatible or otherwise, but people don't actually use this for anything
but keeping their sanity. ("Which version of libc is this? Oh, 2.2.6.")

Package maintainers make the following assumptions:

- the .h header files on the system will work perfectly with the library
version you get when linking.

- the library used at runtime will work perfectly with the API provided by
the header files used at compile time.

You can change the soname when adding an extra API function without breaking
these two assumptions. Note that you can also install header files from an
old library version, and link with a newer one, without breaking this
assumption - and you'll still be able to run with a library as old as the
one compatible with your header files!

So, when I use the new function supported only in version 1.1, but version
1.1 has the same soname (1) as version 1.0, how do I enforce that my binary
will only be used on systems with library 1.1 or higher? Package
dependencies. In Debian format, for example, I can say that my package
depends on:

libc6 (>= 2.2.6)

That is, libc with a soname libc.so.6, version 2.2.6 or greater.

In source-based systems like gentoo, I simply can't compile the program
unless I have a newer version of the headers, which means I need a newer
version of the library. Of course, they have source-level "build
dependencies" to make sure this is true.

libc's ELF tricks are not for forward compatibility - they're for backward
compatibility. The problem is that "statfs" in one version of libc has a
totally different API than "statfs" in the next version, so they play tricks
to make sure I get the one I got during link time. It would have been much
less evil to have the headers enforce this instead (#define statfs
statfs_2_2_6), but that would have required foresight, something programmers
are notoriously bad at.

Have fun,

Avery

[Pierre Phaneuf]

Post by Avery Pennarun
In http://www.townhall.com/columnists/alanreynolds/ar20040325.shtml,

Huh?

Post by Avery Pennarun

If you add a function, like I previously said, this is not forward
compatible, so you have to rename the library.

This is not actually true, and since a lot of your later argument relies on
this, it will get you into trouble. There are reasons you need XPLC, but
this isn't it.

This is actually true, ask Stéphane, who got screwed by that at some
point in his development of ExchangeIT!

It only *looks* like it's all right, and it's clearly the "least bad"
failure, and that's why it is commonly held as "not bad enough to
warrant bumping the soname".

Post by Avery Pennarun
Major sonames are for breakage in backward compatibility only; things that
are backward but not forward compatible do *not* generally change the
soname, unless the package maintainer is an idiot (and some are, of course).
Every library also has a "minor" soname (which is not actually an ELF
feature; it's just the filename) which changes with every change,
incompatible or otherwise, but people don't actually use this for anything
but keeping their sanity. ("Which version of libc is this? Oh, 2.2.6.")

For reducing the confusion, I use "soname" to refer to the ELF attribute
of the same name, and "filename" to refer to the filename of a shared
library.

That said, the filename has no incidence whatsoever on binary
compatibility issues.

Post by Avery Pennarun
- the .h header files on the system will work perfectly with the library
version you get when linking.
- the library used at runtime will work perfectly with the API provided by
the header files used at compile time.
You can change the soname when adding an extra API function without breaking
these two assumptions. Note that you can also install header files from an
old library version, and link with a newer one, without breaking this
assumption - and you'll still be able to run with a library as old as the
one compatible with your header files!

Yes, but adding a function makes the old library incompatible with your
header files. Users of this function will happily crash if linked with
an older version of the library. This is a broken ABI.

The tricky part with this is that it's not a full breakage, as you point
out: if you avoid using the new function, your program will work
perfectly with the old library, and also, programs linked with the old
library will certainly work perfectly with the new library.

This is explicitly where a system like XPLC helps so much, because it
can express support for both the old and the new version at the same
time, while safely preventing code using the new function from trying to
call it using the old library (because it will not support the IID it
will request).

Post by Avery Pennarun
So, when I use the new function supported only in version 1.1, but version
1.1 has the same soname (1) as version 1.0, how do I enforce that my binary
will only be used on systems with library 1.1 or higher? Package
dependencies. In Debian format, for example, I can say that my package
libc6 (>= 2.2.6)
That is, libc with a soname libc.so.6, version 2.2.6 or greater.

Okay, so *some* packaging systems have features to deal with this
problem, but this can't be relied upon very much. Mac OS X and Windows
package managers don't know anything about this, for example.

Still, this is a bit annoying, because the soname is supposed to be a
promise that your program will work safely, and if you add a function
without bumping the soname, it's just not safe. But at the same time,
it's annoying, because my program compiled with version 1.0 will work
safely with version 1.1, but bumping the soname would make that
impossible, for seemingly no good reason. That's why XPLC is especially
sweet, it makes all of that possible and safe, without uselessly
preventing you from using the newer version of the library.

This could have been easily improved with a DT_ALT_SONAME ELF attribute,
which would make ldconfig generate additionnal symlinks pointing at the
shared library. These would be other sonames that the library would be
compatible with, so that when you'd install version 1.1 (with the added
function), it would be free to bump the soname version while putting the
old soname in a DT_ALT_SONAME attribute, thus not breaking programs
linked with the old version.

The program with this approach is that a program linked with the new
version, but that would NOT use the new function in it, would still have
the newer soname specified in its dependencies, for no good reason. ELF
versioning does not have this problem (the linker puts the appropriate
version requirement depending on the symbols that *actually* get used),
but again, ELF versioning is non-portable.

Post by Avery Pennarun
libc's ELF tricks are not for forward compatibility - they're for backward
compatibility. The problem is that "statfs" in one version of libc has a
totally different API than "statfs" in the next version, so they play tricks
to make sure I get the one I got during link time. It would have been much
less evil to have the headers enforce this instead (#define statfs
statfs_2_2_6), but that would have required foresight, something programmers
are notoriously bad at.

Yeah, I agree that you could use this technique for C headers, and it
would be much more portable than the ELF tricks it uses, but it would
still be victim of the problem of missing symbols, which on modern
systems get resolved lazily and thus would report a missing symbol error
sometimes during the execution of the program, including a catastrophic
halt of the program.

The ELF versioning features have the added benefit of preventing the
program from starting if it would crash later.

[Avery Pennarun]

Post by Pierre Phaneuf
Huh?

Erm, I'm an idiot. I meant this:

http://xplc.sourceforge.net/doc/emails/20040323-pcolijn.html

Post by Pierre Phaneuf
This is actually true, ask Stéphane, who got screwed by that at some
point in his development of ExchangeIT!

Well, I'd be curious to know how.

Post by Pierre Phaneuf
Yes, but adding a function makes the old library incompatible with your
header files. Users of this function will happily crash if linked with
an older version of the library. This is a broken ABI.

But if your packaging system is working correctly, it's impossible to
accidentally link against the old library if you use the new header files.

Post by Pierre Phaneuf
This is explicitly where a system like XPLC helps so much, because it
can express support for both the old and the new version at the same
time, while safely preventing code using the new function from trying to
call it using the old library (because it will not support the IID it
will request).

In other words, it duplicates the functionality of rpm, dpkg, and/or apt.

Post by Pierre Phaneuf
Okay, so *some* packaging systems have features to deal with this
problem, but this can't be relied upon very much. Mac OS X and Windows
package managers don't know anything about this, for example.

I don't know about MacOS X. Windows, however, has no "package manager" to
speak of, so every application always includes all the libraries it might
need. But Windows has an actual *api* for checking dll versions and making
sure you have a newest one - it may be gross, but if you have a newer
version of msvcrt40.dll than someone else, your installer will overwrite
the old one automatically. If your msvcrt40.dll is newer than their
msvcrt31.dll, it won't. Sounds perfect. (It isn't perfect, unfortunately,
but for exactly the same reasons that XPLC can't fix it - people are idiots,
and don't understand what binary compatibility requires.)

Post by Pierre Phaneuf
Still, this is a bit annoying, because the soname is supposed to be a
promise that your program will work safely, and if you add a function
without bumping the soname, it's just not safe. But at the same time,
it's annoying, because my program compiled with version 1.0 will work
safely with version 1.1, but bumping the soname would make that
impossible, for seemingly no good reason. That's why XPLC is especially
sweet, it makes all of that possible and safe, without uselessly
preventing you from using the newer version of the library.

You have a solution in search of a problem here. *Nobody* in Linux has this
problem unless they run Slackware, and Windows users seem to have gotten
themselves out of it too. Maybe you can sell it to MacOS X users, but I
imagine they also have some clever plan.

In Linux, anyway, someone is already managing their "minor" library
dependencies independently of the ELF system, and it works great.

Have fun,

Avery