[xplc-general] module versioning

[Pierre Phaneuf]

Okay, so I discussed a bit of this with Stéphane last friday and today,
and I'm still not sure what to do exactly...

There are two possible approaches:

- module gives you an IModule* (the current way)
- module has a ModuleInfo struct

The IModule Way
===============

Pros:

This is more forward-compatible, meaning that a module could always have
the option of trying to be loadable with older module loaders, if it
wanted to spend the effort. It would be forward-compatible in the same
way as other XPLC components can, by providing compatibility wrappers
when the loader requests an older interface.

Cons:

Heavier modules with more code in them. Less code is simpler, and I can
appreciate simple.

There is a circular reference in this design: the module cannot be
unloaded as long as there are references to objects with code from this
module, and the IModule* comes from this module.

The ModuleInfo Way
==================

Pros:

Lightweight and simple, it would be easy to make tools that analyze modules.

Cons:

If the loader encounters a module with a larger major version than it
supports, it can only croak.

-----------------------------------------------------------------

I like the ModuleInfo way a lot, as it is very simple and robust, but
having to deal with these version numbers isn't very XPLCish, which
actually has its own way of dealing with versioning (the IModule way
uses it).

The way I see this is like this. Implementation of IModule will likely
come from libxplc-cxx anyway. Supposing there is a major change in the
interface which would require bumping the major version, the IModule
implementation would have to provide a way to emulate the older
interface, for older module loaders. It could also decide to obsolete an
old interface by not implementing it anymore, but there would be little
reason to do so other than if this old interface become difficult to
emulate in the context of the current one.

If it is possible to emulate the old interface in the new one, it
implies that it only adds in a compatible way (not requiring extra
actions from the loader that it didn't do before, only optionally
requiring them to gain access to some new feature). So it sounds
plausible to me that any feature that could stay forward-compatible
using the IModule way could be done using the ModuleInfo way (my theory
is that it might be provable), and that the only times we'd need to bump
the major number would for changes in interface that couldn't possibly
be done, even with IModule. Does that sound right?

In this case, putting all this compatibility gunk in every single module
sounds like a lot of duplicate code going on when it could all be in one
place.

There is also nothing in particular that forces a module to follow the
very latest module major version, if it doesn't need the feature. The
macros could be done in such a way to easily allow usage of the oldest
interface that supports what the user needs.

I really, truly expect that bumping the major number should be a large
even, akin to bumping the soname of Linux's libc (anyone remembers
libc.so.5?).

There are some cases where you'd be just terminally screwed, but that's
the way it sometimes is. For example, a module that would use the
reflection information could still be validly loaded by a loader that
doesn't know about the reflection information and there would be just
nothing to play with. Not a catastrophic failure, but it'd be nice to
have a way of notifying the user... Have the function calls send the
maximum version numbers that the loader supports, would that be useful?
It couldn't react very much, there's not much space for interaction at
this level... Maybe if we added a bool return value to loadModule, which
could tell us that the loading "failed" (in the sense that the module
would be useless), allowing for possible user reporting?

I'm showing an obvious bias toward the ModuleInfo strategy, but I feel
that I don't really have super-solid arguments, so it's not clear.

[Stephane Lajoie]

Post by Pierre Phaneuf
The IModule Way
===============
This is more forward-compatible, meaning that a module could always have
the option of trying to be loadable with older module loaders, if it
wanted to spend the effort. It would be forward-compatible in the same
way as other XPLC components can, by providing compatibility wrappers
when the loader requests an older interface.

Those wrappers could be provided by xplc-cxx (which I bet most people
will link statically anyway) so it wouldn't be a big effort on the
part of the module author to be compatible with every loader ever
released.

I suppose the same kind of thing could be done for older ModuleInfo
though...

Post by Pierre Phaneuf
Heavier modules with more code in them. Less code is simpler, and I can
appreciate simple.

Imagine xplc taking off in a major way, but Debian still taking
forever to make a release. In five years, everybody could be writing
xplc modules but they would have to include compatibility code to make
them work with the loader in xplc 0.5.0. So yes, this is a "cons"
against IModule, but I think it applies equally well to ModuleInfo.

Post by Pierre Phaneuf
There is a circular reference in this design: the module cannot be
unloaded as long as there are references to objects with code from this
module, and the IModule* comes from this module.

The loader doesn't have to keep the IModule reference forever. It
could use it only for updating its cache and then release it. However,
that would require a separate way to create object instances. Maybe a
hybrid approache could work? More about that below.

Post by Pierre Phaneuf
The ModuleInfo Way
==================
Lightweight and simple, it would be easy to make tools that analyze modules.

Except in five years, when it would become relatively complex due to
different ModuleInfo versions. Not any less complex than an IModule +
IReflection + IWhatever, but at least with interfaces you leverage
something that people will have to learn anyway if they want to use xplc.

Post by Pierre Phaneuf
If the loader encounters a module with a larger major version than it
supports, it can only croak.

Watch that future Debian stable croak all the time :).

Post by Pierre Phaneuf
The way I see this is like this. Implementation of IModule will likely
come from libxplc-cxx anyway. Supposing there is a major change in the
interface which would require bumping the major version, the IModule
implementation would have to provide a way to emulate the older
interface, for older module loaders. It could also decide to obsolete an
old interface by not implementing it anymore, but there would be little
reason to do so other than if this old interface become difficult to
emulate in the context of the current one.
If it is possible to emulate the old interface in the new one, it
implies that it only adds in a compatible way (not requiring extra
actions from the loader that it didn't do before, only optionally
requiring them to gain access to some new feature). So it sounds
plausible to me that any feature that could stay forward-compatible
using the IModule way could be done using the ModuleInfo way (my theory
is that it might be provable), and that the only times we'd need to bump
the major number would for changes in interface that couldn't possibly
be done, even with IModule. Does that sound right?

Not to me. The object that provides this new-fangled
IVeryDifferentModule could still provide IModule and anything else
that was dreamed up between the two. This is the only thing that
fundamentally distinguishes IModule from ModuleInfo: compatibility
both ways for as long as people care to provide it, without disgusting
hacks. It's also the only thing that distinguishes xplc from a simple
wrapper around dlopen, but I don't have to remind *you* of that :).

Post by Pierre Phaneuf
In this case, putting all this compatibility gunk in every single module
sounds like a lot of duplicate code going on when it could all be in one
place.

That doesn't make any sense. If you could predict right now all the
compatibility gunk that could ever be needed you could put it in
xplc.so and be done with it. The fact that we can't predict all that
is the whole reason we're having this discussion: module loading
implies an interface boundary between xplc and modules, and you want
to be able to rev each side of the fence independently. Fortunately,
you already solved this problem with IObject::getInterface :).

Post by Pierre Phaneuf
There is also nothing in particular that forces a module to follow the
very latest module major version, if it doesn't need the feature. The
macros could be done in such a way to easily allow usage of the oldest
interface that supports what the user needs.

How many people accidentally force you to upgrade some libraries when
you install their software these days? The annoying thing with giving
people choices is that you also have to provide a default for those
who can't be bothered to be careful. In this case, the default would
likely be the new interface: you wouldn't have defined it unless you
thought it was better... I think a reasonable default for module
authors would be to always link xplc-cxx statically and have it
emulate everything that needs emulation.

Post by Pierre Phaneuf
I really, truly expect that bumping the major number should be a large
even, akin to bumping the soname of Linux's libc (anyone remembers
libc.so.5?).

Excellent. That means my suggestion of linking xplc-cxx statically and
including every emulation wrapper until the end of time will not
result in huge modules :).

Post by Pierre Phaneuf
There are some cases where you'd be just terminally screwed, but that's
the way it sometimes is. For example, a module that would use the
reflection information could still be validly loaded by a loader that
doesn't know about the reflection information and there would be just
nothing to play with. Not a catastrophic failure, but it'd be nice to
have a way of notifying the user... Have the function calls send the
maximum version numbers that the loader supports, would that be useful?
It couldn't react very much, there's not much space for interaction at
this level... Maybe if we added a bool return value to loadModule, which
could tell us that the loading "failed" (in the sense that the module
would be useless), allowing for possible user reporting?

The only reason you're looking into the module is to update the module
loader's internal cache. The module loader could report an error if
the updated cache is completely empty after the update (because the
module is useless to that loader). Unless you add HRESULTs everywhere
(which I don't advocate), you will never be able to report the exact
reason for something failing (especially if somebody finds a new way
of failing that you didn't think of before).

Post by Pierre Phaneuf
I'm showing an obvious bias toward the ModuleInfo strategy, but I feel
that I don't really have super-solid arguments, so it's not clear.

Here's my hybrid suggestion:

struct XPLC_ObjectEntry {
GUID guid;
IObject *(*createObject)();
};

Modules would have an array of those in read-only memory that allow
the module loader to quickly create any object supported by the module.

Also, a second module entrypoint could provide an object with IModule,
IMonikerMapper, ICategorySupplier and/or any other object discovery
interface that is invented in the future. This entrypoint could be
optional and it would only be used to refresh the cache of the xplc
module loader as a supplement to the static table.

This is quite simple and efficient in most usual cases, yet doesn't
expose any extensibility mechanism except xplc stuff. We all agree
that xplc should take over the world as an extensibility mechanism so
let's preach by example! :)

[Pierre Phaneuf]

Post by Stephane Lajoie
Those wrappers could be provided by xplc-cxx (which I bet most people
will link statically anyway) so it wouldn't be a big effort on the
part of the module author to be compatible with every loader ever
released.

Yes, that was my idea... I think I'm converting back to IModule, but not
the same way I did it originally.

Post by Stephane Lajoie
I suppose the same kind of thing could be done for older ModuleInfo
though...

If you have different entry points, or somehow provide for a "chain" of
ModuleInfo. More work in the module, and harder to automate, unlike
IModule. A bigger binary size footprint though, but I guess it's not as
important.

Post by Stephane Lajoie
Imagine xplc taking off in a major way, but Debian still taking
forever to make a release. In five years, everybody could be writing
xplc modules but they would have to include compatibility code to
make them work with the loader in xplc 0.5.0. So yes, this is a
"cons" against IModule, but I think it applies equally well to
ModuleInfo.

Ouch, quite the practical and realistic example!

The way it is, as long as we never bump up the major number, ModuleInfo
sounds better (other than looking kinda kludgy, not using the same
backward/forward-compatibility mechanism as the rest of XPLC), but as
soon as you bump the major number, it becomes a pain and IModule is better.

Of course, I'd rather never bump the major number, only bumping the
minor number, keeping it only in case of a major screw-up, but I'm all
too knowledgeable of major screw-ups actually happening...

Post by Stephane Lajoie
The loader doesn't have to keep the IModule reference forever. It
could use it only for updating its cache and then release it.
However, that would require a separate way to create object
instances. Maybe a hybrid approache could work? More about that
below.

Yes, I was thinking of a ModuleInfo, but with loadModule, unloadModule
and getModule function pointer. This is still all going around my mind...

Post by Stephane Lajoie

Post by Pierre Phaneuf
If the loader encounters a module with a larger major version than
it supports, it can only croak.

Watch that future Debian stable croak all the time :).

Ouch!

Post by Stephane Lajoie

Post by Pierre Phaneuf
In this case, putting all this compatibility gunk in every single
module sounds like a lot of duplicate code going on when it could
all be in one place.

That doesn't make any sense. If you could predict right now all the
compatibility gunk that could ever be needed you could put it in
xplc.so and be done with it. The fact that we can't predict all that
is the whole reason we're having this discussion: module loading
implies an interface boundary between xplc and modules, and you want
to be able to rev each side of the fence independently. Fortunately,
you already solved this problem with IObject::getInterface :).

Well, you *will* need compatibility gunk, that's for sure. If it's using
XPLC-standard gunkification, I suppose that kind of better. :-)

Post by Stephane Lajoie
struct XPLC_ObjectEntry {
GUID guid;
IObject *(*createObject)();
};
Modules would have an array of those in read-only memory that allow
the module loader to quickly create any object supported by the module.
Also, a second module entrypoint could provide an object with
IModule, IMonikerMapper, ICategorySupplier and/or any other object
discovery interface that is invented in the future. This entrypoint
could be optional and it would only be used to refresh the cache of
the xplc module loader as a supplement to the static table.
This is quite simple and efficient in most usual cases, yet doesn't
expose any extensibility mechanism except xplc stuff. We all agree
that xplc should take over the world as an extensibility mechanism so
let's preach by example! :)

I like that...

I find that it matches well the full set of things that a
IServiceHandler can do, but at the same time, I was thinking of
expanding what a IServiceHandler can do anyway (add a getMetaInfo method
or something, to find out stuff about stuff in a general way). Maybe I'm
just terminally insane, but I'm still having fun, so what the heck.

In any case, grepping for "unstable" gives you enough hits that you
ought to have figured out that it can change, by now... :-)

[Avery Pennarun]

Post by Pierre Phaneuf
- module gives you an IModule* (the current way)
- module has a ModuleInfo struct

My problem is that you talk about these as if they are different things.
IModule is a struct with some stuff. ModuleInfo is also a struct,
potentially with some different stuff.

I propose that you start with ModuleInfo now, then one day a sub-revision
could include a member that's an IModule*. Or alternatively, call it
IModule right now, make it a static object in the module (like you would
have done with the struct), derive it from IObject, and put all the fields
you might have put in ModuleInfo into your new IModule. Magic!

Have fun,

Avery

[Pierre Phaneuf]

Post by Avery Pennarun
My problem is that you talk about these as if they are different
things. IModule is a struct with some stuff. ModuleInfo is also a
struct, potentially with some different stuff.

IModule is a *pointer* to a struct, and the contract for it specifically
says that you can't assume anything other than a vptr being there.
But... (there's always a but!)

Post by Avery Pennarun
I propose that you start with ModuleInfo now, then one day a
sub-revision could include a member that's an IModule*. Or
alternatively, call it IModule right now, make it a static object in
the module (like you would have done with the struct), derive it from
IObject, and put all the fields you might have put in ModuleInfo into
your new IModule. Magic!

More like evil! ;-)

But I get your point, starting with ModuleInfo and adding a getIModule
member as a minor version makes full sense. Ah, the power of laziness
and YAGNI at work!

ObSourceForgeWhining: CVS is dead for the rest of the weekend, argh!