Methods of revealing the hidden self

It took me a long time to realise something that should have been obvious. Of course it takes me a long time to realise all sorts of obvious things (much to my wife's annoyance), but I expect that's besides the point, at least this point.

The point we'll finally get to might be something called “revealing the hidden self”, but before we get there we must first get to the point where we might understand its significance. You'll allow me to ramble on before I get to the obvious point I hope.

Dobbin, a helpful horse whose self will be revealed in due course

Let's take some generic bit of object oriented code (C++ in this case):

class Horse {
    void canter();
} dobbin;

dobbin.canter();

The body of canter(), whatever it may be, gets given a context to work within. Of course this context is the object that it is being called on, dobbin in our case. If we think about what happens as this code passes through the compiler then one stage might look like this sort of C code¹ [1I no longer remember the C rules for the use of void. I think it's OK to have it as a return type, at least in ANSI/ISO C.Also, the reason for translating into a C style syntax is because although C++ is no longer compiled into C which is then compiled into assembly to be turned into machine code and executed as micro-code, remembering that this translation process used to happen can help us to understand why C++ is the way it is. And besides, we all know C is just portable assembler anyway.]:

struct Horse {
} dobbin;
void Horse_canter( Horse* );

Horse_canter( &dobbin );

A couple of things are worthy of note here. Firstly we have to mangle the name of the function because it's no longer nested inside the data structure. This is technically important and is something I'll come back to a bit later.

Of more immediate interest though is that canter() is really just another global function that takes a Horse as its first parameter. This first parameter is of course hidden from us when we look at the C++ version, but it re-appears automagically within the body of canter() where it gets the name this.

This isn't the obvious thing.

The hidden self revealed

What if we want to add another method, gallop()² [2I don't really know enough about horses to tell whether they always canter at the same speed or not. I do know from seeing them race that they gallop at different speeds. In the case that this doesn't make sense from a zoological perspective imagine the horse breed to be that very rare Pedagogical Horse. They're a lot like wooden horses but have blackboards attached.]?

class Horse {
    void canter();
    void gallop( int speed );
} dobbin;

dobbin.canter();
dobbin.gallop( 20 );

Somewhere inside the compiler on its way to becoming machine code it is going to go through a stage that looks something like this:

struct Horse {
} dobbin;
void Horse_walk( Horse* );
void Horse_gallop( Horse*, int );

Horse_walk( &dobbin );
Horse_gallop_int( &dobbin, 20 );

We'll worry about how these function names are made later, for now it's enough to know that they are made.

The secret of the hidden self

Within the body of each method we will have this made available. Again it is done automagically by the compiler for us. This is the secret of the hidden self.

The “self” is the minimum context that the method has at its disposable. Even a method that takes no parameters has a hidden one, the self. We cannot deliver a message to any object without implicitly knowing which object it should arrive at and until it does arrive it hasn't been delivered.

And every message has to be delivered somewhere (except for those that get lost, but a discussion about the “Lost & Found” department in object oriented programming will have to wait). Once the message arrives and triggers the execution of some code (the method) then the self has to be made explicit. But right up until that point the language hides it. The self is tucked away, out of sight and out of mind.

This still isn't the obvious thing.

Using the hidden self

What if we were doing all of this in a dynamic language like Javascript?

function Horse() {
    this.canter = function() {};
    this.gallop = function( speed ) {};
}

var dobbin = new Horse();
dobbin.canter();
dobbin.gallop( 20 );

The dynamic version doesn't look all that different, but of course the mechanics behind the scenes are quite different³ [3Javascript doesn't go through the same process as C++. Specifically there's no linking phase and all of the functions are always first order citizens—we can see this from the constructor and the way we've built the instance, but all of that is besides the point.]. What it does do though is the same trick as C++. It hides the self and brings it back out to us when we need it inside the implementations of canter() and gallop().

Javascript also lets us do one trick that C++ doesn't though:

function Horse() {
    this.canter = function() {};
    this.gallop = function( speed ) {};
}

var dobbin = new Horse();
dobbin.canter();
dobbin.gallop( 20 );

dobbin.trot = function() {};
dobbin.trot();

What we've just done is to extend the class without the original constructor knowing anything about it. That's pretty clever and of course it's one of the great things about using dynamic languages. Shame we can't do it in C++.

Now we get to the obvious thing.

Actually we can do it in C++. The syntax is different, but the effect is the same. Let me repeat that, from a design perspective it does exactly the same thing.

class Horse {
    void canter();
    void gallop( int );
} dobbin;

dobbin.canter();
dobbin.gallop( 20 );

void trot( Horse &dobbin );
trot( dobbin );

I can hear you shouting. If you calm down a moment I'll explain myself.

Of course it isn't remotely the same syntactically, but conceptually it is exactly the same. It'll be clearer if we think about the C like intermediate⁴ [4I'm glossing over something important here. Don't worry about it now, it'll be in the first appendix.]:

struct Horse {
} dobbin;
void Horse_walk( Horse *horse );
void Horse_gallop( Horse *horse, int );
void trot( Horse &horse );

Horse_walk( &dobbin );
Horse_gallop( &dobbin, 20 );
trot( dobbin );

The only real difference is that we're passing that hidden parameter, the hidden self, out in the open and we're having to do it ourselves with no help from the compiler⁵ [5I'm going to completely ignore the difference between pass by reference and pass by pointer. It simply isn't relevant. I hope we won't have a problem with that.].

If you take the time to think about this a bit more you'll notice that this opens a whole world of new possibilities.

What “revealing the hidden self” tells us is that we don't need to make every method part of the class. We can leave the class uncluttered and put them outside the class where it just so happens we can have as many of them as we want and we can keep adding new ones without recourse to the original class.

By making the hidden self explicit we're not losing anything at all in an object oriented sense, but we're gaining a lot by losing clutter in our classes. Our classes can become even more cohesive than they were before.

And we all know (in the voice of Homer Simpson) “Mmmmmm cohesion.”

Making peace with the hidden self

One of the things that this should start us thinking about is why we've got canter() and gallop() in the Horse class at all? What if we did something a bit more egalitarian, something a bit more like this⁶ [6Take a look at the second appendix for a bit more discussion about this. Again, I'm pulling a fast one, but don't worry about it for now.]:

class Horse {
    void move( int leg, int how_far );
} dobbin;

void trot( Horse & );
void canter( Horse & );
void gallop( Horse &, int );

Many of you will spot that this now looks more like structured programming than object oriented programming. Actually that's fine. We're doing this because it makes the program easier to work with, easier to understand and easier to maintain. All laudable goals we should be putting more effort into.

If Paul Graham were ever to read this page though, he would by now be laughing so hard he'd not be able to write a macro for a whole week. LISP has been doing exactly this since before Paul Graham was compiled⁷ [7Of course deep down it'd really be because he's jealous that he can't talk about “revealing the hidden self” when programming. In LISP it was never hidden in the first place.].

Well, I did say it was obvious.

Appendix A

I'm glossing over something important here. Don't worry about it now, it'll be in the first appendix.

I said I was glossing over something important. The important thing that I was glossing over sits somewhere between overloading and something called “Koenig lookup”.

We had this bit of C++:

class Horse {
    void canter();
    void gallop( int );
} dobbin;

dobbin.walk();
dobbin.gallop( 20 );

void trot( Horse &horse );

trot( dobbin );

When we've been translating these into a C style intermediate we've been having to mangle the method names. This is because in C every function requires a unique name⁸ [8It's actually more correct to say that the link phase requires the unique names, but it isn't really relevant what requires the unique names. The important thing is that unique names are needed.] so we have to do something to method names to make them unique. We've been using a fairly simple scheme that just involves prefixing the class name (and presumably we'd also prefix any outer class or namespace names too).

This isn't quite enough though because C++ allows us to overload methods and free standing functions. We also need to include the arguments in the name mangling and, because methods can come in const and non-const forms we also have to add that in. This means that the function the C intermediate sees is really something more like this⁹ [9Real-world C++ compilers do this sort of name mangling, but the algorithm that they use isn't as simple as the one we're using. Because each compiler tends to have its own name mangling scheme you can't always link code compiled by different compilers—often even different versions of the same compiler.]:

void gallop_Horse_int( Horse &horse, int );

Note that we've now used Horse after the function name because it describes an argument. The types of the other arguments follow in order, but a namespace name would still appear first.

Because of this we can see that we're not actually limited to the number of gallop() methods we need to add. We can add one for each class that we have in our program (and we can even add ones for built-in types) because each will receive unique names. We can also several to the same class with the same name. All of this is called overloading.

So what is the Koenig lookup all about? In order to choose the right version of gallop() it isn't good enough to just work out the mangled name that the arguments we give should use because there are all sorts of type conversion rules. Some of these rules are built-in (i.e the promotion of int to long) and others we can specify (either through an appropriate constructor or a cast operator).

The net effect is that the compiler needs some algorithm to use when an exact match isn't found. This algorithm is called Koenig lookup.

If you want to think about this in pure object oriented terms then the Koenig lookup the compiler does is just another message dispatcher.

Of course if you look at LISP you'll find an analogous algorithm whose purpose is exactly the same—that of working out which bit of code to deliver both the message and the self to, except its often “selves” rather than “self” (and that is called multiple dispatch, something I'll go into another time).

Not the final addendum

As ever there is something lurking just below the surface that is worth pulling out and examining more closely.

The name mangling is done by the compiler. What this means in practice is that when we use gallop() in our code then the compiler looks for the best match that it knows about. Even if we have a perfect match, if the compiler doesn't know about it (because we forgot to include it in the header, or didn't include the headers we should have done) then the compiler will generate the wrong mangled name.

The linker will link it properly and we'll spend days looking for a really subtle and hard to find bug. The code will run because the rules guarantee that the types are compatible, but the program will do the wrong thing. Ouch!

Appendix B

Take a look at the second appendix for a bit more discussion about this. Again, I'm pulling a fast one, but don't worry about it for now.

We can see that there are all sorts of methods that we have in a class that are fundemental to how the instances operate. They control the internal state, do the error checking and all sorts of other things without which the object we make from that class are completely and utterly useless.

We also end up with a load of fluff that is in there not because it is critical to the object's function but because it's a convenient place to drop the code.

When Andrei Alexandrescu examined the C++ std::basic_string<> design he realised that the class has exactly this pattern—a number of methods that form the core of the implementation and a number that can be easily expressed in terms of that core.

Now Alex is a much cleverer developer than I am and he doesn't anywhere (at least to my knowledge) say that it is a mistake to include those secondary members as part of the std::basic_string<> class. Surely this means I'm an idiot?

I hope not. The fact that a certain method can be implemented in terms of some core of functionality only makes it a candidate for implementation outside the class. It does not mean that it should be implemented outside the class.

When making the choice about where to implement a method there is a long list of things that need to be considered. The method's requirement of private information on the instances is just one of the things that needs to be considered. Other factors include:

• Other possible implementations of the core class implementation. Often two members can be implemented in terms of each other, for example operator ==() and operator !=(). We shouldn't make one an outsider just because we can.
• Increasing the cohesiveness of a class is bi-directional, it wants to draw things in as well as push things out. For maximum cohesiveness we have to include the core functions that the object requires irrespective of how they are implemented.
• There is a difference in syntax. Although they are equivalent they are not the same. Sometimes this matters and sometimes it matters a lot.

To say that having the methods outside of the class reduces their visibility is I think irrelevant. Any automated documentation tool is designed to support the idioms we use. If there is an idiom we want to use which has no automated documentation support then the tool has a bug. There is no reason for these external methods not be listed alongside the class that they are logically grouped with. In UML for example you might use a stereotype of <<External>>.

The final fast one

Of course the final fast one I pulled was the sudden introduction of a move() method which presumably is used by the other methods to move the legs. Normally we would hide this method from outsiders, but now we are exposing it.

Whether or not this is really an appropriate thing to do needs to be weighed carefully. The issues are less important for smaller projects or where the class is used in a single context, but they become much more important on large projects or where Horse is part of a library that supports many applications.

Whether or not we should expose it raises questions about the encapsulation of the object. The question I feel is most important here has to do with internal versus external representation.

As I hint at in Encapsulation is a Good Thing™ (and will explain much more fully in another article), encapsulation is used to hide the implementation from the external interface. If the method we have to expose is only relevant due to our implementation it is a poor candidate for exposure.

More specifically, if we can conceive of another implementation that would invalidate the method (either remove it entirely or invalidate its interface—by that I mean arguments) then we probably shouldn't do it¹⁰ [10This issue of interface stability relates to problems of leaky abstraction and really deserves its own explanation which I shall have to give elsewhere.].

For our Horse.move() example though I can conceive of a Blacksmith.reshod( Horse ) that would want to move the legs without getting trampled on. Maybe in this example exposing move() actually makes the class more useful.