C++ Reflection: Type MetaData: Introduction

Thinking about how to take things a step further in terms of what your tools can do for you is really important in increasing productivity. This is where a form of reflection can come in very handy. I call the reflection system I'm working on by shorthand of "meta" or "MetaData", though the proper terminology for it would be something more like Class MetaData or Type Reflection. So when I say MetaData I'm referring to data about data, and specifically data about the types of data within your C++ code.

Having a MetaData system allows for information about your data types to be saved during run-time for use during run-time. The C++ compiler compiles away a lot of information, and a lot of this information is very useful. So, a MetaData system saves this information from being compiled away.

So what is the use of such a system? Where here are the things I've constructed so far: simple and powerful debugging features; automated serialization for any type registered with Meta; automic function binding to Lua; Variant and RefVariant class types (objects that can hold any type of data registered within Meta). However these aren't the only things I'll be using the MetaData system for. One could also apply the MetaData to make simple object factories with hardly any code, or property viewing tables for editors with ease. I'm sure there's even more uses as well that I just haven't quite come to terms with. In a way, systems like this can generate tools and functionality for the developer whenever a new class or data type is introduced.

Before we start I want to let the reader know that such a system can be very efficient, efficient enough to run wonderfully within a real-time application like a game. I myself don't know a lot about efficiency or optimization at a low-level, though I do know other programmers who've made such Reflection systems that are very, very fast. So make sure not to have any pre-built misconceptions about a C++ Reflection system before moving on.

I started learning about how to construct a MetaData system from this post. You can take a look if you like, though it won't be necessary if you just want to learn what I'm trying to teach here. In that post, by a fellow student here at DigiPen, he goes over how to get started with MetaData, but leaves a lot to yet be learned. I'll be modelling my post after his quite a bit, as he does have good content structure for an introduction post.

I'll attempt to document what I've learned here as honestly, there's not any resources on constructing a MetaData system like this anywhere as far as I know. The closest thing I could find was from a Game Programming Gems book, the 5th one chapter 1.4, although it required all classes that wanted to participate to inherit from the MetaData class. This isn't really sufficient if you want to reflect class or structure types you don't have source code access to, and it doesn't support the built-in types at all.

Getting Started
To start lets talk about the overall structure of what a MetaData system looks like. I think it's going to best to draw a diagram of what will be achieved from this article:

Diagram of entire MetaData system layout.

In the above diagram the MetaData objects are the key object in which all important operations are performed. A MetaData object is a non-templated class, allowing them to be stored in a data structure. There is one MetaData object for each type of data registered to the system, and the MetaData object of a corresponding type is a representation of that type. It stores information about whether or not it is a Plain Old Data type (POD), the size of the type, it's members and methods, and name. Inheritance information can be stored along with multiple inheritance information, though I haven't even bothered adding that feature in yet as it's not actually very useful and quite difficult to do properly.

The MetaCreator is a templated class that manages the creation of a single unique MetaData instance. It then adds its instance into the MetaManager which contains it in some sort of map.

The MetaManager is a non-templated class that contains all of the created MetaData instances, and can perform search operations on them to find specific types. I use a map of strings to instances, so I can search by string identifier. I've also placed some other small utility functions into my MetaManager as well.

Client Code
Before we get started writing anything, I'd like to try to show some example client code to exemplify why I've taken all this time to make a MetaData system in the first place.

GameObject *obj = FACTORY->CreateObject( "SampleObject" );

// FACTORY->CreateObject code
GameObject *ObjectFactory::CreateObject( const std::string& fileName )
{
  SERIALIZER->OpenFile( fileName );
  GameObject *obj = DESERIALIZE( GameObject );
  SERIALIZER->CloseFile( );
  IDObject( obj );
  LUA_SYSTEM->PushGameObject( obj );
  return obj;
}

// LUA_SYSTEM->PushGameObject code
void LuaSystem::PushGameObject( GameObject *obj )
{
  lua_getglobal( L , "GameObjects" );
    
  lua_pushinteger( L, obj->GetID( ) );
  LuaReference *luaObj = (LuaReference *)lua_newuserdata( L, sizeof( LuaReference ) );
  luaL_setmetatable( L, META_TYPE( GameObject )->MetaTableName( ) );
  luaObj->ID = obj->GetID( );
  luaObj->meta = META_TYPE( GameObject ); // grab MetaData instance of this type
    
  lua_settable( L, 1 ); // Stack index 1[3] = 2

  // Clear the stack
  lua_settop( L, 0 );
}

As you can see I have a nice DESERIALIZE macro that can deserialize any type of data registered within within the Reflection. My entire serialization file (includes in and out) is only about 400 lines of code, and I implemented my own custom file format. I also have a LuaReference data type, which contains a handle and a MetaData instance, and allows any class to be sent to Lua via handle. Because of my Meta system I can write very generic and powerful code pretty easily.

Getting Started
The first I recommend starting with is to reflect the size of a type, and the name of a type. Here's what a very simple  MetaData class could like:

//
// MetaData
// Purpose: Object for holding various info about any C++ type for the MetaData reflection system.
//
class MetaData
  {
  public:
    MetaData( std::string string, unsigned val ) : name( string ), size( val ) {}
    ~MetaData( )

    const std::string& Name( void ) const { return name; }
    unsigned Size( void ) const { return size; }

  private:
    std::string name;
    unsigned size;
}

This simple class just stores the size and name of a type of data. The next thing required is the ability to create a unique instance of MetaData. This requires a templated class called the MetaCreator.

template <typename Metatype>
class MetaCreator
{
public:
  MetaCreator( std::string name, unsigned size )
  {
    Init( name, size );
  }

  static void Init( std::string name, unsigned size )
  {
    Get( )->Init( name, size );
  }

  // Ensure a single instance can exist for this class type
  static MetaData *Get( void )
  {
    static MetaData instance;
    return &instance;
  }
};

You should be able to see that by passing in a type, perhaps <int> to the MetaCreator, a single MetaData instance becomes available from the Get function, and is associated with the MetaCreator<int> class. Any type can be passed into the Metatype typename. The MetaCreator constructor initializes the MetaData instance. This is important. In the future you'll have MetaData for a class that contains MetaData for POD types. However because of out-of-order initialization, some of the types required to construct your class MetaData instance might not be initialized (as in the Init function will not have been called) yet. However if you use the MetaManager<>::Get( ) function, you can retrieve a pointer to the memory location that will be initialized once the MetaCreator of that specific type is constructed. It should be noted that the construction of a MetaCreator happens within a macro, so that there's absolutely no way of screwing up the type registration (the inside of the macro will become quite... ugly).

Lastly you'll need a place to store all of your MetaData instances: the MetaManager!

//
// MetaManager
// Purpose: Just a collection of some functions for management of all the
//          various MetaData objects.
//
class MetaManager
{
public:
  typedef std::map<std::string, const MetaData *> MetaMap;

  // Insert a MetaData into the map of objects
  static void RegisterMeta( const MetaData *instance );

  // Retrieve a MetaData instance by string name from the map of MetaData objects
  static const MetaData *Get( std::string name ); // NULL if not found

  // Safe and easy singleton for map of MetaData objects
  static MetaMap& GetMap( void )
  {
    // Define static map here, so no need for explicit definition
    static MetaMap map;
    return map;
  }
};

And there we have a nice way to store all of our MetaData instances. The Get function is most useful in retrieving a MetaData instance of a type by string name. Now that we have our three major facilities setup, we can talk about the macros involved in actually registering a type within the MetaData system.

  //
  // META_TYPE
  // Purpose: Retrieves the proper MetaData instance of an object by type.
  //
#define META_TYPE( TYPE ) (MetaCreator<TYPE>::Get( ))

  //
  // META
  // Purpose: Retrieves the proper MetaData instance of an object by an object's type.
  //
#define META( OBJECT ) (MetaCreator<decltype( OBJECT )>::Get( ))

  //
  // META_STR
  // Purpose : Finds a MetaData instance by string name
  //
#define META_STR( STRING ) (MetaManager::Get( STRING ))

  //
  // DEFINE_META
  // Purpose : Defines a MetaCreator for a specific type of data
  //
#define DEFINE_META( TYPE ) \
  MetaCreator<TYPE> NAME_GENERATOR( )( #TYPE, sizeof( TYPE ) )

So far so good. Using the DEFINE_META macro it's pretty easy to add a type to the MetaData system, simply do DEFINE_META( type );. The decltype might be confusing, as it's new. decltype simply returns the type of an object. This allows the user to retrieve a MetaData instance that corresponds to an object's type, without knowing what the object's type is; this lets very generic code be easily written.

NAME_GENERATOR is a bit tricky. Every single instance of a MetaCreator needs to be constructed at global scope- this is the only way to get the Init( ) function to be called, without having to place your DEFINE_META macro in some sort of code scope. Without the constructor of the MetaCreator calling Init, the only way to have any sort of code run by using the DEFINE_META macro is to place it within some scope that is run sometime after main executes. This makes the use of the DEFINE_META macro harder. If you create the MetaCreator at global scope, then you can have the constructor's code run before main executes at all, making the DEFINE_META macro very easy and simple to use.

So then comes the issue of "what do I call my MetaCreator?" arises. The first thing you might think of is, just call it MetaCreator and make it static. This hides the MetaCreator at file scope, allowing the DEFINE_META macro to be used once per file without any naming conflicts. However, what if you need more than one DEFINE_META in a file? The next solution I thought of was to use token pasting: ## operator. Here's an example usage of the token pasting technique:

DEFINE_META( TYPE ) \
  MetaCreator<TYPE> Creator##TYPE( )( #TYPE, sizeof( TYPE ), false )

The only problem with this strategy is that you cannot pass in type names with special characters or spaces, as that won't result in a proper token name. The last solution is to use some sort of unique name generation technique. There are two macros __LINE__ and __FILE__ that can be used to generate a unique name each time the macro is used, so long as it is not used twice on the same line of the same file. The __LINE__ and __FILE__ definitions are a bit tedious to use, but I think I have them working properly, like this:

#define PASTE( _, __ )  _##__
#define NAME_GENERATOR_INTERNAL( _ ) PASTE( GENERATED_NAME, _ )
#define NAME_GENERATOR( ) NAME_GENERATOR_INTERNAL( __COUNTER__ )

You have to feed in the __COUNTER__ definition carefully in order to make sure they are translated by the preprocessor into their respective values. A resulting token could look like: GENERATED_NAME__29. This is perfect for creating all of the MetaCreators at global scope on the stack. Without the use of some sort of trick like this, it can be very annoying to have to use a function call to register your MetaData.

Alternatively there are __FILE__ and __LINE__ macros, but they aren't really necessary as the __COUNTER__ does everything we need. The __COUNTER__ however is, I believe, not actually standard.

So far everything is very straightforward, as far as I can tell. Please ask any questions you have in the comments!

It should be noted that const, &, and * types all create different MetaData instances. As such, there is a trick that can be used to strip these off of an object when using the META macro. This will be covered in a subsequent article.

Example Uses
Now lets check out some example client code of what our code can actually do!

DEFINE_META( int );
DEFINE_META( float );
DEFINE_META( double );
DEFINE_META( std::string );

void main( void )
{
  std::cout << META_TYPE( int )->Name( ); // output: "int"
  std::cout << META_TYPE( float )->Size( ); // output: "4"

  std::string word = "This is a word";
  std::cout << META( word )->Name( ); // output: "std::string"
  std::cout << META_STR( "double" )->Size( ); // output: "8"
  if(META( word ) != META_TYPE( int ) // address comparison
  {
    std::cout << "This object is not an int!";
  }
}

And there we have it! An extremely easy to use DEFINE_META macro that stores name and size information of any type, including class and struct types.

Future Posts
For future posts I look forward to writing about automated Lua binding, Variants and RefVariants, automated Serialization, factory usage, messaging, and perhaps other topics as well! These topics are really rather huge, so please by patient in that it may take a while to cover everything.

Link to second article in series.

Simple Smart Pointers

This is going to be the first of many posts abandoning C and heading into C++. I stuck around in C for as long as could, but now heading into my Sophomore year it's time to make the switch.

There are many different implementations and goals of what a smart pointer can be or attempt to achieve. As far as I can see, the term "smart pointer" is a pretty general term that means some sort of extra functionality wrapped around normal pointers. However the smart pointer should be able to behave just like a normal pointer, so overloading of the * and -> operators will be done.

In previous game projects (the ones I written in C in my portfolio) had an issue where when a bit of data was deleted or modified, all other code required to be made aware of such act. This required extra work by the programmer to ensure that no pointers within the program could be accessed if the data they pointed to was modified in an undesirable way (perhaps deleted or converted to another type, or moved). The smart pointer implementation will solve this problem by making it seem like all smart pointers to a single location update each other automatically. Additionally, I wanted to add a little more functionality for ease of use and convenience, of which I'll talk about later in the article.

Here's an example of a problem situation that can arise from misusing ordinary pointers:

SomeFunction
{
  ObjectAddress = newly allocated object
}

Some Other Function
{
  Delete data at ObjectAdress
}

Last Function
{
  ObjectAddress.GetName( ) // Is object deleted yet?
}

I decided to make my smart pointers handle based. A handle system is where some sort of identifier (ID) is mapped to a value by mapping functionality, i.e. std::map. This is extremely useful, as one can now place a layer of indirection between an identifier and its associated value. The reason this is useful goes into many different scenarios, but pertaining to smart pointers we can have a single point of access to the address a smart pointer holds.

template <typename TYPE>
class SmartPtr
{
public:
  SmartPtr( handle val = NULL ): ID( val ) {};
  ~SmartPtr( );

  TYPE *GetPtr( void ) const;
  handle GetID( void ) const { return ID; }
  TYPE *operator->( void );
  TYPE operator*( void );

private:
  handle ID;
  static HandleMap<TYPE> handleMap;
};

Here are the barebones for our smart pointer. The SmartPtr holds an ID, where handle is a typedef'd integer. GetPtr( ) indexes the handleMap (handleMap should be a class to wrap around std::map, to provide inserting and erasing methods) with the SmartPtr's ID to retrieve the pointer value the ID maps to. The SmartPtr itself is templated for a specific type of pointer, this way each different type of data to be pointed to has a different statically defined handleMap. This is a benefit to the amount of unique handles your game can generate. The SmartPtr can be constructed with a handle value, and has a gettor for the handle ID itself.

Now whenever the pointer mapped by a specific ID is modified within the handleMap, all subsequent requests to that value will be updated throughout all other SmartPtr instances with the same ID. This is because the only point of access to the pointer value is through the map itself, meaning there's actually only one pointer mapped by an ID at any given time, although there can be many SmartPtrs with the ID mapping to that single value.

However you may be wondering about how to generate the handle ID itself, as each one needs to be unique. It's very simple: start your first handle at the integer value of 1, and then for every subsequent ID mapped to a new value, add one to the previously used ID. This will give you millions of unique IDs, which is well than more enough for basically any game ever, especially in this case as each type has its own handleMap altogether. The ID value of 0 can be reserved for a NULL pointer to allow to check at any given time if a SmartPtr's handle is valid.

As for the implementation of -> and *, you should use your utility GetPtr( ) function for them. -> will simply require a returning of GetPtr( ), and * will dereference the value returned by GetPtr( ), or return NULL if the value from GetPtr( ) is NULL. In this way your SmartPtr class will behave exactly like an ordinary pointer.

Additionally operator overloads for comparisons, or even pointer arithmetic, can be implemented as well. I implemented equality comparisons.

Lastly, I needed a way to handle inserting a pointer into the handleMap, and consequently erasing it from the map as well. I decided to make the SmartPtr handle this automatically with its constructor and destructor. Currently the example SmartPtr interface I've shown above supports constructing a SmartPtr from an already-generated handle, assuming the handle generation is external the SmartPtr itself. However overloading the constructor to receive a TYPE pointer allows for insertion of this TYPE * into the handleMap automatically. Here's an example:

template <typename TYPE>
SmartPtr<TYPE>::SmartPtr( TYPE *ptr )
{
  ID = handleMap.InsertPtr( ptr );
  clearOnDestruct = true;
}

The constructor inserts the ptr of TYPE into the handleMap by calling a method called InsertPtr( ). The return value is used to initialize the SmartPtr's ID data member. A bool called clearOnDestruct is required to be set, and this will be talked about later.

The InsertPtr( ) is a custom-defined method for your handleMap class. It should just insert a new pointer with a newly generated ID. On destruction of the SmartPtr, if the bool clearOnDestruct is set, the pointer can erase the slot in the handleMap for that ID. This way SmartPtr can contain a pointer for you, perhaps within a linked list or some other data structure.

Lastly, a method called DeletePtr( ) should be implemented to allow delete to be called on the data the SmartPtr contains. This SmartPtr implementation isn't supposed to act as any sort of garbage collection: no reference counting is done. Heap memory contained by a SmartPtr must be explicitly deleted with DeletePtr( ) by design. This way SmartPtr's can easily contain heap allocated memory as well. Here's a diagram showing the flow of a SmartPtr:

Flow of a smart pointer. Can either reference an ID (left), or contain an inserted pointer (right).

Like I said, there are many different ways to implement smart pointers, and many different things they can achieve. However the details here are just what I decided to use to solve specific problems for projects I work on.

Additional reading:

• C++ for Game Programmers - Noel Llopis: Chapter 12
• Modern C++ Design - Andrei Alexandrescu: Chapter 7