How to keep things around

When developing games in Love2D I love working in a live environment that can be interactively updated. The key tool for interactive gamedev with Fennel is the REPL.

While I rarely interact with the REPL directly, outside of testing out new functions, I use the inbuilt command ,reload extensively to reload the module files I am working on. Unlike playing around in the REPL, the changes made to the module files themselves are persisted between game reloads and can be version controlled through git.

To see the processes referenced below in action, check out the associated love2d project persistence.

What is a module?

Fennel compiles to Lua, and to Lua (5.1+) a module is just a value (normally a table) in the global table package.loaded.

The default way to load a package in lua is to use the require function. Looking at the C source code for require, found in loadlib, we can see the runtime checks to see if the module has already been loaded. If it has been it returns the module value. If not, it sees if there is a loader that matches the form of the module name passed to require (loaders were referred to as searchers in Lua5.1).

A loader simply returns a value to be stored in the package.loaded table. Most read a file, execute it and return its final value. If there is a loader require calls it and stores its return value in package.loaded

static int ll_require (lua_State *L) {
  const char *name = luaL_checkstring(L, 1);
  lua_settop(L, 1);  /* LOADED table will be at index 2 */
  lua_getfield(L, LUA_REGISTRYINDEX, LUA_LOADED_TABLE);
  lua_getfield(L, 2, name);  /* LOADED[name] */
  if (lua_toboolean(L, -1))  /* is it there? */
    return 1;  /* package is already loaded */
  /* else must load package */
  lua_pop(L, 1);  /* remove 'getfield' result */
  findloader(L, name);
  lua_rotate(L, -2, 1);  /* function <-> loader data */
  lua_pushvalue(L, 1);  /* name is 1st argument to module loader */
  lua_pushvalue(L, -3);  /* loader data is 2nd argument */
  /* stack: ...; loader data; loader function; mod. name; loader data */
  lua_call(L, 2, 1);  /* run loader to load module */
  /* stack: ...; loader data; result from loader */
  if (!lua_isnil(L, -1))  /* non-nil return? */
    lua_setfield(L, 2, name);  /* LOADED[name] = returned value */
  else
    lua_pop(L, 1);  /* pop nil */
  if (lua_getfield(L, 2, name) == LUA_TNIL) {   /* module set no value? */
    lua_pushboolean(L, 1);  /* use true as result */
    lua_copy(L, -1, -2);  /* replace loader result */
    lua_setfield(L, 2, name);  /* LOADED[name] = true */
  }
  lua_rotate(L, -2, 1);  /* loader data <-> module result  */
  return 2;  /* return module result and loader data */
}

/* }

Reloading a module

The magic of ,reload is that it updates the module value in the global table with the current version of the associated module file. Other modules that reference this module will now reference the new version. If you break your game into logical chunks, and use the idiomatic approach of returning a table at the end of your module file, you can reload specific parts of your codebase at runtime and see the changes in real time. This is interactive game development at its finest.

The simplicity of this approach begins to break down when we begin to handle state. There are three areas of complication:

  1. Storing stateful data in a module that you want to persist through a reload;

  2. Updating a metatable of a table that has already been created; and

  3. Managing the changes in state layout between reloads.

This post provides answers for issues 1 and 2. For issue 3, check out how Erlang / OTP handles code change.

Persisting through a reload

Parameters that you want to tweak interactively in a module may be stored as top level local variables in the module. When you reload the module these parameters will be updated. These could be constants used in your simulation, like gravity or vision distance, or parameters used to specify the environment your program is running in, like release mode vs dev mode.

Like other locals, if you store stateful data in your module as a local, it will be over ridden when you reload. For example, this could be an issue if you are using a module to hold and manage a collection of game elements (a singleton in OOP parlance). If you store the collection of game elements in that module, it will be reset whenever you reload. This could crash your running game or lead to subtle bugs.

I use two distinct approaches to ensure that this sort of stateful data persists through a reload. Either I store the data in a separate state module, which is never reloaded, or I use a macro that lets me persist data across reloads (tucking it away behind a global variable that survives the purge).

Storing the data in a separate state module gives the developer a single point to see the entire state of the running game. You can also include functions in the state module that let it reload, save, load etc.

Using the persist macro is useful when you’re storing state that should be retained within the module, whose initialization is handled by the module, and when you want strong(er) guarantees that nothing outside the module will touch the data. Keeping everything easily accessible in the state module is a temptation leaning on the doorbell of spaghetti code.

Below is an example of a persist macro, which checks to see if the local variable has previously been defined and prevents it from being overridden when the module reloads.

;; persist.fnlm -- a macro file
(fn persist [handle literal]
  ;; check to see if there is a params file at the top level
  ;; if there is and it has persist set to true persist this 
  ;; local, if not treat it like a normal local.
  (local (ok params?) (pcall (fn [] (require :params))))
  (local params (if ok params? {:persist true}))
  (if params.persist
    `(local ,handle
          (let [module# (or ... :entry-module)]
            (print (string.format "Persiting %s %s" module# ,(tostring handle)))
            (when (not _G._persist) (tset _G :_persist {}))
            (when (not (. _G._persist module#)) (tset _G :_persist module# {}))
            (when (not (. _G._persist module# ,(tostring handle)))
              (tset _G :_persist module# ,(tostring handle) ,literal))
            (. _G :_persist module# ,(tostring handle))))
    `(local ,handle ,literal)))

Updating a metatable

Metatables are one of the few tools lua provides to associate functions implicitly with data without being of the data. Like all collections in lua, a metatable is just a table. If the table is stored as a local variable at the top level of a module it will be overwritten when the module is reloaded. A new table will be created with no association with the old table.

In short, if you’re using your table as a metatable, reloading the module will not update the metatables of previously created tables.

All new tables created with the updated metatable will reference the new members, and the old tables, created with the previous metatable will reference the old members. This leads to a bunch of subtle bugs that really put a damper on interactive development with metatables.

To get hotloading working with metatables, what we need to do instead is persist the table, like we did using the persist macro, but rather than disregarding any changes, we update the members of the persisted table. This keeps the table reference intact while updating its members.

Because tables are references, when you can change the members of the table, all places where it is being referenced will now use those new members. This lets us ensure that all previously created elements and all new elements that use the metatable will be using the same metatable reference with the same members.

Below is an example of a hot-table macro, which checks to see if the local table has previously been created and if it has been it overwrites the members of the table rather than creating a new table.

(fn hot-table [handle literal]
  ;; check to see if there is a params file at the top level
  ;; if there is and it has persist set to true rather than
  ;; setting this as a local, overwrite the members of the
  ;; already existing table
  (local (ok params?) (pcall (fn [] (require :params))))
  (local params (if ok params? {:persist true}))  
  (if params.persist
    `(local ,handle
          (let [module# (or ... :entry-module)]
            (print (string.format "Replacing %s %s" module# ,(tostring handle)))
            (when (not _G._persist) (tset _G :_persist {}))
            (when (not (. _G._persist module#)) (tset _G :_persist module# {}))
            (when (not (. _G._persist module# ,(tostring handle)))
              (tset _G :_persist module# ,(tostring handle) {}))
            (each [key# value# (pairs ,literal)]
              (tset _G :_persist module# ,(tostring handle)  key# value#))
            (. _G :_persist module# ,(tostring handle))))
    `(local ,handle ,literal)))

Closing remarks

Interactive development brings joy to gamedev. Being able to poke and prod your game as its running helps you quickly iterate on the feel and appearance of your game. Working at the module level lets you work in larger chunks that are carried through between game reloads. Making sure you manage your state is key to ensuring that hot loading your modules doesn’t crash your game, or put you in an inconsistent / broken game state.

I hope the two techniques I presented here help you in your interactive gamedev journey and good luck jamming!