Weekly Lua Post: package.preload

Lua is designed for embedding into larger host programs. Last time I talked about how to make C functions available to Lua, now I want to talk about how to bundle Lua libraries in with your host program.

First, we have to understand what it means to load a library. Most Lua libraries are packaged into modules (which I'll get into later), one module per file. You call require and pass it a module name, and it returns the module (usually as a table containing some functions).

Notably, you're not passing a file name to require, you're passing a module name. The fact that these modules map to files is just sort of coincidence. In actuality, require looks in several different places (locations on package.path, C libraries on package.cpath, and so on) for the file that contains the module. But, even before it does that, it checks if you've provided a loader for that module. You can do that by putting it in a table, package.loaders.

So let's say you have a module, Penlight, in pl.lua. You want to embed this in your app. So, you store it in your binary as a string, and provide a C function that returns the string (actually you'd use some kind of data file, but I'm keeping this simple). The first place it'll look for the loader is package.preload['pl'], so put this there:

package.preload['pl'] = function()
    local str = load_pl_from_c()
    local fn = loadstring(str)
    return fn()
end

Now, when you call require 'pl' it will execute this function, read the text of the pl library from wherever the host stored it, and run it. And, if you're testing your script outside the host program, it will still work; just don't add anything to preload and it will just read the file from the path.

It's easy to see how you could build a whole package system with this: byte-compile the files first and run them through zlib, put them all in a data file, since package.preload is a table that means you can put a metatable on it, etc. One idea I had was to tie this to curl and have it load packages from a repository over http.