Load source code asynchronously

[jeffayle]

This isn't a recommendation for the language, rather something I've been working on myself that I wanted to get feedback on. Currently in my wren runtime, I am using stdio to load wren modules, but everywhere else I/O is dispatched with io_uring. Loading modules synchronously keeps everything simple and has very few drawbacks. I would like it though if I could unify all I/O. I also want to keep the changes that I have to make to wren itself as generic as possible.

These are the modifications I've made to Wren 0.4.0 to make it happen:

In wren_core.wren, I added two methods to the System class. One registers a callback and the other calls all registered callbacks:

class System {
  [...]
  static attachPreimport(fn) {
    if (!__preimport) {
        __preimport = []
    }
    __preimport.add(fn)
  }

  static executePreimport(module) {
    if (!__preimport) return
    for (fn in __preimport) {
      fn.call(module)
    }
  }
}

Then I modified the compiler to call this method every time a module is imported:

static void import(Compiler* compiler)
{
  ignoreNewlines(compiler);
  consume(compiler, TOKEN_STRING, "Expect a string after 'import'.");

  // Execute pre-import method
  WrenVM* vm = compiler->parser->vm;
  Value coreModuleValue = wrenMapGet(vm->modules, CONST_STRING(vm, ""));
  ObjModule* coreModule = (void*)AS_OBJ(coreModuleValue);
  ObjString* moduleString = (void*)AS_STRING(compiler->parser->previous.value);
  const char* resolvedName =
    vm->config.resolveModuleFn(vm, compiler->parser->module->name->value,
    moduleString->value);
  Value resolvedValue = wrenNewString(vm, resolvedName);
  DEALLOCATE(vm, (char*)resolvedName);
  int resolvedConstant = addConstant(compiler, resolvedValue);
  int systemSymbol =
    wrenSymbolTableFind(&coreModule->variableNames, "System", 6);
  emitShortArg(compiler, CODE_LOAD_MODULE_VAR, systemSymbol);
  emitShortArg(compiler, CODE_CONSTANT, resolvedConstant);
  int preimportMethodSymbol =
    wrenSymbolTableEnsure(compiler->parser->vm,
      &compiler->parser->vm->methodNames,
      "executePreimport(_)", 19);
  emitShortArg(compiler, CODE_CALL_1, preimportMethodSymbol);
  emitOp(compiler, CODE_POP);

  // Load the module.
  emitShortArg(compiler, CODE_IMPORT_MODULE, resolvedConstant);
  [...]

I want my callback to deal with fully resolved module names, so I do that resolution at compile time. Since the resolution is done, I use that name in the import instruction and modify the wren_vm.c so that it doesn't try to resolve the name again:

static Value resolveModule(WrenVM* vm, Value name)
{
  // If the host doesn't care to resolve, leave the name alone.
  if (vm->config.resolveModuleFn == NULL) return name;

  ObjFiber* fiber = vm->fiber;
  ObjFn* fn = fiber->frames[fiber->numFrames - 1].closure->fn;
  ObjString* importer = fn->module->name;
  
  const char* resolved = AS_CSTRING(name);
  [...]

The last line here is the only thing i modified, i made it so that the resolved name is just the name again. This function can be cleaned up because it really has to do nothing, but I've mostly left all the checks in tact for now because it was the easiest thing to do.

On the runtime side, I moved most of my module loading logic into that preimport callback. It reads the source code from the file (because this is regular wren now, it has full access to the asynchronous I/O scheduler) and then saves it to the struct i had attached to the wren vm.

my module load function looks like this:

static WrenLoadModuleResult
load_module(WrenVM* vm, const char* module_name) {
    struct wren* wren = wrenGetUserData(vm);
    WrenLoadModuleResult res = {};
    if (wren->last_source_code) {
        res.userData = wren->last_source_code;
        res.source = res.userData;
        wren->last_source_code = NULL;
        res.onComplete = load_module_finish;
    } else {
        res.source = load_internal_module(module_name);
    }
    return res;
}
load_module_finish
    (WrenVM* vm, const char* module_name, WrenLoadModuleResult res)
{
    free(res.userData);
}

This last bit feels very roundabout. My thought was that once the source code was loaded, I could compile it by running wrenInterpret instead of saving the source code for later when import was called. This only works if importing the module doesn't try to perform I/O, which will result in wrenInterpret returning before the module is completely loaded.