Lemonodor: Lisp DLLs in ACL

March 07, 2002

Lisp DLLs in ACL

Yesterday a fellow programmer, Bob Lane, reminded me of a serious bug in ACL's ability to create Windows DLLs.

We have a natural language processor/task execution system that is written in Lisp. In order to make it easy to use from within other programs (e.g. our telematics simulator), I used ACL's facility for building Windows DLLs.

Creating a DLL around lisp is not too difficult. You really end up writing a tiny wrapper in C, which compiles to the actual DLL (for DPMA, there's about 500 lines of C that compiles to a 70K library, with all debugging info). The wrapper code makes a call that starts your lisp image running, and that's pretty much it.

Well, it's actually somewhat more complicated than that. Every lisp function that you want to be able to use from outside the DLL has to have a C wrapper.

For example, DPMA has its own task description language, which is basically a way to write special kinds of programs, which we call RAPs. There is an interpreter that runs, processing the RAPs associated with the active tasks. In the DPMA engine, there is a lisp function that runs one "step" of the interpreter. I wanted the DLL to export a method for calling that function. In the C wrapper, I wrote

int WINAPI DPMAInterpreter_Step(TDPMAInterpreterStatus *status)
{
  int success;
  dtrace(ENTER, "DPMAInterpreter_Step: &status=%d", status);
  if (!ldpma_interpreter_step)
    success = -1;
  else
    success = (*ldpma_interpreter_step)(status);
  dtrace(EXIT, "DPMAInterpreter_Step: success=%d, status=%d", success, *status);
  return success;
}

This provides an exported C function that calls the lisp function, a pointer to which is in the variable ldpma_interpreter_step.

I don't mind too much that I have to write a C wrapper around every lisp function. It's not a bad place to vet arguments from the C world and do translation of lisp return values.

Wart #1: Callbacks

However, the way that the ldpma_interpreter_step function pointer gets the right value in the first place does seem like a bit of a wart to me. It works like this:

You export an InitializeDPMA C function from your DLL, that users of the DLL have to call first thing.
InitializeDPMA starts up ACL and sends a string to the lisp image to be evaled. In my case, "(dpma-initialize)".
The dpma-initialize function has to make a foreign function call to pass an array of function pointers corresponding to the lisp functions you want to be able to call. It has to call a well-known function you export from the C wrapper.
The C function has to save away the pointers to lisp functions for use by the wrapper functions.

The lisp code for passing the function pointers back looks like

(flet ((rf (f) (ff:register-function f nil t)))
  (let* ((cb (list (rf #'DPMA-EXIT)
		   (rf #'DPMA-INTERPRETER-START)
		   (rf #'DPMA-INTERPRETER-STOP)
		   (rf #'DPMA-INTERPRETER-REGISTER-STATUS)
		   (rf #'DPMA-KNOWLEDGE-LOAD-FILE)
		   (rf #'DPMA-SHOW-REGISTER)
		   (rf #'DPMA-SHOW-GET-LEVEL)
		   (rf #'DPMA-SHOW-SET-LEVEL)
		   (rf #'DPMA-AGENDA-GET)
		   (rf #'DPMA-AGENDA-INSTALL-GOAL)
		   (rf #'DPMA-AGENDA-RESET)
		   (rf #'DPMA-ERROR-REGISTER)
		   (rf #'DPMA-WARNING-REGISTER)
		   (rf #'DPMA-PRIMITIVE-POST-INPUT-TEXT)
		   (rf #'DPMA-PRIMITIVE-REGISTER-TIME)
		   (rf #'DPMA-SKILL-SYSTEM-REGISTER)
		   (rf #'DPMA-SKILL-SYSTEM-POST-EVENT)
		   (rf #'DPMA-INTERPRETER-STEP)
		   (rf #'DPMA-KNOWLEDGE-RESET)
		   ))
	 (cb-array (make-array (length cb)
			       :element-type 'fixnum
			       :initial-contents cb)))
    (set_lisp_callbacks cb-array)))

The C code for receiving them looks like

/* Not part of the API; intended only to be used by Lisp */

void WINAPI set_lisp_callbacks(int *cb)
{
  ldpma_exit = (LDCB_EXIT) cb[0];
  ldpma_interpreter_start = (LDCB_INTERPRETER_START) cb[1];
  ldpma_interpreter_stop = (LDCB_INTERPRETER_STOP) cb[2];
  ldpma_interpreter_register_status = (LDCB_INTERPRETER_REGISTER_STATUS) cb[3];
  ldpma_knowledge_load_file = (LDCB_KNOWLEDGE_LOAD_FILE) cb[4];
  ldpma_show_register = (LDCB_SHOW_REGISTER) cb[5];
  ldpma_show_get_level = (LDCB_SHOW_GET_LEVEL) cb[6];
  ldpma_show_set_level = (LDCB_SHOW_SET_LEVEL) cb[7];
  ldpma_agenda_get = (LDCB_AGENDA_GET) cb[8];
  ldpma_agenda_install_goal = (LDCB_AGENDA_INSTALL_GOAL) cb[9];
  ldpma_agenda_reset = (LDCB_AGENDA_RESET) cb[10];
  ldpma_error_register = (LDCB_ERROR_REGISTER) cb[11];
  ldpma_warning_register = (LDCB_WARNING_REGISTER) cb[12];
  ldpma_primitive_post_input_text = (LDCB_PRIMITIVE_POST_INPUT_TEXT) cb[13];
  ldpma_primitive_register_time = (LDCB_PRIMITIVE_REGISTER_TIME) cb[14];
  ldpma_skill_system_register = (LDCB_SKILL_SYSTEM_REGISTER) cb[15];
  ldpma_skill_system_post_event = (LDCB_SKILL_SYSTEM_POST_EVENT) cb[16];
  ldpma_interpreter_step = (LDCB_INTERPRETER_STEP) cb[17];
  ldpma_knowledge_reset = (LDCB_KNOWLEDGE_RESET) cb[18];
}

where all the LDCB... things are typedefs for a pointer to a function with the appropriate arguments and return values.

Blech. You can see how this would be unwieldy if you had even a few dozen lisp functions you wanted to export from the DLL.

Wart #2: Callbacks

Yeah. Callbacks again, but this time instead of C-to-lisp, the problem is with lisp-to-C.

The DPMA C API allows people to register error handlers, which are called if something goes wrong during RAP interpretation, for example. The problem is that given a pointer to a C function, I don't see any way in ACL to directly call that function from lisp.

Any problem in computer science can be solved with another layer of indirection. -- David Wheeler

What I do is define and export functions from my C code that accept C function pointers and whatever arguments the functions expect, and call those from lisp.

void WINAPI hack_error(TDPMAErrorMethod m, const char *msg)
{
  dtrace(ENTER, "hack_error");
  (*m)(msg);
  dtrace(EXIT, "hack_error");
}

Blech again.

Finally, the Bug

I discovered that the DLL created by ACL doesn't clean up after itself properly. The TerminateLisp function that is supposed to shut down the lisp image doesn't work. According to Franz it is due to a design flaw that isn't easy to fix.

Possibly related to the problem with cleanup is a problem with re-initialization. The issue comes up when developing a Visual Basic application that makes use of the lisp DLL: The typical VB development process involves starting the VB IDE and working semi-interactively. Starting your VB app, pausing it, stopping it, editing it, and restarting. It's not entirely dissimilar from how development is done in lisp. Unfortunately, this doesn't work if your VB app uses a DLL produced by ACL.

Once you stop your lisp-dll-using VB application, you can't start it again. Your app will crash if you try (eventually). You have to completely exit the VB IDE, start it up again, and reload your project. It is exactly as annoying as if you had to exit lisp and reload your code every 5 minutes.

Franz has no solution. They say "Visual Basic is keeping the acli601.dll loaded in its memory between each run of your project and not bringing in a new one so that lisp can start approppriately. InitializeLisp is not designed to start lisp with the previous dll in Visual Basic's menmory."

They do suggest a workaround, however: instead of building a normal DLL, build a lisp OLE object. That's not necessarily a bad idea, it's just not what I want, and it doesn't assuage my annoyance at the fact that the normal DLLs are buggy.

Posted by jjwiseman at March 07, 2002 11:19 AM

Comments