For more humane computing

Classes are not interactions!

Some time ago on #lisp at freenode phoe asked interesting design question. So interesting in fact, that I decided that It deserves a post on a blog as it highlights why multimethods are such a good idea.

Let's consider the following: we want to design a message passing system, and we want to use a clean table approach. That is, we will do this from scratch, avoiding mimicking existing solutions in hope that this can result in an overall better design.

In our system messages are passed asynchronously, and they either succeed (in one way or another) or fail (again, in multiple ways). We would like to keep it extendable and structured, so it is general enough to be reusable and would be easy on our little, soft brains.

This is clearly nothing new. Writing programs that are easy for a human to work with is a lot harder than simply writing programs that will just run on a machine. Generations of programmers tackled this issue and each had their own sets of solutions. I would not dare to say that I have the definite answer, but I think we can agree upon general guidelines:

  1. Given that functionality of the system remains the same, fewer elements is better than more elements.
  2. Simpler elements are in general better than complex elements.
  3. State of elements should be easy to inspect and manage. If immutability does not complicate the overall design, it is probably a good idea.

It is worth noting that none of those rules dictates a distinctive technical choice, but it will help to judge system that we are coming up with it purely on how easy to understand it is.

Let's get back to our message passing system. To start we will first consider something basic, like sending a message. But now, how we will handle the response? It is not merely the question of how we will obtain the result, but also, how we will tie it with the rest of the system. It is possible for instance to pass callback along with the message itself, but that only solves issues with asynchronous nature of the system. We still need to dispatch logic of the response. 

(let* ((message (obtain-message))
       (response-handler (lambda (response) ; async, therefore callback
                           (do-stuff response))))
  (send-message message response-handler))

  ;; alternativly, we can do this slightly differently...

(let* ((message (obtain-message))
       (response (send-message message)) ; non blocking
       (response-handler (lambda ()
                           (do-stuff response))))
  ;; this approach may be slightly more natural.
  ;; Response is now more like a future, so it should have force function.
  (attach-on-response response response-handler))

Implementing large CASE form in the do-stuff function is clearly not the greatest idea.

Phoe stumbled upon the same issue. What he imagined is API that will return an object that will contain everything needed to fully handle changes in the system. It is not a bad idea. It should be easy to debug, easier even to understand. There are multiple ways to achieve this though.

Initial idea from phoe was to have a separate response class for each request class. The response would be returned just after calling send, and then it would be possible to call a blocking generic function handle. Concrete implementation of handle function will perform any changes needed in the system. 

(defgeneric handle (response extra-context))

(defgeneric make-response (message status))

(defclass fundamental-message ()
  ())

(defclass fundamental-response ()
  ((%data :initarg :data
          :reader data)))

(defclass hello-message (fundamental-message)
  ((%target :initarg :target
            :reader target)))

(defclass success-hello (fundamental-response)
  ())

(defclass error-hello (fundamental-response)
  ())

(defmethod make-response ((message hello-message) status)
  (check-type status list)
  (let* ((response-type (first status))
         (response-data (rest status)))
    (case response-type
      (:success (make-instance 'success-hello :data response-data))
      (:error (make-instance 'error-hello :data response-data)))))

(defmethod handle ((response success-hello) extra-context)
  (format extra-context "~a~%" "Sending hello message worked."))

(defmethod handle ((response error-hello) extra-context)
  (format extra-context "~a~%" "Sending hello message failed."))

(let* ((message (make-instance 'hello-message :target 'world))
       (response-handler (lambda (response) (handle response t))))
  (send-message message response-handler) ; non blocking
  ;; I miss response-as-future approach :(
  )

The issue here is that we would have to define not just each message class but also separate response classes. In the example above I defined just two, and then got bored. Also, we are back to the callbacks approach.

But here is the main point: interactions should not be represented as classes. At least not always in Common Lisp. Instead, they can be functions. Multimethods to be exact. To demonstrate, let's consider make-response function first. 

(defgeneric make-response-with-type-and-data (message response-type response-data))

(defmethod make-response ((message fundamental-message) status)
  (check-type status list)
  (let* ((response-type (first status))
         (response-data (rest status)))
    (make-response-with-type-and-data message response-type response-data)))

(defmethod make-response-with-type-and-data ((message hello-message)
                                             (type (eql :error))
                                             data)
  (make-instance 'error-hello :data data))

(defmethod make-response-with-type-and-data ((message hello-message)
                                             (type (eql :success))
                                             data)
  (make-instance 'success-hello :data data))

It becomes quite clear that We don't have to create a separate response class for every possible request, though. We will create just one, containing both the message and a response. In the handle function we will wait for the response to arrive, and this time we will delegate any further processing to yet another function called handle-with-message-and-response. Burden of implementing this function is consequently moved to the higher level code. A complete demonstration of this approach can be seen below. 

(defgeneric handle (response extra-context))

(defgeneric fill-response (response data))

(defgeneric handle-with-message-and-response (message response-type
                                              response-data extra-context))

(defgeneric attach-to-response (response callback))

(defclass fundamental-message ()
  ())

(defclass response ()
  ((%message :initarg :message
             :reader message)
   (%response-type :initarg :response-type
                   :initform nil
                   :accessor response-type)
   (%response-data :initarg :response-data
                   :initform nil
                   :accessor response-data)
   (%lock :initform (bt:make-lock)
          :documentation "For synchronization."
          :reader lock)
   (%condition-variable :initform (bt:make-condition-variable)
                        :documentation "For synchronization."
                        :reader condition-variable)
   (%callbacks-list :initform nil
                    :initarg :callbacks-list
                    :accessor callbacks-list)))

(defmethod fill-response ((response response) data)
  (check-type data list)
  (let ((response-type (first data))
        (response-data (rest data)))
    (assert (null (eql response-type nil)))
    (bt:with-lock-held ((lock response))
      (assert (null (response-type response)))
      (setf (response-type response) response-type
            (response-data response) response-data)
      (map nil (lambda (x) (funcall x response))
           (callbacks-list response))
      (bt:condition-notify (condition-variable response)))
    response))

(defmethod attach-to-response ((response response) callback)
  (alexandria:ensure-functionf callback)
  (let ((is-not-filled t))
    (bt:with-lock-held ((lock response))
      (let ((response-type (response-type response)))
        (push callback (callbacks-list response))
        (setf is-not-filled (null response-type))))
    (unless is-not-filled
      (funcall callback response)))
  response)

(defmethod handle ((response response) extra-context)
  (bt:with-lock-held ((lock response))
    (when (null (response-type response))
      (bt:condition-wait (condition-variable response)
                         (lock response)))
    (handle-with-message-and-response (message response)
                                      (response-type response)
                                      (response-data response)
                                      extra-context)))

#|
The actual application logic is reduced to just pure essentials.
|#

(defclass hello-message (fundamental-message)
  ((%target :initarg :target
            :reader target)))

(defmethod handle-with-message-and-response ((message hello-message)
                                             (type (eql :error))
                                             data
                                             extra-context)
  (format extra-context
          "Saying hello to ~a failed :("
          (target message)))

(defmethod handle-with-message-and-response ((message hello-message)
                                             (type (eql :success))
                                             data
                                             extra-context)
  (format extra-context
          "Saying hello to ~a :)"
          (target message)))

(let* ((message (make-instance 'hello-message :target 'world))
       (response (send-message message))) ; non blocking
  (handle response t)
  ;; or if we still want to use callbacks we can...
  ;; (attach-to-response response
  ;;                     (lambda (response)
  ;;                       (handle response t)))
  )

This burden is light though. There is no boiler plate code at all!

We did not sacrifice anything. According to the rules we have outlined, our design is sound. However what is appealing to me on a more aesthetic level is how "alive" system seems to be. Instead of pulling wires, we let parts interact and magic to happen. It makes things more... fun.