UP | HOME
Land of Lisp

Zhao Wei

How can man die better than facing fearful odds, for the ashes of his fathers and the temples of his Gods? -- By Horatius.

Common-Lisp programm in large

1. Introduction

One of the problem during learning Common-lisp is how to

  • Use other function from other files.
  • How to specify dependencies between different packages including external ones.

Other language like Elixir has built in tools to handle all of this, like mix. But in CL, there is no offical support.
It seems the current mainstream is to use quicklisp + ASDF.

2. How to organize files into packages

2.1. Basics about packages

  • Packages must be registered first before their use.
    • Use package qualifier to access symbols in other packages, such as foo:bar which means a symbol bar in package foo.
    • This is true only if bar is an external symbol of foo, that is, a symbol that is supposed to be visited outside of foo.
    • A reference to an internal symbol requires the intentionally clumsier syntax foo::bar. So, use :: qualifier will let us access any symbols from the named packages.
    • All symbols in the current package can be accessed without a package qualifier.
  • How to access the name-to-symbol mappings in a packages
    • find-symbol
    • intern
  • Current package is defined in global variable *package*.
  • Keywords symbols are written with names starting with a colon “:”. They are interned in the package named KEYWORD and automatically exported.
  • Uninterned symbols are written with a leading #:.
  • Reader translates textual names into symbol objects.
  • A symbol can be accessible in two ways:
    • name-to-symbol table
      • home package
    • inherits from other packages
      • external symbols
  • 3 standard packages
    • common-lisp
    • common-lisp-user
    • keyword
      • Allow us to access keyword symbols without an explicit keyword: package qualification.
  • In the REPL buffer in SLIME you can also change packages with a REPL shortcut. Type a comma , and then enter change-package at the Command: prompt.

2.2. Examples about organize files into different files as packages

  1. Example-01
    1. we could group useful common functions into another package as library

      (defpackage :com.gigamonkeys.text-db
        (:use :common-lisp)
        (:export
         :open-db
         :save
         :store))
      
      • The :use command within defpackage is used to inherit packages.
        • When we inherit packages, all of the external symbols in the inherited packages will be treated as internal symbols in the package we are defining.
      • The :export clause specifies names that will be external in COM.GIGAMONKEYS.TEXT-DB and thus accessible in package that :use it.
        • Any package that inherits com.gigamonkeys.text-db can access these symbols without a package qualifier.
    2. our application

      (defpackage :com.gigamonkeys.email-db
        (:use
         :common-lisp
         :com.gigamonkeys.text-db
         :com.acme.text)
        (:import-from :com.acme.email :parse-email-address)
        (:shadow :build-index)
        (:shadowing-import-from :com.gigamonkeys.text-db :save))
      
      (in-package :com.gigamonkeys.email-db)
      
      • Switch between packages using in-package.
        • So, when we want to intern symbols in a particular package, we need to first switch to that package via in-package form.
      • Now, code written in COM.GIGAMONKEYS.EMAIL-DB can use the unqualified names to refer to the exported symbols from both COMMON-LISP and COM.GIGAMONKEYS.TEST-DB.
      • :import-from is for scenario in which we just want to import only one function from a certain package.
      • About shadow
        • :shadow is the opposite of :import-from, in which it specifies the only function we don’t want to use. So we shadow it. The result is the symbol build-index from other packages will be ignored.
        • :shadowing-import-from solve the case in which there are two packages with same symbols and we want to shadow one of them (solve the ambiguity).
    3. How to export all symbols from a package

      ;; Suppose there is package FOO
      ;; Once the package is created, and all symbols in it created, e.g., by loading your code that implements the package
      ;; We can export any symbols
      ;; (do-all-symbols (sym (find-package :foo)) (export sym))
      (let ((pack (find-package :foo)))
        (do-all-symbols (sym pack)
          (when (eql (symbol-package sym) pack)
            (export sym))))
      
  2. use-package commands is very similar to :use within defpackage

    ;; With only one argument, use-package will add my-package-2 to the use list of the current package:
    (use-package my-package-2)
    
    ;; With two arguments, use-package will add my-package-2 to the use list of my-package-1:
    (use-package my-package-1 my-package-2)
    
    • We can get a package’s use list by calling package-use-list.

      (package-use-list :cl-user)
      (#<PACKAGE "COMMON-LISP"> #<PACKAGE "SB-ALIEN"> #<PACKAGE "SB-DEBUG">
                 #<PACKAGE "SB-EXT"> #<PACKAGE "SB-GRAY"> #<PACKAGE "SB-PROFILE">)
      
  3. Example-02
    1. We will work with the following files:
      • utilities.lisp, it will contains some of our own user-defined functions.
      • quant.lisp, it will does some calculations and makes use of utilities.lisp.
      • main.lisp, a file that references the two above.
      • packages.lisp, a file that combines the above and brings everything together into a cohesive whole so that it can be run as one program.
    2. In utilities.lisp

      (in-package :utilities)
      
      (defun list-sum (my-list)
        (reduce #'+ my-list))
      
    3. In quant.lisp

      (in-package :quant)
      (ql:quickload 'cl-csv)
      
      ;; Change the below path to where you sae your utilities file:
      (load "./utilities.lisp")
      
      ;; Change the below path to where you save your csv file:
      (defparameter my-filename "./my-csv.csv")
      (defparameter csv-file (cl-csv:read-csv (pathname my-filename)))
      (defparameter my-sum (list-sum (list 1 2 3 4 5)))
      
      • Notice, we reference the list-sum function in the utilities package without a qualifier.
      • In contrast, we reference the read-csv function with a package qualifier. Why, see below.
    4. In main.lisp

      (in-package :main)
      
      ;; Change the below path to where you save your quant file:
      (load "./quant.lisp")
      
      (print (quant::list-sum (list 1 2 3 4 5 6 7 8 9 10)))
      (print csv-file)
      (print quant::my-sum)
      
      • It loads the above quant.lisp file, which means it also indirectly loads utilities.lisp.
      • Notice package qualifier with the double colon for list-sum
      • Notice the csv-file from quant.lisp is directly accessed without qualifier.
    5. In package.lisp

      (in-package :cl)
      
      (defpackage :utilities
        (:use :cl)
        (:export :list-sum))
      
      (defpackage :quant
        (:use :cl :utilities)
        (:export :csv-file))
      
      (defpackage :main
        (:use :cl :quant))
      
      ;; Change filepath to where you save your main.lisp:
      (load "./main.lisp")
      
      • We define our 3 packages not in their own files.
        • Note how we segregated package definitions from the actual code we write. This is because we need to define our packages in order, so it is easier to do this together in one place.
      • Notice the this file starts with (in-package :cl) to mak sure we can access it since defpackage is defined in the CL.
    6. Summary
      • All of our files shoud have an in-package form at the top.
      • We can run our program by simply loading and running package.lisp file. (This is how we construct and run our project if we only use plain lisp files as packages. There is a more convonient way describe in ASDF.)
      • It is different from asdf: Another thing to focus on is the difference between a system (as in asdf) and a package. These come together in asdf’s package-inferred-systems, which infer dependencies between files using the package dependencies, at the cost of having a separate package for each file.

3. ASDF – an improved way to use packages

3.1. What is system defined by ASDF

  • A system is a collection of Lisp files that together constitute an application or a library, and that should therefore be managed as a whole.
  • A system definition describes which source files make up the system, what the dependencies among them are, and the order they should be compiled and loaded in.

3.2. How to write a system definition

  1. The kind of files in system
    1. Normal .lisp file
    2. Special .lisp file – package or packages.lisp
      The package/packages.lisp file let us define and use different functions across different packages(files).
      • The pupose of this is to seperate different functions into different namespaces to make them together without conflict.
      • Notice, the package.lisp only let us use different functions from other packages(import) and expose functions(export). It doesn’t automatically download packages.
      • So, if we only use package.lisp in our project. It will NOT be able to
        • Download dependencies.
        • Recompile project to update changes from different files convoniently.
      • The .asd file helps us to solve above problems.
    3. The .asd file – defines our system.
      The .asd file helps us to
      • Solve dependencies. For example, to download missing libraries(packages) from online. Again, packages.lisp only describes what functions are imported and exported, nothing else.
        • It is like “using” or “import” at the beginning of files in other programming language.
        • How the packages(libraries or assemblies) are used to support our project is defined in “.asd” file (like CMakeLists.txt in C++ or .csproj in C#) with extra feature: it helps us to automatically download those dependencies if they are missing.
  2. The general step:
    1. Write whatever function or code in normal .lisp files as normal but with addition “(in-package :<package-name>)”. For example
      • In a tools.lisp file

        (in-package :tools)
        
        (defun another-function ()
          (print "from tool"))
        
      • In a file foobar.lisp file

        (in-package :foobar)
        
        (defun some-function ()
          (tools:another-function)
          (print "hello world1"))
        
    2. Import and export different function from different packages by defining them in “package/packages.lisp” file. Such as:

      (defpackage :tools
        (:use #:cl)
        (:export #:another-function))
      
      (defpackage :foobar
        (:use #:common-lisp #:tools)
        (:export
         #:some-function))
      
    3. Define dependencies and source files of our system.
      In a “.asd” file

      (asdf:defsystem "foobar"
        :depends-on (#:vecto)
        :components (
                     (:file "package")
                     (:file "tools")
                     (:file "foobar")))
      

3.3. How to use/update a system

  1. Create the system from above example
    • As we installed SLIME in emacs, we could just slime-load file by “C-c C-l” to load the .asd file.
    • During loading, it will install missing dependencies automatically.
  2. Use the system or update system after files are changed
    • In SLIME REPL, just eval (ql:quickload "foobar") or re-eval a updated specific function.
    • Then, invoke the functions from that package: (foobar:some-function).

4. Use quickproject to setup a common-lisp project quickly

Steup a ASDF compatible project could be speed up by using quicklisp

  • It helps us to setup a ASDF project for us.
  • Example

    (ql:quickload "quickproject")
    (quickproject:make-project "~/code/common-lisp-programming/common-lisp-in-action/leetcode"
                               :depends-on '(trivia))
    
    • This will create a leetcode system as our starting point.

5. References