Core User Manual

This manual documents the CC core software. The core is a Lisp development system used to build next-gen applications. It is composed of interconnected modules providing an exceptionally powerful development ecosystem.

1. Introduction

Welcome to the core, the primary software development ecosystem of CC.

The core provides the foundation of all applications we produce. It is optimized for rapid development of next-gen applications.

The core is highly opinionated - and these opinions are consistently hard-coded into our development practices.

For example:

Linux is the only platform worth supporting
- We make no efforts to support cross-compilation or porting of our software to devices or operating systems the developer can't fully control such as Windows/MacOS/iOS/Android. We always assume the developer is running a GNU/Linux host.
Portability is over-rated
- CL implementations (lisps) share a base language but many parts of the ANSI CL spec are left up to interpretation: threads, OS, GC, MOP, etc
- Many lisp packages attempt to account for as many lisps as possible via conditional compilation and wrappers - this clutters up the code base, is difficult to test, and is error-prone.
- Instead of accounting for multiple compilers, we realize that we only need one - SBCL currently. The core does not contain any support for alternate lisps.
- 'One true lisp' is all we need, and we're only concerned with internal consistency - we are never bound by the choices in implementation made by other lisps (although there is much to learn from them).
Introspection is under-rated
- Black boxes (proprietary codes) are disrespectful to developers. Computer systems should be transparent by nature and make minimal efforts to 'hide' internal representations of data and code.
- Never hide a part of the system, unless you have a very good reason for doing so.
Follow the Bleeding Edge
- We pull and rebuild the latest changes for all of our dependencies frequently
- We host mirrors of all our major dependencies
- Core development is always based on latest changes, can optionally downgrade for release builds where stability is valued
- incorporate pre-release APIs and our own custom patches as needed.

2. Overview

The core itself is not an application but is complex enough that it merits its own dedicated manual. There are dedicated manuals for the applications which are part of the core, so what we are concerned with here is exploring, managing, and hacking on the source code itself. We are directing our attention inwards, towards the core instead of extending our gaze away from it to see where it leads.

This manual is often technical, but occassionally theoretical in nature and intended for software developers with expertise in Common Lisp, the Linux kernel, and Emacs.

The Core is a Lisp system. Common Lisp is our language of choice and there is a significant amount of Emacs Lisp code too. The only other language present at this time is C (0.1%).

2.1. Source Code

The source for each module is located in its own directory. The STD module lives under the std/ directory, FFI modules such as ROCKSDB under the ffi/ directory and library modules like RDB under the lib/ directory. The top-level directory is dedicated to the CORE module itself.

2.2. Modules

Core modules loosely equate to a lisp system and its 'sub-systems' - for example the CORE module contains the CORE system and CORE/TESTS, CORE/USER, etc sub-systems. They are effectively a container of one or more systems.

Developers may import ASDF systems provided by a module and use the packages it contains, or use the compiled lisp images directly.

A full module system is planned which leverages the *MODULES* variable, in a similar fashion to ASDF with support for more direct usage of modules in day-to-day programming.

2.3. Features

The core contains many features which are included in the compiled image by default.

The following features have special meaning at build-time where they determine which modules get loaded into the resulting core image.

:std     ; the STD library - always required
:prelude ; the PRELUDE library - on by default
:gui     ; the GUI module - optional
:web     ; the WEB module - optional
:full    ; ALL modules

The rest of the features provided by the car are also tracked in the *FEATURES* list but should not need to be updated manually.

*features*
;; *ffi-features*
;; *core-features*

:KEYUTILS

:UBLKSRV

:SNDFILE

:SSH2

:TREE-SITTER-JSON

:PRELUDE

:USER

:TREE-SITTER-RUST

:TREE-SITTER-JAVASCRIPT

:TREE-SITTER-C

:TREE-SITTER-ALIEN

:TREE-SITTER

:SWANK

:SKEL

:CORE

:BLAKE3

:RDB

:ROCKSDB

:RFC3912

:RFC3492

:NET

CFFI-FEATURES:FLAT-NAMESPACE

CFFI-FEATURES:X86-64

CFFI-FEATURES:UNIX

:CFFI

CFFI-SYS::FLAT-NAMESPACE

:CLI

:IO

:KBD

:EVDEV

:XKBCOMMON

:BTRFSUTIL

:BTRFS

:ZSTD

:URING

:FLEXI-STREAMS

:RFC8141

:RFC6874

:RFC3986

:SPLIT-SEQUENCE

:IRONCLAD-ASSEMBLY

:IRONCLAD-SB-VM-EA

:IRONCLAD-FAST-MOD64-ARITHMETIC

:IRONCLAD-FAST-MOD32-ARITHMETIC

:BORDEAUX-THREADS

:GLOBAL-VARS

ALEXANDRIA::SEQUENCE-EMPTYP

:NAMED-READTABLES

CHIPZ-SYSTEM:GRAY-STREAMS

:SBCL-USES-SB-ROTATE-BYTE

:THREAD-SUPPORT

:CL-PPCRE

:STD

:QUICKLISP

:ASDF3.3

:ASDF3.2

:ASDF3.1

:ASDF3

:ASDF2

:ASDF

:OS-UNIX

:NON-BASE-CHARS-EXIST-P

:ASDF-UNICODE

:ARENA-ALLOCATOR

:X86-64

:MARK-REGION-GC

:64-BIT

:ANSI-CL

:COMMON-LISP

:ELF

:IEEE-FLOATING-POINT

:LINUX

:LITTLE-ENDIAN

:PACKAGE-LOCAL-NICKNAMES

:SB-CORE-COMPRESSION

:SB-LDB

:SB-PACKAGE-LOCKS

:SB-THREAD

:SB-UNICODE

:SBCL

:UNIX

2.4. Images

Core Images refer to the binary file resulting from a call to SAVE-LISP-AND-DIE. According to SBCL this file contains

enough information to restart a Lisp process later in the same state, in the file of the specified name.

See the function docs for details - there are many options available which determine how the image can be used directly or dynamically from other programs.

The images we produce are most often executable and contain the full SBCL runtime at their disposal. Note that these images can be quite large - multiple gigabytes in some cases, and so care should be taken when enabling additional build-time features.

3. Getting Started

3.0.1. Bootstrap

Bootstrapping the core can be done via an existing core image or the skel binary in a 'self-bootstrapping' fashion. See the infra project documentation for details on that process.

Bootstrapping from SBCL is quite easy too. All that is needed is to get the source, load the project skelfile and run sk-make.

# clone the source code from a mirror
git clone https://codeberg.org/c-c/core
cd core
sbcl

Then in the SBCL REPL:

(ql:quickload :skel)
(in-package :sk-user)
(init-skel)
(sk-make *skel-project* :all)

:ALL

Installation can be done from the same REPL session. By default the core binary will be written to /usr/local/bin/ and a series of symlinks will be created in the same directory - similar to Busybox binaries. Note that this may not work in a SLIME repl since sudo is called and might need input at the terminal.

(sk-make *skel-project* :install-core)

You should see some output like the following:

skel -> core -> /usr/local/bin/
homer -> core -> /usr/local/bin/
packy -> core -> /usr/local/bin/
rdb -> core -> /usr/local/bin/
organ -> core -> /usr/local/bin/
vc -> core -> /usr/local/bin/
gen -> core -> /usr/local/bin/
swm -> core -> /usr/local/bin/

Dependencies
If you do choose to build yourself without using infra, make sure you're on GNU/Linux and have the required packages installed.
- Required
  - linux-headers
  - gcc
  - sbcl
  - emacs
  - quicklisp
  - rocksdb
  - zstd
  - mercurial
  - openssl
- Optional
  - cuda
  - evdev
  - jack
  - libjpeg-turbo
  - xkbcommon
  - ublksrv
  - keyutils
  - wasmer
  - sndfile
  - libssh2
  - btrfsutils
  - podman
  - qemu
  - gstreamer
  - rustls
  - tree-sitter
  - blake3
  - liburing
  - alsa
  - ffmpeg
  - glib
  - chromaprint
  - openblas (blas/lapack)

3.0.2. Configuration

The next thing you'll want to take care of is user configuration. You can run any of the core apps without a configuration and it will install a default one for you.

The only other required development dependency which needs configuring is Emacs.

Core
The core is a single binary which dispatches on argv[0] from the CLI. Each unique symlink to the core binary is treated as a separate dispatch to a 'main' function like so:
```
(define-multi-main dispatch-core
  (progn (in-package :core-lisp)
         #+nil ...)
  (:skel (bin/skel::start-skel))
  (:homer (bin/homer::start-homer))
  (:mpk (bin/mpk::start-mpk))
  #+nil ...)
```
The current apps that read a configuration file on startup are:
- skel - project management: .skelrc
- homer - home management: .homerc
- mpk - media production kit: .mpkrc
These apps all use the same configuration object protocol from the OBJ/CONFIG package and are printed to various S-Expression. Most commonly the 'pretty' format is used which is simply a plist with implied outer parentheses:
```
:key1 val1
:key2 val2
```
Canonically that would be:
```
(:key1 val1
 :key2 val2)
```

Emacs

If you don't have your own Emacs config you can install the default CC Emacs configuration using the skel command.

Warning

this will overwrite files in your ~/.emacs.d directory:

cd emacs && skel install

default.el -> /home/ellis/.emacs.d/default.el
early-init.el -> /home/ellis/.emacs.d/early-init.el
init.el -> /home/ellis/.emacs.d/init.el
keys.el -> /home/ellis/.emacs.d/keys.el
util.el -> /home/ellis/.emacs.d/util.el
bar.el -> /home/ellis/.emacs.d/lib/bar.el
c2.el -> /home/ellis/.emacs.d/lib/c2.el
exec-path-from-shell.el -> /home/ellis/.emacs.d/lib/exec-path-from-shell.el
graph.el -> /home/ellis/.emacs.d/lib/graph.el
inbox.el -> /home/ellis/.emacs.d/lib/inbox.el
mpk.el -> /home/ellis/.emacs.d/lib/mpk.el
org-expiry.el -> /home/ellis/.emacs.d/lib/org-expiry.el
publish-init.el -> /home/ellis/.emacs.d/lib/publish-init.el
publish.el -> /home/ellis/.emacs.d/lib/publish.el
scrum.el -> /home/ellis/.emacs.d/lib/scrum.el
skel.el -> /home/ellis/.emacs.d/lib/skel.el
skt.el -> /home/ellis/.emacs.d/lib/skt.el
slime-cape.el -> /home/ellis/.emacs.d/lib/slime-cape.el
slime-company.el -> /home/ellis/.emacs.d/lib/slime-company.el
slime-repl-ansi-color.el -> /home/ellis/.emacs.d/lib/slime-repl-ansi-color.el
sxp.el -> /home/ellis/.emacs.d/lib/sxp.el
ulang.el -> /home/ellis/.emacs.d/lib/ulang.el
uml-mode.el -> /home/ellis/.emacs.d/lib/uml-mode.el
x86-lookup.el -> /home/ellis/.emacs.d/lib/x86-lookup.el

The default config supports user customizations in a file with the same name as the current user, so when the current $USER is ellis then we check for the file ellis.el and load it if it exists.

4. Testing

Each module in the core contains its own test suite located in the tests.lisp file or tests directory and isolated to its own ASDF system. The ASDF system is always the module name prefixed by /tests - for example the IO/TESTS system is responsible for testing the IO module system.

In addition to the test suite we have a set of experimental benchmarks. These are kept in the top-level bench directory.

4.1. Regression Tests

Testing the core can be done from the CLI using skel:

# skel compile tests
skel run tests

or from lisp:

(ql:quickload :core/tests)
(rt:run-all-tests)

4.2. Benchmarks

using skel:

# skel compile bench
skel run bench db-stress

or from lisp:

(ql:quickload :core/bench)
(use-package :core/bench)
(run-benchmark :db-stress)

#<PACKAGE "CORE/BENCH/DB/STRESS">

5. Contributing

The core is currently a solo project and I don't plan on accepting contributions in the near future. The core is its own private island, isolated from the rest of the Common Lisp ecosystem and in almost all cases contributions would be better directed towards other packages.

See Awesome CL for a list of projects which could use additional support, and take note the comments scattered throughout the core source code which often reference popular packages from the ecosystem.