1. tasks

1.1. core

Archived entries from file /home/ellis/comp/org/plan/tasks/core.org

1.1.1. DONE dot 0.1 [1/1]   lisp dot

  • State "TODO" from [2023-12-20 Wed 18:27]
  • State "DONE" from "NOPE" [2024-08-16 Fri 22:38]
  • implemented in dat/dot.lisp
  1. NOPE PARSE-SPECIALIZER-USING-CLASS
    • State "TODO" from [2023-12-20 Wed 18:28]
    • experimental SB-MOP generic, may be useful in places where we would typically use (eql OBJECT) as a specializer for GRAPH.

1.1.2. DONE migrate to sb-grovel [9/9]   lisp ffi

  • State "TODO" from [2023-11-27 Mon 23:42]
  • State "DONE" from "TODO" [2024-02-04 Sun 20:30]

:ID: bb365025-e5e7-4407-acba-32e30d1a245a

lisp/ffi/

  1. DONE blake3
    • State "TODO" from [2023-11-29 Wed 19:00]
    • State "DONE" from "TODO" [2023-12-09 Sat 01:57]

    :ID: 656c2683-8780-407c-b955-3defc4959595

  2. DONE bqn
    • State "TODO" from [2023-11-29 Wed 19:00]
    • State "DONE" from "TODO" [2023-12-09 Sat 01:57]

    :ID: b4757746-525b-49e4-b8ac-677375867d8f

  3. DONE btrfs
    • State "TODO" from [2023-11-29 Wed 19:00]
    • State "DONE" from "TODO" [2023-12-09 Sat 01:57]

    :ID: b3b411fe-6b2d-4338-9e23-83536336a487

  4. DONE k
    • State "TODO" from [2023-11-29 Wed 19:00]
    • State "DONE" from "TODO" [2023-12-09 Sat 01:57]

    :ID: 8a463084-efa0-4535-8239-b1c4955bde4c

  5. DONE rocksdb
    • State "TODO" from [2023-11-29 Wed 19:00]
    • State "DONE" from "TODO" [2023-12-09 Sat 01:57]

    :ID: 2d4d70af-228f-424c-8374-9fc8ed5c6f92

  6. DONE tree-sitter
    • State "TODO" from [2023-11-29 Wed 19:00]
    • State "DONE" from "TODO" [2023-12-09 Sat 01:57]

    :ID: 0cdf3412-cb00-4069-8e1b-b49c736377cf

  7. DONE uring
    • State "TODO" from [2023-11-29 Wed 19:00]
    • State "DONE" from "TODO" [2023-12-09 Sat 01:57]

    :ID: 45a52407-ccaf-4ea1-99c4-d6f7ba0b2731

  8. DONE zstd
    • State "TODO" from [2023-11-29 Wed 19:00]
    • State "DONE" from "TODO" [2023-12-09 Sat 01:57]

    :ID: 6ade8dd8-83b1-470d-a0da-a352ce4c2d06

  9. DONE xkb
    • State "TODO" from [2023-11-29 Wed 19:27]
    • State "WIP" from "TODO" [2023-12-10 Sun 22:47]
    • State "DONE" from "WIP" [2024-02-04 Sun 20:30]

    :ID: af778a5f-4834-4f0c-ae53-36d35cb98d8c

    • State "TODO" from [2024-08-13 Tue 22:06]

1.1.3. DONE writebatch

  • State "TODO" from [2024-08-16 Fri 22:56]
  • State "WIP" from "TODO" [2024-08-19 Mon 20:01]

CLOCK: [2024-08-19 Mon 20:01]--[2024-08-19 Mon 21:03] (1:02)

CLOCK: [2024-08-20 Tue 19:41]--[2024-08-20 Tue 20:16] (0:35)

  • State "DONE" from "WIP" [2024-08-20 Tue 20:19]
  1. file:///usr/local/include/rocksdb/c.h#org408b022
    • Refiled on [2024-08-18 Sun 21:44]

1.1.4. DONE logger

  • State "TODO" from [2024-08-16 Fri 22:56]
  • State "DONE" from "TODO" [2024-08-18 Sun 22:00]

1.1.5. DONE merge-operator callbacks   rocksdb

  • State "TODO" from [2024-08-10 Sat 22:21]
  • State "WIP" from "TODO" [2024-08-14 Wed 21:50]
  • State "DONE" from "WIP" [2024-08-16 Fri 22:52]
  • Note taken on [2024-08-16 Fri 22:52]
    still needs testing with actual data

1.1.6. DONE compactionfilter callbacks   rocksdb

  • State "TODO" from [2024-08-14 Wed 21:50]
  • State "WIP" from "TODO" [2024-08-14 Wed 21:50]

CLOCK: [2024-08-18 Sun 18:51]--[2024-08-18 Sun 19:46] (0:00)

  • State "DONE" from "WIP" [2024-08-18 Sun 19:47]

https://github.com/facebook/rocksdb/wiki/Compaction-Filter

(rocksdb-compactionfilterfactory-create nil (alien-sap (alien-callable-function 'rocksdb-destructor))
                                        (alien-sap (alien-callable-function 'rocksdb-create-compaction-filter-never))
                                        (alien-sap (alien-callable-function 'rocksdb-name)))

1.1.7. DONE comparator callbacks   rocksdb

  • State "WIP" from [2024-08-14 Wed 21:51]
  • State "DONE" from "WIP" [2024-08-18 Sun 19:47]

1.1.8. DONE associative-merge implementation

  • Refiled on [2024-08-18 Sun 19:48]
  • State "DONE" from "TODO" [2024-11-07 Thu 21:45]
  • need to figure out what delete-value requires
    • simply passes state + value and value-length to a function
    • when the value is null it doesn't exist in db (noop) ?
    • unclear when the merge op actually runs in relation to deletion op - before or after?
  • https://docs.rs/rocks/latest/rocks/merge_operator/index.html
  1. DONE concat-merge
    • State "TODO" from [2024-08-18 Sun 20:42]
    • State "DONE" from "TODO" [2024-11-07 Thu 20:41]
    • Note taken on [2024-11-07 Thu 20:41]
      well it-s about time.. should have known the SUCCESS arg was the return value that the docs referenced..
  2. DONE index-merge
    • State "TODO" from [2024-08-18 Sun 20:44]
    • State "WIP" from "TODO" [2024-11-07 Thu 20:59]
    • State "DONE" from "WIP" [2024-11-07 Thu 21:44]

1.1.9. DONE incf/decf-merge

  • State "TODO" from [2024-08-18 Sun 20:43]
  • State "TEST" from "TODO" [2024-11-07 Thu 21:00]
  • State "TBD" from "TEST" [2024-11-07 Thu 21:44]
  • Note taken on [2024-11-07 Thu 21:44]
    use the STATE param for this.
  • State "DONE" from "TBD" [2024-12-28 Sat 23:00]
  • Note taken on [2024-12-28 Sat 23:00]
    see index-merge

1.1.10. DONE define-merge-operator   macro

  • Refiled on [2024-08-18 Sun 19:53]
  • State "TODO" from "TODO" [2024-08-18 Sun 22:01]
  • State "TEST" from "TODO" [2024-08-18 Sun 22:02]

CLOCK: [2024-08-21 Wed 20:34]--[2024-08-21 Wed 21:13] (0:39)

  • State "DONE" from "TEST" [2024-08-21 Wed 21:21]
  • Note taken on [2024-08-21 Wed 21:21]
    will need to do actual tests merging key/vals when we implement associative merge API
  • define macro which eases the process of making custom merge operators
(define-merge-operator :incf nil
  :full nil
  :partial nil)
(create-incf-mergeoperator)
#<SB-ALIEN-INTERNALS:ALIEN-VALUE :SAP #X75D288000C30 :TYPE (*
                                                            (SB-ALIEN:STRUCT
                                                             ROCKSDB::ROCKSDB-MERGEOPERATOR-T))>

1.1.11. DONE zstd

  • State "PROJECT" from [2024-09-18 Wed 21:50]
  • Note taken on [2024-09-19 Thu 21:57]
    can probably improve error handling but don't need to address until we are well under way with IO/FLATE.
  • State "DONE" from "PROJECT" [2024-11-02 Sat 23:33]
  1. DONE zstd cstream tests
    • State "DONE" from "TODO" [2024-09-18 Wed 21:50]
    • v2 api
  2. DONE zdict
    • State "TODO" from [2024-09-19 Thu 21:57]
    • Note taken on [2024-10-04 Fri 20:38]
      well this is annoying, there is one function which requires passing struct by value: ZDICTfinalizeDictionary
    • State "DONE" from "TODO" [2024-11-02 Sat 23:33]
    • A few structs that we can't grovel
    • A bunch of functions for pre-training a dictionary

1.1.12. DONE reference impls

  • State "RESEARCH" from [2024-09-19 Thu 22:00]
  • State "DONE" from "RESEARCH" [2024-09-19 Thu 23:19]

1.1.13. DONE retire x.lisp   lisp skel

  • State "TODO" from [2024-07-09 Tue 22:24]
  • State "TBD" from "TODO" [2024-07-27 Sat 02:41]
  • State "TODO" from "TBD" [2024-08-16 Fri 22:58]
  • Note taken on [2024-09-16 Mon 21:52]
    actually let's move it to infra/scripts..
  • Note taken on [2024-09-16 Mon 22:10]
    on second look we really should start afresh lol
  • State "DONE" from "TODO" [2024-09-16 Mon 22:10]

with skel taking on more responsibility we no longer need a dedicated build tool.

  • migrate all functionality to skelfile
  • cleanup readme with installation instructions (with and without infra)
  • easy to break the skel binary and overwrite a LKG version, preventing future builds.
    • just be more careful
    • sanity checks
    • repurpose x.lisp to recover from situations like these specifically

1.1.14. DONE implement globals *cli* *opts* *args* etc

  • State "TODO" from [2024-07-27 Sat 02:39]
  • State "DONE" from "TODO" [2024-07-27 Sat 02:40]

1.1.15. DONE opt/arg parsing

  • State "FIXME" from [2024-08-21 Wed 21:23]
  • State "WIP" from "FIXME" [2024-09-11 Wed 18:30]
  • State "TEST" from "WIP" [2024-09-14 Sat 00:25]
  • State "DONE" from "TEST" [2024-10-15 Tue 19:38]
  • Note taken on [2024-10-15 Tue 19:39]
    took far longer than expected to track this down - root cause was the zerop function. should be in much better shape now.
  • do not need to handle multiple commands
    • what do we want to do about subcommands?
  • do not need to support 'global' option for commands
  • use cli-parse to test.

1.1.16. DONE multipoint-images   lisp bin sys

  • State "TODO" from [2024-07-28 Sun 21:01]
  • State "WIP" from "TODO" [2024-09-14 Sat 17:13]
  • State "DONE" from "WIP" [2024-09-14 Sat 18:49]
  • multiple entrypoint lisp images
  • create multiple links to core lisp image
  • support multiple low-level methods?
    • naive implementation is a single entry point with a switch dispatched on arg0.
    • could also create a thin shim in Rust/Zig and call the appropriate entrypoint after loading the core.
  • for now let's put this in lib/cli so that we can use defmain

1.1.17. DONE make-sql-data-frame

  • State "FIX" from [2024-08-05 Mon 21:13]
  • State "DONE" from "FIX" [2024-08-07 Wed 21:05]
  • finish implementation

1.1.18. DONE git   vc git

  • State "TODO" from [2023-12-16 Sat 00:55]
  • State "DONE" from "TODO" [2024-08-10 Sat 01:34]

1.1.19. DONE hg   vc hg

  • State "TODO" from [2023-12-16 Sat 00:56]
  • State "DONE" from "TODO" [2024-08-10 Sat 01:34]

1.1.20. DONE eval rules from CLI

  • State "TODO" from [2023-12-16 Sat 00:56]
  • State "DONE" from "TODO" [2024-08-05 Mon 21:06]
  • need new command or flag for rules
  • evaluate forms instead of looking in skelfile

1.1.21. DONE refactor system definition

  • State "TODO" from [2023-12-13 Wed 15:17]
  • State "DONE" from "TODO" [2023-12-13 Wed 19:38]
  • remove package-inferred
  • minimize number of systems
  • use pkg.lisp files

1.1.22. DONE cleanup fu

  • State "TODO" from [2023-12-13 Wed 19:39]
  • State "DONE" from "TODO" [2025-01-18 Sat 20:06]

1.1.23. DONE CLTL2 impl

  • State "TODO" from [2024-02-29 Thu 16:09]
  • State "DONE" from "TODO" [2025-01-18 Sat 20:07]

1.1.24. DONE list-package-symbols, etc

  • State "TODO" from [2024-02-29 Thu 16:09]
  • State "DONE" from "TODO" [2024-08-07 Wed 21:07]

1.1.25. DONE std/tests refactor

  • State "TODO" from [2024-07-27 Sat 02:42]
  • State "DONE" from "TODO" [2024-10-15 Tue 19:45]
  • a bit too large, time to shake it up
  • need separation of concerns (files) for task-pool/threaded tests in particular

1.1.26. DONE tasks

  • State "TODO" from [2024-08-10 Sat 01:29]
  • Note taken on [2024-10-15 Tue 19:45]
    currently includes jobs, supervisors, task-pool, kernel - a bit overloaded
  • State "DONE" from "TODO" [2025-01-18 Sat 20:07]

1.1.27. DONE more sb-alien utils   ffi

  • State "TODO" from [2024-09-19 Thu 23:21]
  • State "DONE" from "TODO" [2025-01-18 Sat 20:08]

1.1.28. NOPE uri   net

  • State "TODO" from [2023-12-17 Sun 01:00]
  • Note taken on [2023-12-20 Wed 18:26]
    the franz impl doesn't play well with SBCL, at least with our hacks. I'm not sure how faithfully I want to follow the RFCs anyhow.. maybe we go for the dead simple approach and focus on MOP for a path and URI protocol.
  • mostly just need the tests and to finish porting the Franz implementation

1.1.29. DONE db   db

[2024-09-16 Mon 14:41] <- org graph edges

  • State "TODO" from [2023-12-20 Wed 18:26]
  • State "DONE" from "TODO" [2024-08-05 Mon 21:08]
  • Note taken on [2024-08-05 Mon 21:08]
    XDB2 is no longer used - instead provide a simple abstract API which builds up to high-level systems like RDB
  • ported from XDB2

1.1.30. DONE aggregate physical plan

  • State "DONE" from "TODO" [2024-08-03 Sat 01:00]

1.1.31. DONE Physical Expressions

  • State "TODO" from [2024-08-02 Fri 15:53]
  • State "TODO" from [2024-08-02 Fri 15:53]
  • State "DONE" from "TODO" [2024-08-03 Sat 01:00]

1.1.32. DONE crew   proto

  • State "TODO" from [2023-12-17 Sun 01:01]
  • State "DONE" from "TODO" [2024-08-10 Sat 01:31]

1.1.33. DONE http   proto

  • State "TODO" from [2023-12-17 Sun 01:01]
  • State "DONE" from "TODO" [2024-08-05 Mon 21:07]

1.1.34. DONE dns   proto

  • State "TODO" from [2023-12-17 Sun 01:01]
  • State "DONE" from "TODO" [2024-08-05 Mon 21:07]

1.1.35. DONE tlv   codec

  • State "TODO" from [2023-12-17 Sun 01:01]
  • State "DONE" from "TODO" [2024-08-07 Wed 21:08]

1.1.36. DONE osc   codec

  • State "TODO" from [2023-12-17 Sun 01:03]
  • State "DONE" from "TODO" [2024-08-07 Wed 21:08]

1.1.37. DONE Woo/+

  • State "RESEARCH" from [2024-08-10 Sat 01:23]
  • State "DONE" from "RESEARCH" [2025-01-18 Sat 20:08]

1.1.38. DONE fixture api

  • State "TODO" from [2024-08-13 Tue 22:22]
  • State "DONE" from "TODO" [2025-01-18 Sat 20:09]

1.1.39. DONE temporary files and directories

  • Refiled on [2024-08-28 Wed 21:35]

CLOCK: [2024-08-31 Sat 20:39]--[2024-08-31 Sat 22:14] (1:35)

  • State "DONE" from "TODO" [2024-08-31 Sat 22:14]

1.1.40. DONE org graph edges

  • State "TODO" from [2024-08-27 Tue 18:20]
  • Note taken on [2024-09-14 Sat 16:30]
    dblock will come later. for now we are going with the same strategy as org-super-links - using a drawer. instead of "BACKLINKS" our drawer is called simply LINKS.
  • State "WIP" from "TODO" [2024-09-14 Sat 16:31]

CLOCK: [2024-09-17 Tue 19:50]--[2024-09-17 Tue 20:20] (0:30)

CLOCK: [2024-09-21 Sat 13:56]--[2024-09-21 Sat 17:44] (3:48)

CLOCK: [2024-09-25 Wed 18:54]--[2024-09-25 Wed 19:14] (0:20)

  • State "DONE" from "WIP" [2025-02-16 Sun 19:18]
  • Note taken on [2025-02-16 Sun 19:18]
    initial implementation up and running
  • list of links between other headings
    • <= links to this item
    • => links from this item
    • == siblings
    • ^= parent
    • v= child
  • how should it look when exported?
  • org-graph-link-use-timestamp
  • org-graph-link properties? edge properties?
    • keyvals :key val - should probably align with whatever we use for CONSTRAINTS, for now read as sxp
  • EXPORTEDGES property?
* example
:PROPERTIES:
:ID:       id1
:CUSTOM_ID: demo
:END:
:EDGES:
<- bar
[2024-09-14 Sat 22:37] <- bar
[2024-09-14 Sat 22:50] :key val :foo (bar baz) <- bar
[2024-09-14 Sat 22:37] -> foo
[2024-09-14 Sat 22:39] -- sibling
[2024-09-14 Sat 22:39] >> parent
[2024-09-14 Sat 22:39] << child
:END:

1.1.41. DONE org-follow-location   org

  • State "TODO" from [2024-08-21 Wed 20:37]

CLOCK: [2024-08-29 Thu 18:24]--[2024-08-29 Thu 18:51] (0:27)

  • State "WIP" from "REFILE" [2024-08-29 Thu 18:24]
  • State "DONE" from "WIP" [2024-08-29 Thu 23:46]
  • follow LOCATION property of heading at point
  • respect parent LOCATION, buffer local settings, etc

1.1.42. DONE org-set-location

  • State "TODO" from [2024-08-29 Thu 23:46]
  • State "DONE" from "TODO" [2024-09-08 Sun 12:23]

similar to org-set-effort, to be called automatically on capture.

  • should prompt for directory then file - skip file with C-u or with RET
    • simpler to just prompt for value with completion using org-read-property-value

1.1.43. DONE flate [0/0]

  • State "TODO" from [2024-09-19 Thu 21:55]
  • State "PROJECT" from "TODO" [2024-09-19 Thu 21:55]
  • State "DONE" from "PROJECT" [2025-01-18 Sat 20:48]

1.1.44. DONE review GNU stow API

  • State "TODO" from [2023-12-24 Sun 19:32]
  • State "DONE" from "TODO" [2024-08-05 Mon 21:06]

homepage 'symlink farm manager' rust impl: https://github.com/qboileau/rstow

1.1.45. DONE consolidate with skel   skel

  • State "TODO" from [2023-12-24 Sun 19:32]
  • State "DONE" from "TODO" [2025-01-18 Sat 20:49]

1.1.46. DONE cli tests crashing

  • State "FIXME" from "TODO" [2024-09-18 Wed 12:25]
  • State "DONE" from "FIXME" [2025-01-18 Sat 20:49]

1.1.47. DONE :scripts regression   infra

  • State "DONE" from "TODO" [2025-01-18 Sat 20:49]

1.1.48. NOPE readline   cli

  • State "TODO" from [2024-08-10 Sat 22:23]
  • Note taken on [2025-04-26 Sat 20:22]
    we have removed readline in favor of the lisp-native cli/linedit.

1.1.49. DONE impl rocksdb/slicetransform callbacks

  • State "TODO" from [2024-08-19 Mon 16:17]
  • New deadline from "[2024-08-27 Tue]" on [2024-08-27 Tue 17:30]
  • Refiled on [2024-08-27 Tue 17:31]
  • New deadline from "[2024-09-07 Sat]" on [2024-11-02 Sat 23:34]
  • State "TEST" from "TODO" [2024-11-03 Sun 00:23]
  • State "DONE" from "TEST" [2025-02-26 Wed 18:55]

rocksdb/slicetransform.lisp

1.1.50. DONE OCI projects [4/4]   pod packy skel

  • State "RESEARCH" from [2024-08-01 Thu 12:35]
  • State "DONE" from "RESEARCH" [2025-04-26 Sat 20:27]
  • Note taken on [2025-04-26 Sat 20:27]
    A bit too much for us to consider right now - ultimately I do not feel that OCI containers are a one-size-fits-all solution for image-based development.
  1. DONE oras
    • State "RESEARCH" from [2024-08-01 Thu 12:36]
    • State "DONE" from "RESEARCH" [2025-04-26 Sat 20:29]
  2. DONE skopeo
    • State "RESEARCH" from [2024-08-01 Thu 12:36]
    • State "DONE" from "RESEARCH" [2025-04-26 Sat 20:29]
  3. DONE zot
    • State "RESEARCH" from [2024-08-01 Thu 12:36]
    • State "DONE" from "RESEARCH" [2025-04-26 Sat 20:29]
  4. DONE sigstore
    • State "RESEARCH" from [2024-08-01 Thu 12:30]
    • State "DONE" from "RESEARCH" [2025-04-26 Sat 20:29]

1.1.51. DONE PMem.io

  • State "RESEARCH" from "TODO" [2024-09-03 Tue 11:22]
  • State "DONE" from "RESEARCH" [2025-01-17 Fri 18:19]
  • consider use cases in Lisp images - pre-loading and coordinating shared fasls and core images
  • no longer maintained by intel

1.1.52. NOPE encrypted btrfs luks/dm-crypt

  • State "RESEARCH" from "TODO" [2024-09-29 Sun 01:57]

1.1.53. DONE XDP

  • State "RESEARCH" from [2024-09-20 Fri 22:34]
  • State "DONE" from "RESEARCH" [2025-04-26 Sat 20:39]

A Gentle Introduction to XDP | Datadog GitHub - xdp-project/xdp-tools: Utilities and example programs for use with XDP

1.1.54. DONE DPDK

  • State "RESEARCH" from [2024-09-20 Fri 22:35]
  • State "DONE" from "RESEARCH" [2025-04-26 Sat 20:41]

Home - DPDK GitHub - DPDK/dpdk: Data Plane Development Kit

By leveraging DPDK libraries, developers can create optimized packet processing paths, manage timers for executing functions asynchronously, and utilize a wide range of drivers and libraries tailored for fast packet processing.

librteeal – Environment Abstraction Layer: Provides the foundational API for DPDK, facilitating access to hardware resources such as memory, timers, and logs.

librtemempool – Memory Pool Manager: Manages memory pools for efficient and speedy packet handling.

librtering – Ring Buffer Manager: Implements lock-free FIFO queues, enabling high-speed communication between various DPDK components.

librtembuf – Packet Buffer Management: Handles packet buffers, which are crucial for packet transmission and reception.

librteethdev – Ethernet Device API: Offers an API for configuring and querying Ethernet devices. It supports various operations, including sending and receiving packets.

librtenet – Network Helper Library: Provides helper APIs for dealing with network protocols.

librteipfrag – IP Fragmentation and Reassembly: Handles fragmentation and reassembly of IP packets, supporting both IPv4 and IPv6.

librtekni – Kernel Network Interface: Facilitates communication between DPDK applications and the Linux kernel networking stack, primarily used for debugging or interfacing with existing Linux network services.

1.1.55. DONE eBPF   net

  • State "RESEARCH" from [2024-08-21 Wed 22:40]
  • State "DONE" from "RESEARCH" [2025-04-26 Sat 20:47]

https://docs.kernel.org/bpf/standardization/instruction-set.html#bpf-instruction-set-architecture-isa https://mostlynerdless.de/blog/2023/12/31/hello-ebpf-developing-ebpf-apps-in-java-1/ https://github.com/iovisor/bcc GitHub - mikeroyal/eBPF-Guide bpftrace/tools/README.md at master · bpftrace/bpftrace · GitHub

  • bpftrace seems quite powerful, will be useful for sysmon activities

where would ebpf support go?

probably io.. but maybe also log, net.. syn?

1.1.56. DONE batman-adv   net

  • State "RESEARCH" from [2024-08-23 Fri 21:16]
  • State "DONE" from "RESEARCH" [2025-03-12 Wed 17:15]

Doc-overview - batman-adv - Open Mesh

1.1.57. NOPE libyear   lisp skel

  • State "TODO" from [2024-08-01 Thu 12:27]

Not that useful in our current environment (NIH)

1.1.58. DONE Level 1

  • State "TODO" from [2025-05-10 Sat 22:34]
  • State "DONE" from "TODO" [2025-05-12 Mon 22:56]

1.1.59. DONE Level 2

  • State "TODO" from [2025-05-10 Sat 22:34]
  • State "WIP" from "TODO" [2025-05-12 Mon 22:56]
  • State "DONE" from "WIP" [2025-05-14 Wed 18:05]

1.1.60. DONE Level 3

  • State "TODO" from [2025-05-10 Sat 22:34]
  • State "DONE" from "TODO" [2025-05-18 Sun 20:30]

1.1.61. DONE LAPACK (f77blas.h)

  • State "TODO" from [2025-05-18 Sun 21:07]
  • State "WIP" from "TODO" [2025-05-18 Sun 21:08]

CLOCK: [2025-05-18 Sun 21:08]--[2025-05-18 Sun 23:08] (2:00)

  • State "DONE" from "WIP" [2025-05-19 Mon 11:52]
  • Note taken on [2025-05-24 Sat 18:55]
    this does NOT include the lapack.h functions

1.1.62. DONE LAPACK codegen

  • State "TODO" from [2025-05-24 Sat 18:55]
  • State "DONE" from "TODO" [2025-05-25 Sun 22:29]
  • Note taken on [2025-05-25 Sun 22:30]
    see lapack-ffi.lisp

lapack.h is massive and impractical to attempt lisp bindings from scratch.

instead, our current approach is based on tree-sitter:

(tree-sitter:load-tree-sitter)
(tree-sitter:load-tree-sitter-alien)
(tree-sitter:load-tree-sitter-c)
#P"/usr/local/lib/libtree-sitter-c.so"
(syn/ts::ts-file-query :c #p"/usr/include/openblas/lapack.h" "(declaration)")
#<SB-ALIEN-INTERNALS:ALIEN-VALUE :SAP #X71384C005800 :TYPE (*
                                                            (SB-ALIEN:STRUCT
                                                             TREE-SITTER:TS-QUERY-CURSOR))>

this segfaults:

(in-package :syn/ts)
(sb-alien:with-alien ((m (* tree-sitter::ts-query-match) (sb-alien:make-alien tree-sitter::ts-query-match)))
  (sb-sys::with-pinned-objects (m)
    (tree-sitter::ts-query-cursor-next-match
     (syn/ts::ts-file-query :c #p"/usr/include/openblas/lapack.h" "(declaration)") m)))

(let ((tree (flatten (syn/ts::parse-file :c #p"/usr/include/openblas/lapack.h")))
      (dx 0)
      (ret))
  (loop with dx = 0
        for di = (print (position :declaration tree :start dx))
        if (null di) 
        do (return ret)
        else
        do (progn
             (setf dx (incf di))
             (let ((ei (print (position :declaration tree :start dx))))
               (push (subseq tree di ei) ret)))))
(setq 
 *lapack*
 (let* ((str (read-file #p"/usr/include/openblas/lapack.h"))
        (tree (flatten (syn/ts::parse-string :c str :consume t :produce-cst nil)))
        (dx 0)
        (ret))
   (setq *lapack.h* str)
   (loop with dx = 0
         for di = (position :declaration tree :start dx)
         if (null di) 
         do (return ret)
         else
         do (progn
              (setf dx di)
              (incf dx)
              (let ((ei (position :declaration tree :start dx)))
                (push (subseq tree di ei) ret))))))
(list (length *lapack*) (length *lapack.h*))
1332 747544
(labels ((lastr (x start end)
           (apply 'subseq *lapack.h* (subseq x start end)))
         (lasym (str) (symbolicate (string-upcase (substitute #\- #\_ str))))
         (lasym* (str)
           (let ((len (length str)))
             (list '* (lasym (subseq str 0 
                                     (if (char= (char str (1- len)) #\*)
                                         (1- len)
                                         len))))))
         (lasyms (x start end)
           (symbolicate (string-upcase (substitute #\- #\_ (lastr x start end)))))
         (laparm (str)
           (let ((a (nreverse (split-sequence #\space str))))
             (list (if (string-equal (first a) "T")
                       (gensym "T")
                       (lasym (first a)))
                   (if (third a)
                       ;; type param (could be X const* or const X*)
                       (if (string-equal (third a) "const")
                           (lasym* (second a))
                           (lasym* (third a)))
                       (when (second a)
                         (lasym* (second a)))))))
         (gen-lapack-args (p x)
           (let* ((pa (position :parameter-list x)) ;; param start
                  (pb (position :preproc-ifdef x)) ;; param end
                  (params (subseq x pa pb)))
             (loop with p1 = 0
                   for pos = (position :parameter-declaration params :start p1)
                   while pos
                   do (setf p1 (1+ pos))
                   collect (laparm (lastr params (+ pos 1) (+ pos 3)))))))
  (setq *code*
        (mapcar
         (lambda (x)
           (let ((p (position :function-declarator x)))
             (list* 'deflapack
                    ;; routine name
                    (lasyms x (+ p 5) (+ p 7))
                    ;; return-type
                    (lasyms x (- p 3) (- p 1))
                    ;; list of function parameters (NAME TYPE)
                    (remove-if (lambda (x) (eql (car x) 'fortran-strlen)) (gen-lapack-args p x)))))
         *lapack*)))
(length *code*)
1332
(with-output-to-file (f file) 
  (write-line ";;; lapack-ffi.lisp --- LAPACK Alien Routines -*- buffer-read-only:t -*-" f)
  (write-line "(in-package :lapack)" f)
  (mapc (lambda (x) (write x :stream f :case :downcase) (terpri f)) *code*))

1.1.63. DONE linedit path completion

  • State "FIXME" from [2025-05-11 Sun 00:59]
  • State "DONE" from "FIXME" [2025-05-11 Sun 18:36]
  • Note taken on [2025-05-11 Sun 18:36]
    turns out STD/OS:WITH-DIRECTORY-ITERATOR was the culprit - re-arranged a null-alien check and good to go.

currently segfaults - should be easy to fix

1.1.64. DONE sxp 0.2   lisp sxp

  • State "TODO" from [2023-12-12 Tue 19:49]
  • State "DONE" from "TODO" [2025-05-16 Fri 23:20]
  1. NOPE WRAP-OBJECT   skel
    • State "TODO" from [2023-12-12 Tue 19:49]
    • Note taken on [2025-05-16 Fri 23:20]
      using obj/ast instead - lisp is data is code
    • use to replace LOAD-AST for sk-project and friends
    • requires mop
    • take a look at UNWRAP-OBJECT too, may need rewrite for a better API
    1. NOPE sxp standard methods
      • State "TODO" from [2024-08-13 Tue 22:06]
      • hash-table
      • classoid
      • vector
      • boolean
      • obj/
        • color
        • uri

1.1.65. DONE toml impl

  • State "TODO" from [2023-12-24 Sun 19:28]
  • State "DONE" from "TODO" [2025-04-30 Wed 18:30]
  • Refiled on [2025-04-30 Wed 18:32]

the toml parser on quicklisp is based on esrap, make a comparable one using our parser module.

1.1.66. DONE split out skel rules

  • State "TODO" from [2024-08-11 Sun 00:10]
  • State "DONE" from "TODO" [2025-05-16 Fri 22:45]
  • core/rule.lisp

1.1.67. DONE fix obj/query stack overflow on compile

  • State "TODO" from [2025-04-28 Mon 14:55]
  • State "DONE" from "TODO" [2025-05-20 Tue 18:33]
  • Note taken on [2025-05-20 Tue 18:35]
    soft fix - commented out shit
Control stack exhausted (no more space for function call frames).
This is probably due to heavily nested or infinitely recursive function
calls, or a tail call that SBCL cannot or has not optimized away.

PROCEED WITH CAUTION.
   [Condition of type SB-KERNEL::CONTROL-STACK-EXHAUSTED]

 849: ((:METHOD SB-PCL::COMPUTE-EFFECTIVE-SLOT-DEFINITION-INITARGS (SB-PCL::SLOT-CLASS T)) #<STANDARD-CLASS OBJ/QUERY:LOGICAL-QUERY-PLAN> (#<STANDARD-DIRECT-SLOT-DEFINITION OBJ/AST:AST> #<STANDARD-DIRECT-SL..
      Locals:
	SB-PCL::ALLOCATION = :INSTANCE
	SB-PCL::ALLOCATION-CLASS = #<STANDARD-CLASS OBJ/QUERY:LOGICAL-QUERY-PLAN>
	SB-PCL::ALLOCP = T
	CLASS = #<STANDARD-CLASS OBJ/QUERY:LOGICAL-QUERY-PLAN>
	SB-PCL::DIRECT-SLOTDS = (#<STANDARD-DIRECT-SLOT-DEFINITION OBJ/AST:AST> #<STANDARD-DIRECT-SLOT-DEFINITION OBJ/AST:AST> #<STANDARD-DIRECT-SLOT-DEFINITION OBJ/AST:AST>)
	DOCUMENTATION = NIL
	SB-PCL::DOCUMENTATIONP = NIL
	SB-PCL::INITARGS = (:AST)
	SB-PCL::INITFORM = NIL
	SB-PCL::INITFUNCTION = NIL
	SB-PCL::INITP = NIL
	SB-PCL::NAME = OBJ/AST:AST
	SB-PCL::NAMEP = T

 855: (SB-THREAD::CALL-WITH-RECURSIVE-LOCK #<FUNCTION (FLET SB-THREAD::WITH-RECURSIVE-LOCK-THUNK :IN SB-PCL::UPDATE-CLASS) {739064260D1B}> #<MUTEX "PCL global mutex" taken owner=main thread>)
      Locals:
	GOT-IT = NIL
	MUTEX = #<MUTEX "PCL global mutex" taken owner=main thread>
	SB-C::THING = #<FUNCTION (FLET SB-THREAD::WITH-RECURSIVE-LOCK-THUNK :IN SB-PCL::UPDATE-CLASS) {739064260D1B}>

1.1.68. DONE exfil from obj

  • State "OUTLINE" from [2025-05-20 Tue 18:37]
  • State "DONE" from "WIP" [2025-05-26 Mon 18:52]
  • things lib/math will need from obj:
    • plan
    • store
    • graph
    • srv?
    • schema?
  • may go to std

1.1.69. DONE Ecosystem

  • State "RESEARCH" from [2025-05-08 Thu 18:27]
  • State "WIP" from "RESEARCH" [2025-05-08 Thu 20:53]

CLOCK: [2025-05-08 Thu 20:55]--[2025-05-08 Thu 22:55] (2:00)

  • State "DONE" from "WIP" [2025-05-21 Wed 21:17]

1.1.70. DONE refactor libs into math

  • State "TODO" from [2025-05-08 Thu 18:31]
  • State "DONE" from "TODO" [2025-05-26 Mon 18:15]
  • obj should NOT depend on math
    • math depends on obj for tensor
  • accumulators? hmm..
    • need to be in math but also used by queries which is in obj
    • may need to split out obj/query to Q
      • done
    • moved to std/seq
  • math depends on ffi, depended on by dsp,net,apps,etc
  • should probably have the most tests of all libs
  • we want OBJ/TENSOR - see tensor

1.1.71. DONE symbol cleanup

  • State "TODO" from [2025-04-26 Sat 22:33]
  • State "DONE" from "TODO" [2025-04-26 Sat 22:46]

delete unbound non-anaphoric symbols

(let ((ret))
  (do-external-symbols (s :std ret)
    (when (not (or (boundp s) (fboundp s) (cadr (multiple-value-list (typexpand s))) (find-class s nil)))
      (push s ret))))
TIME-REMAINING PID-T TIMESPEC IT LOFF-T STAGES %P TASK GID-T .I %THREAD %A MAYBE-INLINE JOBS TASKS TIMEVAL WINSIZE UID-T

1.1.72. DONE review previous state of std/tasks

  • State "TODO" from [2025-04-28 Mon 17:56]
  • State "DONE" from "TODO" [2025-05-06 Tue 21:57]

We were doing some things right - like our semaphore worker waking

  • already added this to BROADCAST-WORK (comment suggestion by lmj)
  • mailbox stuff not as useful (speed is a concern over more primitive cas queues and spinlocks)

1.1.73. DONE explain   doc

  • State "TODO" from [2024-08-03 Sat 01:01]
  • State "WIP" from "TODO" [2025-04-27 Sun 19:33]
  • State "DONE" from "WIP" [2025-04-27 Sun 19:37]
  • generic done
  • maybe simple version in std/fmt

1.1.74. DONE explore   doc

  • State "TODO" from "TODO" [2024-08-12 Mon 22:26]
  • State "WIP" from "TODO" [2025-04-27 Sun 19:33]
  • State "DONE" from "WIP" [2025-04-27 Sun 19:37]
  • generic done

1.1.75. DONE define-alien-struct

  • State "DONE" from "TODO" [2025-05-02 Fri 13:54]
  • Refiled on [2025-05-02 Fri 13:54]
  • convenience for defining alien structs, especially with the same name as lisp type

1.1.76. DONE remove makefile

  • State "TODO" from [2024-08-16 Fri 22:44]
  • State "DONE" from "TODO" [2025-05-14 Wed 15:58]

1.2. infra

1.2.1. DONE dictd

  • State "TODO" from [2024-08-12 Mon 17:55]
  • Note taken on [2024-08-12 Mon 17:55]
    basically done - serving on compiler.company:2628 with route through https
  • State "DONE" from "TODO" [2024-08-12 Mon 18:02]

1.2.2. DONE reintegrate pod and box   vc

  • State "TODO" from [2024-08-18 Sun 16:14]
  • State "DONE" from "TODO" [2024-08-20 Tue 19:24]
  • [X] pod
  • [X] box
  • [X] .hgsub

1.2.3. DONE deploy [4/4]   lisp

  • State "TODO" from [2023-12-30 Sat 19:20]
  • State "DONE" from "TODO" [2025-05-04 Sun 19:41]
  • [X] collect artifacts
  • [X] pack
  • [X] send output to DIST
  • [X] Adapt to skel deploy
  • assumptions
    • host is qualified builder
    • all dependendencies installed on system (not in stash)
    • current user has access to DIST
  • done in org skelfile

1.2.4. DONE publish [4/4]   lisp emacs

  • State "TODO" from [2024-06-21 Fri 15:00]
  • State "DONE" from "TODO" [2025-05-04 Sun 19:41]
  • done in org skelfile
  • [X] load dependencies (htmlize,default.el,babel.org)
  • [X] load publish.el
  • [X] set publish-directory
  • [X] eval publish

1.2.5. DONE tree-sitter-pack

  • State "TODO" from [2024-01-24 Wed 22:59]
  • State "DONE" from "TODO" [2025-02-15 Sat 16:23]
  • pull-tree-sitter-langs.sh
  • install-tree-sitter-langs.sh (OG, calls pull)
  • make dist/tree-sitter

1.3. demo

2. roadmap

2.1. DONE 2024

  • State "DONE" from [2025-06-11 Wed 22:40]

2.2. DONE migrate VC off of EC2   opsec vc code

  • State "WIP" from [2024-05-06 Mon 16:43]
  • Note taken on [2024-09-03 Tue 16:52]
    removed dependency on heptapod, running plain ol hgweb behind proxy
  • State "DONE" from "WIP" [2024-09-03 Tue 22:13]
  • Note taken on [2024-09-03 Tue 22:13]
    next up: clonebundles,packy,git hosting

We no longer need to maintain an instance of our forge in AWS now that we have enough compute in-house.

vc.compiler.company will be migrate to rurik on a dedicated pod (requires ubuntu).

2.3. DONE Wireguard VPN   net opsec

  • State "TODO" from [2024-05-06 Mon 16:48]
  • State "WIP" from "TODO" [2024-05-06 Mon 16:48]
  • Note taken on [2024-08-12 Mon 18:21]
    all that is left is to configure for VM and containers
  • State "DONE" from "WIP" [2025-02-26 Wed 19:15]

Wireguard is a fast, secure, and easy to configure VPN library. All our primary devices have been configured and tested with two available endpoints. Our goal is to migrate all existing devices, then make appropriate accomodations for Podman containers, and finally KVM.

2.4. TODO IO   lisp

  • State "TBD" from [2024-05-06 Mon 16:50]
  • State "TODO" from "TBD" [2024-08-28 Wed 21:49]

iouring is the Linux library we have chosen to spend some time getting comfortable with and for which we've written incomplete Common Lisp bindings - URING. This library will soon be developed in parallel to the higher-level IO library - similar to the development of ROCKSDB and RDB.

Over the next few months our IO ecosystem will slowly expand. As a short-term goal, we will implement simple binary streams backed by iouring.

2.5. DONE skel

  • State "WIP" from "TBD" [2024-05-06 Mon 17:06]
  • Note taken on [2024-08-12 Mon 18:21]
    makefiles are now replaced by skelfile rules. next step is SKEL-VM for optimize rule execution plans.
  • State "DONE" from "WIP" [2025-02-26 Wed 19:15]

The pieces are there, just needs glue.

A usable SKEL system is a top priority for this year. To get there we need to give love to many of the underlying core libraries too - most of all cli.

The two top priorities are VC commands and implementing a functional replacement for makefiles as used in the infra project.

2.6. DONE orgdb   mvp

  • State "TBD" from [2024-08-18 Sun 01:58]
  • Note taken on [2024-08-28 Wed 21:49]
    see graph.el, org-graph-init.lisp
  • Note taken on [2025-02-26 Wed 19:15]
    renamed to org-graph-db (graph.el) see infra/scripts/org-graph-db-init.lisp and core/emacs/lib/graph.el
  • State "DONE" from "TBD" [2025-02-26 Wed 19:32]
  • similar idea to org-roam
  • stored in rocksdb columns

2.7. DONE infra

  • State "TODO" from [2024-05-06 Mon 17:06]
  • State "DONE" from "TODO" [2024-08-12 Mon 18:23]

It is time to move away from makefiles and start integrating our own programs into the system bootstrap process. This iteration of the infra project will have a few POSIX shell scripts for checking the host and grabbing a pre-compiled lisp as well as a compressed infra.core image.

3. product