1. tasks

1.1. core

Archived entries from file /home/ellis/comp/org/plan/tasks/core.org

1.1.1. DONE dot 0.1 [1/1]   lisp dot

  • State "TODO" from [2023-12-20 Wed 18:27]
  • State "DONE" from "NOPE" [2024-08-16 Fri 22:38]
  • implemented in dat/dot.lisp
  1. NOPE PARSE-SPECIALIZER-USING-CLASS
    • State "TODO" from [2023-12-20 Wed 18:28]
    • experimental SB-MOP generic, may be useful in places where we would typically use (eql OBJECT) as a specializer for GRAPH.

1.1.2. DONE migrate to sb-grovel [9/9]   lisp ffi

  • State "TODO" from [2023-11-27 Mon 23:42]
  • State "DONE" from "TODO" [2024-02-04 Sun 20:30]

:ID: bb365025-e5e7-4407-acba-32e30d1a245a

lisp/ffi/

  1. DONE blake3
    • State "TODO" from [2023-11-29 Wed 19:00]
    • State "DONE" from "TODO" [2023-12-09 Sat 01:57]

    :ID: 656c2683-8780-407c-b955-3defc4959595

  2. DONE bqn
    • State "TODO" from [2023-11-29 Wed 19:00]
    • State "DONE" from "TODO" [2023-12-09 Sat 01:57]

    :ID: b4757746-525b-49e4-b8ac-677375867d8f

  3. DONE btrfs
    • State "TODO" from [2023-11-29 Wed 19:00]
    • State "DONE" from "TODO" [2023-12-09 Sat 01:57]

    :ID: b3b411fe-6b2d-4338-9e23-83536336a487

  4. DONE k
    • State "TODO" from [2023-11-29 Wed 19:00]
    • State "DONE" from "TODO" [2023-12-09 Sat 01:57]

    :ID: 8a463084-efa0-4535-8239-b1c4955bde4c

  5. DONE rocksdb
    • State "TODO" from [2023-11-29 Wed 19:00]
    • State "DONE" from "TODO" [2023-12-09 Sat 01:57]

    :ID: 2d4d70af-228f-424c-8374-9fc8ed5c6f92

  6. DONE tree-sitter
    • State "TODO" from [2023-11-29 Wed 19:00]
    • State "DONE" from "TODO" [2023-12-09 Sat 01:57]

    :ID: 0cdf3412-cb00-4069-8e1b-b49c736377cf

  7. DONE uring
    • State "TODO" from [2023-11-29 Wed 19:00]
    • State "DONE" from "TODO" [2023-12-09 Sat 01:57]

    :ID: 45a52407-ccaf-4ea1-99c4-d6f7ba0b2731

  8. DONE zstd
    • State "TODO" from [2023-11-29 Wed 19:00]
    • State "DONE" from "TODO" [2023-12-09 Sat 01:57]

    :ID: 6ade8dd8-83b1-470d-a0da-a352ce4c2d06

  9. DONE xkb
    • State "TODO" from [2023-11-29 Wed 19:27]
    • State "WIP" from "TODO" [2023-12-10 Sun 22:47]
    • State "DONE" from "WIP" [2024-02-04 Sun 20:30]

    :ID: af778a5f-4834-4f0c-ae53-36d35cb98d8c

    • State "TODO" from [2024-08-13 Tue 22:06]

1.1.3. DONE writebatch

  • State "TODO" from [2024-08-16 Fri 22:56]
  • State "WIP" from "TODO" [2024-08-19 Mon 20:01]

CLOCK: [2024-08-19 Mon 20:01]--[2024-08-19 Mon 21:03] (1:02)

CLOCK: [2024-08-20 Tue 19:41]--[2024-08-20 Tue 20:16] (0:35)

  • State "DONE" from "WIP" [2024-08-20 Tue 20:19]
  1. file:///usr/local/include/rocksdb/c.h#org408b022
    • Refiled on [2024-08-18 Sun 21:44]

1.1.4. DONE logger

  • State "TODO" from [2024-08-16 Fri 22:56]
  • State "DONE" from "TODO" [2024-08-18 Sun 22:00]

1.1.5. DONE merge-operator callbacks   rocksdb

  • State "TODO" from [2024-08-10 Sat 22:21]
  • State "WIP" from "TODO" [2024-08-14 Wed 21:50]
  • State "DONE" from "WIP" [2024-08-16 Fri 22:52]
  • Note taken on [2024-08-16 Fri 22:52]
    still needs testing with actual data

1.1.6. DONE compactionfilter callbacks   rocksdb

  • State "TODO" from [2024-08-14 Wed 21:50]
  • State "WIP" from "TODO" [2024-08-14 Wed 21:50]

CLOCK: [2024-08-18 Sun 18:51]--[2024-08-18 Sun 19:46] (0:00)

  • State "DONE" from "WIP" [2024-08-18 Sun 19:47]

https://github.com/facebook/rocksdb/wiki/Compaction-Filter

(rocksdb-compactionfilterfactory-create nil (alien-sap (alien-callable-function 'rocksdb-destructor))
                                        (alien-sap (alien-callable-function 'rocksdb-create-compaction-filter-never))
                                        (alien-sap (alien-callable-function 'rocksdb-name)))

1.1.7. DONE comparator callbacks   rocksdb

  • State "WIP" from [2024-08-14 Wed 21:51]
  • State "DONE" from "WIP" [2024-08-18 Sun 19:47]

1.1.8. DONE associative-merge implementation

  • Refiled on [2024-08-18 Sun 19:48]
  • State "DONE" from "TODO" [2024-11-07 Thu 21:45]
  • need to figure out what delete-value requires
    • simply passes state + value and value-length to a function
    • when the value is null it doesn't exist in db (noop) ?
    • unclear when the merge op actually runs in relation to deletion op - before or after?
  • https://docs.rs/rocks/latest/rocks/merge_operator/index.html
  1. DONE concat-merge
    • State "TODO" from [2024-08-18 Sun 20:42]
    • State "DONE" from "TODO" [2024-11-07 Thu 20:41]
    • Note taken on [2024-11-07 Thu 20:41]
      well it-s about time.. should have known the SUCCESS arg was the return value that the docs referenced..
  2. DONE index-merge
    • State "TODO" from [2024-08-18 Sun 20:44]
    • State "WIP" from "TODO" [2024-11-07 Thu 20:59]
    • State "DONE" from "WIP" [2024-11-07 Thu 21:44]

1.1.9. DONE incf/decf-merge

  • State "TODO" from [2024-08-18 Sun 20:43]
  • State "TEST" from "TODO" [2024-11-07 Thu 21:00]
  • State "TBD" from "TEST" [2024-11-07 Thu 21:44]
  • Note taken on [2024-11-07 Thu 21:44]
    use the STATE param for this.
  • State "DONE" from "TBD" [2024-12-28 Sat 23:00]
  • Note taken on [2024-12-28 Sat 23:00]
    see index-merge

1.1.10. DONE define-merge-operator   macro

  • Refiled on [2024-08-18 Sun 19:53]
  • State "TODO" from "TODO" [2024-08-18 Sun 22:01]
  • State "TEST" from "TODO" [2024-08-18 Sun 22:02]

CLOCK: [2024-08-21 Wed 20:34]--[2024-08-21 Wed 21:13] (0:39)

  • State "DONE" from "TEST" [2024-08-21 Wed 21:21]
  • Note taken on [2024-08-21 Wed 21:21]
    will need to do actual tests merging key/vals when we implement associative merge API
  • define macro which eases the process of making custom merge operators
(define-merge-operator :incf nil
  :full nil
  :partial nil)
(create-incf-mergeoperator)
#<SB-ALIEN-INTERNALS:ALIEN-VALUE :SAP #X75D288000C30 :TYPE (*
                                                            (SB-ALIEN:STRUCT
                                                             ROCKSDB::ROCKSDB-MERGEOPERATOR-T))>

1.1.11. DONE zstd

  • State "PROJECT" from [2024-09-18 Wed 21:50]
  • Note taken on [2024-09-19 Thu 21:57]
    can probably improve error handling but don't need to address until we are well under way with IO/FLATE.
  • State "DONE" from "PROJECT" [2024-11-02 Sat 23:33]
  1. DONE zstd cstream tests
    • State "DONE" from "TODO" [2024-09-18 Wed 21:50]
    • v2 api
  2. DONE zdict
    • State "TODO" from [2024-09-19 Thu 21:57]
    • Note taken on [2024-10-04 Fri 20:38]
      well this is annoying, there is one function which requires passing struct by value: ZDICTfinalizeDictionary
    • State "DONE" from "TODO" [2024-11-02 Sat 23:33]
    • A few structs that we can't grovel
    • A bunch of functions for pre-training a dictionary

1.1.12. DONE reference impls

  • State "RESEARCH" from [2024-09-19 Thu 22:00]
  • State "DONE" from "RESEARCH" [2024-09-19 Thu 23:19]

1.1.13. DONE retire x.lisp   lisp skel

  • State "TODO" from [2024-07-09 Tue 22:24]
  • State "TBD" from "TODO" [2024-07-27 Sat 02:41]
  • State "TODO" from "TBD" [2024-08-16 Fri 22:58]
  • Note taken on [2024-09-16 Mon 21:52]
    actually let's move it to infra/scripts..
  • Note taken on [2024-09-16 Mon 22:10]
    on second look we really should start afresh lol
  • State "DONE" from "TODO" [2024-09-16 Mon 22:10]

with skel taking on more responsibility we no longer need a dedicated build tool.

  • migrate all functionality to skelfile
  • cleanup readme with installation instructions (with and without infra)
  • easy to break the skel binary and overwrite a LKG version, preventing future builds.
    • just be more careful
    • sanity checks
    • repurpose x.lisp to recover from situations like these specifically

1.1.14. DONE implement globals *cli* *opts* *args* etc

  • State "TODO" from [2024-07-27 Sat 02:39]
  • State "DONE" from "TODO" [2024-07-27 Sat 02:40]

1.1.15. DONE opt/arg parsing

  • State "FIXME" from [2024-08-21 Wed 21:23]
  • State "WIP" from "FIXME" [2024-09-11 Wed 18:30]
  • State "TEST" from "WIP" [2024-09-14 Sat 00:25]
  • State "DONE" from "TEST" [2024-10-15 Tue 19:38]
  • Note taken on [2024-10-15 Tue 19:39]
    took far longer than expected to track this down - root cause was the zerop function. should be in much better shape now.
  • do not need to handle multiple commands
    • what do we want to do about subcommands?
  • do not need to support 'global' option for commands
  • use cli-parse to test.

1.1.16. DONE multipoint-images   lisp bin sys

  • State "TODO" from [2024-07-28 Sun 21:01]
  • State "WIP" from "TODO" [2024-09-14 Sat 17:13]
  • State "DONE" from "WIP" [2024-09-14 Sat 18:49]
  • multiple entrypoint lisp images
  • create multiple links to core lisp image
  • support multiple low-level methods?
    • naive implementation is a single entry point with a switch dispatched on arg0.
    • could also create a thin shim in Rust/Zig and call the appropriate entrypoint after loading the core.
  • for now let's put this in lib/cli so that we can use defmain

1.1.17. DONE make-sql-data-frame

  • State "FIX" from [2024-08-05 Mon 21:13]
  • State "DONE" from "FIX" [2024-08-07 Wed 21:05]
  • finish implementation

1.1.18. DONE git   vc git

  • State "TODO" from [2023-12-16 Sat 00:55]
  • State "DONE" from "TODO" [2024-08-10 Sat 01:34]

1.1.19. DONE hg   vc hg

  • State "TODO" from [2023-12-16 Sat 00:56]
  • State "DONE" from "TODO" [2024-08-10 Sat 01:34]

1.1.20. DONE eval rules from CLI

  • State "TODO" from [2023-12-16 Sat 00:56]
  • State "DONE" from "TODO" [2024-08-05 Mon 21:06]
  • need new command or flag for rules
  • evaluate forms instead of looking in skelfile

1.1.21. DONE refactor system definition

  • State "TODO" from [2023-12-13 Wed 15:17]
  • State "DONE" from "TODO" [2023-12-13 Wed 19:38]
  • remove package-inferred
  • minimize number of systems
  • use pkg.lisp files

1.1.22. DONE cleanup fu

  • State "TODO" from [2023-12-13 Wed 19:39]
  • State "DONE" from "TODO" [2025-01-18 Sat 20:06]

1.1.23. DONE CLTL2 impl

  • State "TODO" from [2024-02-29 Thu 16:09]
  • State "DONE" from "TODO" [2025-01-18 Sat 20:07]

1.1.24. DONE list-package-symbols, etc

  • State "TODO" from [2024-02-29 Thu 16:09]
  • State "DONE" from "TODO" [2024-08-07 Wed 21:07]

1.1.25. DONE std/tests refactor

  • State "TODO" from [2024-07-27 Sat 02:42]
  • State "DONE" from "TODO" [2024-10-15 Tue 19:45]
  • a bit too large, time to shake it up
  • need separation of concerns (files) for task-pool/threaded tests in particular

1.1.26. DONE tasks

  • State "TODO" from [2024-08-10 Sat 01:29]
  • Note taken on [2024-10-15 Tue 19:45]
    currently includes jobs, supervisors, task-pool, kernel - a bit overloaded
  • State "DONE" from "TODO" [2025-01-18 Sat 20:07]

1.1.27. DONE more sb-alien utils   ffi

  • State "TODO" from [2024-09-19 Thu 23:21]
  • State "DONE" from "TODO" [2025-01-18 Sat 20:08]

1.1.28. NOPE uri   net

  • State "TODO" from [2023-12-17 Sun 01:00]
  • Note taken on [2023-12-20 Wed 18:26]
    the franz impl doesn't play well with SBCL, at least with our hacks. I'm not sure how faithfully I want to follow the RFCs anyhow.. maybe we go for the dead simple approach and focus on MOP for a path and URI protocol.
  • mostly just need the tests and to finish porting the Franz implementation

1.1.29. DONE db   db

[2024-09-16 Mon 14:41] <- org graph edges

  • State "TODO" from [2023-12-20 Wed 18:26]
  • State "DONE" from "TODO" [2024-08-05 Mon 21:08]
  • Note taken on [2024-08-05 Mon 21:08]
    XDB2 is no longer used - instead provide a simple abstract API which builds up to high-level systems like RDB
  • ported from XDB2

1.1.30. DONE aggregate physical plan

  • State "DONE" from "TODO" [2024-08-03 Sat 01:00]

1.1.31. DONE Physical Expressions

  • State "TODO" from [2024-08-02 Fri 15:53]
  • State "TODO" from [2024-08-02 Fri 15:53]
  • State "DONE" from "TODO" [2024-08-03 Sat 01:00]

1.1.32. DONE crew   proto

  • State "TODO" from [2023-12-17 Sun 01:01]
  • State "DONE" from "TODO" [2024-08-10 Sat 01:31]

1.1.33. DONE http   proto

  • State "TODO" from [2023-12-17 Sun 01:01]
  • State "DONE" from "TODO" [2024-08-05 Mon 21:07]

1.1.34. DONE dns   proto

  • State "TODO" from [2023-12-17 Sun 01:01]
  • State "DONE" from "TODO" [2024-08-05 Mon 21:07]

1.1.35. DONE tlv   codec

  • State "TODO" from [2023-12-17 Sun 01:01]
  • State "DONE" from "TODO" [2024-08-07 Wed 21:08]

1.1.36. DONE osc   codec

  • State "TODO" from [2023-12-17 Sun 01:03]
  • State "DONE" from "TODO" [2024-08-07 Wed 21:08]

1.1.37. DONE Woo/+

  • State "RESEARCH" from [2024-08-10 Sat 01:23]
  • State "DONE" from "RESEARCH" [2025-01-18 Sat 20:08]

1.1.38. DONE fixture api

  • State "TODO" from [2024-08-13 Tue 22:22]
  • State "DONE" from "TODO" [2025-01-18 Sat 20:09]

1.1.39. DONE temporary files and directories

  • Refiled on [2024-08-28 Wed 21:35]

CLOCK: [2024-08-31 Sat 20:39]--[2024-08-31 Sat 22:14] (1:35)

  • State "DONE" from "TODO" [2024-08-31 Sat 22:14]

1.1.40. DONE org graph edges

  • State "TODO" from [2024-08-27 Tue 18:20]
  • Note taken on [2024-09-14 Sat 16:30]
    dblock will come later. for now we are going with the same strategy as org-super-links - using a drawer. instead of "BACKLINKS" our drawer is called simply LINKS.
  • State "WIP" from "TODO" [2024-09-14 Sat 16:31]

CLOCK: [2024-09-17 Tue 19:50]--[2024-09-17 Tue 20:20] (0:30)

CLOCK: [2024-09-21 Sat 13:56]--[2024-09-21 Sat 17:44] (3:48)

CLOCK: [2024-09-25 Wed 18:54]--[2024-09-25 Wed 19:14] (0:20)

  • State "DONE" from "WIP" [2025-02-16 Sun 19:18]
  • Note taken on [2025-02-16 Sun 19:18]
    initial implementation up and running
  • list of links between other headings
    • <= links to this item
    • => links from this item
    • == siblings
    • ^= parent
    • v= child
  • how should it look when exported?
  • org-graph-link-use-timestamp
  • org-graph-link properties? edge properties?
    • keyvals :key val - should probably align with whatever we use for CONSTRAINTS, for now read as sxp
  • EXPORTEDGES property?
* example
:PROPERTIES:
:ID:       id1
:CUSTOM_ID: demo
:END:
:EDGES:
<- bar
[2024-09-14 Sat 22:37] <- bar
[2024-09-14 Sat 22:50] :key val :foo (bar baz) <- bar
[2024-09-14 Sat 22:37] -> foo
[2024-09-14 Sat 22:39] -- sibling
[2024-09-14 Sat 22:39] >> parent
[2024-09-14 Sat 22:39] << child
:END:

1.1.41. DONE org-follow-location   org

  • State "TODO" from [2024-08-21 Wed 20:37]

CLOCK: [2024-08-29 Thu 18:24]--[2024-08-29 Thu 18:51] (0:27)

  • State "WIP" from "REFILE" [2024-08-29 Thu 18:24]
  • State "DONE" from "WIP" [2024-08-29 Thu 23:46]
  • follow LOCATION property of heading at point
  • respect parent LOCATION, buffer local settings, etc

1.1.42. DONE org-set-location

  • State "TODO" from [2024-08-29 Thu 23:46]
  • State "DONE" from "TODO" [2024-09-08 Sun 12:23]

similar to org-set-effort, to be called automatically on capture.

  • should prompt for directory then file - skip file with C-u or with RET
    • simpler to just prompt for value with completion using org-read-property-value

1.1.43. DONE flate [0/0]

  • State "TODO" from [2024-09-19 Thu 21:55]
  • State "PROJECT" from "TODO" [2024-09-19 Thu 21:55]
  • State "DONE" from "PROJECT" [2025-01-18 Sat 20:48]

1.1.44. DONE review GNU stow API

  • State "TODO" from [2023-12-24 Sun 19:32]
  • State "DONE" from "TODO" [2024-08-05 Mon 21:06]

homepage 'symlink farm manager' rust impl: https://github.com/qboileau/rstow

1.1.45. DONE consolidate with skel   skel

  • State "TODO" from [2023-12-24 Sun 19:32]
  • State "DONE" from "TODO" [2025-01-18 Sat 20:49]

1.1.46. DONE cli tests crashing

  • State "FIXME" from "TODO" [2024-09-18 Wed 12:25]
  • State "DONE" from "FIXME" [2025-01-18 Sat 20:49]

1.1.47. DONE :scripts regression   infra

  • State "DONE" from "TODO" [2025-01-18 Sat 20:49]

1.1.48. NOPE readline   cli

  • State "TODO" from [2024-08-10 Sat 22:23]
  • Note taken on [2025-04-26 Sat 20:22]
    we have removed readline in favor of the lisp-native cli/linedit.

1.1.49. DONE impl rocksdb/slicetransform callbacks

  • State "TODO" from [2024-08-19 Mon 16:17]
  • New deadline from "[2024-08-27 Tue]" on [2024-08-27 Tue 17:30]
  • Refiled on [2024-08-27 Tue 17:31]
  • New deadline from "[2024-09-07 Sat]" on [2024-11-02 Sat 23:34]
  • State "TEST" from "TODO" [2024-11-03 Sun 00:23]
  • State "DONE" from "TEST" [2025-02-26 Wed 18:55]

rocksdb/slicetransform.lisp

1.1.50. DONE OCI projects [4/4]   pod packy skel

  • State "RESEARCH" from [2024-08-01 Thu 12:35]
  • State "DONE" from "RESEARCH" [2025-04-26 Sat 20:27]
  • Note taken on [2025-04-26 Sat 20:27]
    A bit too much for us to consider right now - ultimately I do not feel that OCI containers are a one-size-fits-all solution for image-based development.
  1. DONE oras
    • State "RESEARCH" from [2024-08-01 Thu 12:36]
    • State "DONE" from "RESEARCH" [2025-04-26 Sat 20:29]
  2. DONE skopeo
    • State "RESEARCH" from [2024-08-01 Thu 12:36]
    • State "DONE" from "RESEARCH" [2025-04-26 Sat 20:29]
  3. DONE zot
    • State "RESEARCH" from [2024-08-01 Thu 12:36]
    • State "DONE" from "RESEARCH" [2025-04-26 Sat 20:29]
  4. DONE sigstore
    • State "RESEARCH" from [2024-08-01 Thu 12:30]
    • State "DONE" from "RESEARCH" [2025-04-26 Sat 20:29]

1.1.51. DONE PMem.io

  • State "RESEARCH" from "TODO" [2024-09-03 Tue 11:22]
  • State "DONE" from "RESEARCH" [2025-01-17 Fri 18:19]
  • consider use cases in Lisp images - pre-loading and coordinating shared fasls and core images
  • no longer maintained by intel

1.1.52. NOPE encrypted btrfs luks/dm-crypt

  • State "RESEARCH" from "TODO" [2024-09-29 Sun 01:57]

1.1.53. DONE XDP

  • State "RESEARCH" from [2024-09-20 Fri 22:34]
  • State "DONE" from "RESEARCH" [2025-04-26 Sat 20:39]

A Gentle Introduction to XDP | Datadog GitHub - xdp-project/xdp-tools: Utilities and example programs for use with XDP

1.1.54. DONE DPDK

  • State "RESEARCH" from [2024-09-20 Fri 22:35]
  • State "DONE" from "RESEARCH" [2025-04-26 Sat 20:41]

Home - DPDK GitHub - DPDK/dpdk: Data Plane Development Kit

By leveraging DPDK libraries, developers can create optimized packet processing paths, manage timers for executing functions asynchronously, and utilize a wide range of drivers and libraries tailored for fast packet processing.

librteeal – Environment Abstraction Layer: Provides the foundational API for DPDK, facilitating access to hardware resources such as memory, timers, and logs.

librtemempool – Memory Pool Manager: Manages memory pools for efficient and speedy packet handling.

librtering – Ring Buffer Manager: Implements lock-free FIFO queues, enabling high-speed communication between various DPDK components.

librtembuf – Packet Buffer Management: Handles packet buffers, which are crucial for packet transmission and reception.

librteethdev – Ethernet Device API: Offers an API for configuring and querying Ethernet devices. It supports various operations, including sending and receiving packets.

librtenet – Network Helper Library: Provides helper APIs for dealing with network protocols.

librteipfrag – IP Fragmentation and Reassembly: Handles fragmentation and reassembly of IP packets, supporting both IPv4 and IPv6.

librtekni – Kernel Network Interface: Facilitates communication between DPDK applications and the Linux kernel networking stack, primarily used for debugging or interfacing with existing Linux network services.

1.1.55. DONE eBPF   net

  • State "RESEARCH" from [2024-08-21 Wed 22:40]
  • State "DONE" from "RESEARCH" [2025-04-26 Sat 20:47]

https://docs.kernel.org/bpf/standardization/instruction-set.html#bpf-instruction-set-architecture-isa https://mostlynerdless.de/blog/2023/12/31/hello-ebpf-developing-ebpf-apps-in-java-1/ https://github.com/iovisor/bcc GitHub - mikeroyal/eBPF-Guide bpftrace/tools/README.md at master · bpftrace/bpftrace · GitHub

  • bpftrace seems quite powerful, will be useful for sysmon activities

where would ebpf support go?

probably io.. but maybe also log, net.. syn?

1.1.56. DONE batman-adv   net

  • State "RESEARCH" from [2024-08-23 Fri 21:16]
  • State "DONE" from "RESEARCH" [2025-03-12 Wed 17:15]

Doc-overview - batman-adv - Open Mesh

1.1.57. NOPE libyear   lisp skel

  • State "TODO" from [2024-08-01 Thu 12:27]

Not that useful in our current environment (NIH)

1.1.58. DONE Level 1

  • State "TODO" from [2025-05-10 Sat 22:34]
  • State "DONE" from "TODO" [2025-05-12 Mon 22:56]

1.1.59. DONE Level 2

  • State "TODO" from [2025-05-10 Sat 22:34]
  • State "WIP" from "TODO" [2025-05-12 Mon 22:56]
  • State "DONE" from "WIP" [2025-05-14 Wed 18:05]

1.1.60. DONE Level 3

  • State "TODO" from [2025-05-10 Sat 22:34]
  • State "DONE" from "TODO" [2025-05-18 Sun 20:30]

1.1.61. DONE LAPACK (f77blas.h)

  • State "TODO" from [2025-05-18 Sun 21:07]
  • State "WIP" from "TODO" [2025-05-18 Sun 21:08]

CLOCK: [2025-05-18 Sun 21:08]--[2025-05-18 Sun 23:08] (2:00)

  • State "DONE" from "WIP" [2025-05-19 Mon 11:52]
  • Note taken on [2025-05-24 Sat 18:55]
    this does NOT include the lapack.h functions

1.1.62. DONE LAPACK codegen

  • State "TODO" from [2025-05-24 Sat 18:55]
  • State "DONE" from "TODO" [2025-05-25 Sun 22:29]
  • Note taken on [2025-05-25 Sun 22:30]
    see lapack-ffi.lisp

lapack.h is massive and impractical to attempt lisp bindings from scratch.

instead, our current approach is based on tree-sitter:

(tree-sitter:load-tree-sitter)
(tree-sitter:load-tree-sitter-alien)
(tree-sitter:load-tree-sitter-c)
#P"/usr/local/lib/libtree-sitter-c.so"
(syn/ts::ts-file-query :c #p"/usr/include/openblas/lapack.h" "(declaration)")
#<SB-ALIEN-INTERNALS:ALIEN-VALUE :SAP #X71384C005800 :TYPE (*
                                                            (SB-ALIEN:STRUCT
                                                             TREE-SITTER:TS-QUERY-CURSOR))>

this segfaults:

(in-package :syn/ts)
(sb-alien:with-alien ((m (* tree-sitter::ts-query-match) (sb-alien:make-alien tree-sitter::ts-query-match)))
  (sb-sys::with-pinned-objects (m)
    (tree-sitter::ts-query-cursor-next-match
     (syn/ts::ts-file-query :c #p"/usr/include/openblas/lapack.h" "(declaration)") m)))

(let ((tree (flatten (syn/ts::parse-file :c #p"/usr/include/openblas/lapack.h")))
      (dx 0)
      (ret))
  (loop with dx = 0
        for di = (print (position :declaration tree :start dx))
        if (null di) 
        do (return ret)
        else
        do (progn
             (setf dx (incf di))
             (let ((ei (print (position :declaration tree :start dx))))
               (push (subseq tree di ei) ret)))))
(setq 
 *lapack*
 (let* ((str (read-file #p"/usr/include/openblas/lapack.h"))
        (tree (flatten (syn/ts::parse-string :c str :consume t :produce-cst nil)))
        (dx 0)
        (ret))
   (setq *lapack.h* str)
   (loop with dx = 0
         for di = (position :declaration tree :start dx)
         if (null di) 
         do (return ret)
         else
         do (progn
              (setf dx di)
              (incf dx)
              (let ((ei (position :declaration tree :start dx)))
                (push (subseq tree di ei) ret))))))
(list (length *lapack*) (length *lapack.h*))
1332 747544
(labels ((lastr (x start end)
           (apply 'subseq *lapack.h* (subseq x start end)))
         (lasym (str) (symbolicate (string-upcase (substitute #\- #\_ str))))
         (lasym* (str)
           (let ((len (length str)))
             (list '* (lasym (subseq str 0 
                                     (if (char= (char str (1- len)) #\*)
                                         (1- len)
                                         len))))))
         (lasyms (x start end)
           (symbolicate (string-upcase (substitute #\- #\_ (lastr x start end)))))
         (laparm (str)
           (let ((a (nreverse (split-sequence #\space str))))
             (list (if (string-equal (first a) "T")
                       (gensym "T")
                       (lasym (first a)))
                   (if (third a)
                       ;; type param (could be X const* or const X*)
                       (if (string-equal (third a) "const")
                           (lasym* (second a))
                           (lasym* (third a)))
                       (when (second a)
                         (lasym* (second a)))))))
         (gen-lapack-args (p x)
           (let* ((pa (position :parameter-list x)) ;; param start
                  (pb (position :preproc-ifdef x)) ;; param end
                  (params (subseq x pa pb)))
             (loop with p1 = 0
                   for pos = (position :parameter-declaration params :start p1)
                   while pos
                   do (setf p1 (1+ pos))
                   collect (laparm (lastr params (+ pos 1) (+ pos 3)))))))
  (setq *code*
        (mapcar
         (lambda (x)
           (let ((p (position :function-declarator x)))
             (list* 'deflapack
                    ;; routine name
                    (lasyms x (+ p 5) (+ p 7))
                    ;; return-type
                    (lasyms x (- p 3) (- p 1))
                    ;; list of function parameters (NAME TYPE)
                    (remove-if (lambda (x) (eql (car x) 'fortran-strlen)) (gen-lapack-args p x)))))
         *lapack*)))
(length *code*)
1332
(with-output-to-file (f file) 
  (write-line ";;; lapack-ffi.lisp --- LAPACK Alien Routines -*- buffer-read-only:t -*-" f)
  (write-line "(in-package :lapack)" f)
  (mapc (lambda (x) (write x :stream f :case :downcase) (terpri f)) *code*))

1.1.63. DONE linedit path completion

  • State "FIXME" from [2025-05-11 Sun 00:59]
  • State "DONE" from "FIXME" [2025-05-11 Sun 18:36]
  • Note taken on [2025-05-11 Sun 18:36]
    turns out STD/OS:WITH-DIRECTORY-ITERATOR was the culprit - re-arranged a null-alien check and good to go.

currently segfaults - should be easy to fix

1.1.64. DONE sxp 0.2   lisp sxp

  • State "TODO" from [2023-12-12 Tue 19:49]
  • State "DONE" from "TODO" [2025-05-16 Fri 23:20]
  1. NOPE WRAP-OBJECT   skel
    • State "TODO" from [2023-12-12 Tue 19:49]
    • Note taken on [2025-05-16 Fri 23:20]
      using obj/ast instead - lisp is data is code
    • use to replace LOAD-AST for sk-project and friends
    • requires mop
    • take a look at UNWRAP-OBJECT too, may need rewrite for a better API
    1. NOPE sxp standard methods
      • State "TODO" from [2024-08-13 Tue 22:06]
      • hash-table
      • classoid
      • vector
      • boolean
      • obj/
        • color
        • uri

1.1.65. DONE toml impl

  • State "TODO" from [2023-12-24 Sun 19:28]
  • State "DONE" from "TODO" [2025-04-30 Wed 18:30]
  • Refiled on [2025-04-30 Wed 18:32]

the toml parser on quicklisp is based on esrap, make a comparable one using our parser module.

1.1.66. DONE split out skel rules

  • State "TODO" from [2024-08-11 Sun 00:10]
  • State "DONE" from "TODO" [2025-05-16 Fri 22:45]
  • core/rule.lisp

1.1.67. DONE fix obj/query stack overflow on compile

  • State "TODO" from [2025-04-28 Mon 14:55]
  • State "DONE" from "TODO" [2025-05-20 Tue 18:33]
  • Note taken on [2025-05-20 Tue 18:35]
    soft fix - commented out shit
Control stack exhausted (no more space for function call frames).
This is probably due to heavily nested or infinitely recursive function
calls, or a tail call that SBCL cannot or has not optimized away.

PROCEED WITH CAUTION.
   [Condition of type SB-KERNEL::CONTROL-STACK-EXHAUSTED]

 849: ((:METHOD SB-PCL::COMPUTE-EFFECTIVE-SLOT-DEFINITION-INITARGS (SB-PCL::SLOT-CLASS T)) #<STANDARD-CLASS OBJ/QUERY:LOGICAL-QUERY-PLAN> (#<STANDARD-DIRECT-SLOT-DEFINITION OBJ/AST:AST> #<STANDARD-DIRECT-SL..
      Locals:
	SB-PCL::ALLOCATION = :INSTANCE
	SB-PCL::ALLOCATION-CLASS = #<STANDARD-CLASS OBJ/QUERY:LOGICAL-QUERY-PLAN>
	SB-PCL::ALLOCP = T
	CLASS = #<STANDARD-CLASS OBJ/QUERY:LOGICAL-QUERY-PLAN>
	SB-PCL::DIRECT-SLOTDS = (#<STANDARD-DIRECT-SLOT-DEFINITION OBJ/AST:AST> #<STANDARD-DIRECT-SLOT-DEFINITION OBJ/AST:AST> #<STANDARD-DIRECT-SLOT-DEFINITION OBJ/AST:AST>)
	DOCUMENTATION = NIL
	SB-PCL::DOCUMENTATIONP = NIL
	SB-PCL::INITARGS = (:AST)
	SB-PCL::INITFORM = NIL
	SB-PCL::INITFUNCTION = NIL
	SB-PCL::INITP = NIL
	SB-PCL::NAME = OBJ/AST:AST
	SB-PCL::NAMEP = T

 855: (SB-THREAD::CALL-WITH-RECURSIVE-LOCK #<FUNCTION (FLET SB-THREAD::WITH-RECURSIVE-LOCK-THUNK :IN SB-PCL::UPDATE-CLASS) {739064260D1B}> #<MUTEX "PCL global mutex" taken owner=main thread>)
      Locals:
	GOT-IT = NIL
	MUTEX = #<MUTEX "PCL global mutex" taken owner=main thread>
	SB-C::THING = #<FUNCTION (FLET SB-THREAD::WITH-RECURSIVE-LOCK-THUNK :IN SB-PCL::UPDATE-CLASS) {739064260D1B}>

1.1.68. DONE exfil from obj

  • State "OUTLINE" from [2025-05-20 Tue 18:37]
  • State "DONE" from "WIP" [2025-05-26 Mon 18:52]
  • things lib/math will need from obj:
    • plan
    • store
    • graph
    • srv?
    • schema?
  • may go to std

1.1.69. DONE Ecosystem

  • State "RESEARCH" from [2025-05-08 Thu 18:27]
  • State "WIP" from "RESEARCH" [2025-05-08 Thu 20:53]

CLOCK: [2025-05-08 Thu 20:55]--[2025-05-08 Thu 22:55] (2:00)

  • State "DONE" from "WIP" [2025-05-21 Wed 21:17]

1.1.70. DONE refactor libs into math

  • State "TODO" from [2025-05-08 Thu 18:31]
  • State "DONE" from "TODO" [2025-05-26 Mon 18:15]
  • obj should NOT depend on math
    • math depends on obj for tensor
  • accumulators? hmm..
    • need to be in math but also used by queries which is in obj
    • may need to split out obj/query to Q
      • done
    • moved to std/seq
  • math depends on ffi, depended on by dsp,net,apps,etc
  • should probably have the most tests of all libs
  • we want OBJ/TENSOR - see tensor

1.1.71. DONE symbol cleanup

  • State "TODO" from [2025-04-26 Sat 22:33]
  • State "DONE" from "TODO" [2025-04-26 Sat 22:46]

delete unbound non-anaphoric symbols

(let ((ret))
  (do-external-symbols (s :std ret)
    (when (not (or (boundp s) (fboundp s) (cadr (multiple-value-list (typexpand s))) (find-class s nil)))
      (push s ret))))
TIME-REMAINING PID-T TIMESPEC IT LOFF-T STAGES %P TASK GID-T .I %THREAD %A MAYBE-INLINE JOBS TASKS TIMEVAL WINSIZE UID-T

1.1.72. DONE review previous state of std/tasks

  • State "TODO" from [2025-04-28 Mon 17:56]
  • State "DONE" from "TODO" [2025-05-06 Tue 21:57]

We were doing some things right - like our semaphore worker waking

  • already added this to BROADCAST-WORK (comment suggestion by lmj)
  • mailbox stuff not as useful (speed is a concern over more primitive cas queues and spinlocks)

1.1.73. DONE explain   doc

  • State "TODO" from [2024-08-03 Sat 01:01]
  • State "WIP" from "TODO" [2025-04-27 Sun 19:33]
  • State "DONE" from "WIP" [2025-04-27 Sun 19:37]
  • generic done
  • maybe simple version in std/fmt

1.1.74. DONE explore   doc

  • State "TODO" from "TODO" [2024-08-12 Mon 22:26]
  • State "WIP" from "TODO" [2025-04-27 Sun 19:33]
  • State "DONE" from "WIP" [2025-04-27 Sun 19:37]
  • generic done

1.1.75. DONE define-alien-struct

  • State "DONE" from "TODO" [2025-05-02 Fri 13:54]
  • Refiled on [2025-05-02 Fri 13:54]
  • convenience for defining alien structs, especially with the same name as lisp type

1.1.76. DONE remove makefile

  • State "TODO" from [2024-08-16 Fri 22:44]
  • State "DONE" from "TODO" [2025-05-14 Wed 15:58]

1.2. infra

1.2.1. DONE dictd

  • State "TODO" from [2024-08-12 Mon 17:55]
  • Note taken on [2024-08-12 Mon 17:55]
    basically done - serving on compiler.company:2628 with route through https
  • State "DONE" from "TODO" [2024-08-12 Mon 18:02]

1.2.2. DONE reintegrate pod and box   vc

  • State "TODO" from [2024-08-18 Sun 16:14]
  • State "DONE" from "TODO" [2024-08-20 Tue 19:24]
  • [X] pod
  • [X] box
  • [X] .hgsub

1.2.3. DONE deploy [4/4]   lisp

  • State "TODO" from [2023-12-30 Sat 19:20]
  • State "DONE" from "TODO" [2025-05-04 Sun 19:41]
  • [X] collect artifacts
  • [X] pack
  • [X] send output to DIST
  • [X] Adapt to skel deploy
  • assumptions
    • host is qualified builder
    • all dependendencies installed on system (not in stash)
    • current user has access to DIST
  • done in org skelfile

1.2.4. DONE publish [4/4]   lisp emacs

  • State "TODO" from [2024-06-21 Fri 15:00]
  • State "DONE" from "TODO" [2025-05-04 Sun 19:41]
  • done in org skelfile
  • [X] load dependencies (htmlize,default.el,babel.org)
  • [X] load publish.el
  • [X] set publish-directory
  • [X] eval publish

1.2.5. DONE tree-sitter-pack

  • State "TODO" from [2024-01-24 Wed 22:59]
  • State "DONE" from "TODO" [2025-02-15 Sat 16:23]
  • pull-tree-sitter-langs.sh
  • install-tree-sitter-langs.sh (OG, calls pull)
  • make dist/tree-sitter

1.3. demo

2. roadmap

3. product