• implemented in dat/dot.lisp

:ID: bb365025-e5e7-4407-acba-32e30d1a245a

lisp/ffi/

https://github.com/facebook/rocksdb/wiki/Compaction-Filter

(rocksdb-compactionfilterfactory-create nil (alien-sap (alien-callable-function 'rocksdb-destructor))
                                        (alien-sap (alien-callable-function 'rocksdb-create-compaction-filter-never))
                                        (alien-sap (alien-callable-function 'rocksdb-name)))

• need to figure out what delete-value requires
  • simply passes state + value and value-length to a function
  • when the value is null it doesn't exist in db (noop) ?
  • unclear when the merge op actually runs in relation to deletion op - before or after?
• https://docs.rs/rocks/latest/rocks/merge_operator/index.html

• define macro which eases the process of making custom merge operators

(define-merge-operator :incf nil
  :full nil
  :partial nil)
(create-incf-mergeoperator)

#<SB-ALIEN-INTERNALS:ALIEN-VALUE :SAP #X75D288000C30 :TYPE (*
                                                            (SB-ALIEN:STRUCT
                                                             ROCKSDB::ROCKSDB-MERGEOPERATOR-T))>

• document in Commentary section
• also consider https://github.com/antirez/linenoise
• RLWRAP settings for SBCL · GitHub
• CIEL Is an Extended Lisp
• GitHub - sharplispers/linedit: Readline-style line-editor for Common Lisp.

with skel taking on more responsibility we no longer need a dedicated build tool.

• migrate all functionality to skelfile
• cleanup readme with installation instructions (with and without infra)
• easy to break the skel binary and overwrite a LKG version, preventing future builds.
  • just be more careful
  • sanity checks
  • repurpose x.lisp to recover from situations like these specifically

• do not need to handle multiple commands
  • what do we want to do about subcommands?
• do not need to support 'global' option for commands
• use cli-parse to test.

• multiple entrypoint lisp images
• create multiple links to core lisp image
• support multiple low-level methods?
  • naive implementation is a single entry point with a switch dispatched on arg0.
  • could also create a thin shim in Rust/Zig and call the appropriate entrypoint after loading the core.
• for now let's put this in lib/cli so that we can use defmain

• finish implementation

• need new command or flag for rules
• evaluate forms instead of looking in skelfile

• remove package-inferred
• minimize number of systems
• use pkg.lisp files

• a bit too large, time to shake it up
• need separation of concerns (files) for task-pool/threaded tests in particular

• refer to cffi and uffi
• cffi/src/types.lisp at 32c90d4a9a01e809f591276c058e5b3c7f39b154 · cffi/cffi · GitHub

• mostly just need the tests and to finish porting the Franz implementation

[2024-09-16 Mon 14:41] <- org graph edges

• ported from XDB2

• list of links between other headings
  • <= links to this item
  • => links from this item
  • == siblings
  • ^= parent
  • v= child
• how should it look when exported?
• org-graph-link-use-timestamp
• org-graph-link properties? edge properties?
  • keyvals :key val - should probably align with whatever we use for CONSTRAINTS, for now read as sxp
• EXPORT_EDGES property?

* example
:PROPERTIES:
:ID:       id1
:CUSTOM_ID: demo
:END:
:EDGES:
<- bar
[2024-09-14 Sat 22:37] <- bar
[2024-09-14 Sat 22:50] :key val :foo (bar baz) <- bar
[2024-09-14 Sat 22:37] -> foo
[2024-09-14 Sat 22:39] -- sibling
[2024-09-14 Sat 22:39] >> parent
[2024-09-14 Sat 22:39] << child
:END:

• follow LOCATION property of heading at point
• respect parent LOCATION, buffer local settings, etc

similar to org-set-effort, to be called automatically on capture.

• should prompt for directory then file - skip file with C-u or with RET
  • simpler to just prompt for value with completion using org-read-property-value

homepage 'symlink farm manager' rust impl: https://github.com/qboileau/rstow

rocksdb/slicetransform.lisp

• consider use cases in Lisp images - pre-loading and coordinating shared fasls and core images
• no longer maintained by intel

A Gentle Introduction to XDP | Datadog GitHub - xdp-project/xdp-tools: Utilities and example programs for use with XDP

• xdp-paper/xdp-the-express-data-path.pdf at main · xdp-project/xdp-paper · GitHub
• BPF and XDP Reference Guide — Cilium 1.18.0-dev documentation
• will require linking to libbpf and libxdp, needs elf loader (or use xdp-tools/iproute2)
• blocked by bpf impl in lisp

Home - DPDK GitHub - DPDK/dpdk: Data Plane Development Kit

By leveraging DPDK libraries, developers can create optimized packet processing paths, manage timers for executing functions asynchronously, and utilize a wide range of drivers and libraries tailored for fast packet processing.

librte_eal – Environment Abstraction Layer: Provides the foundational API for DPDK, facilitating access to hardware resources such as memory, timers, and logs.

librte_mempool – Memory Pool Manager: Manages memory pools for efficient and speedy packet handling.

librte_ring – Ring Buffer Manager: Implements lock-free FIFO queues, enabling high-speed communication between various DPDK components.

librte_mbuf – Packet Buffer Management: Handles packet buffers, which are crucial for packet transmission and reception.

librte_ethdev – Ethernet Device API: Offers an API for configuring and querying Ethernet devices. It supports various operations, including sending and receiving packets.

librte_net – Network Helper Library: Provides helper APIs for dealing with network protocols.

librte_ipfrag – IP Fragmentation and Reassembly: Handles fragmentation and reassembly of IP packets, supporting both IPv4 and IPv6.

librte_kni – Kernel Network Interface: Facilitates communication between DPDK applications and the Linux kernel networking stack, primarily used for debugging or interfacing with existing Linux network services.

https://docs.kernel.org/bpf/standardization/instruction-set.html#bpf-instruction-set-architecture-isa https://mostlynerdless.de/blog/2023/12/31/hello-ebpf-developing-ebpf-apps-in-java-1/ https://github.com/iovisor/bcc GitHub - mikeroyal/eBPF-Guide bpftrace/tools/README.md at master · bpftrace/bpftrace · GitHub

• bpftrace seems quite powerful, will be useful for sysmon activities

where would ebpf support go?

probably io.. but maybe also log, net.. syn?

Doc-overview - batman-adv - Open Mesh

Not that useful in our current environment (NIH)

lapack.h is massive and impractical to attempt lisp bindings from scratch.

instead, our current approach is based on tree-sitter:

(tree-sitter:load-tree-sitter)
(tree-sitter:load-tree-sitter-alien)
(tree-sitter:load-tree-sitter-c)

#P"/usr/local/lib/libtree-sitter-c.so"

(syn/ts::ts-file-query :c #p"/usr/include/openblas/lapack.h" "(declaration)")

#<SB-ALIEN-INTERNALS:ALIEN-VALUE :SAP #X71384C005800 :TYPE (*
                                                            (SB-ALIEN:STRUCT
                                                             TREE-SITTER:TS-QUERY-CURSOR))>

this segfaults:

(in-package :syn/ts)
(sb-alien:with-alien ((m (* tree-sitter::ts-query-match) (sb-alien:make-alien tree-sitter::ts-query-match)))
  (sb-sys::with-pinned-objects (m)
    (tree-sitter::ts-query-cursor-next-match
     (syn/ts::ts-file-query :c #p"/usr/include/openblas/lapack.h" "(declaration)") m)))

(let ((tree (flatten (syn/ts::parse-file :c #p"/usr/include/openblas/lapack.h")))
      (dx 0)
      (ret))
  (loop with dx = 0
        for di = (print (position :declaration tree :start dx))
        if (null di) 
        do (return ret)
        else
        do (progn
             (setf dx (incf di))
             (let ((ei (print (position :declaration tree :start dx))))
               (push (subseq tree di ei) ret)))))

(setq 
 *lapack*
 (let* ((str (read-file #p"/usr/include/openblas/lapack.h"))
        (tree (flatten (syn/ts::parse-string :c str :consume t :produce-cst nil)))
        (dx 0)
        (ret))
   (setq *lapack.h* str)
   (loop with dx = 0
         for di = (position :declaration tree :start dx)
         if (null di) 
         do (return ret)
         else
         do (progn
              (setf dx di)
              (incf dx)
              (let ((ei (position :declaration tree :start dx)))
                (push (subseq tree di ei) ret))))))
(list (length *lapack*) (length *lapack.h*))

(labels ((lastr (x start end)
           (apply 'subseq *lapack.h* (subseq x start end)))
         (lasym (str) (symbolicate (string-upcase (substitute #\- #\_ str))))
         (lasym* (str)
           (let ((len (length str)))
             (list '* (lasym (subseq str 0 
                                     (if (char= (char str (1- len)) #\*)
                                         (1- len)
                                         len))))))
         (lasyms (x start end)
           (symbolicate (string-upcase (substitute #\- #\_ (lastr x start end)))))
         (laparm (str)
           (let ((a (nreverse (split-sequence #\space str))))
             (list (if (string-equal (first a) "T")
                       (gensym "T")
                       (lasym (first a)))
                   (if (third a)
                       ;; type param (could be X const* or const X*)
                       (if (string-equal (third a) "const")
                           (lasym* (second a))
                           (lasym* (third a)))
                       (when (second a)
                         (lasym* (second a)))))))
         (gen-lapack-args (p x)
           (let* ((pa (position :parameter-list x)) ;; param start
                  (pb (position :preproc-ifdef x)) ;; param end
                  (params (subseq x pa pb)))
             (loop with p1 = 0
                   for pos = (position :parameter-declaration params :start p1)
                   while pos
                   do (setf p1 (1+ pos))
                   collect (laparm (lastr params (+ pos 1) (+ pos 3)))))))
  (setq *code*
        (mapcar
         (lambda (x)
           (let ((p (position :function-declarator x)))
             (list* 'deflapack
                    ;; routine name
                    (lasyms x (+ p 5) (+ p 7))
                    ;; return-type
                    (lasyms x (- p 3) (- p 1))
                    ;; list of function parameters (NAME TYPE)
                    (remove-if (lambda (x) (eql (car x) 'fortran-strlen)) (gen-lapack-args p x)))))
         *lapack*)))
(length *code*)

1332

(with-output-to-file (f file) 
  (write-line ";;; lapack-ffi.lisp --- LAPACK Alien Routines -*- buffer-read-only:t -*-" f)
  (write-line "(in-package :lapack)" f)
  (mapc (lambda (x) (write x :stream f :case :downcase) (terpri f)) *code*))

currently segfaults - should be easy to fix

the toml parser on quicklisp is based on esrap, make a comparable one using our parser module.

• core/rule.lisp

Control stack exhausted (no more space for function call frames).
This is probably due to heavily nested or infinitely recursive function
calls, or a tail call that SBCL cannot or has not optimized away.

PROCEED WITH CAUTION.
   [Condition of type SB-KERNEL::CONTROL-STACK-EXHAUSTED]

 849: ((:METHOD SB-PCL::COMPUTE-EFFECTIVE-SLOT-DEFINITION-INITARGS (SB-PCL::SLOT-CLASS T)) #<STANDARD-CLASS OBJ/QUERY:LOGICAL-QUERY-PLAN> (#<STANDARD-DIRECT-SLOT-DEFINITION OBJ/AST:AST> #<STANDARD-DIRECT-SL..
      Locals:
	SB-PCL::ALLOCATION = :INSTANCE
	SB-PCL::ALLOCATION-CLASS = #<STANDARD-CLASS OBJ/QUERY:LOGICAL-QUERY-PLAN>
	SB-PCL::ALLOCP = T
	CLASS = #<STANDARD-CLASS OBJ/QUERY:LOGICAL-QUERY-PLAN>
	SB-PCL::DIRECT-SLOTDS = (#<STANDARD-DIRECT-SLOT-DEFINITION OBJ/AST:AST> #<STANDARD-DIRECT-SLOT-DEFINITION OBJ/AST:AST> #<STANDARD-DIRECT-SLOT-DEFINITION OBJ/AST:AST>)
	DOCUMENTATION = NIL
	SB-PCL::DOCUMENTATIONP = NIL
	SB-PCL::INITARGS = (:AST)
	SB-PCL::INITFORM = NIL
	SB-PCL::INITFUNCTION = NIL
	SB-PCL::INITP = NIL
	SB-PCL::NAME = OBJ/AST:AST
	SB-PCL::NAMEP = T

 855: (SB-THREAD::CALL-WITH-RECURSIVE-LOCK #<FUNCTION (FLET SB-THREAD::WITH-RECURSIVE-LOCK-THUNK :IN SB-PCL::UPDATE-CLASS) {739064260D1B}> #<MUTEX "PCL global mutex" taken owner=main thread>)
      Locals:
	GOT-IT = NIL
	MUTEX = #<MUTEX "PCL global mutex" taken owner=main thread>
	SB-C::THING = #<FUNCTION (FLET SB-THREAD::WITH-RECURSIVE-LOCK-THUNK :IN SB-PCL::UPDATE-CLASS) {739064260D1B}>

• things lib/math will need from obj:
  • plan
  • store
  • graph
  • srv?
  • schema?
• may go to std

• Common Lisp Statistics · GitHub
• GitHub - Lisp-Stat/lisp-stat: Lisp-Stat main system
• GitHub - takagi/cl-cuda: Cl-cuda is a library to use NVIDIA CUDA in Common Lisp programs.
• GitHub - quil-lang/magicl: Matrix Algebra proGrams In Common Lisp.
• SRFI 231: Intervals and Generalized Arrays
• lisplab
• matlisp
• and next.. pola.rs
• use 3d-math patches as a way to learn more about what we will need
• we need a lot more than just matrix acceleration.
• llvm-project/mlir/docs/Dialects/Linalg/_index.md · JuliaLang/llvm-project
• LAPACK? ye probably - be aware of symbol rewrites caused by FC
  • solved in blas (wip)

• What is the difference between Tensors and Matrices? · Learn Tensorflow

  A matrix is a two-dimensional grid of size n×m that contains numbers: you can add and subtract matrices of the same size, multiply one matrix with another as long as the sizes are compatible((n×m)×(m×p)=n×p)((n×m)×(m×p)=n×p), and multiply an entire matrix by a constant.

  A vector is a matrix with just one row or column (but see below), vector has direction, but tensor has no direction.

  A tensor is often thought of as a generalized matrix. That is, it could be

  a 1-D matrix, like a vector, which is actually such a tensor, a 3-D matrix (something like a cube of numbers), a 0-D matrix (a single number), or a higher dimensional structure that is harder to visualize. The dimension of the tensor is called its rank.

  Any rank-2 tensor can be represented as a matrix, but not every matrix is really a rank-2 tensor. The numerical values of a tensor’s matrix representation depend on what transformation rules have been applied to the entire system.

• obj should NOT depend on math
  • math depends on obj for tensor
• accumulators? hmm..
  • need to be in math but also used by queries which is in obj
  • may need to split out obj/query to Q
    • done
  • moved to std/seq
• math depends on ffi, depended on by dsp,net,apps,etc
• should probably have the most tests of all libs
• we want OBJ/TENSOR - see tensor

delete unbound non-anaphoric symbols

(let ((ret))
  (do-external-symbols (s :std ret)
    (when (not (or (boundp s) (fboundp s) (cadr (multiple-value-list (typexpand s))) (find-class s nil)))
      (push s ret))))

We were doing some things right - like our semaphore worker waking

• already added this to BROADCAST-WORK (comment suggestion by lmj)
• mailbox stuff not as useful (speed is a concern over more primitive cas queues and spinlocks)

• generic done
• maybe simple version in std/fmt

• generic done

• convenience for defining alien structs, especially with the same name as lisp type

• [X] pod
• [X] box
• [X] .hgsub

• [X] collect artifacts
• [X] pack
• [X] send output to DIST
• [X] Adapt to skel deploy
• assumptions
  • host is qualified builder
  • all dependendencies installed on system (not in stash)
  • current user has access to DIST
• done in org skelfile

• done in org skelfile
• [X] load dependencies (htmlize,default.el,babel.org)
• [X] load publish.el
• [X] set publish-directory
• [X] eval publish

• pull-tree-sitter-langs.sh
• install-tree-sitter-langs.sh (OG, calls pull)
• make dist/tree-sitter