[M8] reusable libraries: install layout + cargoxx-path deps

2026-05-17 18:13:15 +00:00
parent fdf97861a4
commit e6c39914b3
25 changed files with 932 additions and 21 deletions
--- a/docs/library-reuse-and-publish.md
+++ b/docs/library-reuse-and-publish.md
@@ -0,0 +1,448 @@
+# Library reuse + public registry
+
+Design plan for two interlocking features:
+
+1. **Library reuse** — cargoxx-built libraries are consumable by other
+   cargoxx projects, by plain CMake projects (`find_package`), and by
+   plain clang invocations (`pkg-config`).
+2. **Public registry + publish** — a Gitea-hosted package registry
+   (`cargoxx-pkgs`), a `cargoxx publish` CLI, and self-hosted CI that
+   validates PRs, builds, pushes derivations to a binary cache, and
+   auto-merges.
+
+Phase 1 covers library reuse end-to-end with local path and git deps.
+Phase 2 layers the registry on top, with cache-backed substitution so
+consumers don't recompile.
+
+---
+
+## Part 1 — reusable libraries
+
+### 1.1 Producer install layout
+
+```
+$out/
+├── bin/                                    # binaries (existing path)
+├── lib/
+│   ├── libmylib.a                          # archive of impl bits
+│   ├── pkgconfig/mylib.pc                  # pkg-config descriptor
+│   └── cmake/mylib/
+│       ├── mylibConfig.cmake               # entry point for find_package
+│       ├── mylibConfigVersion.cmake        # semver gate
+│       └── mylibTargets.cmake              # IMPORTED tgt + CXX_MODULES FILE_SET
+└── include/mylib/                          # public headers + source-form modules
+```
+
+Covers three consumption paths:
+
+- **cargoxx → cargoxx**: consumer's CMakeLists emits
+  `find_package(mylib CONFIG REQUIRED)` + link against `mylib::mylib`.
+  `$out` is on `CMAKE_PREFIX_PATH` via `buildInputs`.
+- **plain CMake**: same `find_package` call, manual `HINTS` if not on
+  the default prefix path.
+- **plain clang + pkg-config**: works for non-module libraries
+  (header-only or `.a` with extern API). Module libraries require a
+  CMake-aware consumer — industry-wide pkg-config has no story for
+  module BMIs.
+
+### 1.2 Codegen changes (`src/codegen/cmake.cpp`)
+
+When `layout.library` is set, append:
+
+```cmake
+include(GNUInstallDirs)
+include(CMakePackageConfigHelpers)
+
+install(TARGETS mylib
+        EXPORT mylibTargets
+        FILE_SET CXX_MODULES DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/mylib
+        ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR})
+install(EXPORT mylibTargets
+        FILE mylibTargets.cmake
+        NAMESPACE mylib::
+        DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/mylib)
+
+# Config.cmake.in written inline so we don't need an off-tree template file.
+file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/mylibConfig.cmake.in [[
+@PACKAGE_INIT@
+include(CMakeFindDependencyMacro)
+@FIND_DEPENDENCY_BLOCK@
+include("${CMAKE_CURRENT_LIST_DIR}/mylibTargets.cmake")
+check_required_components(mylib)
+]])
+configure_package_config_file(
+    ${CMAKE_CURRENT_BINARY_DIR}/mylibConfig.cmake.in
+    ${CMAKE_CURRENT_BINARY_DIR}/mylibConfig.cmake
+    INSTALL_DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/mylib)
+write_basic_package_version_file(
+    ${CMAKE_CURRENT_BINARY_DIR}/mylibConfigVersion.cmake
+    VERSION ${PROJECT_VERSION} COMPATIBILITY SameMajorVersion)
+install(FILES
+    ${CMAKE_CURRENT_BINARY_DIR}/mylibConfig.cmake
+    ${CMAKE_CURRENT_BINARY_DIR}/mylibConfigVersion.cmake
+    DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/mylib)
+
+# pkg-config descriptor — also inline.
+file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/mylib.pc.in [[
+prefix=@CMAKE_INSTALL_PREFIX@
+exec_prefix=${prefix}
+libdir=${prefix}/lib
+includedir=${prefix}/include
+
+Name: @PROJECT_NAME@
+Version: @PROJECT_VERSION@
+Description: @PROJECT_DESCRIPTION@
+Cflags: -I${includedir}
+Libs: -L${libdir} -lmylib
+]])
+configure_file(${CMAKE_CURRENT_BINARY_DIR}/mylib.pc.in
+               ${CMAKE_CURRENT_BINARY_DIR}/mylib.pc @ONLY)
+install(FILES ${CMAKE_CURRENT_BINARY_DIR}/mylib.pc
+        DESTINATION ${CMAKE_INSTALL_LIBDIR}/pkgconfig)
+```
+
+`@FIND_DEPENDENCY_BLOCK@` is computed by codegen — one `find_dependency(X)`
+line per cargoxx-source dep, so the transitive graph reconstitutes when
+a consumer does `find_package(mylib …)`.
+
+Binaries also get an install rule so `cmake --install` handles
+everything in one shot:
+
+```cmake
+install(TARGETS mylib_bin RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR})
+install(TARGETS tool      RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR})
+```
+
+### 1.3 `buildCppPackage.installPhase` switches to `cmake --install`
+
+```nix
+installPhase = ''
+  cmake --install build/release --prefix $out
+'';
+```
+
+Replaces the `cp -a build/release/bin/. $out/bin/` hack — now bins,
+library archive, headers, cmake config, pc file all land in the right
+place from one install command.
+
+### 1.4 Manifest schema — discriminated dep source
+
+Backward-compat (string form unchanged):
+
+```toml
+[dependencies]
+sqlite = "*"                                            # nixpkgs / linkdb (today)
+fmt = { version = ">=10" }                              # nixpkgs explicit (today)
+mylib = { path = "../mylib" }                           # cargoxx local path  ← NEW
+remote = { git = "https://gitea/me/remote", rev = "abc123" }  # cargoxx git    ← NEW
+shared = { version = "1.0", registry = "cargoxx" }      # cargoxx registry   ← NEW
+```
+
+`manifest::Dependency` grows discriminated source fields. Parser rejects
+unknown keys in dep tables (today silently ignored).
+
+### 1.5 Lockfile schema additions
+
+For cargoxx-source deps the lockfile carries source provenance and the
+recipe shape (`find_package`, `targets`) — codegen consumes the latter
+unchanged, so cargoxx deps look like any other linkdb recipe to the
+emitted CMakeLists.
+
+Path deps: record relative path.
+Git deps: record `(url, commit, sha256)`.
+Registry deps: record only `(registry_rev, registry_attr)`; the source
+URL + sha256 live in the registry repo, not the consumer's lockfile.
+
+### 1.6 `buildCppPackage` recursion
+
+```nix
+buildCargoxxDep = p:
+  if p.source_kind == "path" then
+    buildCppPackage { src = src + ("/" + p.path); }
+
+  else if p.source_kind == "git" then
+    let depSrc = pkgs.fetchgit {
+          inherit (p) url;
+          rev = p.commit;
+          hash = p.sha256;
+        };
+    in buildCppPackage { src = depSrc; }
+
+  else if p.source_kind == "registry" then
+    # Pull the prebuilt $out from the registry flake — no source fetch,
+    # no recompile. The registry's `packages.<system>.<attr>` derivation
+    # is content-addressed and substituted from the binary cache.
+    let registry = builtins.getFlake
+          "git+https://gitea/.../cargoxx-pkgs?rev=${lock.registry_rev}";
+    in registry.packages.${system}.${p.registry_attr}
+
+  else throw "unknown source_kind: ${p.source_kind}";
+
+cargoxxDepDrvs = map buildCargoxxDep
+                     (builtins.filter (p: p ? source_kind) lock.package);
+```
+
+`cargoxxDepDrvs` join `buildInputs`. nixpkgs's `cmakeConfigurePhase`
+automatically prepends every `buildInputs` `$out` to `CMAKE_PREFIX_PATH`,
+so consumer `find_package(<dep> CONFIG REQUIRED)` resolves without
+extra plumbing.
+
+### 1.7 Resolver flow for cargoxx-source deps
+
+```
+manifest dep → cmd_build::resolve_dep()
+  ├─ source kind == Nixpkgs/Auto: existing resolver chain
+  ├─ source kind == CargoxxPath:
+  │    read ${path}/Cargoxx.toml, take [package].name/version,
+  │    synthesize Recipe { find_package = "<name> CONFIG REQUIRED",
+  │                        targets      = ["<name>::<name>"] }
+  ├─ source kind == CargoxxGit:
+  │    fetch the repo at rev (resolver cache),
+  │    same as CargoxxPath against the checkout
+  └─ source kind == CargoxxRegistry:
+       fetch the registry's recipe TOML, take name + version,
+       synthesize the same Recipe shape.
+```
+
+The recipe shape — `find_package(<name> CONFIG REQUIRED)` plus
+`<name>::<name>` target — matches what the producer's installed
+`<name>Config.cmake` defines. Codegen needs no cargoxx-dep special case.
+
+### 1.8 Phasing
+
+| Phase | Deliverable |
+|---|---|
+| 1a | install rules + Config.cmake + .pc; `cmake --install` in buildCppPackage |
+| 1b | `{ path = "..." }` deps end-to-end (manifest parse + lock + recursive buildCppPackage); e2e fixture |
+| 1c | `{ git = "...", rev = "..." }` deps via `pkgs.fetchgit` |
+| 1d | `{ version = "1.0", registry = "cargoxx" }` deps (gated on Phase 2 being live) |
+
+---
+
+## Part 2 — Gitea-hosted registry
+
+### 2.1 `cargoxx-pkgs` repo layout
+
+```
+cargoxx-pkgs/
+├── flake.nix
+├── recipes/
+│   └── <name>/
+│       ├── maintainers.txt          # gitea usernames, one per line
+│       ├── meta.toml                # description, homepage, license, repo
+│       └── versions/
+│           ├── 1.0.0.toml
+│           └── 1.0.1.toml
+└── .gitea/workflows/
+    ├── validate-pr.yml
+    └── auto-merge.yml
+```
+
+### 2.2 Recipe `versions/<v>.toml`
+
+```toml
+schema = 1
+name = "mylib"
+version = "1.0.0"
+
+[source]
+type   = "git"
+url    = "https://gitea/me/mylib"
+commit = "abc123…"                   # 40-char
+sha256 = "sha256-…"                  # FOD pin
+
+[dependencies]
+fmt      = "10.2"
+otherlib = { version = "0.3", registry = "cargoxx" }
+
+[lock]
+nixpkgs_rev     = "…"
+flake_utils_rev = "…"
+cargoxx_rev     = "…"
+
+[meta]
+description = "…"
+homepage    = "https://…"
+license     = "MIT"
+```
+
+### 2.3 `cargoxx-pkgs` flake (the substitution surface)
+
+```nix
+{
+  description = "cargoxx package registry";
+  inputs.nixpkgs.url = "github:NixOS/nixpkgs/<rev>";
+  inputs.flake-utils.url = "github:numtide/flake-utils";
+  inputs.cargoxx.url = "git+https://gitea/.../cargoxx?rev=<rev>";
+
+  outputs = { self, nixpkgs, flake-utils, cargoxx }:
+    flake-utils.lib.eachDefaultSystem (system:
+      let
+        pkgs = import nixpkgs { inherit system; };
+
+        mkPackage = recipeFile:
+          let r = builtins.fromTOML (builtins.readFile recipeFile);
+          in cargoxx.lib.${system}.buildCppPackage {
+               src = pkgs.fetchgit {
+                 inherit (r.source) url;
+                 rev    = r.source.commit;
+                 hash   = r.source.sha256;
+               };
+             };
+
+        # Enumerate recipes/<name>/versions/<v>.toml, build packagesAttrs.
+        packagesAttrs = …;
+      in {
+        packages = packagesAttrs;
+      });
+}
+```
+
+CI evaluates this flake, builds the new attr, pushes `$out` to the
+binary cache. Consumers later get a cache hit for the SAME
+derivation hash via `builtins.getFlake` at the same `registry_rev`.
+
+### 2.4 Publishing flow (`cargoxx publish`)
+
+```
+cargoxx publish [--dry-run]
+1. Validate Cargoxx.toml has name, version, description, license, repository.
+2. Validate working tree clean; current commit on a remote branch.
+3. Validate Cargoxx.lock pins every cargoxx-source dep to a registry version
+   (path = "..." deps disallowed in publish).
+4. Compute recipe TOML:
+     [source].url    = git remote URL
+     [source].commit = HEAD commit
+     [source].sha256 = sha256 of `git archive HEAD`
+     [dependencies]  = mirror Cargoxx.toml
+     [lock]          = mirror Cargoxx.lock pins
+5. Clone cargoxx-pkgs (cached under ~/.cache/cargoxx/registry-clone).
+6. Create branch publish/<name>-<version>; new package → also create
+   recipes/<name>/{maintainers.txt,meta.toml}.
+7. Write recipes/<name>/versions/<version>.toml; commit.
+8. Push branch; `tea pr create` (tea login is already set up).
+```
+
+### 2.5 Validation bot (`.gitea/workflows/validate-pr.yml`)
+
+Runs on every PR via self-hosted runners:
+
+1. **Schema check**: PR touches only `recipes/<name>/**`; recipe TOML
+   validates.
+2. **Source fixity**: re-fetch `[source].url` at `[source].commit`,
+   recompute sha256, compare.
+3. **Build**: `nix build .#packages.x86_64-linux.<name>`.
+4. **Cache push**:
+   ```
+   nix copy --to "file:///srv/cargoxx-cache?secret-key=$KEY" \
+     .#packages.x86_64-linux.<name>
+   ```
+   `/srv/cargoxx-cache` served by nginx as the binary cache.
+5. **Dependency closure**: every `[dependencies]` entry exists at the
+   same PR HEAD.
+6. **Maintainer match**:
+   - New package: PR author becomes maintainer (write
+     `maintainers.txt`).
+   - Existing package: PR author appears in `maintainers.txt`. Else
+     label `needs-human-review`.
+7. On all-green + maintainer-match: label `auto-merge`.
+
+`auto-merge.yml` triggers on the `auto-merge` label and merges via
+the Gitea API.
+
+Maintainer transfer is a separate PR editing `maintainers.txt`. Same
+validation: PR author must already be on the list.
+
+### 2.6 Resolving registry deps (`cargoxx add foo`)
+
+```
+cargoxx add foo[@<version>]
+  1. Fetch https://gitea/.../cargoxx-pkgs/raw/branch/master/index.json
+     (the index, emitted by CI, maps name → list of versions).
+  2. Pick the highest matching version (or @<version>).
+  3. Fetch recipes/foo/versions/<v>.toml.
+  4. Write Cargoxx.toml:  foo = { version = "<v>", registry = "cargoxx" }.
+  5. Write Cargoxx.lock:  source_kind = "registry",
+                           registry_attr = "foo_<safe_ver>".
+  6. Also persist lock.registry_rev = <current master commit>.
+```
+
+The bundled cargoxx wrapper has `git`, `curl` on PATH, so this works
+on non-Nix hosts too.
+
+### 2.7 Binary cache config
+
+Once the cache URL + signing key are decided, the cargoxx wrapper's
+`NIX_CONFIG` (in `flake.nix`'s `cargoxxNixConfig`) gains:
+
+```
+substituters         = https://cache.cargoxx.<gitea-domain>/... https://cache.nixos.org
+trusted-public-keys  = cache.cargoxx.<gitea-domain>:<base64>= cache.nixos.org:6NCH…=
+```
+
+Set via `--set-default`, so a user with a different setup can
+override by exporting `NIX_CONFIG`.
+
+### 2.8 Hosting
+
+| Concern | Choice |
+|---|---|
+| Repo host | Self-hosted Gitea (your instance). |
+| CI | Gitea Actions on self-hosted runners (your existing pool). |
+| Binary cache | Static-HTTPS to start (`nix copy --to file:///srv/cargoxx-cache`); migrate to `attic` (https://github.com/zhaofengli/attic) if scale demands. |
+| Publishing auth | `tea login` (already configured). |
+| Architectures | `x86_64-linux` only for v1. aarch64/darwin are follow-ups. |
+
+### 2.9 Phasing
+
+| Phase | Deliverable |
+|---|---|
+| 2a | `cargoxx-pkgs` repo skeleton + `flake.nix` (zero recipes). |
+| 2b | `.gitea/workflows/validate-pr.yml`; manual recipe PRs. |
+| 2c | `cargoxx publish` CLI; `auto-merge.yml`. |
+| 2d | `cargoxx add` ↔ registry index; CI cache push; wrapper substituters config. |
+
+---
+
+## Module BMI strategy
+
+Following the libstdc++/libc++ model in nixpkgs: ship **source `.cppm`**
+via `install(TARGETS … FILE_SET CXX_MODULES …)`, ship the **compiled
+archive** (`libfoo.a`) of impl bits. Consumers regenerate the BMI in
+their own build (cmake's `FILE_SET CXX_MODULES` handles it when
+`find_package(foo)` is called). The compiled archive is what's
+cached — the BMI is cheap to regenerate against an already-cached
+archive.
+
+cargoxx already enables `CMAKE_CXX_MODULE_STD ON`, which CMake uses to
+rebuild the `std` BMI in each consumer project. The same machinery
+covers cargoxx libraries.
+
+---
+
+## Why `buildCppPackage` doesn't fetch source for registry deps
+
+The registry flake (`cargoxx-pkgs`) owns the
+`source-fetch → cargoxx-build → cmake-install` pipeline. Its
+`packages.<system>.<name>` is an input-addressed derivation: same
+sources + same toolchain pins → same hash → cache hit.
+
+Consumers reference that derivation via `builtins.getFlake` at the
+locked `registry_rev`. Nix's binary cache substitutes the `$out`
+directly — no source clone, no compile. Source fetch + build live
+inside the registry's eval, run once per recipe at CI time. This is
+the same model as nixpkgs/`cache.nixos.org`: you don't fetch fmt's
+source to use it from nixpkgs.
+
+---
+
+## Open follow-ups
+
+1. Gitea domain + cache URL (for wrapper config in Phase 2d).
+2. Signing key generation
+   (`nix-store --generate-binary-cache-key cache.cargoxx.<domain>
+    <secret> <public>`); private into runner secrets, public into
+   wrapper config.
+3. Multi-arch (`aarch64-linux`, `*-darwin`) — Phase 3+.
+4. Conan-style wrap recipes for non-cargoxx libraries — design now,
+   defer implementation.