mozart/cargoxx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e6c39914b3398379b1f749614580fbc3a83d75bb
cargoxx/docs/library-reuse-and-publish.md

16 KiB
Raw Blame History

Library reuse + public registry

Design plan for two interlocking features:

  1. Library reuse — cargoxx-built libraries are consumable by other cargoxx projects, by plain CMake projects (find_package), and by plain clang invocations (pkg-config).
  2. Public registry + publish — a Gitea-hosted package registry (cargoxx-pkgs), a cargoxx publish CLI, and self-hosted CI that validates PRs, builds, pushes derivations to a binary cache, and auto-merges.

Phase 1 covers library reuse end-to-end with local path and git deps. Phase 2 layers the registry on top, with cache-backed substitution so consumers don't recompile.

Part 1 — reusable libraries

1.1 Producer install layout

$out/
├── bin/                                    # binaries (existing path)
├── lib/
│   ├── libmylib.a                          # archive of impl bits
│   ├── pkgconfig/mylib.pc                  # pkg-config descriptor
│   └── cmake/mylib/
│       ├── mylibConfig.cmake               # entry point for find_package
│       ├── mylibConfigVersion.cmake        # semver gate
│       └── mylibTargets.cmake              # IMPORTED tgt + CXX_MODULES FILE_SET
└── include/mylib/                          # public headers + source-form modules

Covers three consumption paths:

  • cargoxx → cargoxx: consumer's CMakeLists emits find_package(mylib CONFIG REQUIRED) + link against mylib::mylib. $out is on CMAKE_PREFIX_PATH via buildInputs.
  • plain CMake: same find_package call, manual HINTS if not on the default prefix path.
  • plain clang + pkg-config: works for non-module libraries (header-only or .a with extern API). Module libraries require a CMake-aware consumer — industry-wide pkg-config has no story for module BMIs.

1.2 Codegen changes (src/codegen/cmake.cpp)

When layout.library is set, append:

include(GNUInstallDirs)
include(CMakePackageConfigHelpers)

install(TARGETS mylib
        EXPORT mylibTargets
        FILE_SET CXX_MODULES DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/mylib
        ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR})
install(EXPORT mylibTargets
        FILE mylibTargets.cmake
        NAMESPACE mylib::
        DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/mylib)

# Config.cmake.in written inline so we don't need an off-tree template file.
file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/mylibConfig.cmake.in [[
@PACKAGE_INIT@
include(CMakeFindDependencyMacro)
@FIND_DEPENDENCY_BLOCK@
include("${CMAKE_CURRENT_LIST_DIR}/mylibTargets.cmake")
check_required_components(mylib)
]])
configure_package_config_file(
    ${CMAKE_CURRENT_BINARY_DIR}/mylibConfig.cmake.in
    ${CMAKE_CURRENT_BINARY_DIR}/mylibConfig.cmake
    INSTALL_DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/mylib)
write_basic_package_version_file(
    ${CMAKE_CURRENT_BINARY_DIR}/mylibConfigVersion.cmake
    VERSION ${PROJECT_VERSION} COMPATIBILITY SameMajorVersion)
install(FILES
    ${CMAKE_CURRENT_BINARY_DIR}/mylibConfig.cmake
    ${CMAKE_CURRENT_BINARY_DIR}/mylibConfigVersion.cmake
    DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/mylib)

# pkg-config descriptor — also inline.
file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/mylib.pc.in [[
prefix=@CMAKE_INSTALL_PREFIX@
exec_prefix=${prefix}
libdir=${prefix}/lib
includedir=${prefix}/include

Name: @PROJECT_NAME@
Version: @PROJECT_VERSION@
Description: @PROJECT_DESCRIPTION@
Cflags: -I${includedir}
Libs: -L${libdir} -lmylib
]])
configure_file(${CMAKE_CURRENT_BINARY_DIR}/mylib.pc.in
               ${CMAKE_CURRENT_BINARY_DIR}/mylib.pc @ONLY)
install(FILES ${CMAKE_CURRENT_BINARY_DIR}/mylib.pc
        DESTINATION ${CMAKE_INSTALL_LIBDIR}/pkgconfig)

@FIND_DEPENDENCY_BLOCK@ is computed by codegen — one find_dependency(X) line per cargoxx-source dep, so the transitive graph reconstitutes when a consumer does find_package(mylib …).

Binaries also get an install rule so cmake --install handles everything in one shot:

install(TARGETS mylib_bin RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR})
install(TARGETS tool      RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR})

1.3 buildCppPackage.installPhase switches to cmake --install

installPhase = ''
  cmake --install build/release --prefix $out
'';

Replaces the cp -a build/release/bin/. $out/bin/ hack — now bins, library archive, headers, cmake config, pc file all land in the right place from one install command.

1.4 Manifest schema — discriminated dep source

Backward-compat (string form unchanged):

[dependencies]
sqlite = "*"                                            # nixpkgs / linkdb (today)
fmt = { version = ">=10" }                              # nixpkgs explicit (today)
mylib = { path = "../mylib" }                           # cargoxx local path  ← NEW
remote = { git = "https://gitea/me/remote", rev = "abc123" }  # cargoxx git    ← NEW
shared = { version = "1.0", registry = "cargoxx" }      # cargoxx registry   ← NEW

manifest::Dependency grows discriminated source fields. Parser rejects unknown keys in dep tables (today silently ignored).

1.5 Lockfile schema additions

For cargoxx-source deps the lockfile carries source provenance and the recipe shape (find_package, targets) — codegen consumes the latter unchanged, so cargoxx deps look like any other linkdb recipe to the emitted CMakeLists.

Path deps: record relative path. Git deps: record (url, commit, sha256). Registry deps: record only (registry_rev, registry_attr); the source URL + sha256 live in the registry repo, not the consumer's lockfile.

1.6 buildCppPackage recursion

buildCargoxxDep = p:
  if p.source_kind == "path" then
    buildCppPackage { src = src + ("/" + p.path); }

  else if p.source_kind == "git" then
    let depSrc = pkgs.fetchgit {
          inherit (p) url;
          rev = p.commit;
          hash = p.sha256;
        };
    in buildCppPackage { src = depSrc; }

  else if p.source_kind == "registry" then
    # Pull the prebuilt $out from the registry flake — no source fetch,
    # no recompile. The registry's `packages.<system>.<attr>` derivation
    # is content-addressed and substituted from the binary cache.
    let registry = builtins.getFlake
          "git+https://gitea/.../cargoxx-pkgs?rev=${lock.registry_rev}";
    in registry.packages.${system}.${p.registry_attr}

  else throw "unknown source_kind: ${p.source_kind}";

cargoxxDepDrvs = map buildCargoxxDep
                     (builtins.filter (p: p ? source_kind) lock.package);

cargoxxDepDrvs join buildInputs. nixpkgs's cmakeConfigurePhase automatically prepends every buildInputs $out to CMAKE_PREFIX_PATH, so consumer find_package(<dep> CONFIG REQUIRED) resolves without extra plumbing.

1.7 Resolver flow for cargoxx-source deps

manifest dep → cmd_build::resolve_dep()
  ├─ source kind == Nixpkgs/Auto: existing resolver chain
  ├─ source kind == CargoxxPath:
  │    read ${path}/Cargoxx.toml, take [package].name/version,
  │    synthesize Recipe { find_package = "<name> CONFIG REQUIRED",
  │                        targets      = ["<name>::<name>"] }
  ├─ source kind == CargoxxGit:
  │    fetch the repo at rev (resolver cache),
  │    same as CargoxxPath against the checkout
  └─ source kind == CargoxxRegistry:
       fetch the registry's recipe TOML, take name + version,
       synthesize the same Recipe shape.

The recipe shape — find_package(<name> CONFIG REQUIRED) plus <name>::<name> target — matches what the producer's installed <name>Config.cmake defines. Codegen needs no cargoxx-dep special case.

1.8 Phasing

Phase Deliverable
1a install rules + Config.cmake + .pc; cmake --install in buildCppPackage
1b { path = "..." } deps end-to-end (manifest parse + lock + recursive buildCppPackage); e2e fixture
1c { git = "...", rev = "..." } deps via pkgs.fetchgit
1d { version = "1.0", registry = "cargoxx" } deps (gated on Phase 2 being live)

Part 2 — Gitea-hosted registry

2.1 cargoxx-pkgs repo layout

cargoxx-pkgs/
├── flake.nix
├── recipes/
│   └── <name>/
│       ├── maintainers.txt          # gitea usernames, one per line
│       ├── meta.toml                # description, homepage, license, repo
│       └── versions/
│           ├── 1.0.0.toml
│           └── 1.0.1.toml
└── .gitea/workflows/
    ├── validate-pr.yml
    └── auto-merge.yml

2.2 Recipe versions/<v>.toml

schema = 1
name = "mylib"
version = "1.0.0"

[source]
type   = "git"
url    = "https://gitea/me/mylib"
commit = "abc123…"                   # 40-char
sha256 = "sha256-…"                  # FOD pin

[dependencies]
fmt      = "10.2"
otherlib = { version = "0.3", registry = "cargoxx" }

[lock]
nixpkgs_rev     = "…"
flake_utils_rev = "…"
cargoxx_rev     = "…"

[meta]
description = "…"
homepage    = "https://…"
license     = "MIT"

2.3 cargoxx-pkgs flake (the substitution surface)

{
  description = "cargoxx package registry";
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/<rev>";
  inputs.flake-utils.url = "github:numtide/flake-utils";
  inputs.cargoxx.url = "git+https://gitea/.../cargoxx?rev=<rev>";

  outputs = { self, nixpkgs, flake-utils, cargoxx }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system; };

        mkPackage = recipeFile:
          let r = builtins.fromTOML (builtins.readFile recipeFile);
          in cargoxx.lib.${system}.buildCppPackage {
               src = pkgs.fetchgit {
                 inherit (r.source) url;
                 rev    = r.source.commit;
                 hash   = r.source.sha256;
               };
             };

        # Enumerate recipes/<name>/versions/<v>.toml, build packagesAttrs.
        packagesAttrs = …;
      in {
        packages = packagesAttrs;
      });
}

CI evaluates this flake, builds the new attr, pushes $out to the binary cache. Consumers later get a cache hit for the SAME derivation hash via builtins.getFlake at the same registry_rev.

2.4 Publishing flow (cargoxx publish)

cargoxx publish [--dry-run]
1. Validate Cargoxx.toml has name, version, description, license, repository.
2. Validate working tree clean; current commit on a remote branch.
3. Validate Cargoxx.lock pins every cargoxx-source dep to a registry version
   (path = "..." deps disallowed in publish).
4. Compute recipe TOML:
     [source].url    = git remote URL
     [source].commit = HEAD commit
     [source].sha256 = sha256 of `git archive HEAD`
     [dependencies]  = mirror Cargoxx.toml
     [lock]          = mirror Cargoxx.lock pins
5. Clone cargoxx-pkgs (cached under ~/.cache/cargoxx/registry-clone).
6. Create branch publish/<name>-<version>; new package → also create
   recipes/<name>/{maintainers.txt,meta.toml}.
7. Write recipes/<name>/versions/<version>.toml; commit.
8. Push branch; `tea pr create` (tea login is already set up).

2.5 Validation bot (.gitea/workflows/validate-pr.yml)

Runs on every PR via self-hosted runners:

  1. Schema check: PR touches only recipes/<name>/**; recipe TOML validates.
  2. Source fixity: re-fetch [source].url at [source].commit, recompute sha256, compare.
  3. Build: nix build .#packages.x86_64-linux.<name>.
  4. Cache push: 
    nix copy --to "file:///srv/cargoxx-cache?secret-key=$KEY" \
  .#packages.x86_64-linux.<name>
    /srv/cargoxx-cache served by nginx as the binary cache.
  5. Dependency closure: every [dependencies] entry exists at the same PR HEAD.
  6. Maintainer match:
    • New package: PR author becomes maintainer (write maintainers.txt).
    • Existing package: PR author appears in maintainers.txt. Else label needs-human-review.
  7. On all-green + maintainer-match: label auto-merge.

auto-merge.yml triggers on the auto-merge label and merges via the Gitea API.

Maintainer transfer is a separate PR editing maintainers.txt. Same validation: PR author must already be on the list.

2.6 Resolving registry deps (cargoxx add foo)

cargoxx add foo[@<version>]
  1. Fetch https://gitea/.../cargoxx-pkgs/raw/branch/master/index.json
     (the index, emitted by CI, maps name → list of versions).
  2. Pick the highest matching version (or @<version>).
  3. Fetch recipes/foo/versions/<v>.toml.
  4. Write Cargoxx.toml:  foo = { version = "<v>", registry = "cargoxx" }.
  5. Write Cargoxx.lock:  source_kind = "registry",
                           registry_attr = "foo_<safe_ver>".
  6. Also persist lock.registry_rev = <current master commit>.

The bundled cargoxx wrapper has git, curl on PATH, so this works on non-Nix hosts too.

2.7 Binary cache config

Once the cache URL + signing key are decided, the cargoxx wrapper's NIX_CONFIG (in flake.nix's cargoxxNixConfig) gains:

substituters         = https://cache.cargoxx.<gitea-domain>/... https://cache.nixos.org
trusted-public-keys  = cache.cargoxx.<gitea-domain>:<base64>= cache.nixos.org:6NCH…=

Set via --set-default, so a user with a different setup can override by exporting NIX_CONFIG.

2.8 Hosting

Concern Choice
Repo host Self-hosted Gitea (your instance).
CI Gitea Actions on self-hosted runners (your existing pool).
Binary cache Static-HTTPS to start (nix copy --to file:///srv/cargoxx-cache); migrate to attic (https://github.com/zhaofengli/attic) if scale demands.
Publishing auth tea login (already configured).
Architectures x86_64-linux only for v1. aarch64/darwin are follow-ups.

2.9 Phasing

Phase Deliverable
2a cargoxx-pkgs repo skeleton + flake.nix (zero recipes).
2b .gitea/workflows/validate-pr.yml; manual recipe PRs.
2c cargoxx publish CLI; auto-merge.yml.
2d cargoxx add ↔ registry index; CI cache push; wrapper substituters config.

Module BMI strategy

Following the libstdc++/libc++ model in nixpkgs: ship source .cppm via install(TARGETS … FILE_SET CXX_MODULES …), ship the compiled archive (libfoo.a) of impl bits. Consumers regenerate the BMI in their own build (cmake's FILE_SET CXX_MODULES handles it when find_package(foo) is called). The compiled archive is what's cached — the BMI is cheap to regenerate against an already-cached archive.

cargoxx already enables CMAKE_CXX_MODULE_STD ON, which CMake uses to rebuild the std BMI in each consumer project. The same machinery covers cargoxx libraries.

Why buildCppPackage doesn't fetch source for registry deps

The registry flake (cargoxx-pkgs) owns the source-fetch → cargoxx-build → cmake-install pipeline. Its packages.<system>.<name> is an input-addressed derivation: same sources + same toolchain pins → same hash → cache hit.

Consumers reference that derivation via builtins.getFlake at the locked registry_rev. Nix's binary cache substitutes the $out directly — no source clone, no compile. Source fetch + build live inside the registry's eval, run once per recipe at CI time. This is the same model as nixpkgs/cache.nixos.org: you don't fetch fmt's source to use it from nixpkgs.

Open follow-ups

  1. Gitea domain + cache URL (for wrapper config in Phase 2d).
  2. Signing key generation (nix-store --generate-binary-cache-key cache.cargoxx.<domain> <secret> <public>); private into runner secrets, public into wrapper config.
  3. Multi-arch (aarch64-linux, *-darwin) — Phase 3+.
  4. Conan-style wrap recipes for non-cargoxx libraries — design now, defer implementation.
Reference in New Issue View Git Blame Copy Permalink