449 lines
16 KiB
Markdown
449 lines
16 KiB
Markdown
# Library reuse + public registry
|
|
|
|
Design plan for two interlocking features:
|
|
|
|
1. **Library reuse** — cargoxx-built libraries are consumable by other
|
|
cargoxx projects, by plain CMake projects (`find_package`), and by
|
|
plain clang invocations (`pkg-config`).
|
|
2. **Public registry + publish** — a Gitea-hosted package registry
|
|
(`cargoxx-pkgs`), a `cargoxx publish` CLI, and self-hosted CI that
|
|
validates PRs, builds, pushes derivations to a binary cache, and
|
|
auto-merges.
|
|
|
|
Phase 1 covers library reuse end-to-end with local path and git deps.
|
|
Phase 2 layers the registry on top, with cache-backed substitution so
|
|
consumers don't recompile.
|
|
|
|
---
|
|
|
|
## Part 1 — reusable libraries
|
|
|
|
### 1.1 Producer install layout
|
|
|
|
```
|
|
$out/
|
|
├── bin/ # binaries (existing path)
|
|
├── lib/
|
|
│ ├── libmylib.a # archive of impl bits
|
|
│ ├── pkgconfig/mylib.pc # pkg-config descriptor
|
|
│ └── cmake/mylib/
|
|
│ ├── mylibConfig.cmake # entry point for find_package
|
|
│ ├── mylibConfigVersion.cmake # semver gate
|
|
│ └── mylibTargets.cmake # IMPORTED tgt + CXX_MODULES FILE_SET
|
|
└── include/mylib/ # public headers + source-form modules
|
|
```
|
|
|
|
Covers three consumption paths:
|
|
|
|
- **cargoxx → cargoxx**: consumer's CMakeLists emits
|
|
`find_package(mylib CONFIG REQUIRED)` + link against `mylib::mylib`.
|
|
`$out` is on `CMAKE_PREFIX_PATH` via `buildInputs`.
|
|
- **plain CMake**: same `find_package` call, manual `HINTS` if not on
|
|
the default prefix path.
|
|
- **plain clang + pkg-config**: works for non-module libraries
|
|
(header-only or `.a` with extern API). Module libraries require a
|
|
CMake-aware consumer — industry-wide pkg-config has no story for
|
|
module BMIs.
|
|
|
|
### 1.2 Codegen changes (`src/codegen/cmake.cpp`)
|
|
|
|
When `layout.library` is set, append:
|
|
|
|
```cmake
|
|
include(GNUInstallDirs)
|
|
include(CMakePackageConfigHelpers)
|
|
|
|
install(TARGETS mylib
|
|
EXPORT mylibTargets
|
|
FILE_SET CXX_MODULES DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/mylib
|
|
ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR})
|
|
install(EXPORT mylibTargets
|
|
FILE mylibTargets.cmake
|
|
NAMESPACE mylib::
|
|
DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/mylib)
|
|
|
|
# Config.cmake.in written inline so we don't need an off-tree template file.
|
|
file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/mylibConfig.cmake.in [[
|
|
@PACKAGE_INIT@
|
|
include(CMakeFindDependencyMacro)
|
|
@FIND_DEPENDENCY_BLOCK@
|
|
include("${CMAKE_CURRENT_LIST_DIR}/mylibTargets.cmake")
|
|
check_required_components(mylib)
|
|
]])
|
|
configure_package_config_file(
|
|
${CMAKE_CURRENT_BINARY_DIR}/mylibConfig.cmake.in
|
|
${CMAKE_CURRENT_BINARY_DIR}/mylibConfig.cmake
|
|
INSTALL_DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/mylib)
|
|
write_basic_package_version_file(
|
|
${CMAKE_CURRENT_BINARY_DIR}/mylibConfigVersion.cmake
|
|
VERSION ${PROJECT_VERSION} COMPATIBILITY SameMajorVersion)
|
|
install(FILES
|
|
${CMAKE_CURRENT_BINARY_DIR}/mylibConfig.cmake
|
|
${CMAKE_CURRENT_BINARY_DIR}/mylibConfigVersion.cmake
|
|
DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/mylib)
|
|
|
|
# pkg-config descriptor — also inline.
|
|
file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/mylib.pc.in [[
|
|
prefix=@CMAKE_INSTALL_PREFIX@
|
|
exec_prefix=${prefix}
|
|
libdir=${prefix}/lib
|
|
includedir=${prefix}/include
|
|
|
|
Name: @PROJECT_NAME@
|
|
Version: @PROJECT_VERSION@
|
|
Description: @PROJECT_DESCRIPTION@
|
|
Cflags: -I${includedir}
|
|
Libs: -L${libdir} -lmylib
|
|
]])
|
|
configure_file(${CMAKE_CURRENT_BINARY_DIR}/mylib.pc.in
|
|
${CMAKE_CURRENT_BINARY_DIR}/mylib.pc @ONLY)
|
|
install(FILES ${CMAKE_CURRENT_BINARY_DIR}/mylib.pc
|
|
DESTINATION ${CMAKE_INSTALL_LIBDIR}/pkgconfig)
|
|
```
|
|
|
|
`@FIND_DEPENDENCY_BLOCK@` is computed by codegen — one `find_dependency(X)`
|
|
line per cargoxx-source dep, so the transitive graph reconstitutes when
|
|
a consumer does `find_package(mylib …)`.
|
|
|
|
Binaries also get an install rule so `cmake --install` handles
|
|
everything in one shot:
|
|
|
|
```cmake
|
|
install(TARGETS mylib_bin RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR})
|
|
install(TARGETS tool RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR})
|
|
```
|
|
|
|
### 1.3 `buildCppPackage.installPhase` switches to `cmake --install`
|
|
|
|
```nix
|
|
installPhase = ''
|
|
cmake --install build/release --prefix $out
|
|
'';
|
|
```
|
|
|
|
Replaces the `cp -a build/release/bin/. $out/bin/` hack — now bins,
|
|
library archive, headers, cmake config, pc file all land in the right
|
|
place from one install command.
|
|
|
|
### 1.4 Manifest schema — discriminated dep source
|
|
|
|
Backward-compat (string form unchanged):
|
|
|
|
```toml
|
|
[dependencies]
|
|
sqlite = "*" # nixpkgs / linkdb (today)
|
|
fmt = { version = ">=10" } # nixpkgs explicit (today)
|
|
mylib = { path = "../mylib" } # cargoxx local path ← NEW
|
|
remote = { git = "https://gitea/me/remote", rev = "abc123" } # cargoxx git ← NEW
|
|
shared = { version = "1.0", registry = "cargoxx" } # cargoxx registry ← NEW
|
|
```
|
|
|
|
`manifest::Dependency` grows discriminated source fields. Parser rejects
|
|
unknown keys in dep tables (today silently ignored).
|
|
|
|
### 1.5 Lockfile schema additions
|
|
|
|
For cargoxx-source deps the lockfile carries source provenance and the
|
|
recipe shape (`find_package`, `targets`) — codegen consumes the latter
|
|
unchanged, so cargoxx deps look like any other linkdb recipe to the
|
|
emitted CMakeLists.
|
|
|
|
Path deps: record relative path.
|
|
Git deps: record `(url, commit, sha256)`.
|
|
Registry deps: record only `(registry_rev, registry_attr)`; the source
|
|
URL + sha256 live in the registry repo, not the consumer's lockfile.
|
|
|
|
### 1.6 `buildCppPackage` recursion
|
|
|
|
```nix
|
|
buildCargoxxDep = p:
|
|
if p.source_kind == "path" then
|
|
buildCppPackage { src = src + ("/" + p.path); }
|
|
|
|
else if p.source_kind == "git" then
|
|
let depSrc = pkgs.fetchgit {
|
|
inherit (p) url;
|
|
rev = p.commit;
|
|
hash = p.sha256;
|
|
};
|
|
in buildCppPackage { src = depSrc; }
|
|
|
|
else if p.source_kind == "registry" then
|
|
# Pull the prebuilt $out from the registry flake — no source fetch,
|
|
# no recompile. The registry's `packages.<system>.<attr>` derivation
|
|
# is content-addressed and substituted from the binary cache.
|
|
let registry = builtins.getFlake
|
|
"git+https://gitea/.../cargoxx-pkgs?rev=${lock.registry_rev}";
|
|
in registry.packages.${system}.${p.registry_attr}
|
|
|
|
else throw "unknown source_kind: ${p.source_kind}";
|
|
|
|
cargoxxDepDrvs = map buildCargoxxDep
|
|
(builtins.filter (p: p ? source_kind) lock.package);
|
|
```
|
|
|
|
`cargoxxDepDrvs` join `buildInputs`. nixpkgs's `cmakeConfigurePhase`
|
|
automatically prepends every `buildInputs` `$out` to `CMAKE_PREFIX_PATH`,
|
|
so consumer `find_package(<dep> CONFIG REQUIRED)` resolves without
|
|
extra plumbing.
|
|
|
|
### 1.7 Resolver flow for cargoxx-source deps
|
|
|
|
```
|
|
manifest dep → cmd_build::resolve_dep()
|
|
├─ source kind == Nixpkgs/Auto: existing resolver chain
|
|
├─ source kind == CargoxxPath:
|
|
│ read ${path}/Cargoxx.toml, take [package].name/version,
|
|
│ synthesize Recipe { find_package = "<name> CONFIG REQUIRED",
|
|
│ targets = ["<name>::<name>"] }
|
|
├─ source kind == CargoxxGit:
|
|
│ fetch the repo at rev (resolver cache),
|
|
│ same as CargoxxPath against the checkout
|
|
└─ source kind == CargoxxRegistry:
|
|
fetch the registry's recipe TOML, take name + version,
|
|
synthesize the same Recipe shape.
|
|
```
|
|
|
|
The recipe shape — `find_package(<name> CONFIG REQUIRED)` plus
|
|
`<name>::<name>` target — matches what the producer's installed
|
|
`<name>Config.cmake` defines. Codegen needs no cargoxx-dep special case.
|
|
|
|
### 1.8 Phasing
|
|
|
|
| Phase | Deliverable |
|
|
|---|---|
|
|
| 1a | install rules + Config.cmake + .pc; `cmake --install` in buildCppPackage |
|
|
| 1b | `{ path = "..." }` deps end-to-end (manifest parse + lock + recursive buildCppPackage); e2e fixture |
|
|
| 1c | `{ git = "...", rev = "..." }` deps via `pkgs.fetchgit` |
|
|
| 1d | `{ version = "1.0", registry = "cargoxx" }` deps (gated on Phase 2 being live) |
|
|
|
|
---
|
|
|
|
## Part 2 — Gitea-hosted registry
|
|
|
|
### 2.1 `cargoxx-pkgs` repo layout
|
|
|
|
```
|
|
cargoxx-pkgs/
|
|
├── flake.nix
|
|
├── recipes/
|
|
│ └── <name>/
|
|
│ ├── maintainers.txt # gitea usernames, one per line
|
|
│ ├── meta.toml # description, homepage, license, repo
|
|
│ └── versions/
|
|
│ ├── 1.0.0.toml
|
|
│ └── 1.0.1.toml
|
|
└── .gitea/workflows/
|
|
├── validate-pr.yml
|
|
└── auto-merge.yml
|
|
```
|
|
|
|
### 2.2 Recipe `versions/<v>.toml`
|
|
|
|
```toml
|
|
schema = 1
|
|
name = "mylib"
|
|
version = "1.0.0"
|
|
|
|
[source]
|
|
type = "git"
|
|
url = "https://gitea/me/mylib"
|
|
commit = "abc123…" # 40-char
|
|
sha256 = "sha256-…" # FOD pin
|
|
|
|
[dependencies]
|
|
fmt = "10.2"
|
|
otherlib = { version = "0.3", registry = "cargoxx" }
|
|
|
|
[lock]
|
|
nixpkgs_rev = "…"
|
|
flake_utils_rev = "…"
|
|
cargoxx_rev = "…"
|
|
|
|
[meta]
|
|
description = "…"
|
|
homepage = "https://…"
|
|
license = "MIT"
|
|
```
|
|
|
|
### 2.3 `cargoxx-pkgs` flake (the substitution surface)
|
|
|
|
```nix
|
|
{
|
|
description = "cargoxx package registry";
|
|
inputs.nixpkgs.url = "github:NixOS/nixpkgs/<rev>";
|
|
inputs.flake-utils.url = "github:numtide/flake-utils";
|
|
inputs.cargoxx.url = "git+https://gitea/.../cargoxx?rev=<rev>";
|
|
|
|
outputs = { self, nixpkgs, flake-utils, cargoxx }:
|
|
flake-utils.lib.eachDefaultSystem (system:
|
|
let
|
|
pkgs = import nixpkgs { inherit system; };
|
|
|
|
mkPackage = recipeFile:
|
|
let r = builtins.fromTOML (builtins.readFile recipeFile);
|
|
in cargoxx.lib.${system}.buildCppPackage {
|
|
src = pkgs.fetchgit {
|
|
inherit (r.source) url;
|
|
rev = r.source.commit;
|
|
hash = r.source.sha256;
|
|
};
|
|
};
|
|
|
|
# Enumerate recipes/<name>/versions/<v>.toml, build packagesAttrs.
|
|
packagesAttrs = …;
|
|
in {
|
|
packages = packagesAttrs;
|
|
});
|
|
}
|
|
```
|
|
|
|
CI evaluates this flake, builds the new attr, pushes `$out` to the
|
|
binary cache. Consumers later get a cache hit for the SAME
|
|
derivation hash via `builtins.getFlake` at the same `registry_rev`.
|
|
|
|
### 2.4 Publishing flow (`cargoxx publish`)
|
|
|
|
```
|
|
cargoxx publish [--dry-run]
|
|
1. Validate Cargoxx.toml has name, version, description, license, repository.
|
|
2. Validate working tree clean; current commit on a remote branch.
|
|
3. Validate Cargoxx.lock pins every cargoxx-source dep to a registry version
|
|
(path = "..." deps disallowed in publish).
|
|
4. Compute recipe TOML:
|
|
[source].url = git remote URL
|
|
[source].commit = HEAD commit
|
|
[source].sha256 = sha256 of `git archive HEAD`
|
|
[dependencies] = mirror Cargoxx.toml
|
|
[lock] = mirror Cargoxx.lock pins
|
|
5. Clone cargoxx-pkgs (cached under ~/.cache/cargoxx/registry-clone).
|
|
6. Create branch publish/<name>-<version>; new package → also create
|
|
recipes/<name>/{maintainers.txt,meta.toml}.
|
|
7. Write recipes/<name>/versions/<version>.toml; commit.
|
|
8. Push branch; `tea pr create` (tea login is already set up).
|
|
```
|
|
|
|
### 2.5 Validation bot (`.gitea/workflows/validate-pr.yml`)
|
|
|
|
Runs on every PR via self-hosted runners:
|
|
|
|
1. **Schema check**: PR touches only `recipes/<name>/**`; recipe TOML
|
|
validates.
|
|
2. **Source fixity**: re-fetch `[source].url` at `[source].commit`,
|
|
recompute sha256, compare.
|
|
3. **Build**: `nix build .#packages.x86_64-linux.<name>`.
|
|
4. **Cache push**:
|
|
```
|
|
nix copy --to "file:///srv/cargoxx-cache?secret-key=$KEY" \
|
|
.#packages.x86_64-linux.<name>
|
|
```
|
|
`/srv/cargoxx-cache` served by nginx as the binary cache.
|
|
5. **Dependency closure**: every `[dependencies]` entry exists at the
|
|
same PR HEAD.
|
|
6. **Maintainer match**:
|
|
- New package: PR author becomes maintainer (write
|
|
`maintainers.txt`).
|
|
- Existing package: PR author appears in `maintainers.txt`. Else
|
|
label `needs-human-review`.
|
|
7. On all-green + maintainer-match: label `auto-merge`.
|
|
|
|
`auto-merge.yml` triggers on the `auto-merge` label and merges via
|
|
the Gitea API.
|
|
|
|
Maintainer transfer is a separate PR editing `maintainers.txt`. Same
|
|
validation: PR author must already be on the list.
|
|
|
|
### 2.6 Resolving registry deps (`cargoxx add foo`)
|
|
|
|
```
|
|
cargoxx add foo[@<version>]
|
|
1. Fetch https://gitea/.../cargoxx-pkgs/raw/branch/master/index.json
|
|
(the index, emitted by CI, maps name → list of versions).
|
|
2. Pick the highest matching version (or @<version>).
|
|
3. Fetch recipes/foo/versions/<v>.toml.
|
|
4. Write Cargoxx.toml: foo = { version = "<v>", registry = "cargoxx" }.
|
|
5. Write Cargoxx.lock: source_kind = "registry",
|
|
registry_attr = "foo_<safe_ver>".
|
|
6. Also persist lock.registry_rev = <current master commit>.
|
|
```
|
|
|
|
The bundled cargoxx wrapper has `git`, `curl` on PATH, so this works
|
|
on non-Nix hosts too.
|
|
|
|
### 2.7 Binary cache config
|
|
|
|
Once the cache URL + signing key are decided, the cargoxx wrapper's
|
|
`NIX_CONFIG` (in `flake.nix`'s `cargoxxNixConfig`) gains:
|
|
|
|
```
|
|
substituters = https://cache.cargoxx.<gitea-domain>/... https://cache.nixos.org
|
|
trusted-public-keys = cache.cargoxx.<gitea-domain>:<base64>= cache.nixos.org:6NCH…=
|
|
```
|
|
|
|
Set via `--set-default`, so a user with a different setup can
|
|
override by exporting `NIX_CONFIG`.
|
|
|
|
### 2.8 Hosting
|
|
|
|
| Concern | Choice |
|
|
|---|---|
|
|
| Repo host | Self-hosted Gitea (your instance). |
|
|
| CI | Gitea Actions on self-hosted runners (your existing pool). |
|
|
| Binary cache | Static-HTTPS to start (`nix copy --to file:///srv/cargoxx-cache`); migrate to `attic` (https://github.com/zhaofengli/attic) if scale demands. |
|
|
| Publishing auth | `tea login` (already configured). |
|
|
| Architectures | `x86_64-linux` only for v1. aarch64/darwin are follow-ups. |
|
|
|
|
### 2.9 Phasing
|
|
|
|
| Phase | Deliverable |
|
|
|---|---|
|
|
| 2a | `cargoxx-pkgs` repo skeleton + `flake.nix` (zero recipes). |
|
|
| 2b | `.gitea/workflows/validate-pr.yml`; manual recipe PRs. |
|
|
| 2c | `cargoxx publish` CLI; `auto-merge.yml`. |
|
|
| 2d | `cargoxx add` ↔ registry index; CI cache push; wrapper substituters config. |
|
|
|
|
---
|
|
|
|
## Module BMI strategy
|
|
|
|
Following the libstdc++/libc++ model in nixpkgs: ship **source `.cppm`**
|
|
via `install(TARGETS … FILE_SET CXX_MODULES …)`, ship the **compiled
|
|
archive** (`libfoo.a`) of impl bits. Consumers regenerate the BMI in
|
|
their own build (cmake's `FILE_SET CXX_MODULES` handles it when
|
|
`find_package(foo)` is called). The compiled archive is what's
|
|
cached — the BMI is cheap to regenerate against an already-cached
|
|
archive.
|
|
|
|
cargoxx already enables `CMAKE_CXX_MODULE_STD ON`, which CMake uses to
|
|
rebuild the `std` BMI in each consumer project. The same machinery
|
|
covers cargoxx libraries.
|
|
|
|
---
|
|
|
|
## Why `buildCppPackage` doesn't fetch source for registry deps
|
|
|
|
The registry flake (`cargoxx-pkgs`) owns the
|
|
`source-fetch → cargoxx-build → cmake-install` pipeline. Its
|
|
`packages.<system>.<name>` is an input-addressed derivation: same
|
|
sources + same toolchain pins → same hash → cache hit.
|
|
|
|
Consumers reference that derivation via `builtins.getFlake` at the
|
|
locked `registry_rev`. Nix's binary cache substitutes the `$out`
|
|
directly — no source clone, no compile. Source fetch + build live
|
|
inside the registry's eval, run once per recipe at CI time. This is
|
|
the same model as nixpkgs/`cache.nixos.org`: you don't fetch fmt's
|
|
source to use it from nixpkgs.
|
|
|
|
---
|
|
|
|
## Open follow-ups
|
|
|
|
1. Gitea domain + cache URL (for wrapper config in Phase 2d).
|
|
2. Signing key generation
|
|
(`nix-store --generate-binary-cache-key cache.cargoxx.<domain>
|
|
<secret> <public>`); private into runner secrets, public into
|
|
wrapper config.
|
|
3. Multi-arch (`aarch64-linux`, `*-darwin`) — Phase 3+.
|
|
4. Conan-style wrap recipes for non-cargoxx libraries — design now,
|
|
defer implementation.
|