Collector

The lunar collect CLI for collecting SDLC metadata into a component's JSON, and the environment available to Lunar collectors.

The Collector Bash SDK is a set of Lunar CLI subcommands that allow you to collect SDLC metadata from within a collector, or from external systems, such as your CI system directly.

Collector environment

Earthly Lunar executes collectors in an environment set up with the following variables:

• LUNAR_HUB_HOST: The host of the Lunar Hub.
• LUNAR_HUB_TOKEN: The authentication token for the Lunar Hub.
• LUNAR_HUB_GRPC_PORT: The gRPC port of the Lunar Hub.
• LUNAR_HUB_HTTP_PORT: The HTTP port of the Lunar Hub.
• LUNAR_HUB_INSECURE: Whether to skip SSL verification of the Lunar Hub.
• LUNAR_BIN_DIR: The directory where the Lunar CLI is installed.
• LUNAR_COLLECTOR_NAME: The name of the collector.
• LUNAR_COLLECTOR_CI_PIPELINE: The CI pipeline of the component that the collector is running for (if this is a CI collector). 🚧 Coming Soon
• LUNAR_COMPONENT_ID: The ID of the component that the collector is running for in github.com/.../... format.
• LUNAR_COMPONENT_DOMAIN: The domain of the component.
• LUNAR_COMPONENT_OWNER: The owner of the component.
• LUNAR_COMPONENT_HEAD_BRANCH: The head branch of the PR (branch that contains the changes), if applicable.
• LUNAR_COMPONENT_BASE_BRANCH: The base branch of the PR (branch to merge the changes into), if applicable.
• LUNAR_COMPONENT_PR: The PR number of the component, if applicable.
• LUNAR_COMPONENT_TAGS: The tags of the component as a JSON array.
• LUNAR_COMPONENT_GIT_SHA: The Git SHA of the component that the collector is running for.
• LUNAR_COMPONENT_META: The component's .meta (an arbitrary key→value map, e.g. pagerduty/service-id) as a JSON object. Only set when the component has meta.
• LUNAR_SECRET_<name>: Any secret configured for collectors — either via HUB_COLLECTOR_SECRETS=NAME:VALUE,NAME2:VALUE2 or at runtime using lunar secret set <name> <value>.

CI collector environment

For ci-{before,after}-job, ci-{before,after}-step and ci-{before,after}-command hooks the following entries are also available:

• LUNAR_CI: The CI provider identifier (github, buildkite, ...).
• LUNAR_CI_PIPELINE_RUN_ID: Unique identifier for the current pipeline run. Pipeline are the top-level units of execution in a CI system. Also known as "workflow" in some CIs.
• LUNAR_CI_PIPELINE_RUN_ATTEMPT: Pipeline run attempt (1-based).
• LUNAR_CI_PIPELINE_DEFINITION_REF: URL to the pipeline source code.
• LUNAR_CI_PIPELINE_NAME: Name of the pipeline, if available.
• LUNAR_CI_PIPELINE_TRIGGERED_BY_RUN_ID: For a pipeline run started by another pipeline's run, that run's ID (e.g. in GitHub Actions, the run behind an on: workflow_run trigger).
• LUNAR_CI_JOB_ID: Job id. Jobs are units of parallel execution within a pipeline. Also known as "stages" in some CIs.
• LUNAR_CI_JOB_NAME: Job name, if available.

For ci-{before,after}-step and ci-{before,after}-command hooks the following entries are also available:

• LUNAR_CI_STEP_INDEX: Step index (1-based). Steps are the sequential units of execution within a job.
• LUNAR_CI_STEP_ID: Step ID (the step's id: in the GitHub Actions workflow), when the step has one.
• LUNAR_CI_GITHUB_ACTION: The action a GitHub Actions uses: step runs, as <owner>/<repo>[/<subdirectory>]@<ref> (e.g. actions/checkout@v4, github/codeql-action/analyze@v3; the ref as written, a SHA when pinned). Empty for steps that run a script, a local action, a docker:// image, or a composite or Docker container action.

For ci-{before,after}-command hooks the following entries are also available:

• LUNAR_CI_COMMAND: Command and arguments of the hooked command, as a JSON string.
• LUNAR_CI_COMMAND_BIN: Binary name only (no directory).
• LUNAR_CI_COMMAND_BIN_DIR: Directory containing the binary.
• LUNAR_CI_COMMAND_ARGS: Command arguments (excluding the binary) as a JSON array.
• LUNAR_CI_COMMAND_PID: Process ID of the hooked command.
• LUNAR_CI_COMMAND_PPID: Parent process PID of the hooked command.

after-ci-pipeline collector environment

An after-ci-pipeline collector runs on a Lunar runner for each finished CI workflow run. It gets the collector environment above, with LUNAR_COMPONENT_ID and LUNAR_COMPONENT_GIT_SHA naming the component and the commit the run built, plus the run itself in:

• LUNAR_CI_PIPELINE_RUN_ID: The workflow run ID.
• LUNAR_CI_PIPELINE_RUN_ATTEMPT: The run attempt (1-based). A re-run is a new attempt.
• LUNAR_CI_PIPELINE_NAME: The workflow name.
• LUNAR_CI_PIPELINE_RUN_CONCLUSION: How the run ended (success, failure, cancelled, ...).

For a run another pipeline's run started, LUNAR_COMPONENT_GIT_SHA is the commit the chain started from (details), and these are set too:

• LUNAR_CI_PIPELINE_HEAD_SHA: The commit the CI system recorded the run at.
• LUNAR_CI_PIPELINE_TRIGGERED_BY_RUN_ID: The run that started it.
• LUNAR_CI_PIPELINE_ORIGIN_SOURCE: How the Hub found the chain's commit: tracer, run-name, or unresolved (then LUNAR_COMPONENT_GIT_SHA is the CI system's commit).

lunar collect CLI command

• Form:

  lunar collect [--component <name>] [--sha <commit>] [--pr <number>] \
                [--json] [--array-append] [--pretty] \
                <json-path> <value> [<json-path> <value> ...]

lunar collect collects SDLC metadata into a component's JSON, taking one or more <json-path> <value> pairs. Inside a collector the target component and commit default to the current run, so you collect by passing the paths and values to record — use --json for non-string values like numbers:

lunar collect --json .testing.coverage.percentage 87.4
lunar collect .testing.coverage.source.tool codecov

Out-of-band collection

Collection can also happen outside of Lunar collectors — for example, in a CD pipeline, to associate release information with a previously built component's JSON. Pass --component and --sha to collect against a specific component and commit:

lunar collect --component github.com/my-org/my-repo --sha "$COMMIT_SHA" \
  .release.tag v1.2.3 \
  .release.image registry.example.com/my-app:v1.2.3

The Hub records the collection and re-evaluates policies for that SHA; verify the merged result with lunar component get-json <component> --git-sha <sha>.

With CI job identity, the write must target the job's commit or pull request; pass --pr for a pull request. A write from a branch the component does not track, without --pr, is stored but not evaluated, and only lunar component get-json --git-sha returns it. A write with --pr is evaluated for that pull request once Lunar Hub has recorded the commit in it; a pull request whose base branch no component tracks, such as a stacked one, is stored but not evaluated.

--component <name>

• Type: string
• Optional

The component the metadata is associated with, in github.com/org/repo form. Defaults to $LUNAR_COMPONENT_ID (set automatically inside a collector); pass it to collect against a specific component.

--sha <commit>

• Type: string
• Optional

The full 40-character commit SHA to key the collection on. Defaults to $GITHUB_SHA, which GitHub Actions sets automatically — in any other CI, including GitLab, pass it explicitly (for example --sha "$CI_COMMIT_SHA").

--pr <number>

• Type: int
• Optional

Optional PR number, to collect against a PR rather than a branch commit.

--json

The --json flag is used to specify that the value should be interpreted as a JSON object, JSON array, or JSON scalar (string, number, etc). If the flag is not provided, the value is interpreted as a raw string.

Example:

lunar collect --json '.foo.bar1' 'true'              # interprets as a JSON scalar (boolean)
lunar collect '.foo.bar2' 'true'                     # interprets as a raw string
lunar collect --json '.foo.bar3' '{"baz": "value 3"}' # interprets as a JSON object
lunar collect --json '.foo.bar4' '["value 4"]'        # interprets as a JSON array

Will result in:

{
  "foo": {
    "bar1": true,
    "bar2": "true",
    "bar3": {"baz": "value 3"},
    "bar4": ["value 4"]
  }
}

Note: Without --json, values are always treated as raw strings regardless of their content. For example, lunar collect '.foo.bar' '{"key": "val"}' would store the literal string "{\"key\": \"val\"}", not a JSON object.

--array-append

The --array-append flag is used to specify that the value should be appended to an array at the specified JSON path. If the path does not exist, a new array will be created. If the path exists and is not an array, the value will be replaced with a new array containing the existing value and the new value.

Since arrays are concatenated during the component JSON merge, using --array-append is equivalent to wrapping the value in a JSON array. The following two commands are equivalent:

lunar collect --array-append '.foo.bar' 'a value'
lunar collect --json '.foo.bar' '["a value"]'

Example:

lunar collect --json --array-append '.foo.bar' '{"baz": "value 1"}'
lunar collect --json --array-append '.foo.bar' '{"baz": "value 2"}'

Will result in:

{
  "foo": {
    "bar": [
      {"baz": "value 1"},
      {"baz": "value 2"}
    ]
  }
}

--follow-processed-sha

Write to the component's recorded processing SHA instead of the requested SHA. If commit Y references processing commit X, this flag merges your JSON into X unchanged. Without the flag, collection writes to Y.

The source must be known and available. If it cannot be resolved unambiguously, collection returns an error. Older Hubs ignore this flag and write to the requested SHA.

For example, collect deployment information at the processing commit:

lunar collect --component github.com/acme/monorepo/api \
  --sha "$GITHUB_SHA" --follow-processed-sha --json \
  .deployment.attempts.run_123 '{"status":"approved","author":"deploy-bot"}'

--pretty | -p

Pretty-print the JSON output.

<json-path>

• Type: string

The JSON path is used to specify the location in the JSON object where the value should be stored. The path is specified using dot notation, where each level of the object is separated by a dot. For example, .foo.bar.baz would specify the baz property of the bar object, which is a property of the foo object.

<value>

By default, the value is interpreted as a raw string. If the --json flag is provided, the value is interpreted as a JSON object, JSON array, or JSON scalar (string, number, etc).

If the value is -, the value is read from standard input. This is useful for piping the output of a command into the lunar collect command.