> For the complete documentation index, see [llms.txt](https://docs-lunar.earthly.dev/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs-lunar.earthly.dev/configuration/hub-configuration/data-retention.md).

# Data Retention

What a Lunar Hub install deletes on its own, what it keeps for the life of the install, and the environment variables that change either.

What a Lunar Hub install deletes on its own, what it keeps forever, and which knobs change that.

## Do you need this page?

Read it if you operate a Hub and want to bound its storage growth, or if you need to state a retention window to satisfy a compliance requirement. If you run [Lunar Dedicated](/install/lunar-hub/dedicated/overview.md), Earthly sets these for your install and you can skip to [What no window covers](#what-no-window-covers) to see what retention does not reach.

This page is about data the Hub stores. It is not about the CI Tracer's local files, which the [CLI configuration reference](/install/ci-tracer/configuration-reference.md) covers.

## The short version

**Run history is capped by a retention window, but the feature is off until you turn it on.** Set `HUB_RETENTION_ENABLED=true` and the Hub ages out runs, policy results, and collection records older than 90 days. Leave it off, the default, and those tables grow with your commit volume and nothing removes them.

Queue rows, catalog history, and on-disk working files are capped separately and always have been.

## Run and trend retention

The customer-facing contract is two windows rather than a knob per table, because there are only two questions to answer: how far back can you see a run, and how far back do your trends go.

| Variable                        | Default | Governs                                                                                                                  |
| ------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------ |
| `HUB_RETENTION_ENABLED`         | `false` | Master switch. Nothing below deletes anything while this is off                                                          |
| `HUB_RETENTION_RUNS`            | `90d`   | Script runs, policy runs, policy assertions and rollups, collection records, catalog JSON items, merged collection blobs |
| `HUB_RETENTION_DERIVED`         | `90d`   | Materialized trends, component scores, the runs listing                                                                  |
| `HUB_RETENTION_CONFIG`          | `90d`   | Superseded manifest generations, and the components, domains and script definitions they defined                         |
| `HUB_RETENTION_CASCADE_ENABLED` | `false` | Whether to prune those generations at all                                                                                |

Windows are written in days — `90d`, `365d`. Go durations such as `2160h` still parse, but days are what these knobs mean and the Hub reports them back that way.

**The runs and derived windows must be equal in this version, and the Hub refuses to boot if they are not.** Setting a longer derived window looks like it should work and does not: two of the derived surfaces are rebuilt from the run tables, so they truncate themselves back to the runs window within a day, silently. Rejecting the config at boot is better than accepting one the Hub cannot deliver. Divergence becomes supported once those surfaces are fixed.

### Pruning superseded config

Every time a repository's config is published, the Hub writes a new manifest generation with its own components, domains, and script definitions. Those accumulate, and the components underneath them are the bulk of it.

`HUB_RETENTION_CASCADE_ENABLED` turns on removing them, and it is separate from `HUB_RETENTION_ENABLED` because it is a different kind of deletion. The runs window removes *history* — rows describing something that happened. This removes the *definitions* those rows point at, and it is the only part of retention that deletes a component.

**A generation is pruned when it is both superseded and older than `HUB_RETENTION_CONFIG`.** It has its own window rather than following the runs window, because config churn and run history are kept for different reasons.

**The newest generation for a repository is never pruned, at any age.** It is that repository's live configuration, so removing it would leave collectors and policies with nothing to run against. There is no setting that overrides this.

A generation whose runs have not aged out yet is skipped rather than pruned, and picked up on a later pass once the runs sweep has caught up.

**Retention is off by default and stays off through an upgrade.** Turning it on for the first time on an established install has years of accumulated rows to remove, so it deliberately does not do that in one pass. `HUB_RETENTION_BATCH_SIZE` (5000) and `HUB_RETENTION_MAX_BATCHES_PER_RUN` (100) bound each run, and `HUB_RETENTION_INTERVAL` (1h) sets how often it runs. The backlog drains over days. Watch the Hub logs for `budget_exhausted` — while it is true there is still a backlog, and when it goes false the install has reached steady state.

### Commits, pull requests, and repositories

Aged on their own clocks rather than the runs window, because they are not runs. Commits age on when they were committed, pull requests on when they were opened, and **repositories on when Lunar last synced them** — a repository is an identity, so "not seen in the window" is the condition that means it is gone, where "created long ago" only means it is old. An active repository is never removed for age.

A commit is removed only once nothing still names it. That check is wider than it looks: thirteen tables reference a commit by its SHA and only two by a foreign key, so the database alone would not stop a delete that orphans, for example, a component's current head pointer. All of them are checked before a commit is removed.

## Reclaiming deleted space

Deleting rows caps growth. It does not shrink the database file: the freed space becomes reusable by that table, and stays allocated.

Handing it back to the operating system takes a compaction pass, which holds an exclusive lock on the table while it runs. Readers block for the duration and then continue — it is a stall, not an error — but on a large table that stall is measured in minutes, so it is off by default and wants a maintenance window.

| Variable                            | Default              | Governs                                                       |
| ----------------------------------- | -------------------- | ------------------------------------------------------------- |
| `HUB_RETENTION_VACUUM_ENABLED`      | `false`              | Master switch for compaction                                  |
| `HUB_RETENTION_VACUUM_SCHEDULE`     | `0 3 * * *`          | When it checks, not when it acts                              |
| `HUB_RETENTION_VACUUM_MIN_BYTES`    | `1073741824` (1 GiB) | Minimum reclaimable space to justify the stall                |
| `HUB_RETENTION_VACUUM_MIN_RATIO`    | `0.2`                | Minimum fraction of the table that is reclaimable             |
| `HUB_RETENTION_VACUUM_LOCK_TIMEOUT` | `5s`                 | How long to wait for the lock before giving up until tomorrow |
| `HUB_RETENTION_VACUUM_TIMEOUT`      | `2h`                 | Cap on one compaction                                         |

**It runs nightly and should do nothing on almost every run.** A table is compacted only when it clears both thresholds, and only one table is compacted per night. Expect it to matter once, after retention first drains a large backlog, and rarely after that.

Leave the lock timeout short. The exclusive lock queues behind whatever transaction is already running, and every query that arrives meanwhile queues behind it, so a compaction that waits patiently is how a stall becomes an outage. Giving up and retrying tomorrow costs nothing.

## Queue retention

Terminal jobs are the highest-churn rows in the system. The Hub's queue and the script operator's queue each keep their own, and each is tuned separately.

| Variable                            | Default | Governs                                            |
| ----------------------------------- | ------- | -------------------------------------------------- |
| `HUB_QUEUE_COMPLETED_JOB_RETENTION` | `24h`   | Jobs that finished successfully                    |
| `HUB_QUEUE_CANCELLED_JOB_RETENTION` | `24h`   | Jobs cancelled before completing                   |
| `HUB_QUEUE_DISCARDED_JOB_RETENTION` | `168h`  | Jobs that exhausted their retries                  |
| `OPERATOR_COMPLETED_JOB_RETENTION`  | `24h`   | As above, for the script-execution queue           |
| `OPERATOR_CANCELLED_JOB_RETENTION`  | `24h`   | Script-execution jobs cancelled before completing  |
| `OPERATOR_DISCARDED_JOB_RETENTION`  | `168h`  | Script-execution jobs that exhausted their retries |

Nothing reads a terminal job. The durable record of a run is `hub.snippet_runs`, so shortening these windows loses no history you can query. Discarded jobs are the exception worth keeping longer, since they are failures someone may want to inspect.

**Reach for the operator knobs first on a large fleet.** `operator_queue.river_job` takes one row per script-execution batch and churns harder than any other table.

## Catalog history

`HUB_CATALOG_VERSION_RETENTION` (default `90d`) bounds how long captured catalog versions are kept for the SQL API `catalog` view. Set it to `0` to disable pruning.

Each version stores a complete catalog document, roughly 1 MB at 30,000 components, so the window multiplies against your capture rate rather than your component count alone. The newest version is always kept regardless of age, so the view never goes empty.

## On-disk working files

These govern the Hub's volume, not the database.

| Variable                         | Default   | Governs                                    |
| -------------------------------- | --------- | ------------------------------------------ |
| `HUB_INSTALL_FILE_MAX_AGE_DAYS`  | `10`      | Manifest directories, pruned by age        |
| `HUB_INSTALL_FILE_MAX_DISK_SIZE` | `0` (off) | The same directories, pruned by total size |
| `HUB_BUNDLE_MAX_AGE`             | `2h`      | Policy bundles staged for execution        |
| `HUB_BUNDLE_NO_DELETE`           | `false`   | Keeps bundles for debugging                |

The Hub writes a directory per manifest generation, so on a busy config this is the volume's fastest-growing tenant. The size arm is off by default because a byte cap only makes sense against a known volume size. Set it if your volume is small enough that age alone could still fill it.

`HUB_BUNDLE_MAX_AGE` covers a scratch directory rather than the persistent volume, so it is rarely the knob you want.

## Script pods

Pods are reaped by the operator on a separate set of timers: `OPERATOR_TERMINAL_POD_GRACE_PERIOD` (5m), `OPERATOR_MAX_POD_AGE` (12h), `OPERATOR_RUNNING_POD_TIMEOUT` (6h), and `OPERATOR_PENDING_POD_TIMEOUT` (10m). `OPERATOR_RETAIN_FAILED_PODS` keeps failed pods for inspection and is off by default.

## Object storage

**Lunar never deletes from your buckets.** Run logs and run-bundle archives stay until something outside Lunar removes them. Nothing on this page changes that: the windows above govern database rows, not objects.

On a self-hosted install, set S3 lifecycle rules yourself, and check them against your own retention obligations first, since both buckets can contain credentials, script source, or PII surfaced from CI runs. The [self-hosted prerequisites](/install/lunar-hub/self-hosted/prerequisites.md#step-4-provision-s3-compatible-object-storage) cover the bucket layout. On [Lunar Dedicated](/install/lunar-hub/dedicated/overview.md) the buckets live in your own account, so the rules are still yours to set — agree them with Earthly during setup rather than assuming either side has.

## What no window covers

`HUB_RETENTION_RUNS` reaches the run tables and their dependants. These are outside it, at any setting:

* The newest manifest generation for each repository, and everything under it

Everything else is covered, though not all of it by the runs window: components, domains, and script definitions go with [the generation that defined them](#pruning-superseded-config) under `HUB_RETENTION_CONFIG`, and commits, pull requests, and repositories have [their own clocks](#commits-pull-requests-and-repositories). Both need `HUB_RETENTION_CASCADE_ENABLED`.

### What `0` means

**`0` is not one sentinel.** It means different things to different knobs on this page, and several of them mean the opposite of what you would expect.

| Knob                                             | `0` means                                                                                                                                                    |
| ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `HUB_RETENTION_RUNS` / `_DERIVED`                | Keep forever. Retention is disabled                                                                                                                          |
| `HUB_RETENTION_CONFIG`                           | Keep every superseded generation                                                                                                                             |
| `HUB_INSTALL_FILE_MAX_AGE_DAYS`                  | Keep forever. The pruner returns early                                                                                                                       |
| `HUB_CATALOG_VERSION_RETENTION`                  | Keep forever. Pruning is disabled                                                                                                                            |
| `HUB_QUEUE_*` / `OPERATOR_*` job retention       | **River's own default** — 24h, 24h, 168h. Not forever                                                                                                        |
| `HUB_BUNDLE_MAX_AGE`                             | **Delete every bundle on the next pass**, including one staged for execution                                                                                 |
| Pod timeouts (`OPERATOR_*`)                      | **Reap immediately**                                                                                                                                         |
| `HUB_RETENTION_VACUUM_LOCK_TIMEOUT` / `_TIMEOUT` | **Rejected at startup.** Not a sentinel — to Postgres a zero `lock_timeout` means wait forever, which is the behaviour the compaction section argues against |

The queue knobs are the trap worth naming, because the advice above is to keep discarded jobs longer than the rest: setting `HUB_QUEUE_DISCARDED_JOB_RETENTION=0` to keep them indefinitely silently gets you seven days. **To keep terminal jobs forever, set `-1ns`.** River's infinite sentinel is `-1`, and the value is parsed as a duration, so a bare `-1` is rejected for having no unit.

To keep bundles, use `HUB_BUNDLE_NO_DELETE=true` rather than a zero age.

## Sizing

Two things drive Hub database growth, and only one of them scales with the size of your fleet.

**Run history scales with commit volume, bounded by the retention window.** A component that never changes costs almost nothing; one receiving fifty commits a day writes runs fifty times a day. Estimate from your commit and merge rate over one window, not from your component count. With retention off, drop the window and the figure has no ceiling.

**Catalog history scales with capture rate and component count together**, at roughly 1 MB per captured version at 30,000 components.

The queue tables are bounded by the retention windows above and reach a steady state. If they do not, that is a backlog rather than a retention problem, and the queue-depth metrics on the Hub's `/metrics` endpoint will show it.

## Related

* [Day 2 Operations](/install/lunar-hub/self-hosted/day-2-operations.md) — upgrades, secret rotation, backup, and the diagnostics bundle
* [Prerequisites](/install/lunar-hub/self-hosted/prerequisites.md) — database and bucket sizing before install


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs-lunar.earthly.dev/configuration/hub-configuration/data-retention.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.