# prod-tools

Pull production diagnostics onto the dev box and reduce them to something
readable. Read-only with respect to production: every transfer is an rsync
**pull**, and the only commands run remotely are `find`, `du`, `test` and `ls`.

| | |
|---|---|
| `sync-prod-logs.sh` | rsync-pulls the logs into `projects/production/logs/` |
| `prod-log-digest.sh` | reduces that tree to `projects/production/digest/latest.md` |

## Setup

`rsync` is not in the linuxserver/code-server image. It is installed by
`codeserver-setup/install.sh`, which also creates `/config/secrets/` — so after
a container redeploy, run the rehydrate script before expecting this to work.

```bash
# 1. write the credentials template (also creates the directory)
./sync-prod-logs.sh --list

# 2. fill in PROD_HOST and drop a key beside it
vi /config/secrets/prod-sync.env
ssh-keygen -t ed25519 -f /config/secrets/prod_id_ed25519 -N ''
ssh-copy-id -i /config/secrets/prod_id_ed25519.pub root@<prod>

# 3. confirm what is reachable before moving any bytes
./sync-prod-logs.sh --list

# 4. go
./sync-prod-logs.sh
```

The credentials file lives at `/config/secrets/prod-sync.env`, deliberately
**outside `/config/workspace`** — none of the three git repos can reach it, so
it cannot be committed by accident. It is written `chmod 600`, and the sync
re-tightens it (and any `prod_id_*` key beside it) on every run: a key with
loose permissions is silently ignored by ssh, which then falls through to a
password prompt and hangs a cron run forever.

## Usage

```bash
./sync-prod-logs.sh                     # everything readable, then digest
./sync-prod-logs.sh --list              # probe only: exists? readable? how big?
./sync-prod-logs.sh --dry-run           # show the transfers, move nothing
./sync-prod-logs.sh --only web-error,cron
./sync-prod-logs.sh --since 30          # widen the window on dated sources
./sync-prod-logs.sh --full              # no date window at all
./sync-prod-logs.sh --domlogs           # add per-vhost Apache logs (BIG)
./sync-prod-logs.sh --install-cron      # hourly, quiet
./prod-log-digest.sh --top 40 > /tmp/d.md   # re-digest without re-syncing
```

Source names for `--only`: `web-error`, `cron`, `deploy`, `tenant-error`,
`mysql`, `forensics`, `messages`, `sar`, `domlogs`.

## Design notes

**Probe before transfer.** Each source is checked for existence and
readability first. rsync's own error for an unreadable path is a bare exit 23,
which says nothing about which source or why; a root-only log the account
cannot read is reported as `DENIED` and the run continues rather than failing.
A glob that matches a mix reports `PARTIAL` and names the unreadable paths —
the interesting tenant can easily be the one that is unreadable.

**Rotation is archived, appends are not.** See the note in
`projects/production/README.md`. This is the one piece of state the tool keeps.

**`--since` is implemented with a remote `find`, not an rsync filter**, because
rsync has no mtime filter. The file list is built on prod and fed through
`--files-from`.

**Per-tenant globs use `rsync -R`.** `companies/*/error_log` matches one file
per tenant, all named `error_log`; pulled flat they overwrite each other and
the mirror keeps whichever tenant transferred last. A `/./` marker in the
source path plus `-R` recreates the tenant directory under the destination.

**The digest has no `pipefail`.** Nearly every pipeline in it ends in `head`,
which closes the pipe and SIGPIPEs the producer; under `pipefail` that reads as
failure and every truncated section would report itself empty.

**Grouping strips more than the timestamp.** The `[client 1.2.3.4:5678]` field
and the `, referer: <url>` tail differ on every single occurrence, and Apache
escapes the newlines in a stack trace so the whole trace rides along on one
line. Without removing all three, one bug reports as four separate rows —
which is exactly how the first version split 279 identical fatals into
176 / 73 / 14 / 12.

**mawk, not gawk.** This container has mawk, so no `asorti()` — sorting is done
by `sort(1)`.