Where to run a command¶
| Scope | Where | Command | Preview |
|---|---|---|---|
| One host | SSH on the host | sudo lakedrops-emergency ACTION … |
--dry-run |
| One client | Client pipeline, ChatOps or ascr |
emergency ACTION … |
--preview |
| All clients | Workstation with Docker | lda-fleet -- ACTION … |
--preview |
On the host, waf rule and waf update are two words; for ascr and
lda-fleet they are waf-rule and waf-update. Everything else is the
same: the actions shutdown, block, throttle, deny, waf-rule,
waf-update, challenge, exec, lift, status, with the tool's
options (exec needs toolkit 0.38.0). Lists
are comma-separated.
One host¶
The break-glass path: it works without GitLab and the runners.
-
SSH to the host over NetBird (SSH is NetBird-only on the hosts), then:
1 2 3
sudo lakedrops-emergency status sudo lakedrops-emergency shutdown --id drupal-sa-off --reason "Drupal core security release" \ --image '*drupal-php*' --dry-run -
A host whose role
traefikpredates the tool has no/usr/local/sbin/lakedrops-emergency:ascr traefik --limit HOSTinstalls it (the first run restarts Traefik andsecurity-txtonce).
Reference: One host.
One client¶
ascr emergency ACTION … runs the tool on every host of the group
traefik of the client. Three ways to start it:
| Way | How |
|---|---|
| Pipeline | Client project > Build > Pipelines > Run pipeline on main, variable ASCR_COMMAND = emergency … |
| ChatOps | run ascr emergency … |
| Workstation | ascr emergency … |
1 | |
ASCR_COMMANDis split like a POSIX shell and nothing is expanded: quote text with spaces;*drupal-php*needs no quotes. A rule with backticks still needs quotes for its spaces.--limit HOSTnarrows it to one host.--previewis the tool's dry run on each host.ascr's own--dry-runonly prints the ansible command.- Each host prints the tool's output in the job log. A host without the tool fails with a message; the others go on.
waf-rule --fileis a path relative to whereascrruns: from a pipeline, the file must be in the client repository.
Reference: One client.
All clients¶
lda-fleet starts the same ascr emergency command in one pipeline per
client and reports per client and host.
Prerequisites: a workstation with Docker; your personal access token
with scope api, role Developer or higher in the client projects, allowed
to run pipelines on their default branch.
1 2 3 4 | |
Use the image of the toolkit release the clients run (_commit in each
client's .copier-answers.yml): 0.38.0 from 2026-11-18. To run the lda-fleet …
commands of these pages as written, define a shell function once:
1 2 3 4 | |
In a toolkit checkout, uv run lda-fleet … does the same.
Usage: lda-fleet [--only PATH,…] [--exclude PATH,…] [--yes] [--dry-run] [--no-wait] [--gitlab-url URL] -- ARGUMENTS
| Option | |
|---|---|
--only PATH,…, --exclude PATH,… |
Narrow the clients, e.g. --only v2/local,v2/mobimo. Today the clients are v2/lakedrops, v2/local, v2/mobimo, v2/speicher. |
--dry-run |
List the clients and the ASCR_COMMAND, start nothing. Not the tool's preview: that is --preview after --. |
--yes |
Do not ask for confirmation. |
--no-wait |
Start the pipelines, print their URLs and stop. |
-- ARGUMENTS |
The arguments of ascr emergency. |
- Exit status:
0every job succeeded and no host failed or was unreachable;1a job or host failed, or the confirmation was declined;2usage or API error before any pipeline started. - About 15 seconds from the start to the end of a one-host job.
- From a phone: SSH to the workstation and run it inside
tmux. Ctrl-C or a dropped connection stops only the waiting; the pipelines go on.
Reference: All clients: lda-fleet,
cli/README.md.
Toolkit 0.38.0 for exec¶
The action exec comes with toolkit 0.38.0, released on the evening of
2026-11-17 (Europe/Berlin). It needs that release in two places:
| Where | How it gets there |
|---|---|
The client image (ascr) |
Renovate proposes the release to the client repositories at 22:17; the copier merge requests merge themselves once their pipelines pass, which builds the client image. |
The hosts (lakedrops-emergency) |
The clients' scheduled runs at 01:00 install it with the role traefik. Sooner, once the client image has it: ascr traefik --limit HOST. |
Before that, exec is refused: by the client's ascr as an unknown
action, or by an older tool on the host (exit 2, nothing run). A client
whose merge request has not merged by 01:00 keeps the older image, and its
scheduled site stops at the toolkit release guard: merge it, then run
ascr traefik.
Reference: Releasing.
Not reached¶
None of the above reaches these hosts; act on them by hand:
- hosts of a client without the client template, today
tineon(thxr1, thdbm1, thdbo1, thwv1); - hosts outside the group
traefik.
Reference: Hosts without the tool.
Check: status¶
| Scope | Command |
|---|---|
| One host | sudo lakedrops-emergency status (--json for JSON) |
| One client | ASCR_COMMAND = emergency status |
| All clients | lda-fleet -- status |
status lists the measures and the problems and never changes anything.
While a measure is active, ascr sanity check keeps the issue
[<host>] Emergency measures open.
Lifting¶
1 2 3 | |
lift succeeds and changes nothing where the id is not active, so lifting
on all clients is safe. See lift.
Logs¶
Every change goes to the host's journal under the identifier
lakedrops-emergency, with who started it (the CI job's URL or the user),
and is shipped to OpenObserve:
1 | |