Skip to content

Where to run a command

Scope Where Command Preview
One host SSH on the host sudo lakedrops-emergency ACTION … --dry-run
One client Client pipeline, ChatOps or ascr emergency ACTION … --preview
All clients Workstation with Docker lda-fleet -- ACTION … --preview

On the host, waf rule and waf update are two words; for ascr and lda-fleet they are waf-rule and waf-update. Everything else is the same: the actions shutdown, block, throttle, deny, waf-rule, waf-update, challenge, exec, lift, status, with the tool's options (exec needs toolkit 0.38.0). Lists are comma-separated.

One host

The break-glass path: it works without GitLab and the runners.

  • SSH to the host over NetBird (SSH is NetBird-only on the hosts), then:

    1
    2
    3
    sudo lakedrops-emergency status
    sudo lakedrops-emergency shutdown --id drupal-sa-off --reason "Drupal core security release" \
      --image '*drupal-php*' --dry-run
    
  • A host whose role traefik predates the tool has no /usr/local/sbin/lakedrops-emergency: ascr traefik --limit HOST installs it (the first run restarts Traefik and security-txt once).

Reference: One host.

One client

ascr emergency ACTION … runs the tool on every host of the group traefik of the client. Three ways to start it:

Way How
Pipeline Client project > Build > Pipelines > Run pipeline on main, variable ASCR_COMMAND = emergency …
ChatOps run ascr emergency …
Workstation ascr emergency …
1
ASCR_COMMAND = emergency shutdown --id drupal-sa-off --reason "Drupal core security release" --image *drupal-php* --title "Security maintenance" --preview
  • ASCR_COMMAND is split like a POSIX shell and nothing is expanded: quote text with spaces; *drupal-php* needs no quotes. A rule with backticks still needs quotes for its spaces.
  • --limit HOST narrows it to one host.
  • --preview is the tool's dry run on each host. ascr's own --dry-run only prints the ansible command.
  • Each host prints the tool's output in the job log. A host without the tool fails with a message; the others go on.
  • waf-rule --file is a path relative to where ascr runs: from a pipeline, the file must be in the client repository.

Reference: One client.

All clients

lda-fleet starts the same ascr emergency command in one pipeline per client and reports per client and host.

Prerequisites: a workstation with Docker; your personal access token with scope api, role Developer or higher in the client projects, allowed to run pipelines on their default branch.

1
2
3
4
read -rs GITLAB_TOKEN && export GITLAB_TOKEN
docker run --rm -it --env GITLAB_TOKEN \
  registry.lakedrops.com/ansible/playbooks/toolkit/ee:0.38.0 \
  lda-fleet -- status

Use the image of the toolkit release the clients run (_commit in each client's .copier-answers.yml): 0.38.0 from 2026-11-18. To run the lda-fleet … commands of these pages as written, define a shell function once:

1
2
3
4
lda-fleet() {
  docker run --rm -it --env GITLAB_TOKEN \
    registry.lakedrops.com/ansible/playbooks/toolkit/ee:0.38.0 lda-fleet "$@"
}

In a toolkit checkout, uv run lda-fleet … does the same.

Usage: lda-fleet [--only PATH,…] [--exclude PATH,…] [--yes] [--dry-run] [--no-wait] [--gitlab-url URL] -- ARGUMENTS

Option
--only PATH,…, --exclude PATH,… Narrow the clients, e.g. --only v2/local,v2/mobimo. Today the clients are v2/lakedrops, v2/local, v2/mobimo, v2/speicher.
--dry-run List the clients and the ASCR_COMMAND, start nothing. Not the tool's preview: that is --preview after --.
--yes Do not ask for confirmation.
--no-wait Start the pipelines, print their URLs and stop.
-- ARGUMENTS The arguments of ascr emergency.
  • Exit status: 0 every job succeeded and no host failed or was unreachable; 1 a job or host failed, or the confirmation was declined; 2 usage or API error before any pipeline started.
  • About 15 seconds from the start to the end of a one-host job.
  • From a phone: SSH to the workstation and run it inside tmux. Ctrl-C or a dropped connection stops only the waiting; the pipelines go on.

Reference: All clients: lda-fleet, cli/README.md.

Toolkit 0.38.0 for exec

The action exec comes with toolkit 0.38.0, released on the evening of 2026-11-17 (Europe/Berlin). It needs that release in two places:

Where How it gets there
The client image (ascr) Renovate proposes the release to the client repositories at 22:17; the copier merge requests merge themselves once their pipelines pass, which builds the client image.
The hosts (lakedrops-emergency) The clients' scheduled runs at 01:00 install it with the role traefik. Sooner, once the client image has it: ascr traefik --limit HOST.

Before that, exec is refused: by the client's ascr as an unknown action, or by an older tool on the host (exit 2, nothing run). A client whose merge request has not merged by 01:00 keeps the older image, and its scheduled site stops at the toolkit release guard: merge it, then run ascr traefik.

Reference: Releasing.

Not reached

None of the above reaches these hosts; act on them by hand:

  • hosts of a client without the client template, today tineon (thxr1, thdbm1, thdbo1, thwv1);
  • hosts outside the group traefik.

Reference: Hosts without the tool.

Check: status

Scope Command
One host sudo lakedrops-emergency status (--json for JSON)
One client ASCR_COMMAND = emergency status
All clients lda-fleet -- status

status lists the measures and the problems and never changes anything. While a measure is active, ascr sanity check keeps the issue [<host>] Emergency measures open.

Lifting

1
2
3
sudo lakedrops-emergency lift --id ID   # one host
ascr emergency lift --id ID             # one client (ASCR_COMMAND = emergency lift --id ID)
lda-fleet -- lift --id ID               # all clients

lift succeeds and changes nothing where the id is not active, so lifting on all clients is safe. See lift.

Logs

Every change goes to the host's journal under the identifier lakedrops-emergency, with who started it (the CI job's URL or the user), and is shipped to OpenObserve:

1
journalctl -t lakedrops-emergency