Skip to content

Measures

One section per measure: what it does, when to use it, the command on one host, in one client and on all clients (where to run them), how to check it from outside, how to lift it, and the caveats. Details are in the toolkit's runbook docs/emergency.md.

Every example is the preview

The examples end with --dry-run (host) or --preview (client, all clients): they print what would be written or run and change nothing. Run the same command again without it to apply it.

Placeholders in upper case: ID, HOST, ADDRESS, CIDR, PERMISSION, MODULE, CONFIG, KEY, VALUE; www.example.com stands for a site's name. Exit codes of the tool: 0 ok, 1 failure (rolled back; for exec: a command failed), 2 invalid input, nothing written or run.

shutdown

Routes, does not stop: the containers keep running, every request on websecure gets the notice page, 503 with Retry-After: 3600 and Cache-Control: no-store, for every method. The ACME challenge, /.well-known/security.txt, the redirect of web and the NetBird entrypoints keep working.

When: a vulnerable app without a patch must go offline; one site is attacked or may be compromised.

Host:

1
2
sudo lakedrops-emergency shutdown --id drupal-sa-off --reason "Drupal core security release" \
  --image '*drupal-php*' --title "Security maintenance" --dry-run

One client (ASCR_COMMAND):

1
emergency shutdown --id drupal-sa-off --reason "Drupal core security release" --image *drupal-php* --title "Security maintenance" --preview

All clients:

1
2
lda-fleet -- shutdown --id drupal-sa-off --reason "Drupal core security release" \
  --image '*drupal-php*' --title "Security maintenance" --preview
Option
--all Every name of the host; stands alone.
--host NAME,… These names.
--stack GLOB,… The routers of the compose projects that match, e.g. 'drupal_*'.
--image GLOB,… The routers of every compose project with a container whose image matches, e.g. '*drupal-php*'.
--router NAME@docker,… These routers; an unknown one is refused.
--title, --message, --link https://… The notice page. The page adds fixed texts in English and German.
--expires DATE ISO 8601 date (2026-11-20) or date-time with a time zone. A reminder only: nothing is lifted then.
--bypass CIDR,… TCP peers that still reach the sites, e.g. the office to check a fix. Useless behind a CDN.

--host, --stack, --image and --router combine. Text must not contain {{, {% or {#.

Check: curl -sI https://www.example.com/ answers HTTP/2 503 with retry-after: 3600.

Warning

A selection by --image or --stack is resolved again every 30 seconds: a stack deployed during the incident, a patched one too, is shut down within a minute. Change the selection (same type and id) or lift to release it.

Reference: shutdown, Suspected compromise.

block

A virtual patch at Traefik: the router emergency-block-<id> answers 403 (text/plain, Cache-Control: no-store) to the requests that the rule matches; every other request passes.

When: the exploit shows in the path, the query, the method or the headers (Path or body?).

Host:

1
2
sudo lakedrops-emergency block --id drupal-sa-block --reason "SA-CORE-2026-0XX probes" \
  --rule 'PathPrefix(`/example/path`) || QueryRegexp(`param`, `pattern`)' --dry-run

One client (ASCR_COMMAND):

1
emergency block --id drupal-sa-block --reason "SA-CORE-2026-0XX probes" --rule 'PathPrefix(`/example/path`) || QueryRegexp(`param`, `pattern`)' --preview

All clients:

1
2
lda-fleet -- block --id drupal-sa-block --reason "SA-CORE-2026-0XX probes" \
  --rule 'PathPrefix(`/example/path`) || QueryRegexp(`param`, `pattern`)' --preview
Option
--rule Traefik's matchers, see below.
--host NAME,… Only these names. Without it, every name of the host.

The rule allows only the matchers Path, PathPrefix, PathRegexp, Query, QueryRegexp, Header, HeaderRegexp, Method, Host and HostRegexp, combined with &&, ||, ! and parentheses, their arguments in backticks or double quotes. ClientIP is refused. A rule that Traefik rejects is rolled back. In a shell, put the rule in single quotes: backticks would run a command.

Check: curl -s -o /dev/null -w '%{http_code}\n' 'https://www.example.com/example/path' answers 403.

Test with two or three requests at most

Five blocked requests in quick succession ban the address for 4 hours (longer when it comes back) on every router of the host, the office included. Test from a fleet server on the CrowdSec allowlist, which is never banned.

Reference: block.

Path or body?

Traefik's matchers see the name, the method, the path, the query and the headers, never the body. Compare a request of the exploit (advisory, proof of concept, access log) with a legitimate one:

  • They differ in the path, query, method or a header: block.
  • They differ only in the body (e.g. a POST to a login form or a JSON API that real clients use as well): waf rule. A block of the path would block the real clients too.

Reference: Path or body?.

waf rule

A virtual patch in CrowdSec's AppSec, which sees the body: an in-band rule in CrowdSec's format. A matching request gets 403 and raises an alert. It applies below traefik_crowdsec.allow_paths too.

When: the exploit shows only in the request body.

The rule file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
name: lakedrops/emergency-drupal-sa-waf
description: "SA-CORE-2026-0XX: <what the rule matches>"
rules:
  - and:
      - zones: [URI]
        transform: [lowercase, urldecode, normalize_path]
        match: {type: regex, value: '^/example/path'}
      - zones: [RAW_BODY]
        match: {type: contains, value: example}
labels: {type: exploit, service: http, behavior: "http:exploit", confidence: 3, spoofable: 0, label: "SA-CORE-2026-0XX"}
  • Only the keys name, description, rules and labels.
  • name starts with lakedrops/emergency-, then lower-case letters, digits, ., _ and -. One name per incident id.
  • A regex must not contain a literal ": write \x22.

Host (the file on the host):

1
2
sudo lakedrops-emergency waf rule --id drupal-sa-waf --reason "SA-CORE-2026-0XX body" \
  --file /tmp/drupal-sa-waf.yaml --dry-run

One client (ASCR_COMMAND; the file must be in the client repository):

1
emergency waf-rule --id drupal-sa-waf --reason "SA-CORE-2026-0XX body" --file rules/drupal-sa-waf.yaml --preview

All clients (the file in every client repository, see waf rule on all clients):

1
2
lda-fleet --only v2/lakedrops,v2/local,v2/mobimo -- waf-rule --id drupal-sa-waf \
  --reason "SA-CORE-2026-0XX body" --file rules/drupal-sa-waf.yaml --preview

The tool converts the YAML to JSON, runs crowdsec -t, restarts CrowdSec (about 6 seconds, during which AppSec lets every request through), checks its log, and rolls back on failure (exit 1).

Check from an address outside the fleet (e.g. a phone hotspot): curl -s -o /dev/null -w '%{http_code}\n' -X POST --data 'example' 'https://www.example.com/example/path' answers 403, and cscli alerts list (see status) names the rule.

Not from a fleet server

The fleet's addresses are on the CrowdSec allowlist, and AppSec skips them: a test from a server of the fleet always passes.

Reference: waf rule, Drupal login probes (the role's own rule, same structure).

waf update

cscli hub update and cscli hub upgrade (never --force). Prints the AppSec rules that are new (new: <name> <version>) or changed (changed: <name> <old> -> <new>), then checks and restarts CrowdSec like waf rule. Prints unchanged and restarts nothing when there is nothing new. No id, nothing to lift.

When: the CrowdSec hub publishes a virtual patch (crowdsecurity/vpatch-CVE-…) for the CVE.

Host:

1
sudo lakedrops-emergency waf update --dry-run

One client (ASCR_COMMAND):

1
emergency waf-update --preview

All clients:

1
lda-fleet -- waf-update --preview

Check on the host:

1
docker compose --project-directory /opt/traefik exec crowdsec cscli appsec-rules list | grep -i vpatch

An upgraded rule that fails the check

If crowdsec -t rejects an upgraded rule, CrowdSec keeps running with the rules it loaded and is not restarted (exit 1). Do not restart it until the hub item is fixed.

Reference: waf update, Disabled AppSec rules.

exec

Runs one command in every running container that the selection matches, one container after the other, as docker exec without a shell, a TTY or stdin. It knows nothing about the apps: drush is just a command. Needs toolkit 0.38.0 (where to run).

When: the mitigation lives inside the apps: uninstall a module, revoke a permission, change a setting, until the patch is deployed. Also read-only questions to every site, such as its core version.

Host:

1
2
sudo lakedrops-emergency exec --reason "SA-CORE-2026-0XX mitigation" --image '*drupal-php*' \
  --cmd "drush role:perm:remove anonymous 'PERMISSION'" --dry-run

One client (ASCR_COMMAND):

1
emergency exec --reason "SA-CORE-2026-0XX mitigation" --image *drupal-php* --cmd "drush role:perm:remove anonymous 'PERMISSION'" --preview

All clients:

1
2
lda-fleet -- exec --reason "SA-CORE-2026-0XX mitigation" --image '*drupal-php*' \
  --cmd "drush role:perm:remove anonymous 'PERMISSION'" --preview
Option
--reason TEXT Required, one line: it goes to the journal.
--cmd "COMMAND ARGS…" Required. One line, split like a POSIX shell into the arguments of docker exec: no shell, nothing expanded. For a shell, write sh -c '…'. Must not contain {{, {% or {#.
--image GLOB,… Containers whose image matches.
--stack GLOB,… Containers of the compose projects that match.
--service GLOB,… Containers of the compose services that match.
--container GLOB,… Containers whose name matches.
--user USER Run as this user (name or uid, optionally :group); default: the container's user.
--timeout SECONDS Per container, 1 to 3600; default 300.

Selection: at least one of --image, --stack, --service and --container; there is no implicit "all". Only running containers. Selectors of different kinds intersect; the globs within one list are alternatives. This is the opposite of shutdown, whose selections add up: --stack 'drupal_*' --image '*drupal-php*' means the php containers of the Drupal stacks for exec, but every router of both for shutdown. A union would widen what exec runs in: every container of those stacks (database, web server, mailpit, …), where the command does not belong.

--image '*drupal-php*' takes the php container of every running docker4drupal stack, staging and review stacks included; narrow it with --stack if needed.

Output: per container a header, the command's output and a result line (executed NAME: exit 0, failed NAME: exit N, timed out NAME after 300 s), then a summary. One failing container does not stop the others. Exit 0 when every container exited 0 or none matched (no running container matches), 1 when one failed or timed out, 2 invalid input, nothing run.

Check: the command's own output in the job log, or a second, read-only exec, e.g. --cmd "drush config:get CONFIG KEY".

Not a measure: no --id, nothing is stored, nothing to lift, and status and the sanity issue do not show it; the journal has it. The way back is another exec with the inverse command, e.g. drush role:perm:add anonymous 'PERMISSION'.

Never put a secret into --cmd

The command goes to the host's journal, which is shipped to OpenObserve, and to the CI job log.

A timeout does not stop the command

On a timeout the tool stops waiting and reports the container as timed out, but the process inside the container may still be running.

Reference: exec.

throttle

Traefik's rateLimit and inFlightReq in emergency@file, on every router of websecure, for every name.

When: a flood of requests.

Host:

1
sudo lakedrops-emergency throttle --id flood-1 --reason "HTTP flood" --average 10 --burst 50 --inflight 200 --dry-run

One client (ASCR_COMMAND):

1
emergency throttle --id flood-1 --reason "HTTP flood" --average 10 --burst 50 --inflight 200 --preview

All clients:

1
lda-fleet -- throttle --id flood-1 --reason "HTTP flood" --average 10 --burst 50 --inflight 200 --preview
Option
--average N Requests per period and client.
--burst N Requests a client may send at once; default --average.
--period Period of --average: 1s (default), 10s, 1m, …
--inflight N Concurrent requests per name; beyond N every client of the name gets 429.

At least one of --average and --inflight is required. The key of the rate limit is the TCP peer (an IPv6 client per /64); on a host that only a CDN reaches, it is the visitor. Ten 429 in quick succession ban the address for 4 hours.

Check: status shows the limits and the key.

Reference: throttle.

deny

CrowdSec ban decisions per address and range. They apply within 60 seconds on every router with the bouncer. The host needs CrowdSec.

When: known attacking addresses or ranges.

Host:

1
sudo lakedrops-emergency deny --id attack-1 --reason "Known attackers" --ip ADDRESS --range CIDR --duration 24h --dry-run

One client (ASCR_COMMAND):

1
emergency deny --id attack-1 --reason "Known attackers" --ip ADDRESS --range CIDR --duration 24h --preview

All clients:

1
lda-fleet -- deny --id attack-1 --reason "Known attackers" --ip ADDRESS --range CIDR --duration 24h --preview
Option
--ip A,… Addresses, IPv4 or IPv6.
--range CIDR,… Networks. --ip and --range may be combined.
--duration 24h, 90m, 1h30m, at most 8760h. Without it, until lift.

Check: status lists the addresses with their end; on the host, docker compose --project-directory /opt/traefik exec crowdsec cscli decisions list.

Reference: deny, CrowdSec bouncer.

deny --nft

The addresses become members of the nftables table inet lakedrops_emergency and are dropped before Docker: at once, open connections too, no CrowdSec needed. NetBird still reaches the host.

When: known addresses that must not reach the host at all, at once.

Host:

1
sudo lakedrops-emergency deny --id attack-2 --reason "Flood from one network" --nft --range CIDR --duration 24h --dry-run

One client (ASCR_COMMAND):

1
emergency deny --id attack-2 --reason "Flood from one network" --nft --range CIDR --duration 24h --preview

All clients:

1
lda-fleet -- deny --id attack-2 --reason "Flood from one network" --nft --range CIDR --duration 24h --preview

Needs the host firewall (firewall.json of the role firewall). Refused (exit 2): overlaps with another incident's members, with traefik_trusted_proxies (CDN ranges) or within one call, /0, a range with host bits, an IPv4-mapped IPv6 address.

Check on the host: sudo nft list table inet lakedrops_emergency.

Reference: deny --nft.

challenge

CrowdSec's bot challenge (an alpha feature of CrowdSec): a GET of a selected name gets a proof-of-work page; a browser that passes gets a cookie for 12 hours. Not a CVE tool.

When: a flood from many addresses that look like visitors (a botnet), where throttle per address does not help. Never for Nextcloud names.

Host:

1
sudo lakedrops-emergency challenge --id bots-1 --reason "Bot flood" --host www.example.com --exempt-path /webhook/ --dry-run

One client (ASCR_COMMAND):

1
emergency challenge --id bots-1 --reason "Bot flood" --host www.example.com --exempt-path /webhook/ --preview

All clients:

1
lda-fleet -- challenge --id bots-1 --reason "Bot flood" --host www.example.com --exempt-path /webhook/ --preview
Option
--host NAME,… The names; required. Nextcloud names are refused.
--exempt-path P,… Path prefixes without the challenge, e.g. /webhook/ for webhooks that use GET.

Check in a real browser from outside the fleet: the challenge page once, then the site.

Warning

  • Solve it by hand in a real browser, once per app type, before using it in production; automated browsers never pass.
  • Eleven unsolved pages ban the client for 4 hours, on every name of the host. Exempt monitors with --exempt-path.
  • The page has about 320 KB: it makes a flood that fills the uplink worse.

Reference: challenge.

lift

Removes every measure of the id, its notice page, its decisions or members. The sites answer again within seconds, a denied address within a minute, a member of deny --nft at once. Succeeds and changes nothing where the id is not active. A deny, waf rule or challenge needs a running crowdsec to be lifted. exec cannot be lifted: run the inverse command.

Host:

1
sudo lakedrops-emergency lift --id ID --dry-run

One client (ASCR_COMMAND; add --limit HOST for one host):

1
emergency lift --id ID --preview

All clients:

1
lda-fleet -- lift --id ID --preview

Reference: Lifting.

status

Lists the measures (id, since, what they do, reason, expires) and the problems. It never changes anything.

Host:

1
2
sudo lakedrops-emergency status
sudo lakedrops-emergency status --json

One client (ASCR_COMMAND):

1
emergency status

All clients:

1
lda-fleet -- status

CrowdSec's alerts on the host:

1
docker compose --project-directory /opt/traefik exec crowdsec cscli alerts list

Reference: Status.