Measures¶
One section per measure: what it does, when to use it, the command on one host, in one client and on all clients (where to run them), how to check it from outside, how to lift it, and the caveats. Details are in the toolkit's runbook docs/emergency.md.
Every example is the preview
The examples end with --dry-run (host) or --preview (client, all
clients): they print what would be written or run and change nothing.
Run the same command again without it to apply it.
Placeholders in upper case: ID, HOST, ADDRESS, CIDR, PERMISSION,
MODULE, CONFIG, KEY, VALUE; www.example.com stands for a site's
name. Exit codes of the tool: 0 ok, 1 failure (rolled back; for exec:
a command failed), 2 invalid input, nothing written or run.
shutdown¶
Routes, does not stop: the containers keep running, every request on
websecure gets the notice page, 503 with Retry-After: 3600 and
Cache-Control: no-store, for every method. The ACME challenge,
/.well-known/security.txt, the redirect of web and the NetBird
entrypoints keep working.
When: a vulnerable app without a patch must go offline; one site is attacked or may be compromised.
Host:
1 2 | |
One client (ASCR_COMMAND):
1 | |
All clients:
1 2 | |
| Option | |
|---|---|
--all |
Every name of the host; stands alone. |
--host NAME,… |
These names. |
--stack GLOB,… |
The routers of the compose projects that match, e.g. 'drupal_*'. |
--image GLOB,… |
The routers of every compose project with a container whose image matches, e.g. '*drupal-php*'. |
--router NAME@docker,… |
These routers; an unknown one is refused. |
--title, --message, --link https://… |
The notice page. The page adds fixed texts in English and German. |
--expires DATE |
ISO 8601 date (2026-11-20) or date-time with a time zone. A reminder only: nothing is lifted then. |
--bypass CIDR,… |
TCP peers that still reach the sites, e.g. the office to check a fix. Useless behind a CDN. |
--host, --stack, --image and --router combine. Text must not
contain {{, {% or {#.
Check: curl -sI https://www.example.com/ answers HTTP/2 503 with
retry-after: 3600.
Warning
A selection by --image or --stack is resolved again every 30
seconds: a stack deployed during the incident, a patched one too, is
shut down within a minute. Change the selection (same type and id) or
lift to release it.
Reference: shutdown,
Suspected compromise.
block¶
A virtual patch at Traefik: the router emergency-block-<id> answers 403
(text/plain, Cache-Control: no-store) to the requests that the rule
matches; every other request passes.
When: the exploit shows in the path, the query, the method or the headers (Path or body?).
Host:
1 2 | |
One client (ASCR_COMMAND):
1 | |
All clients:
1 2 | |
| Option | |
|---|---|
--rule |
Traefik's matchers, see below. |
--host NAME,… |
Only these names. Without it, every name of the host. |
The rule allows only the matchers Path, PathPrefix, PathRegexp,
Query, QueryRegexp, Header, HeaderRegexp, Method, Host and
HostRegexp, combined with &&, ||, ! and parentheses, their arguments
in backticks or double quotes. ClientIP is refused. A rule that Traefik
rejects is rolled back. In a shell, put the rule in single quotes: backticks
would run a command.
Check: curl -s -o /dev/null -w '%{http_code}\n' 'https://www.example.com/example/path'
answers 403.
Test with two or three requests at most
Five blocked requests in quick succession ban the address for 4 hours (longer when it comes back) on every router of the host, the office included. Test from a fleet server on the CrowdSec allowlist, which is never banned.
Reference: block.
Path or body?¶
Traefik's matchers see the name, the method, the path, the query and the headers, never the body. Compare a request of the exploit (advisory, proof of concept, access log) with a legitimate one:
- They differ in the path, query, method or a header:
block. - They differ only in the body (e.g. a POST to a login form or a JSON API
that real clients use as well):
waf rule. Ablockof the path would block the real clients too.
Reference: Path or body?.
waf rule¶
A virtual patch in CrowdSec's AppSec, which sees the body: an in-band rule
in CrowdSec's format. A matching request gets 403 and raises an alert. It
applies below traefik_crowdsec.allow_paths too.
When: the exploit shows only in the request body.
The rule file:
1 2 3 4 5 6 7 8 9 10 | |
- Only the keys
name,description,rulesandlabels. namestarts withlakedrops/emergency-, then lower-case letters, digits,.,_and-. One name per incident id.- A regex must not contain a literal
": write\x22.
Host (the file on the host):
1 2 | |
One client (ASCR_COMMAND; the file must be in the client repository):
1 | |
All clients (the file in every client repository, see
waf rule on all clients):
1 2 | |
The tool converts the YAML to JSON, runs crowdsec -t, restarts CrowdSec
(about 6 seconds, during which AppSec lets every request through), checks
its log, and rolls back on failure (exit 1).
Check from an address outside the fleet (e.g. a phone hotspot):
curl -s -o /dev/null -w '%{http_code}\n' -X POST --data 'example' 'https://www.example.com/example/path'
answers 403, and cscli alerts list (see
status) names the rule.
Not from a fleet server
The fleet's addresses are on the CrowdSec allowlist, and AppSec skips them: a test from a server of the fleet always passes.
Reference: waf rule,
Drupal login probes
(the role's own rule, same structure).
waf update¶
cscli hub update and cscli hub upgrade (never --force). Prints the
AppSec rules that are new (new: <name> <version>) or changed
(changed: <name> <old> -> <new>), then checks and restarts CrowdSec like
waf rule. Prints unchanged and restarts nothing when there is nothing
new. No id, nothing to lift.
When: the CrowdSec hub publishes a virtual patch
(crowdsecurity/vpatch-CVE-…) for the CVE.
Host:
1 | |
One client (ASCR_COMMAND):
1 | |
All clients:
1 | |
Check on the host:
1 | |
An upgraded rule that fails the check
If crowdsec -t rejects an upgraded rule, CrowdSec keeps running with
the rules it loaded and is not restarted (exit 1). Do not restart it
until the hub item is fixed.
Reference: waf update,
Disabled AppSec rules.
exec¶
Runs one command in every running container that the selection matches,
one container after the other, as docker exec without a shell, a TTY or
stdin. It knows nothing about the apps: drush is just a command. Needs
toolkit 0.38.0 (where to run).
When: the mitigation lives inside the apps: uninstall a module, revoke a permission, change a setting, until the patch is deployed. Also read-only questions to every site, such as its core version.
Host:
1 2 | |
One client (ASCR_COMMAND):
1 | |
All clients:
1 2 | |
| Option | |
|---|---|
--reason TEXT |
Required, one line: it goes to the journal. |
--cmd "COMMAND ARGS…" |
Required. One line, split like a POSIX shell into the arguments of docker exec: no shell, nothing expanded. For a shell, write sh -c '…'. Must not contain {{, {% or {#. |
--image GLOB,… |
Containers whose image matches. |
--stack GLOB,… |
Containers of the compose projects that match. |
--service GLOB,… |
Containers of the compose services that match. |
--container GLOB,… |
Containers whose name matches. |
--user USER |
Run as this user (name or uid, optionally :group); default: the container's user. |
--timeout SECONDS |
Per container, 1 to 3600; default 300. |
Selection: at least one of --image, --stack, --service and
--container; there is no implicit "all". Only running containers.
Selectors of different kinds intersect; the globs within one list are
alternatives. This is the opposite of shutdown, whose
selections add up: --stack 'drupal_*' --image '*drupal-php*' means the
php containers of the Drupal stacks for exec, but every router of both
for shutdown. A union would widen what exec runs in: every container of
those stacks (database, web server, mailpit, …), where the command does not
belong.
--image '*drupal-php*' takes the php container of every running
docker4drupal stack, staging and review stacks included; narrow it with
--stack if needed.
Output: per container a header, the command's output and a result line
(executed NAME: exit 0, failed NAME: exit N, timed out NAME after 300 s),
then a summary. One failing container does not stop the others. Exit 0
when every container exited 0 or none matched (no running container
matches), 1 when one failed or timed out, 2 invalid input, nothing run.
Check: the command's own output in the job log, or a second, read-only
exec, e.g. --cmd "drush config:get CONFIG KEY".
Not a measure: no --id, nothing is stored, nothing to lift, and
status and the sanity issue do not show it; the journal has it. The way
back is another exec with the inverse command, e.g.
drush role:perm:add anonymous 'PERMISSION'.
Never put a secret into --cmd
The command goes to the host's journal, which is shipped to OpenObserve, and to the CI job log.
A timeout does not stop the command
On a timeout the tool stops waiting and reports the container as timed out, but the process inside the container may still be running.
Reference: exec.
throttle¶
Traefik's rateLimit and inFlightReq in emergency@file, on every
router of websecure, for every name.
When: a flood of requests.
Host:
1 | |
One client (ASCR_COMMAND):
1 | |
All clients:
1 | |
| Option | |
|---|---|
--average N |
Requests per period and client. |
--burst N |
Requests a client may send at once; default --average. |
--period |
Period of --average: 1s (default), 10s, 1m, … |
--inflight N |
Concurrent requests per name; beyond N every client of the name gets 429. |
At least one of --average and --inflight is required. The key of the
rate limit is the TCP peer (an IPv6 client per /64); on a host that only a
CDN reaches, it is the visitor. Ten 429 in quick succession ban the
address for 4 hours.
Check: status shows the limits and the key.
Reference: throttle.
deny¶
CrowdSec ban decisions per address and range. They apply within 60 seconds on every router with the bouncer. The host needs CrowdSec.
When: known attacking addresses or ranges.
Host:
1 | |
One client (ASCR_COMMAND):
1 | |
All clients:
1 | |
| Option | |
|---|---|
--ip A,… |
Addresses, IPv4 or IPv6. |
--range CIDR,… |
Networks. --ip and --range may be combined. |
--duration |
24h, 90m, 1h30m, at most 8760h. Without it, until lift. |
Check: status lists the addresses with their end; on the host,
docker compose --project-directory /opt/traefik exec crowdsec cscli decisions list.
Reference: deny,
CrowdSec bouncer.
deny --nft¶
The addresses become members of the nftables table
inet lakedrops_emergency and are dropped before Docker: at once, open
connections too, no CrowdSec needed. NetBird still reaches the host.
When: known addresses that must not reach the host at all, at once.
Host:
1 | |
One client (ASCR_COMMAND):
1 | |
All clients:
1 | |
Needs the host firewall (firewall.json of the role firewall). Refused
(exit 2): overlaps with another incident's members, with
traefik_trusted_proxies (CDN ranges) or within one call, /0, a range
with host bits, an IPv4-mapped IPv6 address.
Check on the host: sudo nft list table inet lakedrops_emergency.
Reference: deny --nft.
challenge¶
CrowdSec's bot challenge (an alpha feature of CrowdSec): a GET of a selected name gets a proof-of-work page; a browser that passes gets a cookie for 12 hours. Not a CVE tool.
When: a flood from many addresses that look like visitors (a botnet),
where throttle per address does not help. Never for Nextcloud names.
Host:
1 | |
One client (ASCR_COMMAND):
1 | |
All clients:
1 | |
| Option | |
|---|---|
--host NAME,… |
The names; required. Nextcloud names are refused. |
--exempt-path P,… |
Path prefixes without the challenge, e.g. /webhook/ for webhooks that use GET. |
Check in a real browser from outside the fleet: the challenge page once, then the site.
Warning
- Solve it by hand in a real browser, once per app type, before using it in production; automated browsers never pass.
- Eleven unsolved pages ban the client for 4 hours, on every name of the
host. Exempt monitors with
--exempt-path. - The page has about 320 KB: it makes a flood that fills the uplink worse.
Reference: challenge.
lift¶
Removes every measure of the id, its notice page, its decisions or
members. The sites answer again within seconds, a denied address within a
minute, a member of deny --nft at once. Succeeds and changes nothing where
the id is not active. A deny, waf rule or challenge needs a running
crowdsec to be lifted. exec cannot be lifted: run the inverse
command.
Host:
1 | |
One client (ASCR_COMMAND; add --limit HOST for one host):
1 | |
All clients:
1 | |
Reference: Lifting.
status¶
Lists the measures (id, since, what they do, reason, expires) and the
problems. It never changes anything.
Host:
1 2 | |
One client (ASCR_COMMAND):
1 | |
All clients:
1 | |
CrowdSec's alerts on the host:
1 | |
Reference: Status.