feat(webhooks): circuit breaker auto-disables misbehaving subscriptions
After DECNET_WEBHOOK_CIRCUIT_THRESHOLD (default 5) consecutive failed
deliveries, the worker calls trip_webhook_circuit(uuid, ts) which
flips enabled=False and stamps auto_disabled_at. The worker sets its
reload flag so the next dispatch epoch stops consuming events for the
tripped sub entirely — one dead receiver can't poison the shared
egress pool anymore.
Operator clears the trip via PATCH — setting enabled=True when the
sub was previously disabled clears auto_disabled_at, zeros
consecutive_failures, and clears last_error. Admin-pause → re-enable
hits the same path harmlessly.
Three observable states now distinguishable in the UI:
- Active enabled=True, auto_disabled_at=NULL
- Admin-paused enabled=False, auto_disabled_at=NULL
- Tripped enabled=False, auto_disabled_at=<ts>
UI surfaces a TRIPPED · <ts> chip on the row (red, alert-styled) and
a "N TRIPPED" count in the page header. Hover tooltip tells the
operator how to reset ("Re-enable via Edit").
record_webhook_failure now returns the new consecutive_failures count
so the worker can compare against the threshold without a second
roundtrip. trip_webhook_circuit is idempotent — re-tripping just
re-stamps auto_disabled_at.
Closes THREAT_MODEL WH-02 and DEBT-037 §1.
This commit is contained in:
@@ -29,6 +29,7 @@ interface WebhookRow {
|
||||
last_success_at: string | null;
|
||||
last_failure_at: string | null;
|
||||
last_error: string | null;
|
||||
auto_disabled_at: string | null;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
warnings: string[];
|
||||
@@ -116,6 +117,10 @@ const Webhooks: React.FC = () => {
|
||||
() => webhooks.filter((w) => w.consecutive_failures > 0).length,
|
||||
[webhooks],
|
||||
);
|
||||
const trippedCount = useMemo(
|
||||
() => webhooks.filter((w) => w.auto_disabled_at).length,
|
||||
[webhooks],
|
||||
);
|
||||
|
||||
const fetchWebhooks = async () => {
|
||||
try {
|
||||
@@ -290,6 +295,7 @@ const Webhooks: React.FC = () => {
|
||||
<h1>WEBHOOKS</h1>
|
||||
<span className="page-sub">
|
||||
{webhooks.length} CONFIGURED · {enabledCount} ENABLED
|
||||
{trippedCount > 0 && ` · ${trippedCount} TRIPPED`}
|
||||
{failCount > 0 && ` · ${failCount} FAILING`}
|
||||
{insecureCount > 0 && ` · ${insecureCount} INSECURE`}
|
||||
</span>
|
||||
@@ -424,6 +430,14 @@ const Webhooks: React.FC = () => {
|
||||
<span className={`wh-chip ${w.enabled ? '' : 'status-disabled'}`}>
|
||||
{w.enabled ? 'ENABLED' : 'DISABLED'}
|
||||
</span>
|
||||
{w.auto_disabled_at && (
|
||||
<span
|
||||
className="wh-chip status-fail"
|
||||
title={`Circuit tripped at ${formatDate(w.auto_disabled_at)}. Re-enable via Edit to reset.`}
|
||||
>
|
||||
TRIPPED · {formatDate(w.auto_disabled_at)}
|
||||
</span>
|
||||
)}
|
||||
{w.consecutive_failures > 0 && (
|
||||
<span className="wh-chip status-fail" title={w.last_error || ''}>
|
||||
FAIL · {w.consecutive_failures}
|
||||
|
||||
Reference in New Issue
Block a user