feat(profiler/behave_shell): emit cognitive.feedback_loop_engagement
BEHAVE-EXTRACTOR.md Phase A Step 7. The orthogonal axis — does the
operator's pause-after-command correlate with bytes of output they
just saw? Splits HUMAN/CLAUDE-CL (closed_loop) from LW-sim/CLAUDE-FF
(fire_and_forget); cuts ACROSS the LLM/human axis.
* _features/cognitive.py:feedback_loop_engagement(ctx) emits one
Observation in {closed_loop, fire_and_forget, unknown}.
* Pearson correlation between ctx.output_per_cmd[i] and
ctx.inter_cmd_iats[i] (paired by construction in Step 4); via
statistics.correlation with constant-series fallback to "unknown".
* r > FEEDBACK_CORRELATION_MIN (0.30) → closed_loop; otherwise
(zero, negative, or undefined) → fire_and_forget.
* First primitive that depends on output events: zero output events
in the shard or fewer than FEEDBACK_MIN_PAIRS (5) pairs → emit
"unknown" at confidence 1.0 (the absence-of-data is itself a
high-confidence answer). Zero-command session skips entirely.
Tests: no-output → unknown, few-pairs → unknown, strong positive r
→ closed_loop, constant pace → fire_and_forget/unknown,
negative r → fire_and_forget.
This commit is contained in:
@@ -13,6 +13,7 @@ from decnet_behave_core.spec.envelope import Observation
|
||||
from decnet.profiler.behave_shell._ctx import SessionContext
|
||||
from decnet.profiler.behave_shell._features.cognitive import (
|
||||
command_branch_diversity,
|
||||
feedback_loop_engagement,
|
||||
inter_command_latency_class,
|
||||
)
|
||||
from decnet.profiler.behave_shell._features.motor import (
|
||||
@@ -27,4 +28,5 @@ FEATURES: tuple[FeatureFn, ...] = (
|
||||
paste_burst_rate,
|
||||
inter_command_latency_class,
|
||||
command_branch_diversity,
|
||||
feedback_loop_engagement,
|
||||
)
|
||||
|
||||
@@ -16,6 +16,8 @@ from decnet.profiler.behave_shell._ctx import SessionContext
|
||||
from decnet.profiler.behave_shell._features._emit import make_observation
|
||||
from decnet.profiler.behave_shell._thresholds import (
|
||||
BRANCH_DIVERSITY_LINEAR_MIN,
|
||||
FEEDBACK_CORRELATION_MIN,
|
||||
FEEDBACK_MIN_PAIRS,
|
||||
INTER_CMD_DELIBERATE_MAX,
|
||||
INTER_CMD_INSTANT_MAX,
|
||||
INTER_CMD_LLM_HEAVYWEIGHT_MAX,
|
||||
@@ -100,3 +102,53 @@ def command_branch_diversity(ctx: SessionContext) -> Iterator[Observation]:
|
||||
value=value,
|
||||
confidence=0.80,
|
||||
)
|
||||
|
||||
|
||||
def feedback_loop_engagement(ctx: SessionContext) -> Iterator[Observation]:
|
||||
"""Emit ``cognitive.feedback_loop_engagement``.
|
||||
|
||||
Pearson correlation between ``output_per_cmd[i]`` (bytes the
|
||||
operator saw before the next command) and
|
||||
``inter_cmd_iats[i]`` (the pause that followed). closed_loop
|
||||
operators read more before pausing more; fire_and_forget operators
|
||||
pace independently of output. CUTS ACROSS the LLM/human axis —
|
||||
closed-loop LLMs and reading humans both score closed_loop.
|
||||
|
||||
First primitive that depends on output events: zero output events
|
||||
in the shard → emit ``unknown`` at confidence 1.0 (no honest
|
||||
correlation possible) and exit.
|
||||
"""
|
||||
pairs = list(zip(ctx.output_per_cmd, ctx.inter_cmd_iats))
|
||||
if not ctx.output_events or len(pairs) < FEEDBACK_MIN_PAIRS:
|
||||
if not ctx.commands:
|
||||
return
|
||||
yield make_observation(
|
||||
ctx,
|
||||
primitive="cognitive.feedback_loop_engagement",
|
||||
value="unknown",
|
||||
confidence=1.0,
|
||||
)
|
||||
return
|
||||
xs = [float(p[0]) for p in pairs]
|
||||
ys = [float(p[1]) for p in pairs]
|
||||
try:
|
||||
r = statistics.correlation(xs, ys)
|
||||
except statistics.StatisticsError:
|
||||
# Constant series on either axis — correlation undefined.
|
||||
yield make_observation(
|
||||
ctx,
|
||||
primitive="cognitive.feedback_loop_engagement",
|
||||
value="unknown",
|
||||
confidence=1.0,
|
||||
)
|
||||
return
|
||||
if r > FEEDBACK_CORRELATION_MIN:
|
||||
value = "closed_loop"
|
||||
else:
|
||||
value = "fire_and_forget"
|
||||
yield make_observation(
|
||||
ctx,
|
||||
primitive="cognitive.feedback_loop_engagement",
|
||||
value=value,
|
||||
confidence=0.75,
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user