Show original
Enjoyed this article?
Support Lily

AI translation
― Visualizing Silent Failures
Enjoyed this article?
Support Lily
前回の autopilot 記事 introduced a configuration for running Claude unattended from launchd. This time, we're discussing what happens behind the scenes ―― the problem of unattended jobs silently stopping without anyone noticing, and how to handle it.
The real danger of unattended automation isn't bugs, but silent failures. Even if a launchd job crashes with exit 78, or a hooks script loses execute permissions, or agentmemory keeps a port open in a "half-alive state" with no worker present, the surface appears unchanged. There's a real example in the comments of `automation-health.sh` where this state persisted for 6 days without anyone noticing.
When you build an autonomous loop, the number of things to monitor grows. In my setup alone:
I have no desire to manually verify that all of these are working correctly. But running `launchctl list` individually doesn't give you the "big picture."
What's needed is a single command that comprehensively checks all automation and reports back with 3 values: GREEN / WARN / FAIL.
The core of `~/.claude/scripts/automation-health.sh` is simple.
fail=0; warn=0
ok() { printf " ${GRN}✓${RST} %s\n" "$1"; }
wn() { printf " ${YEL}⚠${RST} %s\n" "$1"; warn=$((warn+1)); }
ng() { printf " ${RED}✗${RST} %s\n" "$1"; fail=$((fail+1)); }Each check item is evaluated as one of `ok` / `wn` / `ng`, and at the end the judgment is made like this:
if [ "$fail" -gt 0 ]; then
printf " ${RED}✗ FAIL${RST} red=%d warn=%d\n\n" "$fail" "$warn"; exit 1
elif [ "$warn" -gt 0 ]; then
printf " ${YEL}⚠ WARN${RST} warn=%d (致命的問題なし)\n\n" "$warn"; exit 0
else
printf " ${GRN}✓ ALL GREEN${RST} 全自動化が正常稼働\n\n"; exit 0
fiBy separating exit codes, you can chain subsequent processing from `daily-brief.sh` or CI using `||`.
The script is divided into 9 sections. Here are the main judgments extracted from the actual code.
for job in com.shun.skill-harvest com.shun.skill-curate; do
line=$(launchctl list 2>/dev/null | grep -E "\b${job}\b")
if [ -z "$line" ]; then
ng "$job: 未ロード (launchctl load し直しが必要)"
else
exitc=$(echo "$line" | awk '{print $2}')
if [ "$exitc" = "0" ] || [ "$exitc" = "-" ]; then
ok "$job: ロード済 / last exit=$exitc"
else
ng "$job: last exit=$exitc (前回失敗)"
fi
fi
doneThe second column of `launchctl list` is the last exit code. `-` means "not yet started (normal)", 0 means success, anything else means failure.
hooks=(pre_git_guard.sh pre_secrets_check.sh pre_env_guard.sh \
post_audit_log.sh post_format.sh post_tsc_check.sh \
stop_notify.sh user_prompt_submit.sh)
for h in "${hooks[@]}"; do
f="$CLAUDE/hooks/$h"
if [ ! -f "$f" ]; then ng "$h: 不在"
elif [ ! -x "$f" ]; then wn "$h: 実行権限なし (chmod +x 推奨)"
else ok "$h"
fi
doneEven if you write a hook in settings.json, if the implementation script isn't `chmod +x`, it silently skips. We detect this with WARN.
hlog="$CLAUDE/skills/auto/.harvest.log"
a=$(age_h "$hlog") # (now - mtime) / 3600 を返すヘルパ
if [ "$a" -ge 0 ] && [ "$a" -le 48 ]; then
ok "最終稼働 ${a}h前: ${last##*] }"
else
wn "最終稼働 ${a}h前 (>48h: cron停止の疑い): ${last##*] }"
fi
nskills=$(find "$CLAUDE/skills/auto" -maxdepth 2 -name SKILL.md | wc -l | tr -d ' ')
ok "生成済 auto-skill: ${nskills} 個"If the log's mtime is 48+ hours old, it's WARN. "Log exists" and "recently ran" are different things, so we check mtime.
cnt=$(find "$CONV" -name '*.md' ! -name 'INDEX.md' | wc -l | tr -d ' ')
ok "会話ログ ${cnt} 件 / 最新 ${la}h前"
ia=$(age_h "$idx")
if [ "$ia" -le 24 ]; then ok "INDEX.md 鮮度 ${ia}h前"
else wn "INDEX.md が ${ia}h前 (extract が更新していない可能性)"; fiif [ -f "$hot" ] && grep -q "recent:start auto-updated" "$hot"; then
ha=$(age_h "$hot"); ok "hot.md 自動更新マーカー有 / ${ha}h前"
else wn "hot.md の自動更新マーカーが見つからない"; fi
# index.md カバレッジ: 実ファイル数 vs 記載ページ数
real=$(find "$VAULT" -name '*.md' -not -path '*/.*' | wc -l | tr -d ' ')
stated=$(grep -oE '総ページ数:[0-9]+' "$vidx" | grep -oE '[0-9]+' | head -1)
if [ "$stated" = "$real" ]; then ok "index.md カバレッジ一致 (${real}p)"
else wn "index.md 記載 ${stated:-?}p ≠ 実 ${real}p"; fifor f in now.md recent.md archive.md; do
[ -f "$REMEMBER/$f" ] && ok "$f 存在" || wn "$f 不在 (必須層)"
done
dup=$(grep -vE '^\s*$|^##|^#' "$nowf" | sort | uniq -c | sort -rn | head -1 | awk '{print $1}')
if [ "${dup:-0}" -ge 3 ]; then
wn "now.md に同一要約が ${dup} 回 (consolidate 遅延の兆候)"
else
ok "now.md 重複なし (consolidate 健全)"
fiIf `now.md` consolidate lags, the same summary gets appended repeatedly. We early-detect 3+ duplicates with WARN.
cdir_mb=$(( ${cdir_total_mb:-0} - ${disabled_mb:-0} ))
if [ "${cdir_mb:-0}" -lt 2000 ]; then ok "~/.claude 実効 ${cdir_mb}MB"
elif [ "${cdir_mb:-0}" -lt 5000 ]; then wn "~/.claude 実効 ${cdir_mb}MB (>2GB: 大きめ)"
else ng "~/.claude 実効 ${cdir_mb}MB (>5GB: 異常肥大)"; fi
orphan=$(find "$CLAUDE" -maxdepth 1 -name 'security_warnings_state_*.json' -mtime +7 | wc -l | tr -d ' ')
if [ "${orphan:-0}" -ge 5 ]; then
wn "security_warnings_state_*.json が ${orphan} 個 (>7日前: backups/ へ退避推奨)"
fiReversible cache (`.disabled-cache`) is excluded from effective capacity when judging.
declare -a CRON_JOBS=(
"weekly cleanup-misc:$CLAUDE/logs/cleanup-misc.log:192"
"weekly env-audit:$CLAUDE/logs/env-audit-latest.md:192"
"monthly plugin-purge:$CLAUDE/logs/plugin-purge.log:744"
"weekly plugin-auto-disable:$CLAUDE/logs/plugin-auto-disable.log:192"
"weekly dotfiles-snapshot:$CLAUDE/logs/dotfiles-snapshot.log:192"
"weekly agents-index:$CLAUDE/logs/agents-index.log:192"
"daily plugin-usage:$CLAUDE/scripts/plugin-audit-latest.md:48"
)
for entry in "${CRON_JOBS[@]}"; do
IFS=":" read -r name path budget <<< "$entry"
a=$(age_h "$path")
if [ "$a" -le "$budget" ]; then ok "$name: ${a}h前 (budget ${budget}h)"
else wn "$name: ${a}h前 (budget ${budget}h 超過 — launchd 配送失敗の可能性)"; fi
doneFor weekly jobs, 192h (8 days) is the budget; for monthly, 744h (31 days). Exceeding that triggers WARN.
Only the agentmemory server check has two-stage judgment.
am_line=$(launchctl list 2>/dev/null | awk '$3=="com.shun.agentmemory"{print $1" "$2}')
am_pid=${am_line%% *}; am_exit=${am_line##* }
if [ -z "$am_line" ]; then
wn "com.shun.agentmemory が launchd に未ロード"
elif [ "$am_pid" = "-" ] && [ "$am_exit" != "0" ]; then
ng "com.shun.agentmemory 停止中 (last exit=$am_exit) — クラッシュループの可能性"
else
am_code=$(curl -s -o /dev/null -w '%{http_code}' -m 3 \
http://localhost:3111/agentmemory/health 2>/dev/null || echo 000)
if [ "$am_code" = "200" ]; then ok "稼働中 (pid=$am_pid / health 200)"
elif [ "$am_code" = "000" ]; then ng "プロセスは居るが port 3111 無応答"
else ng "port 3111 は開くが /agentmemory/health=$am_code — worker 不在の半生状態"; fi
fiThe reason is in the comments:
A launchd exit 78 crash loop went unnoticed for 6 days, and the "half-alive state" where the port opens but the worker is absent and all APIs return 404 is invisible to liveness monitoring.
Checking only launchd status can't detect the case where "the process exists but isn't actually working." The reason we confirm HTTP 200 with `curl` is here:
If you don't check both launchd PID status and HTTP health endpoints, you'll miss the "appears to be running but all APIs return 404" half-alive state. If you're running MCP servers or API servers under launchd, always add the extra step of hitting the `/health` endpoint.
An often-overlooked item is the 9th section: double-execution check.
cron_sh=$(crontab -l 2>/dev/null | grep -vE '^[[:space:]]*#' | \
grep -oE '/[^ ]+\.sh' | xargs -n1 basename | sort -u)
launchd_sh=$(grep -hoE '/[^<> ]+\.sh' \
~/Library/LaunchAgents/com.shun.*.plist | xargs -n1 basename | sort -u)
dup=$(comm -12 <(printf '%s\n' "$cron_sh") <(printf '%s\n' "$launchd_sh") | \
grep -vE '^[[:space:]]*$')
if [ -n "$dup" ]; then
ng "cron と launchd の両方に登録され二重実行されるスクリプト: $(printf '%s' "$dup" | tr '\n' ' ')"
fiThe comment says "2026-06-01: During cron→launchd migration, forgot to remove cron, resulting in both being registered = double execution every cycle." This is the pattern of forgetting to remove the old crontab after migration.
`daily-brief.sh` launches from launchd at 8:00 AM every morning, with just one line embedded in the environment health at the top.
echo "## 🏥 環境ヘルス"
run_to 60 ~/.claude/scripts/automation-health.sh 2>&1 | tail -3 | head -1`tail -3 | head -1` extracts only the final summary line (`✓ ALL GREEN` / `⚠ WARN` / `✗ FAIL`).
Furthermore, on the `daily-brief.sh` side, the `## 🚨 要対応` section appears only when live connectivity probes detect RED.
RED_FLAGS=""
flag_red() { RED_FLAGS="${RED_FLAGS}\n- ❌ $1"; }
if [ -n "$RED_FLAGS" ]; then
echo "## 🚨 要対応(ライブ疎通でRED)"
printf '%b\n' "$RED_FLAGS"
fiWhen there are no problems, this section doesn't exist. It's designed as a "section that appears only during failures." The brief is written to `~/Desktop/Daily Brief/today-brief-YYYYMMDD.md`, so when you open the desktop, RED jumps into view.
Using the exit 1 from `automation-health.sh`, you can add similar patterns in a launchd wrapper.
bash ~/.claude/scripts/automation-health.sh || touch ~/Desktop/AUTOMATION_FAILED.md
bash ~/.claude/scripts/automation-health.sh && rm -f ~/Desktop/AUTOMATION_FAILED.mdThe file exists only during failures and disappears when fixed ―― the desktop tells you the state without needing to check the dashboard.
Next time, I plan to write about integrating this health check with improvement logs generated by autopilot, and automatically generating weekly reports ―― the mechanism for weekly report auto-generation.
Lily (@bokuwalily) ― Individual developer. Building automation infrastructure with Claude Code while mass-producing iOS apps and web services
◼︎ Apps I've made are summarized at **ポートフォリオ**📱
◼︎ New releases and development insights shared on X **@bokuwalily**🐦
◼︎ OSS: **github.com/bokuwalily**🐙
Your ❤️ and shares are my motivation!
前回の autopilot 記事で launchd から Claude を無人で走らせる構成を紹介しました。今回はその裏側 ―― 無人ジョブが黙って止まっていた事実を、誰も気づかないまま放置してしまう問題とその対処の話です。
無人自動化の本当の怖さはバグではなく、サイレント失敗です。launchd のジョブが exit 78 でクラッシュしていても、hooks のスクリプトが実行権限を失っていても、agentmemory がポートを開いたまま worker 不在の「半生状態」で動いていても、表面上は何も起きていないように見えます。この状態が6日間続いて誰にも気づかれなかった実例が `automation-health.sh` のコメントに残っています。
自律ループを組むと監視対象が増えます。私の構成だけでも:
これらが全部正常かどうかを手で確認する気にはなれません。かといって個別に `launchctl list` を叩いても「全体像」が見えない。
必要なのは 全自動化をまとめて点検して、GREEN / WARN / FAIL の3値で教えてくれる1コマンドです。
`~/.claude/scripts/automation-health.sh` のコアはシンプルです。
fail=0; warn=0
ok() { printf " ${GRN}✓${RST} %s\n" "$1"; }
wn() { printf " ${YEL}⚠${RST} %s\n" "$1"; warn=$((warn+1)); }
ng() { printf " ${RED}✗${RST} %s\n" "$1"; fail=$((fail+1)); }チェック項目を `ok` / `wn` / `ng` のいずれかで評価していき、最後にこう判定します。
if [ "$fail" -gt 0 ]; then
printf " ${RED}✗ FAIL${RST} red=%d warn=%d\n\n" "$fail" "$warn"; exit 1
elif [ "$warn" -gt 0 ]; then
printf " ${YEL}⚠ WARN${RST} warn=%d (致命的問題なし)\n\n" "$warn"; exit 0
else
printf " ${GRN}✓ ALL GREEN${RST} 全自動化が正常稼働\n\n"; exit 0
fiexit code を分けることで、`daily-brief.sh` や CI から `||` で後続処理を繋げられます。
スクリプトは9節に分かれています。実コードから主要な判定を抜粋します。
for job in com.shun.skill-harvest com.shun.skill-curate; do
line=$(launchctl list 2>/dev/null | grep -E "\b${job}\b")
if [ -z "$line" ]; then
ng "$job: 未ロード (launchctl load し直しが必要)"
else
exitc=$(echo "$line" | awk '{print $2}')
if [ "$exitc" = "0" ] || [ "$exitc" = "-" ]; then
ok "$job: ロード済 / last exit=$exitc"
else
ng "$job: last exit=$exitc (前回失敗)"
fi
fi
done`launchctl list` の2列目が last exit code です。`-` は「まだ起動していない(正常)」、0 は成功、それ以外は失敗です。
hooks=(pre_git_guard.sh pre_secrets_check.sh pre_env_guard.sh \
post_audit_log.sh post_format.sh post_tsc_check.sh \
stop_notify.sh user_prompt_submit.sh)
for h in "${hooks[@]}"; do
f="$CLAUDE/hooks/$h"
if [ ! -f "$f" ]; then ng "$h: 不在"
elif [ ! -x "$f" ]; then wn "$h: 実行権限なし (chmod +x 推奨)"
else ok "$h"
fi
donesettings.json に hook を書いても、実体スクリプトが `chmod +x` されていないと silently skip されます。これを WARN で検出します。
hlog="$CLAUDE/skills/auto/.harvest.log"
a=$(age_h "$hlog") # (now - mtime) / 3600 を返すヘルパ
if [ "$a" -ge 0 ] && [ "$a" -le 48 ]; then
ok "最終稼働 ${a}h前: ${last##*] }"
else
wn "最終稼働 ${a}h前 (>48h: cron停止の疑い): ${last##*] }"
fi
nskills=$(find "$CLAUDE/skills/auto" -maxdepth 2 -name SKILL.md | wc -l | tr -d ' ')
ok "生成済 auto-skill: ${nskills} 個"ログの mtime が 48h 以上古ければ WARN。「ログがある」と「最近走った」は別の話なので mtime で見ます。
cnt=$(find "$CONV" -name '*.md' ! -name 'INDEX.md' | wc -l | tr -d ' ')
ok "会話ログ ${cnt} 件 / 最新 ${la}h前"
ia=$(age_h "$idx")
if [ "$ia" -le 24 ]; then ok "INDEX.md 鮮度 ${ia}h前"
else wn "INDEX.md が ${ia}h前 (extract が更新していない可能性)"; fiif [ -f "$hot" ] && grep -q "recent:start auto-updated" "$hot"; then
ha=$(age_h "$hot"); ok "hot.md 自動更新マーカー有 / ${ha}h前"
else wn "hot.md の自動更新マーカーが見つからない"; fi
# index.md カバレッジ: 実ファイル数 vs 記載ページ数
real=$(find "$VAULT" -name '*.md' -not -path '*/.*' | wc -l | tr -d ' ')
stated=$(grep -oE '総ページ数:[0-9]+' "$vidx" | grep -oE '[0-9]+' | head -1)
if [ "$stated" = "$real" ]; then ok "index.md カバレッジ一致 (${real}p)"
else wn "index.md 記載 ${stated:-?}p ≠ 実 ${real}p"; fifor f in now.md recent.md archive.md; do
[ -f "$REMEMBER/$f" ] && ok "$f 存在" || wn "$f 不在 (必須層)"
done
dup=$(grep -vE '^\s*$|^##|^#' "$nowf" | sort | uniq -c | sort -rn | head -1 | awk '{print $1}')
if [ "${dup:-0}" -ge 3 ]; then
wn "now.md に同一要約が ${dup} 回 (consolidate 遅延の兆候)"
else
ok "now.md 重複なし (consolidate 健全)"
fi`now.md` は consolidate が遅れると同じ要約が繰り返し追記されます。3回以上の重複を WARN で早期検知します。
cdir_mb=$(( ${cdir_total_mb:-0} - ${disabled_mb:-0} ))
if [ "${cdir_mb:-0}" -lt 2000 ]; then ok "~/.claude 実効 ${cdir_mb}MB"
elif [ "${cdir_mb:-0}" -lt 5000 ]; then wn "~/.claude 実効 ${cdir_mb}MB (>2GB: 大きめ)"
else ng "~/.claude 実効 ${cdir_mb}MB (>5GB: 異常肥大)"; fi
orphan=$(find "$CLAUDE" -maxdepth 1 -name 'security_warnings_state_*.json' -mtime +7 | wc -l | tr -d ' ')
if [ "${orphan:-0}" -ge 5 ]; then
wn "security_warnings_state_*.json が ${orphan} 個 (>7日前: backups/ へ退避推奨)"
fireversible cache(`.disabled-cache`)は実効容量から除外して判定します。
declare -a CRON_JOBS=(
"weekly cleanup-misc:$CLAUDE/logs/cleanup-misc.log:192"
"weekly env-audit:$CLAUDE/logs/env-audit-latest.md:192"
"monthly plugin-purge:$CLAUDE/logs/plugin-purge.log:744"
"weekly plugin-auto-disable:$CLAUDE/logs/plugin-auto-disable.log:192"
"weekly dotfiles-snapshot:$CLAUDE/logs/dotfiles-snapshot.log:192"
"weekly agents-index:$CLAUDE/logs/agents-index.log:192"
"daily plugin-usage:$CLAUDE/scripts/plugin-audit-latest.md:48"
)
for entry in "${CRON_JOBS[@]}"; do
IFS=":" read -r name path budget <<< "$entry"
a=$(age_h "$path")
if [ "$a" -le "$budget" ]; then ok "$name: ${a}h前 (budget ${budget}h)"
else wn "$name: ${a}h前 (budget ${budget}h 超過 — launchd 配送失敗の可能性)"; fi
done週次ジョブなら 192h(8日)、月次なら 744h(31日)を budget として、それを超えたら WARN です。
agentmemory サーバのチェックだけ、判定が2段階になっています。
am_line=$(launchctl list 2>/dev/null | awk '$3=="com.shun.agentmemory"{print $1" "$2}')
am_pid=${am_line%% *}; am_exit=${am_line##* }
if [ -z "$am_line" ]; then
wn "com.shun.agentmemory が launchd に未ロード"
elif [ "$am_pid" = "-" ] && [ "$am_exit" != "0" ]; then
ng "com.shun.agentmemory 停止中 (last exit=$am_exit) — クラッシュループの可能性"
else
am_code=$(curl -s -o /dev/null -w '%{http_code}' -m 3 \
http://localhost:3111/agentmemory/health 2>/dev/null || echo 000)
if [ "$am_code" = "200" ]; then ok "稼働中 (pid=$am_pid / health 200)"
elif [ "$am_code" = "000" ]; then ng "プロセスは居るが port 3111 無応答"
else ng "port 3111 は開くが /agentmemory/health=$am_code — worker 不在の半生状態"; fi
fiコメントに理由が書いてあります。
launchd の exit 78 クラッシュループが6日間誰にも気づかれず、さらに「ポートは開くが worker 不在で全API 404」の半生状態は死活監視では見えない。
launchd の状態チェックだけでは「プロセスがいるが実際には動いていない」ケースを検出できません。`curl` で HTTP 200 を確認する理由はここにあります。
launchd の PID チェックと HTTP ヘルスエンドポイントの両方を見ないと、「起動しているように見えて全 API が 404」という半生状態を見逃します。MCP サーバや API サーバを launchd で動かすなら、必ず `/health` エンドポイントを叩く一手間を足してください。
見落としがちな項目が9番目の二重実行チェックです。
cron_sh=$(crontab -l 2>/dev/null | grep -vE '^[[:space:]]*#' | \
grep -oE '/[^ ]+\.sh' | xargs -n1 basename | sort -u)
launchd_sh=$(grep -hoE '/[^<> ]+\.sh' \
~/Library/LaunchAgents/com.shun.*.plist | xargs -n1 basename | sort -u)
dup=$(comm -12 <(printf '%s\n' "$cron_sh") <(printf '%s\n' "$launchd_sh") | \
grep -vE '^[[:space:]]*$')
if [ -n "$dup" ]; then
ng "cron と launchd の両方に登録され二重実行されるスクリプト: $(printf '%s' "$dup" | tr '\n' ' ')"
fiコメントには「2026-06-01: cron→launchd 移行で cron を消し忘れて両方に登録=毎サイクル二重実行が発生」とあります。移行後に古い crontab を消し忘れたパターンです。
`daily-brief.sh` は毎朝 8:00 に launchd から起動し、冒頭の環境ヘルスに1行だけ組み込みます。
echo "## 🏥 環境ヘルス"
run_to 60 ~/.claude/scripts/automation-health.sh 2>&1 | tail -3 | head -1`tail -3 | head -1` は最終サマリ行(`✓ ALL GREEN` / `⚠ WARN` / `✗ FAIL` の行)だけを切り出しています。
さらに `daily-brief.sh` 側では、ライブ疎通プローブで RED を検出したときだけ `## 🚨 要対応` セクションが現れます。
RED_FLAGS=""
flag_red() { RED_FLAGS="${RED_FLAGS}\n- ❌ $1"; }
if [ -n "$RED_FLAGS" ]; then
echo "## 🚨 要対応(ライブ疎通でRED)"
printf '%b\n' "$RED_FLAGS"
fi問題がないときこのセクションは存在しません。**「失敗中だけ現れる節」**として設計されています。brief は `~/Desktop/Daily Brief/today-brief-YYYYMMDD.md` に書き出されるので、デスクトップを開いたときに RED が目に入る形になります。
`automation-health.sh` の exit 1 を使えば、launchd のラッパーで同様のパターンを追加できます。
bash ~/.claude/scripts/automation-health.sh || touch ~/Desktop/AUTOMATION_FAILED.md
bash ~/.claude/scripts/automation-health.sh && rm -f ~/Desktop/AUTOMATION_FAILED.md失敗中だけファイルが存在し、直ったら消える ―― ダッシュボードを見なくてもデスクトップが状態を教えてくれます。
次回は、このヘルスチェックと autopilot が生成した改善ログを統合して、週次で何が変わったかをまとめる ―― 週次レポート自動生成の仕組みを書く予定です。
Lily(@bokuwalily)― 個人開発者。Claude Code で自動化基盤を組みながら、iOSアプリやWebサービスを量産しています
◼︎作ったアプリは **ポートフォリオ** にまとめています📱
◼︎新着・開発の裏側は X **@bokuwalily** で発信しています🐦
◼︎OSS: **github.com/bokuwalily**🐙
皆さんの ❤️ やシェアが励みになります!