Monitoring Alert Agent
Checking whether a URL returns 200 is not agent work. It's a loop and an if-statement, it costs nothing, and it should run every minute forever. The agent earns its keep in the thirty seconds after the check fails: pulling recent logs, noticing the disk filled up an hour before the crash, and writing an alert that says "the site is down, here's probably why, here's the likely fix" instead of just "DOWN."
That split is the workflow-versus-agent decision from §3.6 in miniature. Deterministic checking stays deterministic. The model only runs when there's something to diagnose, which also means you're not paying inference costs to confirm, sixty times an hour, that everything is fine.
Two rules keep this from becoming its own incident. The agent suggests fixes; it never applies them. An agent that restarts services on its own diagnosis will eventually restart the wrong thing at the worst time, and you'll spend a weekend learning why. And alerts are deduplicated through a state file, because the fastest way to stop reading alerts is to receive forty identical ones. An ignored monitoring system is decoration.
Prerequisites
- Python 3.10+ with CrewAI installed: `pip install crewai crewai-tools`.
- An LLM API key exported as an environment variable.
- Something to watch: a URL, plus wherever its logs live.
"""
Monitoring Alert Agent
- A plain function does the actual check (cheap, deterministic).
- On failure, the agent gathers context, diagnoses, and DRAFTS an
alert with a suggested fix.
- A state file dedupes alerts; recovery clears the state.
- The agent never restarts, deletes, or fixes anything itself.
"""
import json
import urllib.request
from datetime import datetime
from pathlib import Path
from crewai import Agent, Task, Crew, Process
from crewai_tools import tool
TARGET_URL = "[https://your-site.example.com/health]"
LOG_FILE = Path("[/var/log/your-app/app.log]")
STATE = Path("state/alert_state.json") # {"open_incident": bool, "since": ...}
def check_target() -> dict:
"""The deterministic check. Not a tool: the agent never decides
whether to check. Cron decides. Returns status the code acts on."""
try:
req = urllib.request.Request(TARGET_URL, method="GET")
with urllib.request.urlopen(req, timeout=10) as resp:
return {"ok": 200 <= resp.status < 400, "status": resp.status}
except Exception as e:
return {"ok": False, "status": None, "error": str(e)}
# ---------- Tools the agent uses AFTER a failure ----------
@tool("read_recent_logs")
def read_recent_logs(lines: int = 100) -> str:
"""Return the last N log lines. Stub: swap for journalctl, your
platform's log API, or `docker logs` as appropriate."""
if not LOG_FILE.exists():
return "(no log file found at configured path)"
return "\n".join(LOG_FILE.read_text().splitlines()[-lines:])
@tool("read_recent_deploys")
def read_recent_deploys() -> str:
"""What changed lately? Stub: swap for `git log --since=2.days`,
your CI history, or your platform's deploy list. 'What changed'
answers most incidents faster than any log line."""
return "[stub: return recent deploys/commits with timestamps]"
@tool("draft_alert")
def draft_alert(summary: str, evidence: str, suggested_fix: str) -> str:
"""Draft the alert for a human. Stub: swap for email, Slack, or
SMS. Even wired to a real channel, it alerts a human; it does
not act."""
print("\n[ALERT DRAFT]")
print(f"Summary: {summary}")
print(f"Evidence:\n{evidence}")
print(f"Suggested fix (for a HUMAN to apply): {suggested_fix}")
print("[END ALERT]\n")
return "alert drafted"
# ---------- Agent ----------
on_call_analyst = Agent(
role="On-call Analyst",
goal=(
"When the check fails, figure out the most likely cause from "
"logs and recent changes, then draft one clear alert with a "
"suggested fix."
),
backstory=(
"You are calm at 3am. You report what the evidence shows and "
"clearly label guesses as guesses. You NEVER run commands that "
"change state: no restarts, no deletes, no config edits. You "
"diagnose; a human decides."
),
tools=[read_recent_logs, read_recent_deploys, draft_alert],
verbose=True,
)
diagnose_task = Task(
description=(
"The health check for {url} just failed with: {failure}.\n"
"1. Read recent logs and recent deploys.\n"
"2. Form a most-likely cause. Cite the exact log lines or "
"deploy that support it. If the evidence is thin, say so.\n"
"3. Draft ONE alert: a one-line summary, the evidence, and a "
"suggested fix a human can apply. If you genuinely cannot "
"tell, the honest alert is 'down, cause unknown, checked X "
"and Y'. Do not invent a confident story."
),
agent=on_call_analyst,
expected_output="Confirmation that one alert was drafted.",
)
def main():
result = check_target()
state = json.loads(STATE.read_text()) if STATE.exists() else {}
STATE.parent.mkdir(parents=True, exist_ok=True)
if result["ok"]:
if state.get("open_incident"):
print(f"[RECOVERED] {TARGET_URL} at {datetime.now()}")
STATE.write_text(json.dumps({"open_incident": False}))
return
if state.get("open_incident"):
# Already alerted for this incident. Stay quiet until recovery.
print("[SUPPRESSED] incident already alerted")
return
STATE.write_text(json.dumps(
{"open_incident": True, "since": str(datetime.now())}))
crew = Crew(agents=[on_call_analyst], tasks=[diagnose_task],
process=Process.sequential, verbose=True)
crew.kickoff(inputs={"url": TARGET_URL, "failure": json.dumps(result)})
if __name__ == "__main__":
main()Adaptation notes:
- Schedule
main()with cron or launchd every 1 to 5 minutes. The agent only runs on the first failure of an incident, so the steady-state cost is one HTTP request. - Watching something other than a URL? Replace
check_targetwith any function returning{"ok": bool, ...}: a disk-space threshold, a queue depth, a "did the nightly backup file appear" check. Everything downstream stays the same. - Wire
draft_alertto email or Slack once the printed drafts have been sensible for a week. Notifying a human is a safe automation; the line the agent never crosses is changing state on the server. - Add a recovery notification in the
RECOVEREDbranch. "It's back, 12 minutes of downtime" is the difference between a system you trust and one you have to go check on. - The mistake people make: letting the agent apply its own suggested fix, because the suggestions were right five times in a row. The sixth suggestion restarts your database mid-backup. Diagnosis and action have different blast radii; keep the human between them.