feat: kontinuierliche LXC↔Mac-Diagnose + Auto-Retry

Diagnose-Sichtbarkeit (Plan A):
- Job.step_started_at + last_heartbeat_at, Heartbeat-Task pingt alle 3 s
  während laufender Mac-Calls
- Mac-Worker hält per X-Job-Id Header einen Log-Ringbuffer pro Job
  (200 Zeilen, 1 h TTL); GET /api/jobs/{id}/log liefert ihn aus
- LXC-Endpoint GET /api/jobs/{id}/diag bündelt Job-Status, Stage-Sekunden,
  Heartbeat-Alter und Mac-Worker-Log
- Frontend: Live-Timing im Status-Pill ("läuft seit Xs · letzter Ping vor Ys"),
  klappbares Diagnose-Panel mit Mac-Log für FAILED + langlaufende Jobs

Robustheit (Plan B):
- MacClient: einmaliger Retry mit 3 s Backoff bei RemoteProtocolError /
  ConnectError / ReadError für summarize/protocol/export
- Mac-Worker /api/preload heizt Ollama vor (keep_alive 30 m, num_predict 1);
  Pipeline ruft Preload vor summarize, damit Modell-Reload nicht mitten
  in der Generierung passiert (Hauptursache "Server disconnected")

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-05-14 09:12:27 +00:00
co-authored by Claude Opus 4.7
parent 354e4c57eb
commit f408a555f8
9 changed files with 558 additions and 48 deletions
+19
View File
@@ -120,6 +120,8 @@ async def retry_job(
job.status = JobStatus.QUEUED
job.progress = 0
job.error = ""
job.step_started_at = None
job.last_heartbeat_at = None
session.add(job)
session.commit()
session.refresh(job)
@@ -128,6 +130,21 @@ async def retry_job(
return _job_dict(job, user)
@router.get("/{job_id}/diag")
async def diag(job_id: int, session: Session = Depends(get_session), user: User = Depends(current_user)):
"""Diagnose-Bündel: Job-Status + letzte Mac-Worker-Logzeilen für genau diesen Job."""
from app.core.mac_client import MacClient
job = _require_access(session.get(Job, job_id), user)
client = MacClient()
mac_log = await client.fetch_log(job_id, last=200)
return {
"job": _job_dict(job, user),
"step_started_at": job.step_started_at.isoformat() if job.step_started_at else None,
"last_heartbeat_at": job.last_heartbeat_at.isoformat() if job.last_heartbeat_at else None,
"mac_log": mac_log,
}
@router.get("/{job_id}/download/{kind}")
def download(job_id: int, kind: str, session: Session = Depends(get_session), user: User = Depends(current_user)):
job = _require_access(session.get(Job, job_id), user)
@@ -158,6 +175,8 @@ def _job_dict(j: Job, user: User, owner_username: str | None = None) -> dict:
"profile": j.profile or "meeting",
"created_at": j.created_at.isoformat(),
"updated_at": j.updated_at.isoformat(),
"step_started_at": j.step_started_at.isoformat() if j.step_started_at else None,
"last_heartbeat_at": j.last_heartbeat_at.isoformat() if j.last_heartbeat_at else None,
"has": {
"transcript": bool(j.transcript_path),
"summary": bool(j.summary_path),