gke-ai-troubleshooting-tpu-mxla-hang

Diagnose GKE Cloud TPU multi-slice training hangs (Megascale `HANG_DETECTED` logs and hang monitored events) using the ML Diagnostics `Megascale XLA (MXLA) Hang Analyzer` (`gcloud alpha mldiagnostics monitored-events`) and 1-minute Cloud Monitoring multi-slice latency metrics (`kubernetes.io/container/multislice/*`). Distinguishes XLA compiler/HLO launch divergence and host data-input stalls from TPU chip, SparseCore, ICI, or network fabric faults. Use when multi-slice TPU training jobs freeze w

Install

KitScout test notes

Not tested by KitScout yet. When our harness runs it, pros, cons and the tasks we tried appear here, with the date and evidence.

Similar tools

• observability-and-instrumentation 117,221 ★
  Instruments code so production behavior is visible and diagnosable. Use when adding logging, metrics, tracing,
• honeycomb-tracing 113,505 ★
  Trace Paperclip runs, requests, and full CI suites in Honeycomb. Use to find traces, diagnose latency, design
• Netdata 81,392 ★
  Real-time infrastructure monitoring with metrics, logs, alerts, and ML-based anomaly detection.
• azure-monitor-opentelemetry-ts 49,672 ★
  Auto-instrument Node.js applications with distributed tracing, metrics, and logs.
• devops-troubleshooter 49,672 ★
  Expert DevOps troubleshooter specializing in rapid incident response, advanced debugging, and modern observabi