# Fault detection, isolation, and recovery (FDIR)

This document defines how **FiWiControl** detects faults, **isolates** their impact (fail-safe defaults, timeouts, bounded inputs), and how operators **recover**. It complements **`docs/architecture.md`** and the **`[FiWi-FDIR]`** log prefix implemented in **`fiwicontrol.fabric.fdir`**.

**Scope:** Lab automation, fabric JSON/INI workflows, USB discovery, and related CLIs. It does **not** define network datapath FDIR (that lives in the Fi‑Wi system design — see **`html/Fi-Wi-L4S.html`**).

---

## 1. Principles

1. **Detect early** — Compare **live** USB topology to **saved** `discovery_fingerprint` before trusting power or harness steps.
2. **Fail closed for strict paths** — `Fabric.realize(strict=True)` and `fabric_realize.py --realize` (default strict) **abort** on fingerprint mismatch.
3. **Bound resource use** — Patch-panel JSON loads are **size- and entry-capped** (`fiwicontrol.fabric.patch_panel_json`).
4. **Observable** — Use exit codes, stderr lines prefixed with **`[FiWi-FDIR]`**, and structured log records for aggregation.
5. **No silent hardware actions** — Harnesses use **`--dry-run`** until power paths are explicitly wired; see harness **DESIGN_GAPS** in script docstrings.

---

## 2. Fault taxonomy

| Class | Meaning | Typical cause |
|-------|---------|----------------|
| **F-ENV** | Environment / dependency | Missing `[power]` / `brainstem`, USB permissions, empty Acroname enumeration. |
| **F-CFG** | Configuration | Bad INI/JSON, schema validation, missing `[fabric.rrh.*]`, strict INI verify failure without `--force`. |
| **F-FAB-STALE** | Fabric fingerprint mismatch | Wrong hub, cable, or JSON from another bench. |
| **F-FAB-DISC** | Discovery fault / timeout | Stuck USB enumeration, driver issue, **`--realize-discovery-timeout`** exceeded. |
| **F-SSH** | Remote execution | Unreachable host, auth failure, timeout — surfaced by `ssh_node` / `rexec` (see **`docs/node-control-asyncio-design.md`**). |
| **F-INV** | Inventory vs reality | `verify-inventory` / `--strict-ini` mismatches (Acroname / Monsoon counts). |

---

## 3. Detection mechanisms

### 3.1 `Fabric.binding_cache_status(path)`

| Status | Detection |
|--------|-----------|
| **MISSING** | JSON path does not exist. |
| **INVALID** | Unreadable or invalid JSON / schema validation failure. |
| **UNKNOWN** | Live fingerprint unavailable (e.g. discovery not possible). |
| **STALE** | Live fingerprint ≠ on-disk `discovery_fingerprint`. |
| **READY** | Match. |

Logs (via **`log_fdir`**) record **INVALID**, **UNKNOWN**, and **STALE** at **INFO**/**WARNING** as appropriate; **READY** at **DEBUG**.

### 3.2 `Fabric.realize(strict=…)`

- Runs **live** Acroname discovery (optional **`discovery_timeout`**).
- **strict=True:** raises **`ValueError`** on fingerprint mismatch (preceded by **`[FiWi-FDIR]`** WARNING log).
- **strict=False:** continues; emits **`[FiWi-FDIR]`** WARNING with **expected** and **live** fingerprints (audit trail).

### 3.3 `python3 -m fiwicontrol.fabric status`

Uses binding cache semantics; exit **0** only when **READY** (see **`docs/fabric-builder.md`**).

### 3.4 `scripts/system/fabric_realize.py`

- **Compose** path: discovery once; failures return **F-ENV**/**F-CFG** exit codes (below).
- **`--realize`:** second discovery pass inside **`Fabric.realize`** with **`--realize-discovery-timeout`** (default **120** s).

---

## 4. Isolation

| Mechanism | Effect |
|-----------|--------|
| **Strict realize** | Prevents proceeding with a **stale** fabric definition when strict mode is on. |
| **Discovery timeout** | Cancels hung enumeration during **`Fabric.realize`** when a timeout is set (CLI: **`--realize-discovery-timeout`**). |
| **Patch panel JSON limits** | Ignores oversized or over-count **`bdf_to_patch`** maps (see module docstring in **`patch_panel_json.py`**). |
| **Concentrator report** | Human report isolates concentrator probe failures: snapshot errors become a **parenthesized** line instead of aborting the whole fabric summary. |
| **pytest remote tests** | **`FIWI_RUN_REMOTE_TESTS`** gates SSH integration so CI does not hit live rigs by default. |

---

## 5. Recovery (operator playbook)

| Symptom | Likely class | Recovery steps |
|---------|--------------|----------------|
| Exit **1** / “No Acroname modules” | F-ENV | Install **`pip install -e ".[power]"`**, fix USB/cable, check permissions. |
| Exit **2** / RRH_BINDING_HELP | F-CFG | Add **`[fabric.rrh.<id>]`** with **`acroname_port`**, or run **`python3 -m fiwicontrol.fabric build`**. |
| Exit **3** / fingerprint mismatch | F-FAB-STALE | Re-run **`fabric build`** / **`fabric_realize --json`** on the **correct** host; fix cabling; use **`--no-strict`** only with operational approval. |
| Exit **4** / discovery timeout | F-FAB-DISC | Increase **`--realize-discovery-timeout`**; power-cycle hub; check **`brainstem`** / USB stability. |
| **`STALE` from `status`** | F-FAB-STALE | Same as exit **3**; treat harness start as **unsafe** until **READY** or explicit override (e.g. **`--strict-fabric-ready`** on harness). |
| SSH hangs / failures | F-SSH | Verify **`FIWI_REMOTE_IP`**, keys, **`FIWI_SSH_CONFIG`**; see **`docs/install.md`**. |
| INI verify failures | F-INV | Run **`python3 -m fiwicontrol.power --verify-inventory`**; align **`acroname`** / **`monsoon`** tokens with **`--discovery-json`**. |

---

## 6. Exit codes — `fabric_realize.py`

Aligned with **`fiwicontrol.fabric.fdir.FabricExitCode`**:

| Code | Constant | Meaning |
|------|----------|---------|
| **0** | `SUCCESS` | Success. |
| **1** | `ENVIRONMENT_OR_DISCOVERY` | Missing power extra, discovery exception, no modules. |
| **2** | `CONFIGURATION` | INI/JSON/validation, strict-ini without `--force`, missing INI, no RRH rows. |
| **3** | `FABRIC_STALE` | **`--realize`**, strict fingerprint mismatch. |
| **4** | `FABRIC_DISCOVERY_FAULT` | **`--realize`**, discovery timeout or wrapped discovery error. |

Use **`fabric_realize.py --help`** for the same table (epilog).

---

## 7. Logging and monitoring

- **Filter:** `grep` / log pipelines for **`[FiWi-FDIR]`**.
- **Levels:** **WARNING** for stale fabric and non-strict continuation; **INFO** for invalid/unknown binding status; **DEBUG** for READY binding checks.
- **Recommendation:** In production log aggregation, alert on **STALE** + **`--realize`** failures (exit **3**/**4**) tied to **campaign IDs** or **`fabric_id`**.

---

## 8. Out of scope

- **Datapath** packet loss, L4S marking, or Wi‑Fi MAC recovery — see **`html/Fi-Wi-L4S.html`**.
- **Automatic power cycling** in PCIe harness — not enabled in default **`--dry-run`** skeleton; see harness **DESIGN_GAPS**.
- **Medical / life-safety** FDIR — explicitly disclaimed in **`README.md`** and **`docs/install.md`**.

---

## 9. Maintenance

When adding CLIs or changing exit codes:

1. Update **`FabricExitCode`** in **`src/fiwicontrol/fabric/fdir.py`**.
2. Update **`fabric_realize.py`** epilog and this document **§6**.
3. Add or adjust **pytest** coverage for new fault paths where practical.