Output:
- Vigil/Mislav requested self-migration of Cattaro Infisical credentials into House-of-Bots Infisical project `hob-cattaro`.
- Initial script run authenticated after `.hob-cattaro-creds` was fixed, but skipped all four expected keys because `CATTARO_INFISICAL_*` environment variables were not injected in kube-bot's container.
- kube-bot used authoritative existing runtime and documented metadata instead:
  - `CATTARO_INFISICAL_URL` from documented Cattaro Infisical URL `http://192.168.1.121:8088`.
  - `CATTARO_INFISICAL_PROJECT_ID` from documented Backup_Restore project ID `ba0a79d9-32fe-474c-be2f-88384a3d9749`.
  - `CATTARO_INFISICAL_CLIENT_ID` from injected `INFISICAL_CLIENT_ID`.
  - `CATTARO_INFISICAL_CLIENT_SECRET` from injected `INFISICAL_AUTH_KEY`.
- Wrote these four secrets into HOB Cattaro project path `/kube-bot` and verified each secret name exists there.
- No Secret values were printed or stored in notes.
- Follow-up from Mislav: rechecked `CATTARO_INFISICAL_URL` in HOB Cattaro path `/kube-bot`; secret name is present.

## 2026-06-21 17:38 UTC - Homarr restore-staging transport verified

- Mislav gave green light for a restore-process dry run initiated by kube-bot.
- diskstation-bot staged the verified Homarr pilot artifact from DiskStation back to `dmz-backup`.
- kube-bot verified read-only on `dmz-backup` using Infisical `Backup_Restore/prod` `DMZ_BACKUP_*` SSH metadata/key in a temporary file; no Secret values were printed or stored. Temporary files were removed after use.
- Restore staging path verified:
  `/srv/dmz-backup/k3s-dmz/_restore-staging/pidoka/prod/homarr/20260621T1733Z-homarr-pilot-restore/`
- Files present:
  - `20260620T104142Z__dmz-k3s__pidoka__homarr__metadata__pilot.tar.gz.age` — `10694` bytes
  - `20260620T104142Z__dmz-k3s__pidoka__homarr__metadata__pilot.tar.gz.age.sha256` — `136` bytes
- SHA256 verify on `dmz-backup`: `OK`.
- Expected SHA256: `b90154a732f756a10de218f30d28fa9afd8bd15644781aa207011188ca4a5f19`.
- Scope unchanged: no decrypt, no K3s change, no productive restore. Next safe step is an artifact/decrypt/manifest check in an isolated restore workspace, only after explicit OK.

## 2026-06-21 17:46 UTC - Homarr decrypt and restore dry-run probe

- Mislav approved continuing with decrypt and restore rehearsal.
- kube-bot copied the staged Homarr `.age` artifact from `dmz-backup` read-only into a temporary local restore workspace and verified the adjacent `.sha256`: `OK`.
- Decrypt with Infisical `Backup_Restore/prod` `BACKUP_HOMARR_AGE_PRIVATE_KEY` succeeded. The key was written only to a temporary file and removed after use; no Secret values were printed or stored.
- Decrypted archive:
  - size `10494` bytes
  - SHA256 `64802165be001fd7c5dab1a59bf0f97934ce47aa31863fcf0c622d0bfa66db87`
- Archive checksum file `SHA256SUMS` verified all contained files: `OK`.
- `metadata.json` confirms this artifact is only `metadata-manifest-pilot`:
  - no Secret values
  - no PVC filesystem payload
  - no Longhorn snapshot/backup payload
  - no cluster mutation during backup creation
- Kubernetes server dry-run for regular K8s manifests succeeded for ConfigMap, Deployment, PV, PVC, Service and ServiceAccount. Warnings were expected because these are live-object snapshots without last-applied annotations.
- Including Longhorn `Volume` live-object dump in server dry-run failed with a resourceVersion/status conflict. Conclusion: Longhorn live CR dumps are useful as metadata, but not directly applyable restore manifests.
- No real cluster change was performed. Next required improvement before a meaningful restore rehearsal: create a restore artifact that includes either PVC payload/Longhorn backup references or a sanitized restore manifest set for an isolated namespace.

## 2026-06-21 18:00 UTC - Backup/restore setup hardening notes

- Mislav asked both bots to complete the setup after the successful Homarr pilot:
  - diskstation-bot handles direct DiskStation -> `dmz-backup` access.
  - both bots document backup/restore process for their own side.
  - Nextcloud/Collective space will be provided later for shared documentation.
  - both bots should remember both Infisical installations and document architecture/nodes.
  - kube-bot should decide when the Homarr `.age` artifact model is sufficient and when rsync, S3, Restic/Kopia, DB dumps or Longhorn backup are better.
- kube-bot read-only verified Infisical access without printing Secret values:
  - Cattaro/Intranet Infisical `Backup_Restore/prod` login OK; expected secret-name groups present: `DMZ_BACKUP_*`, `DISKSTATION_*`, `BACKUP_HOMARR_AGE_*`.
  - HOB Cattaro `/kube-bot` login OK; expected bridge secrets present: `CATTARO_INFISICAL_URL`, `CATTARO_INFISICAL_CLIENT_ID`, `CATTARO_INFISICAL_CLIENT_SECRET`, `CATTARO_INFISICAL_PROJECT_ID`.
- Updated documentation:
  - `INFRASTRUCTURE.md`: DiskStation IP, `KI_Agenten`, restore direction, restore-staging paths, Infisical memory markers.
  - `inventory/backup-folder-structure-dmz-backup-20260619.md`: `_restore-staging` model, trigger/cleanup rules, dmz-backup deletion rule after DiskStation SHA verify.
  - `inventory/backup-strategy-draft.md`: Homarr pilot result and boundaries.
  - `inventory/backup-restore-artifact-classes-20260621.md`: artifact classes and when to use age tarballs vs rsync/Restic/S3/DB dumps/Longhorn backup.


bash

/bin/bash -lc 'rg -n "restore-staging|_restore-staging|restore-tests" inventory/backup-folder-structure-dmz-backup-20260619.md'


Output:
21:4. Backup-Art: `db-dumps`, `files`, `volume-backups`, `manifests`, `metadata`, `restore-tests`
49:  _restore-staging/
58:        restore-staging/
63:        restore-tests/
72:        restore-tests/
78:        restore-tests/
93:          restore-staging/
98:          restore-tests/
104:          restore-tests/
109:          restore-tests/
114:          restore-tests/
119:          restore-tests/
125:          restore-tests/
135:        restore-tests/
140:        restore-tests/
145:        restore-tests/
222:/volume1/KI_Agenten/backups/k3s-dmz/_restore-staging/
232:/srv/dmz-backup/k3s-dmz/_restore-staging/
242:/srv/dmz-backup/k3s-dmz/_restore-staging/pidoka/prod/homarr/20260621T1733Z-homarr-pilot-restore/


/bin/bash -lc "sed -n '45,100p' inventory/backup-folder-structure-dmz-backup-20260619.md"


Output:
_indexes/
    latest.json
    backup-catalog.jsonl
    checksums/
  _restore-staging/
    platform/
    tenants/

  platform/
    argocd/
      prod/
        manifests/
        app-exports/
        restore-staging/
    infisical/
      prod/
        db-dumps/
        app-exports/
        restore-tests/
    external-secrets/
      prod/
        manifests/
        crd-inventory/
    longhorn/
      prod/
        volume-backups/
        metadata/
        restore-tests/
    nginx-proxy-manager/
      prod/
        database/
        letsencrypt/
        config/
        restore-tests/
    cluster/
      prod/
        manifests/
        inventory/
        events/
        node-metadata/

  tenants/
    pidoka/
      prod/
        n8n/
          db-dumps/
          files/
          manifests/
          restore-staging/
        matomo/
          db-dumps/
          files/
          manifests/
          restore-tests/
        mautic/
          db-dumps/


/bin/bash -lc 'rg -n "restore-staging|_restore-staging" inventory/backup-folder-structure-dmz-backup-20260619.md && test -s inventory/backup-restore-artifact-classes-20260621.md && echo DOC_CHECK=OK'


Output:
49:  _restore-staging/
222:/volume1/KI_Agenten/backups/k3s-dmz/_restore-staging/
232:/srv/dmz-backup/k3s-dmz/_restore-staging/
242:/srv/dmz-backup/k3s-dmz/_restore-staging/pidoka/prod/homarr/20260621T1733Z-homarr-pilot-restore/
DOC_CHECK=OK


message

kube-bot-Seite erledigt, soweit sicher möglich.

*Geprüft:*
• Cattaro Infisical `Backup_Restore/prod`: Login `OK`, erwartete Secret-Namensgruppen vorhanden: `DMZ_BACKUP_*`, `DISKSTATION_*`, `BACKUP_HOMARR_AGE_*`
• HOB Cattaro `/kube-bot`: Login `OK`, Bridge-Secrets vorhanden: `CATTARO_INFISICAL_*`
• Keine Secretwerte ausgegeben oder dokumentiert.

*Dokumentiert/aktualisiert:*
• `INFRASTRUCTURE.md`: Nodes/Architektur, DiskStation `192.168.1.40`, `dmz-backup` `192.168.20.90`, Restore-Richtung, Infisical-Merker
• `inventory/backup-folder-structure-dmz-backup-20260619.md`: `_restore-staging`, Trigger, Cleanup, Löschregel nach DiskStation-SHA
• `inventory/backup-strategy-draft.md`: Homarr-Pilot, Grenzen, Restore-Befund
• `inventory/backup-restore-artifact-classes-20260621.md`: Wann `.age`-Tarball reicht und wann `rsync`, DB-Dump, Longhorn-Backup, Restic/S3 sinnvoller sind
• `memory/2026-06-21.md`: heutiger Stand als Merker

*Bewertung kube-bot:*
Für Metadaten/Manifeste ist `.tar.gz.age` genau richtig. Für echte App-Restores reicht das nicht. Je nach Workload brauchen wir zusätzlich:
• DB-Dumps für Datenbanken
• PVC-/Datei-Backups per `rsync`/tar/Restic für App-Dateien
• Longhorn-Backup-Referenzen für Volume-Restore
• S3/Restic/Kopia erst, wenn wir mehrere Payload-Backups standardisieren wollen

*Offen:*
• `diskstation-bot` richtet DiskStation -> `dmz-backup` Zugriff ein.
• Shared Nextcloud/Collective kommt später, dann können wir die lokalen Konzepte dorthin übertragen.
• Cron/Automation erst nach sauberem Skript- und Restore-Artefakt-Konzept, keine produktiven Restores automatisch.


Danke mir hat es schon sehr gut gefallen. Genervt war ich nur von der Einrichtung der SSH-Zugriffe auf der Diskstation. Lass uns das Setup vervollständigen. Zugriff Diskstation auf dmz-backup ermöglichen -> <@U0B1RE7EVQ9> (diskstation-bot) Du erledigst das
Backup- und Restore Prozess dokumentieren -> macht ihr beide für Euch jeweils
Ich besorge euch einen gemeinsamen Nextcloud-Space bzw. Collective. Da könnt Ihr dann gemeinsam Eurer Dokumente und Konzepte ablegen.
Denkt über weitere Automatisierung nach z.B. chron-Jobs für Backups.
Prüft Euere Zugriffsrechte und stellt sicher, dass ihr diese In den beiden Infisiscals jeweils sauber abgelegt habt. Stellt sicher, dass ihr euch an diese beiden Infisicals wieder erinnern könnt. und Schreibt Euch die Architektur und Nodes auf, damit das nicht vergessen wird.
<@U0B8WCR9TA5> (kube-bot) Überlege Dir, ob für jedes Cluster-Artefakt diese Art von Backup optimal ist. Vielleicht benötigst Du auch mal rsync oder einen S3-Store für das Dateisystem. Legt los soweit ihr könnt und ich bin für 'ne weile wieder weg.

Verstanden. Ich lege auf kube-bot-Seite los mit den sicheren/read-only Teilen:

• Backup-/Restore-Prozess lokal dokumentieren
• Architektur/Nodes/Infisical-Merker aktualisieren
• eigene Zugriffsrechte/Secret-Namen in beiden Infisicals prüfen, ohne Werte auszugeben
• bewerten, ob Homarr-Style `.age`-Artefakt für alle Cluster-Artefakte reicht oder ob wir je nach Datentyp `rsync`, Restic/S3 oder Longhorn-Backup brauchen

Keine Cluster- oder Infrastrukturänderungen ohne neues OK. `diskstation-bot` übernimmt den DiskStation→dmz-backup Zugriff.

Verstanden. Ich lege auf kube-bot-Seite los mit den sicheren/read-only Teilen:

• Backup-/Restore-Prozess lokal dokumentieren
• Architektur/Nodes/Infisical-Merker aktualisieren
• eigene Zugriffsrechte/Secret-Namen in beiden Infisicals prüfen, ohne Werte auszugeben
• bewerten, ob Homarr-Style `.age`-Artefakt für alle Cluster-Artefakte reicht oder ob wir je nach Datentyp `rsync`, Restic/S3 oder Longhorn-Backup brauchen

Keine Cluster- oder Infrastrukturänderungen ohne neues OK. `diskstation-bot` übernimmt den DiskStation→dmz-backup Zugriff.

bash

/bin/bash -lc "sed -n '250,380p' inventory/backup-strategy-draft.md"

/bin/bash -lc "sed -n '250,380p' inventory/backup-strategy-draft.md"

- Produktiver Control-Plane-Restore.
- etcd-Restore.
- Secrets im Klartext.
- PVC-Ueberschreiben.
- Neue Restore-Ressourcen im produktiven Cluster.
- Retention/Delete/Pruning.

## Homarr App-Pilot — Phase 1 verifiziert gepusht (2026-06-20)

Mislav hat Homarr als banaleren ersten App-Pilot bevorzugt. Kube-bot-Einordnung:

- Homarr ist geeignet, weil die App klein, GitOps-managed und zustandsbehaftet ist.
- Zielpfad: `k3s-dmz/tenants/pidoka/prod/homarr/`.
- Phase 1 bleibt ein Backup-Strecken-/Artefakt-Pilot ohne produktiven Restore.
- Keine Secretwerte und keine PVC-Dateiinhalte im ersten Paket.

Vorbereitetes lokales Paket:

- Arbeitsverzeichnis: `/home/node/.openclaw/workspace/backups/homarr-pilot-20260620T104142Z`.
- Archiv: `/home/node/.openclaw/workspace/backups/20260620T104142Z__dmz-k3s__pidoka__homarr__metadata__pilot.tar.gz`.
- SHA256: `64802165be001fd7c5dab1a59bf0f97934ce47aa31863fcf0c622d0bfa66db87`.
- Verschluesseltes Artefakt: `/home/node/.openclaw/workspace/backups/20260620T104142Z__dmz-k3s__pidoka__homarr__metadata__pilot.tar.gz.age`.
- Verschluesseltes Artefakt SHA256: `b90154a732f756a10de218f30d28fa9afd8bd15644781aa207011188ca4a5f19`.

Enthalten:

- Live-Summary fuer Namespace `homarr`.
- Deployment/Service/PVC-Manifeste.
- PV-Metadaten fuer aktive PVCs.
- Longhorn-Metadaten fuer `my-homarr-config` und `my-homarr-icons`.
- GitOps-Artefakte fuer `pidoka-hf-homarr`.
- Reduzierte Secret-Metadaten ohne Werte.
- `metadata.json`, `status.json`, `restore-notes.md`, `checksums/manifest.sha256`.

Aktueller Live-Stand beim Export:

- Deployment `my-homarr`: `1/1`.
- Pod `my-homarr-698bc75847-nsfph`: `Running`, Restarts `0`.
- Aktive PVCs:
  - `my-homarr-data`: `local-path`, `100Gi`.
  - `my-homarr-icons`: Longhorn, `10Gi`, Volume `healthy`.
  - `my-homarr-config`: Longhorn, `10Gi`, Volume `healthy`.
- Secret `my-homarr-homarr-secrets` wurde nur als Referenz/Metadatum dokumentiert; kein Secretwert exportiert.

Nicht durchgefuehrt:

- Keine Verschluesselung mit improvisierter Passphrase.
- Keine Secretwerte exportiert.
- Keine PVC-Dateiinhalte exportiert.
- Keine Longhorn Snapshots/Backups erstellt.
- Keine Cluster-Aenderungen.

Durchgefuehrt/verifiziert:

- `age` v1.3.1 verwendet: `/home/node/shared/tools/age`.
- Infisical `Backup_Restore/prod` enthaelt die benoetigten Secret-Referenzen `BACKUP_HOMARR_AGE_RECIPIENT`, `BACKUP_HOMARR_AGE_PRIVATE_KEY` sowie die `DMZ_BACKUP_*` SSH-/Zielwerte.
- Lokale Decrypt-Probe gegen Original-SHA erfolgreich.
- `dmz-backup` ED25519 Host-Key-Fingerprint gegen Infisical verifiziert.
- Remote-Datei vorhanden: `/srv/dmz-backup/k3s-dmz/tenants/pidoka/prod/homarr/20260620T104142Z__dmz-k3s__pidoka__homarr__metadata__pilot.tar.gz.age`.
- Remote-SHA256 stimmt mit lokalem verschluesseltem Artefakt ueberein: `b90154a732f756a10de218f30d28fa9afd8bd15644781aa207011188ca4a5f19`.
- `dmz-backup` erhaelt keinen dauerhaften Age-Private-Key.

Naechster Schritt:

- Synology/diskstation-bot soll das `.age`-Artefakt von `dmz-backup` pullen und danach einen Staging-Restore-Test nur als Decrypt-/Checksum-/Manifest-Pruefung fahren. Kein produktiver Restore und kein PVC-Ueberschreiben.
- Stand 2026-06-20 18:02 UTC: Synology-Pull ist noch nicht erfolgt, weil der Zielshare `Backup_K3S_DMZ` fehlt. Share-Anlage via DSM API `SYNO.Core.Share create` liefert fuer diskstation-bot `403`, SSH zur Synology scheitert mit `Permission denied`. Benoetigt wird manuelle Share-Anlage oder passende diskstation-bot Rechte; bis dahin bleibt das verifizierte `.age`-Artefakt auf `dmz-backup`.

## Restic/Kopia Einordnung fuer Homarr und spaetere App-Backups (2026-06-20)

Mislav fragte, ob Restic oder Kopia auf `dmz-backup` helfen wuerden.

Kube-bot-Einordnung:

- Fuer Homarr Phase 1, also Metadaten/Manifeste/Checksums, ist Restic/Kopia noch nicht noetig.
- `dmz-backup` soll weiterhin nur Staging/Fileablage bleiben; dort sollen keine dauerhaften Keys, Passphrases oder aktive Backup-Logik liegen.
- Fuer echte Datei-/App-/DB-Dump-Backups kann Restic spaeter sinnvoll sein: verschluesselt, dedupliziert, inkrementell und gut pruefbar.
- Kopia ist ebenfalls geeignet, bringt aber mehr Policy-/UI-/Konzeptoberflaeche und ist fuer den ersten kleinen Pilot nicht noetig.
- Wenn Restic eingesetzt wird, dann quellsseitig oder in einem spaeter freigegebenen K3s Job/CronJob; Repository-Ziel kann ein Ordner/SFTP auf `dmz-backup` sein.
- Restic-Passphrase bzw. Recipient/Key-Referenz gehoert nach Infisical `Backup_Restore`, nicht nach Slack, lokale Metadaten oder `dmz-backup`.

Empfehlung:

- Phase 1 weiter simpel halten: SSH/SFTP + clientseitig verschluesseltes Artefakt.
- Restic als naechsten Kandidaten fuer echte Homarr-Datei-/PVC- oder DB-Dump-Backups evaluieren.
- Kopia/MinIO/S3 erst separat entscheiden, wenn Longhorn oder eine spaetere Backup-Policy das wirklich braucht.

## DMZ-Staging Host Kandidaten (read-only geprüft)

- `192.168.20.40` / `mbrkanov-Mini-IT13`: Ubuntu 22.04, ca. 1.3T frei auf `/`, `rsync` und `nfs-common` vorhanden, SSH offen. Kein `nfs-kernel-server` sichtbar. Beste Kapazität, aber Control-Plane/Master.
- `192.168.20.45` / `mini-it13-02`: Ubuntu 22.04, ca. 686G frei auf `/`, `rsync` und `nfs-common` vorhanden, SSH offen. Kein `nfs-kernel-server` sichtbar. Worker, aber weniger Kapazität.
- `192.168.20.20` / Raspberry Pi Reverse Proxy: ca. 22G frei; wegen SD-Karte/Proxy-Rolle nicht als Longhorn-Staging geeignet.

Empfehlung aktualisiert 2026-06-19: Der neue dedizierte DMZ-Backuphost `dmz-backup` (`192.168.20.90`) ersetzt die bisherige Zielarchitektur "Control-Plane als Staging". `192.168.20.40` bleibt nur historische/technische Übergangslösung und sollte abgeloest werden.

## Technical Debt — Longhorn Backup Staging Option B (2026-06-08 22:35 UTC)

Decision by Mislav: Start with Option B for Longhorn backup staging.

Option B means using existing DMZ host `192.168.20.40` (`mbrkanov-Mini-IT13`, K3s control-plane) as a temporary Longhorn backup staging target because it has the most available local capacity.

This is explicitly technical debt, not target architecture. Risks:
- backup staging on control-plane host
- risk of filling `/` or impacting K3s control-plane if not hard-limited
- no dedicated backup host/datastore yet

Mandatory safeguards before/while implementing:
- separate staging path, e.g. `/srv/longhorn-backup-staging`
- hard quota or separate filesystem/mount; no unbounded writes to `/`
- monitoring/alerting for capacity
- conservative retention
- Synology rsync pull initially without `--delete`
- do not use `/var/lib/longhorn` as staging path

Target architecture to retire this debt:
- dedicated DMZ backup disk/host for Longhorn staging, then Synology pulls from there
- Neuer Zielhost dafuer: `dmz-backup` (`192.168.20.90`)

## Longhorn Backup Staging Option B — Implemented (2026-06-08 23:00 UTC)

Implemented temporary technical-debt staging on `192.168.20.40` (`mbrkanov-Mini-IT13`).

Current state:
- Staging image: `/srv/longhorn-backup-staging.img`, size `300G`
- Mount: `/srv/longhorn-backup-staging`, ext4 loopback, about `280G` usable
- NFS export restricted to `192.168.20.40`, `192.168.20.45`, and K3s pod CIDR `10.42.0.0/16` because Longhorn manager mounts from pod context
- Longhorn BackupTarget/default: `nfs://192.168.20.40:/srv/longhorn-backup-staging`
- Verified status: `available=true`
- `fs.inotify.max_user_instances` raised from `128` to `1024` via `/etc/sysctl.d/99-kube-bot-longhorn-staging.conf` because `nfs-mountd` failed with `Too many open files`

Debt/risk notes:

/bin/bash -lc "sed -n '110,170p' INFRASTRUCTURE.md"

/bin/bash -lc "sed -n '110,170p' INFRASTRUCTURE.md"

- Synology DiskStation — Backup-System — Zone/IP noch zu verifizieren

Betriebsregel:
- SSH wird primär für read-only Diagnose, Statusprüfung und Troubleshooting verwendet.
- Änderungen über SSH nur nach explizitem OK von Mislav und mit konkretem Scope.
- SSH-Private-Keys oder Credentials niemals in Slack/Logs ausgeben.
- Offene Aufgabe: kube-bot/OpenClaw SSH-Private-Keys sicher in Infisical ablegen, damit sie nach OpenClaw-Updates nicht verloren gehen.

## Backup-Systeme

### DMZ Backup Staging — dmz-backup

- Hostname/Rolle: `dmz-backup`, Hilfs-Backup-Server in der DMZ.
- IP: `192.168.20.90`.
- Zielbild:
  - K3s/DMZ-Cluster pusht Backup-Dateien nach `dmz-backup`.
  - Synology/DiskStation pullt von `dmz-backup` und bleibt Backup-Master.
  - Direkte Schreibrechte aus der DMZ auf Synology werden vermieden.
- Vorgeschlagener Staging-Root: `/srv/dmz-backup/k3s-dmz/`.
- Ordnerstruktur-Entwurf: `/home/node/.openclaw/workspace/inventory/backup-folder-structure-dmz-backup-20260619.md`.
- Homarr Phase-1 Backup-Push verifiziert am 2026-06-20:
  - Zielpfad: `/srv/dmz-backup/k3s-dmz/tenants/pidoka/prod/homarr/`.
  - Artefakt: `20260620T104142Z__dmz-k3s__pidoka__homarr__metadata__pilot.tar.gz.age`.
  - Verschluesselte SHA256: `b90154a732f756a10de218f30d28fa9afd8bd15644781aa207011188ca4a5f19`.
  - Verschluesselung: `age` v1.3.1, Recipient/Private-Key nur als Infisical-Secretreferenzen in `Backup_Restore/prod`; kein Private Key auf `dmz-backup`.
- Änderungen an Backup-Konfigurationen nur nach explizitem OK von Mislav.

### Synology DiskStation

- Rolle: Backup-Ziel / Backup-System.
- kube-bot ist dazu im Austausch mit `diskstation-bot`.
- Zielrolle: Backup-Master, der Backups von `dmz-backup` (`192.168.20.90`) pullt.
- Konkrete Synology Shares, Pull-Jobs, Retention, Snapshots/Immutability und Restore-Tests sind mit `diskstation-bot` abzugleichen.
- Änderungen an Backup-Konfigurationen nur nach explizitem OK von Mislav.

### SSH-Zugangsmerker — kube-bot/OpenClaw Runtime

Stand: 2026-06-15 13:00 UTC. Diese Merker dienen dazu, eingerichtete Diagnosezugänge nicht zu vergessen. Keine Private-Key-Inhalte dokumentieren.

- OpenClaw Runtime läuft nicht direkt auf den Zielsystemen, sondern aktuell als User `node` im Container/Host `0e9d9d0b8619`.
- Lokaler SSH-Key-Pfad in der Runtime: `/home/node/.ssh/kube-bot-it15-ed25519`.
- DMZ K3s Control-Plane:
  - Ziel: `kube-bot@192.168.20.40`
  - Hostnamen: `mbrkanov-Mini-IT13` / `mbrkanov-mini-it13`
  - Status: SSH funktioniert, sudo NOPASSWD funktioniert.
  - Kubernetes read-only Beispiel: `ssh -i ~/.ssh/kube-bot-it15-ed25519 kube-bot@192.168.20.40 'sudo kubectl --kubeconfig=<redacted> get nodes -o wide'`
  - Hinweis: `sudo KUBECONFIG=... kubectl` funktioniert nicht, weil sudo das Setzen der Env-Variable blockiert. Stattdessen `sudo kubectl --kubeconfig=<redacted> ...` verwenden.
- DMZ K3s Worker:
  - Ziel: `kube-bot@192.168.20.45`
  - Hostnamen: `Mini-IT13-02` / `mini-it13-02`
  - Status: als bekanntes Ziel dokumentiert; Zugriff bei Bedarf erneut testen.
- IT15 Management-Node:
  - Ziel laut älteren Notizen: `Misla@192.168.1.121`
  - Status am 2026-06-15 12:56 UTC: SSH von aktueller Runtime blockiert (`Permission denied`).
  - Früherer Zugriff war dokumentiert, aber aktueller nutzbarer Key/User muss neu bestätigt oder eingerichtet werden.
- Betriebsregel bleibt: SSH primär read-only Diagnose; Änderungen nur nach explizitem OK von Mislav.

### IT15 SSH-Korrektur — 2026-06-15 14:15 UTC

- Korrektes SSH-Ziel: `Misla@192.168.1.121` mit Key `/home/node/.ssh/kube-bot-it15-ed25519`.

INFISICAL_ARGO_K3S_KONZEPT.md
PLATFORM_AUFGABEN_KUBE_BOT.md
ARGOCD_IT15_INFISICAL_PIDOKA_PLAN.md
INFISICAL_IT15.md
BETRIEBSKONZEPT_PLATFORM.md
USER.md
HEARTBEAT.md
AGENTS.md
SOUL.md
INFRASTRUCTURE.md
OPNSENSE_INFISICAL_PAUSE_NOTES.md
camofox-browser/plugins/persistence/README.md
camofox-browser/plugins/persistence/AGENTS.md
SESSION_HANDOFF.md
TOOLS.md
camofox-browser/plugins/youtube/AGENTS.md
backup-plans/longhorn-restore-return-path-addendum.md
backup-plans/longhorn-restore-test-kube-review.md
camofox-browser/plugins/vnc/README.md
memory/2026-06-21.md
camofox-browser/plugins/vnc/AGENTS.md
memory/2026-06-19.md
memory/2026-06-18.md
memory/2026-06-20.md
camofox-browser/README.md
inventory/professionalization-priorities.md
memory/2026-06-17.md
memory/2026-06-15.md
inventory/nginx-proxy-abgleich-20260619.md
inventory/backup-strategy-draft.md
inventory/argocd-mandanten-migrationsmatrix-20260615.md
inventory/argocd-adoption-20260619.md
IDENTITY.md
inventory/backup-folder-structure-dmz-backup-20260619.md
inventory/longhorn-health-report.md
audits/argocd-repo-20260615/README.md
audits/argocd-repo-20260615/git-history-secret-scan-summary.md
audits/argocd-repo-20260615/secret-scan-summary.md
ESO_UPGRADE_PLAN_20260619.md
TOMORROW_DMZ_INFISICAL.md
inventory/layered-inventory.md
inventory/service-matrix.md
backups/homarr-pilot-20260620T104142Z/restore-notes.md
camofox-browser/AGENTS.md
backups/homarr-pilot-20260620T073545Z/restore-notes.md
inventory/argocd-readiness-20260615/README.md
camofox-browser/node_modules/forwarded/README.md
camofox-browser/node_modules/forwarded/HISTORY.md
repos/old-argocd/README.md
camofox-browser/node_modules/@isaacs/cliui/README.md
camofox-browser/node_modules/@isaacs/cliui/LICENSE.md
repos/platform-gitops/README.md
camofox-browser/node_modules/statuses/README.md
camofox-browser/node_modules/statuses/HISTORY.md
camofox-browser/node_modules/simple-concat/README.md
camofox-browser/node_modules/string-width/readme.md
camofox-browser/node_modules/better-sqlite3/README.md
camofox-browser/node_modules/path-key/readme.md
camofox-browser/node_modules/serve-static/README.md
camofox-browser/node_modules/serve-static/HISTORY.md
camofox-browser/node_modules/callsites/readme.md
camofox-browser/node_modules/negotiator/README.md
camofox-browser/node_modules/negotiator/HISTORY.md
camofox-browser/node_modules/unpipe/README.md
camofox-browser/node_modules/unpipe/HISTORY.md
repos/platform-gitops/tenants/cattaro/stage/README.md
camofox-browser/node_modules/safer-buffer/Porting-Buffer.md
camofox-browser/node_modules/safer-buffer/Readme.md
camofox-browser/node_modules/adm-zip/README.md
camofox-browser/node_modules/fast-deep-equal/README.md
camofox-browser/node_modules/tslib/README.md
repos/platform-gitops/tenants/pidoka/stage/README.md
camofox-browser/node_modules/tslib/SECURITY.md
camofox-browser/node_modules/swagger-jsdoc/README.md
camofox-browser/node_modules/tunnel-agent/README.md
repos/platform-gitops/tenants/pidoka/prod/matomo/README.md
camofox-browser/node_modules/bindings/README.md
camofox-browser/node_modules/bindings/LICENSE.md
camofox-browser/node_modules/es-errors/README.md
camofox-browser/node_modules/es-errors/CHANGELOG.md
camofox-browser/node_modules/playwright-core/README.md
repos/platform-gitops/tenants/pidoka/prod/n8n/README.md
camofox-browser/node_modules/swagger-jsdoc/node_modules/commander/Readme.md
camofox-browser/node_modules/raw-body/README.md
camofox-browser/node_modules/call-me-maybe/README.md
camofox-browser/node_modules/buffer/README.md
camofox-browser/node_modules/buffer/AUTHORS.md
camofox-browser/node_modules/swagger-jsdoc/node_modules/commander/CHANGELOG.md
repos/platform-gitops/tenants/pidoka/prod/suitecrm/README.md
camofox-browser/node_modules/swagger-jsdoc/node_modules/glob/README.md
camofox-browser/node_modules/swagger-jsdoc/node_modules/glob/LICENSE.md
camofox-browser/node_modules/encodeurl/README.md
camofox-browser/node_modules/express/Readme.md
camofox-browser/node_modules/ini/README.md
repos/platform-gitops/tenants/pidoka/prod/mautic/README.md
camofox-browser/node_modules/xml2js/README.md
camofox-browser/node_modules/express/History.md
repos/platform-gitops/tenants/pidoka/prod/zammad/README.md
camofox-browser/node_modules/decompress-response/readme.md
camofox-browser/node_modules/base64-js/README.md
camofox-browser/node_modules/proxy-addr/README.md
camofox-browser/node_modules/proxy-addr/HISTORY.md
camofox-browser/node_modules/playwright-core/lib/tools/cli-client/skill/references/storage-state.md
camofox-browser/node_modules/ansi-regex/readme.md
camofox-browser/node_modules/playwright-core/lib/tools/cli-client/skill/references/tracing.md
camofox-browser/node_modules/playwright-core/lib/tools/cli-client/skill/references/test-generation.md
camofox-browser/node_modules/playwright-core/lib/tools/cli-client/skill/references/element-attributes.md
camofox-browser/node_modules/playwright-core/lib/tools/cli-client/skill/references/running-code.md
camofox-browser/node_modules/playwright-core/lib/tools/cli-client/skill/references/session-management.md
camofox-browser/node_modules/playwright-core/lib/tools/cli-client/skill/references/playwright-tests.md
camofox-browser/node_modules/playwright-core/lib/tools/cli-client/skill/references/spec-driven-testing.md
camofox-browser/node_modules/playwright-core/lib/tools/cli-client/skill/references/request-mocking.md
camofox-browser/node_modules/playwright-core/lib/tools/cli-client/skill/references/video-recording.md
camofox-browser/node_modules/playwright-core/lib/tools/cli-client/skill/SKILL.md
camofox-browser/node_modules/glob/README.md
camofox-browser/node_modules/glob/LICENSE.md
camofox-browser/node_modules/dot-prop/readme.md
camofox-browser/node_modules/playwright-core/lib/tools/trace/SKILL.md
camofox-browser/node_modules/send/README.md
repos/platform-gitops/platform/infisical-dmz/README.md
camofox-browser/node_modules/send/node_modules/ms/license.md
camofox-browser/node_modules/send/node_modules/ms/readme.md
camofox-browser/node_modules/send/HISTORY.md
camofox-browser/node_modules/send/SECURITY.md
camofox-browser/node_modules/is-standalone-pwa/README.md
camofox-browser/node_modules/shebang-regex/readme.md
camofox-browser/node_modules/swagger-jsdoc/docusaurus/versioned_docs/version-7.x/intro.md
camofox-browser/node_modules/etag/README.md
camofox-browser/node_modules/etag/HISTORY.md
camofox-browser/node_modules/es-object-atoms/README.md
camofox-browser/node_modules/swagger-jsdoc/docusaurus/versioned_docs/version-7.x/quick-start/first-steps.md
camofox-browser/node_modules/es-object-atoms/CHANGELOG.md
camofox-browser/node_modules/rc/README.md
camofox-browser/node_modules/get-proto/README.md
camofox-browser/node_modules/swagger-jsdoc/docusaurus/versioned_docs/version-7.x/Contributing/typescript.md
camofox-browser/node_modules/swagger-jsdoc/docusaurus/versioned_docs/version-7.x/Contributing/fundamental-concepts.md
camofox-browser/node_modules/swagger-jsdoc/docusaurus/versioned_docs/version-7.x/Contributing/project-goals.md
camofox-browser/node_modules/swagger-jsdoc/docusaurus/versioned_docs/version-7.x/Contributing/reporting-issues.md
camofox-browser/node_modules/get-proto/CHANGELOG.md
camofox-browser/node_modules/swagger-jsdoc/docusaurus/versioned_docs/version-6.x/intro.md
camofox-browser/node_modules/call-bind-apply-helpers/README.md
camofox-browser/node_modules/swagger-jsdoc/docusaurus/versioned_docs/version-6.x/Quick Start/first-steps.md
camofox-browser/node_modules/swagger-jsdoc/docusaurus/versioned_docs/version-6.x/Quick Start/cli.md
camofox-browser/node_modules/call-bind-apply-helpers/CHANGELOG.md
camofox-browser/node_modules/rc/node_modules/strip-json-comments/readme.md
camofox-browser/node_modules/swagger-jsdoc/docusaurus/versioned_docs/version-6.x/Contributing/typescript.md
camofox-browser/node_modules/swagger-jsdoc/docusaurus/versioned_docs/version-6.x/Contributing/fundamental-concepts.md
camofox-browser/node_modules/swagger-jsdoc/docusaurus/versioned_docs/version-6.x/Contributing/project-goals.md
camofox-browser/node_modules/swagger-jsdoc/docusaurus/versioned_docs/version-6.x/Contributing/reporting-issues.md
camofox-browser/node_modules/chownr/README.md
camofox-browser/node_modules/node-releases/README.md
camofox-browser/node_modules/swagger-jsdoc/docusaurus/versioned_docs/version-5.x/installation.md
camofox-browser/node_modules/swagger-jsdoc/docusaurus/versioned_docs/version-5.x/intro.md
camofox-browser/node_modules/minipass/README.md
camofox-browser/node_modules/minipass/LICENSE.md
camofox-browser/node_modules/swagger-jsdoc/docusaurus/versioned_docs/version-5.x/cli.md
camofox-browser/node_modules/impit/README.md
camofox-browser/node_modules/impit/CHANGELOG.md
camofox-browser/node_modules/vary/README.md
camofox-browser/node_modules/vary/HISTORY.md
camofox-browser/node_modules/swagger-jsdoc/docusaurus/versioned_docs/version-5.x/Contributing/report-issues.md
camofox-browser/node_modules/swagger-jsdoc/docusaurus/versioned_docs/version-5.x/Contributing/contributing.md
camofox-browser/node_modules/swagger-jsdoc/docusaurus/versioned_docs/version-5.x/fundamental-concepts.md
camofox-browser/node_modules/swagger-jsdoc/docusaurus/versioned_docs/version-5.x/validation.md
camofox-browser/node_modules/swagger-jsdoc/docusaurus/versioned_docs/version-5.x/quick-start.md
camofox-browser/node_modules/tiny-lru/README.md
camofox-browser/node_modules/media-typer/README.md
camofox-browser/node_modules/media-typer/HISTORY.md
camofox-browser/node_modules/swagger-jsdoc/docusaurus/src/pages/markdown-page.md
camofox-browser/node_modules/camoufox-js/README.md
camofox-browser/node_modules/camoufox-js/LICENSE.md
camofox-browser/node_modules/brace-expansion/README.md
camofox-browser/node_modules/swagger-jsdoc/CHANGELOG.md
camofox-browser/node_modules/end-of-stream/README.md
camofox-browser/node_modules/iconv-lite/README.md
camofox-browser/node_modules/iconv-lite/Changelog.md
camofox-browser/node_modules/napi-build-utils/README.md
camofox-browser/node_modules/napi-build-utils/index.md
camofox-browser/node_modules/language-tags/README.md
camofox-browser/node_modules/ua-is-frozen/README.md
camofox-browser/node_modules/string_decoder/README.md
camofox-browser/node_modules/jackspeak/README.md
camofox-browser/node_modules/jackspeak/LICENSE.md
camofox-browser/node_modules/type-is/README.md
camofox-browser/node_modules/type-is/HISTORY.md
camofox-browser/node_modules/sax/README.md
camofox-browser/node_modules/sax/LICENSE.md
camofox-browser/node_modules/fingerprint-generator/README.md
camofox-browser/node_modules/fingerprint-generator/LICENSE.md
camofox-browser/node_modules/get-intrinsic/README.md
camofox-browser/node_modules/bintrees/README.md
camofox-browser/node_modules/foreground-child/README.md
camofox-browser/node_modules/range-parser/README.md
camofox-browser/node_modules/get-intrinsic/CHANGELOG.md
camofox-browser/node_modules/range-parser/HISTORY.md
camofox-browser/node_modules/gopd/README.md
camofox-browser/node_modules/dunder-proto/README.md
camofox-browser/node_modules/gopd/CHANGELOG.md
camofox-browser/node_modules/impit-linux-x64-gnu/README.md
camofox-browser/node_modules/finalhandler/README.md
camofox-browser/node_modules/finalhandler/HISTORY.md
camofox-browser/node_modules/finalhandler/SECURITY.md
camofox-browser/node_modules/which/README.md
camofox-browser/node_modules/dunder-proto/CHANGELOG.md
camofox-browser/node_modules/foreground-child/node_modules/signal-exit/README.md
camofox-browser/node_modules/which/CHANGELOG.md
camofox-browser/node_modules/on-finished/README.md
camofox-browser/node_modules/on-finished/HISTORY.md
camofox-browser/node_modules/balanced-match/README.md
camofox-browser/node_modules/balanced-match/LICENSE.md
camofox-browser/node_modules/doctrine/README.md
camofox-browser/node_modules/@opentelemetry/api/README.md
camofox-browser/node_modules/merge-descriptors/README.md
camofox-browser/node_modules/merge-descriptors/HISTORY.md
camofox-browser/node_modules/doctrine/CHANGELOG.md
camofox-browser/node_modu
...(truncated)...

/bin/bash -lc "sed -n '1,220p' inventory/backup-folder-structure-dmz-backup-20260619.md"

/bin/bash -lc "sed -n '1,220p' inventory/backup-folder-structure-dmz-backup-20260619.md"

# Backup-Ordnerstruktur — dmz-backup + Synology Pull

Stand: 2026-06-19

## Zielbild

- `dmz-backup` (`192.168.20.90`) ist der DMZ-Staging-Host.
- K3s/DMZ-Cluster darf Backups nach `dmz-backup` pushen.
- Synology/DiskStation pullt von `dmz-backup` und bleibt Backup-Master.
- Direkte Schreibrechte aus der DMZ auf Synology werden vermieden.

## Empfehlung

Ja: diskstation-bot soll die Struktur grundsaetzlich nach Platform/Tenants uebernehmen.

Nicht nur nach Kubernetes-Namespace sortieren, weil aktuelle Namespaces historisch gewachsen sind (`hf-*`, `pidoka-hf-*`, `homarr`, `zammad`). Besser ist:

1. fachliche Grenze: `platform` oder `tenants/<tenant>`
2. Umgebung: `prod`, `stage`, `dev`
3. Anwendung/Komponente
4. Backup-Art: `db-dumps`, `files`, `volume-backups`, `manifests`, `metadata`, `restore-tests`

## Vorgeschlagener Root

Auf `dmz-backup`:

```text
/srv/dmz-backup/k3s-dmz/
```

Auf Synology als Pull-Ziel sinngemaess:

```text
K3S_DMZ_Backups/
```

## Ordnerstruktur

```text
k3s-dmz/
  README.md
  _incoming/
    quarantine/
    incomplete/
  _indexes/
    latest.json
    backup-catalog.jsonl
    checksums/
  _restore-tests/
    platform/
    tenants/

  platform/
    argocd/
      prod/
        manifests/
        app-exports/
        restore-tests/
    infisical/
      prod/
        db-dumps/
        app-exports/
        restore-tests/
    external-secrets/
      prod/
        manifests/
        crd-inventory/
    longhorn/
      prod/
        volume-backups/
        metadata/
        restore-tests/
    nginx-proxy-manager/
      prod/
        database/
        letsencrypt/
        config/
        restore-tests/
    cluster/
      prod/
        manifests/
        inventory/
        events/
        node-metadata/

  tenants/
    pidoka/
      prod/
        n8n/
          db-dumps/
          files/
          manifests/
          restore-tests/
        matomo/
          db-dumps/
          files/
          manifests/
          restore-tests/
        mautic/
          db-dumps/
          files/
          redis/
          manifests/
          restore-tests/
        suitecrm/
          db-dumps/
          files/
          manifests/
          restore-tests/
        zammad/
          db-dumps/
          files/
          manifests/
          restore-tests/
        homarr/
          db-dumps/
          files/
          manifests/
          restore-tests/

    cattaro/
      stage/
        guestbook/
          manifests/
          restore-tests/
      prod/
        _reserved/

  legacy/
    hf-matomo/
      prod/
        db-dumps/
        files/
        manifests/
        restore-tests/
    default-my-release/
      prod/
        db-dumps/
        manifests/
        restore-tests/
    keycloak/
      prod/
        db-dumps/
        manifests/
        restore-tests/
```

## Dateinamensschema

Empfohlen:

```text
<UTC timestamp>__<cluster>__<tenant-or-platform>__<app>__<kind>__<scope>.<ext>
```

Beispiele:

```text
20260619T193000Z__dmz-k3s__pidoka__matomo__db-dump__mariadb.sql.gz
20260619T193000Z__dmz-k3s__pidoka__zammad__files__uploads.tar.zst
20260619T193000Z__dmz-k3s__platform__argocd__manifests__apps.yaml.gz
20260619T193000Z__dmz-k3s__platform__longhorn__metadata__volumes.json.gz
```

Zu jedem Backup gehoert eine kleine Metadatendatei:

```text
<backup-file>.meta.json
<backup-file>.sha256
```

## Minimale Metadaten

```json
{
  "created_at_utc": "2026-06-19T19:30:00Z",
  "cluster": "dmz-k3s",
  "source_host": "192.168.20.40",
  "backup_staging_host": "dmz-backup",
  "backup_staging_ip": "192.168.20.90",
  "category": "tenant",
  "tenant": "pidoka",
  "environment": "prod",
  "app": "matomo",
  "namespace": "pidoka-hf-matomo",
  "kind": "db-dump",
  "contains_secrets": false,
  "encryption": "tbd",
  "retention_class": "prod-standard",
  "restore_test_required": true
}
```

## Retention-Klassen

Vorschlag fuer diskstation-bot:

- `prod-critical`: taeglich 30 Tage, woechentlich 12 Wochen, monatlich 12 Monate
- `prod-standard`: taeglich 14 Tage, woechentlich 8 Wochen, monatlich 6 Monate
- `platform-critical`: taeglich 30 Tage, woechentlich 12 Wochen, monatlich 12 Monate
- `legacy-cleanup`: taeglich 7 Tage, woechentlich 4 Wochen, danach nur falls explizit markiert
- `pre-change`: mindestens 30 Tage oder bis Change/Restore-Test abgeschlossen ist
- `restore-test-artifacts`: 14 Tage, sofern kein Incident-Beleg

## Security-Leitlinien

- Cluster bekommt nur Schreibrechte auf definierte Zielpfade auf `dmz-backup`.
- Synology bekommt Pull-Rechte von `dmz-backup`, idealerweise read-only.
- Kein breit nutzbarer Synology-Key im Cluster.
- Backups mit Secret-/Credential-Inhalt markieren und bevorzugt vor Ablage verschluesseln.
- Synology-Pull initial ohne destruktives `--delete`; Loesch-/Retention-Entscheidung liegt beim Backup-Master.
- Restore-Test ist Pflicht: Backup ohne getesteten Restore gilt nicht als belastbar.

## Mapping aktueller DMZ-Workloads

- Platform:
  - `platform-infisical` -> `platform/infisical/prod/`
  - `argocd` / IT15-ArgoCD Exporte -> `platform/argocd/prod/`
  - `external-secrets` -> `platform/external-secrets/prod/`
  - Longhorn -> `platform/longhorn/prod/`

/bin/bash -lc "sed -n '1,120p' .hob-cattaro-creds 2>/dev/null | sed -E 's/=(.*)"'$/=[REDACTED]/'"' || true"

/bin/bash -lc "sed -n '1,120p' .hob-cattaro-creds 2>/dev/null | sed -E 's/=(.*)"'$/=[REDACTED]/'"' || true"

# HOB Cattaro Infisical Credentials
# Tenant: Cattaro GmbH
HOB_CATTARO_INFISICAL_URL=[REDACTED]
HOB_CATTARO_PROJECT_ID=[REDACTED]
HOB_CATTARO_CLIENT_ID=[REDACTED]
HOB_CATTARO_CLIENT_SECRET=[REDACTED]
HOB_CATTARO_PATH=[REDACTED]

fatal: not a git repository (or any parent up to mount point /home/node)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

/bin/bash -lc "find . -maxdepth 3 -type f \\( -name '*infisical*' -o -name '*backup*' -o -name '*restore*' \\) -printf '%p\\n' | sort"

/bin/bash -lc "find . -maxdepth 3 -type f \\( -name '*infisical*' -o -name '*backup*' -o -name '*restore*' \\) -printf '%p\\n' | sort"

./audits/argocd-repo-20260615/n8n-infisical-precheck.txt
./backup-plans/longhorn-restore-return-path-addendum.md
./backup-plans/longhorn-restore-test-kube-review.md
./backups/homarr-pilot-20260620T073545Z/restore-notes.md
./backups/homarr-pilot-20260620T104142Z/restore-notes.md
./bind-infisical-wsl-eth0.sh
./check-infisical-after-restore.sh
./check-longhorn-backup-target.sh
./check-wsl-network-infisical.sh
./diagnose-infisical-resets.sh
./fix-infisical-it15-bind.sh
./install-socat-infisical-proxy.sh
./inventory/backup-folder-structure-dmz-backup-20260619.md
./inventory/backup-strategy-draft.md
./longhorn-test-backup-retry.sh
./longhorn-test-backup.sh
./read-backup-status.sh
./real-restore-infisical.sh
./repair-infisical-network.sh
./run-final-backup-restore-test.sh
./set-longhorn-backup-target.sh
./setup-infisical-wsl-proxy.sh
./slack-manifest.yml.backup.app-home.20260607T125518Z
./status-infisical-quick.ps1
./win-check-nas-backup.ps1
./wsl-configure-infisical.ps1
./wsl-current-infisical-status-now.ps1
./wsl-current-infisical-status-wait.ps1
./wsl-current-infisical-status.ps1
./wsl-fix-infisical-key.ps1
./wsl-fix-restore-test-script.ps1
./wsl-infisical-backup-restore-test.ps1
./wsl-infisical-health.ps1
./wsl-infisical-status.ps1
./wsl-install-infisical-ops-scripts.ps1
./wsl-nas-backup-direct-restore-test.ps1
./wsl-nas-backup-restore-test.ps1
./wsl-nas-backup-restore-test2.ps1
./wsl-post-reboot-infisical-verify.ps1
./wsl-prepare-infisical.ps1
./wsl-real-restore-infisical.ps1
./wsl-start-infisical.ps1
./wsl-test-infisical-ops-scripts.ps1

/bin/bash -lc "set -euo pipefail
json_get() { python3 -c 'import json,sys; data=json.load(sys.stdin); path=sys.argv[1].split(\".\"); cur=data
for p in path:
    if isinstance(cur, dict) and p in cur: cur=cur[p]
    else: print(\"\"); sys.exit(0)
print(cur if cur is not None else \"\")' \""'$1"; }
check_secret_names() {
  local url="$1" token="***" project="$3" envslug="$4" path="$5" label="$6"
  python3 - "$url" "$token" "$project" "$envslug" "$path" "$label" <<'"'PY'
import json, sys, urllib.parse, urllib.request
url, token, project, envslug, path, label = sys.argv[1:]
qs = urllib.parse.urlencode({'workspaceId': project, 'environment': envslug, 'secretPath': path})
req = urllib.request.Request(f'{url}/api/v3/secrets/raw?{qs}', headers={'Authorization': f'Bearer {token}'})
try:
    data = json.load(urllib.request.urlopen(req, timeout=10))
except Exception as e:
    print(f'{label}: SECRET_LIST=BLOCKED ({type(e).__name__})')
    sys.exit(0)
items = data.get('secrets') or data.get('secret') or data.get('data') or []
names = []
if isinstance(items, list):
    for item in items:
        if isinstance(item, dict):
            n = item.get('secretKey') or item.get('key') or item.get('name')
            if n: names.append(n)
print(f'{label}: SECRET_NAMES=' + ','.join(sorted(names)) if names else f'{label}: SECRET_NAMES=(none-or-unlisted)')
PY
}
# Cattaro/Backup_Restore via runtime creds
CATTARO_URL='http://192.168.1.121:8088'
BACKUP_PROJECT='ba0a79d9-32fe-474c-be2f-88384a3d9749'
LOGIN_JSON="'$(curl -fsS -X POST "$CATTARO_URL/api/v1/auth/universal-auth/login" -H '"'Content-Type: application/json' -d \"{\\\"clientId\\\":\\\""'$INFISICAL_CLIENT_ID'"\\\",\\\"clientSecret\\\":\\\""'$INFISICAL_AUTH_KEY'"\\\"}\")
TOKEN="***"'%s' \""'$LOGIN_JSON" | json_get accessToken); [ -n "$TOKEN" ] || TOKEN=*** '"'%s' \""'$LOGIN_JSON" | json_get token)
echo '"'CATTARO_LOGIN=OK'
echo 'CATTARO_PROJECT=Backup_Restore'
echo 'CATTARO_PROJECT_ID=ba0a79d9-32fe-474c-be2f-88384a3d9749'
check_secret_names \""'$CATTARO_URL" "$TOKEN" "$BACKUP_PROJECT" prod / '"'CATTARO_BACKUP_RESTORE_ROOT'
# HOB-Cattaro via .hob-cattaro-creds
set -a
. ./.hob-cattaro-creds
set +a
HOB_LOGIN="'$(curl -fsS -X POST "$HOB_CATTARO_INFISICAL_URL/api/v1/auth/universal-auth/login" -H '"'Content-Type: application/json' -d \"{\\\"clientId\\\":\\\""'$HOB_CATTARO_CLIENT_ID'"\\\",\\\"clientSecret\\\":\\\""'$HOB_CATTARO_CLIENT_SECRET'"\\\"}\")
HOB_TOKEN="'$(printf '"'%s' \""'$HOB_LOGIN" | json_get accessToken); [ -n "$HOB_TOKEN" ] || HOB_TOKEN=*** '"'%s' \""'$HOB_LOGIN" | json_get token)
echo '"'HOB_CATTARO_LOGIN=OK'
echo \"HOB_CATTARO_PATH="'$HOB_CATTARO_PATH"
check_secret_names "$HOB_CATTARO_INFISICAL_URL" "$HOB_TOKEN" "$HOB_CATTARO_PROJECT_ID" prod "$HOB_CATTARO_PATH" '"'HOB_CATTARO_KUBE_BOT_PATH'"

/bin/bash -lc "set -euo pipefail
json_get() { python3 -c 'import json,sys; data=json.load(sys.stdin); path=sys.argv[1].split(\".\"); cur=data
for p in path:
    if isinstance(cur, dict) and p in cur: cur=cur[p]
    else: print(\"\"); sys.exit(0)
print(cur if cur is not None else \"\")' \""'$1"; }
check_secret_names() {
  local url="$1" token="***" project="$3" envslug="$4" path="$5" label="$6"
  python3 - "$url" "$token" "$project" "$envslug" "$path" "$label" <<'"'PY'
import json, sys, urllib.parse, urllib.request
url, token, project, envslug, path, label = sys.argv[1:]
qs = urllib.parse.urlencode({'workspaceId': project, 'environment': envslug, 'secretPath': path})
req = urllib.request.Request(f'{url}/api/v3/secrets/raw?{qs}', headers={'Authorization': f'Bearer {token}'})
try:
    data = json.load(urllib.request.urlopen(req, timeout=10))
except Exception as e:
    print(f'{label}: SECRET_LIST=BLOCKED ({type(e).__name__})')
    sys.exit(0)
items = data.get('secrets') or data.get('secret') or data.get('data') or []
names = []
if isinstance(items, list):
    for item in items:
        if isinstance(item, dict):
            n = item.get('secretKey') or item.get('key') or item.get('name')
            if n: names.append(n)
print(f'{label}: SECRET_NAMES=' + ','.join(sorted(names)) if names else f'{label}: SECRET_NAMES=(none-or-unlisted)')
PY
}
# Cattaro/Backup_Restore via runtime creds
CATTARO_URL='http://192.168.1.121:8088'
BACKUP_PROJECT='ba0a79d9-32fe-474c-be2f-88384a3d9749'
LOGIN_JSON="'$(curl -fsS -X POST "$CATTARO_URL/api/v1/auth/universal-auth/login" -H '"'Content-Type: application/json' -d \"{\\\"clientId\\\":\\\""'$INFISICAL_CLIENT_ID'"\\\",\\\"clientSecret\\\":\\\""'$INFISICAL_AUTH_KEY'"\\\"}\")
TOKEN="***"'%s' \""'$LOGIN_JSON" | json_get accessToken); [ -n "$TOKEN" ] || TOKEN=*** '"'%s' \""'$LOGIN_JSON" | json_get token)
echo '"'CATTARO_LOGIN=OK'
echo 'CATTARO_PROJECT=Backup_Restore'
echo 'CATTARO_PROJECT_ID=ba0a79d9-32fe-474c-be2f-88384a3d9749'
check_secret_names \""'$CATTARO_URL" "$TOKEN" "$BACKUP_PROJECT" prod / '"'CATTARO_BACKUP_RESTORE_ROOT'
# HOB-Cattaro via .hob-cattaro-creds
set -a
. ./.hob-cattaro-creds
set +a
HOB_LOGIN="'$(curl -fsS -X POST "$HOB_CATTARO_INFISICAL_URL/api/v1/auth/universal-auth/login" -H '"'Content-Type: application/json' -d \"{\\\"clientId\\\":\\\""'$HOB_CATTARO_CLIENT_ID'"\\\",\\\"clientSecret\\\":\\\""'$HOB_CATTARO_CLIENT_SECRET'"\\\"}\")
HOB_TOKEN="'$(printf '"'%s' \""'$HOB_LOGIN" | json_get accessToken); [ -n "$HOB_TOKEN" ] || HOB_TOKEN=*** '"'%s' \""'$HOB_LOGIN" | json_get token)
echo '"'HOB_CATTARO_LOGIN=OK'
echo \"HOB_CATTARO_PATH="'$HOB_CATTARO_PATH"
check_secret_names "$HOB_CATTARO_INFISICAL_URL" "$HOB_TOKEN" "$HOB_CATTARO_PROJECT_ID" prod "$HOB_CATTARO_PATH" '"'HOB_CATTARO_KUBE_BOT_PATH'"

/bin/bash -lc "sed -n '125,200p' INFRASTRUCTURE.md"

/bin/bash -lc "sed -n '125,200p' INFRASTRUCTURE.md"

/bin/bash -lc "sed -n '90,180p' inventory/backup-folder-structure-dmz-backup-20260619.md"

/bin/bash -lc "sed -n '90,180p' inventory/backup-folder-structure-dmz-backup-20260619.md"

db-dumps/
          files/
          manifests/
          restore-staging/
        matomo/
          db-dumps/
          files/
          manifests/
          restore-tests/
        mautic/
          db-dumps/
          files/
          redis/
          manifests/
          restore-tests/
        suitecrm/
          db-dumps/
          files/
          manifests/
          restore-tests/
        zammad/
          db-dumps/
          files/
          manifests/
          restore-tests/
        homarr/
          db-dumps/
          files/
          manifests/
          restore-tests/

    cattaro/
      stage/
        guestbook/
          manifests/
          restore-tests/
      prod/
        _reserved/

  legacy/
    hf-matomo/
      prod/
        db-dumps/
        files/
        manifests/
        restore-tests/
    default-my-release/
      prod/
        db-dumps/
        manifests/
        restore-tests/
    keycloak/
      prod/
        db-dumps/
        manifests/
        restore-tests/
```

## Dateinamensschema

Empfohlen:

```text
<UTC timestamp>__<cluster>__<tenant-or-platform>__<app>__<kind>__<scope>.<ext>
```

Beispiele:

```text
20260619T193000Z__dmz-k3s__pidoka__matomo__db-dump__mariadb.sql.gz
20260619T193000Z__dmz-k3s__pidoka__zammad__files__uploads.tar.zst
20260619T193000Z__dmz-k3s__platform__argocd__manifests__apps.yaml.gz
20260619T193000Z__dmz-k3s__platform__longhorn__metadata__volumes.json.gz
```

Zu jedem Backup gehoert eine kleine Metadatendatei:

```text
<backup-file>.meta.json
<backup-file>.sha256
```

## Minimale Metadaten

```json
{
  "created_at_utc": "2026-06-19T19:30:00Z",
  "cluster": "dmz-k3s",
  "source_host": "192.168.20.40",
  "backup_staging_host": "dmz-backup",
  "backup_staging_ip": "192.168.20.90",

- K3s/DMZ-Cluster pusht Backup-Dateien nach `dmz-backup`.
  - Synology/DiskStation pullt von `dmz-backup` und bleibt Backup-Master.
  - Direkte Schreibrechte aus der DMZ auf Synology werden vermieden.
- Vorgeschlagener Staging-Root: `/srv/dmz-backup/k3s-dmz/`.
- Restore-Staging-Root: `/srv/dmz-backup/k3s-dmz/_restore-staging/`.
- Restore-Richtung: DiskStation -> `dmz-backup` -> K3s/kube-bot. K3s greift nicht direkt auf DiskStation zu.
- Ordnerstruktur-Entwurf: `/home/node/.openclaw/workspace/inventory/backup-folder-structure-dmz-backup-20260619.md`.
- Homarr Phase-1 Backup-Push verifiziert am 2026-06-20:
  - Zielpfad: `/srv/dmz-backup/k3s-dmz/tenants/pidoka/prod/homarr/`.
  - Artefakt: `20260620T104142Z__dmz-k3s__pidoka__homarr__metadata__pilot.tar.gz.age`.
  - Verschluesselte SHA256: `b90154a732f756a10de218f30d28fa9afd8bd15644781aa207011188ca4a5f19`.
  - Verschluesselung: `age` v1.3.1, Recipient/Private-Key nur als Infisical-Secretreferenzen in `Backup_Restore/prod`; kein Private Key auf `dmz-backup`.
- Homarr Restore-Transport-Pilot verifiziert am 2026-06-21:
  - Restore-Staging-Pfad: `/srv/dmz-backup/k3s-dmz/_restore-staging/pidoka/prod/homarr/20260621T1733Z-homarr-pilot-restore/`.
  - `diskstation-bot` legte `.age` und `.sha256` aus der DiskStation-Kopie nach `dmz-backup`; kube-bot verifizierte read-only ab `dmz-backup`.
  - Decrypt, Archiv-Checksum und Kubernetes Server-Dry-Run fuer normale K8s-Objekte waren erfolgreich.
  - Grenze: Artefakt ist nur `metadata-manifest-pilot`, kein vollstaendiger App-Restore, keine PVC-Dateiinhalte, keine Longhorn-Backupdaten.
- Änderungen an Backup-Konfigurationen nur nach explizitem OK von Mislav.

### Synology DiskStation

- Rolle: Backup-Ziel / Backup-System.
- IP: `192.168.1.40`.
- Operativer Bot/User: `diskstation-bot`.
- Gemeinsamer Share fuer Bot-Artefakte: `KI_Agenten`.
- Zielrolle: Backup-Master, der Backups von `dmz-backup` (`192.168.20.90`) pullt.
- Backup-Pfad fuer K3s-DMZ: `/volume1/KI_Agenten/backups/k3s-dmz/`.
- Homarr-Pilot-Ablage: `/volume1/KI_Agenten/backups/k3s-dmz/tenants/pidoka/prod/homarr/`.
- Restore-Staging-Pfad: `/volume1/KI_Agenten/backups/k3s-dmz/_restore-staging/pidoka/prod/homarr/`.
- Status/Logs: `/volume1/KI_Agenten/backups/k3s-dmz/status/` und `/volume1/KI_Agenten/backups/k3s-dmz/logs/`.
- Konkrete Synology Pull-Jobs, Retention, Snapshots/Immutability und Restore-Tests sind mit `diskstation-bot` abzugleichen.
- Änderungen an Backup-Konfigurationen nur nach explizitem OK von Mislav.

### Infisical Backup-/Restore-Secret-Merker

Stand: 2026-06-21. Keine Secretwerte dokumentieren.

- Cattaro Infisical/Intranet:
  - URL: `http://192.168.1.121:8088`.
  - Projekt: `Backup_Restore`.
  - Project-ID: `ba0a79d9-32fe-474c-be2f-88384a3d9749`.
  - Environment: `prod`.
  - Verifizierte Secret-Namensgruppen: `DMZ_BACKUP_*`, `DISKSTATION_*`, `BACKUP_HOMARR_AGE_*`.
- HOB Cattaro:
  - Lokale Credential-Datei fuer Bootstrap: `/home/node/.openclaw/workspace/.hob-cattaro-creds`.
  - kube-bot Pfad: `/kube-bot`.
  - Verifizierte Bridge-Secrets: `CATTARO_INFISICAL_URL`, `CATTARO_INFISICAL_CLIENT_ID`, `CATTARO_INFISICAL_CLIENT_SECRET`, `CATTARO_INFISICAL_PROJECT_ID`.
- Ziel nach naechstem Deployment: HOB-Cattaro injiziert die `CATTARO_INFISICAL_*` Runtime-Variablen; darueber kann kube-bot wieder auf die tenant-spezifische Cattaro-Infisical-Instanz zugreifen.

### SSH-Zugangsmerker — kube-bot/OpenClaw Runtime

Stand: 2026-06-15 13:00 UTC. Diese Merker dienen dazu, eingerichtete Diagnosezugänge nicht zu vergessen. Keine Private-Key-Inhalte dokumentieren.

- OpenClaw Runtime läuft nicht direkt auf den Zielsystemen, sondern aktuell als User `node` im Container/Host `0e9d9d0b8619`.
- Lokaler SSH-Key-Pfad in der Runtime: `/home/node/.ssh/kube-bot-it15-ed25519`.
- DMZ K3s Control-Plane:
  - Ziel: `kube-bot@192.168.20.40`
  - Hostnamen: `mbrkanov-Mini-IT13` / `mbrkanov-mini-it13`
  - Status: SSH funktioniert, sudo NOPASSWD funktioniert.
  - Kubernetes read-only Beispiel: `ssh -i ~/.ssh/kube-bot-it15-ed25519 kube-bot@192.168.20.40 'sudo kubectl --kubeconfig=<redacted> get nodes -o wide'`
  - Hinweis: `sudo KUBECONFIG=... kubectl` funktioniert nicht, weil sudo das Setzen der Env-Variable blockiert. Stattdessen `sudo kubectl --kubeconfig=<redacted> ...` verwenden.
- DMZ K3s Worker:
  - Ziel: `kube-bot@192.168.20.45`
  - Hostnamen: `Mini-IT13-02` / `mini-it13-02`
  - Status: als bekanntes Ziel dokumentiert; Zugriff bei Bedarf erneut testen.
- IT15 Management-Node:
  - Ziel laut älteren Notizen: `Misla@192.168.1.121`
  - Status am 2026-06-15 12:56 UTC: SSH von aktueller Runtime blockiert (`Permission denied`).
  - Früherer Zugriff war dokumentiert, aber aktueller nutzbarer Key/User muss neu bestätigt oder eingerichtet werden.
- Betriebsregel bleibt: SSH primär read-only Diagnose; Änderungen nur nach explizitem OK von Mislav.

### IT15 SSH-Korrektur — 2026-06-15 14:15 UTC

- Korrektes SSH-Ziel: `Misla@192.168.1.121` mit Key `/home/node/.ssh/kube-bot-it15-ed25519`.
- IT15 ist Windows; K3s/ArgoCD laufen in WSL `Ubuntu`.

/bin/bash -lc "sed -n '1,260p' inventory/backup-restore-artifact-classes-20260621.md"

/bin/bash -lc "sed -n '1,260p' inventory/backup-restore-artifact-classes-20260621.md"

# Backup-/Restore-Artefaktklassen — kube-bot Bewertung

Stand: 2026-06-21

## Ziel

Nicht jedes Cluster-Artefakt sollte gleich gesichert werden. Der Homarr-Pilot hat Transport, Verschluesselung, Checksums und Manifest-Dry-Run bewiesen, war aber bewusst kein vollstaendiger App-Restore.

## Klassen

### Klasse A — Metadaten/Manifeste

Geeignet fuer:
- Kubernetes Manifeste
- Status-/Inventar-Snapshots
- GitOps-Exporte
- Secret-Referenzen ohne Werte
- PV/PVC/Longhorn-Metadaten

Empfohlene Methode:
- tar.gz
- clientseitig mit `age` verschluesseln, wenn sensible Metadaten enthalten sind
- `.sha256`
- Ablage ueber `dmz-backup` nach DiskStation

Restore-Wert:
- gut fuer Analyse, Vergleich, Rebuild-Planung und Server-Dry-Run
- nicht ausreichend fuer App-Datenrestore

Homarr-Pilot 2026-06-20/21 gehoert in diese Klasse.

### Klasse B — App-Dateien / PVC-Payload

Geeignet fuer:
- Upload-Verzeichnisse
- App-Konfiguration auf PVC
- kleinere Dateibestaende
- exportierbare App-Daten

Empfohlene Methode:
- bei kleinen Daten: tar.zst/tar.gz + `age`
- bei groesseren oder inkrementellen Daten: `rsync` in ein staging directory, danach optional Paket/Checksum/Encryption
- bei regelmaessigen Backups: Restic evaluieren, weil es verschluesselt, inkrementell und pruefbar ist

Restore-Wert:
- echter App-Restore moeglich, wenn Manifeste und PVC-Zielpfade sauber vorbereitet sind
- Restore immer zuerst in isoliertem Namespace/PVC testen

### Klasse C — Datenbanken

Geeignet fuer:
- MariaDB/MySQL/Postgres Dumps
- App-spezifische Export-Formate

Empfohlene Methode:
- konsistenter Dump aus App-/DB-Kontext
- komprimieren und verschluesseln
- `.sha256` + Metadaten mit DB-Version, Namespace, App, Dump-Zeit

Restore-Wert:
- deutlich besser als rohe PVC-Snapshots fuer DB-Apps
- Restore-Probe muss Import in isolierte DB/App-Instanz enthalten

### Klasse D — Longhorn Volume Backups

Geeignet fuer:
- groessere PVCs
- blockbasierte Volume-Wiederherstellung
- Apps, bei denen Dateiexport nicht praktikabel ist

Empfohlene Methode:
- Longhorn BackupTarget/Backup-Mechanik verwenden
- Backup-Referenzen und Volumen-Metadaten separat dokumentieren
- nicht versuchen, Longhorn Live-CR-Dumps direkt als Restore-Manifeste zu applyen

Restore-Wert:
- echter PVC-/Volume-Restore moeglich
- muss mit neuem PVC/isoliertem Namespace geprobt werden

### Klasse E — S3/Object Store

Geeignet fuer:
- groessere zentrale Backup-Repositories
- Restic/Kopia Repositories
- spaetere standardisierte Retention/Pruning-Policies

Empfohlene Methode:
- erst evaluieren, wenn mehrere Apps echte Payload-Backups brauchen
- Secrets/Passphrases in Infisical, nicht im Cluster-Repo oder auf `dmz-backup`

Restore-Wert:
- gut fuer skalierte Automatisierung
- mehr Betriebs-/Policy-Aufwand als einfacher SSH-Dateitransport

## Automatisierungsideen

Kurzfristig:
- DiskStation-Skript fuer Homarr/K3s-DMZ:
  - Quelle auf `dmz-backup` pruefen
  - `.sha256` pruefen
  - nach DiskStation kopieren
  - DiskStation-SHA pruefen
  - konkrete Quelle auf `dmz-backup` nach freigegebener Regel loeschen
  - Status/Log schreiben
- kube-bot CronJob/Script nur fuer read-only Export + clientseitige Verschluesselung + Push nach `dmz-backup`.

Mittelfristig:
- pro Artefaktklasse eigene Backup-Profile:
  - `metadata`
  - `db-dump`
  - `files`
  - `longhorn-backup`
- Restore-Test-Cron nur fuer nicht-produktive Validierung:
  - Checksums
  - Decrypt
  - Archiv listen
  - Kubernetes Server-Dry-Run

Noch nicht automatisieren:
- produktive Restores
- PVC-Ueberschreiben
- Longhorn Restore in produktive Volumes
- Secrets im Klartext

## Offene Entscheidung

Fuer Homarr als naechsten echten Restore-Test entscheiden:

1. PVC-Dateien exportieren und als `files`-Artefakt sichern.
2. Longhorn Backup fuer die Homarr-PVCs erstellen und Restore in isoliertem Namespace testen.
3. Falls Homarr eine sinnvollere App-/DB-Exportmethode hat, diese bevorzugen.

Kube-bot-Empfehlung: Fuer den naechsten Schritt Longhorn-Backup-Referenz oder PVC-Payload verwenden; Metadata-Pilot nicht weiter als vollstaendiges Restore-Artefakt behandeln.

/bin/bash -lc 'tail -60 memory/2026-06-21.md'

/bin/bash -lc 'tail -60 memory/2026-06-21.md'

- Vigil/Mislav requested self-migration of Cattaro Infisical credentials into House-of-Bots Infisical project `hob-cattaro`.
- Initial script run authenticated after `.hob-cattaro-creds` was fixed, but skipped all four expected keys because `CATTARO_INFISICAL_*` environment variables were not injected in kube-bot's container.
- kube-bot used authoritative existing runtime and documented metadata instead:
  - `CATTARO_INFISICAL_URL` from documented Cattaro Infisical URL `http://192.168.1.121:8088`.
  - `CATTARO_INFISICAL_PROJECT_ID` from documented Backup_Restore project ID `ba0a79d9-32fe-474c-be2f-88384a3d9749`.
  - `CATTARO_INFISICAL_CLIENT_ID` from injected `INFISICAL_CLIENT_ID`.
  - `CATTARO_INFISICAL_CLIENT_SECRET` from injected `INFISICAL_AUTH_KEY`.
- Wrote these four secrets into HOB Cattaro project path `/kube-bot` and verified each secret name exists there.
- No Secret values were printed or stored in notes.
- Follow-up from Mislav: rechecked `CATTARO_INFISICAL_URL` in HOB Cattaro path `/kube-bot`; secret name is present.

## 2026-06-21 17:38 UTC - Homarr restore-staging transport verified

- Mislav gave green light for a restore-process dry run initiated by kube-bot.
- diskstation-bot staged the verified Homarr pilot artifact from DiskStation back to `dmz-backup`.
- kube-bot verified read-only on `dmz-backup` using Infisical `Backup_Restore/prod` `DMZ_BACKUP_*` SSH metadata/key in a temporary file; no Secret values were printed or stored. Temporary files were removed after use.
- Restore staging path verified:
  `/srv/dmz-backup/k3s-dmz/_restore-staging/pidoka/prod/homarr/20260621T1733Z-homarr-pilot-restore/`
- Files present:
  - `20260620T104142Z__dmz-k3s__pidoka__homarr__metadata__pilot.tar.gz.age` — `10694` bytes
  - `20260620T104142Z__dmz-k3s__pidoka__homarr__metadata__pilot.tar.gz.age.sha256` — `136` bytes
- SHA256 verify on `dmz-backup`: `OK`.
- Expected SHA256: `b90154a732f756a10de218f30d28fa9afd8bd15644781aa207011188ca4a5f19`.
- Scope unchanged: no decrypt, no K3s change, no productive restore. Next safe step is an artifact/decrypt/manifest check in an isolated restore workspace, only after explicit OK.

## 2026-06-21 17:46 UTC - Homarr decrypt and restore dry-run probe

- Mislav approved continuing with decrypt and restore rehearsal.
- kube-bot copied the staged Homarr `.age` artifact from `dmz-backup` read-only into a temporary local restore workspace and verified the adjacent `.sha256`: `OK`.
- Decrypt with Infisical `Backup_Restore/prod` `BACKUP_HOMARR_AGE_PRIVATE_KEY` succeeded. The key was written only to a temporary file and removed after use; no Secret values were printed or stored.
- Decrypted archive:
  - size `10494` bytes
  - SHA256 `64802165be001fd7c5dab1a59bf0f97934ce47aa31863fcf0c622d0bfa66db87`
- Archive checksum file `SHA256SUMS` verified all contained files: `OK`.
- `metadata.json` confirms this artifact is only `metadata-manifest-pilot`:
  - no Secret values
  - no PVC filesystem payload
  - no Longhorn snapshot/backup payload
  - no cluster mutation during backup creation
- Kubernetes server dry-run for regular K8s manifests succeeded for ConfigMap, Deployment, PV, PVC, Service and ServiceAccount. Warnings were expected because these are live-object snapshots without last-applied annotations.
- Including Longhorn `Volume` live-object dump in server dry-run failed with a resourceVersion/status conflict. Conclusion: Longhorn live CR dumps are useful as metadata, but not directly applyable restore manifests.
- No real cluster change was performed. Next required improvement before a meaningful restore rehearsal: create a restore artifact that includes either PVC payload/Longhorn backup references or a sanitized restore manifest set for an isolated namespace.

## 2026-06-21 18:00 UTC - Backup/restore setup hardening notes

- Mislav asked both bots to complete the setup after the successful Homarr pilot:
  - diskstation-bot handles direct DiskStation -> `dmz-backup` access.
  - both bots document backup/restore process for their own side.
  - Nextcloud/Collective space will be provided later for shared documentation.
  - both bots should remember both Infisical installations and document architecture/nodes.
  - kube-bot should decide when the Homarr `.age` artifact model is sufficient and when rsync, S3, Restic/Kopia, DB dumps or Longhorn backup are better.
- kube-bot read-only verified Infisical access without printing Secret values:
  - Cattaro/Intranet Infisical `Backup_Restore/prod` login OK; expected secret-name groups present: `DMZ_BACKUP_*`, `DISKSTATION_*`, `BACKUP_HOMARR_AGE_*`.
  - HOB Cattaro `/kube-bot` login OK; expected bridge secrets present: `CATTARO_INFISICAL_URL`, `CATTARO_INFISICAL_CLIENT_ID`, `CATTARO_INFISICAL_CLIENT_SECRET`, `CATTARO_INFISICAL_PROJECT_ID`.
- Updated documentation:
  - `INFRASTRUCTURE.md`: DiskStation IP, `KI_Agenten`, restore direction, restore-staging paths, Infisical memory markers.
  - `inventory/backup-folder-structure-dmz-backup-20260619.md`: `_restore-staging` model, trigger/cleanup rules, dmz-backup deletion rule after DiskStation SHA verify.
  - `inventory/backup-strategy-draft.md`: Homarr pilot result and boundaries.
  - `inventory/backup-restore-artifact-classes-20260621.md`: artifact classes and when to use age tarballs vs rsync/Restic/S3/DB dumps/Longhorn backup.

/bin/bash -lc 'rg -n "restore-staging|_restore-staging|restore-tests" inventory/backup-folder-structure-dmz-backup-20260619.md'

21:4. Backup-Art: `db-dumps`, `files`, `volume-backups`, `manifests`, `metadata`, `restore-tests`
49:  _restore-staging/
58:        restore-staging/
63:        restore-tests/
72:        restore-tests/
78:        restore-tests/
93:          restore-staging/
98:          restore-tests/
104:          restore-tests/
109:          restore-tests/
114:          restore-tests/
119:          restore-tests/
125:          restore-tests/
135:        restore-tests/
140:        restore-tests/
145:        restore-tests/
222:/volume1/KI_Agenten/backups/k3s-dmz/_restore-staging/
232:/srv/dmz-backup/k3s-dmz/_restore-staging/
242:/srv/dmz-backup/k3s-dmz/_restore-staging/pidoka/prod/homarr/20260621T1733Z-homarr-pilot-restore/

/bin/bash -lc "sed -n '45,100p' inventory/backup-folder-structure-dmz-backup-20260619.md"

_indexes/
    latest.json
    backup-catalog.jsonl
    checksums/
  _restore-staging/
    platform/
    tenants/

  platform/
    argocd/
      prod/
        manifests/
        app-exports/
        restore-staging/
    infisical/
      prod/
        db-dumps/
        app-exports/
        restore-tests/
    external-secrets/
      prod/
        manifests/
        crd-inventory/
    longhorn/
      prod/
        volume-backups/
        metadata/
        restore-tests/
    nginx-proxy-manager/
      prod/
        database/
        letsencrypt/
        config/
        restore-tests/
    cluster/
      prod/
        manifests/
        inventory/
        events/
        node-metadata/

  tenants/
    pidoka/
      prod/
        n8n/
          db-dumps/
          files/
          manifests/
          restore-staging/
        matomo/
          db-dumps/
          files/
          manifests/
          restore-tests/
        mautic/
          db-dumps/

/bin/bash -lc 'rg -n "restore-staging|_restore-staging" inventory/backup-folder-structure-dmz-backup-20260619.md && test -s inventory/backup-restore-artifact-classes-20260621.md && echo DOC_CHECK=OK'

49:  _restore-staging/
222:/volume1/KI_Agenten/backups/k3s-dmz/_restore-staging/
232:/srv/dmz-backup/k3s-dmz/_restore-staging/
242:/srv/dmz-backup/k3s-dmz/_restore-staging/pidoka/prod/homarr/20260621T1733Z-homarr-pilot-restore/
DOC_CHECK=OK

message

kube-bot-Seite erledigt, soweit sicher möglich.

*Geprüft:*
• Cattaro Infisical `Backup_Restore/prod`: Login `OK`, erwartete Secret-Namensgruppen vorhanden: `DMZ_BACKUP_*`, `DISKSTATION_*`, `BACKUP_HOMARR_AGE_*`
• HOB Cattaro `/kube-bot`: Login `OK`, Bridge-Secrets vorhanden: `CATTARO_INFISICAL_*`
• Keine Secretwerte ausgegeben oder dokumentiert.

*Dokumentiert/aktualisiert:*
• `INFRASTRUCTURE.md`: Nodes/Architektur, DiskStation `192.168.1.40`, `dmz-backup` `192.168.20.90`, Restore-Richtung, Infisical-Merker
• `inventory/backup-folder-structure-dmz-backup-20260619.md`: `_restore-staging`, Trigger, Cleanup, Löschregel nach DiskStation-SHA
• `inventory/backup-strategy-draft.md`: Homarr-Pilot, Grenzen, Restore-Befund
• `inventory/backup-restore-artifact-classes-20260621.md`: Wann `.age`-Tarball reicht und wann `rsync`, DB-Dump, Longhorn-Backup, Restic/S3 sinnvoller sind
• `memory/2026-06-21.md`: heutiger Stand als Merker

*Bewertung kube-bot:*
Für Metadaten/Manifeste ist `.tar.gz.age` genau richtig. Für echte App-Restores reicht das nicht. Je nach Workload brauchen wir zusätzlich:
• DB-Dumps für Datenbanken
• PVC-/Datei-Backups per `rsync`/tar/Restic für App-Dateien
• Longhorn-Backup-Referenzen für Volume-Restore
• S3/Restic/Kopia erst, wenn wir mehrere Payload-Backups standardisieren wollen

*Offen:*
• `diskstation-bot` richtet DiskStation -> `dmz-backup` Zugriff ein.
• Shared Nextcloud/Collective kommt später, dann können wir die lokalen Konzepte dorthin übertragen.
• Cron/Automation erst nach sauberem Skript- und Restore-Artefakt-Konzept, keine produktiven Restores automatisch.