Output:
Chunk ID: 9f0764
Wall time: 1.5934 seconds
Process exited with code 0
Original token count: 708
Output:
20260617T135148Z
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS   IMAGES           SELECTOR
deployment.apps/matomo-db   1/1     1            1           282d   mariadb      mariadb:latest   app=matomo-db
deployment.apps/matomo      1/1     1            1           282d   matomo       matomo:latest    app=matomo

NAME                             READY   STATUS    RESTARTS   AGE   IP           NODE                 NOMINATED NODE   READINESS GATES
pod/matomo-db-6766699c55-xjg8d   1/1     Running   0          45d   10.42.0.61   mbrkanov-mini-it13   <none>           <none>
pod/matomo-868fb8cdbd-t8kqm      1/1     Running   0          45d   10.42.0.62   mbrkanov-mini-it13   <none>           <none>

NAME                                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE    VOLUMEMODE
persistentvolumeclaim/matomo-pvc      Bound    pvc-f6714be0-39b1-47e6-baa2-2e1ae801d907   1Gi        RWO            longhorn       282d   Filesystem
persistentvolumeclaim/matomo-db-pvc   Bound    pvc-61155551-a7de-4c0c-b558-f86437232148   1Gi        RWO            longhorn       282d   Filesystem

NAME                                       CLASS     HOSTS                   ADDRESS                       PORTS   AGE
ingress.networking.k8s.io/matomo-ingress   traefik   matomo.heckenfels.com   192.168.20.40,192.168.20.45   80      282d
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS   IMAGES                                                                                SELECTOR
deployment.apps/matomo      0/0     0            0           7h6m   matomo       docker.io/library/matomo@sha256:21b8bae018cd51a05f36bc6beac3861bb314d131df690c47bba1648aad2b8d7e    app=matomo
deployment.apps/matomo-db   0/0     0            0           7h6m   mariadb      docker.io/library/mariadb@sha256:e0236fc6386e7eacd9359e59d0a078bd7aa0d18280d36d13061121bedeaee903   app=matomo-db

NAME                                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE    VOLUMEMODE
persistentvolumeclaim/matomo-db-pvc   Bound    pvc-2c086a12-6f3b-494a-b982-e2405f7649af   1Gi        RWO            longhorn       7h6m   Filesystem
persistentvolumeclaim/matomo-pvc      Bound    pvc-e96e1f56-a534-49e2-b2ee-b39c6e523af5   1Gi        RWO            longhorn       7h6m   Filesystem

NAME                                               STORE           REFRESH INTERVAL   STATUS         READY
externalsecret.external-secrets.io/matomo-secret   dmz-infisical   1h                 SecretSynced   True

NAME                                            AGE    STATUS   CAPABILITIES   READY
secretstore.external-secrets.io/dmz-infisical   7h6m   Valid    ReadOnly       True


Output:
Chunk ID: 165d27
Wall time: 0.4165 seconds
Process exited with code 0
Original token count: 44
Output:
pidoka-hf-matomo           Synced   Healthy   9a8b373abdf4f0bb1e5d3959c2ceaee50b87584f
pidoka-hf-matomo-secrets   Synced   Healthy   9a8b373abdf4f0bb1e5d3959c2ceaee50b87584f


Output:
Chunk ID: 2cc1b1
Wall time: 0.0000 seconds
Process exited with code 0
Original token count: 15
Output:
 M tenants/pidoka/prod/n8n/external-secrets/n8n-secret.yaml


Vorprüfung passt: alt läuft, neu ist geparkt und ArgoCD grün. Ich friere jetzt die alte Matomo-Web-App ein, erstelle einen finalen Dump und sichere das alte Ingress-Manifest für Rollback.


ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 '
set -euo pipefail
TS=20260617T135148Z
BACKUP=/home/kube-bot/backups/matomo-cutover-$TS
mkdir -p "$BACKUP"
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo get ingress matomo-ingress -o yaml > "$BACKUP/old-matomo-ingress.yaml"
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo scale deployment matomo --replicas=0
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo rollout status deployment/matomo --timeout=120s || true
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo get deploy matomo matomo-db
# DB dump without printing credentials.
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo exec deployment/matomo-db -- sh -lc '\''mysqldump -h127.0.0.1 -uroot -p"$MYSQL_ROOT_PASSWORD" "$MYSQL_DATABASE"'\'' | gzip -c > "$BACKUP/matomo-db.sql.gz"
# HTML snapshot after app freeze.
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo exec deployment/matomo-db -- true >/dev/null
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo exec deployment/matomo-db -- sh -lc '\''true'\'' >/dev/null
# Use a short pod to archive the app PVC on the control-plane where backup path exists.
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo delete pod matomo-cutover-html-backup --ignore-not-found=true >/dev/null
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml apply -f - <<YAML
apiVersion: v1
kind: Pod
metadata:
  name: matomo-cutover-html-backup
  namespace: hf-matomo
spec:
  restartPolicy: Never
  nodeSelector:
    kubernetes.io/hostname: mbrkanov-mini-it13
  containers:
    - name: backup
      image: alpine:3.20
      command: ["/bin/sh", "-lc"]
      args:
        - tar -czf /backup/matomo-html.tar.gz -C /html . && echo html-backup-ok
      volumeMounts:
        - name: html
          mountPath: /html
          readOnly: true
        - name: backup
          mountPath: /backup
  volumes:
    - name: html
      persistentVolumeClaim:
        claimName: matomo-pvc
    - name: backup
      hostPath:
        path: $BACKUP
        type: Directory
YAML
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo wait --for=jsonpath={.status.phase}=Succeeded pod/matomo-cutover-html-backup --timeout=300s
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo logs matomo-cutover-html-backup
cd "$BACKUP"
sha256sum matomo-db.sql.gz matomo-html.tar.gz old-matomo-ingress.yaml > SHA256SUMS
sha256sum -c SHA256SUMS
printf "BACKUP=%s\n" "$BACKUP"
'


Output:
Chunk ID: dce1da
Wall time: 1.6074 seconds
Process exited with code 127
Original token count: 73
Output:
deployment.apps/matomo scaled
deployment "matomo" successfully rolled out
NAME        READY   UP-TO-DATE   AVAILABLE   AGE
matomo      0/0     0            0           282d
matomo-db   1/1     1            1           282d
sh: 1: mysqldump: not found
command terminated with exit code 127


Freeze ist aktiv: alte Matomo-Web-App steht jetzt auf `0/0`, alte DB läuft weiter. Der erste Dump-Befehl ist sauber fehlgeschlagen, weil `mysqldump` im MariaDB-Container fehlt; ich wechsle auf `mariadb-dump`/verfügbare Tools und lasse den Freeze bestehen.


ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 'sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo exec deployment/matomo-db -- sh -lc '\''command -v mariadb-dump || true; command -v mysqldump || true; command -v mariadb || true; command -v mysql || true'\''; sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo get deploy,pod -o wide'


ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 'ls -la /home/kube-bot/backups/matomo-cutover-20260617T135148Z 2>/dev/null || true'


Output:
Chunk ID: cad30f
Wall time: 1.5020 seconds
Process exited with code 0
Original token count: 158
Output:
/usr/bin/mariadb-dump
/usr/bin/mariadb
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS   IMAGES           SELECTOR
deployment.apps/matomo-db   1/1     1            1           282d   mariadb      mariadb:latest   app=matomo-db
deployment.apps/matomo      0/0     0            0           282d   matomo       matomo:latest    app=matomo

NAME                             READY   STATUS    RESTARTS   AGE   IP           NODE                 NOMINATED NODE   READINESS GATES
pod/matomo-db-6766699c55-xjg8d   1/1     Running   0          45d   10.42.0.61   mbrkanov-mini-it13   <none>           <none>


Output:
Chunk ID: 780536
Wall time: 1.2789 seconds
Process exited with code 0
Original token count: 63
Output:
total 16
drwxrwxr-x 2 kube-bot kube-bot 4096 Jun 17 15:52 .
drwxrwxr-x 5 kube-bot kube-bot 4096 Jun 17 15:52 ..
-rw-rw-r-- 1 kube-bot kube-bot   20 Jun 17 15:52 matomo-db.sql.gz
-rw-rw-r-- 1 kube-bot kube-bot 1508 Jun 17 15:52 old-matomo-ingress.yaml


Ich lösche den kaputten 20-Byte-Dump und erstelle den finalen Dump mit `mariadb-dump`. Danach sichere ich das HTML-PVC.


ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 '
set -euo pipefail
BACKUP=/home/kube-bot/backups/matomo-cutover-20260617T135148Z
rm -f "$BACKUP/matomo-db.sql.gz" "$BACKUP/matomo-html.tar.gz" "$BACKUP/SHA256SUMS"
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo exec deployment/matomo-db -- sh -lc '\''mariadb-dump -h127.0.0.1 -uroot -p"$MYSQL_ROOT_PASSWORD" "$MYSQL_DATABASE"'\'' | gzip -c > "$BACKUP/matomo-db.sql.gz"
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo delete pod matomo-cutover-html-backup --ignore-not-found=true >/dev/null
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml apply -f - <<YAML
apiVersion: v1
kind: Pod
metadata:
  name: matomo-cutover-html-backup
  namespace: hf-matomo
spec:
  restartPolicy: Never
  nodeSelector:
    kubernetes.io/hostname: mbrkanov-mini-it13
  containers:
    - name: backup
      image: alpine:3.20
      command: ["/bin/sh", "-lc"]
      args:
        - tar -czf /backup/matomo-html.tar.gz -C /html . && echo html-backup-ok
      volumeMounts:
        - name: html
          mountPath: /html
          readOnly: true
        - name: backup
          mountPath: /backup
  volumes:
    - name: html
      persistentVolumeClaim:
        claimName: matomo-pvc
    - name: backup
      hostPath:
        path: $BACKUP
        type: Directory
YAML
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo wait --for=jsonpath={.status.phase}=Succeeded pod/matomo-cutover-html-backup --timeout=300s
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo logs matomo-cutover-html-backup
cd "$BACKUP"
sha256sum matomo-db.sql.gz matomo-html.tar.gz old-matomo-ingress.yaml > SHA256SUMS
sha256sum -c SHA256SUMS
ls -lh "$BACKUP"
'


Output:
Chunk ID: 61ba8f
Wall time: 30.0006 seconds
Process running with session ID 50223
Original token count: 10
Output:
pod/matomo-cutover-html-backup created


Finaler DB-Dump ist geschrieben, HTML-Backup-Pod läuft gerade. Ich warte auf die Checksummenprüfung.


Output:
Chunk ID: a1fc8d
Wall time: 120.0006 seconds
Process running with session ID 50223
Original token count: 0
Output:


Das HTML-Backup dauert länger als erwartet, vermutlich wegen PVC/Longhorn-I/O. Ich prüfe den Pod parallel; Freeze bleibt aktiv.


ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 'sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo get pod matomo-cutover-html-backup -o wide; sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo describe pod matomo-cutover-html-backup | tail -50; ls -lh /home/kube-bot/backups/matomo-cutover-20260617T135148Z'


Output:
Chunk ID: 47509a
Wall time: 5.0006 seconds
Process running with session ID 50223
Original token count: 0
Output:


Output:
Chunk ID: 5d02eb
Wall time: 1.0431 seconds
Process exited with code 0
Original token count: 719
Output:
NAME                         READY   STATUS    RESTARTS   AGE     IP           NODE                 NOMINATED NODE   READINESS GATES
matomo-cutover-html-backup   1/1     Running   0          2m48s   10.42.0.70   mbrkanov-mini-it13   <none>           <none>
    Image ID:      docker.io/library/alpine@sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -lc
    Args:
      tar -czf /backup/matomo-html.tar.gz -C /html . && echo html-backup-ok
    State:          Running
      Started:      Wed, 17 Jun 2026 15:53:31 +0200
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /backup from backup (rw)
      /html from html (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xszcj (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  html:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  matomo-pvc
    ReadOnly:   false
  backup:
    Type:          HostPath (bare host directory volume)
    Path:          /home/kube-bot/backups/matomo-cutover-20260617T135148Z
    HostPathType:  Directory
  kube-api-access-xszcj:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              kubernetes.io/hostname=mbrkanov-mini-it13
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason                  Age    From                     Message
  ----    ------                  ----   ----                     -------
  Normal  Scheduled               2m47s  default-scheduler        Successfully assigned hf-matomo/matomo-cutover-html-backup to mbrkanov-mini-it13
  Normal  SuccessfulAttachVolume  2m31s  attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-f6714be0-39b1-47e6-baa2-2e1ae801d907"
  Normal  Pulled                  2m11s  kubelet                  Container image "alpine:3.20" already present on machine
  Normal  Created                 2m11s  kubelet                  Created container backup
  Normal  Started                 2m10s  kubelet                  Started container backup
total 8.2M
-rw-rw-r-- 1 kube-bot kube-bot 781K Jun 17 15:52 matomo-db.sql.gz
-rw-r--r-- 1 root     root     7.4M Jun 17 15:55 matomo-html.tar.gz
-rw-rw-r-- 1 kube-bot kube-bot 1.5K Jun 17 15:52 old-matomo-ingress.yaml


HTML-Archiv wächst, Backup-Pod läuft normal. Ich lasse ihn fertig schreiben, bevor ich die Checksummen akzeptiere.


Output:
Chunk ID: b20fe6
Wall time: 128.1987 seconds
Process exited with code 1
Original token count: 20
Output:
error: timed out waiting for the condition on pods/matomo-cutover-html-backup


Der `kubectl wait` ist am Timeout ausgelaufen, nicht der Backup-Pod selbst. Ich prüfe jetzt, ob das Archiv vollständig geschrieben ist oder ob der Pod hängt; falls nötig nutze ich den bereits vorhandenen letzten geprüften HTML-Snapshot und nur den frischen DB-Dump.


ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 'sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo get pod matomo-cutover-html-backup -o wide; sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo logs matomo-cutover-html-backup --tail=20 || true; ls -lh /home/kube-bot/backups/matomo-cutover-20260617T135148Z; stat -c "%s %y %n" /home/kube-bot/backups/matomo-cutover-20260617T135148Z/matomo-html.tar.gz'


ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 'ls -lh /home/kube-bot/backups/matomo-20260617T060849Z/matomo-html.tar.gz /home/kube-bot/backups/matomo-20260617T060849Z/matomo-db.sql.gz 2>/dev/null || true'


Output:
Chunk ID: 44fbeb
Wall time: 1.3399 seconds
Process exited with code 0
Original token count: 149
Output:
NAME                         READY   STATUS    RESTARTS   AGE     IP           NODE                 NOMINATED NODE   READINESS GATES
matomo-cutover-html-backup   1/1     Running   0          5m16s   10.42.0.70   mbrkanov-mini-it13   <none>           <none>
total 22M
-rw-rw-r-- 1 kube-bot kube-bot 781K Jun 17 15:52 matomo-db.sql.gz
-rw-r--r-- 1 root     root      21M Jun 17 15:58 matomo-html.tar.gz
-rw-rw-r-- 1 kube-bot kube-bot 1.5K Jun 17 15:52 old-matomo-ingress.yaml
21637097 2026-06-17 15:58:09.871977076 +0200 /home/kube-bot/backups/matomo-cutover-20260617T135148Z/matomo-html.tar.gz


Output:
Chunk ID: 1e1578
Wall time: 1.0946 seconds
Process exited with code 0
Original token count: 57
Output:
-rw-rw-r-- 1 kube-bot kube-bot 773K Jun 17 08:08 /home/kube-bot/backups/matomo-20260617T060849Z/matomo-db.sql.gz
-rw-rw-r-- 1 kube-bot kube-bot  85M Jun 17 08:10 /home/kube-bot/backups/matomo-20260617T060849Z/matomo-html.tar.gz


Das HTML-Archiv ist noch im Aufbau und noch kleiner als der frühere Snapshot, also warte ich weiter. Der Freeze ist aktiv; neue Writes sind dadurch weiterhin blockiert.


sleep 120
ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 'sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo get pod matomo-cutover-html-backup -o wide; sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo logs matomo-cutover-html-backup --tail=20 || true; ls -lh /home/kube-bot/backups/matomo-cutover-20260617T135148Z; stat -c "%s %y %n" /home/kube-bot/backups/matomo-cutover-20260617T135148Z/matomo-html.tar.gz'


Output:
Chunk ID: bba651
Wall time: 30.0021 seconds
Process running with session ID 40520
Original token count: 0
Output:


Ich warte noch auf die 120-Sekunden-Prüfung. Danach entscheide ich, ob der HTML-Backup-Pod weiterläuft oder ob ich auf eine schnellere `kubectl cp`/Direkt-Tar-Variante wechsle.


Output:
Chunk ID: e44384
Wall time: 85.5435 seconds
Process exited with code 0
Original token count: 149
Output:
NAME                         READY   STATUS    RESTARTS   AGE     IP           NODE                 NOMINATED NODE   READINESS GATES
matomo-cutover-html-backup   1/1     Running   0          7m28s   10.42.0.70   mbrkanov-mini-it13   <none>           <none>
total 60M
-rw-rw-r-- 1 kube-bot kube-bot 781K Jun 17 15:52 matomo-db.sql.gz
-rw-r--r-- 1 root     root      59M Jun 17 16:00 matomo-html.tar.gz
-rw-rw-r-- 1 kube-bot kube-bot 1.5K Jun 17 15:52 old-matomo-ingress.yaml
61368297 2026-06-17 16:00:21.706640647 +0200 /home/kube-bot/backups/matomo-cutover-20260617T135148Z/matomo-html.tar.gz


Archiv ist inzwischen bei 59 MB und wächst weiter; der frühere Snapshot hatte 85 MB. Ich lasse den laufenden Prozess noch fertig werden, statt ein teilweise geschriebenes Archiv zu verwenden.


sleep 90
ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 'sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo get pod matomo-cutover-html-backup -o jsonpath="phase={.status.phase}{\"\\n\"}"; sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo logs matomo-cutover-html-backup --tail=20 || true; ls -lh /home/kube-bot/backups/matomo-cutover-20260617T135148Z/matomo-html.tar.gz'


Output:
Chunk ID: 357e62
Wall time: 30.0014 seconds
Process running with session ID 77250
Original token count: 0
Output:


Weiterhin im Wartefenster für das HTML-Archiv. Ich melde den nächsten Status direkt nach der Prüfung.


Output:
Chunk ID: f61286
Wall time: 57.4479 seconds
Process exited with code 0
Original token count: 32
Output:
phase=Running
-rw-r--r-- 1 root root 71M Jun 17 16:02 /home/kube-bot/backups/matomo-cutover-20260617T135148Z/matomo-html.tar.gz


Der HTML-Tar läuft immer noch, jetzt 71 MB. Da er fortschreitet und keine Fehler zeigt, lasse ich ihn fertigschreiben; ich breche nicht ab, damit der Cutover auf einem vollständigen finalen Snapshot basiert.


Output:
Chunk ID: c76bd6
Wall time: 30.0020 seconds
Process running with session ID 22467
Original token count: 0
Output:


Ich prüfe nach weiteren 90 Sekunden wieder. Alter Matomo-Webzugriff bleibt währenddessen absichtlich eingefroren; DB und Rollback-PVCs bleiben erhalten.


Output:
Chunk ID: 979351
Wall time: 55.4414 seconds
Process exited with code 0
Original token count: 32
Output:
phase=Running
-rw-r--r-- 1 root root 73M Jun 17 16:03 /home/kube-bot/backups/matomo-cutover-20260617T135148Z/matomo-html.tar.gz


Das HTML-Backup ist ungewöhnlich langsam und wächst nur noch wenig. Ich prüfe jetzt, ob es an einem bestimmten Dateibereich hängt; falls es nicht sauber fertig wird, nehme ich für den Cutover den bereits geprüften HTML-Snapshot und den frischen DB-Dump, weil Matomo-Daten primär in der DB liegen.


ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 'sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo exec matomo-cutover-html-backup -- sh -lc '\''ps -ef; du -sh /html 2>/dev/null || true; find /html -xdev -type f | wc -l'\'''


ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 'stat -c "%s %Y %n" /home/kube-bot/backups/matomo-cutover-20260617T135148Z/matomo-html.tar.gz; sleep 15; stat -c "%s %Y %n" /home/kube-bot/backups/matomo-cutover-20260617T135148Z/matomo-html.tar.gz'


Output:
Chunk ID: 6b5c63
Wall time: 20.0011 seconds
Process running with session ID 66283
Original token count: 90
Output:
PID   USER     TIME  COMMAND
    1 root      0:00 /bin/sh -lc tar -czf /backup/matomo-html.tar.gz -C /html . && echo html-backup-ok
    7 root      0:01 tar -czf /backup/matomo-html.tar.gz -C /html .
    8 root      0:17 gzip -f
    9 root      0:00 sh -lc ps -ef; du -sh /html 2>/dev/null || true; find /html -xdev -type f | wc -l
   15 root      0:00 ps -ef


Output:
Chunk ID: 5d6119
Wall time: 16.1605 seconds
Process exited with code 0
Original token count: 47
Output:
78634844 1781705041 /home/kube-bot/backups/matomo-cutover-20260617T135148Z/matomo-html.tar.gz
78995292 1781705058 /home/kube-bot/backups/matomo-cutover-20260617T135148Z/matomo-html.tar.gz


Output:
Chunk ID: ca8551
Wall time: 63.8411 seconds
Process exited with code 0
Original token count: 5
Output:
239.2M	/html
11354


Der Prozess lebt: `/html` hat ca. 239 MB und 11.354 Dateien; gzip schreibt weiter. Das Archiv wächst noch, also warte ich auf sauberen Abschluss.


sleep 120
ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 'sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo get pod matomo-cutover-html-backup -o jsonpath="phase={.status.phase}{\"\\n\"}"; sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo logs matomo-cutover-html-backup --tail=20 || true; ls -lh /home/kube-bot/backups/matomo-cutover-20260617T135148Z/matomo-html.tar.gz'


Output:
Chunk ID: f27e23
Wall time: 30.0011 seconds
Process running with session ID 26850
Original token count: 0
Output:


Ich bin noch im Backup-Schritt, nicht hängen geblieben. Nächste Statusprüfung läuft; sobald das Archiv fertig ist, gehen Restore und Cutover deutlich schneller.


Output:
Chunk ID: e740e7
Wall time: 86.6116 seconds
Process exited with code 0
Original token count: 32
Output:
phase=Running
-rw-r--r-- 1 root root 78M Jun 17 16:07 /home/kube-bot/backups/matomo-cutover-20260617T135148Z/matomo-html.tar.gz


Das finale HTML-Archiv kommt nicht in sinnvoller Zeit zum Abschluss. Ich breche nur diesen HTML-Backup-Pod ab und verwende für den Cutover den bereits geprüften HTML-Restore im neuen PVC; für die wirklich veränderlichen Daten nehme ich den frischen finalen DB-Dump aus dem Wartungsfenster.


ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 '
set -euo pipefail
BACKUP=/home/kube-bot/backups/matomo-cutover-20260617T135148Z
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo delete pod matomo-cutover-html-backup --ignore-not-found=true
mv -f "$BACKUP/matomo-html.tar.gz" "$BACKUP/matomo-html.tar.gz.incomplete" || true
cd "$BACKUP"
sha256sum matomo-db.sql.gz old-matomo-ingress.yaml > SHA256SUMS
sha256sum -c SHA256SUMS
ls -lh "$BACKUP"
'


Output:
Chunk ID: c80e7b
Wall time: 30.0007 seconds
Process running with session ID 61472
Original token count: 11
Output:
pod "matomo-cutover-html-backup" deleted


Der hängende HTML-Backup-Pod wird gerade gelöscht. Ich warte auf Abschluss und prüfe dann den finalen DB-Dump per Checksumme.


Output:
Chunk ID: ff2428
Wall time: 0.0000 seconds
Process exited with code 0
Original token count: 85
Output:
matomo-db.sql.gz: OK
old-matomo-ingress.yaml: OK
total 79M
-rw-rw-r-- 1 kube-bot kube-bot  173 Jun 17 16:08 SHA256SUMS
-rw-rw-r-- 1 kube-bot kube-bot 781K Jun 17 15:52 matomo-db.sql.gz
-rw-r--r-- 1 root     root      79M Jun 17 16:08 matomo-html.tar.gz.incomplete
-rw-rw-r-- 1 kube-bot kube-bot 1.5K Jun 17 15:52 old-matomo-ingress.yaml


Finaler DB-Dump und Rollback-Ingress sind verifiziert. Ich restore jetzt nur die Staging-DB frisch; das Staging-HTML-PVC bleibt der bereits geprüfte Restore.


Finalen Produktionsstand sichern und Schreibzugriff einfrieren

Staging-PVCs neu aus finalem Backup restoren

Staging intern mit gepinnten Images verifizieren

Ingress/Traffic auf neuen Namespace umschalten

Produktions- und Rollback-Status dokumentieren


Output: Plan updated


ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 '
set -euo pipefail
BACKUP=/home/kube-bot/backups/matomo-cutover-20260617T135148Z
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n pidoka-hf-matomo scale deployment matomo matomo-db --replicas=0
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n pidoka-hf-matomo delete pod matomo-db-pvc-cleanup --ignore-not-found=true >/dev/null
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml apply -f - <<YAML
apiVersion: v1
kind: Pod
metadata:
  name: matomo-db-pvc-cleanup
  namespace: pidoka-hf-matomo
spec:
  restartPolicy: Never
  nodeSelector:
    kubernetes.io/hostname: mbrkanov-mini-it13
  containers:
    - name: cleanup
      image: alpine:3.20
      command: ["/bin/sh", "-lc"]
      args:
        - find /db -mindepth 1 -maxdepth 1 -exec rm -rf {} + && echo db-cleanup-ok
      volumeMounts:
        - name: db
          mountPath: /db
  volumes:
    - name: db
      persistentVolumeClaim:
        claimName: matomo-db-pvc
YAML
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n pidoka-hf-matomo wait --for=jsonpath={.status.phase}=Succeeded pod/matomo-db-pvc-cleanup --timeout=180s
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n pidoka-hf-matomo logs matomo-db-pvc-cleanup
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n pidoka-hf-matomo scale deployment matomo-db --replicas=1
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n pidoka-hf-matomo rollout status deployment/matomo-db --timeout=300s
'


Output:
Chunk ID: a7611f
Wall time: 30.0003 seconds
Process running with session ID 50876
Original token count: 25
Output:
deployment.apps/matomo scaled
deployment.apps/matomo-db scaled
pod/matomo-db-pvc-cleanup created


Staging-DB-PVC wird gerade geleert und danach mit dem finalen Dump neu aufgebaut. Ich beobachte den Cleanup/Rollout.


Output:
Chunk ID: 650f59
Wall time: 85.6175 seconds
Process exited with code 0
Original token count: 58
Output:
pod/matomo-db-pvc-cleanup condition met
db-cleanup-ok
deployment.apps/matomo-db scaled
Waiting for deployment "matomo-db" rollout to finish: 0 of 1 updated replicas are available...
deployment "matomo-db" successfully rolled out


Staging-DB ist frisch gestartet. Ich importiere jetzt den finalen DB-Dump aus dem Wartungsfenster und prüfe danach Tabellen/Sites.


ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 'set -o pipefail; gzip -dc /home/kube-bot/backups/matomo-cutover-20260617T135148Z/matomo-db.sql.gz | sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n pidoka-hf-matomo exec -i deployment/matomo-db -- sh -lc '\''mariadb -h127.0.0.1 -P3306 -uroot -p"$MARIADB_ROOT_PASSWORD" "$MARIADB_DATABASE"'\'' && echo final-db-import-ok
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n pidoka-hf-matomo exec deployment/matomo-db -- sh -lc '\''mariadb -h127.0.0.1 -P3306 -uroot -p"$MARIADB_ROOT_PASSWORD" "$MARIADB_DATABASE" -N -e "SELECT COUNT(*) FROM information_schema.tables WHERE table_schema=DATABASE(); SELECT COUNT(*) FROM matomo_site;"'\'''


Output:
Chunk ID: 485ff0
Wall time: 9.0979 seconds
Process exited with code 1
Original token count: 51
Output:
ERROR 2002 (HY000): Can't connect to server on '127.0.0.1' (115)
command terminated with exit code 1
ERROR 2002 (HY000): Can't connect to server on '127.0.0.1' (115)
command terminated with exit code 1


MariaDB ist vom Kubernetes-Rollout her verfügbar, nimmt aber intern noch keine TCP-Verbindung an. Das hatten wir vorhin schon gesehen; ich warte auf Port 3306 und importiere dann erneut.


for i in $(seq 1 60); do
  out=$(ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 'sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n pidoka-hf-matomo exec deployment/matomo-db -- sh -lc '\''(ss -ltn 2>/dev/null || netstat -ltn 2>/dev/null || true) | grep -q 3306 && echo port-open || true; ps -ef | grep mariadbd | grep -v grep | sed "s/  */ /g"'\''')
  echo "$out" | sed -n '1,3p'
  echo "$out" | grep -q 'port-open' && exit 0
  sleep 5
done
exit 1


Output:
Chunk ID: 552344
Wall time: 30.0009 seconds
Process running with session ID 32386
Original token count: 585
Output:
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K


DB initialisiert wieder das frische Datenverzeichnis; das ist erwartbar und dauert einige Minuten. Ich bleibe am Poll, bis Port 3306 offen ist.


Output:
Chunk ID: 3ceab7
Wall time: 120.0016 seconds
Process running with session ID 32386
Original token count: 2633
Output:
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K


MariaDB hängt erneut lange in der Initialisierung des frisch geleerten PVCs. Das ist der gleiche Effekt wie vorhin; ich warte noch kurz auf Port 3306. Wenn es zu lange dauert, stoppe ich und rolle Freeze zurück, statt das Wartungsfenster unnötig zu verlängern.


Output:
Chunk ID: aeebc2
Wall time: 120.0012 seconds
Process running with session ID 32386
Original token count: 2133
Output:
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 74 72 0 14:10 ? 00:00:00 /usr/sbin/mariadbd --lc-messages-dir=/usr/share/mariadb/english/.. --bootstrap --silent-startup --basedir=/usr --datadir=/var/lib/mysql/ --log-warnings=0 --enforce-storage-engine= --plugin-dir=/usr/lib/mysql/plugin --old-mode=UTF8_IS_UTF8MB3 --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 --loose-innodb_buffer_pool_dump_at_shutdown=0 --max_allowed_packet=8M --net_buffer_length=16K
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 633 1 1 14:15 ? 00:00:00 mariadbd --skip-networking --default-time-zone=SYSTEM --socket=/run/mysqld/mysqld.sock --wsrep_on=OFF --expire-logs-days=0 --skip-slave-start --loose-innodb_buffer_pool_load_at_startup=0 --skip-ssl --ssl-cert= --ssl-key= --ssl-ca=
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 633 1 0 14:15 ? 00:00:00 mariadbd --skip-networking --default-time-zone=SYSTEM --socket=/run/mysqld/mysqld.sock --wsrep_on=OFF --expire-logs-days=0 --skip-slave-start --loose-innodb_buffer_pool_load_at_startup=0 --skip-ssl --ssl-cert= --ssl-key= --ssl-ca=
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 633 1 0 14:15 ? 00:00:00 mariadbd --skip-networking --default-time-zone=SYSTEM --socket=/run/mysqld/mysqld.sock --wsrep_on=OFF --expire-logs-days=0 --skip-slave-start --loose-innodb_buffer_pool_load_at_startup=0 --skip-ssl --ssl-cert= --ssl-key= --ssl-ca=
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 633 1 0 14:15 ? 00:00:00 mariadbd --skip-networking --default-time-zone=SYSTEM --socket=/run/mysqld/mysqld.sock --wsrep_on=OFF --expire-logs-days=0 --skip-slave-start --loose-innodb_buffer_pool_load_at_startup=0 --skip-ssl --ssl-cert= --ssl-key= --ssl-ca=
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 633 1 3 14:15 ? 00:00:01 mariadbd --skip-networking --default-time-zone=SYSTEM --socket=/run/mysqld/mysqld.sock --wsrep_on=OFF --expire-logs-days=0 --skip-slave-start --loose-innodb_buffer_pool_load_at_startup=0 --skip-ssl --ssl-cert= --ssl-key= --ssl-ca=
mysql 764 1 0 14:15 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 633 1 3 14:15 ? 00:00:01 mariadbd --skip-networking --default-time-zone=SYSTEM --socket=/run/mysqld/mysqld.sock --wsrep_on=OFF --expire-logs-days=0 --skip-slave-start --loose-innodb_buffer_pool_load_at_startup=0 --skip-ssl --ssl-cert= --ssl-key= --ssl-ca=
mysql 764 1 0 14:15 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 633 1 2 14:15 ? 00:00:01 mariadbd --skip-networking --default-time-zone=SYSTEM --socket=/run/mysqld/mysqld.sock --wsrep_on=OFF --expire-logs-days=0 --skip-slave-start --loose-innodb_buffer_pool_load_at_startup=0 --skip-ssl --ssl-cert= --ssl-key= --ssl-ca=
mysql 764 1 0 14:15 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd


Die DB ist jetzt in der zweiten Initialisierungsphase (`--skip-networking`), also Fortschritt. Ich warte noch bis zum normalen Servermodus, dann importiere ich direkt.


Output:
Chunk ID: cf38bc
Wall time: 43.8606 seconds
Process exited with code 0
Original token count: 420
Output:
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 633 1 2 14:15 ? 00:00:01 mariadbd --skip-networking --default-time-zone=SYSTEM --socket=/run/mysqld/mysqld.sock --wsrep_on=OFF --expire-logs-days=0 --skip-slave-start --loose-innodb_buffer_pool_load_at_startup=0 --skip-ssl --ssl-cert= --ssl-key= --ssl-ca=
mysql 764 1 0 14:15 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 633 1 2 14:15 ? 00:00:01 mariadbd --skip-networking --default-time-zone=SYSTEM --socket=/run/mysqld/mysqld.sock --wsrep_on=OFF --expire-logs-days=0 --skip-slave-start --loose-innodb_buffer_pool_load_at_startup=0 --skip-ssl --ssl-cert= --ssl-key= --ssl-ca=
mysql 764 1 0 14:15 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 633 1 1 14:15 ? 00:00:01 mariadbd --skip-networking --default-time-zone=SYSTEM --socket=/run/mysqld/mysqld.sock --wsrep_on=OFF --expire-logs-days=0 --skip-slave-start --loose-innodb_buffer_pool_load_at_startup=0 --skip-ssl --ssl-cert= --ssl-key= --ssl-ca=
mysql 1 0 0 14:10 ? 00:00:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mariadbd
mysql 633 1 1 14:15 ? 00:00:01 mariadbd --skip-networking --default-time-zone=SYSTEM --socket=/run/mysqld/mysqld.sock --wsrep_on=OFF --expire-logs-days=0 --skip-slave-start --loose-innodb_buffer_pool_load_at_startup=0 --skip-ssl --ssl-cert= --ssl-key= --ssl-ca=
mysql 1 0 0 14:10 ? 00:00:00 mariadbd
mysql 1 0 0 14:10 ? 00:00:00 mariadbd
port-open
mysql 1 0 0 14:10 ? 00:00:01 mariadbd


Port 3306 ist offen. Ich importiere jetzt den finalen Dump und verifiziere die DB-Zähler.


Output:
Chunk ID: 7fbca9
Wall time: 30.0004 seconds
Process running with session ID 55482
Original token count: 0
Output:


Finaler DB-Import läuft. Beim letzten Mal dauerte der Commit-Teil rund zwei Minuten; ich bleibe dran.


Output:
Chunk ID: c7a8d4
Wall time: 120.0010 seconds
Process running with session ID 55482
Original token count: 0
Output:


Import läuft weiter, keine Fehlermeldung. Ich prüfe parallel die DB-Prozessliste, ohne den Import zu unterbrechen.


ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 'sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n pidoka-hf-matomo exec deployment/matomo-db -- sh -lc '\''ps -ef | grep -E "maria|mysql" | grep -v grep; mariadb -h127.0.0.1 -P3306 -uroot -p"$MARIADB_ROOT_PASSWORD" -e "SHOW PROCESSLIST;" 2>/dev/null | sed -E "s/(Password|password|secret|key)[^[:space:]]*/[REDACTED]/g" || true'\'''


Output:
Chunk ID: d5ad96
Wall time: 5.0008 seconds
Process running with session ID 55482
Original token count: 0
Output:


Output:
Chunk ID: 0101cc
Wall time: 1.3348 seconds
Process exited with code 0
Original token count: 148
Output:
mysql          1       0  9 14:10 ?        00:00:51 mariadbd
root         948       0  0 14:17 ?        00:00:00 sh -lc mariadb -h127.0.0.1 -P3306 -uroot -p"$MARIADB_ROOT_PASSWORD" "$MARIADB_DATABASE"
root         957     948  0 14:17 ?        00:00:00 mariadb -h127.0.0.1 -P3306 -uroot -px xxxxxxxxx matomo
Id	User	Host	db	Command	Time	State	Info	Progress
3	root	127.0.0.1:56254	matomo	Query	7	Commit	INSERT INTO `matomo_log_link_visit_action` VALUES\n(1,1,'�\t���H�',1,0,0,NULL,1,'2025-09-07 19:05:50'	0.000
4	root	127.0.0.1:47050	NULL	Query	0	starting	SHOW PROCESSLIST	0.000


Import ist aktiv und arbeitet gerade im `Commit` auf Matomo-Logtabellen. Ich lasse ihn fertig laufen.


Output:
Chunk ID: 5b04bb
Wall time: 119.7426 seconds
Process exited with code 0
Original token count: 6
Output:
final-db-import-ok
53
2


Finaler DB-Restore ist verifiziert: Import OK, 53 Tabellen, 2 Sites. Ich starte jetzt Staging-App intern und prüfe HTTP, danach schalte ich den Ingress um.


Output: Plan updated


ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 '
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n pidoka-hf-matomo scale deployment matomo --replicas=1
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n pidoka-hf-matomo rollout status deployment/matomo --timeout=240s
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n pidoka-hf-matomo exec deployment/matomo -- sh -lc '\''curl -sS -o /dev/null -w "http_code=%{http_code}\\n" http://127.0.0.1/'\''
sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n pidoka-hf-matomo get pod -l "app in (matomo,matomo-db)" -o jsonpath="{range .items[*]}{.metadata.name} imageID={.status.containerStatuses[0].imageID} restarts={.status.containerStatuses[0].restartCount}{\"\\n\"}{end}"
'


Output:
Chunk ID: 939129
Wall time: 30.0007 seconds
Process running with session ID 78536
Original token count: 31
Output:
deployment.apps/matomo scaled
Waiting for deployment "matomo" rollout to finish: 0 of 1 updated replicas are available...


Staging-App startet wieder; Image ist bereits gepinnt. Ich warte auf Rollout und HTTP-Test.


Output:
Chunk ID: a23b69
Wall time: 17.8284 seconds
Process exited with code 0
Original token count: 86
Output:
deployment "matomo" successfully rolled out
http_code=200
matomo-db-7b9bdc7b94-2hv2t imageID=docker.io/library/mariadb@sha256:e0236fc6386e7eacd9359e59d0a078bd7aa0d18280d36d13061121bedeaee903 restarts=0
matomo-6fb4766964-znjvw imageID=docker.io/library/matomo@sha256:21b8bae018cd51a05f36bc6beac3861bb314d131df690c47bba1648aad2b8d7e restarts=0


Interner Check ist grün: HTTP `200`, beide Pods mit produktiven Digests, `0` Restarts. Ich bereite jetzt die Ingress-Umschaltung vor: altes Ingress sichern ist erledigt, neues Ingress kommt in den neuen Namespace.


Output: Plan updated


ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 'sed -n "1,220p" /home/kube-bot/backups/matomo-cutover-20260617T135148Z/old-matomo-ingress.yaml'


Output:
Chunk ID: a1425c
Wall time: 1.3148 seconds
Process exited with code 0
Original token count: 377
Output:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{},"labels":{"io.portainer.kubernetes.application.kind":"content","io.portainer.kubernetes.application.name":"d-hf-matomo","io.portainer.kubernetes.application.owner":"admin","io.portainer.kubernetes.application.stack":"d-hf-matomo","io.portainer.kubernetes.application.stackid":"5"},"name":"matomo-ingress","namespace":"hf-matomo"},"spec":{"rules":[{"host":"matomo.heckenfels.com","http":{"paths":[{"backend":{"service":{"name":"matomo","port":{"number":80}}},"path":"/","pathType":"Prefix"}]}}]}}
  creationTimestamp: "2025-09-07T15:04:56Z"
  generation: 1
  labels:
    io.portainer.kubernetes.application.kind: content
    io.portainer.kubernetes.application.name: d-hf-matomo
    io.portainer.kubernetes.application.owner: admin
    io.portainer.kubernetes.application.stack: d-hf-matomo
    io.portainer.kubernetes.application.stackid: "5"
  name: matomo-ingress
  namespace: hf-matomo
  resourceVersion: "80999146"
  uid: fc699ea2-7711-405e-a7a1-2b852fede85a
spec:
  ingressClassName: traefik
  rules:
  - host: matomo.heckenfels.com
    http:
      paths:
      - backend:
          service:
            name: matomo
            port:
              number: 80
        path: /
        pathType: Prefix
status:
  loadBalancer:
    ingress:
    - ip: 192.168.20.40
    - ip: 192.168.20.45


Ich übernehme die alte Host-Regel `matomo.heckenfels.com` als GitOps-Ingress im neuen Namespace. Beim Cutover lösche ich danach das alte Ingress und synchronisiere die neue App; Rollback ist das gesicherte alte Ingress-Manifest.


Input:
*** Begin Patch
*** Update File: /home/node/.openclaw/workspace/repos/platform-gitops/tenants/pidoka/prod/matomo/app.yaml
@@
 apiVersion: v1
 kind: Service
 metadata:
   name: matomo
@@
   ports:
     - protocol: TCP
       port: 80
       targetPort: 80
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: matomo-ingress
+  namespace: pidoka-hf-matomo
+  annotations:
+    argocd.argoproj.io/sync-wave: "5"
+spec:
+  ingressClassName: traefik
+  rules:
+    - host: matomo.heckenfels.com
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: matomo
+                port:
+                  number: 80
*** End Patch


Success. Updated the following files:
M /home/node/.openclaw/workspace/repos/platform-gitops/tenants/pidoka/prod/matomo/app.yaml

@@ -183 +183,22 @@
       targetPort: 80
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: matomo-ingress
+  namespace: pidoka-hf-matomo
+  annotations:
+    argocd.argoproj.io/sync-wave: "5"
+spec:
+  ingressClassName: traefik
+  rules:
+    - host: matomo.heckenfels.com
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: matomo
+                port:
+                  number: 80


Output:
Exit code: 0
Wall time: 0 seconds
Output:
Success. Updated the following files:
M /home/node/.openclaw/workspace/repos/platform-gitops/tenants/pidoka/prod/matomo/app.yaml


Ingress ist lokal im GitOps-Manifest ergänzt. Ich committe und pushe nur diese Matomo-Änderung; die n8n-Diff bleibt weiterhin getrennt.


git -C /home/node/.openclaw/workspace/repos/platform-gitops diff -- tenants/pidoka/prod/matomo/app.yaml
git -C /home/node/.openclaw/workspace/repos/platform-gitops add tenants/pidoka/prod/matomo/app.yaml && git -C /home/node/.openclaw/workspace/repos/platform-gitops commit -m "Add Matomo cutover ingress"
set +x
python3 - <<'PY' > /tmp/git-basic-header
import base64, json, os, urllib.request, urllib.parse
it15='http://192.168.1.121:8088'
def req(method,url,token=None,data=None):
    h={'Content-Type':'application/json'}
    if token: h['Authorization']='Bearer '+token
    r=urllib.request.Request(url, data=None if data is None else json.dumps(data).encode(), headers=h, method=method)
    with urllib.request.urlopen(r, timeout=15) as resp: return json.loads(resp.read().decode() or '{}')
login=req('POST',it15+'/api/v1/auth/universal-auth/login',data={'clientId':os.environ['INFISICAL_CLIENT_ID'],'clientSecret':os.environ['INFISICAL_AUTH_KEY']})
qs=urllib.parse.urlencode({'projectId':'c5e8661f-4b7a-43e1-80b0-8674f30950dc','environment':'prod','secretPath':'/platform/gitops/github'})
obj=req('GET',it15+'/api/v4/secrets/GITHUB_PAT_PLATFORM_GITOPS?'+qs,login['accessToken'])
tok=obj['secret']['secretValue']
print('AUTHORIZATION: basic '+base64.b64encode(('x-access-token:'+tok).encode()).decode())
PY
hdr=$(cat /tmp/git-basic-header)
rm -f /tmp/git-basic-header
git -C /home/node/.openclaw/workspace/repos/platform-gitops -c http.https://github.com/.extraheader="$hdr" push


Output:
Chunk ID: 0911ea
Wall time: 1.1633 seconds
Process exited with code 0
Original token count: 230
Output:
diff --git a/tenants/pidoka/prod/matomo/app.yaml b/tenants/pidoka/prod/matomo/app.yaml
index 3fd0110..55358a8 100644
--- a/tenants/pidoka/prod/matomo/app.yaml
+++ b/tenants/pidoka/prod/matomo/app.yaml
@@ -181,3 +181,24 @@ spec:
     - protocol: TCP
       port: 80
       targetPort: 80
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: matomo-ingress
+  namespace: pidoka-hf-matomo
+  annotations:
+    argocd.argoproj.io/sync-wave: "5"
+spec:
+  ingressClassName: traefik
+  rules:
+    - host: matomo.heckenfels.com
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: matomo
+                port:
+                  number: 80
[main 5851072] Add Matomo cutover ingress
 1 file changed, 21 insertions(+)
To https://github.com/GitMasterOne/platform-gitops.git
   9a8b373..5851072  main -> main


Commit `5851072` ist gepusht. Ich lösche jetzt das alte Ingress in `hf-matomo`, synchronisiere `pidoka-hf-matomo` auf den neuen Commit und teste den Host.


ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 'sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo delete ingress matomo-ingress'
ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new Misla@192.168.1.121 'wsl.exe -d Ubuntu -e bash -lc "kubectl -n argocd annotate application pidoka-hf-matomo argocd.argoproj.io/refresh=hard --overwrite && kubectl -n argocd patch application pidoka-hf-matomo --type merge -p '\''{\"operation\":{\"sync\":{\"revision\":\"main\"}}}'\''"'


Output:
Chunk ID: 0b4113
Wall time: 2.0073 seconds
Process exited with code 0
Original token count: 38
Output:
ingress.networking.k8s.io "matomo-ingress" deleted
application.argoproj.io/pidoka-hf-matomo annotated
application.argoproj.io/pidoka-hf-matomo patched


Traffic-Umschaltung läuft: altes Ingress ist gelöscht, neue ArgoCD-Sync-Operation ist angestoßen. Ich poll jetzt ArgoCD und Kubernetes-Ingress.


for i in $(seq 1 12); do ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new Misla@192.168.1.121 'wsl.exe -d Ubuntu -e bash -lc "kubectl -n argocd get application pidoka-hf-matomo -o custom-columns=NAME:.metadata.name,SYNC:.status.sync.status,HEALTH:.status.health.status,REV:.status.sync.revision,PHASE:.status.operationState.phase --no-headers"'; sleep 5; done


for i in $(seq 1 12); do ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 'sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml get ingress -A | grep matomo || true'; sleep 5; done


Output:
Chunk ID: b4a241
Wall time: 30.0015 seconds
Process running with session ID 90643
Original token count: 137
Output:
pidoka-hf-matomo   OutOfSync   Healthy   585107236d5ec2abe5b8ffd591fb095a444dcb7a   Running
pidoka-hf-matomo   Synced   Healthy   585107236d5ec2abe5b8ffd591fb095a444dcb7a   Succeeded
pidoka-hf-matomo   Synced   Healthy   585107236d5ec2abe5b8ffd591fb095a444dcb7a   Succeeded
pidoka-hf-matomo   Synced   Healthy   585107236d5ec2abe5b8ffd591fb095a444dcb7a   Succeeded
pidoka-hf-matomo   Synced   Healthy   585107236d5ec2abe5b8ffd591fb095a444dcb7a   Succeeded
pidoka-hf-matomo   Synced   Healthy   585107236d5ec2abe5b8ffd591fb095a444dcb7a   Succeeded


Output:
Chunk ID: f56d10
Wall time: 30.0014 seconds
Process running with session ID 64231
Original token count: 117
Output:
pidoka-hf-matomo   matomo-ingress      traefik   matomo.heckenfels.com     192.168.20.40,192.168.20.45   80      3s
pidoka-hf-matomo   matomo-ingress      traefik   matomo.heckenfels.com     192.168.20.40,192.168.20.45   80      9s
pidoka-hf-matomo   matomo-ingress      traefik   matomo.heckenfels.com     192.168.20.40,192.168.20.45   80      15s
pidoka-hf-matomo   matomo-ingress      traefik   matomo.heckenfels.com     192.168.20.40,192.168.20.45   80      21s


ArgoCD ist auf `5851072` `Synced / Healthy`, und das Matomo-Ingress existiert jetzt nur im neuen Namespace `pidoka-hf-matomo`. Ich warte die Polls aus und teste dann den Host über den Cluster/Ingress-Pfad.


Output:
Chunk ID: c97dc8
Wall time: 30.2278 seconds
Process exited with code 0
Original token count: 137
Output:
pidoka-hf-matomo   Synced   Healthy   585107236d5ec2abe5b8ffd591fb095a444dcb7a   Succeeded
pidoka-hf-matomo   Synced   Healthy   585107236d5ec2abe5b8ffd591fb095a444dcb7a   Succeeded
pidoka-hf-matomo   Synced   Healthy   585107236d5ec2abe5b8ffd591fb095a444dcb7a   Succeeded
pidoka-hf-matomo   Synced   Healthy   585107236d5ec2abe5b8ffd591fb095a444dcb7a   Succeeded
pidoka-hf-matomo   Synced   Healthy   585107236d5ec2abe5b8ffd591fb095a444dcb7a   Succeeded
pidoka-hf-matomo   Synced   Healthy   585107236d5ec2abe5b8ffd591fb095a444dcb7a   Succeeded


Output:
Chunk ID: f919f8
Wall time: 10.0639 seconds
Process exited with code 0
Original token count: 205
Output:
pidoka-hf-matomo   matomo-ingress      traefik   matomo.heckenfels.com     192.168.20.40,192.168.20.45   80      28s
pidoka-hf-matomo   matomo-ingress      traefik   matomo.heckenfels.com     192.168.20.40,192.168.20.45   80      34s
pidoka-hf-matomo   matomo-ingress      traefik   matomo.heckenfels.com     192.168.20.40,192.168.20.45   80      40s
pidoka-hf-matomo   matomo-ingress      traefik   matomo.heckenfels.com     192.168.20.40,192.168.20.45   80      47s
pidoka-hf-matomo   matomo-ingress      traefik   matomo.heckenfels.com     192.168.20.40,192.168.20.45   80      54s
pidoka-hf-matomo   matomo-ingress      traefik   matomo.heckenfels.com     192.168.20.40,192.168.20.45   80      60s
pidoka-hf-matomo   matomo-ingress      traefik   matomo.heckenfels.com     192.168.20.40,192.168.20.45   80      66s


Ingress-Umschaltung ist sichtbar stabil. Ich teste jetzt intern per Host-Header gegen Traefik und prüfe, dass alte App weiterhin eingefroren bleibt.


ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 'curl -sS -o /tmp/matomo-cutover-http.out -w "traefik_40=%{http_code}\n" -H "Host: matomo.heckenfels.com" http://192.168.20.40/; head -c 120 /tmp/matomo-cutover-http.out | tr "\n" " " ; echo; curl -sS -o /tmp/matomo-cutover-http2.out -w "traefik_45=%{http_code}\n" -H "Host: matomo.heckenfels.com" http://192.168.20.45/; head -c 120 /tmp/matomo-cutover-http2.out | tr "\n" " " ; echo'


ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 'sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n hf-matomo get deploy,pod,ingress -o wide; sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n pidoka-hf-matomo get deploy,pod,svc,ingress,pvc -o wide'


ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 'sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n pidoka-hf-matomo logs deployment/matomo --tail=80 | sed -E "s/(password|PASSWORD|secret|SECRET|key|KEY)([= :]+)[^[:space:]]+/\\1\\2[REDACTED]/g"'


Output:
Chunk ID: c00e68
Wall time: 1.5910 seconds
Process exited with code 0
Original token count: 18
Output:
traefik_40=404
404 page not found
traefik_45=404
404 page not found


Output:
Chunk ID: 061a51
Wall time: 1.5927 seconds
Process exited with code 0
Original token count: 599
Output:
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS   IMAGES           SELECTOR
deployment.apps/matomo-db   1/1     1            1           282d   mariadb      mariadb:latest   app=matomo-db
deployment.apps/matomo      0/0     0            0           282d   matomo       matomo:latest    app=matomo

NAME                             READY   STATUS    RESTARTS   AGE   IP           NODE                 NOMINATED NODE   READINESS GATES
pod/matomo-db-6766699c55-xjg8d   1/1     Running   0          45d   10.42.0.61   mbrkanov-mini-it13   <none>           <none>
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                                                                                SELECTOR
deployment.apps/matomo-db   0/0     0            0           7h40m   mariadb      docker.io/library/mariadb@sha256:e0236fc6386e7eacd9359e59d0a078bd7aa0d18280d36d13061121bedeaee903   app=matomo-db
deployment.apps/matomo      0/0     0            0           7h40m   matomo       docker.io/library/matomo@sha256:21b8bae018cd51a05f36bc6beac3861bb314d131df690c47bba1648aad2b8d7e    app=matomo

NAME                        READY   STATUS      RESTARTS   AGE   IP           NODE                 NOMINATED NODE   READINESS GATES
pod/matomo-db-pvc-cleanup   0/1     Completed   0          16m   10.42.0.71   mbrkanov-mini-it13   <none>           <none>

NAME                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE     SELECTOR
service/matomo      ClusterIP   10.43.37.72     <none>        80/TCP     7h41m   app=matomo
service/matomo-db   ClusterIP   10.43.215.111   <none>        3306/TCP   7h41m   app=matomo-db

NAME                                       CLASS     HOSTS                   ADDRESS                       PORTS   AGE
ingress.networking.k8s.io/matomo-ingress   traefik   matomo.heckenfels.com   192.168.20.40,192.168.20.45   80      89s

NAME                                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE     VOLUMEMODE
persistentvolumeclaim/matomo-db-pvc   Bound    pvc-2c086a12-6f3b-494a-b982-e2405f7649af   1Gi        RWO            longhorn       7h40m   Filesystem
persistentvolumeclaim/matomo-pvc      Bound    pvc-e96e1f56-a534-49e2-b2ee-b39c6e523af5   1Gi        RWO            longhorn       7h40m   Filesystem


Output:
Chunk ID: 360798
Wall time: 10.0006 seconds
Process running with session ID 69287
Original token count: 0
Output:


Ich sehe den Fehler: ArgoCD hat beim Sync die neuen Deployments wieder auf `replicas=0` gesetzt, weil das noch so im GitOps-Manifest steht. Ingress zeigt daher 404. Ich korrigiere jetzt GitOps auf `replicas=1` für neue Matomo-App und DB, pushe und sync erneut.


Output:
Chunk ID: 95dc9c
Wall time: 1.8452 seconds
Process exited with code 0
Original token count: 11
Output:
error: timed out waiting for the condition


Explanation: Ingress ist umgeschaltet, GitOps muss noch replicas=1 setzen, damit der neue Service Backends hat.


Output: Plan updated


Ich korrigiere nur Matomo `replicas` im GitOps-Manifest; n8n bleibt weiterhin unangetastet.


Input:
*** Begin Patch
*** Update File: /home/node/.openclaw/workspace/repos/platform-gitops/tenants/pidoka/prod/matomo/app.yaml
@@
-  replicas: 0
+  replicas: 1
@@
-  replicas: 0
+  replicas: 1
*** End Patch


@@ -40,3 +40,3 @@
 spec:
-  replicas: 0
+  replicas: 1
   selector:
@@ -128,3 +128,3 @@
 spec:
-  replicas: 0
+  replicas: 1
   selector:


Output:
Exit code: 0
Wall time: 0 seconds
Output:
Success. Updated the following files:
M /home/node/.openclaw/workspace/repos/platform-gitops/tenants/pidoka/prod/matomo/app.yaml


git -C /home/node/.openclaw/workspace/repos/platform-gitops diff -- tenants/pidoka/prod/matomo/app.yaml
git -C /home/node/.openclaw/workspace/repos/platform-gitops add tenants/pidoka/prod/matomo/app.yaml && git -C /home/node/.openclaw/workspace/repos/platform-gitops commit -m "Enable Matomo cutover workloads"
set +x
python3 - <<'PY' > /tmp/git-basic-header
import base64, json, os, urllib.request, urllib.parse
it15='http://192.168.1.121:8088'
def req(method,url,token=None,data=None):
    h={'Content-Type':'application/json'}
    if token: h['Authorization']='Bearer '+token
    r=urllib.request.Request(url, data=None if data is None else json.dumps(data).encode(), headers=h, method=method)
    with urllib.request.urlopen(r, timeout=15) as resp: return json.loads(resp.read().decode() or '{}')
login=req('POST',it15+'/api/v1/auth/universal-auth/login',data={'clientId':os.environ['INFISICAL_CLIENT_ID'],'clientSecret':os.environ['INFISICAL_AUTH_KEY']})
qs=urllib.parse.urlencode({'projectId':'c5e8661f-4b7a-43e1-80b0-8674f30950dc','environment':'prod','secretPath':'/platform/gitops/github'})
obj=req('GET',it15+'/api/v4/secrets/GITHUB_PAT_PLATFORM_GITOPS?'+qs,login['accessToken'])
tok=obj['secret']['secretValue']
print('AUTHORIZATION: basic '+base64.b64encode(('x-access-token:'+tok).encode()).decode())
PY
hdr=$(cat /tmp/git-basic-header)
rm -f /tmp/git-basic-header
git -C /home/node/.openclaw/workspace/repos/platform-gitops -c http.https://github.com/.extraheader="$hdr" push


Output:
Chunk ID: 13c1a9
Wall time: 1.1441 seconds
Process exited with code 0
Original token count: 183
Output:
diff --git a/tenants/pidoka/prod/matomo/app.yaml b/tenants/pidoka/prod/matomo/app.yaml
index 55358a8..bd57021 100644
--- a/tenants/pidoka/prod/matomo/app.yaml
+++ b/tenants/pidoka/prod/matomo/app.yaml
@@ -38,7 +38,7 @@ metadata:
   annotations:
     argocd.argoproj.io/sync-wave: "3"
 spec:
-  replicas: 0
+  replicas: 1
   selector:
     matchLabels:
       app: matomo-db
@@ -126,7 +126,7 @@ metadata:
   annotations:
     argocd.argoproj.io/sync-wave: "4"
 spec:
-  replicas: 0
+  replicas: 1
   selector:
     matchLabels:
       app: matomo
[main d625118] Enable Matomo cutover workloads
 1 file changed, 2 insertions(+), 2 deletions(-)
To https://github.com/GitMasterOne/platform-gitops.git
   5851072..d625118  main -> main


Commit `d625118` ist gepusht. Ich synchronisiere jetzt erneut; danach sollte der Ingress Backends haben.


ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new Misla@192.168.1.121 'wsl.exe -d Ubuntu -e bash -lc "kubectl -n argocd annotate application pidoka-hf-matomo argocd.argoproj.io/refresh=hard --overwrite && kubectl -n argocd patch application pidoka-hf-matomo --type merge -p '\''{\"operation\":{\"sync\":{\"revision\":\"main\"}}}'\''"'


Output:
Chunk ID: 05a379
Wall time: 0.7452 seconds
Process exited with code 0
Original token count: 25
Output:
application.argoproj.io/pidoka-hf-matomo annotated
application.argoproj.io/pidoka-hf-matomo patched


Sync läuft. Ich prüfe jetzt ArgoCD, Pods und den Host-Test in kurzen Polls.


for i in $(seq 1 15); do echo poll=$i; ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new Misla@192.168.1.121 'wsl.exe -d Ubuntu -e bash -lc "kubectl -n argocd get application pidoka-hf-matomo -o custom-columns=SYNC:.status.sync.status,HEALTH:.status.health.status,REV:.status.sync.revision,PHASE:.status.operationState.phase --no-headers"'; ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 'sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n pidoka-hf-matomo get deploy,pod -o wide'; sleep 8; done


for i in $(seq 1 15); do echo poll=$i; ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 'curl -sS -o /tmp/matomo-cutover-http.out -w "http40=%{http_code}\n" -H "Host: matomo.heckenfels.com" http://192.168.20.40/; curl -sS -o /tmp/matomo-cutover-http2.out -w "http45=%{http_code}\n" -H "Host: matomo.heckenfels.com" http://192.168.20.45/'; sleep 8; done


Output:
Chunk ID: bd626a
Wall time: 30.0010 seconds
Process running with session ID 91031
Original token count: 814
Output:
poll=1
OutOfSync   Progressing   d6251183310cc6e3446fb2bb77d91450ddfb66e4   Running
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                                                                                SELECTOR
deployment.apps/matomo      0/0     0            0           7h41m   matomo       docker.io/library/matomo@sha256:21b8bae018cd51a05f36bc6beac3861bb314d131df690c47bba1648aad2b8d7e    app=matomo
deployment.apps/matomo-db   0/1     1            0           7h41m   mariadb      docker.io/library/mariadb@sha256:e0236fc6386e7eacd9359e59d0a078bd7aa0d18280d36d13061121bedeaee903   app=matomo-db

NAME                             READY   STATUS              RESTARTS   AGE   IP           NODE                 NOMINATED NODE   READINESS GATES
pod/matomo-db-pvc-cleanup        0/1     Completed           0          18m   10.42.0.71   mbrkanov-mini-it13   <none>           <none>
pod/matomo-db-7b9bdc7b94-sk4b6   0/1     ContainerCreating   0          5s    <none>       mini-it13-02         <none>           <none>
poll=2
OutOfSync   Progressing   d6251183310cc6e3446fb2bb77d91450ddfb66e4   Running
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                                                                                SELECTOR
deployment.apps/matomo      0/0     0            0           7h41m   matomo       docker.io/library/matomo@sha256:21b8bae018cd51a05f36bc6beac3861bb314d131df690c47bba1648aad2b8d7e    app=matomo
deployment.apps/matomo-db   0/1     1            0           7h41m   mariadb      docker.io/library/mariadb@sha256:e0236fc6386e7eacd9359e59d0a078bd7aa0d18280d36d13061121bedeaee903   app=matomo-db

NAME                             READY   STATUS              RESTARTS   AGE   IP           NODE                 NOMINATED NODE   READINESS GATES
pod/matomo-db-pvc-cleanup        0/1     Completed           0          18m   10.42.0.71   mbrkanov-mini-it13   <none>           <none>
pod/matomo-db-7b9bdc7b94-sk4b6   0/1     ContainerCreating   0          15s   <none>       mini-it13-02         <none>           <none>
poll=3
OutOfSync   Progressing   d6251183310cc6e3446fb2bb77d91450ddfb66e4   Running
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                                                                                SELECTOR
deployment.apps/matomo      0/0     0            0           7h41m   matomo       docker.io/library/matomo@sha256:21b8bae018cd51a05f36bc6beac3861bb314d131df690c47bba1648aad2b8d7e    app=matomo
deployment.apps/matomo-db   0/1     1            0           7h41m   mariadb      docker.io/library/mariadb@sha256:e0236fc6386e7eacd9359e59d0a078bd7aa0d18280d36d13061121bedeaee903   app=matomo-db

NAME                             READY   STATUS              RESTARTS   AGE   IP           NODE                 NOMINATED NODE   READINESS GATES
pod/matomo-db-pvc-cleanup        0/1     Completed           0          18m   10.42.0.71   mbrkanov-mini-it13   <none>           <none>
pod/matomo-db-7b9bdc7b94-sk4b6   0/1     ContainerCreating   0          25s   <none>       mini-it13-02         <none>           <none>
poll=4


Output:
Chunk ID: fc7f3a
Wall time: 30.0013 seconds
Process running with session ID 41499
Original token count: 29
Output:
poll=1
http40=404
http45=404
poll=2
http40=404
http45=404
poll=3
http40=404
http45=404
poll=4
http40=404
http45=404


Die neue DB startet noch, deshalb bleibt der Host aktuell 404. Das ist erwartbar, bis ArgoCD erst DB und dann App hochgezogen hat. Ich bleibe im Poll.


Output:
Chunk ID: 64140b
Wall time: 60.0016 seconds
Process running with session ID 91031
Original token count: 2081
Output:
OutOfSync   Progressing   d6251183310cc6e3446fb2bb77d91450ddfb66e4   Running
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                                                                                SELECTOR
deployment.apps/matomo      0/0     0            0           7h41m   matomo       docker.io/library/matomo@sha256:21b8bae018cd51a05f36bc6beac3861bb314d131df690c47bba1648aad2b8d7e    app=matomo
deployment.apps/matomo-db   0/1     1            0           7h42m   mariadb      docker.io/library/mariadb@sha256:e0236fc6386e7eacd9359e59d0a078bd7aa0d18280d36d13061121bedeaee903   app=matomo-db

NAME                             READY   STATUS              RESTARTS   AGE   IP           NODE                 NOMINATED NODE   READINESS GATES
pod/matomo-db-pvc-cleanup        0/1     Completed           0          18m   10.42.0.71   mbrkanov-mini-it13   <none>           <none>
pod/matomo-db-7b9bdc7b94-sk4b6   0/1     ContainerCreating   0          35s   <none>       mini-it13-02         <none>           <none>
poll=5
Synced   Progressing   d6251183310cc6e3446fb2bb77d91450ddfb66e4   Running
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                                                                                SELECTOR
deployment.apps/matomo-db   1/1     1            1           7h42m   mariadb      docker.io/library/mariadb@sha256:e0236fc6386e7eacd9359e59d0a078bd7aa0d18280d36d13061121bedeaee903   app=matomo-db
deployment.apps/matomo      0/1     1            0           7h42m   matomo       docker.io/library/matomo@sha256:21b8bae018cd51a05f36bc6beac3861bb314d131df690c47bba1648aad2b8d7e    app=matomo

NAME                             READY   STATUS              RESTARTS   AGE   IP            NODE                 NOMINATED NODE   READINESS GATES
pod/matomo-db-pvc-cleanup        0/1     Completed           0          18m   10.42.0.71    mbrkanov-mini-it13   <none>           <none>
pod/matomo-db-7b9bdc7b94-sk4b6   1/1     Running             0          45s   10.42.1.215   mini-it13-02         <none>           <none>
pod/matomo-6fb4766964-8gb4k      0/1     ContainerCreating   0          2s    <none>        mini-it13-02         <none>           <none>
poll=6
Synced   Progressing   d6251183310cc6e3446fb2bb77d91450ddfb66e4   Running
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                                                                                SELECTOR
deployment.apps/matomo-db   1/1     1            1           7h42m   mariadb      docker.io/library/mariadb@sha256:e0236fc6386e7eacd9359e59d0a078bd7aa0d18280d36d13061121bedeaee903   app=matomo-db
deployment.apps/matomo      0/1     1            0           7h42m   matomo       docker.io/library/matomo@sha256:21b8bae018cd51a05f36bc6beac3861bb314d131df690c47bba1648aad2b8d7e    app=matomo

NAME                             READY   STATUS              RESTARTS   AGE   IP            NODE                 NOMINATED NODE   READINESS GATES
pod/matomo-db-pvc-cleanup        0/1     Completed           0          19m   10.42.0.71    mbrkanov-mini-it13   <none>           <none>
pod/matomo-db-7b9bdc7b94-sk4b6   1/1     Running             0          56s   10.42.1.215   mini-it13-02         <none>           <none>
pod/matomo-6fb4766964-8gb4k      0/1     ContainerCreating   0          13s   <none>        mini-it13-02         <none>           <none>
poll=7
Synced   Progressing   d6251183310cc6e3446fb2bb77d91450ddfb66e4   Running
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                                                                                SELECTOR
deployment.apps/matomo-db   1/1     1            1           7h42m   mariadb      docker.io/library/mariadb@sha256:e0236fc6386e7eacd9359e59d0a078bd7aa0d18280d36d13061121bedeaee903   app=matomo-db
deployment.apps/matomo      0/1     1            0           7h42m   matomo       docker.io/library/matomo@sha256:21b8bae018cd51a05f36bc6beac3861bb314d131df690c47bba1648aad2b8d7e    app=matomo

NAME                             READY   STATUS              RESTARTS   AGE   IP            NODE                 NOMINATED NODE   READINESS GATES
pod/matomo-db-pvc-cleanup        0/1     Completed           0          19m   10.42.0.71    mbrkanov-mini-it13   <none>           <none>
pod/matomo-db-7b9bdc7b94-sk4b6   1/1     Running             0          66s   10.42.1.215   mini-it13-02         <none>           <none>
pod/matomo-6fb4766964-8gb4k      0/1     ContainerCreating   0          23s   <none>        mini-it13-02         <none>           <none>
poll=8
Synced   Progressing   d6251183310cc6e3446fb2bb77d91450ddfb66e4   Running
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                                                                                SELECTOR
deployment.apps/matomo-db   1/1     1            1           7h42m   mariadb      docker.io/library/mariadb@sha256:e0236fc6386e7eacd9359e59d0a078bd7aa0d18280d36d13061121bedeaee903   app=matomo-db
deployment.apps/matomo      0/1     1            0           7h42m   matomo       docker.io/library/matomo@sha256:21b8bae018cd51a05f36bc6beac3861bb314d131df690c47bba1648aad2b8d7e    app=matomo

NAME                             READY   STATUS              RESTARTS   AGE   IP            NODE                 NOMINATED NODE   READINESS GATES
pod/matomo-db-pvc-cleanup        0/1     Completed           0          19m   10.42.0.71    mbrkanov-mini-it13   <none>           <none>
pod/matomo-db-7b9bdc7b94-sk4b6   1/1     Running             0          76s   10.42.1.215   mini-it13-02         <none>           <none>
pod/matomo-6fb4766964-8gb4k      0/1     ContainerCreating   0          33s   <none>        mini-it13-02         <none>           <none>
poll=9
Synced   Healthy   d6251183310cc6e3446fb2bb77d91450ddfb66e4   Succeeded
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                                                                                SELECTOR
deployment.apps/matomo-db   1/1     1            1           7h42m   mariadb      docker.io/library/mariadb@sha256:e0236fc6386e7eacd9359e59d0a078bd7aa0d18280d36d13061121bedeaee903   app=matomo-db
deployment.apps/matomo      1/1     1            1           7h42m   matomo       docker.io/library/matomo@sha256:21b8bae018cd51a05f36bc6beac3861bb314d131df690c47bba1648aad2b8d7e    app=matomo

NAME                             READY   STATUS      RESTARTS   AGE   IP            NODE                 NOMINATED NODE   READINESS GATES
pod/matomo-db-pvc-cleanup        0/1     Completed   0          19m   10.42.0.71    mbrkanov-mini-it13   <none>           <none>
pod/matomo-db-7b9bdc7b94-sk4b6   1/1     Running     0          86s   10.42.1.215   mini-it13-02         <none>           <none>
pod/matomo-6fb4766964-8gb4k      1/1     Running     0          43s   10.42.1.216   mini-it13-02         <none>           <none>
poll=10
Synced   Healthy   d6251183310cc6e3446fb2bb77d91450ddfb66e4   Succeeded
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                                                                                SELECTOR
deployment.apps/matomo-db   1/1     1            1           7h43m   mariadb      docker.io/library/mariadb@sha256:e0236fc6386e7eacd9359e59d0a078bd7aa0d18280d36d13061121bedeaee903   app=matomo-db
deployment.apps/matomo      1/1     1            1           7h43m   matomo       docker.io/library/matomo@sha256:21b8bae018cd51a05f36bc6beac3861bb314d131df690c47bba1648aad2b8d7e    app=matomo

NAME                             READY   STATUS      RESTARTS   AGE   IP            NODE                 NOMINATED NODE   READINESS GATES
pod/matomo-db-pvc-cleanup        0/1     Completed   0          19m   10.42.0.71    mbrkanov-mini-it13   <none>           <none>
pod/matomo-db-7b9bdc7b94-sk4b6   1/1     Running     0          97s   10.42.1.215   mini-it13-02         <none>           <none>
pod/matomo-6fb4766964-8gb4k      1/1     Running     0          54s   10.42.1.216   mini-it13-02         <none>           <none>


Output:
Chunk ID: 8ac7e8
Wall time: 50.5773 seconds
Process exited with code 0
Original token count: 82
Output:
poll=5
http40=404
http45=404
poll=6
http40=404
http45=404
poll=7
http40=404
http45=404
poll=8
http40=404
http45=404
poll=9
http40=404
http45=404
poll=10
http40=200
http45=200
poll=11
http40=200
http45=200
poll=12
http40=200
http45=200
poll=13
http40=200
http45=200
poll=14
http40=200
http45=200
poll=15
http40=200
http45=200


Cutover ist jetzt funktional: ArgoCD `Synced / Healthy`, beide neuen Pods `1/1`, und Traefik auf beiden Nodes liefert HTTP `200`. Ich mache jetzt Schlusschecks: alte Produktion eingefroren, neue Logs, DB-Zähler, externe Domain soweit erreichbar.


ssh -i /home/node/.ssh/kube-bot-it15-ed25519 -o BatchMode=yes -o StrictHostKeyChecking=accept-new kube-bot@192.168.20.40 'sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n pidoka-hf-matomo exec deployment/matomo-db -- sh -lc '\''mariadb -h127.0.0.1 -P3306 -uroot -p"$MARIADB_ROOT_PASSWORD" "$MARIADB_DATABASE" -N -e "SELECT COUNT(*) FROM information_schema.tables WHERE table_schema=DATABASE(); SELECT COUNT(*) FROM matomo_site;"'\''; sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n pidoka-hf-matomo logs deployment/matomo --tail=80 | sed -E "s/(password|PASSWORD|secret|SECRET|key|KEY)([= :]+)[^[:space:]]+/\\1\\2[REDACTED]/g"'


curl -sS -o /tmp/matomo-public.out -w 'public_http=%{http_code}\n' --max-time 15 http://matomo.heckenfels.com/ || true; head -c 120 /tmp/matomo-public.out 2>/dev/null | tr '\n' ' '; echo


Output:
Chunk ID: 114689
Wall time: 0.0000 seconds
Process exited with code 0
Original token count: 1490
Output:
poll=11
Synced   Healthy   d6251183310cc6e3446fb2bb77d91450ddfb66e4   Succeeded
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                                                                                SELECTOR
deployment.apps/matomo-db   1/1     1            1           7h43m   mariadb      docker.io/library/mariadb@sha256:e0236fc6386e7eacd9359e59d0a078bd7aa0d18280d36d13061121bedeaee903   app=matomo-db
deployment.apps/matomo      1/1     1            1           7h43m   matomo       docker.io/library/matomo@sha256:21b8bae018cd51a05f36bc6beac3861bb314d131df690c47bba1648aad2b8d7e    app=matomo

NAME                             READY   STATUS      RESTARTS   AGE    IP            NODE                 NOMINATED NODE   READINESS GATES
pod/matomo-db-pvc-cleanup        0/1     Completed   0          19m    10.42.0.71    mbrkanov-mini-it13   <none>           <none>
pod/matomo-db-7b9bdc7b94-sk4b6   1/1     Running     0          106s   10.42.1.215   mini-it13-02         <none>           <none>
pod/matomo-6fb4766964-8gb4k      1/1     Running     0          63s    10.42.1.216   mini-it13-02         <none>           <none>
poll=12
Synced   Healthy   d6251183310cc6e3446fb2bb77d91450ddfb66e4   Succeeded
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                                                                                SELECTOR
deployment.apps/matomo-db   1/1     1            1           7h43m   mariadb      docker.io/library/mariadb@sha256:e0236fc6386e7eacd9359e59d0a078bd7aa0d18280d36d13061121bedeaee903   app=matomo-db
deployment.apps/matomo      1/1     1            1           7h43m   matomo       docker.io/library/matomo@sha256:21b8bae018cd51a05f36bc6beac3861bb314d131df690c47bba1648aad2b8d7e    app=matomo

NAME                             READY   STATUS      RESTARTS   AGE    IP            NODE                 NOMINATED NODE   READINESS GATES
pod/matomo-db-pvc-cleanup        0/1     Completed   0          20m    10.42.0.71    mbrkanov-mini-it13   <none>           <none>
pod/matomo-db-7b9bdc7b94-sk4b6   1/1     Running     0          116s   10.42.1.215   mini-it13-02         <none>           <none>
pod/matomo-6fb4766964-8gb4k      1/1     Running     0          73s    10.42.1.216   mini-it13-02         <none>           <none>
poll=13
Synced   Healthy   d6251183310cc6e3446fb2bb77d91450ddfb66e4   Succeeded
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                                                                                SELECTOR
deployment.apps/matomo-db   1/1     1            1           7h43m   mariadb      docker.io/library/mariadb@sha256:e0236fc6386e7eacd9359e59d0a078bd7aa0d18280d36d13061121bedeaee903   app=matomo-db
deployment.apps/matomo      1/1     1            1           7h43m   matomo       docker.io/library/matomo@sha256:21b8bae018cd51a05f36bc6beac3861bb314d131df690c47bba1648aad2b8d7e    app=matomo

NAME                             READY   STATUS      RESTARTS   AGE    IP            NODE                 NOMINATED NODE   READINESS GATES
pod/matomo-db-pvc-cleanup        0/1     Completed   0          20m    10.42.0.71    mbrkanov-mini-it13   <none>           <none>
pod/matomo-db-7b9bdc7b94-sk4b6   1/1     Running     0          2m5s   10.42.1.215   mini-it13-02         <none>           <none>
pod/matomo-6fb4766964-8gb4k      1/1     Running     0          82s    10.42.1.216   mini-it13-02         <none>           <none>
poll=14
Synced   Healthy   d6251183310cc6e3446fb2bb77d91450ddfb66e4   Succeeded
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                                                                                SELECTOR
deployment.apps/matomo-db   1/1     1            1           7h43m   mariadb      docker.io/library/mariadb@sha256:e0236fc6386e7eacd9359e59d0a078bd7aa0d18280d36d13061121bedeaee903   app=matomo-db
deployment.apps/matomo      1/1     1            1           7h43m   matomo       docker.io/library/matomo@sha256:21b8bae018cd51a05f36bc6beac3861bb314d131df690c47bba1648aad2b8d7e    app=matomo

NAME                             READY   STATUS      RESTARTS   AGE     IP            NODE                 NOMINATED NODE   READINESS GATES
pod/matomo-db-pvc-cleanup        0/1     Completed   0          20m     10.42.0.71    mbrkanov-mini-it13   <none>           <none>
pod/matomo-db-7b9bdc7b94-sk4b6   1/1     Running     0          2m15s   10.42.1.215   mini-it13-02         <none>           <none>
pod/matomo-6fb4766964-8gb4k      1/1     Running     0          92s     10.42.1.216   mini-it13-02         <none>           <none>
poll=15
Synced   Healthy   d6251183310cc6e3446fb2bb77d91450ddfb66e4   Succeeded
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                                                                                SELECTOR
deployment.apps/matomo-db   1/1     1            1           7h43m   mariadb      docker.io/library/mariadb@sha256:e0236fc6386e7eacd9359e59d0a078bd7aa0d18280d36d13061121bedeaee903   app=matomo-db
deployment.apps/matomo      1/1     1            1           7h43m   matomo       docker.io/library/matomo@sha256:21b8bae018cd51a05f36bc6beac3861bb314d131df690c47bba1648aad2b8d7e    app=matomo

NAME                             READY   STATUS      RESTARTS   AGE     IP            NODE                 NOMINATED NODE   READINESS GATES
pod/matomo-db-pvc-cleanup        0/1     Completed   0          20m     10.42.0.71    mbrkanov-mini-it13   <none>           <none>
pod/matomo-db-7b9bdc7b94-sk4b6   1/1     Running     0          2m25s   10.42.1.215   mini-it13-02         <none>           <none>
pod/matomo-6fb4766964-8gb4k      1/1     Running     0          102s    10.42.1.216   mini-it13-02         <none>           <none>