[08:21:19] I was paged with "HarborComponentDown" but it resolved by itself in a few mins [08:21:37] looks like tools-static is also down, and that did not page [08:24:38] I can ssh to tools-static-15 and nothing seems obviously broken [08:25:20] I'll try rebooting that VM anyway [08:29:32] the reboot fixed it ¯\_(ツ)_/¯ [09:22:22] I think it might have been stuck on nfs: [09:22:23] `Jun 21 08:25:31 tools-static-15 systemd[1]: Failed unmounting mnt-nfs-labstore\x2dsecondary\x2dtools\x2dproject.mount - /mnt/nfs/labstore-secondary-tools-project.` [09:23:47] hiccup last night it seems `Jun 21 00:13:30 tools-static-15 kernel: nfs: server tools-nfs.svc.tools.eqiad1.wikimedia.cloud not responding, still trying` [09:28:05] tools-prometheus-8 had issues too it seems :/ [09:55:33] started T397563, it seems to me that there was some network issue that affected at least the tool-prometheus-8 vm [09:55:34] T397563: [infra] 2025-06-21 tools-prometheus-8 stopped responding for a bit - https://phabricator.wikimedia.org/T397563 [10:04:12] Hmm... during the nsf hiccup, there was also some other issues, like gitlab runners failed to pull images it seems [10:04:13] https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-backend/-/jobs/542326 [10:04:27] I'll open a task for the hiccup too [10:14:57] created T397566 to try to see if there's any relation between things that happened tonight :/ [10:14:58] T397566: [infra] 2025-06-21 Several correlated poetntially network issues during the night - https://phabricator.wikimedia.org/T397566