[01:12:19] FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [02:07:19] RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [02:15:19] FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [03:10:19] RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [06:24:08] (03update) 10samwilson: Add GitLab CI to test extension installation [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/2 [06:40:32] (03update) 10samwilson: Add GitLab CI to test extension installation [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/2 [06:42:16] 10Tool-archive-externa-links, 10Wikidata, 10Wikidata-Gadgets: [Documentation] Le service Internet Archive ne parvient pas à archiver tous les liens externes via ArchiveExternaLinks - https://phabricator.wikimedia.org/T401760#11133614 (10poro26) 05Open→03Resolved [06:48:12] (03update) 10samwilson: Add GitLab CI to test extension installation [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/2 [07:16:50] (03update) 10samwilson: Add GitLab CI to test extension installation [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/2 [07:38:06] (03update) 10samwilson: Add GitLab CI to test extension installation [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/2 [08:14:50] FIRING: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:19:51] RESOLVED: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:05:04] (03update) 10samwilson: Add GitLab CI to test extension installation [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/2 [10:17:07] (03update) 10samwilson: Add GitLab CI to test extension installation [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/2 [11:05:26] (03update) 10samwilson: Add GitLab CI to test extension installation [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/2 [11:14:25] (03update) 10samwilson: Add GitLab CI to test extension installation [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/2 [11:20:15] (03update) 10samwilson: Add GitLab CI to test extension installation [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/2 [11:20:54] (03update) 10damian: Ensure reuse_from components are re-run [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/125 (https://phabricator.wikimedia.org/T403285) [11:25:08] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [components-api] reuse_from components are not explicitly re-created in jobs-api - https://phabricator.wikimedia.org/T403285#11133718 (10DamianZaremba) Added a test case - currently there is no test coverage of the delete + recreate logic (`jobs` is alw... [11:34:42] (03update) 10damian: Add validated type for git urls [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/121 [12:02:50] (03open) 10damian: Add validated type for git urls [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/127 (https://phabricator.wikimedia.org/T402764) [12:04:43] (03update) 10damian: Add `source` support to ToolConfig [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/127 (https://phabricator.wikimedia.org/T402764) [12:05:26] (03open) 10damian: _resolve_ref - use GitPython [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/128 [12:06:50] 06cloud-services-team, 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [components-api] allow specifying `source_repo`+`ref` for the config - https://phabricator.wikimedia.org/T402764#11133741 (10DamianZaremba) Rebased for the conflicts from `reuse_from`. Moved all MRs to the target repo - will ne... [12:14:36] 06cloud-services-team, 10Toolforge: [webservice] returning default 404 even though webservice is healthy - https://phabricator.wikimedia.org/T403168#11133754 (10DamianZaremba) >>! In T403168#11133440, @bd808 wrote: >>>! In T403168#11131903, @DamianZaremba wrote: >> Health check is returning a good response: >>... [12:16:21] (03update) 10samwilson: Add GitLab CI to test extension installation [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/2 [15:13:56] 06cloud-services-team, 10Toolforge, 07Upstream: Build service tools built with --use-latest-versions are missing /etc/mime.types - https://phabricator.wikimedia.org/T403303#11133849 (10LucasWerkmeister) Okay, it’s also reproducible with the upstream builders, `pack-cli build --builder docker.io/heroku/builde... [15:27:45] 06cloud-services-team, 10Toolforge: [components-api] restart rather than delete/create continuous jobs - https://phabricator.wikimedia.org/T403321 (10DamianZaremba) 03NEW [15:29:35] 06cloud-services-team, 10Toolforge: [components-api] restart rather than delete/create continuous jobs - https://phabricator.wikimedia.org/T403321#11133870 (10DamianZaremba) Rather than changing `delete_job_if_exists` to `restart_job_if_exists` which would cause it to be restarted twice if the job configuratio... [15:32:32] 06cloud-services-team, 10Toolforge, 07Upstream: Build service tools built with --use-latest-versions are missing /etc/mime.types - https://phabricator.wikimedia.org/T403303#11133872 (10LucasWerkmeister) [15:37:16] 06cloud-services-team, 10Toolforge: [components-api] restart rather than delete/create continuous jobs - https://phabricator.wikimedia.org/T403321#11133875 (10DamianZaremba) [15:38:41] 06cloud-services-team, 10Toolforge: [components-api] restart rather than delete/create continuous jobs - https://phabricator.wikimedia.org/T403321#11133879 (10DamianZaremba) Checking `jobs-api` the `PATCH` endpoint actually just calls `delete` and `create` rather than doing an update.... so this problem goes a... [16:01:14] 06cloud-services-team, 10Toolforge: [builds-api] return image digest - https://phabricator.wikimedia.org/T403322 (10DamianZaremba) 03NEW [16:08:22] 10VPS-project-Codesearch: Codesearch down/unreachable (2025-08-30) - https://phabricator.wikimedia.org/T403323 (10LucasWerkmeister) 03NEW [16:10:32] 06cloud-services-team, 10Toolforge: [builds-api] return image digest - https://phabricator.wikimedia.org/T403322#11133917 (10DamianZaremba) This is already parsed out of the report generated by pack and shoved into the logs by `./deployment/chart/scripts/results.sh` Tekton supports `results`, so that logic ca... [16:16:41] 10VPS-project-Codesearch: Codesearch down/unreachable (2025-08-30) - https://phabricator.wikimedia.org/T403323#11133918 (10LucasWerkmeister) Huh. The tab just loaded for me… after waiting for something like ten minutes. Did anyone do something? [16:19:57] 06cloud-services-team, 10Toolforge: [webservice] returning default 404 even though webservice is healthy - https://phabricator.wikimedia.org/T403168#11133920 (10DamianZaremba) Adjusted the webservice config so negative dns responses are basically ignored, which is a workaround for jobs-api/components-api doing... [16:27:41] 06cloud-services-team, 10Toolforge: [webservice] returning default 404 even though webservice is healthy - https://phabricator.wikimedia.org/T403168#11133927 (10DamianZaremba) Just as a record of this behaviour (also describe in T403321): ` [WARNING] (1) : Server kubernetes/service is DOWN, reason: Layer4 tim... [16:31:19] 10VPS-project-Codesearch: Codesearch down/unreachable (2025-08-30) - https://phabricator.wikimedia.org/T403323#11133930 (10SomeRandomDeveloper) [16:54:32] 10Tool-techcontribs: add "Shell groups" - https://phabricator.wikimedia.org/T403325 (10Novem_Linguae) 03NEW [18:02:06] 10Tool-techcontribs: add "Shell groups" - https://phabricator.wikimedia.org/T403325#11134008 (10taavi) →14Duplicate dup:03T393322 [18:02:07] 10Tool-techcontribs: Show admin groups in tech-contribs - https://phabricator.wikimedia.org/T393322#11134010 (10taavi) [18:44:28] 10VPS-project-Codesearch: Codesearch down/unreachable (2025-08-30) - https://phabricator.wikimedia.org/T403323#11134019 (10Ladsgroup) 05Open→03Resolved It's back online now. On its own. I didn't do anything. Mayyybe related (kern.log): ` 2025-08-30T17:49:03.566472+00:00 codesearch9 kernel: [1141187.60789... [19:28:32] 10VPS-project-Codesearch: Codesearch down/unreachable (2025-08-30) - https://phabricator.wikimedia.org/T403323#11134044 (10A_smart_kitten) > If someone is willing to build an anti-AI front for it, by all means. random idea, maybe Codesearch could use ? @musikanimal added... [19:30:34] 10VPS-project-Codesearch: Codesearch down/unreachable (2025-08-30) - https://phabricator.wikimedia.org/T403323#11134046 (10LucasWerkmeister) IMHO it would make more sense to gate it behind an OAuth login, just like [Global Search](https://global-search.toolforge.org/). This way we don’t waste CPU time on peo...