[00:39:57] FIRING: ProbeDown: Service labweb-ssl:7443 has failed probes (http_labweb-ssl_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#labweb-ssl:7443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:44:57] RESOLVED: ProbeDown: Service labweb-ssl:7443 has failed probes (http_labweb-ssl_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#labweb-ssl:7443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [01:30:33] (03open) 10bd808: Convert project for use with buildpacks [toolforge-repos/phabricator-bug-status] - 10https://gitlab.wikimedia.org/toolforge-repos/phabricator-bug-status/-/merge_requests/2 (https://phabricator.wikimedia.org/T142237) [01:33:19] (03update) 10bd808: Convert project for use with buildpacks [toolforge-repos/phabricator-bug-status] - 10https://gitlab.wikimedia.org/toolforge-repos/phabricator-bug-status/-/merge_requests/2 (https://phabricator.wikimedia.org/T142237) [01:34:51] (03open) 10raymond-ndibe: logs-api,logs: add logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T400917) [01:38:00] 10Toolforge (Toolforge iteration 25): add more logs tests to toolforge-deploy - https://phabricator.wikimedia.org/T418326 (10Raymond_Ndibe) 03NEW [01:38:29] (03update) 10raymond-ndibe: logs-api,logs: add logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T400917 https://phabricator.wikimedia.org/T418326) [01:38:56] (03update) 10raymond-ndibe: logs-api,logs: add logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326) [01:39:07] (03update) 10raymond-ndibe: logs-api,logs: add logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326) [01:44:31] (03PS1) 10Salmanfadi: T371092: Use +show for Markdown files in Gitiles links [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1243302 [01:45:56] (03update) 10raymond-ndibe: logs-api,logs: add logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326) [01:48:04] (03update) 10raymond-ndibe: logs-api,logs: add logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326) [01:48:20] (03update) 10bd808: Convert project for use with buildpacks [toolforge-repos/phabricator-bug-status] - 10https://gitlab.wikimedia.org/toolforge-repos/phabricator-bug-status/-/merge_requests/2 (https://phabricator.wikimedia.org/T142237 https://phabricator.wikimedia.org/T380127 https://phabricator.wikimedia.org/T408108) [01:49:12] (03update) 10raymond-ndibe: logs-api,logs: add logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326) [01:51:11] (03PS2) 10Salmanfadi: T371092: Use +show for Markdown files in Gitiles links [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1243302 (https://phabricator.wikimedia.org/T371092) [01:52:52] 10VPS-project-Codesearch, 10Gerrit, 07good first task, 13Patch-For-Review: Codesearch links to Gitiles for Markdown files show rendering instead of source - https://phabricator.wikimedia.org/T371092#11649219 (10Salmanfadi) I have submitted a patch for this task: https://gerrit.wikimedia.org/r/c/labs/codese... [01:54:00] 10Tool-Phabricator-bug-status: Move away from deprecated `maniphest.query` Conduit API - https://phabricator.wikimedia.org/T418328 (10bd808) 03NEW [02:07:09] (03update) 10raymond-ndibe: logs-api,logs: add logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326) [02:24:09] (03update) 10raymond-ndibe: logs-api,logs: add logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326) [02:48:37] (03update) 10raymond-ndibe: logs-api,logs: add logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326) [02:53:44] (03update) 10raymond-ndibe: logs-api,logs: add logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326) [03:03:49] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.05% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [03:09:27] (03update) 10raymond-ndibe: logs-api,logs: add logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326) [03:25:19] (03update) 10raymond-ndibe: logs-api,logs: add logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326) [03:35:14] (03update) 10raymond-ndibe: logs-api,logs: add logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326) [03:44:15] 10Toolforge (Toolforge iteration 25): add more logs tests to toolforge-deploy - https://phabricator.wikimedia.org/T418326#11649319 (10Raymond_Ndibe) 05Open→03In progress [03:46:55] 10VPS-project-Phabricator, 06collaboration-services: phabricator.wmcloud.org account verification request: ajhalili2006 - https://phabricator.wikimedia.org/T418224#11649321 (10ajhalili2006) >>! In T418224#11644511, @Aklapper wrote: > Thanks for your interest! What //exactly// do you plan to do in the test inst... [03:47:50] 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: add more logs tests to toolforge-deploy - https://phabricator.wikimedia.org/T418326#11649322 (10Raymond_Ndibe) [04:14:44] (03open) 10gopavasanth: fix: remove responsive settings from campaign table DataTable initialization [toolforge-repos/isa] - 10https://gitlab.wikimedia.org/toolforge-repos/isa/-/merge_requests/66 (https://phabricator.wikimedia.org/T418228) [04:34:42] 10Tool-campwiz-nxt: CampWiz Nxt Redesign: Root Path - https://phabricator.wikimedia.org/T415408#11649381 (10Adityash2906) @Nokib_Sarkar The backend is down i guess even i am getting 404 error while setting up the frontend locally [05:33:55] (03update) 10raymond-ndibe: logs-api,logs: test start, end params for logs-api and jobs-api log endpoint [repos/cloud/toolforge/toolforge-deploy] (add_logs_api_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917) [05:33:58] (03open) 10raymond-ndibe: logs-api,logs: test start, end params for logs-api and jobs-api log endpoint [repos/cloud/toolforge/toolforge-deploy] (add_logs_api_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917) [05:47:25] (03update) 10raymond-ndibe: support start and end params in logs endpoint [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/267 (https://phabricator.wikimedia.org/T400917) [06:01:37] Change on 12wikitech.wikimedia.org a page Help:Toolforge/My first NodeJS OAuth tool was modified, changed by Hugoa-wikis link https://wikitech.wikimedia.org/w/index.php?diff=2383912 edit summary: Fix typo [06:08:02] (03update) 10raymond-ndibe: logs-api,logs: test start, end params for logs-api and jobs-api log endpoint [repos/cloud/toolforge/toolforge-deploy] (add_logs_api_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917) [06:21:59] (03update) 10raymond-ndibe: logs-api,logs: test start, end params for logs-api and jobs-api log endpoint [repos/cloud/toolforge/toolforge-deploy] (add_logs_api_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917) [06:31:03] (03update) 10raymond-ndibe: logs-api,logs: test start, end params for logs-api and jobs-api log endpoint [repos/cloud/toolforge/toolforge-deploy] (add_logs_api_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917) [06:37:43] (03update) 10raymond-ndibe: logs-api,logs: test start, end params for logs-api and jobs-api log endpoint [repos/cloud/toolforge/toolforge-deploy] (add_logs_api_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917) [06:49:09] (03update) 10raymond-ndibe: logs-api,logs: test start, end params for logs-api and jobs-api log endpoint [repos/cloud/toolforge/toolforge-deploy] (add_logs_api_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917) [07:03:49] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 5.272% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [07:45:10] (03CR) 10A smart kitten: "recheck [trigger CI]" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1243302 (https://phabricator.wikimedia.org/T371092) (owner: 10Salmanfadi) [08:04:41] (03CR) 10A smart kitten: "thanks for the patch!" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1243302 (https://phabricator.wikimedia.org/T371092) (owner: 10Salmanfadi) [08:51:43] 10Cloud-VPS (Project-requests): Cloud VPS Project Request: ph-student-encyclopedia - https://phabricator.wikimedia.org/T418213#11649767 (10taavi) 05Open→03Declined This wiki does not seem directly related to Wikimedia projects and thus is out of scope for WMCS. [09:05:53] 06cloud-services-team, 10Toolforge: Make webservice provision HTTPRoutes for each tool - https://phabricator.wikimedia.org/T418349 (10taavi) 03NEW [09:08:26] 06cloud-services-team, 10Toolforge: Make webservice provision HTTPRoutes for each tool - https://phabricator.wikimedia.org/T418349#11649795 (10taavi) [09:08:27] 06cloud-services-team, 10Toolforge: Pick and install a Gateway API implementation for Toolforge - https://phabricator.wikimedia.org/T418274#11649797 (10taavi) [09:08:30] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Update access control logic for migration from Ingress to Gateway API objects - https://phabricator.wikimedia.org/T418276#11649796 (10taavi) [09:08:47] (03update) 10taavi: kubernetes: Add support for HTTPRoute resource [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/89 (https://phabricator.wikimedia.org/T418275 https://phabricator.wikimedia.org/T418349) [09:08:52] (03update) 10taavi: kubernetes: Add support for HTTPRoute resource [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/89 (https://phabricator.wikimedia.org/T418349) [09:35:18] Change on 12wikitech.wikimedia.org a page Help:Toolforge/Kubernetes was modified, changed by Hugoa-wikis link https://wikitech.wikimedia.org/w/index.php?diff=2383929 edit summary: Fix typo [09:38:50] 06cloud-services-team, 10Toolforge: Pick and install a Gateway API implementation for Toolforge - https://phabricator.wikimedia.org/T418274#11649867 (10taavi) >>! In T418274#11647548, @dcaro wrote: > Some notes from the k8s sig today when asking about Gateway API: > * Chainguard will try to maintain the nginx-... [09:44:55] (03open) 10dcaro: toolforge_deploy: support non-image components [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/310 [09:45:22] (03update) 10dcaro: toolforge_deploy: support non-image components [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/310 [09:47:53] (03open) 10dcaro: add deployment docs [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/20 [09:48:16] (03update) 10dcaro: add deployment docs [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/20 [09:48:36] (03update) 10dcaro: docs: added standard deployment notes [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/20 [09:49:09] (03update) 10dcaro: docs: added standard deployment notes [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/20 [09:49:13] (03approved) 10fnegri: toolforge_deploy: support non-image components [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/310 (owner: 10dcaro) [09:49:15] (03update) 10dcaro: docs: added standard deployment notes [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/20 [09:50:20] (03merge) 10dcaro: toolforge_deploy: support non-image components [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/310 [09:53:19] (03open) 10dcaro: docs: update deployment docs [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/44 [09:53:51] (03update) 10dcaro: docs: update deployment docs [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/44 [09:54:17] (03update) 10dcaro: docs: update deployment docs [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/44 [09:54:55] 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [docs] update all readmes with the same deployment docs - https://phabricator.wikimedia.org/T407477#11649896 (10dcaro) [09:56:41] (03update) 10dcaro: docs: update deployment docs [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/44 [09:57:10] (03open) 10dcaro: docs: update deployment docs [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/36 [09:57:29] (03update) 10dcaro: docs: update deployment docs [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/36 [10:01:25] (03open) 10taavi: Validate Gateway API HTTPRoute resources [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/37 (https://phabricator.wikimedia.org/T418276) [10:04:16] (03open) 10dcaro: docs: update deployment instructions [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/70 [10:04:42] (03update) 10dcaro: docs: update deployment instructions [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/70 [10:06:35] (03update) 10taavi: Validate Gateway API HTTPRoute resources [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/37 (https://phabricator.wikimedia.org/T418276) [10:11:35] (03open) 10dcaro: docs: update the deployment docs using standard deploy [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/153 [10:12:15] (03update) 10dcaro: docs: update the deployment docs using standard deploy [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/153 [10:12:49] 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [docs] update all readmes with the same deployment docs - https://phabricator.wikimedia.org/T407477#11649917 (10dcaro) [10:13:19] 10Cloud-VPS (Project-requests): Cloud VPS Project Request: ph-student-encyclopedia - https://phabricator.wikimedia.org/T418213#11649918 (10Mr.Besya) I apologize for the lack of clarity regarding the project's scope. I understand that WMCS is not for general hosting. My primary goal is actually technical: I w... [10:13:34] 10Cloud-VPS (Project-requests): Cloud VPS Project Request: ph-student-encyclopedia - https://phabricator.wikimedia.org/T418213#11649919 (10Mr.Besya) 05Declined→03Stalled [10:14:29] 10Cloud-VPS (Project-requests): Cloud VPS Project Request: ph-student-encyclopedia - https://phabricator.wikimedia.org/T418213#11649922 (10Mr.Besya) I am moving this task to Stalled while I refine the project's technical goals to ensure they align with the WMCS Scope. I want to focus the project on documenting M... [10:15:07] 10Cloud-VPS (Project-requests): Cloud VPS Project Request: ph-student-encyclopedia - https://phabricator.wikimedia.org/T418213#11649923 (10Mr.Besya) 05Stalled→03Open [10:15:24] 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [docs] update all readmes with the same deployment docs - https://phabricator.wikimedia.org/T407477#11649929 (10dcaro) 05Open→03In progress [10:15:52] 10Cloud-VPS (Project-requests): Cloud VPS Project Request: ph-student-encyclopedia - https://phabricator.wikimedia.org/T418213#11649930 (10Mr.Besya) I am moving this back to Open because I have narrowed the scope to fit WMCS requirements. I realize my previous description was unclear. The project is now strictly... [10:17:09] 10Cloud-VPS (Project-requests): Cloud VPS Project Request: ph-student-encyclopedia - https://phabricator.wikimedia.org/T418213#11649935 (10Mr.Besya) [10:25:06] 10Cloud-VPS (Project-requests): Cloud VPS Project Request: ph-student-encyclopedia - https://phabricator.wikimedia.org/T418213#11649960 (10taavi) 05Open→03Declined > https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_project#Reviews_of_Cloud_VPS_Project_requests wrote: > **"Laptop in the cloud" projects... [10:33:22] 06cloud-services-team, 06SRE Observability, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance, 13Patch-For-Review: [cloudweb] Improve the alerts coming from the LVS setup - https://phabricator.wikimedia.org/T316682#11650016 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Task... [11:03:50] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 4.694% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [11:05:41] 06cloud-services-team, 10Toolforge: Pick and install a Gateway API implementation for Toolforge - https://phabricator.wikimedia.org/T418274#11650115 (10taavi) Looking at all implementations marked as Conformant on the [[ https://gateway-api.sigs.k8s.io/implementations/ | Gateway API website ]]: (tl;dr at the b... [11:16:43] 06cloud-services-team, 10Toolforge: Pick and install a Gateway API implementation for Toolforge - https://phabricator.wikimedia.org/T418274#11650188 (10dcaro) +1 for Istio, quite nice that the benchmark used 3k routes xd, we have around that number, pretty nice. Kgateway seems promising, but yep, a bit new, w... [11:45:02] 06cloud-services-team, 10Toolforge: Pick and install a Gateway API implementation for Toolforge - https://phabricator.wikimedia.org/T418274#11650294 (10taavi) [11:56:28] FIRING: InstanceDown: Project tools instance tools-puppetserver-01 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [11:57:10] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22395760068 (https://github.com/cluebotng/component-configs/commits/c6093c4ed72aba8fa453b2f67e48d1effeaabb4b) [11:57:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [12:01:28] RESOLVED: InstanceDown: Project tools instance tools-puppetserver-01 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [12:23:44] 10Cloud-VPS (Project-requests): Cloud VPS Project Request: ph-student-encyclopedia - https://phabricator.wikimedia.org/T418213#11650420 (10Mr.Besya) 05Declined→03Open [12:23:44] 10Cloud-VPS (Project-requests): Cloud VPS Project Request: ph-student-encyclopedia - https://phabricator.wikimedia.org/T418213#11650421 (10Mr.Besya) I have read the Help:Cloud_VPS_project guidelines regarding single-user projects. To address the concerns of fairness and inclusivity: This is not a personal develo... [12:53:35] RESOLVED: DiskSpace: Disk space cloudbackup1004:9100:/srv 4.606% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [12:58:46] 10Cloud-VPS (Project-requests): Cloud VPS Project Request: ph-student-encyclopedia - https://phabricator.wikimedia.org/T418213#11650525 (10Aklapper) 05Open→03Declined Do not reopen this ticket. Thanks. [13:19:17] (03open) 10taavi: gateway-api: Install Gateway API CRDs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1142 (https://phabricator.wikimedia.org/T418275) [13:19:18] (03update) 10taavi: gateway-api: Install Gateway API CRDs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1142 (https://phabricator.wikimedia.org/T418275) [13:21:21] (03update) 10taavi: k8s: Install Gateway API CRDs [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/309 (https://phabricator.wikimedia.org/T418275) [13:22:47] (03update) 10taavi: k8s: Install Gateway API CRDs [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/309 (https://phabricator.wikimedia.org/T418275) [13:23:38] (03update) 10taavi: gateway-api: Install Gateway API CRDs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1142 (https://phabricator.wikimedia.org/T418275) [13:27:08] (03PS1) 10Majavah: toolforge: k8s: Add cookbook to mirror Istio images [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1243818 (https://phabricator.wikimedia.org/T418274) [13:30:41] (03approved) 10dcaro: gateway-api: Install Gateway API CRDs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1142 (https://phabricator.wikimedia.org/T418275) (owner: 10taavi) [13:37:12] (03update) 10taavi: gateway-api: Install Gateway API CRDs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1142 (https://phabricator.wikimedia.org/T418275) [13:38:21] (03merge) 10taavi: gateway-api: Install Gateway API CRDs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1142 (https://phabricator.wikimedia.org/T418275) [13:46:13] 10Cloud-VPS (Project-requests): Cloud VPS Project Request: ph-student-encyclopedia - https://phabricator.wikimedia.org/T418213#11650715 (10taavi) [13:49:16] (03update) 10taavi: k8s: Install Gateway API CRDs [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/309 (https://phabricator.wikimedia.org/T418275) [13:49:17] (03update) 10taavi: kind: Use port 30000 for Istio-based tool gateway [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/311 (https://phabricator.wikimedia.org/T418274) [13:49:18] (03open) 10taavi: kind: Use port 30000 for Istio-based tool gateway [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/311 (https://phabricator.wikimedia.org/T418274) [13:49:22] (03update) 10taavi: kind: Use port 30000 for Istio-based tool gateway [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/311 (https://phabricator.wikimedia.org/T418274) [13:53:09] (03approved) 10dcaro: k8s: Install Gateway API CRDs [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/309 (https://phabricator.wikimedia.org/T418275) (owner: 10taavi) [13:53:28] (03approved) 10dcaro: kind: Use port 30000 for Istio-based tool gateway [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/311 (https://phabricator.wikimedia.org/T418274) (owner: 10taavi) [13:57:35] (03merge) 10taavi: k8s: Install Gateway API CRDs [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/309 (https://phabricator.wikimedia.org/T418275) [13:57:36] (03update) 10taavi: kind: Use port 30000 for Istio-based tool gateway [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/311 (https://phabricator.wikimedia.org/T418274) [13:57:38] (03update) 10taavi: kind: Use port 30000 for Istio-based tool gateway [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/311 (https://phabricator.wikimedia.org/T418274) [14:01:28] (03merge) 10taavi: kind: Use port 30000 for Istio-based tool gateway [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/311 (https://phabricator.wikimedia.org/T418274) [14:02:06] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component gateway-api [14:07:37] (03CR) 10David Caro: [C:03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1243818 (https://phabricator.wikimedia.org/T418274) (owner: 10Majavah) [14:08:26] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry for Istio 1.29.0 [14:08:55] !log taavi@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry (exit_code=99) for Istio 1.29.0 [14:15:51] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-28 (2026-02-24 to 2026-03-10)): Fix linter issues discovered during implementation of the OAD example - https://phabricator.wikimedia.org/T414974#11650798 (10HCoplin-WMF) [14:17:30] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component gateway-api [14:21:57] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry for Istio 1.29.0 [14:22:47] !log taavi@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry (exit_code=99) for Istio 1.29.0 [14:24:01] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry for Istio 1.29.0 [14:24:29] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry (exit_code=0) for Istio 1.29.0 [14:25:58] (03CR) 10Majavah: [C:03+2] toolforge: k8s: Add cookbook to mirror Istio images [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1243818 (https://phabricator.wikimedia.org/T418274) (owner: 10Majavah) [14:29:34] (03Merged) 10jenkins-bot: toolforge: k8s: Add cookbook to mirror Istio images [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1243818 (https://phabricator.wikimedia.org/T418274) (owner: 10Majavah) [14:53:52] (03open) 10nurahwakili: login link for work page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/181 [15:26:03] (03open) 10taavi: Deploy istio and a Gateway [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1143 (https://phabricator.wikimedia.org/T418274) [15:34:17] (03update) 10taavi: kubernetes: Add support for HTTPRoute resource [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/89 (https://phabricator.wikimedia.org/T418349) [16:07:36] (03approved) 10fnegri: kubernetes: Add support for HTTPRoute resource [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/89 (https://phabricator.wikimedia.org/T418349) (owner: 10taavi) [16:12:02] 06cloud-services-team, 10Data-Services, 10Datasets-General-or-Unknown, 06Traffic, 13Patch-For-Review: Move dumps.wikimedia.org HTTP service behind CDN edge - https://phabricator.wikimedia.org/T306550#11651360 (10taavi) >>! In T306550#11624133, @BCornwall wrote: > It would be helpful to state the desired... [16:12:28] (03merge) 10taavi: kubernetes: Add support for HTTPRoute resource [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/89 (https://phabricator.wikimedia.org/T418349) [16:20:55] FIRING: [2x] ProbeDown: Service cloudweb1003:443 has failed probes (http_toolsadmin_wikimedia_org_ip4) #page - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [16:30:55] FIRING: [2x] ProbeDown: Service cloudweb1003:443 has failed probes (http_toolsadmin_wikimedia_org_ip4) #page - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [16:36:01] RESOLVED: [2x] ProbeDown: Service cloudweb1003:443 has failed probes (http_toolsadmin_wikimedia_org_ip4) #page - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [16:40:28] FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [17:08:13] RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [17:12:37] 06cloud-services-team, 10Toolforge, 07Documentation: Create a "my first React app" tutorial for Toolforge - https://phabricator.wikimedia.org/T231950#11651644 (10Hugoa) 05Open→03In progress Hi! Here is a first version of the tutorial: [[ https://wikitech.wikimedia.org/wiki/User:Hugoa-wikis/My_first_React... [17:17:47] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Platform-SRE: Drop support for cl_to, cl_collation and il_to from wikireplicas - https://phabricator.wikimedia.org/T417492#11651659 (10Ottomata) [17:18:54] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Platform-SRE: Drop support for cl_to, cl_collation and il_to from wikireplicas - https://phabricator.wikimedia.org/T417492#11651676 (10Ottomata) @Milimetric @Snwachukwu just double checking: we don't sqoop categorylink related tables from a... [17:25:52] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Platform-SRE: Drop support for cl_to, cl_collation and il_to from wikireplicas - https://phabricator.wikimedia.org/T417492#11651715 (10Milimetric) @Ottomata we do: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/f12678b3a... [17:32:30] 06cloud-services-team, 10Data-Services, 07Documentation: Revise and reformat Portal:Data_Services - https://phabricator.wikimedia.org/T348024#11651743 (10Hugoa) a:03Hugoa [17:40:40] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Platform-SRE: Drop support for cl_to, cl_collation and il_to from wikireplicas - https://phabricator.wikimedia.org/T417492#11651801 (10Ottomata) Hah, sorry! I guess I meant: "the cloud replicas we access for analytics-hadoop ingestion" [17:50:24] (03approved) 10fnegri: jobs-api: add jobs version migration script and docs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/746 (https://phabricator.wikimedia.org/T359649) (owner: 10dcaro) [17:52:48] (03PS3) 10Salmanfadi: frontend: Use +show for Markdown files in Gitiles links [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1243302 (https://phabricator.wikimedia.org/T371092) [18:06:01] (03update) 10dcaro: jobs-api: add jobs version migration script and docs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/746 (https://phabricator.wikimedia.org/T359649) [18:12:52] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version - https://phabricator.wikimedia.org/T359649#11651923 (10fnegri) > I could not reproduce a scenario where recreating two jobs... [18:16:40] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version - https://phabricator.wikimedia.org/T359649#11651941 (10dcaro) > @Raymond_Ndibe @dcaro should we schedule a date for this mig... [18:21:31] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Platform-SRE: Drop support for cl_to, cl_collation and il_to from wikireplicas - https://phabricator.wikimedia.org/T417492#11651954 (10Ottomata) Ah: This is covered by {T416481}. Moving to DE radar. [18:37:29] 10VPS-project-Phabricator, 06collaboration-services: phabricator.wmcloud.org account verification request: ajhalili2006 - https://phabricator.wikimedia.org/T418224#11652020 (10Dzahn) It's fine if you want to test something around ACLs but I would like to point out there is no relation to wiki extensions and th... [18:43:22] 10VPS-project-Phabricator, 06collaboration-services: phabricator.wmcloud.org account verification request: ajhalili2006 - https://phabricator.wikimedia.org/T418224#11652038 (10Dzahn) 05Open→03Resolved a:03Dzahn You have been verified. [20:48:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [21:10:47] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22415923749 (https://github.com/cluebotng/component-configs/commits/a2a4f5ecffad1b49c96c33b5045430a5b75f71bc) [21:10:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [21:11:30] 06cloud-services-team, 10Toolforge: Node.js buildpack selects EOL Node.js version - https://phabricator.wikimedia.org/T418414 (10LucasWerkmeister) 03NEW [21:13:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [21:16:05] 06cloud-services-team, 10Toolforge: Node.js buildpack selects EOL Node.js version - https://phabricator.wikimedia.org/T418414#11652370 (10LucasWerkmeister) I’m guessing the Node buildpack is itself outdated? ` [step-detect] 2026-02-25T20:42:56.651711339Z 3 of 8 buildpacks participating... [21:18:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [21:30:20] (03CR) 10A smart kitten: "recheck" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1243302 (https://phabricator.wikimedia.org/T371092) (owner: 10Salmanfadi) [21:34:00] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [21:39:00] FIRING: [2x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [21:43:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [21:44:00] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [21:45:56] FIRING: SystemdUnitDown: The service unit designate_floating_ip_ptr_records_updater.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [21:50:56] FIRING: [2x] SystemdUnitDown: The service unit designate_floating_ip_ptr_records_updater.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [22:14:32] (03PS1) 10Eevans: Add (phony) password for linked_artifacts Cassandra role [labs/private] - 10https://gerrit.wikimedia.org/r/1243986 (https://phabricator.wikimedia.org/T418420) [22:15:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [22:18:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [22:24:06] Change on 12wikitech.wikimedia.org a page Help:Toolforge was modified, changed by Muusejaamac100 link https://wikitech.wikimedia.org/w/index.php?diff=2384357 edit summary: [+20] Hawd Zone [22:26:28] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance extdist-06 in project extdist - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [22:43:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [22:48:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [23:07:29] FIRING: PuppetAgentNoResources: No Puppet resources found on instance extdist-06 on project extdist - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:13:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [23:18:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [23:24:00] FIRING: [4x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [23:39:00] FIRING: [5x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [23:43:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [23:48:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown