[00:13:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on cloudrabbit2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[00:13:57] <wikibugs>	 06cloud-services-team: PuppetFailure Puppet has failed on cloudrabbit2002-dev:9100 - https://phabricator.wikimedia.org/T393528 (10phaultfinder) 03NEW
[00:14:50] <wmcs-alerts>	 RESOLVED: PrometheusK8sCertExpirySoon: Prometheus k8s certificate is about to expire - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/PrometheusK8sCertExpirySoon  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusK8sCertExpirySoon
[00:28:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on cloudrabbit2003-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[00:28:53] <wikibugs>	 06cloud-services-team: PuppetFailure Puppet has failed on cloudrabbit2003-dev:9100 - https://phabricator.wikimedia.org/T393529 (10phaultfinder) 03NEW
[00:33:48] <jinxer-wm>	 RESOLVED: PuppetFailure: Puppet has failed on cloudrabbit2003-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[00:43:11] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
[00:47:00] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
[01:03:54] <wikibugs>	 06cloud-services-team, 13Patch-For-Review: Rename cloudcontrol200[789]-dev.codfw to cloudrabbit200[123]-dev.codfw - https://phabricator.wikimedia.org/T392539#10798616 (10Andrew) 05Open→03Resolved a:03Andrew
[02:52:19] <jinxer-wm>	 FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[02:52:44] <wikibugs>	 (03update) 10chuckonwumelu: Bug: T390056 importing tools volumes to Tofu [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/19
[02:55:06] <wikibugs>	 (03merge) 10chuckonwumelu: Bug: T390056 importing tools volumes to Tofu [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/19
[02:57:19] <jinxer-wm>	 RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[03:13:19] <jinxer-wm>	 FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[05:00:10] <wikibugs>	 10Tool-techcontribs: Add a link to Phabricator to the footer - https://phabricator.wikimedia.org/T393532 (10Novem_Linguae) 03NEW
[05:03:19] <jinxer-wm>	 FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[05:03:24] <wikibugs>	 06cloud-services-team: HighIOWaitStalling High iowait detected on clouddumps1002:9100. - https://phabricator.wikimedia.org/T393533 (10phaultfinder) 03NEW
[05:03:35] <wikibugs>	 10Tool-techcontribs: hide mediawiki sub-groups from the gerrit groups section - https://phabricator.wikimedia.org/T393534 (10Novem_Linguae) 03NEW
[05:04:45] <wikibugs>	 10Tool-techcontribs: Remove the "uploaded groups" section from Gerrit groups - https://phabricator.wikimedia.org/T393535 (10Novem_Linguae) 03NEW
[05:04:56] <wikibugs>	 10Tool-techcontribs: Remove the "uploaded groups" section from Gerrit groups - https://phabricator.wikimedia.org/T393535#10798777 (10Novem_Linguae)
[05:10:21] <wikibugs>	 10Tool-techcontribs: Add some of the data from ldap.toolforge.org tool - https://phabricator.wikimedia.org/T393536 (10Novem_Linguae) 03NEW
[05:13:57] <wikibugs>	 10Tool-techcontribs: Reduce amount of scrolling and clicks needed to get to the search form - https://phabricator.wikimedia.org/T393537 (10Novem_Linguae) 03NEW
[05:14:27] <wikibugs>	 10Tool-techcontribs: Reduce amount of scrolling and clicks needed to get to the search form - https://phabricator.wikimedia.org/T393537#10798801 (10Novem_Linguae)
[06:05:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[06:58:19] <jinxer-wm>	 RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[06:58:19] <jinxer-wm>	 RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[07:05:19] <jinxer-wm>	 FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[08:02:23] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: lua entry thread aborted: runtime error: /etc/nginx/lua/domainproxy.lua:32: bad request - https://phabricator.wikimedia.org/T393024#10799056 (10taavi) 05Open→03Resolved
[08:05:19] <jinxer-wm>	 RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[08:12:00] <wikibugs>	 06cloud-services-team, 10Toolforge, 07Documentation: Update roadmap for Toolforge Build Service - https://phabricator.wikimedia.org/T393549#10799156 (10A_smart_kitten)
[08:14:16] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Retire explicit 'roots' sudo policies - https://phabricator.wikimedia.org/T392797#10799161 (10taavi) a:03taavi
[08:15:35] <wikibugs>	 (03merge) 10dcaro: [jobs-cli] only send timeout if it's set by the user [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/96 (https://phabricator.wikimedia.org/T389118) (owner: 10raymond-ndibe)
[08:15:36] <wikibugs>	 (03update) 10dcaro: [jobs-cli] health_check and quota refactor [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/97 (https://phabricator.wikimedia.org/T389118) (owner: 10raymond-ndibe)
[08:18:02] <wikibugs>	 (03open) 10dcaro: d/changelog: bump to 16.1.12 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/102 (https://phabricator.wikimedia.org/T389118)
[08:25:51] <wikibugs>	 06cloud-services-team, 10Toolforge, 07Documentation: Update roadmap for Toolforge Build Service - https://phabricator.wikimedia.org/T393549#10799218 (10dcaro) It needs some updating yep, just changed the task to point to the epic (was the parent of those), I'll add also a task for the beta with more details...
[08:33:19] <wikibugs>	 06cloud-services-team, 10Toolforge, 07Documentation: Update roadmap for Toolforge Build Service - https://phabricator.wikimedia.org/T393549#10799281 (10dcaro) Added also some links to the workgroup reports, monthly meetings and changelog just in case.
[08:36:19] <jinxer-wm>	 FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[08:40:06] <wikibugs>	 06cloud-services-team, 14Toolforge (Toolforge iteration 08), 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project: [builds-api,components-api] Automatically deploy the webservice when the image is built - https://phabricator.wikimedia.org/T341065#10799307 (10dcaro) Fyi. this will be implemen...
[08:41:19] <jinxer-wm>	 RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[08:47:47] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
[08:53:19] <jinxer-wm>	 FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[08:59:17] <wikibugs>	 10wikitech.wikimedia.org, 10Wikidata, 10Wikimedia-Interwiki-links, 13Patch-For-Review, 10Wikidata Integration in Wikimedia projects (Kanban Board): Enable interwiki links to/from Wikitech - https://phabricator.wikimedia.org/T290147#10799349 (10Neslihan_Turan_WMDE)
[08:59:29] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
[09:00:18] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
[09:12:04] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
[09:12:33] <wikibugs>	 (03approved) 10dcaro: d/changelog: bump to 16.1.12 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/102 (https://phabricator.wikimedia.org/T389118)
[09:12:37] <wikibugs>	 (03merge) 10dcaro: d/changelog: bump to 16.1.12 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/102 (https://phabricator.wikimedia.org/T389118)
[09:14:29] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api
[09:19:35] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
[09:22:05] <wikibugs>	 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge, 05Cloud-Services-Origin-Team, and 3 others: [Hypothesis] WE6.3.10 start a beta for the push-to-deploy features - https://phabricator.wikimedia.org/T393564 (10dcaro) 03NEW
[09:22:11] <wikibugs>	 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge, 05Cloud-Services-Origin-Team, and 3 others: [Hypothesis] WE6.3.10 start a beta for the push-to-deploy features - https://phabricator.wikimedia.org/T393564#10799397 (10dcaro) 05Open→03In progress p:05Triage→03High
[09:22:53] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api
[09:27:31] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: Enable IPv6 for the Cloud VPS web proxy - https://phabricator.wikimedia.org/T379175#10799406 (10taavi)
[09:33:49] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
[09:39:52] <wikibugs>	 06cloud-services-team, 10Toolforge: Retire explicit 'roots' sudo policies - https://phabricator.wikimedia.org/T392797#10799438 (10taavi) Dropped the toolsbeta policy from LDAP and as far as I can tell the access is still there: `lang=shell-session taavi@toolsbeta-puppetserver-1:~$ sudo -u raymond-ndibe sudo -l...
[09:40:21] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api
[09:41:21] <wikibugs>	 06cloud-services-team, 10Toolforge: Retire explicit 'roots' sudo policies - https://phabricator.wikimedia.org/T392797#10799443 (10taavi) 05Open→03Resolved I dropped it from tools as well, and updated the docs.
[09:42:07] <wikibugs>	 06cloud-services-team, 10Toolforge, 07Documentation: Update roadmap for Toolforge Build Service - https://phabricator.wikimedia.org/T393549#10799446 (10dcaro) And created {T393564} as a placeholder until the doc is reviewed :)
[09:53:40] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
[09:59:36] <wikibugs>	 (03approved) 10dcaro: builds-api: bump to 0.0.188-20250506170015-db3e79d0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/758 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[09:59:39] <wikibugs>	 (03merge) 10dcaro: builds-api: bump to 0.0.188-20250506170015-db3e79d0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/758 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[09:59:49] <wikibugs>	 (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/55 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[09:59:52] <wikibugs>	 (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/55 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[10:05:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[10:05:44] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: envvars-api: bump to 0.0.67-20250507100000-487572b3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/759
[10:06:45] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
[10:13:17] <wikibugs>	 10Toolforge (Toolforge iteration 19): [components-cli,poetry autoupdate] scheduled job is failing to update the poetry deps - https://phabricator.wikimedia.org/T393568 (10dcaro) 03NEW
[10:17:56] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
[10:29:08] <wikibugs>	 10Tool-techcontribs: Remove the "uploaded groups" section from Gerrit groups - https://phabricator.wikimedia.org/T393535#10799568 (10Chlod) Unlikely to do this. Uploaded patches include patches approved via `scap` and uploads of other people's code, which won't show up as owned. I can see some purpose in replaci...
[10:30:24] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
[10:31:24] <wikibugs>	 10Tool-techcontribs: Reduce amount of scrolling and clicks needed to get to the search form - https://phabricator.wikimedia.org/T393537#10799588 (10Chlod) Idea 1 sounds good. But this will be a link and not a button. UI-wise, buttons are not meant for actions which do not change the current page, are not part of...
[10:31:41] <wikibugs>	 10Tool-techcontribs: Reduce amount of scrolling and clicks needed to get to the search form - https://phabricator.wikimedia.org/T393537#10799589 (10Chlod) p:05Triage→03Low
[10:34:14] <wikibugs>	 10Tool-techcontribs: Make section explainers collapsible - https://phabricator.wikimedia.org/T393326#10799611 (10Chlod) p:05Triage→03Low Tech Contribs was actually intended to also cater to non-developers, but it seems this purpose of it has been lesser realized than its usage by developers themselves. I can...
[10:36:00] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
[10:40:39] <wikibugs>	 (03update) 10fnegri: toolsdb: use DNS CNAME instead of A records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/228 (https://phabricator.wikimedia.org/T392831)
[10:44:00] <wikibugs>	 (03approved) 10dcaro: toolsdb: use DNS CNAME instead of A records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/228 (https://phabricator.wikimedia.org/T392831) (owner: 10fnegri)
[10:48:03] <wikibugs>	 (03merge) 10fnegri: toolsdb: use DNS CNAME instead of A records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/228 (https://phabricator.wikimedia.org/T392831)
[10:48:08] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
[10:49:03] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
[10:51:16] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [toolsdb] Use DNS CNAMEs instead of A records - https://phabricator.wikimedia.org/T392831#10799674 (10fnegri) 05In progress→03Resolved Applied and working correctly:  ` ~ $ dig tools.db.svc.wikimedia....
[11:10:11] <wikibugs>	 (03merge) 10taavi: Upgrade dependencies [repos/cloud/cloud-vps/go-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/go-cloudvps/-/merge_requests/4
[11:14:58] <wikibugs>	 (03update) 10galrach600: Eliza data update [toolforge-repos/miss-search] (linkhere_branch) - 10https://gitlab.wikimedia.org/toolforge-repos/miss-search/-/merge_requests/3 (owner: 10eliza189)
[11:17:41] <wikibugs>	 (03update) 10taavi: proxies: handle 400 return code from proxy API [repos/cloud/cloud-vps/go-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/go-cloudvps/-/merge_requests/2 (owner: 10andrew)
[11:19:14] <wikibugs>	 (03update) 10taavi: proxies: handle 400 return code from proxy API [repos/cloud/cloud-vps/go-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/go-cloudvps/-/merge_requests/2 (owner: 10andrew)
[11:53:19] <jinxer-wm>	 RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[11:57:08] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
[12:05:19] <jinxer-wm>	 FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[12:06:58] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [jobs-api] Periodically refresh image-config data - https://phabricator.wikimedia.org/T357112#10799998 (10dcaro) >>! In T357112#10775170, @Raymond_Ndibe wrote: > wondering why we just don't fetch the images from k8s config every...
[12:08:51] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [jobs-api] Periodically refresh image-config data - https://phabricator.wikimedia.org/T357112#10799999 (10dcaro) > Ideally (at some point), we might want to centralize that info, create an API for it, probably in the build servi...
[12:10:19] <jinxer-wm>	 RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[12:10:27] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
[12:13:36] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19): [k8s,infra] Upgrade Toolforge to Uwubernetes (1.30) - https://phabricator.wikimedia.org/T362869#10800016 (10dcaro) 05Open→03In progress
[12:14:38] <wikibugs>	 (03approved) 10dcaro: envvars-api: bump to 0.0.67-20250507100000-487572b3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/759 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[12:14:40] <wikibugs>	 (03merge) 10dcaro: envvars-api: bump to 0.0.67-20250507100000-487572b3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/759 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[12:14:49] <wikibugs>	 (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/29 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[12:14:52] <wikibugs>	 (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/29 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[12:18:04] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: volume-admission: bump to 0.0.66-20250507121502-5d365205 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/760
[12:25:20] <wikibugs>	 10Tool-inteGraality, 10VerySmallGLAM, 10Wikibase (3rd party installations), 06Wikibase Suite Team, 07Product-Feature: integraality for Wikibases? - https://phabricator.wikimedia.org/T294892#10800086 (10JeanFred) >>! In T294892#10779177, @Addshore wrote: > How is it deployed? Primairly a web app thing? >...
[12:29:54] <wikibugs>	 10Tool-inteGraality, 10VerySmallGLAM, 10Wikibase (3rd party installations), 06Wikibase Suite Team, 07Product-Feature: integraality for Wikibases? - https://phabricator.wikimedia.org/T294892#10800110 (10Addshore) > → there would need to be some onboarding to do for each new Wikibase wishing to use integra...
[12:31:23] <wikibugs>	 10Tool-techcontribs, 07Upstream: hide mediawiki sub-groups from the gerrit groups section - https://phabricator.wikimedia.org/T393534#10800116 (10Chlod) No way to do this right now with the existing Gerrit API without downloading a really big amount of data per group, as far as I can tell. This is why this inc...
[12:31:40] <wikibugs>	 10Tool-techcontribs, 07Upstream: hide mediawiki sub-groups from the gerrit groups section - https://phabricator.wikimedia.org/T393534#10800119 (10Chlod) p:05Triage→03Low
[12:35:12] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
[12:41:31] <wikibugs>	 10Tool-inteGraality, 07Documentation: Make architecture diagram for integraality - https://phabricator.wikimedia.org/T393593 (10JeanFred) 03NEW
[12:42:25] <wikibugs>	 10Tool-inteGraality, 07Documentation: Make architecture diagram for integraality - https://phabricator.wikimedia.org/T393593#10800157 (10JeanFred) 05Open→03Resolved I gave a try to Excalidraw and made https://commons.wikimedia.org/wiki/File:Integraality_architecture_diagram.png ; now added to https://w...
[12:47:33] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
[12:48:09] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
[12:57:39] <wmcs-alerts>	 FIRING: QuarryDown: Quarry application is unreachable   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown
[12:58:21] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
[13:13:15] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,neutron
[13:14:06] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,neutron
[13:21:28] <wmcs-alerts>	 FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure
[13:24:11] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,neutron
[13:43:28] <wmcs-alerts>	 FIRING: PuppetAgentFailure: Puppet agent failure detected on instance tools-k8s-worker-nfs-43 in project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
[13:44:28] <wmcs-alerts>	 RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure
[13:46:28] <wmcs-alerts>	 FIRING: [2x] PuppetAgentFailure: Puppet agent failure detected on instance toolsbeta-test-k8s-worker-nfs-10 in project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
[13:48:55] <wikibugs>	 10Data-Services, 06Data-Engineering, 06Data-Platform-SRE: Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#10800408 (10BTullis) I think that this is likely to be uncontentious and relatively easy to achieve. The three tables mentioned appear to be related in s...
[13:53:59] <wikibugs>	 10Data-Services, 06Data-Engineering, 06Data-Platform-SRE: Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#10800442 (10BTullis) From an eyeball of the schemas, I think that there is unlikely to be any need for redaction of data.  We can also see another table...
[14:03:29] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: Enable IPv6 for the Cloud VPS web proxy - https://phabricator.wikimedia.org/T379175#10800503 (10taavi)
[14:03:58] <wmcs-alerts>	 RESOLVED: [10x] PuppetAgentFailure: Puppet agent failure detected on instance tools-k8s-worker-nfs-14 in project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
[14:03:58] <wmcs-alerts>	 RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure
[14:05:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[14:07:58] <wmcs-alerts>	 RESOLVED: [2x] PuppetAgentFailure: Puppet agent failure detected on instance toolsbeta-test-k8s-worker-nfs-10 in project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
[14:16:39] <wikibugs>	 06cloud-services-team, 10Toolforge: [toolsdb] Remove floating IP - https://phabricator.wikimedia.org/T381272#10800555 (10taavi) a:03taavi
[14:16:55] <wikibugs>	 06cloud-services-team, 10Toolforge: [toolsdb] Remove floating IP - https://phabricator.wikimedia.org/T381272#10800556 (10taavi) 05Open→03Resolved
[14:17:56] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
[14:19:39] <wmcs-alerts>	 FIRING: QuarryDown: Quarry application is unreachable   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown
[14:22:07] <wikibugs>	 10Tool-techcontribs, 07Upstream: hide mediawiki sub-groups from the gerrit groups section - https://phabricator.wikimedia.org/T393534#10800573 (10Novem_Linguae) Could do it the hacky way:  ` if ( groups.mediawiki ) {     delete groups.AhoCorasick;     delete groups.at-ease;     delete groups.base-convert;...
[14:30:45] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
[14:32:24] <wikibugs>	 (03approved) 10dcaro: volume-admission: bump to 0.0.66-20250507121502-5d365205 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/760 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[14:32:27] <wikibugs>	 (03merge) 10dcaro: volume-admission: bump to 0.0.66-20250507121502-5d365205 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/760 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[14:34:50] <wikibugs>	 (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/21 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[14:34:52] <wikibugs>	 (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/21 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[14:36:09] <wmcs-alerts>	 RESOLVED: QuarryDown: Quarry application is unreachable   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown
[14:38:50] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: envvars-admission: bump to 0.0.28-20250507143504-a08f3471 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/761
[14:39:41] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
[14:51:37] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
[14:52:41] <wikibugs>	 10Toolforge (Toolforge iteration 19): [components-cli,poetry autoupdate] scheduled job is failing to update the poetry deps - https://phabricator.wikimedia.org/T393568#10800725 (10dcaro) p:05Triage→03Medium
[14:54:10] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
[15:01:20] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: project-proxy puppetserver CA about to expire - https://phabricator.wikimedia.org/T392792#10800769 (10dcaro) p:05Triage→03Medium
[15:04:32] <wikibugs>	 10Toolforge (Toolforge iteration 19): [components-cli,poetry autoupdate] scheduled job is failing to update the poetry deps - https://phabricator.wikimedia.org/T393568#10800796 (10dcaro) Probably related: https://github.com/pypa/wheel/issues/643
[15:06:26] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
[15:07:05] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [jobs-api] Periodically refresh image-config data - https://phabricator.wikimedia.org/T357112#10800818 (10taavi) Note that as soon as `webservice` is gone we can move the config back to jobs-api and avoid all the other complicat...
[15:11:40] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [jobs-api] Periodically refresh image-config data - https://phabricator.wikimedia.org/T357112#10800831 (10dcaro) >>! In T357112#10800818, @taavi wrote: > Note that as soon as `webservice` is gone we can move the config back to j...
[15:16:41] <wikibugs>	 06cloud-services-team, 10Toolforge, 07Documentation: Update roadmap for Toolforge Build Service - https://phabricator.wikimedia.org/T393549#10800846 (10dcaro) @Sascha let me know if that feels like enough info, or what other bits you see missing, thanks for the task!
[15:16:50] <wikibugs>	 06cloud-services-team, 10Toolforge, 07Documentation: Update roadmap for Toolforge Build Service - https://phabricator.wikimedia.org/T393549#10800847 (10dcaro) p:05Triage→03Medium
[15:17:03] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 07Documentation: Update roadmap for Toolforge Build Service - https://phabricator.wikimedia.org/T393549#10800848 (10dcaro)
[15:30:59] <wikibugs>	 (03approved) 10dcaro: envvars-admission: bump to 0.0.28-20250507143504-a08f3471 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/761 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[15:31:01] <wikibugs>	 06cloud-services-team: wmcs-cookbooks: update the openstack restart cookbook to prioritize control nodes - https://phabricator.wikimedia.org/T393610 (10Andrew) 03NEW
[15:31:01] <wikibugs>	 (03merge) 10dcaro: envvars-admission: bump to 0.0.28-20250507143504-a08f3471 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/761 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[15:31:09] <wikibugs>	 (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/161 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[15:31:12] <wikibugs>	 (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/161 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[15:34:16] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.370-20250507153123-03fd02a2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/762
[15:37:39] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
[15:40:30] <wikibugs>	 (03PS1) 10Andrew Bogott: restart_openstack: restart services on cloudvirts last [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1143126 (https://phabricator.wikimedia.org/T393610)
[15:42:35] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,nova
[15:42:37] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
[15:42:38] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,nova
[15:43:09] <wikibugs>	 (03PS2) 10Andrew Bogott: restart_openstack: restart services on cloudvirts last [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1143126 (https://phabricator.wikimedia.org/T393610)
[15:43:14] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,nova
[15:43:17] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,nova
[15:43:35] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,nova
[15:43:38] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,nova
[15:44:19] <wikibugs>	 (03PS3) 10Andrew Bogott: restart_openstack: restart services on cloudvirts last [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1143126 (https://phabricator.wikimedia.org/T393610)
[15:44:47] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,nova
[15:45:20] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,nova
[15:49:44] <wikibugs>	 (03CR) 10Andrew Bogott: "I tested this and it did what I expected." [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1143126 (https://phabricator.wikimedia.org/T393610) (owner: 10Andrew Bogott)
[15:50:15] <wikibugs>	 (03CR) 10FNegri: [C:03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1143126 (https://phabricator.wikimedia.org/T393610) (owner: 10Andrew Bogott)
[15:51:37] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
[15:52:19] <wikibugs>	 10Toolforge (Toolforge iteration 19): [components-cli,poetry autoupdate] scheduled job is failing to update the poetry deps - https://phabricator.wikimedia.org/T393568#10801037 (10dcaro) Using a newer poetry seems to help, looking
[15:54:21] <wikibugs>	 (03CR) 10Andrew Bogott: [C:03+2] restart_openstack: restart services on cloudvirts last [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1143126 (https://phabricator.wikimedia.org/T393610) (owner: 10Andrew Bogott)
[15:57:11] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 07Documentation: Update roadmap for Toolforge Build Service - https://phabricator.wikimedia.org/T393549#10801085 (10Sascha) Sounds good, and good luck with the work! Just curious, is it possible to say when push-to-deploy will be working?
[15:58:26] <wikibugs>	 (03open) 10dcaro: create_poetry_update_mrs: upgrade poetry to latest [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/52 (https://phabricator.wikimedia.org/T393568)
[16:02:43] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
[16:02:45] <wikibugs>	 (03update) 10dcaro: create_poetry_update_mrs: upgrade poetry to latest [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/52 (https://phabricator.wikimedia.org/T393568)
[16:20:30] <wikibugs>	 (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/240 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[16:20:33] <wikibugs>	 (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/240 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[16:22:16] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.104-20250507161948-39977c69 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/763
[16:23:42] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,magnum,heat
[16:24:22] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,magnum,heat
[16:27:07] <wikibugs>	 (03open) 10dcaro: create_toolforge_deploy_mr: skip deployment if flagged [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/53
[16:27:14] <wikibugs>	 (03update) 10dcaro: create_toolforge_deploy_mr: skip deployment if flagged [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/53
[16:28:21] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,magnum,heat
[16:28:34] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,magnum,heat
[16:29:51] <wikibugs>	 (03open) 10dcaro: create_precommit_update_mrs: add the `deploy: skip` header [repos/cloud/cicd/gitlab-ci] (add_skip_deploy_detection) - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/54
[16:30:08] <wikibugs>	 (03update) 10dcaro: create_precommit_update_mrs: add the `deploy: skip` header [repos/cloud/cicd/gitlab-ci] (add_skip_deploy_detection) - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/54
[16:30:20] <wikibugs>	 (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/65 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[16:30:23] <wikibugs>	 (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/65 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[16:30:24] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,magnum
[16:30:35] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,magnum
[16:30:40] <wikibugs>	 (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/64 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[16:30:42] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,heat
[16:31:04] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,heat
[16:31:53] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api
[16:32:22] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.67-20250507163032-f26d5cd1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/764
[16:32:41] <wikibugs>	 (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/64 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[16:32:43] <wikibugs>	 (03unapproved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/64 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[16:41:38] <wikibugs>	 10Tool-techcontribs: visiting techcontribs.toolforge.org/uid/ (with an incomplete URI) redirects to https://localhost:8000 and gives ERR_CONNECTION_REFUSED - https://phabricator.wikimedia.org/T393620 (10Novem_Linguae) 03NEW
[16:44:00] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
[16:44:38] <wikibugs>	 10Tool-techcontribs, 07Upstream: hide mediawiki/wmf sub-groups from the gerrit groups section - https://phabricator.wikimedia.org/T393534#10801354 (10Novem_Linguae)
[16:46:19] <wikibugs>	 (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/64 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[16:46:22] <wikibugs>	 (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/64 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[16:47:33] <wikibugs>	 10Tool-techcontribs: clearer explanation of the difference between "developer account shell name" and "developer account name" - https://phabricator.wikimedia.org/T393622 (10Novem_Linguae) 03NEW
[16:48:17] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.67-20250507163032-f26d5cd1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/764
[16:48:20] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.67-20250507163032-f26d5cd1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/764
[16:52:32] <wikibugs>	 10Tool-techcontribs: in GitHub section, include organizations that the user is a member of - https://phabricator.wikimedia.org/T393623 (10Novem_Linguae) 03NEW
[16:57:36] <wikibugs>	 (03open) 10dcaro: create_toolforge_deploy_mr: update MR title/desc if exists [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/55
[17:01:09] <wikibugs>	 (03update) 10dcaro: create_toolforge_deploy_mr: update MR title/desc if exists [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/55
[17:05:15] <wikibugs>	 (03update) 10dcaro: create_precommit_update_mrs: add the `deploy: skip` header [repos/cloud/cicd/gitlab-ci] (add_skip_deploy_detection) - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/54
[17:07:50] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 07Documentation: Update roadmap for Toolforge Build Service - https://phabricator.wikimedia.org/T393549#10801508 (10dcaro) >>! In T393549#10801085, @Sascha wrote: > Sounds good, and good luck with the work! Just curious, is it possible to say when...
[17:08:12] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 07Documentation: Update roadmap for Toolforge Build Service - https://phabricator.wikimedia.org/T393549#10801511 (10dcaro) a:03dcaro
[17:08:20] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 07Documentation: Update roadmap for Toolforge Build Service - https://phabricator.wikimedia.org/T393549#10801515 (10dcaro) 05Open→03Resolved
[17:08:33] <wikibugs>	 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: Check for diff in services when running diff_with_running_job - https://phabricator.wikimedia.org/T392717#10801518 (10dcaro) p:05Triage→03Medium
[17:09:40] <wikibugs>	 10wikitech.wikimedia.org, 06serviceops-radar, 06SRE, 06SRE Observability: Move meta monitoring off of wikitech-static - https://phabricator.wikimedia.org/T393625 (10andrea.denisse) 03NEW
[17:09:45] <wikibugs>	 10Toolforge (Toolforge iteration 19), 07Epic: [cicd] Streamline toolforge cli deployment and external contributor ci flows - https://phabricator.wikimedia.org/T392524#10801530 (10dcaro)
[17:09:59] <wikibugs>	 10Toolforge (Toolforge iteration 19), 07Epic: [cicd] Streamline toolforge cli deployment and external contributor ci flows - https://phabricator.wikimedia.org/T392524#10801532 (10dcaro) p:05Triage→03Medium
[17:11:27] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19): [builds-builder] Golang buildpack does not allow using Procfiles so can't use custom scripts/entrypoints - https://phabricator.wikimedia.org/T390845#10801536 (10dcaro)
[17:12:21] <wikibugs>	 10wikitech.wikimedia.org, 06serviceops-radar, 06SRE, 13Patch-For-Review, 07SRE-Unowned: Redesign wikitech-static - https://phabricator.wikimedia.org/T376400#10801541 (10andrea.denisse) Hi @RobH @Andrew , we have Meta Monitoring enabled in the Wikitech static Rackspace host. Could you please provide the o...
[17:12:30] <wikibugs>	 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [components-api] Rename the CRDs groups to be `components-api.toolforge.org` - https://phabricator.wikimedia.org/T386829#10801544 (10dcaro) 05Open→03In progress
[17:13:49] <wikibugs>	 10wikitech.wikimedia.org, 06serviceops-radar, 06SRE, 13Patch-For-Review, 07SRE-Unowned: Redesign wikitech-static - https://phabricator.wikimedia.org/T376400#10801548 (10RobH) So I actually have no login rights (and don't need them) for the new AWS hosted wikitech static deployment.  I just pay the AWS bi...
[17:14:48] <wikibugs>	 (03approved) 10dcaro: components-api: bump to 0.0.104-20250507161948-39977c69 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/763 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[17:14:51] <wikibugs>	 (03merge) 10dcaro: components-api: bump to 0.0.104-20250507161948-39977c69 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/763 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[17:15:10] <wikibugs>	 (03update) 10dcaro: api-gateway: bump to 0.0.67-20250507163032-f26d5cd1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/764 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[17:15:13] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
[17:23:01] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Upgrade cloud-vps openstack to version 'Epoxy' - https://phabricator.wikimedia.org/T390914#10801597 (10Andrew) a:03Andrew
[17:27:33] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
[17:27:52] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: Enable IPv6 for the Cloud VPS web proxy - https://phabricator.wikimedia.org/T379175#10801608 (10taavi) Noting for the records that the above-mentioned security group update resulted in https://wikitech.wikimedia.org/wiki/Incidents/2025-05-07_c...
[17:32:38] <wikibugs>	 10Tool-techcontribs: Add reporting on Quips mentions - https://phabricator.wikimedia.org/T393627 (10taavi) 03NEW
[17:37:44] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Upgrade cloud-vps openstack to version 'Epoxy' - https://phabricator.wikimedia.org/T390914#10801649 (10Andrew) Checking release notes for:  [x] glance -- no changes needed [x] magnum -- no changes needed but we should investigate new cluster drivers; heat driver is now depre...
[17:58:43] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.remove_instance for instance tools-legacy-redirector-2
[17:59:19] <jinxer-wm>	 FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[18:00:20] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-legacy-redirector-2
[18:03:16] <wmcs-alerts>	 FIRING: [3x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[18:05:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[18:08:16] <wmcs-alerts>	 FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[18:09:19] <jinxer-wm>	 RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[18:10:51] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: Repurpose 5 config B servers - https://phabricator.wikimedia.org/T380805#10801829 (10Andrew) 05Open→03Resolved a:03Andrew Yes! the other three were repurposed in https://phabricator.wikimedia.org/T392539
[18:11:13] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: wmcs-cookbooks: update the openstack restart cookbook to prioritize control nodes - https://phabricator.wikimedia.org/T393610#10801836 (10Andrew) 05Open→03Resolved
[18:13:16] <wmcs-alerts>	 FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[18:13:31] <wmcs-alerts>	 FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[18:17:26] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudweb.set_maintenance (T390914)
[18:17:33] <stashbot>	 T390914: Upgrade cloud-vps openstack to version 'Epoxy' - https://phabricator.wikimedia.org/T390914
[18:18:08] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) (T390914)
[18:18:16] <wmcs-alerts>	 FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[18:18:31] <wmcs-alerts>	 FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[18:22:19] <jinxer-wm>	 FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[18:23:16] <wmcs-alerts>	 FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[18:23:27] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004.eqiad.wmnet' (T390914)
[18:23:28] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2004.eqiad.wmnet' (T390914)
[18:23:31] <wmcs-alerts>	 FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[18:23:34] <stashbot>	 T390914: Upgrade cloud-vps openstack to version 'Epoxy' - https://phabricator.wikimedia.org/T390914
[18:23:46] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004.codfw.wmnet' (T390914)
[18:23:46] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2004.codfw.wmnet' (T390914)
[18:24:08] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet' (T390914)
[18:27:19] <jinxer-wm>	 RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[18:28:16] <wmcs-alerts>	 FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[18:32:06] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2004-dev.codfw.wmnet' (T390914)
[18:32:12] <stashbot>	 T390914: Upgrade cloud-vps openstack to version 'Epoxy' - https://phabricator.wikimedia.org/T390914
[18:32:13] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet' (T390914)
[18:33:16] <wmcs-alerts>	 RESOLVED: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[18:39:48] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2005-dev.codfw.wmnet' (T390914)
[18:39:56] <stashbot>	 T390914: Upgrade cloud-vps openstack to version 'Epoxy' - https://phabricator.wikimedia.org/T390914
[18:43:51] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet' (T390914)
[18:58:45] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2004-dev.codfw.wmnet' (T390914)
[18:58:52] <stashbot>	 T390914: Upgrade cloud-vps openstack to version 'Epoxy' - https://phabricator.wikimedia.org/T390914
[19:04:19] <jinxer-wm>	 FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[19:04:23] <wikibugs>	 (03open) 10chuckonwumelu: Importing: Floating IPs from Tools & Toolsbeta [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/20
[19:05:13] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfw.wmnet' (T390914)
[19:05:20] <stashbot>	 T390914: Upgrade cloud-vps openstack to version 'Epoxy' - https://phabricator.wikimedia.org/T390914
[19:09:43] <wikibugs>	 10Tool-translatetagger: Put inline chunks of code in tvars instead of leaving them out of translate tags completely. - https://phabricator.wikimedia.org/T393255#10802048 (10theprotonade) I was reading some documentation and it seems that chunks of `<code></code>` should not be wrapped with tvars. Am I missing so...
[19:15:25] <wikibugs>	 (03update) 10chuckonwumelu: Importing: Floating IPs from Tools & Toolsbeta [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/20 (https://phabricator.wikimedia.org/T390056)
[19:19:20] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2005-dev.codfw.wmnet' (T390914)
[19:19:21] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2006-dev.codfw.wmnet' (T390914)
[19:19:28] <stashbot>	 T390914: Upgrade cloud-vps openstack to version 'Epoxy' - https://phabricator.wikimedia.org/T390914
[19:23:27] <wikibugs>	 (03update) 10chuckonwumelu: Importing: Floating IPs from Tools & Toolsbeta [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/20 (https://phabricator.wikimedia.org/T390056)
[19:24:18] <wikibugs>	 (03update) 10chuckonwumelu: Importing: Floating IPs from Tools & Toolsbeta [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/20 (https://phabricator.wikimedia.org/T390056)
[19:34:25] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2006-dev.codfw.wmnet' (T390914)
[19:34:26] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet' (T390914)
[19:34:31] <stashbot>	 T390914: Upgrade cloud-vps openstack to version 'Epoxy' - https://phabricator.wikimedia.org/T390914
[19:42:59] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2006-dev.codfw.wmnet' (T390914)
[19:43:00] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' (T390914)
[19:43:08] <stashbot>	 T390914: Upgrade cloud-vps openstack to version 'Epoxy' - https://phabricator.wikimedia.org/T390914
[19:46:16] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Upgrade cloud-vps openstack to version 'Epoxy' - https://phabricator.wikimedia.org/T390914#10802180 (10RhinosF1)
[19:51:34] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2005-dev.codfw.wmnet' (T390914)
[19:51:41] <stashbot>	 T390914: Upgrade cloud-vps openstack to version 'Epoxy' - https://phabricator.wikimedia.org/T390914
[19:57:10] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet' (T390914)
[19:57:17] <stashbot>	 T390914: Upgrade cloud-vps openstack to version 'Epoxy' - https://phabricator.wikimedia.org/T390914
[19:59:30] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,designate
[20:01:52] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,designate
[20:02:19] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet' (T390914)
[20:02:20] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2005-dev.codfw.wmnet' (T390914)
[20:02:25] <stashbot>	 T390914: Upgrade cloud-vps openstack to version 'Epoxy' - https://phabricator.wikimedia.org/T390914
[20:07:56] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet' (T390914)
[20:07:57] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2006-dev.codfw.wmnet' (T390914)
[20:08:03] <stashbot>	 T390914: Upgrade cloud-vps openstack to version 'Epoxy' - https://phabricator.wikimedia.org/T390914
[20:09:19] <jinxer-wm>	 RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[20:13:36] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2006-dev.codfw.wmnet' (T390914)
[20:13:43] <stashbot>	 T390914: Upgrade cloud-vps openstack to version 'Epoxy' - https://phabricator.wikimedia.org/T390914
[20:29:19] <jinxer-wm>	 FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[20:34:19] <jinxer-wm>	 RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[20:45:25] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1001-dev.eqiad.wmnet' (T390914)
[20:45:31] <stashbot>	 T390914: Upgrade cloud-vps openstack to version 'Epoxy' - https://phabricator.wikimedia.org/T390914
[20:47:47] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Upgrade cloud-vps openstack to version 'Epoxy' - https://phabricator.wikimedia.org/T390914#10802431 (10Andrew) codfw1dev is now running epoxy, and the fullstack test and policy tests are passing.
[20:50:15] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1001-dev.eqiad.wmnet' (T390914)
[20:50:16] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1002-dev.eqiad.wmnet' (T390914)
[20:55:15] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1002-dev.eqiad.wmnet' (T390914)
[20:55:19] <jinxer-wm>	 FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[20:55:25] <stashbot>	 T390914: Upgrade cloud-vps openstack to version 'Epoxy' - https://phabricator.wikimedia.org/T390914
[21:00:19] <jinxer-wm>	 RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[22:05:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[23:00:19] <wikibugs>	 10Tool-kuwikibot, 10Toolhub: Invalid source code and issues URL on https://toolsadmin.wikimedia.org/tools/id/kuwikibot - https://phabricator.wikimedia.org/T361553#10802787 (10bd808) The $HOME of the tool is empty except for normal system generated files. I would guess there was an intent but never an implement...