[01:08:31] <wikibugs>	 10Tool-wikimonitor: Saved edit summaries for faster undo and rollback actions - https://phabricator.wikimedia.org/T428349 (10Gerges) 03NEW
[01:08:53] <wikibugs>	 10Tool-wikimonitor: Saved edit summaries for faster undo and rollback actions - https://phabricator.wikimedia.org/T428349#11991319 (10Gerges) p:05Triage→03Low
[03:32:44] <jinxer-wm>	 FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[03:37:44] <jinxer-wm>	 RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[04:56:50] <wikibugs>	 10VPS-project-voterlists, 10MediaWiki-extensions-SecurePoll, 06Product Safety and Integrity: Switch global election workflow to use voterlists.wmcloud.org? - https://phabricator.wikimedia.org/T423547#11991403 (10SD0001) >>! In T423547#11923739, @jrbs wrote: > Unfortunately I can't seem to use the file with `...
[12:15:10] <wikibugs>	 06cloud-services-team, 10Data-Services: [wikireplicas] Create views for new wiki urwikisource - https://phabricator.wikimedia.org/T415977#11991466 (10Samwilson) #ws_export has also recently started erroring with failures to connect to `urwikisource_p`.
[12:24:13] <wikibugs>	 10Tool-redminbot: Grab list of modules from wiki replicas instead of file - https://phabricator.wikimedia.org/T417783#11991467 (10Redmin) a:03Redmin
[12:28:35] <wikibugs>	 06cloud-services-team, 10Toolforge: Specifying --filelog-stdout or --filelog-stderr requires --filelog - https://phabricator.wikimedia.org/T428354 (10Huji) 03NEW
[12:36:50] <wikibugs>	 (03open) 10r4356th: Grab list of modules from wiki replicas instead of file [toolforge-repos/redminbot] - 10https://gitlab.wikimedia.org/toolforge-repos/redminbot/-/merge_requests/20 (https://phabricator.wikimedia.org/T417783)
[12:44:39] <wikibugs>	 (03update) 10r4356th: Grab list of modules from wiki replicas instead of file [toolforge-repos/redminbot] - 10https://gitlab.wikimedia.org/toolforge-repos/redminbot/-/merge_requests/20 (https://phabricator.wikimedia.org/T417783)
[12:45:56] <wikibugs>	 (03merge) 10r4356th: Grab list of modules from wiki replicas instead of file [toolforge-repos/redminbot] - 10https://gitlab.wikimedia.org/toolforge-repos/redminbot/-/merge_requests/20 (https://phabricator.wikimedia.org/T417783)
[12:55:11] <wikibugs>	 10Tool-redminbot, 13Patch-For-Review: Grab list of modules from wiki replicas instead of file - https://phabricator.wikimedia.org/T417783#11991497 (10Redmin) a:05Redmin→03None
[13:51:09] <wikibugs>	 06cloud-services-team, 10Toolforge: Specifying --filelog-stdout or --filelog-stderr requires --filelog - https://phabricator.wikimedia.org/T428354#11991525 (10Huji)
[13:56:03] <wikibugs>	 06cloud-services-team, 10Toolforge: Specifying --filelog-stdout or --filelog-stderr requires --filelog - https://phabricator.wikimedia.org/T428354#11991526 (10Huji) Regression was caused by [https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/commit/0503db2a06ca2421781f4bc39eb647e8c971a3e3 this chang...
[14:10:25] <wikibugs>	 06cloud-services-team, 10Toolforge: Specifying --filelog-stdout or --filelog-stderr requires --filelog - https://phabricator.wikimedia.org/T428354#11991540 (10HakanIST) Tagging @Raymond_Ndibe
[15:03:19] <wikibugs>	 10Tool-delintbot, 13Patch-For-Review: Fix cases of tags not being closed correctly - https://phabricator.wikimedia.org/T417483#11991571 (10Kavaljeet_Singh) a:05Kavaljeet_Singh→03None
[17:39:58] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
[17:40:57] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06Release-Engineering-Team (Radar): Magnum cluster stuck in DELETE_FAILED status - https://phabricator.wikimedia.org/T428312#11991701 (10Andrew) This is producing an internal permissions error, as though heat isn't allowed to delete the things it just created.
[17:42:22] <jinxer-wm>	 FIRING: [7x] HAProxyBackendUnavailable: HAProxy service glance-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable
[17:47:22] <jinxer-wm>	 RESOLVED: [14x] HAProxyBackendUnavailable: HAProxy service glance-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable
[17:48:51] <wikibugs>	 06cloud-services-team, 10Tool-curator, 10Toolforge: Rotate MariaDB SQL password for tool-curator - https://phabricator.wikimedia.org/T428367 (10DaxServer) 03NEW
[17:55:24] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
[18:11:38] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06Release-Engineering-Team (Radar): Magnum cluster stuck in DELETE_FAILED status - https://phabricator.wikimedia.org/T428312#11991716 (10Andrew) Restarting services didn't help, so the next step is to hook the policy engine and see who and what is not allowed. That might b...