[00:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [00:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:02:06] 10Cloud-VPS (Project-requests), 07affects-Miraheze: Request creation of createwikitest VPS project - https://phabricator.wikimedia.org/T375454#10190162 (10Xaloria) I want to host an wiki in Cloud VPS that's why I'm asking. [02:39:50] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 06serviceops, 13Patch-For-Review: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10190282 (10Bugreporter) [06:00:42] 10Toolforge-standards-committee (Maintainer needed), 10Tools, 10Wikidata: The EditGroups tool needs new maintainer(s) - https://phabricator.wikimedia.org/T376072#10190425 (10Pintoch) That's a move I have already done, and it did help with making the service more stable indeed. The remaining issues I am aware... [06:33:31] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/9 [06:33:49] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/91 [06:33:55] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/91 [06:33:59] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/10 [06:34:09] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/74 [06:34:11] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/125 [06:34:35] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/74 [06:34:36] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/10 [06:34:36] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/64 [06:34:38] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/125 [06:34:39] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/15 [06:34:39] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/64 [06:34:47] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/15 [06:35:02] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/12 [06:35:03] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/12 [06:35:06] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/41 [06:35:09] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/41 [06:35:11] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/30 [06:35:19] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/30 [06:35:27] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/13 [06:35:32] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/13 [06:35:33] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/19 [06:35:34] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/19 [06:35:43] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/113 [06:35:44] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/113 [06:35:45] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/42 [06:35:45] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/192 [06:35:46] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/42 [06:35:47] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/192 [08:05:08] (03merge) 10tkarcher: live scripts from toolforge back to gitlab [toolforge-repos/erinnermich] - 10https://gitlab.wikimedia.org/toolforge-repos/erinnermich/-/merge_requests/2 (owner: 10mirji) [08:06:47] (03CR) 10David Caro: [C:03+2] ceph.undrain_osds_in_chunks: fix the chunking [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076796 (owner: 10David Caro) [08:10:09] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814) [08:10:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:10:17] T372814: Put cloudcephosd10[39-41] into service - https://phabricator.wikimedia.org/T372814 [08:10:23] (03Merged) 10jenkins-bot: ceph.undrain_osds_in_chunks: fix the chunking [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076796 (owner: 10David Caro) [08:12:16] (03close) 10dcaro: build.Start: initialize the envvars array so it does not send nil [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/112 [08:14:37] (03update) 10dcaro: builds-buidler: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 [08:17:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [08:18:26] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814) [08:18:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:18:33] T372814: Put cloudcephosd10[39-41] into service - https://phabricator.wikimedia.org/T372814 [08:23:01] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814) [08:23:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:25:18] (03update) 10dcaro: builds-buidler: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 [08:38:58] FIRING: SystemdUnitDown: The systemd unit wmf_auto_restart_virtlogd.service on node cloudvirt1063 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudvirt1063 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [08:39:10] 06cloud-services-team: SystemdUnitDown Unit wmf_auto_restart_virtlogd.service on node cloudvirt1063 has been down for long. - https://phabricator.wikimedia.org/T376109 (10phaultfinder) 03NEW [08:41:28] (03update) 10dcaro: builds-buidler: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 [08:42:55] (03update) 10dcaro: builds-buidler: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 [08:45:31] (03update) 10dcaro: builds-buidler: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 [08:51:48] (03PS1) 10JMeybohm: kubernetes::worker_containerd: Add registry 'secrets' [labs/private] - 10https://gerrit.wikimedia.org/r/1076973 (https://phabricator.wikimedia.org/T362408) [08:53:45] 06cloud-services-team, 10Cloud-VPS: tofu-infra: add zone templating support to replace wmfkeystonehook - https://phabricator.wikimedia.org/T376110 (10aborrero) 03NEW [08:54:21] 06cloud-services-team, 10Cloud-VPS: tofu-infra: add support for DNS zones created by wmfkeystonehook - https://phabricator.wikimedia.org/T376110#10190667 (10aborrero) [08:54:56] (03CR) 10JMeybohm: [V:03+2 C:03+2] kubernetes::worker_containerd: Add registry 'secrets' [labs/private] - 10https://gerrit.wikimedia.org/r/1076973 (https://phabricator.wikimedia.org/T362408) (owner: 10JMeybohm) [09:08:32] (03update) 10dcaro: builds-buidler: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 [09:14:35] (03update) 10dcaro: builds-buidler: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 [09:15:00] FIRING: Primary cloud switch inbound port utilisation over 80%: Alert for device cloudsw1-e4-eqiad.mgmt.eqiad.wmnet - Primary cloud switch inbound port utilisation over 80% - https://alerts.wikimedia.org/?q=alertname%3DPrimary+cloud+switch+inbound+port+utilisation+over+80%25 [09:15:00] FIRING: Primary cloud switch port utilisation over 80%: Alert for device cloudsw1-d5-eqiad.mgmt.eqiad.wmnet - Primary cloud switch port utilisation over 80% - https://alerts.wikimedia.org/?q=alertname%3DPrimary+cloud+switch+port+utilisation+over+80%25 [09:15:11] 06cloud-services-team: Primary cloud switch port utilisation over 80% Rule: Primary cloud switch port utilisation over 80% Faults: #1: sysObjectID = .1.3.6.1.4.1.2636.1.1.1.4.82.5; sysDescr = Juniper Networks, Inc. qfx5100-48s-6q Ethernet Switch, kerne... - https://phabricator.wikimedia.org/T376019#10190736 [09:15:12] 06cloud-services-team: Primary cloud switch inbound port utilisation over 80% Rule: Primary cloud switch inbound port utilisation over 80% Faults: #1: sysObjectID = .1.3.6.1.4.1.2636.1.1.1.4.82.22; sysDescr = Juniper Networks, Inc. qfx5120-48y-8c Ether... - https://phabricator.wikimedia.org/T376018#10190735 [09:15:39] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [09:15:48] (03update) 10dcaro: builds-buidler: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 [09:18:00] (03update) 10aborrero: projects: refactor into new repository layout [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/69 (https://phabricator.wikimedia.org/T375283) [09:20:00] RESOLVED: Primary cloud switch inbound port utilisation over 80%: Device cloudsw1-e4-eqiad.mgmt.eqiad.wmnet recovered from Primary cloud switch inbound port utilisation over 80% - https://alerts.wikimedia.org/?q=alertname%3DPrimary+cloud+switch+inbound+port+utilisation+over+80%25 [09:20:01] RESOLVED: Primary cloud switch port utilisation over 80%: Device cloudsw1-d5-eqiad.mgmt.eqiad.wmnet recovered from Primary cloud switch port utilisation over 80% - https://alerts.wikimedia.org/?q=alertname%3DPrimary+cloud+switch+port+utilisation+over+80%25 [09:24:09] (03update) 10aborrero: projects: refactor into new repository layout [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/69 (https://phabricator.wikimedia.org/T375283) [09:24:40] (03update) 10dcaro: builds-buidler: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 [09:27:19] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/10 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:27:25] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/10 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:27:32] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/74 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:27:35] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/74 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:27:50] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/192 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:27:53] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/192 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:27:58] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/42 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:28:01] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/42 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:28:07] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/113 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:28:11] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/113 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:28:14] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/13 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:28:18] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/13 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:28:21] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/19 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:28:25] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/19 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:28:42] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/15 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:28:46] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/15 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:28:49] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/12 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:28:53] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/12 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:30:26] (03approved) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/64 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:30:30] (03merge) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/64 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:30:45] (03approved) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/91 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:30:49] (03merge) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/91 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:31:04] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: api-gateway: bump to 0.0.45-20241001092818-fb288a5c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/537 [09:31:09] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: api-gateway: bump to 0.0.45-20241001092818-fb288a5c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/537 [09:31:58] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: components-api: bump to 0.0.19-20241001092739-d389755b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/538 [09:32:30] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: registry-admission: bump to 0.0.52-20241001092859-1f1de83c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/539 [09:32:33] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: registry-admission: bump to 0.0.52-20241001092859-1f1de83c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/539 [09:32:49] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: ingress-admission: bump to 0.0.52-20241001092905-6e9e386c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/540 [09:32:50] (03update) 10aborrero: projects: refactor into new repository layout [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/69 (https://phabricator.wikimedia.org/T375283) [09:32:54] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: ingress-admission: bump to 0.0.52-20241001092905-6e9e386c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/540 [09:33:46] (03approved) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/30 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:33:50] (03merge) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/30 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:34:13] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: volume-admission: bump to 0.0.57-20241001092841-08c6322e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/541 [09:34:26] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission [09:36:09] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: builds-api: bump to 0.0.171-20241001092826-1929cb0c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/542 [09:36:11] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: builds-api: bump to 0.0.171-20241001092826-1929cb0c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/542 [09:38:54] (03PS1) 10David Caro: ceph: use prometheus-node-pinger for ping checks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076978 [09:40:00] (03update) 10aborrero: projects: refactor into new repository layout [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/69 (https://phabricator.wikimedia.org/T375283) [09:40:17] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission [09:40:27] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Improve WMCS NodeDown alerts - https://phabricator.wikimedia.org/T375479#10190805 (10fnegri) 05Open→03In progress [09:42:50] (03CR) 10CI reject: [V:04-1] ceph: use prometheus-node-pinger for ping checks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076978 (owner: 10David Caro) [09:46:05] (03PS1) 10David Caro: toolforge.component.deploy: show the MR comment link [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076980 [09:47:20] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission [09:49:26] (03CR) 10CI reject: [V:04-1] toolforge.component.deploy: show the MR comment link [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076980 (owner: 10David Caro) [09:50:54] 06cloud-services-team, 10Cloud-VPS: tofu-infra: add support for DNS zones created by wmfkeystonehook - https://phabricator.wikimedia.org/T376110#10190832 (10aborrero) p:05Triage→03Medium [09:52:21] (03approved) 10fnegri: projects: refactor into new repository layout [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/69 (https://phabricator.wikimedia.org/T375283) (owner: 10aborrero) [09:52:34] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission [09:52:55] 06cloud-services-team, 10Cloud-VPS, 10Wikidata, 10Wikidata Analytics: Delete the wmdeanalytics Cloud VPS project - https://phabricator.wikimedia.org/T371696#10190830 (10AndrewTavis_WMDE) Hey @taavi 👋 Thanks for the direction here! @ItamarWMDE and I just took a look at the instances for the project and they... [09:53:32] (03update) 10dcaro: builds-buidler: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 [09:53:35] (03merge) 10aborrero: projects: refactor into new repository layout [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/69 (https://phabricator.wikimedia.org/T375283) [09:53:44] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [09:55:07] (03update) 10dcaro: builds-buidler: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 [09:57:05] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [09:59:28] (03PS2) 10David Caro: toolforge.component.deploy: show the MR comment link [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076980 [09:59:44] (03approved) 10dcaro: ingress-admission: bump to 0.0.52-20241001092905-6e9e386c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/540 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:59:47] (03merge) 10dcaro: ingress-admission: bump to 0.0.52-20241001092905-6e9e386c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/540 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [10:00:01] (03update) 10dcaro: api-gateway: bump to 0.0.45-20241001092818-fb288a5c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/537 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [10:00:25] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [10:01:33] (03update) 10dcaro: builds-buidler: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 [10:03:00] (03CR) 10CI reject: [V:04-1] toolforge.component.deploy: show the MR comment link [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076980 (owner: 10David Caro) [10:05:45] (03update) 10dcaro: builds-buidler: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 [10:06:05] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway [10:07:26] (03update) 10dcaro: builds-buidler: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 [10:08:28] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [10:08:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:08:46] FIRING: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_toolserver_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:13:45] RESOLVED: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_toolserver_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:13:57] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway [10:14:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:14:13] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: tofu-infra: refactor repo structure - https://phabricator.wikimedia.org/T375283#10190903 (10fnegri) We discussed this a bit further in a meeting yesterday with @aborrero, some takeaways: * My idea of having only 3 or 4 "core projects" being present in `... [10:17:48] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [10:18:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:23:11] (03update) 10dcaro: volume-admission: bump to 0.0.57-20241001092841-08c6322e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/541 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [10:27:24] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway [10:27:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:27:31] (03PS3) 10David Caro: toolforge.component.deploy: show the MR comment link [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076980 [10:27:43] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [10:30:54] (03CR) 10CI reject: [V:04-1] toolforge.component.deploy: show the MR comment link [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076980 (owner: 10David Caro) [10:31:30] (03update) 10aborrero: network: have projects in the resouce id [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/70 (https://phabricator.wikimedia.org/T375283) [10:32:31] (03PS2) 10David Caro: ceph: use prometheus-node-pinger for ping checks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076978 [10:32:31] (03PS4) 10David Caro: toolforge.component.deploy: show the MR comment link [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076980 [10:32:37] (03update) 10aborrero: network: have projects in the resouce id [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/70 (https://phabricator.wikimedia.org/T375283) [10:32:54] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [10:33:19] (03CR) 10David Caro: [V:03+1] "Tested in cloudcumin and locally:" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076980 (owner: 10David Caro) [10:35:36] (03PS3) 10David Caro: ceph: use prometheus-node-pinger for ping checks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076978 [10:35:36] (03PS5) 10David Caro: toolforge.component.deploy: show the MR comment link [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076980 [10:36:34] (03CR) 10David Caro: "Tested locally:" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076978 (owner: 10David Caro) [10:37:32] (03update) 10aborrero: network: have projects in the resouce id [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/70 (https://phabricator.wikimedia.org/T375283) [10:37:52] (03PS4) 10David Caro: ceph: use prometheus-node-pinger for ping checks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076978 [10:37:52] (03CR) 10David Caro: ceph: use prometheus-node-pinger for ping checks (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076978 (owner: 10David Caro) [10:37:52] (03PS6) 10David Caro: toolforge.component.deploy: show the MR comment link [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076980 [10:38:11] (03CR) 10David Caro: ceph: use prometheus-node-pinger for ping checks (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076978 (owner: 10David Caro) [10:38:12] (03update) 10aborrero: network: have projects in the resouce id [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/70 (https://phabricator.wikimedia.org/T375283) [10:38:51] (03PS5) 10David Caro: ceph: use prometheus-node-pinger for ping checks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076978 [10:38:51] (03PS7) 10David Caro: toolforge.component.deploy: show the MR comment link [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076980 [10:40:29] (03approved) 10dcaro: api-gateway: bump to 0.0.45-20241001092818-fb288a5c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/537 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [10:40:32] (03merge) 10dcaro: api-gateway: bump to 0.0.45-20241001092818-fb288a5c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/537 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [10:40:44] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [10:40:56] (03update) 10dcaro: components-api: bump to 0.0.19-20241001092739-d389755b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/538 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [10:42:22] (03CR) 10CI reject: [V:04-1] toolforge.component.deploy: show the MR comment link [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076980 (owner: 10David Caro) [10:45:39] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [10:46:10] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [10:51:50] (03update) 10aborrero: network: have projects in the resouce id [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/70 (https://phabricator.wikimedia.org/T375283) [10:52:30] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [10:52:49] (03update) 10aborrero: network: have projects in the resouce id [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/70 (https://phabricator.wikimedia.org/T375283) [10:56:08] (03update) 10aborrero: network: have projects in the resouce id [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/70 (https://phabricator.wikimedia.org/T375283) [11:02:38] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: 2024-09-21 NodeDown cloudvirt1063 - https://phabricator.wikimedia.org/T375223#10191121 (10fnegri) [11:02:42] 06cloud-services-team: SystemdUnitDown Unit wmf_auto_restart_virtlogd.service on node cloudvirt1063 has been down for long. - https://phabricator.wikimedia.org/T376109#10191119 (10fnegri) →14Duplicate dup:03T375223 [11:10:27] (03open) 10aborrero: tofu-infra: use project name instead of tenant id [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/71 (https://phabricator.wikimedia.org/T375283) [11:11:12] (03update) 10aborrero: tofu-infra: use project name instead of tenant id [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/71 (https://phabricator.wikimedia.org/T375283) [11:12:29] (03update) 10aborrero: tofu-infra: use project name instead of tenant id [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/71 (https://phabricator.wikimedia.org/T375283) [11:13:01] (03update) 10aborrero: tofu-infra: use project name instead of tenant id [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/71 (https://phabricator.wikimedia.org/T375283) [11:13:20] 06cloud-services-team, 06Infrastructure-Foundations: Define single hiera key to identify WMCS-managed bare metal hosts - https://phabricator.wikimedia.org/T375673#10191156 (10fnegri) p:05Triage→03Medium [11:13:50] 06cloud-services-team, 10Cloud-VPS: Unable to create user in Trove postgres DB - https://phabricator.wikimedia.org/T292195#10191158 (10fnegri) p:05Triage→03Low [11:18:55] (03update) 10aborrero: network: have projects in the resouce id [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/70 (https://phabricator.wikimedia.org/T375283) [11:24:05] 06cloud-services-team, 10Cloud-VPS, 10observability: Determine how to monitor services in cloud-private / cloudlb - https://phabricator.wikimedia.org/T347148#10191207 (10fnegri) p:05Triage→03Medium [11:25:01] 06cloud-services-team, 10Cloud-VPS, 07Upstream: [trove] does not expose amount of disk space used - https://phabricator.wikimedia.org/T354728#10191215 (10fnegri) [11:26:53] 06cloud-services-team, 10Tool-openstack-browser: Trove quota usage not available from the API - https://phabricator.wikimedia.org/T286128#10191213 (10fnegri) →14Duplicate dup:03T354728 [11:28:50] 06cloud-services-team, 10Horizon, 13Patch-For-Review: Horizon should use a proxy to access cloud vps hosted apis - https://phabricator.wikimedia.org/T305453#10191221 (10fnegri) p:05Triage→03Low [11:29:27] 06cloud-services-team, 10Cloud-VPS: TLS support for OpenStack trove - https://phabricator.wikimedia.org/T294118#10191223 (10fnegri) p:05Triage→03Low [11:29:57] 06cloud-services-team, 10Data-Services: NFS-on-ceph: monitoring - https://phabricator.wikimedia.org/T301279#10191224 (10fnegri) p:05Triage→03Low [11:33:19] 06cloud-services-team: Primary cloud switch port utilisation over 80% Rule: Primary cloud switch port utilisation over 80% Faults: #1: sysObjectID = .1.3.6.1.4.1.2636.1.1.1.4.82.5; sysDescr = Juniper Networks, Inc. qfx5100-48s-6q Ethernet Switch, kern... - https://phabricator.wikimedia.org/T376019#10191231 [11:33:21] 06cloud-services-team: Primary cloud switch inbound port utilisation over 80% Rule: Primary cloud switch inbound port utilisation over 80% Faults: #1: sysObjectID = .1.3.6.1.4.1.2636.1.1.1.4.82.22; sysDescr = Juniper Networks, Inc. qfx5120-48y-8c Ethe... - https://phabricator.wikimedia.org/T376018#10191233 [11:40:37] 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops: ☂ Migrate Wikitech to Kubernetes - https://phabricator.wikimedia.org/T292707#10191281 (10Ladsgroup) [11:43:04] (03update) 10aborrero: network: have projects in the resouce id [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/70 (https://phabricator.wikimedia.org/T375283) [11:50:44] (03update) 10aborrero: network: have projects in the resouce id [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/70 (https://phabricator.wikimedia.org/T375283) [11:52:05] (03approved) 10fnegri: tofu-infra: use project name instead of tenant id [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/71 (https://phabricator.wikimedia.org/T375283) (owner: 10aborrero) [11:52:42] (03merge) 10aborrero: tofu-infra: use project name instead of tenant id [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/71 (https://phabricator.wikimedia.org/T375283) [11:55:50] (03approved) 10fnegri: network: have projects in the resouce id [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/70 (https://phabricator.wikimedia.org/T375283) (owner: 10aborrero) [12:00:32] (03update) 10aborrero: network: have projects in the resouce id [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/70 (https://phabricator.wikimedia.org/T375283) [12:00:50] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:02:12] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [12:05:59] (03merge) 10aborrero: network: have projects in the resouce id [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/70 (https://phabricator.wikimedia.org/T375283) [12:10:58] (03open) 10aborrero: utils: fix ports and router path in migration script [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/72 (https://phabricator.wikimedia.org/T375283) [12:12:12] (03merge) 10aborrero: utils: fix ports and router path in migration script [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/72 (https://phabricator.wikimedia.org/T375283) [12:18:08] (03open) 10aborrero: utils/eqiad1-move.sh: fix network references [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/73 (https://phabricator.wikimedia.org/T375283) [12:18:16] (03merge) 10aborrero: utils/eqiad1-move.sh: fix network references [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/73 (https://phabricator.wikimedia.org/T375283) [12:24:29] (03open) 10aborrero: network objects: don't add the project name to the openstack resource name [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/74 (https://phabricator.wikimedia.org/T375283) [12:31:56] (03open) 10aborrero: utils: remove them [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/75 [12:36:01] (03merge) 10aborrero: network objects: don't add the project name to the openstack resource name [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/74 (https://phabricator.wikimedia.org/T375283) [12:36:08] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:36:38] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [12:41:56] 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops, 13Patch-For-Review: mediawiki-config: consolidate labswiki - https://phabricator.wikimedia.org/T371374#10191548 (10jijiki) 05Resolved→03Stalled [12:42:32] 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops: Traffic: Direct wikitech.wikimedia.org to mw-on-k8s - https://phabricator.wikimedia.org/T371358#10191523 (10jijiki) 05Open→03Resolved ATS is done [12:43:34] 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops: Migrate Wikitech's Jobqueue - https://phabricator.wikimedia.org/T371359#10191526 (10jijiki) 05Open→03Resolved Jobs are running on mw-jobrunner [12:43:48] 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops: Apache: Include Wikitech in mw-on-k8s' virtual hosts - https://phabricator.wikimedia.org/T371360#10191556 (10jijiki) 05Open→03Resolved Rolled out the apache change. [12:44:52] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 06serviceops: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10191571 (10jijiki) 05Open→03Resolved LDAP extension has been removed. [12:45:31] 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops: Memcached: Migrate wikitech to main memcached - https://phabricator.wikimedia.org/T371608#10191552 (10jijiki) 05Open→03Resolved Wikitech is using our main memcached cluster, so far so good. [12:46:58] 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops, 13Patch-For-Review: mediawiki-config: consolidate labswiki - https://phabricator.wikimedia.org/T371374#10191542 (10jijiki) 05Open→03Resolved The de-wikitech patch has been merged, however I am leaving this open for a few more days in case something... [12:47:44] 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops: ☂ Migrate Wikitech to Kubernetes - https://phabricator.wikimedia.org/T292707#10191593 (10jijiki) p:05High→03Medium @Ladsgroup and I flipped the relevant switches today, for the time being all seems well. [12:48:27] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 06serviceops: wikitech self-auth: Allow wikitech to use its own internal authentication - https://phabricator.wikimedia.org/T371588#10191595 (10Ladsgroup) 05Open→03Resolved This is done. [12:55:59] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install cloudlb2004-dev - https://phabricator.wikimedia.org/T370678#10191609 (10aborrero) I'm sorry I completely missed the pings here. I will get this done today. [12:56:43] (03approved) 10dcaro: components-api: bump to 0.0.19-20241001092739-d389755b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/538 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [12:56:45] (03merge) 10dcaro: components-api: bump to 0.0.19-20241001092739-d389755b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/538 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [13:03:06] 10wikitech.wikimedia.org: Setting new passwort at wikitech does not work - https://phabricator.wikimedia.org/T376140 (10Ameisenigel) 03NEW [13:03:27] 10wikitech.wikimedia.org: Setting new password at wikitech does not work - https://phabricator.wikimedia.org/T376140#10191635 (10Ameisenigel) [13:09:56] 10wikitech.wikimedia.org: Setting new password at wikitech does not work - https://phabricator.wikimedia.org/T376140#10191639 (10Lucas_Werkmeister_WMDE) > Because of the process to connect this wiki with SUL you have to request a temporary password Are you sure logging in with your “old” Wikitech password doesn... [13:11:15] 10Tool-wikiloves: Armenia in WLM statistic. - https://phabricator.wikimedia.org/T375998#10191640 (10Beko) 05Open→03Resolved a:03Beko [13:30:35] (03approved) 10fnegri: utils: remove them [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/75 (owner: 10aborrero) [13:32:28] 06cloud-services-team, 10wikitech.wikimedia.org, 10Wikimedia-Extension-setup, 07Documentation, 07I18n: Install Translate extension on wikitech - https://phabricator.wikimedia.org/T100313#10191749 (10Bugreporter) This is suggested again in https://wikitech.wikimedia.org/wiki/Talk:Wikitech/SUL-migration#c-... [13:45:27] 10wikitech.wikimedia.org: Setting new password at wikitech does not work - https://phabricator.wikimedia.org/T376140#10191785 (10Ameisenigel) >>! In T376140#10191639, @Lucas_Werkmeister_WMDE hat geschrieben: >> Because of the process to connect this wiki with SUL you have to request a temporary password > > Are... [13:46:26] 10wikitech.wikimedia.org: Setting new password at wikitech does not work - https://phabricator.wikimedia.org/T376140#10191789 (10MoritzMuehlenhoff) I can reproduce this as well if needed for debugging. [13:51:07] RECOVERY - ensure kvm processes are running on cloudvirt1063 is OK: PROCS OK: 1 process with regex args qemu-system-x86_64 https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [13:58:27] 10Tool-yearinreview, 06Indic MediaWiki Developers UG, 06Indic-TechCom: add support for punjabi in yearinreview tool - https://phabricator.wikimedia.org/T369465#10191837 (10Jayprakash12345) 05Open→03Resolved a:03Keerthan_16 https://github.com/indictechcom/wscontest [14:00:37] 10Tool-yearinreview, 06Indic MediaWiki Developers UG, 06Indic-TechCom: add support for punjabi in yearinreview tool - https://phabricator.wikimedia.org/T369465#10191848 (10Jayprakash12345) 05Resolved→03Open a:05Keerthan_16→03None [14:18:03] 10wikitech.wikimedia.org: Setting new password at wikitech does not work - https://phabricator.wikimedia.org/T376140#10191908 (10Ladsgroup) FWIW, lots of people could reset their passwords, that worksTM. I need to figure out what's causing this. [14:33:38] (03update) 10aborrero: utils: remove them [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/75 [15:04:35] 10wikitech.wikimedia.org: Setting new password at wikitech does not work - https://phabricator.wikimedia.org/T376140#10192104 (10Joe) >>! In T376140#10191908, @Ladsgroup wrote: > FWIW, lots of people could reset their passwords, that worksTM. I need to figure out what's causing this. People could before the mov... [15:16:24] (03open) 10taavi: Call accounts by their proper names [toolforge-repos/techcontribs] - 10https://gitlab.wikimedia.org/toolforge-repos/techcontribs/-/merge_requests/1 [15:17:00] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07LDAP, 13Patch-For-Review: Update Wikitech's LDAP credentials to be read-only - https://phabricator.wikimedia.org/T367287#10192144 (10Bugreporter) 05Stalled→03Open Since LdapAuthentication is gone these LDAP credentials s... [15:17:22] 10wikitech.wikimedia.org, 10MediaWiki-extensions-LdapAuthentication, 10MediaWiki-User-login-and-signup, 07Unstewarded-production-error, 07Wikimedia-production-error: Logging in on Wikitech can throw fatal from LdapAuthentication "CAS update failed on us... - https://phabricator.wikimedia.org/T249623#10192171 [15:19:55] 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops: Cleanup: Wikitech code leftovers - https://phabricator.wikimedia.org/T371378#10192186 (10taavi) [15:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:20:47] 06cloud-services-team, 10Striker: Warn cloud users against re-using keys - https://phabricator.wikimedia.org/T208599#10192185 (10Bugreporter) [15:21:10] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07LDAP, 13Patch-For-Review: Update Wikitech's LDAP credentials to be read-only - https://phabricator.wikimedia.org/T367287#10192183 (10taavi) →14Duplicate dup:03T371378 [15:21:57] 10wikitech.wikimedia.org, 10MediaWiki-Blocks, 10MediaWiki-extensions-LdapAuthentication, 07affects-Miraheze, and 2 others: Fatal exception of type "UnexpectedValueException" when attempting to block - https://phabricator.wikimedia.org/T305786#10192197 (10Bugreporter) 05Open→03Declined Extension dis... [15:22:03] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install cloudlb2004-dev - https://phabricator.wikimedia.org/T370678#10192209 (10aborrero) [15:24:22] 06cloud-services-team, 10wikitech.wikimedia.org: Cleanup logging and curl use in wikitech post-block hooks - https://phabricator.wikimedia.org/T222209#10192214 (10Bugreporter) 05Open→03Declined wikitech.php is gone. [15:24:36] 06cloud-services-team, 10wikitech.wikimedia.org, 07LDAP: Striker: use idm for 2fa validation instead of wikitech - https://phabricator.wikimedia.org/T373461#10192234 (10Bugreporter) Still relevant? [15:24:54] 06cloud-services-team, 10wikitech.wikimedia.org, 07LDAP, 13Patch-For-Review: Replace wikitech as source of two-factor auth protection for developer accounts - https://phabricator.wikimedia.org/T359551#10192250 (10Bugreporter) Still relevant? [15:26:00] 10Striker, 10wikitech.wikimedia.org, 10MediaWiki-extensions-OATHAuth: Wikitech 2FA does not appear to allow recovery with recovery codes - https://phabricator.wikimedia.org/T204682#10192225 (10Bugreporter) Still relevant? [15:26:08] 06cloud-services-team, 10wikitech.wikimedia.org, 07LDAP, 13Patch-For-Review: Horizon: use idm for 2fa validation instead of wikitech - https://phabricator.wikimedia.org/T373462#10192246 (10Bugreporter) Still relevant? [15:27:15] 10wikitech.wikimedia.org, 10MediaWiki-File-management: Uploading files to wikitech fails to display files afterwards ("No file by this name exists, but you can upload it.") - https://phabricator.wikimedia.org/T334487#10192263 (10Bugreporter) 05Open→03Declined No longer reproducible [15:27:47] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q1:rack/setup/install cloudlb2004-dev - https://phabricator.wikimedia.org/T370678#10192269 (10aborrero) please @Jhancock.wm try now. [15:28:16] 06cloud-services-team, 10wikitech.wikimedia.org, 07LDAP, 13Patch-For-Review: Replace wikitech as source of two-factor auth protection for developer accounts - https://phabricator.wikimedia.org/T359551#10192275 (10taavi) Striker still has some code that needs to be cleaned up so {T373461} probably needs to... [15:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:32:35] 10wikitech.wikimedia.org: Setting new password at wikitech does not work - https://phabricator.wikimedia.org/T376140#10192297 (10bd808) I experience this error when trying to login as `BryanDavis` which is my historical Wikitech account name and also my volunteer SUL account name. [15:35:59] 10Cloud-Services, 10wikitech.wikimedia.org: Cleanup $wgObjectCaches for wikitech - https://phabricator.wikimedia.org/T158614#10192352 (10Bugreporter) →14Duplicate dup:03T371374 The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/... [15:37:26] (03merge) 10aborrero: utils: remove them [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/75 [15:39:45] 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops, 13Patch-For-Review: mediawiki-config: consolidate labswiki - https://phabricator.wikimedia.org/T371374#10192355 (10Bugreporter) [15:40:23] 10wikitech.wikimedia.org, 10TimedMediaHandler: Support video on wikitech wiki - https://phabricator.wikimedia.org/T174476#10192382 (10Bugreporter) 05Open→03Resolved a:03Bugreporter [15:40:58] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814) [15:41:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:41:05] T372814: Put cloudcephosd10[39-41] into service - https://phabricator.wikimedia.org/T372814 [15:43:22] 10wikitech.wikimedia.org, 06DBA, 10MediaModeration, 06Trust and Safety Product Team, 07Wikimedia-production-error: wikitech/cloudweb hosts cannot access x1 - https://phabricator.wikimedia.org/T366574#10192407 (10Bugreporter) 05Open→03Declined Presumed, please reopen if it is not [15:44:49] 10wikitech.wikimedia.org, 06SRE, 10WikimediaDebug: With XWikimediaDebug enabled, wikitech.wikimedia.org gets redirected to foundation.wikimedia.org until Wikitech is on k8s - https://phabricator.wikimedia.org/T375795#10192430 (10Bugreporter) 05Open→03Declined Reopen if it is still needed. [16:07:56] 10Tool-translatetagger: Improvise footer design for mobile devices - https://phabricator.wikimedia.org/T376169 (10Gopavasanth) 03NEW [16:14:14] 10Tool-toolwatch: Redesign Tool Details Navigation and Button Text - https://phabricator.wikimedia.org/T376173 (10Gopavasanth) 03NEW [16:17:17] (03update) 10dcaro: registry-admission: bump to 0.0.52-20241001092859-1f1de83c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/539 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:17:26] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [16:17:27] dcaro@cloudcumin1001: Failed to log message to wiki. Somebody should check the error logs. [16:20:48] 10Tool-toolwatch: Improvise pagination experience - https://phabricator.wikimedia.org/T376175 (10Gopavasanth) 03NEW [16:23:01] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [16:23:02] dcaro@cloudcumin1001: Failed to log message to wiki. Somebody should check the error logs. [16:23:24] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [16:23:24] dcaro@cloudcumin1001: Failed to log message to wiki. Somebody should check the error logs. [16:27:12] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: Put cloudcephosd10[39-41] into service - https://phabricator.wikimedia.org/T372814#10192815 (10dcaro) 05Open→03Resolved Done! all three upgraded, setup and joined in the cluster. [16:28:38] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [16:28:40] dcaro@cloudcumin1001: Failed to log message to wiki. Somebody should check the error logs. [16:30:34] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install cloudlb2004-dev - https://phabricator.wikimedia.org/T370678#10192858 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm [16:35:09] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 06serviceops: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10192872 (10Pppery) [16:40:03] 10wikitech.wikimedia.org: Setting new password at wikitech does not work - https://phabricator.wikimedia.org/T376140#10192902 (10AntiCompositeNumber) [16:40:45] (03approved) 10dcaro: registry-admission: bump to 0.0.52-20241001092859-1f1de83c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/539 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:40:48] (03merge) 10dcaro: registry-admission: bump to 0.0.52-20241001092859-1f1de83c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/539 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:42:06] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [16:42:07] dcaro@cloudcumin1001: Failed to log message to wiki. Somebody should check the error logs. [16:42:12] (03update) 10dcaro: volume-admission: bump to 0.0.57-20241001092841-08c6322e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/541 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:44:15] (03update) 10dcaro: builds-buidler: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 [16:44:42] !log fnegri@cloudcumin1001 mwoffliner START - Cookbook wmcs.openstack.quota_increase (T375977) [16:44:43] fnegri@cloudcumin1001: Failed to log message to wiki. Somebody should check the error logs. [16:44:44] T375977: Temporary (1-2 weeks) quota increase for disaster recovery exercise - https://phabricator.wikimedia.org/T375977 [16:44:50] !log fnegri@cloudcumin1001 mwoffliner END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T375977) [16:44:51] fnegri@cloudcumin1001: Failed to log message to wiki. Somebody should check the error logs. [16:46:28] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission [16:46:28] dcaro@cloudcumin1001: Failed to log message to wiki. Somebody should check the error logs. [16:49:00] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [16:49:01] dcaro@cloudcumin1001: Failed to log message to wiki. Somebody should check the error logs. [16:50:24] (03update) 10dcaro: tekton: upgrade to v0.59.3 [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/61 [16:50:28] (03update) 10dcaro: tekton: upgrade to v0.59.3 [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/61 [16:50:32] (03update) 10dcaro: tekton: upgrade to v0.59.3 [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/61 [16:50:36] (03update) 10dcaro: tekton: upgrade to v0.59.3 [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/61 [16:51:02] (03update) 10dcaro: tekton: upgrade to v0.59.3 [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/61 [16:51:31] (03update) 10dcaro: all: upgrade to tekton 0.59.X LTS [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/111 (https://phabricator.wikimedia.org/T374908) [16:51:36] (03update) 10dcaro: all: upgrade to tekton 0.59.X LTS [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/111 (https://phabricator.wikimedia.org/T374908) [16:51:40] (03update) 10dcaro: all: upgrade to tekton 0.59.X LTS [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/111 (https://phabricator.wikimedia.org/T374908) [16:53:10] (03update) 10dcaro: builds-buidler: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 [16:54:28] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission [16:54:29] dcaro@cloudcumin1001: Failed to log message to wiki. Somebody should check the error logs. [17:03:19] (03approved) 10dcaro: volume-admission: bump to 0.0.57-20241001092841-08c6322e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/541 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [17:03:22] (03merge) 10dcaro: volume-admission: bump to 0.0.57-20241001092841-08c6322e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/541 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [17:03:35] (03update) 10dcaro: builds-api: bump to 0.0.171-20241001092826-1929cb0c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/542 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [17:03:38] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [17:03:40] dcaro@cloudcumin1001: Failed to log message to wiki. Somebody should check the error logs. [17:06:53] 10wikitech.wikimedia.org, 10Stashbot: stashbot logging to Wikitech fails because of SUL migration - https://phabricator.wikimedia.org/T376176#10193020 (10bd808) [17:07:45] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [17:07:45] dcaro@cloudcumin1001: Failed to log message to wiki. Somebody should check the error logs. [17:07:53] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [17:07:53] dcaro@cloudcumin1001: Failed to log message to wiki. Somebody should check the error logs. [17:11:09] 10wikitech.wikimedia.org, 10Stashbot: stashbot logging to Wikitech fails because of SUL migration - https://phabricator.wikimedia.org/T376176#10193042 (10bd808) [17:11:15] 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops, 13Patch-For-Review: ☂ Migrate Wikitech to Kubernetes - https://phabricator.wikimedia.org/T292707#10193043 (10bd808) [17:12:39] 06cloud-services-team, 10Toolforge, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Unplanned: [registry-admission-webhook] Investigate why helm did not override the selector on the service on deployment - https://phabricator.wikimedia.org/T320665#10193040 (10dcaro) 05Open→03Declined I thin... [17:13:24] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [17:13:26] dcaro@cloudcumin1001: Failed to log message to wiki. Somebody should check the error logs. [17:14:47] (03approved) 10dcaro: builds-api: bump to 0.0.171-20241001092826-1929cb0c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/542 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [17:14:50] (03merge) 10dcaro: builds-api: bump to 0.0.171-20241001092826-1929cb0c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/542 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [17:18:05] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 13Patch-For-Review: Improve WMCS NodeDown alerts - https://phabricator.wikimedia.org/T375479#10193083 (10taavi) >>! In T375479#10172290, @dcaro wrote: > The reason was mostly UX, having two alerts with the same name meaning different things is really co... [17:20:50] 10wikitech.wikimedia.org: OAuth consumers registered locally at Wikitech are no longer configured to be used - https://phabricator.wikimedia.org/T376188 (10bd808) 03NEW [17:22:49] 10wikitech.wikimedia.org, 10Stashbot: stashbot logging to Wikitech fails because of SUL migration - https://phabricator.wikimedia.org/T376176#10193133 (10bd808) OAuth issue split out to {T376188} [17:23:38] (03update) 10dcaro: builds-buidler: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 [17:25:33] 10Cloud-VPS (Quota-requests): Temporary (1-2 weeks) quota increase for disaster recovery exercise - https://phabricator.wikimedia.org/T375977#10193152 (10fnegri) I misread the description and I used the cookbook to increase the Cinder quota. Now I've reverted that and I increased the Trove quota instead as reque... [17:26:49] 10Cloud-VPS (Quota-requests): Temporary (1-2 weeks) quota increase for disaster recovery exercise - https://phabricator.wikimedia.org/T375977#10193154 (10fnegri) 05In progress→03Stalled [17:33:10] (03open) 10fnegri: Increase memory quota for "sqid" [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/543 (https://phabricator.wikimedia.org/T375070) [17:33:49] (03update) 10fnegri: Increase memory quota for "sqid" [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/543 (https://phabricator.wikimedia.org/T375070) [17:37:15] 10wikitech.wikimedia.org, 10Stashbot: stashbot logging to Wikitech fails because of SUL migration - https://phabricator.wikimedia.org/T376176#10193205 (10bd808) I created a new owner-only consumer grant at https://meta.wikimedia.org/wiki/Special:OAuthListConsumers/view/9d357563021064fb204d827a8c8479bb and reco... [17:37:40] 10VPS-project-Wikistats: Add shnwikinews to wikistats - https://phabricator.wikimedia.org/T375437#10193202 (10Dzahn) 05Open→03Resolved ` MariaDB [wikistats]> insert into wikinews (prefix, lang, loclang, method) select prefix,lang,loclang,method from wikipedias where prefix="shn"; Query OK, 1 row affected... [17:38:09] 10VPS-project-Wikistats: Add shnwikinews to wikistats - https://phabricator.wikimedia.org/T375437#10193206 (10Dzahn) https://wikistats.wmcloud.org/display.php?t=wn [17:44:21] 10Toolforge (Quota-requests), 13Patch-For-Review: Request increased quota for sqid Toolforge tool - https://phabricator.wikimedia.org/T375070#10193213 (10fnegri) The merge request above sets `memory: 9`, which will set `limits.memory: "9Gi"`. `requests.memory` is [[ https://gitlab.wikimedia.org/repos/cloud/too... [17:50:46] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install cloudlb2004-dev - https://phabricator.wikimedia.org/T370678#10193222 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm executed... [17:55:47] 10wikitech.wikimedia.org: OAuth consumers registered locally at Wikitech are no longer configured to be used - https://phabricator.wikimedia.org/T376188#10193254 (10bd808) [18:00:39] 10wikitech.wikimedia.org, 13Patch-For-Review: Setting new password at wikitech does not work - https://phabricator.wikimedia.org/T376140#10193268 (10Ladsgroup) Can you try again and let me know? [18:08:55] 10wikitech.wikimedia.org, 13Patch-For-Review: Setting new password at wikitech does not work - https://phabricator.wikimedia.org/T376140#10193289 (10Ameisenigel) >>! In T376140#10193268, @Ladsgroup hat geschrieben: > Can you try again and let me know? Thanks, I have now been able to reset my password. [18:10:29] 10wikitech.wikimedia.org: OAuth consumers registered locally at Wikitech are no longer configured to be used - https://phabricator.wikimedia.org/T376188#10193295 (10bd808) [18:10:32] 10wikitech.wikimedia.org, 10Stashbot: stashbot logging to Wikitech fails because of SUL migration - https://phabricator.wikimedia.org/T376176#10193296 (10bd808) [18:11:51] 10Striker, 10wikitech.wikimedia.org: toolsadmin.wikimedia.org login fails because of missing OAuth grant at Wikitech - https://phabricator.wikimedia.org/T376190#10193293 (10bd808) Created new grant at https://meta.wikimedia.org/wiki/Special:OAuthListConsumers/view/b79c42dc7b90712aaeb8541efb587c88. Put secrets... [18:11:53] 10wikitech.wikimedia.org, 10Stashbot: Stashbot needs new owner-only OAuth grant following Wikitech config change - https://phabricator.wikimedia.org/T376176#10193297 (10bd808) [18:13:14] 10wikitech.wikimedia.org, 10Stashbot: Stashbot needs new owner-only OAuth grant following Wikitech config change - https://phabricator.wikimedia.org/T376176#10193302 (10bd808) 05Open→03Resolved a:03bd808 [18:13:49] 10wikitech.wikimedia.org, 13Patch-For-Review: Setting new password at wikitech does not work - https://phabricator.wikimedia.org/T376140#10193298 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup Awesome! [18:25:36] 10Striker, 10wikitech.wikimedia.org: toolsadmin.wikimedia.org login fails because of missing OAuth grant at Wikitech - https://phabricator.wikimedia.org/T376190#10193335 (10bd808) `'The authorization headers in your request are for a user that does not exist here'` -- The StrikerBot account on Wikitech needs t... [18:31:29] 10Cloud-VPS (Quota-requests): Temporary (1-2 weeks) quota increase for disaster recovery exercise - https://phabricator.wikimedia.org/T375977#10193348 (10Audiodude) Thanks so much! [18:38:05] 10Striker, 10wikitech.wikimedia.org: toolsadmin.wikimedia.org login fails because of missing OAuth grant at Wikitech - https://phabricator.wikimedia.org/T376190#10193372 (10bd808) >>! In T376190#10193335, @bd808 wrote: > `'The authorization headers in your request are for a user that does not exist here'` -- T... [18:43:27] 10Striker, 10wikitech.wikimedia.org: toolsadmin.wikimedia.org login fails because of missing OAuth grant at Wikitech - https://phabricator.wikimedia.org/T376190#10193414 (10bd808) `mwclient.errors.APIError: ('permissiondenied', "You don't have permission to check OATH status.", None)` happening now because I f... [18:52:44] 10Striker, 10wikitech.wikimedia.org: toolsadmin.wikimedia.org login fails because of missing OAuth grant at Wikitech - https://phabricator.wikimedia.org/T376190#10193432 (10bd808) >>! In T376190#10193414, @bd808 wrote: > `mwclient.errors.APIError: ('permissiondenied', "You don't have permission to check OATH s... [18:56:33] 10Striker, 10wikitech.wikimedia.org: toolsadmin.wikimedia.org login fails because of missing OAuth grant at Wikitech - https://phabricator.wikimedia.org/T376190#10193438 (10bd808) >>! In T376190#10193432, @bd808 wrote: > I hacked the grant in the database, but the API is still claiming that StrikerBot does not... [19:33:19] 10wikitech.wikimedia.org, 10Stashbot: Stashbot needs new owner-only OAuth grant following Wikitech config change - https://phabricator.wikimedia.org/T376176#10193547 (10jijiki) [19:33:28] 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops, 13Patch-For-Review: ☂ Migrate Wikitech to Kubernetes - https://phabricator.wikimedia.org/T292707#10193548 (10jijiki) [19:33:58] 10wikitech.wikimedia.org: OAuth consumers registered locally at Wikitech are no longer configured to be used - https://phabricator.wikimedia.org/T376188#10193549 (10jijiki) [19:34:07] 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops, 13Patch-For-Review: ☂ Migrate Wikitech to Kubernetes - https://phabricator.wikimedia.org/T292707#10193550 (10jijiki) [19:43:59] 10Striker, 10wikitech.wikimedia.org: toolsadmin.wikimedia.org login fails because of missing OAuth grant at Wikitech - https://phabricator.wikimedia.org/T376190#10193591 (10bd808) >>! In T376190#10193438, @bd808 wrote: > Still getting `mwclient.errors.APIError: ('permissiondenied', "You don't have permission t... [19:50:14] (03PS1) 10BryanDavis: auth: Disable OATH (2FA) checking [labs/striker] - 10https://gerrit.wikimedia.org/r/1077091 (https://phabricator.wikimedia.org/T376190) [19:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:52:27] (03CR) 10CI reject: [V:04-1] auth: Disable OATH (2FA) checking [labs/striker] - 10https://gerrit.wikimedia.org/r/1077091 (https://phabricator.wikimedia.org/T376190) (owner: 10BryanDavis) [19:59:22] 10wikitech.wikimedia.org: OAuth consumers registered locally at Wikitech are no longer configured to be used - https://phabricator.wikimedia.org/T376188#10193645 (10bd808) The general fix is to: * Make sure you have a SUL account matching the username of the Wikitech account to be fixed * Regain the ability to l... [20:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:20:11] 06cloud-services-team, 06Data-Platform-SRE, 07Ceph: ceph: introduce puppet logic to purge stale keyfiles - https://phabricator.wikimedia.org/T328010#10193733 (10BTullis) [20:23:42] 10wikitech.wikimedia.org, 06SRE: Setup LDAP account - https://phabricator.wikimedia.org/T376209 (10WRai-WMF) 03NEW [20:28:28] 10wikitech.wikimedia.org, 06SRE: Setup LDAP account - https://phabricator.wikimedia.org/T376209#10193753 (10RhinosF1) 05Open→03Invalid Hi, You can create an LDAP / Developer account at https://idm.wikimedia.org/signup/ If you need further help, you may want to jump onto the #wikimedia-tech channel on... [20:39:28] (03PS2) 10BryanDavis: auth: Disable OATH (2FA) checking [labs/striker] - 10https://gerrit.wikimedia.org/r/1077091 (https://phabricator.wikimedia.org/T376190) [20:44:44] (03CR) 10BryanDavis: [C:03+2] "Ugly hack, but I think getting Striker working again is more important than pretty fixes at the moment." [labs/striker] - 10https://gerrit.wikimedia.org/r/1077091 (https://phabricator.wikimedia.org/T376190) (owner: 10BryanDavis) [20:45:59] (03Merged) 10jenkins-bot: auth: Disable OATH (2FA) checking [labs/striker] - 10https://gerrit.wikimedia.org/r/1077091 (https://phabricator.wikimedia.org/T376190) (owner: 10BryanDavis) [20:48:30] 10Cloud-VPS (Project-requests): Request creation of VPS project - https://phabricator.wikimedia.org/T376211 (10SDunlap) 03NEW [20:54:39] 10Cloud-VPS (Project-requests): Request creation of catalyst-dev VPS project - https://phabricator.wikimedia.org/T376211#10193841 (10Pppery) [21:05:53] 10Striker, 10wikitech.wikimedia.org, 13Patch-For-Review: toolsadmin.wikimedia.org login fails because of missing OAuth grant at Wikitech - https://phabricator.wikimedia.org/T376190#10193863 (10bd808) 05Open→03Resolved a:03bd808 Logins work again. The changes: * Attached Wikitech StrikerBot account... [21:25:57] 10Tool-gitlab-account-approval, 10wikitech.wikimedia.org: Gitlabaccountapprovalbot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376214 (10bd808) 03NEW [21:28:29] 10Tool-gitlab-account-approval, 10wikitech.wikimedia.org: Gitlabaccountapprovalbot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376214#10193936 (10bd808) 05Open→03In progress p:05Triage→03Medium a:03bd808 [21:31:58] 10Tool-gitlab-account-approval, 10wikitech.wikimedia.org: Gitlabaccountapprovalbot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376214#10193954 (10bd808) https://meta.wikimedia.org/wiki/Special:OAuthListConsumers/view/b00531a13c75f82bc3375f7f998c782b [21:35:43] 10Tool-gitlab-account-approval, 10wikitech.wikimedia.org: Gitlabaccountapprovalbot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376214#10193973 (10bd808) [21:35:59] 10Tool-gitlab-account-approval, 10wikitech.wikimedia.org: Gitlabaccountapprovalbot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376214#10193974 (10bd808) 05In progress→03Resolved [21:41:57] 10Tool-phab-ban, 10wikitech.wikimedia.org: Phabbanbot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376216 (10bd808) 03NEW [21:42:50] 10Tool-phab-ban, 10wikitech.wikimedia.org: Phabbanbot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376216#10193989 (10bd808) 05Open→03In progress p:05Triage→03High a:03bd808 [21:45:33] 10Tool-phab-ban, 10wikitech.wikimedia.org: Phabbanbot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376216#10193994 (10bd808) https://meta.wikimedia.org/wiki/Special:OAuthListConsumers/view/e6b183d3ecae8e000b66bd219fe52827 [21:50:45] 10Tool-phab-ban, 10wikitech.wikimedia.org: Phabbanbot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376216#10193997 (10bd808) It works! https://wikitech.wikimedia.org/w/index.php?title=Tool:Phab-ban/Log&diff=prev&oldid=2231842 [21:50:52] 10Tool-phab-ban, 10wikitech.wikimedia.org: Phabbanbot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376216#10193998 (10bd808) [21:51:00] 10Tool-phab-ban, 10wikitech.wikimedia.org: Phabbanbot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376216#10193999 (10bd808) 05In progress→03Resolved [21:56:17] 10wikitech.wikimedia.org: ScheduleDeploymentBot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376217 (10bd808) 03NEW [21:58:20] 10wikitech.wikimedia.org: ScheduleDeploymentBot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376217#10194012 (10bd808) 05Open→03In progress p:05Triage→03High a:03bd808 [21:59:06] 10wikitech.wikimedia.org: ScheduleDeploymentBot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376217#10194018 (10bd808) https://meta.wikimedia.org/wiki/Special:OAuthListConsumers/view/304f5c448008b64f0e9d936a6eb44d71 [22:14:33] 10wikitech.wikimedia.org, 13Patch-For-Review: ScheduleDeploymentBot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376217#10194088 (10bd808) https://wikitech.wikimedia.org/w/index.php?diff=2231845 [22:18:11] 10wikitech.wikimedia.org, 13Patch-For-Review: ScheduleDeploymentBot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376217#10194089 (10bd808) 05In progress→03Resolved [22:24:16] 06cloud-services-team, 10wikitech.wikimedia.org: Labslogbot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376220 (10bd808) 03NEW [22:25:27] 10wikitech.wikimedia.org: DeploymentCalendarTool needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376221 (10bd808) 03NEW [22:26:09] 10wikitech.wikimedia.org: MABot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376222 (10bd808) 03NEW [22:27:04] 10wikitech.wikimedia.org: Toolforge Image Bot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376223 (10bd808) 03NEW [22:27:51] 10wikitech.wikimedia.org: Wikitech double redirect bot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376224 (10bd808) 03NEW [22:39:12] 10wikitech.wikimedia.org: OAuth consumers registered locally at Wikitech are no longer configured to be used - https://phabricator.wikimedia.org/T376188#10194194 (10bd808) I made subtasks for the accounts that seemed mostly likely to still be in use. I also have tried to subscribe all of the humans behind these... [22:47:30] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate mwv-builder-03.mediawiki-vagrant.eqiad.wmflabs is about to expire in 23d 23h 58m 34s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [23:04:14] 06cloud-services-team, 10wikitech.wikimedia.org, 10Wikimedia-Extension-setup, 07Documentation, 07I18n: Install Translate extension on wikitech - https://phabricator.wikimedia.org/T100313#10194214 (10bd808) >>! In T100313#10191749, @Bugreporter wrote: > This is suggested again in https://wikitech.wikimedi... [23:04:18] 06cloud-services-team, 10wikitech.wikimedia.org, 10Wikimedia-Extension-setup, 07Documentation, 07I18n: Install Translate extension on wikitech - https://phabricator.wikimedia.org/T100313#10194217 (10bd808) Another point to be considered is how adding E:Translate may affect maintenance of https://wikitech... [23:13:39] 10wikitech.wikimedia.org, 10Wikimedia-Site-requests: fold contentadmin group to sysop in Wikitech - https://phabricator.wikimedia.org/T375950#10194247 (10bd808) 05Stalled→03Open [23:14:09] 10wikitech.wikimedia.org, 06Data Products, 06Data-Engineering, 06DBA, 07Schema-change: Please drop globalblocks table from labswiki - https://phabricator.wikimedia.org/T375783#10194248 (10bd808) 05Stalled→03Open [23:14:30] 10wikitech.wikimedia.org: Enable ConfirmEdit on Wikitech - https://phabricator.wikimedia.org/T194974#10194227 (10bd808) 05Open→03Resolved Done as a side effect of {T371374} [23:17:22] 06cloud-services-team, 10wikitech.wikimedia.org, 07LDAP: Striker: use idm for 2fa validation instead of wikitech - https://phabricator.wikimedia.org/T373461#10194275 (10bd808) 05Open→03Declined See {T359554} as a replacement. [23:17:24] 06cloud-services-team, 10wikitech.wikimedia.org, 07LDAP, 13Patch-For-Review: Horizon: use idm for 2fa validation instead of wikitech - https://phabricator.wikimedia.org/T373462#10194259 (10bd808) 05Open→03Declined >>! In T373462#10103091, @Andrew wrote: > We are probably skipping ahead to idp auth.... [23:20:44] 06cloud-services-team, 10wikitech.wikimedia.org, 07LDAP, 13Patch-For-Review: Replace wikitech as source of two-factor auth protection for developer accounts - https://phabricator.wikimedia.org/T359551#10194298 (10bd808) →14Duplicate dup:03T359552 [23:30:56] 10wikitech.wikimedia.org, 07Epic: [EPIC] Make wikitech more friendly for the multiple audiences it supports - https://phabricator.wikimedia.org/T123425#10194331 (10bd808) 05Open→03Resolved a:03bd808 This #epic has served its purpose. We have come such a long way in the last 8 years that it is difficu... [23:44:30] 10wikitech.wikimedia.org: Audit MediaWiki namespace message overrides for legacy Developer account related information - https://phabricator.wikimedia.org/T376231 (10bd808) 03NEW [23:44:56] 10wikitech.wikimedia.org: Audit MediaWiki namespace message overrides for legacy Developer account related information - https://phabricator.wikimedia.org/T376231#10194374 (10bd808) [23:45:07] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10194375 (10bd808) [23:58:29] 06cloud-services-team, 10wikitech.wikimedia.org, 10Wikimedia-Extension-setup, 07Documentation, 07I18n: Install Translate extension on wikitech - https://phabricator.wikimedia.org/T100313#10194394 (10Bugreporter) >>! In T100313#10194217, @bd808 wrote: > Another point to be considered is how adding E:Trans...