[00:00:06] FIRING: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:05:06] RESOLVED: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:09:02] 10tool-wdlocator: Refresh link for current area - https://phabricator.wikimedia.org/T364287 (10Samwilson) 03NEW [03:42:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:47:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:52:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:57:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:35:47] 14Grid-Engine-to-K8s-Migration: Migrate hazard-bot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319785#9773459 (10Hazard-SJ) 05Declined→03Resolved [04:36:50] 14Grid-Engine-to-K8s-Migration: Migrate hazard-bot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319785#9773460 (10Hazard-SJ) Changed the status since this was in fact mostly done. [05:07:12] 10ToolforgeBundle, 10AhoCorasick, 10at-ease, 10base_convert, and 25 others: Make new releases of all Wikimedia-authored PHP libraries, and bump their usages (mid-2021) - https://phabricator.wikimedia.org/T287972#9773468 (10Jdforrester-WMF) 05Resolved→03Open >>! In T287972#9771259, @Zabe wrote: > Mid-20... [06:14:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:24:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:28:04] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Remove obsolete dummy cert [labs/private] - 10https://gerrit.wikimedia.org/r/1026439 (https://phabricator.wikimedia.org/T360439) (owner: 10Muehlenhoff) [06:51:34] (03PS1) 10Muehlenhoff: Add dummy keytab for install7001 [labs/private] - 10https://gerrit.wikimedia.org/r/1028236 (https://phabricator.wikimedia.org/T364016) [06:54:17] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Add dummy keytab for install7001 [labs/private] - 10https://gerrit.wikimedia.org/r/1028236 (https://phabricator.wikimedia.org/T364016) (owner: 10Muehlenhoff) [07:13:24] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api [07:13:35] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api [07:23:52] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api [07:24:06] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api [07:26:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [07:26:44] 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: [builds-api, envvars-api] add oapi-codegen installation to makefile - https://phabricator.wikimedia.org/T362290#9773583 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/274 envvars-api:... [07:31:28] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [07:42:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:47:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:52:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:57:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:10:05] 06cloud-services-team, 10Toolforge: [infra,k8s] Move to kubernetes PAVs and drop kyverno - https://phabricator.wikimedia.org/T364293 (10dcaro) 03NEW [08:12:55] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [08:13:06] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [08:18:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [08:18:54] 10Toolforge, 03Wikimedia-Hackathon-2024: toolforge jobs load flushes out all jobs - https://phabricator.wikimedia.org/T364204#9773726 (10dcaro) p:05Triage→03Medium [08:19:21] 06cloud-services-team, 10Toolforge: [infra,k8s] Move to kubernetes PAVs and drop kyverno - https://phabricator.wikimedia.org/T364293#9773729 (10dcaro) p:05Triage→03High [08:20:03] 10Toolforge, 03Wikimedia-Hackathon-2024: Add more granular schedule macro's to toolforge jobs - https://phabricator.wikimedia.org/T364210#9773731 (10dcaro) p:05Triage→03Medium [08:20:50] 06cloud-services-team, 10Observability-Alerting, 10SRE Observability (FY2023/2024-Q4): Karma UI shows duplicate alerts - https://phabricator.wikimedia.org/T353457#9773734 (10fgiunchedi) [08:23:28] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [08:24:43] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [08:24:54] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [08:38:22] 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: Golang and Procfile buildpacks not working together as expected - https://phabricator.wikimedia.org/T363417#9773760 (10dcaro) [08:38:54] 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: Golang and Procfile buildpacks not working together as expected - https://phabricator.wikimedia.org/T363417#9773758 (10dcaro) 05Open→03In progress p:05Triage→03Medium [08:39:40] 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: [builds-builder] golang based images get infinite nested loops for procfile entries - https://phabricator.wikimedia.org/T363417#9773764 (10dcaro) [08:45:32] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Remove obsolete stub certs [labs/private] - 10https://gerrit.wikimedia.org/r/1026806 (owner: 10Muehlenhoff) [08:47:56] 06cloud-services-team, 10Toolforge: toolforge: create a PSP migration plan - https://phabricator.wikimedia.org/T364297 (10aborrero) 03NEW [09:12:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:16:48] 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29 - https://phabricator.wikimedia.org/T362868#9773821 (10dcaro) [09:17:18] 10Toolforge: [k8s,infra] Upgrade Toolforge to Uwubernetes (1.30) - https://phabricator.wikimedia.org/T362869#9773822 (10dcaro) [09:17:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:18:13] 06cloud-services-team, 10Toolforge: [infra,k8s] Move to kubernetes PAVs and drop kyverno - https://phabricator.wikimedia.org/T364293#9773823 (10dcaro) [09:22:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:27:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:33:46] 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: [builds-builder] golang based images get infinite nested loops for procfile entries - https://phabricator.wikimedia.org/T363417#9773852 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/45... [09:35:07] 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: [builds-builder] golang based images get infinite nested loops for procfile entries - https://phabricator.wikimedia.org/T363417#9773854 (10CodeReviewBot) project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/t... [09:36:35] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [09:36:51] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [09:59:42] 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: [builds-builder] golang based images get infinite nested loops for procfile entries - https://phabricator.wikimedia.org/T363417#9773898 (10CodeReviewBot) dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/46... [10:31:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [10:36:28] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [10:42:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:52:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:27:04] 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: [builds-builder] golang based images get infinite nested loops for procfile entries - https://phabricator.wikimedia.org/T363417#9774062 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/46... [11:29:26] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [11:29:42] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [12:04:48] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [12:05:11] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [12:22:10] 10Toolforge (Toolforge iteration 09): [toolforge] Investigate authentication - https://phabricator.wikimedia.org/T363983#9774149 (10dcaro) [12:22:21] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1028463 (owner: 10L10n-bot) [12:29:58] 06cloud-services-team, 10Toolforge: toolforge: create a PSP migration plan - https://phabricator.wikimedia.org/T364297#9774204 (10aborrero) the plan could be this: * finish {T362872} * finish {T362050} * finish {T364113} * deploy kyverno (with policies in audit mode) -- https://gitlab.wikimedia.org/repos/clou... [12:34:43] 10Toolforge (Toolforge iteration 09): [toolforge] Investigate authentication - https://phabricator.wikimedia.org/T363983#9774220 (10dcaro) [12:43:41] 10Toolforge (Toolforge iteration 09): [toolforge] Investigate authentication - https://phabricator.wikimedia.org/T363983#9774235 (10aborrero) linking {T277778} for reference, in case is relevant [12:50:31] 06cloud-services-team, 10Toolforge: toolforge: create a PSP migration plan - https://phabricator.wikimedia.org/T364297#9774249 (10aborrero) p:05Triage→03Medium [12:52:07] 06cloud-services-team, 10Toolforge: toolforge: create a PSP migration plan - https://phabricator.wikimedia.org/T364297#9774246 (10aborrero) 05Open→03In progress [12:58:02] 06cloud-services-team, 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: toolforge: review pod templates for PSP replacement - https://phabricator.wikimedia.org/T362050#9774267 (10aborrero) [13:05:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [13:14:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:15:28] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [13:17:59] 10Tools: templatecount tool inaccessible due to 502 Bad Gateway - https://phabricator.wikimedia.org/T172549#9774339 (10Jeff_G) a:05Jeff_G→03None [13:24:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:33:23] 06cloud-services-team, 13Patch-For-Review: Migrate cloudweb, cloudbackup, cloudmetrics physical servers off buster - https://phabricator.wikimedia.org/T332400#9774378 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudbackup1004.eqiad.wmnet with OS bookworm [13:38:27] 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: [builds-builder] golang based images get infinite nested loops for procfile entries - https://phabricator.wikimedia.org/T363417#9774387 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/2... [13:39:08] 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: [builds-builder] golang based images get infinite nested loops for procfile entries - https://phabricator.wikimedia.org/T363417#9774388 (10dcaro) @bd808 This should have been fixed, can you try now? [14:07:01] 06cloud-services-team, 10Toolforge: toolforge: create a PSP migration plan - https://phabricator.wikimedia.org/T364297#9774452 (10aborrero) Updated {T362050} to make sure our pod templates are updated accordingly. Another point to consider: how to back-fill per-tool kyverno policies for existing tools. [14:11:39] 06cloud-services-team, 10Toolforge: toolforge: introduce some logic to backfill maintain-kubeuser resources (like per-tool kyverno policies) - https://phabricator.wikimedia.org/T364312 (10aborrero) 03NEW [14:13:33] 06cloud-services-team, 10Toolforge: toolforge: introduce some logic to backfill maintain-kubeuser resources (like per-tool kyverno policies) - https://phabricator.wikimedia.org/T364312#9774469 (10aborrero) p:05Triage→03Medium [14:16:42] 06cloud-services-team, 13Patch-For-Review: Migrate cloudweb, cloudbackup, cloudmetrics physical servers off buster - https://phabricator.wikimedia.org/T332400#9774476 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host cloudbackup1004.eqiad.wmnet with OS bookworm co... [14:17:52] 06cloud-services-team, 06Infrastructure-Foundations, 10Puppet-Infrastructure, 13Patch-For-Review: puppet servers run out of inodes in puppet code volume - https://phabricator.wikimedia.org/T364047#9774479 (10jhathaway) p:05Triage→03Medium [14:18:37] 06cloud-services-team, 10Toolforge: toolforge: introduce some logic to backfill maintain-kubeuser resources (like per-tool kyverno policies) - https://phabricator.wikimedia.org/T364312#9774480 (10aborrero) [14:21:17] PROBLEM - Host cloudbackup1004 is DOWN: PING CRITICAL - Packet loss = 100% [14:21:43] 10superset.wmcloud.org: Upgrade to 4.0.0 - https://phabricator.wikimedia.org/T364022#9774483 (10rook) Getting the error described in https://github.com/apache/superset/issues/28145 [14:23:09] RECOVERY - Host cloudbackup1004 is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms [14:43:11] 10Tools, 06translatewiki.net, 10Language-Team (Language-2024-April-June), 03Localization Infrastructure FY2023-24, and 2 others: Make Wikidata Image Positions tool translatable on translatewiki.net - https://phabricator.wikimedia.org/T363626#9774565 (10LucasWerkmeister) Hm, I just realized that having the... [14:49:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:54:28] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [15:11:50] 10Tools, 06translatewiki.net, 10Language-Team (Language-2024-April-June), 03Localization Infrastructure FY2023-24, and 2 others: Make Wikidata Image Positions tool translatable on translatewiki.net - https://phabricator.wikimedia.org/T363626#9774695 (10CodeReviewBot) lucaswerkmeister merged https://gitlab.... [15:20:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [15:40:28] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [15:53:34] 06cloud-services-team, 13Patch-For-Review: Migrate cloudweb, cloudbackup, cloudmetrics physical servers off buster - https://phabricator.wikimedia.org/T332400#9774865 (10Andrew) 05Open→03Resolved [16:08:22] (03PS1) 10Andrea Denisse: Revert "ssl: Remove unnecessary dummy key from thanos-query hosts" [labs/private] - 10https://gerrit.wikimedia.org/r/1028567 [16:11:44] (03CR) 10Andrea Denisse: [V:03+2 C:03+2] Revert "ssl: Remove unnecessary dummy key from thanos-query hosts" [labs/private] - 10https://gerrit.wikimedia.org/r/1028567 (owner: 10Andrea Denisse) [16:12:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [16:25:36] 06cloud-services-team, 10Toolforge: Remove old symlinks to trunk/rewrite/compat/pywikipedia in /shared - https://phabricator.wikimedia.org/T192733#9774988 (10Xqt) Not related to Pywikibot: https://codesearch.wmcloud.org/pywikibot/?q=trunk%7Crewrite%7Ccompat%7Cpywikipedia&files=&excludeFiles=&repos= [16:42:28] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [16:46:12] 10superset.wmcloud.org: Upgrade to 4.0.0 - https://phabricator.wikimedia.org/T364022#9775048 (10rook) This appears stuck until the github issue is resolved. Upgrading the install in place does not seem to remedy the issue either. The issue appears in the 3.1.2 version of superset. Installing fresh does not seem... [16:46:22] 10superset.wmcloud.org: Upgrade to 4.0.0 - https://phabricator.wikimedia.org/T364022#9775049 (10rook) 05Open→03Stalled [16:54:49] 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T364188#9775053 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/409 [16:54:59] vivian-rook opened https://github.com/toolforge/paws/pull/409 [16:57:18] vivian-rook opened https://github.com/toolforge/paws/pull/410 [17:08:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [17:12:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:15:16] 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T364188#9775120 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/409 [17:15:32] vivian-rook closed https://github.com/toolforge/paws/pull/409 [17:16:44] 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T364188#9775142 (10rook) 05Open→03Resolved a:03rook [17:17:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:22:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:27:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:28:04] 10Tools, 06translatewiki.net, 10Language-Team (Language-2024-April-June), 03Localization Infrastructure FY2023-24, and 2 others: Make Wikidata Image Positions tool translatable on translatewiki.net - https://phabricator.wikimedia.org/T363626#9775381 (10LucasWerkmeister) Alright, GitLab CI wasn’t as bad as... [18:34:23] 10PAWS: jupyterlab to 4.2.0 - https://phabricator.wikimedia.org/T364327#9775390 (10rook) 05Open→03Stalled [18:34:24] 10PAWS: jupyterlab to 4.2.0 - https://phabricator.wikimedia.org/T364327#9775392 (10rook) Appears to be being downgraded by other packages to 4.1.8. Will wait and see how it resolves after 4.2.0 has been out for a little longer. [19:38:28] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [20:03:22] 06cloud-services-team, 10Toolforge: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#9775711 (10Andrew) I'm striking out the 'keystone projects in ldap' option because keystone doesn't really support that one. [20:07:29] 06cloud-services-team, 10Toolforge: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#9775716 (10Andrew) [20:07:42] 10Cloud-VPS (Debian Buster Deprecation), 06collaboration-services, 13Patch-For-Review: replace buster machines in devtools project - https://phabricator.wikimedia.org/T360964#9775717 (10Dzahn) 05Stalled→03In progress [20:10:06] 10Cloud-VPS (Debian Buster Deprecation), 06collaboration-services, 13Patch-For-Review: replace buster machines in devtools project - https://phabricator.wikimedia.org/T360964#9775747 (10Dzahn) Most issues with deploy-1006 are now fixed. And the remaining one is related to T360470 and the same on both old and... [20:10:30] 06cloud-services-team, 10Toolforge: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#9775748 (10Andrew) My favorite option is 'Automatic creation of per-tool keystone project'. Since that's a simple extension of 'On-demand creation of per-tool... [20:24:04] 06cloud-services-team, 10VPS-project-devtools, 06collaboration-services, 10Puppet (Puppet 7.0): Update devtools project puppetmaster - https://phabricator.wikimedia.org/T360470#9775810 (10Dzahn) Jelto and taavi: Actually we saw a little while ago the puppetmaster was not in sync but hesitated to just g... [20:37:25] 10Cloud-VPS (Debian Buster Deprecation), 06collaboration-services, 13Patch-For-Review: replace buster machines in devtools project - https://phabricator.wikimedia.org/T360964#9775888 (10Dzahn) [20:39:39] 10Cloud-VPS (Debian Buster Deprecation), 06collaboration-services, 13Patch-For-Review: replace buster machines in devtools project - https://phabricator.wikimedia.org/T360964#9775889 (10Dzahn) 05In progress→03Resolved no more buster machines in devtools [21:12:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:17:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks