[02:28:20] 10Toolforge, 10Wikimedia-Site-requests: Audit wmf-config and add toolforge.org as needed where tools.wmflabs.org is used - https://phabricator.wikimedia.org/T285364#9762701 (10Pppery) Anything left to do here? [02:39:25] 10Tools, 10MediaViewer, 10Thumbor, 13Patch-For-Review: Explore moving the Panoviewer gadget/Toolforge tool into production - https://phabricator.wikimedia.org/T138933#9762717 (10tstarling) I made subtasks for what I see as the three main components of spherical panorama support: * {T363786} * {T363885} *... [02:42:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:47:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:52:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:57:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:12:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:22:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:53:20] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Remove obsolete stub cert [labs/private] - 10https://gerrit.wikimedia.org/r/1025777 (https://phabricator.wikimedia.org/T360439) (owner: 10Muehlenhoff) [07:08:55] 10Toolforge (Toolforge iteration 09), 07Upstream: [builds-builder,jobs-api,upstream] Calling nontrivial Procfile commands with arguments results in confusing error (“no such file or directory”) - https://phabricator.wikimedia.org/T356016#9762912 (10dcaro) [07:13:16] 10Toolforge: Golang and Procfile buildpacks not working together as expected - https://phabricator.wikimedia.org/T363417#9762915 (10dcaro) The procfile endpoints don't allow any new arguments if they have any already, this is an upstream issue on the procfile buildpack ({T356016}). There's a note and a workaroun... [07:24:11] 10Toolforge: Golang and Procfile buildpacks not working together as expected - https://phabricator.wikimedia.org/T363417#9762919 (10dcaro) For the bridgebot entry itself, the issue is that the golang buildpack generates on the fly the bridgebot procfile entry already: ` dcaro@urcuchillay$ podman run --entrypoin... [07:32:25] 10Toolforge, 13Patch-For-Review: Golang and Procfile buildpacks not working together as expected - https://phabricator.wikimedia.org/T363417#9762924 (10CodeReviewBot) dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/45 procfile: avoid recursive calls for golang b... [07:32:29] 10Toolforge, 13Patch-For-Review: Golang and Procfile buildpacks not working together as expected - https://phabricator.wikimedia.org/T363417#9762926 (10dcaro) This should fix it on our side until we get the upstream side done: https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/4... [07:37:49] (03PS1) 10Muehlenhoff: Remove obsolete dummy cert [labs/private] - 10https://gerrit.wikimedia.org/r/1026439 (https://phabricator.wikimedia.org/T360439) [08:18:17] 10Tools, 06translatewiki.net, 10Language-Team (Language-2024-April-June), 03Localization Infrastructure FY2023-24, 03Wikimedia-Hackathon-2024: Make Wikidata Image Positions tool translatable on translatewiki.net - https://phabricator.wikimedia.org/T363626#9763029 (10abi_) p:05Triage→03Medium [08:19:03] 10Tools, 06translatewiki.net, 10Language-Team (Language-2024-April-June), 03Localization Infrastructure FY2023-24, 03Wikimedia-Hackathon-2024: Make Wikidata Image Positions tool translatable on translatewiki.net - https://phabricator.wikimedia.org/T363626#9763028 (10abi_) [08:25:28] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [09:21:47] 14Grid-Engine-to-K8s-Migration: Migrate parliamentdiagram from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319955#9763236 (10fnegri) @Gouvernathor can you give me an example of a file that the task is failing to remove? I'll have a look at the permissions. [09:53:58] 10Toolforge (Toolforge iteration 09): [toolforge] Investigate authentication - https://phabricator.wikimedia.org/T363983 (10dcaro) 03NEW [09:54:01] 10Toolforge (Toolforge iteration 09): [toolforge] Investigate authentication - https://phabricator.wikimedia.org/T363983#9763328 (10dcaro) p:05Triage→03High [09:54:37] 10Toolforge: [components-api] Get a skeleton of API webservice and implement `/tool//deploy` with build-only features - https://phabricator.wikimedia.org/T362069#9763331 (10dcaro) [09:54:38] 10Toolforge (Toolforge iteration 09): [toolforge] Investigate authentication - https://phabricator.wikimedia.org/T363983#9763330 (10dcaro) [09:56:38] 10Toolforge: [builds-api] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363808#9763335 (10dcaro) Wait for {T363983}, as it might not be needed [09:56:43] 10Toolforge: [jobs-apii] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363346#9763338 (10dcaro) Wait for {T363983}, as it might not be needed [09:57:12] 10Toolforge: [envvars-api] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363809#9763341 (10dcaro) Wait for {T363983}, as it might not be needed [10:25:35] 14Grid-Engine-to-K8s-Migration: Migrate parliamentdiagram from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319955#9763367 (10Gouvernathor) The files in public_html/svgfiles. You can look inside the parliamentdiagram-beta tool, that's where we do our tests before deployin... [11:26:47] 14Grid-Engine-to-K8s-Migration: Migrate parliamentdiagram from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319955#9763491 (10Gouvernathor) Nvm. They were the files in public_html/svgfiles, but the problem came from the fact that an absolute path to the main tool was used... [14:12:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:17:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:22:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:27:42] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:00:49] FIRING: PuppetDisabled: Puppet disabled on cloudrabbit1003:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [16:00:57] 06cloud-services-team: PuppetDisabled Puppet disabled on cloudrabbit1003:9100 - https://phabricator.wikimedia.org/T364020 (10phaultfinder) 03NEW [16:05:49] FIRING: [3x] PuppetDisabled: Puppet disabled on cloudrabbit1001:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [16:05:54] 06cloud-services-team: PuppetDisabled - https://phabricator.wikimedia.org/T364021 (10phaultfinder) 03NEW [16:06:35] 10superset.wmcloud.org: Upgrade to 4.0.0 - https://phabricator.wikimedia.org/T364022 (10rook) 03NEW [16:45:44] 10superset.wmcloud.org: Upgrade to 4.0.0 - https://phabricator.wikimedia.org/T364022#9764591 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/superset-deploy/pull/21 [16:45:57] vivian-rook opened https://github.com/toolforge/superset-deploy/pull/21 [16:50:12] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [labs/private] - 10https://gerrit.wikimedia.org/r/1025877 (https://phabricator.wikimedia.org/T360414) (owner: 10Andrea Denisse) [16:51:05] (03CR) 10Dzahn: [C:03+1] ssl: Delete dummy TLS key for the Grafana hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1025877 (https://phabricator.wikimedia.org/T360414) (owner: 10Andrea Denisse) [16:51:19] RESOLVED: [2x] PuppetDisabled: Puppet disabled on cloudrabbit1001:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [17:25:21] (03CR) 10Andrea Denisse: [V:03+2] ssl: Delete dummy TLS key for the Grafana hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1025877 (https://phabricator.wikimedia.org/T360414) (owner: 10Andrea Denisse) [17:25:44] (03CR) 10Andrea Denisse: [V:03+2 C:04-2] ssl: Delete dummy TLS key for the Grafana hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1025877 (https://phabricator.wikimedia.org/T360414) (owner: 10Andrea Denisse) [17:25:51] (03CR) 10Andrea Denisse: [V:03+2 C:03+2] ssl: Delete dummy TLS key for the Grafana hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1025877 (https://phabricator.wikimedia.org/T360414) (owner: 10Andrea Denisse) [17:36:14] (03PS1) 10Dzahn: delete fake key for query-preview.wikidata.org [labs/private] - 10https://gerrit.wikimedia.org/r/1026623 (https://phabricator.wikimedia.org/T333656) [17:36:47] (03PS2) 10Dzahn: delete fake key for query-preview.wikidata.org [labs/private] - 10https://gerrit.wikimedia.org/r/1026623 (https://phabricator.wikimedia.org/T333656) [17:39:04] (03PS1) 10Dzahn: delete fake key for thanos-query.discovery.wmnet.key [labs/private] - 10https://gerrit.wikimedia.org/r/1026646 [17:40:51] (03PS1) 10Dzahn: delete fake key for prometheus.wikimedia.org [labs/private] - 10https://gerrit.wikimedia.org/r/1026647 [17:42:59] (03CR) 10Andrea Denisse: [C:03+1] "LGTM, thank you!" [labs/private] - 10https://gerrit.wikimedia.org/r/1026646 (owner: 10Dzahn) [17:43:55] (03CR) 10Andrea Denisse: [C:03+1] "LGTM, thank you!" [labs/private] - 10https://gerrit.wikimedia.org/r/1026647 (owner: 10Dzahn) [17:44:59] (03CR) 10Andrea Denisse: [C:03+1] "LGTM, thank you!" [labs/private] - 10https://gerrit.wikimedia.org/r/1026623 (https://phabricator.wikimedia.org/T333656) (owner: 10Dzahn) [17:47:33] (03PS2) 10Dzahn: delete fake key for thanos-query.discovery.wmnet.key [labs/private] - 10https://gerrit.wikimedia.org/r/1026646 [17:47:38] (03CR) 10Andrea Denisse: [C:03+2] delete fake key for thanos-query.discovery.wmnet.key [labs/private] - 10https://gerrit.wikimedia.org/r/1026646 (owner: 10Dzahn) [17:47:41] (03CR) 10Andrea Denisse: [V:03+2 C:03+2] delete fake key for thanos-query.discovery.wmnet.key [labs/private] - 10https://gerrit.wikimedia.org/r/1026646 (owner: 10Dzahn) [17:47:52] (03PS2) 10Dzahn: delete fake key for prometheus.wikimedia.org [labs/private] - 10https://gerrit.wikimedia.org/r/1026647 [17:47:54] (03CR) 10Andrea Denisse: [C:03+2] delete fake key for prometheus.wikimedia.org [labs/private] - 10https://gerrit.wikimedia.org/r/1026647 (owner: 10Dzahn) [17:47:56] (03CR) 10Andrea Denisse: [V:03+2 C:03+2] delete fake key for prometheus.wikimedia.org [labs/private] - 10https://gerrit.wikimedia.org/r/1026647 (owner: 10Dzahn) [17:50:24] (03PS1) 10Dzahn: delete phabricator-stage-1001.devtools.eqiad.wmflabs.key [labs/private] - 10https://gerrit.wikimedia.org/r/1026649 [17:54:57] (03PS1) 10Dzahn: delete bugzilla.wikimedia.org.key [labs/private] - 10https://gerrit.wikimedia.org/r/1026653 [18:00:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [18:07:40] (03CR) 10Dzahn: [V:03+2 C:03+2] delete fake key for query-preview.wikidata.org [labs/private] - 10https://gerrit.wikimedia.org/r/1026623 (https://phabricator.wikimedia.org/T333656) (owner: 10Dzahn) [18:08:49] (03CR) 10Dzahn: [V:03+2 C:03+2] "instance doesn't exist anymore" [labs/private] - 10https://gerrit.wikimedia.org/r/1026649 (owner: 10Dzahn) [18:14:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:14:50] 10Tools, 06translatewiki.net, 10Language-Team (Language-2024-April-June), 03Localization Infrastructure FY2023-24, and 2 others: Make Wikidata Image Positions tool translatable on translatewiki.net - https://phabricator.wikimedia.org/T363626#9764920 (10Nikerabbit) [18:24:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:42:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:52:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:31:40] 10Tool-itwiki: Aggiungere un'etichetta alle modifiche di JWB - https://phabricator.wikimedia.org/T363923#9765268 (10Melos) L'etichetta porta benefici di verifica postuma degli edit. Sarei per il supporto locale essendo un tool il cui sviluppatore è quasi inattivo (vedi per esempio [[https://github.com/wikimedia-... [21:04:57] 06cloud-services-team, 06Infrastructure-Foundations, 10Puppet-Infrastructure: puppet servers run out of inodes in puppet code volume - https://phabricator.wikimedia.org/T364047 (10Andrew) 03NEW [21:26:51] 10Tools, 10Gerrit, 03Wikimedia-Hackathon-2024: Gerrit reviewer bot should have an option to add people as CC instead of reviewers - https://phabricator.wikimedia.org/T363290#9765404 (10matmarex) [21:28:44] 10Tools, 10Gerrit, 03Wikimedia-Hackathon-2024: Gerrit reviewer bot should have an option to add people as CC instead of reviewers - https://phabricator.wikimedia.org/T363290#9765411 (10matmarex) Thanks for the comments, I renamed the task just to make it clear I'm not planning to break anyone's workflow, but... [21:30:28] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [22:07:40] (03PS1) 10Andrea Denisse: ssl: Remove unnecessary dummy key for the kibana hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1026693 (https://phabricator.wikimedia.org/T360414) [22:08:14] (03CR) 10Andrea Denisse: "To be merged after the CFSSL migration." [labs/private] - 10https://gerrit.wikimedia.org/r/1026693 (https://phabricator.wikimedia.org/T360414) (owner: 10Andrea Denisse) [22:12:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:17:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:22:32] (03CR) 10Dzahn: [V:03+2 C:03+2] delete bugzilla.wikimedia.org.key [labs/private] - 10https://gerrit.wikimedia.org/r/1026653 (owner: 10Dzahn) [22:22:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:27:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:34:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-meta-monitor-1 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:42:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:52:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks