[00:15:58] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/20 [00:15:58] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/20 [00:16:02] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/20 [01:02:04] FIRING: ObjectStorageObjectQuotaFull: Object storage quota by 'objects' is 82.55% full for project toolsbeta-logging - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageObjectQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageObjectQuotaFull [03:59:21] (03open) 10countcount: Add /currentname expires field and serve the walk from a ToolsDB global_renames mirror [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/37 [04:26:46] (03update) 10countcount: Add /currentname expires field and serve the walk from a ToolsDB global_renames mirror [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/37 [04:27:04] (03update) 10countcount: Add /currentname expires field and serve the walk from a ToolsDB global_renames mirror [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/37 [04:53:53] (03update) 10countcount: Add /currentname expires field and serve the walk from a ToolsDB global_renames mirror [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/37 [04:54:13] (03update) 10countcount: Add /currentname expires field and serve the walk from a ToolsDB global_renames mirror [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/37 [05:02:04] FIRING: ObjectStorageObjectQuotaFull: Object storage quota by 'objects' is 83.15% full for project toolsbeta-logging - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageObjectQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageObjectQuotaFull [05:53:29] 10Tool-tome: Agreements in Tome should accommodate oral consent script and incorporate other features - https://phabricator.wikimedia.org/T431192#12088649 (10psubhashish1) 05Openβ†’03Resolved a:03psubhashish1 Features added in current version [05:57:19] 06cloud-services-team, 10Toolforge, 06Toolforge-standards-committee: Request publishable source code and task configuration for JarBot under the Toolforge Right to fork policy - https://phabricator.wikimedia.org/T431237 (10samy) 03NEW [06:08:47] (03merge) 10countcount: Add /currentname expires field and serve the walk from a ToolsDB global_renames mirror [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/37 [06:15:56] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/28 [06:17:29] 06Toolforge-standards-committee: Request publishable source code and task configuration for JarBot under the Toolforge Right to fork policy - https://phabricator.wikimedia.org/T431237#12088696 (10JJMC89) [06:20:54] 06Toolforge-standards-committee: Request publishable source code and task configuration for JarBot under the Toolforge Right to fork policy - https://phabricator.wikimedia.org/T431237#12088698 (10Bugreporter) [06:29:43] 06Toolforge-standards-committee: Request publishable source code and task configuration for JarBot under the Toolforge Right to fork policy - https://phabricator.wikimedia.org/T431237#12088717 (10JJMC89) 05Openβ†’03Invalid Nothing is available to be provided was the tool was deleted years ago. [06:38:53] 06cloud-services-team, 10Toolforge: Disabled Toolforge tool deletions not occurring - https://phabricator.wikimedia.org/T431241 (10JJMC89) 03NEW [06:44:51] (03merge) 10countcount: Update pnpm to v11.10.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/19 (owner: 10renovatebot) [06:45:10] (03merge) 10countcount: Update pnpm to v11.10.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/35 (owner: 10renovatebot) [06:59:49] (03open) 10countcount: Tag releases and gate deploys on green CI [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/36 [07:00:02] (03open) 10countcount: Tag releases and gate prod deploys on green CI [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/38 [07:00:39] (03open) 10countcount: Tag releases and gate deploys on green CI [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/21 [07:00:59] (03open) 10countcount: Add tagged, CI-gated Toolforge deploy script [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/24 [07:01:03] (03open) 10countcount: Add tagged, CI-gated Toolforge deploy script [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/36 [07:01:09] (03open) 10countcount: Add tagged, CI-gated Toolforge deploy script [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/27 [07:03:48] FIRING: PuppetFailure: Puppet has failed on clouddb1028:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [07:08:20] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{P:openstack::codfw1dev::nova::compute::service} AND NOT P{F:kernelversion = 6.12.95}' [07:15:36] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/34 [07:16:30] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [07:47:00] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12088841 (10CWilliams-WMF) > I guess we could add a quick check that only runs umount if /srv is a m... [07:47:24] (03PS1) 10Filippo Giunchedi: cloudvirt: use 'compute service set' to handle maintenance [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1307689 (https://phabricator.wikimedia.org/T424802) [07:47:27] (03PS1) 10Filippo Giunchedi: wmcs_libs: deprecate openstack aggregates handling [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1307690 (https://phabricator.wikimedia.org/T424802) [07:48:51] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: cloudvirt1075 in 'maintenance' aggregate - https://phabricator.wikimedia.org/T424802#12088847 (10fgiunchedi) >>! In T424802#12058115, @fgiunchedi wrote: >>>! In T424802#12054459, @Andrew wrote: >> I'm not sure why this is happening. Most likely it's due... [07:49:02] (03merge) 10samwilson: Move targetLang fetching to within the change handler [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/12 (https://phabricator.wikimedia.org/T430917) [07:49:37] (03merge) 10samwilson: Fix various minor eslint warnings [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/13 [07:51:18] 10Toolforge (Quota-requests), 06Machine-Learning-Team: Request increased quota for wiki-tts Toolforge tool - https://phabricator.wikimedia.org/T425804#12088908 (10isarantopoulos) [07:55:04] (03CR) 10CI reject: [V:04-1] wmcs_libs: deprecate openstack aggregates handling [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1307690 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [07:57:08] 10Cloud-VPS (Project-requests), 06Machine-Learning-Team: Request creation of wikitts VPS project - https://phabricator.wikimedia.org/T425909#12088965 (10isarantopoulos) [07:58:05] (03PS2) 10Filippo Giunchedi: wmcs_libs: deprecate openstack aggregates handling [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1307690 (https://phabricator.wikimedia.org/T424802) [08:03:47] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: cloudvirt1075 in 'maintenance' aggregate - https://phabricator.wikimedia.org/T424802#12088993 (10fgiunchedi) Current situation wrt aggregates `ceph`, `network-ovs` and `maintenance` ` root@cloudcontrol1006:~# wmcs-openstack aggregate show maintenance +... [08:04:05] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'P{P:openstack::codfw1dev::nova::compute::service} AND NOT P{F:kernelversion = 6.12.95}' [08:07:41] (03update) 10samwilson: Switch machine translation from checkbox to button [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/11 (https://phabricator.wikimedia.org/T425082) [08:10:45] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Revisit cloudvirt maintenance story - https://phabricator.wikimedia.org/T424802#12089010 (10fgiunchedi) [08:13:32] (03merge) 10samwilson: Switch machine translation from checkbox to button [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/11 (https://phabricator.wikimedia.org/T425082) [08:15:50] 10Toolforge, 06tools-platform-team: [toolsbeta,infra,logging] We are reaching 86% of quota by objects - https://phabricator.wikimedia.org/T431248 (10dcaro) 03NEW [08:17:01] 10Toolforge, 06tools-platform-team: [pywikibot-buildservice] New upstream release for Pywikibot - https://phabricator.wikimedia.org/T431056#12089034 (10dcaro) [08:18:59] 10Toolforge, 06tools-platform-team: [toolforge] Investigate authentication - https://phabricator.wikimedia.org/T363983#12089048 (10dcaro) Just created a new service in idp.wmcloud.org CAS for local development (so we can authenticate into lima-kilo). The credentials are in the secrets repo (service `local_dev`... [08:22:42] 10Toolforge, 06tools-platform-team: [toolforge] Investigate authentication - https://phabricator.wikimedia.org/T363983#12089051 (10dcaro) For client authentication, we might have to extend the current idp puppet code to allow setting up [[https://datatracker.ietf.org/doc/html/rfc7636 | PKCE]]. [08:36:26] PROBLEM - Host cloudvirt1040 is DOWN: PING CRITICAL - Packet loss = 100% [08:36:39] 10Toolforge, 06tools-platform-team: [infra,o11y] ToolforgeWebHighErrorRate should not page if a single tool is down - https://phabricator.wikimedia.org/T429738#12089139 (10fgiunchedi) >>! In T429738#12085068, @taavi wrote: > Moving the alerting to be based on Istio-level metrics means we lose visibility into a... [08:37:48] (03PS3) 10Arendpieter: Upgrade dev wiki to 1.43 LTS and drop OpenStackManager [labs/striker] - 10https://gerrit.wikimedia.org/r/1295943 (https://phabricator.wikimedia.org/T427824) [08:38:56] RECOVERY - Host cloudvirt1040 is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms [08:43:13] (03PS5) 10Arendpieter: Use IDP for authentication [labs/striker] - 10https://gerrit.wikimedia.org/r/1250537 (https://phabricator.wikimedia.org/T359554) [08:43:48] RESOLVED: PuppetFailure: Puppet has failed on clouddb1028:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [08:45:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [08:54:22] PROBLEM - Host cloudvirt1041 is DOWN: PING CRITICAL - Packet loss = 100% [08:56:10] RECOVERY - Host cloudvirt1041 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [08:56:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1041 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [09:00:33] (03CR) 10David Caro: [C:03+1] "LGTM, did not test it though, but if you did +1 for removing the hackish code I wrote :)" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1307689 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [09:01:42] (03CR) 10David Caro: [C:03+1] cloudvirt: use 'compute service set' to handle maintenance (032 comments) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1307689 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [09:01:49] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1041 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [09:02:06] (03CR) 10David Caro: [C:03+1] ":deletedelete:" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1307690 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [09:07:14] 06cloud-services-team, 10Toolforge: [disable-tool] Disabled Toolforge tool deletions not occurring - https://phabricator.wikimedia.org/T431241#12089208 (10dcaro) [09:07:17] 06cloud-services-team, 10Toolforge: [disable-tools] Disabled Toolforge tool deletions not occurring - https://phabricator.wikimedia.org/T431241#12089209 (10dcaro) [09:10:40] PROBLEM - Host cloudvirt1042 is DOWN: PING CRITICAL - Packet loss = 100% [09:12:56] RECOVERY - Host cloudvirt1042 is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms [09:23:00] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12089262 (10Marostegui) clouddb1028 is now replicating s4 and s6 [09:27:42] (03CR) 10Filippo Giunchedi: "Thank you for the review David !" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1307689 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [09:30:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [09:42:18] (03CR) 10David Caro: [C:03+1] cloudvirt: use 'compute service set' to handle maintenance (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1307689 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [09:52:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1044 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [09:52:59] (03PS2) 10Filippo Giunchedi: cloudvirt: use 'compute service set' to handle maintenance [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1307689 (https://phabricator.wikimedia.org/T424802) [09:52:59] (03PS3) 10Filippo Giunchedi: wmcs_libs: deprecate openstack aggregates handling [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1307690 (https://phabricator.wikimedia.org/T424802) [09:53:19] (03CR) 10Filippo Giunchedi: cloudvirt: use 'compute service set' to handle maintenance (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1307689 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [09:54:49] 06tools-platform-team: [infra,o11y] Fix/remove all failing systemd units from all VMs and enable alerts on failed unit - https://phabricator.wikimedia.org/T431261 (10dcaro) 03NEW [09:55:06] 10Toolforge, 06tools-platform-team: [infra,o11y] Fix/remove all failing systemd units from all VMs and enable alerts on failed unit - https://phabricator.wikimedia.org/T431261#12089460 (10dcaro) [09:55:36] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: [infra,o11y] Fix/remove all failing systemd units from all VMs and enable alerts on failed unit - https://phabricator.wikimedia.org/T431261#12089461 (10dcaro) [09:56:27] (03open) 10countcount: Add /nameusage endpoint finding all rename chains containing a name [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/39 [09:56:30] (03CR) 10CI reject: [V:04-1] cloudvirt: use 'compute service set' to handle maintenance [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1307689 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [09:56:39] (03CR) 10CI reject: [V:04-1] wmcs_libs: deprecate openstack aggregates handling [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1307690 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [09:57:04] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1044 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [09:57:39] 10Toolforge, 06tools-platform-team: [jobs-api] flaky unit test test_get_one_off_job_status - https://phabricator.wikimedia.org/T430550#12089468 (10dcaro) > @dcaro let’s discuss whether I should go ahead with that little refactoring as my first task on Monday? πŸ‘ we use freezegun in other places yep [09:58:07] 10Toolforge, 06tools-platform-team: [jobs-api] flaky unit test test_get_one_off_job_status - https://phabricator.wikimedia.org/T430550#12089470 (10dcaro) p:05Triageβ†’03Low a:03TLepage-WMF [09:58:22] 10Toolforge, 06tools-platform-team: [jobs-api] flaky unit test test_get_one_off_job_status - https://phabricator.wikimedia.org/T430550#12089485 (10dcaro) 05Openβ†’03In progress [10:04:14] (03PS3) 10Filippo Giunchedi: cloudvirt: use 'compute service set' to handle maintenance [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1307689 (https://phabricator.wikimedia.org/T424802) [10:04:14] (03PS4) 10Filippo Giunchedi: wmcs_libs: deprecate openstack aggregates handling [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1307690 (https://phabricator.wikimedia.org/T424802) [10:32:30] (03update) 10countcount: Add /nameusage endpoint finding all rename chains containing a name [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/39 [10:32:52] (03update) 10countcount: Add /nameusage endpoint finding all rename chains containing a name [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/39 [10:37:39] (03merge) 10countcount: Tag releases and gate prod deploys on green CI [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/38 [10:37:52] (03update) 10countcount: Add /nameusage endpoint finding all rename chains containing a name [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/39 [10:56:26] 06tools-platform-team: jobs-api: allow default command for buildservice images - https://phabricator.wikimedia.org/T431273 (10Raymond_Ndibe) 03NEW [10:58:56] 10Toolforge (Push-to-Deploy), 06tools-platform-team: jobs-api: allow default command for buildservice images - https://phabricator.wikimedia.org/T431273#12089770 (10Raymond_Ndibe) [10:59:13] 10Toolforge, 06tools-platform-team: jobs-api: allow default command for buildservice images - https://phabricator.wikimedia.org/T431273#12089771 (10Raymond_Ndibe) [11:08:50] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1048 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [11:12:12] 10Toolforge (Push-to-Deploy), 06tools-platform-team: jobs-api: resolve some env variable differences between jobs-api and webservice-cli - https://phabricator.wikimedia.org/T431274 (10Raymond_Ndibe) 03NEW [11:12:51] 10Toolforge (Push-to-Deploy), 06tools-platform-team: jobs-api: allow default command for buildservice images - https://phabricator.wikimedia.org/T431273#12089815 (10Raymond_Ndibe) [11:13:49] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1048 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [11:15:26] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/28 [11:23:19] 10Toolforge (Push-to-Deploy), 06tools-platform-team: jobs-api: HTTP health-check probes missing User-Agent headers - https://phabricator.wikimedia.org/T431275 (10Raymond_Ndibe) 03NEW [11:28:11] 10Toolforge (Push-to-Deploy), 06tools-platform-team: jobs-api, jobs-cli: investigate shell / exec into jobs-api containers - https://phabricator.wikimedia.org/T431276 (10Raymond_Ndibe) 03NEW [11:30:08] filippo@cloudcumin1001 safe_reboot (PID 1908085) is awaiting input [11:30:47] (03merge) 10countcount: Add /nameusage endpoint finding all rename chains containing a name [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/39 [11:31:36] (03open) 10samwilson: Add appId for wikitree.com requests [toolforge-repos/wabmd] - 10https://gitlab.wikimedia.org/toolforge-repos/wabmd/-/merge_requests/2 [11:32:34] (03merge) 10samwilson: Add appId for wikitree.com requests [toolforge-repos/wabmd] - 10https://gitlab.wikimedia.org/toolforge-repos/wabmd/-/merge_requests/2 [11:48:37] filippo@cloudcumin1001 safe_reboot (PID 1908085) is awaiting input [11:50:15] 10Toolforge (Push-to-Deploy), 06tools-platform-team: jobs-api: significant resources request/limit differences vs. webservice-cli - https://phabricator.wikimedia.org/T431281 (10Raymond_Ndibe) 03NEW [11:53:10] 06cloud-services-team, 10Striker, 06tools-platform-team: Cannot join Toolforge: Striker throws IntegrityError while saving user - https://phabricator.wikimedia.org/T429759#12089975 (10dcaro) [11:58:53] 06cloud-services-team, 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team: nova.exception.GroupAffinityViolation while live-migrating tools-redis-6 - https://phabricator.wikimedia.org/T431282 (10fgiunchedi) 03NEW [12:01:23] 06cloud-services-team, 10Striker, 06tools-platform-team: Cannot join Toolforge: Striker throws IntegrityError while saving user - https://phabricator.wikimedia.org/T429759#12090017 (10taavi) >>! In T429759#12083896, @bd808 wrote: > I'm not seeing a duplicate: > [cut] > Did someone clean this up or is there a... [12:07:08] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1307775 (owner: 10L10n-bot) [12:15:12] 10Cloud-VPS, 06tools-infrastructure-team: Fold special maps proxies back to the normal proxy - https://phabricator.wikimedia.org/T431284 (10taavi) 03NEW [12:15:19] 10Cloud-VPS, 06tools-infrastructure-team: Fold special maps proxies back to the normal proxy - https://phabricator.wikimedia.org/T431284#12090087 (10taavi) [12:15:24] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Convert dynamicproxy from nginx to haproxy - https://phabricator.wikimedia.org/T429930#12090088 (10taavi) [12:15:29] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/28 [12:16:11] 06cloud-services-team, 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team: nova.exception.GroupAffinityViolation while live-migrating tools-redis-6 - https://phabricator.wikimedia.org/T431282#12090089 (10fgiunchedi) [12:17:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1049 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [12:22:49] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1049 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [12:36:36] 06cloud-services-team, 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team: nova.exception.GroupAffinityViolation while live-migrating tools-redis-6 - https://phabricator.wikimedia.org/T431282#12090193 (10fgiunchedi) I'm not sure there's anything immediate to do here as trying again worked. I suspect the... [12:36:50] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1050 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [12:41:49] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1050 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [12:49:13] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/68 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:49:26] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/68 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:49:38] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/25 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:49:39] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/25 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:49:43] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/25 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:49:52] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/85 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:49:56] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/85 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:49:59] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/85 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:50:10] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/100 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:50:11] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/100 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:50:15] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/100 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:50:25] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/335 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:50:28] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/335 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:50:31] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/335 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:50:48] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/113 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:50:49] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/113 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:50:55] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/113 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:53:25] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.27-20260706125005-50ad2df3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1315 [12:54:26] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: api-gateway: bump to 0.0.100-20260706125031-78535732 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1316 [12:54:26] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: api-gateway: bump to 0.0.100-20260706125031-78535732 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1316 [12:54:53] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/335 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:58:58] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.524-20260706125507-61309993 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1317 [12:59:05] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.524-20260706125507-61309993 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1317 [13:05:49] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/55 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:05:50] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/55 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:06:03] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/55 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:06:12] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/328 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:06:12] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/328 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:06:15] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/328 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:06:20] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/175 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:06:22] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/175 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:06:26] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/175 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:06:32] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/54 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:06:35] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/54 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:06:37] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/54 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:06:47] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/26 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:06:51] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/26 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:06:53] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/26 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:07:04] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/99 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:07:08] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/99 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:07:12] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/99 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:07:33] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/85 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:07:40] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/176 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:07:41] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/176 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:07:44] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/176 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:08:00] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/141 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:08:02] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/141 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:08:08] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/141 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:08:48] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/26 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:08:51] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/99 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:09:27] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/54 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:09:34] (03update) 10raymond-ndibe: runtime: remove diff_with_running_job [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/336 (owner: 10dcaro) [13:09:39] (03update) 10raymond-ndibe: runtime: remove diff_with_running_job [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/336 (owner: 10dcaro) [13:10:04] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.201-20260706130638-44ac0f07 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1318 [13:10:07] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.89-20260706130617-79c2f42b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1319 [13:10:11] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.201-20260706130638-44ac0f07 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1318 [13:10:13] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.89-20260706130617-79c2f42b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1319 [13:10:29] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/176 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:11:34] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: api-gateway: bump to 0.0.101-20260706130914-36718366 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1316 [13:11:45] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.28-20260706130905-ce392384 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1315 [13:11:51] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: api-gateway: bump to 0.0.101-20260706130914-36718366 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1316 [13:11:59] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.28-20260706130905-ce392384 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1315 [13:12:24] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.90-20260706130947-85110ab7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1319 [13:12:33] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.90-20260706130947-85110ab7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1319 [13:13:55] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.202-20260706131050-189403b5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1318 [13:14:06] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.202-20260706131050-189403b5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1318 [13:15:08] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/34 [13:16:23] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [13:16:50] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1052 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [13:21:49] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1052 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [13:32:57] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [13:34:08] 06tools-infrastructure-team: NeutronAgentDown firing during cloudvirt.safe_reboot despite silences - https://phabricator.wikimedia.org/T431300 (10fgiunchedi) 03NEW [13:34:58] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: NeutronAgentDown firing during cloudvirt.safe_reboot despite silences - https://phabricator.wikimedia.org/T431300#12090497 (10fgiunchedi) [13:35:21] 10Cloud-VPS, 06tools-infrastructure-team: NeutronAgentDown firing during cloudvirt.safe_reboot despite silences - https://phabricator.wikimedia.org/T431300#12090499 (10taavi) [13:38:47] 10Cloud-VPS, 06tools-infrastructure-team: NeutronAgentDown firing during cloudvirt.safe_reboot despite silences - https://phabricator.wikimedia.org/T431300#12090502 (10fgiunchedi) [13:40:01] filippo@cloudcumin1001 safe_reboot (PID 1908085) is awaiting input [13:44:40] filippo@cloudcumin1001 safe_reboot (PID 1908085) is awaiting input [13:45:13] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [13:49:09] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [13:51:51] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12090533 (10ops-monitoring-bot) Rebooting db-test2001.codfw.wmnet [13:55:21] (03open) 10tlepage: runtimes/k8s: fix flaky test_get_one_off_job_status [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/338 (https://phabricator.wikimedia.org/T430550) [13:56:32] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12090540 (10ops-monitoring-bot) Reboot of db-test2001.codfw.wmnet completed [13:56:40] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12090542 (10ops-monitoring-bot) Rebooting db-test2002.codfw.wmnet [14:00:01] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12090549 (10ops-monitoring-bot) Reboot of db-test2002.codfw.wmnet completed [14:04:13] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [14:04:50] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1053 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [14:08:04] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12090576 (10ops-monitoring-bot) Rebooting db2230.codfw.wmnet [14:09:07] 10Toolforge, 06tools-platform-team: [jobs-api] Replace black with ruff - https://phabricator.wikimedia.org/T431305 (10TLepage-WMF) 03NEW [14:09:28] 10Toolforge, 06tools-platform-team: [jobs-api] Replace black with ruff - https://phabricator.wikimedia.org/T431305#12090591 (10TLepage-WMF) a:03TLepage-WMF [14:09:50] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1053 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [14:11:34] FIRING: NfsAlmostFull: The NFS drive is over 85% capacity (currently 99.37%) at host paws-nfs-2 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DNfsAlmostFull [14:11:49] 10Cloud-VPS, 06tools-infrastructure-team: Consider deprecating the 'suspend' VM feature - https://phabricator.wikimedia.org/T431307 (10fgiunchedi) 03NEW [14:13:53] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12090627 (10ops-monitoring-bot) Reboot of db2230.codfw.wmnet completed [14:14:01] (03update) 10raymond-ndibe: stop passing tool name [repos/cloud/toolforge/jobs-api] (use_job_kind_for_labels) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/337 (owner: 10dcaro) [14:14:04] (03approved) 10raymond-ndibe: stop passing tool name [repos/cloud/toolforge/jobs-api] (use_job_kind_for_labels) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/337 (owner: 10dcaro) [14:14:09] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12090628 (10Marostegui) >>! In T409557#12089262, @Marostegui wrote: > clouddb1028 is now replicating s4 and s6 Fixed some issues with grafana a... [14:14:58] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/34 [14:15:27] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [14:20:43] (03approved) 10dcaro: jobs-api: bump to 0.0.524-20260706125507-61309993 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1317 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:20:47] (03merge) 10dcaro: jobs-api: bump to 0.0.524-20260706125507-61309993 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1317 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:21:44] (03open) 10tlepage: Replace black+isort with ruff [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/339 (https://phabricator.wikimedia.org/T431305) [14:25:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [14:27:13] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12090685 (10CWilliams-WMF) Confirmation was indeed requested when the host was not a clouddb one. @M... [14:28:12] (03update) 10raymond-ndibe: runtime: remove diff_with_running_job [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/336 (owner: 10dcaro) [14:28:13] (03approved) 10raymond-ndibe: runtime: remove diff_with_running_job [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/336 (owner: 10dcaro) [14:28:26] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12090701 (10cmooney) p:05Triageβ†’03Medium [14:29:32] (03update) 10raymond-ndibe: runtime, core: split get_job/get_jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/327 (owner: 10dcaro) [14:32:35] (03approved) 10raymond-ndibe: runtime, core: split get_job/get_jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/327 (owner: 10dcaro) [14:32:37] (03update) 10raymond-ndibe: runtime, core: split get_job/get_jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/327 (owner: 10dcaro) [14:34:21] (03update) 10raymond-ndibe: runtime: remove diff_with_running_job [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/336 (owner: 10dcaro) [14:34:27] (03update) 10raymond-ndibe: components-api: bump to 0.0.202-20260706131050-189403b5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1318 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:34:30] (03approved) 10raymond-ndibe: components-api: bump to 0.0.202-20260706131050-189403b5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1318 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:34:44] (03update) 10raymond-ndibe: components-api: bump to 0.0.202-20260706131050-189403b5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1318 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:35:24] (03merge) 10raymond-ndibe: components-api: bump to 0.0.202-20260706131050-189403b5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1318 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:35:52] (03update) 10raymond-ndibe: api-gateway: bump to 0.0.101-20260706130914-36718366 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1316 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:36:09] 10Tool-lexeme-forms, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team (Kanban Board): Error β€œThe refresh token is invalid.” after migrating Wikidata Lexeme Forms to OAuth 2 - https://phabricator.wikimedia.org/T431146#12090734 (10JTweed-WMF) [14:36:45] (03update) 10raymond-ndibe: jobs-emailer: bump to 0.0.90-20260706130947-85110ab7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1319 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:37:02] (03update) 10raymond-ndibe: logs-api: bump to 0.0.28-20260706130905-ce392384 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1315 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:37:29] (03update) 10raymond-ndibe: k8s: use job_type for label generating functions [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/328 (owner: 10dcaro) [14:37:34] (03update) 10raymond-ndibe: k8s: use job_type for label generating functions [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/328 (owner: 10dcaro) [14:38:11] (03approved) 10raymond-ndibe: k8s: use job_type for label generating functions [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/328 (owner: 10dcaro) [14:40:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [14:46:42] (03update) 10dcaro: runtime, core: split get_job/get_jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/327 [14:48:50] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1055 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [14:50:01] (03open) 10raymond-ndibe: use __ENTRYPOINT__ keyword to support empty k8s command buildservice images [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/340 (https://phabricator.wikimedia.org/T431273) [14:50:02] (03update) 10raymond-ndibe: use __ENTRYPOINT__ keyword to support empty k8s command buildservice images [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/340 (https://phabricator.wikimedia.org/T431273) [14:52:50] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: jobs-api: allow default command for buildservice images - https://phabricator.wikimedia.org/T431273#12090838 (10Raymond_Ndibe) 05Openβ†’03In progress [14:53:50] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1055 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [14:55:28] (03update) 10raymond-ndibe: use __ENTRYPOINT__ keyword to support empty k8s command buildservice images [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/340 (https://phabricator.wikimedia.org/T431273) [14:56:58] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: jobs-api: allow default command for buildservice images - https://phabricator.wikimedia.org/T431273#12090854 (10dcaro) Another option: * Use the `web` command for buildservice webservices, as that's what is set as the default entrypoint... [15:02:49] (03merge) 10dcaro: runtime, core: split get_job/get_jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/327 [15:02:53] (03update) 10dcaro: k8s: use job_type for label generating functions [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/328 [15:02:58] (03update) 10dcaro: runtime: remove diff_with_running_job [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/336 [15:03:36] 10Toolforge, 06tools-platform-team: [code,ci,general] 1st of the month updates - https://phabricator.wikimedia.org/T430995#12090889 (10dcaro) a:03Raymond_Ndibe [15:03:37] 10Toolforge, 06tools-platform-team: [code,ci,general] 1st of the month updates - https://phabricator.wikimedia.org/T430995#12090891 (10dcaro) 05Openβ†’03In progress [15:03:39] 10Toolforge, 06tools-platform-team: [code,ci,general] 1st of the month updates - https://phabricator.wikimedia.org/T430995#12090892 (10dcaro) p:05Triageβ†’03Medium [15:04:46] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.525-20260706150140-734f60da [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1320 (https://phabricator.wikimedia.org/T430071) [15:04:54] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.525-20260706150140-734f60da [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1320 (https://phabricator.wikimedia.org/T430071) [15:06:03] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: jobs-api: allow default command for buildservice images - https://phabricator.wikimedia.org/T431273#12090899 (10Raymond_Ndibe) a:03Raymond_Ndibe [15:07:10] (03merge) 10countcount: Add tagged, CI-gated Toolforge deploy script [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/27 [15:07:28] (03merge) 10countcount: Add tagged, CI-gated Toolforge deploy script [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/36 [15:07:41] (03merge) 10countcount: Add tagged, CI-gated Toolforge deploy script [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/24 [15:08:15] 10Tool-paulina: Migrate SPARQL queries to other data access methods - https://phabricator.wikimedia.org/T426391#12090903 (10Pepe_piton) [15:09:59] 10Tool-paulina: Migrate SPARQL queries to other data access methods - https://phabricator.wikimedia.org/T426391#12090904 (10Pepe_piton) [15:12:50] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1056 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [15:15:51] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/34 [15:16:51] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [15:16:53] (03update) 10tlepage: Replace black+isort with ruff [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/339 (https://phabricator.wikimedia.org/T431305) [15:16:54] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [15:17:45] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/35 [15:17:50] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1056 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [15:18:14] (03update) 10renovatebot: Update python Docker tag to v3.14.6 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [15:18:17] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/23 [15:18:26] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/26 [15:32:50] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1057 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [15:37:50] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1057 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [15:44:22] 10Toolforge (Push-to-Deploy), 06tools-platform-team: jobs-api: resolve some env variable differences between jobs-api and webservice-cli - https://phabricator.wikimedia.org/T431274#12091020 (10Raymond_Ndibe) * `PORT` env should be added. * `HOME` env for buildservice images was added because `HOME` was being s... [15:45:23] 10Toolforge (Push-to-Deploy), 06tools-platform-team: jobs-api: HTTP health-check probes missing User-Agent headers - https://phabricator.wikimedia.org/T431275#12091023 (10Raymond_Ndibe) * we can ignore. doesn't seem like anyone depends on it [15:52:15] 10Cloud-VPS (Debian Bullseye Deprecation), 06collaboration-services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12091053 (10LSobanski) [15:52:50] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1058 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [15:57:42] (03update) 10tlepage: runtimes/k8s: fix flaky test_get_one_off_job_status [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/338 (https://phabricator.wikimedia.org/T430550) [15:57:50] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1058 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [15:57:57] (03update) 10tlepage: runtimes/k8s: fix flaky test_get_one_off_job_status [repos/cloud/toolforge/jobs-api] (black-to-ruff) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/338 (https://phabricator.wikimedia.org/T430550) [16:12:50] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1059 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [16:15:03] (03update) 10renovatebot: Update pnpm to v11.10.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/1 [16:15:07] (03open) 10renovatebot: Update pnpm to v11.10.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/1 [16:17:06] 10Tool-documentation, 10Adiutor, 06Wiki-Mentor-Africa, 07good first task: [Documentation] Update and improve docs for Adiutor tool - https://phabricator.wikimedia.org/T390392#12091127 (10TBurmeister) Thanks @Priyankar22 for working on this! The structure of this document is really great; it makes it very... [16:17:18] 10Tool-documentation, 10Adiutor, 06Wiki-Mentor-Africa, 07good first task: [Documentation] Update and improve docs for Adiutor tool - https://phabricator.wikimedia.org/T390392#12091129 (10TBurmeister) 05Openβ†’03In progress [16:17:50] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1059 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [16:17:53] (03update) 10raymond-ndibe: logs-api: bump to 0.0.28-20260706130905-ce392384 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1315 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:17:55] (03approved) 10raymond-ndibe: logs-api: bump to 0.0.28-20260706130905-ce392384 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1315 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:17:59] (03merge) 10raymond-ndibe: logs-api: bump to 0.0.28-20260706130905-ce392384 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1315 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:18:15] (03update) 10raymond-ndibe: jobs-emailer: bump to 0.0.90-20260706130947-85110ab7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1319 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:18:17] (03approved) 10raymond-ndibe: jobs-emailer: bump to 0.0.90-20260706130947-85110ab7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1319 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:18:22] (03update) 10raymond-ndibe: jobs-emailer: bump to 0.0.90-20260706130947-85110ab7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1319 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:18:57] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.525-20260706150140-734f60da [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1320 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:19:23] (03merge) 10raymond-ndibe: jobs-emailer: bump to 0.0.90-20260706130947-85110ab7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1319 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:19:46] 10Cloud-VPS (Debian Bullseye Deprecation), 06collaboration-services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12091136 (10Dzahn) for the deployment server an issue is also T257317 (also see latest comments there where Raine runs into it in production) [16:19:56] (03update) 10raymond-ndibe: api-gateway: bump to 0.0.101-20260706130914-36718366 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1316 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:19:59] (03approved) 10raymond-ndibe: api-gateway: bump to 0.0.101-20260706130914-36718366 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1316 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:20:02] (03update) 10raymond-ndibe: api-gateway: bump to 0.0.101-20260706130914-36718366 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1316 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:20:31] (03merge) 10raymond-ndibe: api-gateway: bump to 0.0.101-20260706130914-36718366 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1316 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:27:12] (03update) 10tlepage: runtimes/k8s: fix flaky test_get_one_off_job_status [repos/cloud/toolforge/jobs-api] (black-to-ruff) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/338 (https://phabricator.wikimedia.org/T430550) [16:41:33] 10Tool-wmf-openapi-linter, 05MW-1.47-notes (1.47.0-wmf.10; 2026-07-07), 06MW-Interfaces-Team (MediaWiki-Interfaces-Sprints), 07OKR-Work, 13Patch-For-Review: Fix type, media type, and array issues in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428149#12091209 (10AGhirelli-WMF) 05I... [16:52:20] 10Cloud-Services, 06collaboration-services, 06Data-Persistence, 06Infrastructure-Foundations, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12091218 (10Dzahn) [16:52:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1061 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [16:53:54] 10Cloud-Services, 06collaboration-services, 06Data-Persistence, 06Infrastructure-Foundations, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12091221 (10Dzahn) re: contint2002 - This is not the main CI server, that is currently contint1002, but it is still used to a ce... [16:56:03] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [16:57:49] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1061 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [17:06:14] 06cloud-services-team, 10Toolforge: [toolforge-deploy] detect deployed cli versions in lima-kilo - https://phabricator.wikimedia.org/T401649#12091251 (10dcaro) 05Openβ†’03Resolved a:03dcaro This is working for me, I'll close, please reopen with more details if it's still hapenning: ` dcaro@toolslocal:~... [17:08:14] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [17:08:14] (03update) 10dcaro: update handler to return empty list when no deployments [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/174 (owner: 10buxtang) [17:08:41] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:08:46] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [17:12:21] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [lima-kilo] toolforge_deploy does not support merge requests from forks properly - https://phabricator.wikimedia.org/T374045#12091267 (10dcaro) This is working now, tested with https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/m... [17:12:26] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [lima-kilo] toolforge_deploy does not support merge requests from forks properly - https://phabricator.wikimedia.org/T374045#12091268 (10dcaro) 05Openβ†’03Resolved a:03dcaro [17:15:30] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/34 [17:15:56] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [17:20:34] 06cloud-services-team, 10PAWS, 06tools-platform-team, 13Patch-For-Review: PAWS disk space is running out - https://phabricator.wikimedia.org/T429578#12091363 (10Andrew) A bit of documentation added here: https://wikitech.wikimedia.org/w/index.php?title=PAWS%2FGetting_started_with_PAWS&diff=2433391&oldid=22... [17:21:36] 06cloud-services-team, 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance: ceph-mon 16.2.15+ds-0+deb12u1 uses all the RAM - https://phabricator.wikimedia.org/T398389#12091378 (10dcaro) 05Openβ†’03Resolved @Andrew I'll close this as I think it's already fixed, feel free t... [17:22:17] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [17:23:05] (03approved) 10dcaro: jobs-api: bump to 0.0.525-20260706150140-734f60da [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1320 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:23:21] (03update) 10dcaro: jobs-api: bump to 0.0.525-20260706150140-734f60da [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1320 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:24:06] (03merge) 10dcaro: jobs-api: bump to 0.0.525-20260706150140-734f60da [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1320 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:24:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1063 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [17:29:49] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1063 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [17:40:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1064 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [17:45:49] RESOLVED: [2x] NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1063 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [17:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:56:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1065 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [18:01:50] RESOLVED: [2x] NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1064 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [18:08:50] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1066 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [18:13:49] RESOLVED: [2x] NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1065 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [18:24:50] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1067 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [18:29:50] RESOLVED: [2x] NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1066 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [18:39:59] (03PS2) 10MusikAnimal: AppExtension: cache git hash shown in footer of every page [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307265 (https://phabricator.wikimedia.org/T430888) [18:40:18] (03CR) 10MusikAnimal: AppExtension: cache git hash shown in footer of every page (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307265 (https://phabricator.wikimedia.org/T430888) (owner: 10MusikAnimal) [18:40:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1068 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [18:45:50] RESOLVED: [2x] NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1067 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [18:56:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1069 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [19:01:50] RESOLVED: [2x] NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1068 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [19:02:08] 06cloud-services-team, 10Cloud-VPS: Should we care about the efficiency of wmcs.openstack.cloudvirt.safe_reboot? - https://phabricator.wikimedia.org/T431337 (10Andrew) 03NEW [19:05:40] 06cloud-services-team, 10Toolforge, 06Commons: Commons Assessment Checker: fix VI detection + usability/perf improvements - https://phabricator.wikimedia.org/T431059#12091696 (10Suyash.dwivedi) [19:08:04] 06cloud-services-team, 10Toolforge, 06Commons: Commons Assessment Checker: fix VI detection + usability/perf improvements - https://phabricator.wikimedia.org/T431059#12091699 (10Suyash.dwivedi) Hi @aklapper, This is for wikimedia Commons community not only for my personal use. I have changed the tag and... [19:11:53] 06cloud-services-team, 10Toolforge, 06Commons: Commons Assessment Checker: fix VI detection + usability/perf improvements - https://phabricator.wikimedia.org/T431059#12091725 (10Suyash.dwivedi) 05Invalidβ†’03Open [19:12:36] 06cloud-services-team, 10PAWS, 06tools-platform-team: Should we ban use of 'freeroot' on PAWS? - https://phabricator.wikimedia.org/T431340 (10Andrew) 03NEW [19:18:32] 06cloud-services-team, 10PAWS, 06tools-platform-team: Should we ban use of 'freeroot' on PAWS? - https://phabricator.wikimedia.org/T431340#12091756 (10bd808) I believe the historic use case for the bash shell access was running pywikibiot scripts. That should have no need to install a random VM. I would be f... [19:23:07] 06cloud-services-team, 10PAWS, 06tools-platform-team: Should we ban use of 'freeroot' on PAWS? - https://phabricator.wikimedia.org/T431340#12091788 (10bd808) Is the `freeroot` you are seeing @Andrew this https://github.com/foxytouxxx/freeroot project? [19:27:52] 06cloud-services-team, 10PAWS, 06tools-platform-team: Should we ban use of 'freeroot' on PAWS? - https://phabricator.wikimedia.org/T431340#12091803 (10Andrew) >>! In T431340#12091788, @bd808 wrote: > Is the `freeroot` you are seeing @Andrew this https://github.com/foxytouxxx/freeroot project? yes, at least... [19:40:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1072 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [19:45:49] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1072 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [19:50:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [19:54:56] FIRING: SystemdUnitDown: The service unit maintain-dbusers.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [19:59:56] RESOLVED: SystemdUnitDown: The service unit maintain-dbusers.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [20:00:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:00:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1073 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [20:02:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:05:49] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1073 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [20:05:56] FIRING: SystemdUnitDown: The service unit maintain-dbusers.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [20:16:44] FIRING: MaintainDBUsersDown: Maintain-dbusers is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersDown - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersDown [20:25:56] RESOLVED: SystemdUnitDown: The service unit maintain-dbusers.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [20:26:44] RESOLVED: MaintainDBUsersDown: Maintain-dbusers is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersDown - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersDown [20:27:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:40:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:44:56] FIRING: SystemdUnitDown: The service unit maintain-dbusers.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [20:46:49] FIRING: [2x] NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1075 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [20:47:16] 06cloud-services-team, 10Striker, 06tools-platform-team: Cannot join Toolforge: Striker throws IntegrityError while saving user - https://phabricator.wikimedia.org/T429759#12092190 (10bd808) Thanks @taavi. I'm not sure why I didn't figure that out yet. :) The https://ldap.toolforge.org/user/kavaljeetsinghde... [20:51:49] RESOLVED: [2x] NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1075 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [20:53:02] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:53:07] 06cloud-services-team, 10PAWS, 06tools-platform-team: Should we ban use of 'freeroot' on PAWS? - https://phabricator.wikimedia.org/T431340#12092217 (10Andrew) find . -name README.md -exec grep "Foxytoux" {} \; | wc 3401 32322 244072 [20:55:13] RESOLVED: SystemdUnitDown: The service unit maintain-dbusers.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [21:05:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [21:10:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [21:29:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [21:30:08] filippo@cloudcumin1001 safe_reboot (PID 1908085) is awaiting input [21:34:56] FIRING: SystemdUnitDown: The service unit maintain-dbusers.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [21:35:41] filippo@cloudcumin1001 safe_reboot (PID 1908085) is awaiting input [21:44:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [21:44:56] RESOLVED: SystemdUnitDown: The service unit maintain-dbusers.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [22:46:45] 06cloud-services-team, 10Cloud-VPS: Should we care about the efficiency of wmcs.openstack.cloudvirt.safe_reboot? - https://phabricator.wikimedia.org/T431337#12092445 (10BLiviero-WMF) Curious if you have some thoughts about VM migration timing (how long does it take to move them, and are there better/faster pro... [23:42:13] 10Cloud-Services, 06collaboration-services, 06Data-Persistence, 06Infrastructure-Foundations, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12092634 (10Eevans)