[01:13:33] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Add floating IP and vanity domain for azwikimedia project - https://phabricator.wikimedia.org/T419582#11747307 (10Nemoralis) https://mail.wikimedia.az/ is live! Thanks! [01:20:25] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Add floating IP and vanity domain for azwikimedia project - https://phabricator.wikimedia.org/T419582#11747314 (10PixDeVl) (Just as an extra note, mailcow’s default logo branding is still present on dark mode, only light mode is using the Wikimedia logo) [01:23:39] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Add floating IP and vanity domain for azwikimedia project - https://phabricator.wikimedia.org/T419582#11747318 (10Nemoralis) >>! In T419582#11747314, @PixDeVl wrote: > (Just as an extra note, mailcow’s default logo branding is still present on dark mode, onl... [05:52:46] (03CR) 10MusikAnimal: "Thanks for this!" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1260115 (https://phabricator.wikimedia.org/T392531) (owner: 10Novem Linguae) [05:53:00] (03CR) 10MusikAnimal: [C:03+2] set up eslint-config-wikimedia [labs/xtools] - 10https://gerrit.wikimedia.org/r/1260115 (https://phabricator.wikimedia.org/T392531) (owner: 10Novem Linguae) [05:53:39] (03Merged) 10jenkins-bot: set up eslint-config-wikimedia [labs/xtools] - 10https://gerrit.wikimedia.org/r/1260115 (https://phabricator.wikimedia.org/T392531) (owner: 10Novem Linguae) [05:59:06] (03CR) 10MusikAnimal: [C:03+2] "I hit send on accident before I finished my comment, but anyway what I think what we're going for here in the long-run in XTools-land is [" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1260115 (https://phabricator.wikimedia.org/T392531) (owner: 10Novem Linguae) [06:21:32] 10Tool-wikimonitor, 07good first task, 07patch-welcome: Unable to Edit SpEL Condition from Mobile - https://phabricator.wikimedia.org/T420409#11747536 (10Vishaldevops) @Gerges Hi, I have submitted an updated patch that fixes the mobile touch focus issue in the SpEL editor. The issue occurred because the Cod... [07:01:36] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11747630 (10Spandan1104) [07:45:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [07:49:46] (03CR) 10Novem Linguae: "Thanks for the quick review!" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1260115 (https://phabricator.wikimedia.org/T392531) (owner: 10Novem Linguae) [07:50:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [08:06:46] (03PS1) 10Novem Linguae: eslint: autofix several rules [labs/xtools] - 10https://gerrit.wikimedia.org/r/1260605 (https://phabricator.wikimedia.org/T392531) [08:16:05] (03PS2) 10Novem Linguae: eslint: autofix several rules [labs/xtools] - 10https://gerrit.wikimedia.org/r/1260605 (https://phabricator.wikimedia.org/T392531) [08:27:07] (03PS1) 10Novem Linguae: eslint: add spaces inside parentheses and brackets [labs/xtools] - 10https://gerrit.wikimedia.org/r/1260608 (https://phabricator.wikimedia.org/T392531) [08:41:44] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: grafana.wmcloud.org unavailable - failed db migration - https://phabricator.wikimedia.org/T418831#11747840 (10fgiunchedi) [08:41:55] (03PS1) 10Novem Linguae: eslint: autofix several rules [labs/xtools] - 10https://gerrit.wikimedia.org/r/1260616 (https://phabricator.wikimedia.org/T392531) [08:42:01] (03CR) 10MusikAnimal: [C:03+2] set up eslint-config-wikimedia (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1260115 (https://phabricator.wikimedia.org/T392531) (owner: 10Novem Linguae) [08:45:03] (03CR) 10MusikAnimal: "On my local, `npm run lint` throws tons of errors for the `vendor`/ directory. See https://gerrit.wikimedia.org/r/c/labs/xtools/+/1260115/" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1260605 (https://phabricator.wikimedia.org/T392531) (owner: 10Novem Linguae) [08:45:54] (03PS1) 10Novem Linguae: eslint: ignore vendor directory [labs/xtools] - 10https://gerrit.wikimedia.org/r/1260618 (https://phabricator.wikimedia.org/T392531) [08:46:48] (03CR) 10Novem Linguae: set up eslint-config-wikimedia (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1260115 (https://phabricator.wikimedia.org/T392531) (owner: 10Novem Linguae) [08:47:58] (03CR) 10Novem Linguae: "Roger that. Filed https://gerrit.wikimedia.org/r/c/labs/xtools/+/1260618 to fix that, then we can rebase this patch chain on top of that." [labs/xtools] - 10https://gerrit.wikimedia.org/r/1260605 (https://phabricator.wikimedia.org/T392531) (owner: 10Novem Linguae) [08:50:52] 10Tool-centralnotice-banner-editor: Banner element width is relative to grid layout instead of the input value - https://phabricator.wikimedia.org/T421210 (10Oyelola_Victoria) 03NEW [08:53:00] 10Tools: uploadersincat.php PieChart not working and uploader statistics missing - https://phabricator.wikimedia.org/T420815#11747907 (10Nokib_Sarkar) @MilkyM In the meantime, please use https://uploaders-in-cat.toolforge.org/ as an alternative (basic prototype) created by Wiki Loves Folklore Dev Team. [08:53:05] 06cloud-services-team, 10Toolforge: Java missing from PATH in Clojure buildpack on Toolforge - https://phabricator.wikimedia.org/T411486#11747908 (10KBach) I ran into this again. When using `--use-latest-versions`, the workaround (using `/app/.jdk/bin/java`) no longer works. I'll try to locate and use a differ... [08:58:42] (03CR) 10Thiemo Kreuz (WMDE): [C:03+1] "The same is done in other codebases as well. 👍" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1260618 (https://phabricator.wikimedia.org/T392531) (owner: 10Novem Linguae) [09:32:15] 06cloud-services-team, 10Toolforge: Java missing from PATH in Clojure buildpack on Toolforge - https://phabricator.wikimedia.org/T411486#11747994 (10dcaro) >>! In T411486#11747908, @KBach wrote: > I ran into this again. When using `--use-latest-versions`, the previous workaround (using `/app/.jdk/bin/java`) no... [09:42:52] 06cloud-services-team, 10Toolforge: Java missing from PATH in Clojure buildpack on Toolforge - https://phabricator.wikimedia.org/T411486#11748023 (10KBach) Yes, with the help of `#wikimedia-cloud` :) It'd still be great to have it in `$PATH` though! [09:43:14] 06cloud-services-team, 10Toolforge: Java missing from PATH in Clojure buildpack on Toolforge - https://phabricator.wikimedia.org/T411486#11748024 (10dcaro) We might want to upgrade the buildpack, though it's not cloud native, so it needs some adaptation [10:01:21] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: Move all openstack rabbitmq queues to quorum - https://phabricator.wikimedia.org/T421054#11748093 (10fgiunchedi) The oslo setting I mentioned `rabbit_transient_quorum_queue` refers to the transient queues (reply, fanout) that openstack manages on rabbit,... [10:14:18] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23535729399 (https://github.com/cluebotng/component-configs/commits/c1468d960041cd66ab50902f344fec1ac65ddcad) [10:14:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [10:14:35] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23535729395 (https://github.com/cluebotng/component-configs/commits/c1468d960041cd66ab50902f344fec1ac65ddcad) [10:14:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [10:42:50] (03open) 10dcaro: toolforge_deploy: add `local` option to choices [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/315 [10:43:02] (03update) 10dcaro: toolforge_deploy: add `local` option to choices [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/315 [10:48:43] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Persistence: clouddb1013 crashed after the upgrade to mariadb 10.11.16 - https://phabricator.wikimedia.org/T420177#11748227 (10fnegri) There were two similar SEGV in `clouddb1022`: `lang=shell-session fnegri@clouddb1022:~$ sudo journalctl -u... [10:58:42] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Persistence: clouddb1013 crashed after the upgrade to mariadb 10.11.16 - https://phabricator.wikimedia.org/T420177#11748248 (10Marostegui) >>! In T420177#11731899, @jcrespo wrote: > I see some mentions of table corruption the 25 of february.... [11:15:09] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Persistence: clouddb1013 crashed after the upgrade to mariadb 10.11.16 - https://phabricator.wikimedia.org/T420177#11748296 (10fnegri) > When was clouddb1022 upgraded? Upgraded on 2026-03-16, but it was only pooled yesterday. It looks like t... [11:20:21] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Persistence: clouddb1013 crashed after the upgrade to mariadb 10.11.16 - https://phabricator.wikimedia.org/T420177#11748320 (10Marostegui) >>! In T420177#11748296, @fnegri wrote: >> When was clouddb1022 upgraded? > > Upgraded on 2026-03-16,... [11:22:45] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Persistence: clouddb1013 crashed after the upgrade to mariadb 10.11.16 - https://phabricator.wikimedia.org/T420177#11748327 (10fnegri) > Can you pool clouddb1022 back? I'd like to capture some traces. Repooled. > It is very unlikely this is... [11:27:06] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Persistence: clouddb1013 crashed after the upgrade to mariadb 10.11.16 - https://phabricator.wikimedia.org/T420177#11748332 (10Marostegui) >>! In T420177#11748327, @fnegri wrote: >> Can you pool clouddb1022 back? I'd like to capture some trac... [11:36:46] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Persistence: clouddb1013 crashed after the upgrade to mariadb 10.11.16 - https://phabricator.wikimedia.org/T420177#11748361 (10fnegri) > But maybe it is faster to install 10.11.15 and check clouddb1013 was running 10.11.13 before it started... [11:41:37] (03update) 10samwilson: Use HTTP client from Api class for Wikidata, to set User-Agent [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/7 [11:45:51] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Persistence: clouddb1013 crashed after the upgrade to mariadb 10.11.16 - https://phabricator.wikimedia.org/T420177#11748407 (10Marostegui) >>! In T420177#11748361, @fnegri wrote: >> But maybe it is faster to install 10.11.15 and check > > c... [12:19:33] 06cloud-services-team (FY2025/2026-Q3-Q4), 06Infrastructure-Foundations, 10Puppet-Core, 13Patch-For-Review: Puppet types causing issues around nftables::service - https://phabricator.wikimedia.org/T411102#11748569 (10taavi) 05Open→03Resolved [12:25:53] 06cloud-services-team (FY2025/2026-Q3-Q4), 06Infrastructure-Foundations, 10Puppet-Core, 13Patch-For-Review: Puppet types causing issues around nftables::service - https://phabricator.wikimedia.org/T411102#11748578 (10taavi) 05Resolved→03Open Actually, re-opening since we should do the same thing for nf... [12:26:37] (03update) 10raymond-ndibe: build: add use_deprecated_versions [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/155 (https://phabricator.wikimedia.org/T380127) (owner: 10dcaro) [12:27:18] (03approved) 10raymond-ndibe: build: add use_deprecated_versions [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/155 (https://phabricator.wikimedia.org/T380127) (owner: 10dcaro) [12:42:14] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Persistence: clouddb1013 crashed after the upgrade to mariadb 10.11.16 - https://phabricator.wikimedia.org/T420177#11748642 (10Marostegui) After enabling the error log, nothing new gets dumped, just the recovery but not the reason for the cra... [12:49:22] FIRING: HAProxyBackendUnavailable: HAProxy service wikireplica-db-web-x3 backend clouddb1022.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [12:50:04] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Persistence: clouddb1013 crashed after the upgrade to mariadb 10.11.16 - https://phabricator.wikimedia.org/T420177#11748666 (10Marostegui) I've downgraded clouddb1022 to 10.11.15 - I will pool the host back once the tables have been checked a... [12:52:14] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Persistence: clouddb1013 crashed after the upgrade to mariadb 10.11.16 - https://phabricator.wikimedia.org/T420177#11748670 (10Marostegui) clouddb1022 pooled back with 10.11.15 [12:54:22] RESOLVED: HAProxyBackendUnavailable: HAProxy service wikireplica-db-web-x3 backend clouddb1022.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [13:30:01] 10Cloud-VPS (Quota-requests), 06collaboration-services, 10Continuous-Integration-Infrastructure, 07Jenkins: Large temporary quota increase for integration project to ease Jenkins migration - https://phabricator.wikimedia.org/T421139#11748838 (10hashar) 05Open→03Declined I will rebuild them to Bookw... [13:31:05] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: Support nftables in Cloud VPS - https://phabricator.wikimedia.org/T411089#11748845 (10taavi) a:03taavi [13:36:05] 06cloud-services-team, 10Data-Services, 06Data-Persistence: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#11748915 (10Marostegui) @fnegri is this something you plan to work on or you'd like us to do it? This would be low priority for us at the moment as... [13:43:02] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.remove_instance for instance tools-package-builder-04 [13:43:52] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-package-builder-04 [13:44:25] 06cloud-services-team, 10Toolforge: Retire tools-package-builder-04 - https://phabricator.wikimedia.org/T401819#11748981 (10taavi) 05Open→03Resolved [14:07:53] 06cloud-services-team, 10Data-Services, 06Data-Persistence: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#11749132 (10fnegri) @Marostegui I don't plan to work on this immediately, but I might do in the future, depending on other priorities. If I do, I'l... [14:14:42] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Persistence: clouddb1013 crashed after the upgrade to mariadb 10.11.16 - https://phabricator.wikimedia.org/T420177#11749151 (10fnegri) clouddb1023 also crashed, while clouddb1022 was depooled: `lang=shell-session fnegri@clouddb1023:~$ sudo j... [14:16:05] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Persistence: clouddb1013 crashed after the upgrade to mariadb 10.11.16 - https://phabricator.wikimedia.org/T420177#11749154 (10Marostegui) I can downgrade that one to 10.11.15 and see if there's any difference. I'd like to downgrade one versi... [14:19:11] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Persistence: clouddb1013 crashed after the upgrade to mariadb 10.11.16 - https://phabricator.wikimedia.org/T420177#11749159 (10fnegri) I would keep only clouddb1022 on 10.11.15 for now, and clouddb1023 on 10.11.16, to confirm my theory that c... [14:20:20] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Persistence: clouddb1013 crashed after the upgrade to mariadb 10.11.16 - https://phabricator.wikimedia.org/T420177#11749162 (10Marostegui) Yeah, clouddb1022 was already done: T420177#11748670 [14:38:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:54:06] 10Cloud-VPS (Quota-requests), 10Continuous-Integration-Infrastructure, 10Continuous-Integration-Config, 07Jenkins: New flavor for the integration project with more vCPU and ephemeral disk space - https://phabricator.wikimedia.org/T421242 (10hashar) 03NEW [14:55:32] 06cloud-services-team, 10Data-Services, 06Data-Persistence: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#11749330 (10Marostegui) p:05Triage→03Low Thank you! [14:59:40] 10Cloud-VPS (Quota-requests), 10Catalyst: Disk quota increase for catalyst-dev - https://phabricator.wikimedia.org/T420611#11749347 (10Andrew) +1 approved [15:04:15] 10Cloud-VPS (Quota-requests), 10Browser Test Platform, 10Continuous-Integration-Infrastructure, 10Continuous-Integration-Config, 07Jenkins: New flavor for the integration project with more vCPU and ephemeral disk space - https://phabricator.wikimedia.org/T421242#11749366 (10Peter) [15:13:42] 06cloud-services-team, 10Striker, 10Bitu, 06Infrastructure-Foundations, and 2 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11749383 (10bd808) >>! In T420599#11744332, @Dondersmooi wrote: > My question is: should Bitu and... [15:26:22] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 07Epic, 06Release-Engineering-Team (Priority Backlog 📥): Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T401839#11749487 (10bd808) [15:32:07] 10Cloud-VPS (Quota-requests), 10Browser Test Platform, 10Continuous-Integration-Infrastructure, 10Continuous-Integration-Config, 07Jenkins: New flavor for the integration project with more vCPU and ephemeral disk space - https://phabricator.wikimedia.org/T421242#11749518 (10zeljkofilipin) [15:54:47] 06cloud-services-team, 10Striker, 10Bitu, 06Infrastructure-Foundations, and 2 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11749593 (10Dondersmooi) 05Open→03Declined [15:55:03] 06cloud-services-team, 10Striker, 10Bitu, 06Infrastructure-Foundations, and 2 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11749594 (10Dondersmooi) Thank you all for the thorough answers to all my questions! [16:05:00] 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: [harbor,tools] Harbor object usage in S3 is steadily increasing - https://phabricator.wikimedia.org/T418528#11749647 (10Raymond_Ndibe) Note to self: for the link objects cleanup, maybe we can pull the image before the cleanup, and verify that we can pul... [16:18:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:21:43] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,heat [16:22:28] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,heat [16:30:47] 10Tool-curator: Curator should handle invalid CSRF token errors during uploads - https://phabricator.wikimedia.org/T420961#11749842 (10DaxServer) 05Open→03In progress [16:39:51] (03update) 10fnegri: Introduce diff-mode [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/10 (https://phabricator.wikimedia.org/T351637) [16:42:48] (03update) 10fnegri: Introduce diff-mode [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/10 (https://phabricator.wikimedia.org/T351637) [16:48:25] 10Tool-wikimonitor, 07good first task, 07patch-welcome: Unable to Edit SpEL Condition from Mobile - https://phabricator.wikimedia.org/T420409#11749950 (10Gerges) 05Open→03Resolved a:03Gerges Thanks @Vishaldevops :) [17:01:59] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [17:03:09] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for all services [17:08:17] 10Cloud-VPS (Quota-requests), 10Browser Test Platform, 10Continuous-Integration-Infrastructure, 10Continuous-Integration-Config, 07Jenkins: New flavor for the integration project with more vCPU and ephemeral disk space - https://phabricator.wikimedia.org/T421242#11750096 (10hashar) [17:08:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:08:46] (03update) 10dcaro: support since and until params in logs endpoints [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/12 (https://phabricator.wikimedia.org/T400917) (owner: 10raymond-ndibe) [17:16:27] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [17:17:37] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for all services [17:18:15] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [17:21:32] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services [17:25:04] 10Tool-wikimonitor, 07good first task, 07patch-welcome: Display flagged edits count in browser tab title (dynamic update) - https://phabricator.wikimedia.org/T420850#11750214 (10Gerges) Hi @Praffq! 👋 Great to hear that you're interested in working on this issue — welcome! 🎉 Yes, feel free to go ahead and ope... [17:42:54] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Engineering-Radar, and 4 others: Set up x1 replication to an-redacteddb1001 - https://phabricator.wikimedia.org/T407485#11750287 (10Ahoelzl) [18:01:35] FIRING: NetworkOutSaturated: Outgoing network saturation detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DNetworkOutSaturated [18:06:35] RESOLVED: NetworkOutSaturated: Outgoing network saturation detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DNetworkOutSaturated [19:09:22] (03PS1) 10Akoopal: cleanup of nl-gem fields [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1260777 [20:03:49] (03open) 10devjadiya: Prepare Toolforge production deployment; fix notifier reliability and improve... [toolforge-repos/script-publisher] - 10https://gitlab.wikimedia.org/toolforge-repos/script-publisher/-/merge_requests/2 [20:04:11] (03approved) 10devjadiya: Prepare Toolforge production deployment; fix notifier reliability and improve... [toolforge-repos/script-publisher] - 10https://gitlab.wikimedia.org/toolforge-repos/script-publisher/-/merge_requests/2 [20:11:20] (03merge) 10devjadiya: Prepare Toolforge production deployment; fix notifier reliability and improve... [toolforge-repos/script-publisher] - 10https://gitlab.wikimedia.org/toolforge-repos/script-publisher/-/merge_requests/2 [20:25:32] 10Toolforge (Quota-requests): Request increased quota for techacitivty Toolforge tool - https://phabricator.wikimedia.org/T421292 (10Addshore) 03NEW [20:27:19] 10Toolforge (Quota-requests): Request increased quota for techacitivty Toolforge tool - https://phabricator.wikimedia.org/T421292#11750940 (10Addshore) [20:53:13] 10Tool-inteGraality: Function "" is currently not supported by QLever. - https://phabricator.wikimedia.org/T420247#11751074 (10JeanFred) 05Open→03Resolved We are now using SPAQL compatible with both WDQS and QLever [20:57:50] 10Tool-inteGraality: Support for grouping by monolingual text property - https://phabricator.wikimedia.org/T421301 (10JeanFred) 03NEW [21:10:27] 06cloud-services-team, 10Toolforge, 07Tracking-Neverending: Packages to be added to toollabs puppet - https://phabricator.wikimedia.org/T55704#11751186 (10Pppery) [21:16:09] 06cloud-services-team, 10Toolforge, 05Goal, 07Kubernetes, 13Patch-For-Review: Make webservice backend default to kubernetes - https://phabricator.wikimedia.org/T154504#11751243 (10Pppery) [21:16:39] 06cloud-services-team, 10Striker, 07Epic: Manually connect existing Diffusion repos to tools in toolsadmin - https://phabricator.wikimedia.org/T144282#11751260 (10Pppery) [21:18:13] 06cloud-services-team, 10Data-Services, 10Toolforge, 06DBA, 07Tracking-Neverending: Certain tools users create multiple long running queries that take all memory and/or CPU from labsdb hosts, slowing it down and potentially crashing (tracking) - https://phabricator.wikimedia.org/T119601#11751277 (10Pp... [21:18:22] 10Cloud-Services, 07Tracking-Neverending: Measure capacity and utilization of labs services (Tracking) - https://phabricator.wikimedia.org/T107066#11751281 (10Pppery) The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profil... [21:18:35] 06cloud-services-team, 10Toolforge, 07Documentation, 07Tracking-Neverending: Toolforge admin guides (tracking) - https://phabricator.wikimedia.org/T104734#11751283 (10Pppery) [21:19:05] 10Cloud-Services, 06cloud-services-team (Kanban), 07Tracking-Neverending: Find replacements for various things that people were using NFS for but should not have been (Tracking) - https://phabricator.wikimedia.org/T104193#11751291 (10Pppery) The #Cloud-Services project tag is not intended to have any tas... [21:20:47] 10Cloud-Services, 07Tracking-Neverending: Storage capacity & redundancy expansion (tracking) - https://phabricator.wikimedia.org/T85604#11751320 (10Pppery) The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and... [21:21:28] 10Cloud-Services, 10Beta-Cluster-Infrastructure, 07Tracking-Neverending: Beta Cluster <-> WMF Labs policy compliance (tracking) - https://phabricator.wikimedia.org/T114615#11751327 (10Pppery) The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.w... [21:24:30] (03update) 10raymond-ndibe: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [21:25:54] 06cloud-services-team, 10Toolforge, 07Tracking-Neverending: Missing Toolserver features in Tools (tracking) - https://phabricator.wikimedia.org/T60791#11751390 (10Pppery) The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/... [21:26:21] (03update) 10raymond-ndibe: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [21:27:01] (03update) 10raymond-ndibe: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [21:33:56] 10Tool-extjsonuploader: extjsonuploader update-and-upload job fails with "Error when saving : Invalid CSRF token." - https://phabricator.wikimedia.org/T421307 (10Tgr) 03NEW [21:35:51] 10Tool-extjsonuploader: extjsonuploader update-and-upload job fails with "Error when saving : Invalid CSRF token." - https://phabricator.wikimedia.org/T421307#11751542 (10Tgr) We use `toolforge/extjsonuploader`. The [[https://foundation.wikimedia.org/wiki/Policy:Wikimedia_Foundation_User-Agent_Policy|... [21:48:01] (03approved) 10rho: Rename Author attribution signal to Credit [toolforge-repos/wikimedia-attribution] - 10https://gitlab.wikimedia.org/toolforge-repos/wikimedia-attribution/-/merge_requests/1 (owner: 10sarai-wmf) [21:48:49] (03PS2) 10Gergő Tisza: Add contact URL to user agent [labs/tools/extjsonuploader] - 10https://gerrit.wikimedia.org/r/1260820 (https://phabricator.wikimedia.org/T421307) [21:48:49] (03CR) 10Gergő Tisza: "Did not test but it's a trivial change." [labs/tools/extjsonuploader] - 10https://gerrit.wikimedia.org/r/1260820 (https://phabricator.wikimedia.org/T421307) (owner: 10Gergő Tisza) [21:55:16] (03PS1) 10Dzahn: zuul::executor: add missing dummy password [labs/private] - 10https://gerrit.wikimedia.org/r/1260825 (https://phabricator.wikimedia.org/T421232) [21:56:30] (03CR) 10Dzahn: [V:03+2 C:03+2] zuul::executor: add missing dummy password [labs/private] - 10https://gerrit.wikimedia.org/r/1260825 (https://phabricator.wikimedia.org/T421232) (owner: 10Dzahn) [21:56:49] 10Tool-paulina, 10Outreachy (Round 31): Outreachy 31: Features to edit author and work data on Wikidata directly from Paulina - https://phabricator.wikimedia.org/T392429#11751740 (10LGoto) Please close this task as Resolved if complete. Thanks! [21:57:13] 10Tool-paulina, 10Outreachy (Round 31): OAuth Implementation - https://phabricator.wikimedia.org/T412023#11751751 (10LGoto) Please close this task as Resolved if complete. Thanks! [21:57:53] 10Tool-documentation, 10Adiutor, 06Wiki-Mentor-Africa, 07good first task: [Documentation] Update and improve docs for Adiutor tool - https://phabricator.wikimedia.org/T390392#11751765 (10Pppery) a:05CBID2→03None Resetting assignee since progress seems to have stalled [21:59:32] 10Tool-extjsonuploader, 13Patch-For-Review: extjsonuploader update-and-upload job fails with "Error when saving : Invalid CSRF token." - https://phabricator.wikimedia.org/T421307#11751772 (10A_smart_kitten) Potentially coincides a little bit with {T421168} / {T421161}, potentially related? [22:00:45] (03CR) 10Alien4444: [C:03+2] eslint: ignore vendor directory [labs/xtools] - 10https://gerrit.wikimedia.org/r/1260618 (https://phabricator.wikimedia.org/T392531) (owner: 10Novem Linguae) [22:01:32] (03Merged) 10jenkins-bot: eslint: ignore vendor directory [labs/xtools] - 10https://gerrit.wikimedia.org/r/1260618 (https://phabricator.wikimedia.org/T392531) (owner: 10Novem Linguae) [22:19:11] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:42:15] 10Tool-inteGraality, 07Documentation, 07good first task: Add information to InteGraality user guide from Wikipedia Workbook for Cultural Institutions - https://phabricator.wikimedia.org/T376902#11751983 (10Pppery) a:05Ibom99→03None [23:20:25] (03CR) 10Samwilson: [C:03+2] Add contact URL to user agent [labs/tools/extjsonuploader] - 10https://gerrit.wikimedia.org/r/1260820 (https://phabricator.wikimedia.org/T421307) (owner: 10Gergő Tisza) [23:20:56] (03Merged) 10jenkins-bot: Add contact URL to user agent [labs/tools/extjsonuploader] - 10https://gerrit.wikimedia.org/r/1260820 (https://phabricator.wikimedia.org/T421307) (owner: 10Gergő Tisza)