[00:13:35] (03open) 10jaredblumer: Fix Paths Operation Summary Validation and Enable Paths Ruleset Tests [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/6 [00:15:45] (03update) 10jaredblumer: Fix Paths Operation Summary Validation and Enable Paths Ruleset Tests [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/6 [00:16:34] (03update) 10jaredblumer: Fix Paths Operation Summary Validation and Enable Paths Ruleset Tests [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/6 [01:11:00] FIRING: [2x] PuppetCertificateAboutToExpire: Puppet CA certificate fullstackd-20210112012248.admin-monitoring.eqiad1.wikimedia.cloud is about to expire in 19d 0h 15m 34s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [08:12:05] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 07affects-Kiwix-and-openZIM: Temporary quota increase for mwoffliner project - https://phabricator.wikimedia.org/T411751#11482851 (10Benoit74) 05Resolved→03Open Please revert quota change, everything has been migrated, thank you. [08:38:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:27:36] 06cloud-services-team, 10Cloud-VPS: Can't connect to the CloudVPS instance - https://phabricator.wikimedia.org/T413312#11482971 (10fnegri) 05Open→03Resolved a:03taavi This was resolved by @taavi over IRC: `lang=irc 22:40 openstack correctly sees you in the bastion project (https://openstack-... [10:49:55] FIRING: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [10:50:28] FIRING: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [11:08:22] 06cloud-services-team (FY2025/26-Q1-Q2), 10PAWS: PAWS failing to mount volumes - https://phabricator.wikimedia.org/T413428 (10fnegri) 03NEW [11:08:30] 06cloud-services-team (FY2025/26-Q1-Q2), 10PAWS: PAWS failing to mount volumes - https://phabricator.wikimedia.org/T413428#11483019 (10fnegri) 05Open→03In progress p:05Triage→03High [11:10:39] 06cloud-services-team (FY2025/26-Q1-Q2), 10PAWS: PAWS failing to mount volumes - https://phabricator.wikimedia.org/T413428#11483021 (10fnegri) I tried deleting the pod with `kubectl delete pod hub-5bf7498868-gkxgs -n prod`, the new pod is stuck with a slightly different error: ` root@bastion:~# kubectl descri... [11:16:11] 06cloud-services-team (FY2025/26-Q1-Q2), 10PAWS: PAWS failing to mount volumes - https://phabricator.wikimedia.org/T413428#11483023 (10fnegri) An after another `delete` I'm getting an error from the Openstack API: ` Events: Type Reason Age From Messa... [11:31:47] 10Tool-wsindex, 10Wikisource Reader App: Display extra information from Wikidata in Book Details Screen - https://phabricator.wikimedia.org/T406245#11483041 (10Bodhisattwa) a:05Saiphani02→03Muguro [11:32:16] 10Tool-wsindex, 10Wikisource Reader App: Display extra information from Wikidata in Book Details Screen - https://phabricator.wikimedia.org/T406245#11483042 (10Bodhisattwa) [11:32:17] 10Tool-wsindex, 10Wikisource Reader App, 10Outreachy (Round 31): Outreachy 31: Improve the Wikisource Reader App - https://phabricator.wikimedia.org/T405593#11483043 (10Bodhisattwa) [11:36:08] 10Tool-wsindex, 10Wikisource Reader App, 10Outreachy (Round 31): Outreachy 31: Improve the Wikisource Reader App - https://phabricator.wikimedia.org/T405593#11483057 (10Bodhisattwa) [11:41:34] 06cloud-services-team (FY2025/26-Q1-Q2), 10PAWS: PAWS failing to mount volumes - https://phabricator.wikimedia.org/T413428#11483058 (10fnegri) I sshed to node-1 and the only one clouddump is mounted: `lang=shell-session [core@paws-127c-uwce57bvcgrt-node-1 ~]$ mount |grep clouddumps clouddumps1002.wikimedia.or... [11:41:59] 10Tool-wsindex, 10Wikisource Reader App, 10Outreachy (Round 31): Outreachy 31: Improve the Wikisource Reader App - https://phabricator.wikimedia.org/T405593#11483060 (10Bodhisattwa) [11:48:23] 06cloud-services-team (FY2025/26-Q1-Q2), 10PAWS: PAWS failing to mount volumes - https://phabricator.wikimedia.org/T413428#11483065 (10fnegri) I tried using a different mount directory, and mounting clouddumps1002 works fine, but clouddumps1001 is failing: `lang=shell-session // This fails [root@paws-127c-uwc... [11:51:34] 06cloud-services-team (FY2025/26-Q1-Q2), 10PAWS: PAWS failing to mount volumes - https://phabricator.wikimedia.org/T413428#11483069 (10fnegri) 05In progress→03Resolved a:03fnegri Restarting the nfs-server unit in `clouddumps1001` fixed it: `lang=shell-session root@clouddumps1001:~# systemctl restart... [11:54:55] RESOLVED: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [11:55:28] RESOLVED: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [11:55:51] 06cloud-services-team (FY2025/26-Q1-Q2), 10Cloud-VPS, 10Toolforge: If the inactive clouddumps host goes down, it causes a ripple effect on Cloud VPS and Toolforge - https://phabricator.wikimedia.org/T391369#11483075 (10fnegri) Adding a note that PAWS can also be affected by the unavailability of one of the t... [12:29:12] 06cloud-services-team (FY2025/26-Q1-Q2), 10Toolforge: updatetools is frequently timing out - https://phabricator.wikimedia.org/T413431 (10fnegri) 03NEW [12:29:58] 06cloud-services-team, 10Toolforge: updatetools should alert instead of emailing - https://phabricator.wikimedia.org/T413099#11483109 (10fnegri) [12:31:00] 06cloud-services-team, 10Toolforge: updatetools should alert instead of emailing - https://phabricator.wikimedia.org/T413099#11483111 (10fnegri) I renamed this task to be about converting the email-based alerts to prometheus-based alerts, where we can add more details and link to a runbook. I created a subtas... [12:31:19] 06cloud-services-team (FY2025/26-Q1-Q2), 10Toolforge: updatetools is frequently timing out - https://phabricator.wikimedia.org/T413431#11483113 (10fnegri) 05Open→03In progress p:05Triage→03High a:03fnegri [12:33:14] 06cloud-services-team (FY2025/26-Q1-Q2), 10Toolforge: updatetools is frequently timing out - https://phabricator.wikimedia.org/T413431#11483124 (10fnegri) [12:38:56] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:54:07] 06Toolforge-standards-committee, 07User-notice: November 2025 TFSC appointments - https://phabricator.wikimedia.org/T409290#11483202 (10JJMC89) [13:57:49] 06Toolforge-standards-committee: Facilitate Volunteer NDA application process for 2025 Toolforge standards committee appointees - https://phabricator.wikimedia.org/T413434 (10JJMC89) 03NEW [14:03:32] 06Toolforge-standards-committee: Facilitate Volunteer NDA application process for 2025 Toolforge standards committee appointees - https://phabricator.wikimedia.org/T413434#11483224 (10Soda) T366032 exists for me, not sure if that is sufficient [14:48:07] (03approved) 10wikigit: Fix Paths Operation Summary Validation and Enable Paths Ruleset Tests [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/6 (owner: 10jaredblumer) [14:48:10] (03merge) 10wikigit: Fix Paths Operation Summary Validation and Enable Paths Ruleset Tests [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/6 (owner: 10jaredblumer) [14:58:01] (03CR) 10Agamyasamuel: "recheck" [labs/striker] - 10https://gerrit.wikimedia.org/r/1195355 (https://phabricator.wikimedia.org/T405526) (owner: 10Divyaratann Srivastava) [16:27:58] (03open) 10harroyo-wmf: Drop specification for /checkuser/v0/temporaryaccount/{name}/revisions/{ids} [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/7 [16:29:22] (03update) 10harroyo-wmf: Drop specification for /checkuser/v0/temporaryaccount/{name}/revisions/{ids} [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/7 [16:29:52] (03update) 10harroyo-wmf: Drop specification for /checkuser/v0/temporaryaccount/{name}/revisions/{ids} [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/7 [16:30:21] (03update) 10harroyo-wmf: Drop specification for /checkuser/v0/temporaryaccount/{name}/revisions/{ids} [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/7 [16:38:56] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:11:33] 10Tool-Pageviews: Generate 2025 topviews yearly datasets - https://phabricator.wikimedia.org/T413393#11483720 (10MusikAnimal) [20:22:11] 06Toolforge-standards-committee: Facilitate Volunteer NDA application process for 2025 Toolforge standards committee appointees - https://phabricator.wikimedia.org/T413434#11483762 (10bd808) >>! In T413434#11483224, @Soda wrote: > Noting that, T366032 exists for me back when I requested access to logstash, not s... [20:23:02] 06Toolforge-standards-committee: Facilitate Volunteer NDA application process for 2025 Toolforge standards committee appointees - https://phabricator.wikimedia.org/T413434#11483766 (10bd808) [20:23:20] 06Toolforge-standards-committee: Facilitate Volunteer NDA application process for 2025 Toolforge standards committee appointees - https://phabricator.wikimedia.org/T413434#11483768 (10bd808) 05Open→03Resolved [20:23:52] 06Toolforge-standards-committee, 07User-notice: November 2025 TFSC appointments - https://phabricator.wikimedia.org/T409290#11483770 (10bd808) [20:37:13] 06Toolforge-standards-committee, 07User-notice: November 2025 TFSC appointments - https://phabricator.wikimedia.org/T409290#11483818 (10JJMC89) [20:38:56] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:55:27] 06Toolforge-standards-committee: Facilitate Volunteer NDA application process for 2025 Toolforge standards committee appointees - https://phabricator.wikimedia.org/T413434#11483822 (10JJMC89) 05Resolved→03Open Soda still needs to be added to #wmf-nda.