[00:28:28] 10Openstack-Magnum, 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Priority Backlog 📥): Rebuild the deployment-wikikube-v127 magnum cluster with the new magnum-cluster-api driver and k8s v1.36 - https://phabricator.wikimedia.org/T437500#12529369 (10bd808) 05Open→... [00:43:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:48:00] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [00:48:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:33:00] FIRING: [2x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [02:03:00] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [02:11:00] 10Data-Services, 06Data-Persistence, 06DBA: Please include page_props on x4 - https://phabricator.wikimedia.org/T443973#12529464 (10Bugreporter) I don't believe this task is necessary. Instead, you can do so (all three queries can be completed in one minute): (1) Run the following query and fetch **all** (no... [02:17:43] !log tools.cluebotng-trainer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42328251090 (https://github.com/cluebotng/component-configs/commits/9024a7a5b070a1b79f7c9d815774c4afdb9c260f) [02:17:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [02:20:20] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42328251009 (https://github.com/cluebotng/component-configs/commits/9024a7a5b070a1b79f7c9d815774c4afdb9c260f) [02:20:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [02:20:38] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42328251051 (https://github.com/cluebotng/component-configs/commits/9024a7a5b070a1b79f7c9d815774c4afdb9c260f) [02:20:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [03:31:59] FIRING: BackupCoverageDrop: volume backup coverage dropped on cloudbackup2004:9100 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Backup#BackupCoverageDrop - https://grafana.wikimedia.org/d/fisr2p9/wmcs-backup?viewPanel=21&var-kind=volume&var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DBackupCoverageDrop [03:40:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:50:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:17:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:27:10] (03open) 10ibrahemqasim: Add citation template collection dashboard [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/33 [05:27:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:28:21] (03update) 10ibrahemqasim: Add citation template collection dashboard [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/33 [06:03:00] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [06:56:41] 10Tool-k8s-status: k8s-status should have an API - https://phabricator.wikimedia.org/T443994#12529619 (10Legoktm) [07:31:59] FIRING: BackupCoverageDrop: volume backup coverage dropped on cloudbackup2004:9100 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Backup#BackupCoverageDrop - https://grafana.wikimedia.org/d/fisr2p9/wmcs-backup?viewPanel=21&var-kind=volume&var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DBackupCoverageDrop [08:27:25] (03merge) 10countcount: Update dependency oauthlib to v4.0.1 [SECURITY] [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/47 (owner: 10renovatebot) [08:27:28] (03merge) 10countcount: Update dependency oauthlib to v4.0.1 [SECURITY] [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/54 (owner: 10renovatebot) [08:27:30] (03merge) 10countcount: Update dependency oauthlib to v4.0.1 [SECURITY] [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/42 (owner: 10renovatebot) [09:16:51] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/52 [09:17:16] (03update) 10renovatebot: Update python Docker tag to v3.15.2 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [09:17:33] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/45 [10:03:00] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [10:54:07] 10Tools: Bring up apersonbot after the PHP vulns - https://phabricator.wikimedia.org/T442477#12529717 (10Novem_Linguae) 05Open→03Resolved a:03Novem_Linguae [11:31:59] FIRING: BackupCoverageDrop: volume backup coverage dropped on cloudbackup2004:9100 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Backup#BackupCoverageDrop - https://grafana.wikimedia.org/d/fisr2p9/wmcs-backup?viewPanel=21&var-kind=volume&var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DBackupCoverageDrop [11:52:02] 10Tool-itwiki: Quality assurance 2026 on itWiki repositories (Code of Conduct, CITATION.cff, codemeta.json, Software Heritage) - https://phabricator.wikimedia.org/T444000 (10valerio.bozzolan) 03NEW p:05Triage→03Low [12:29:42] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42372350465 (https://github.com/cluebotng/component-configs/commits/48fbc4fed6236580eba63deced350d7a4c952b15) [12:29:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [12:31:50] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819771 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c) [12:31:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [12:31:52] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819762 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c) [12:31:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [12:31:56] !log tools.cluebotng-trainer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819815 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c) [12:31:57] !log tools.cluebot-syncer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819987 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c) [12:31:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [12:31:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [12:33:44] !log tools.toolforge-functional-runner Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42372819824 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c) [12:33:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.toolforge-functional-runner/SAL [12:34:06] !log tools.cluebot-documentation Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42372819864 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c) [12:34:06] !log tools.cluebotng-monitoring Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372358825 (https://github.com/cluebotng/component-configs/commits/0f7031a50a4ebd3ca73184235a595780de8e5139) [12:34:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-documentation/SAL [12:34:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [12:34:25] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42372819896 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c) [12:34:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [12:34:36] !log tools.cluebot3 Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819831 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c) [12:34:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [12:35:00] !log tools.cluebotng-staging Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819817 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c) [12:35:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [12:35:40] !log tools.cluebot-syncer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42372828537 (https://github.com/cluebotng/component-configs/commits/a2245a8c11578a3bf41b97b386148d20cf6d0688) [12:35:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [12:36:44] !log tools.cluebotng-trainer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372856569 (https://github.com/cluebotng/component-configs/commits/571c4fe0c9c2d2f4afad8b7033bd42575c748d8c) [12:36:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [12:37:11] !log tools.toolforge-functional-runner Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42372828492 (https://github.com/cluebotng/component-configs/commits/a2245a8c11578a3bf41b97b386148d20cf6d0688) [12:37:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.toolforge-functional-runner/SAL [12:37:28] !log tools.cluebotng-editsets Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373017677 (https://github.com/cluebotng/component-configs/commits/93823d543554cbb8a9c17f2b2550b5eaf018c960) [12:37:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [12:37:30] !log tools.cluebot3 Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373017594 (https://github.com/cluebotng/component-configs/commits/93823d543554cbb8a9c17f2b2550b5eaf018c960) [12:37:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [12:37:49] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372856570 (https://github.com/cluebotng/component-configs/commits/571c4fe0c9c2d2f4afad8b7033bd42575c748d8c) [12:37:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [12:37:53] !log tools.cluebot-documentation Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42373017638 (https://github.com/cluebotng/component-configs/commits/93823d543554cbb8a9c17f2b2550b5eaf018c960) [12:37:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-documentation/SAL [12:39:04] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372856760 (https://github.com/cluebotng/component-configs/commits/571c4fe0c9c2d2f4afad8b7033bd42575c748d8c) [12:39:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [12:39:07] !log tools.cluebot-syncer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373130794 (https://github.com/cluebotng/component-configs/commits/39841188970476424bb18049268ac40290ae604f) [12:39:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [12:39:32] !log tools.cluebotng-staging Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373017605 (https://github.com/cluebotng/component-configs/commits/93823d543554cbb8a9c17f2b2550b5eaf018c960) [12:39:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [12:39:36] !log tools.cluebotng-monitoring Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373062046 (https://github.com/cluebotng/component-configs/commits/acb2a23da058868d4d36155c4ddac250479dadc2) [12:39:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [12:41:12] !log tools.cluebotng-trainer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373017679 (https://github.com/cluebotng/component-configs/commits/93823d543554cbb8a9c17f2b2550b5eaf018c960) [12:41:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [12:41:59] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42373303402 (https://github.com/cluebotng/component-configs/commits/0266e661f49a5220ee270323d5bd7eb2617ca4f2) [12:42:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [12:43:37] !log tools.cluebotng-staging Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373224069 (https://github.com/cluebotng/component-configs/commits/b7987054470c43be35c9e93b1e91e43119048538) [12:43:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [12:43:53] !log tools.cluebot3 Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373444118 (https://github.com/cluebotng/component-configs/commits/4b7bb61eba3f6cf3781522588941262f6b984ed5) [12:43:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [12:44:04] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373224062 (https://github.com/cluebotng/component-configs/commits/b7987054470c43be35c9e93b1e91e43119048538) [12:44:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [12:44:49] !log tools.cluebotng-trainer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373423759 (https://github.com/cluebotng/component-configs/commits/0bc0cef490c8c2afb008ff906644dc7fb977bdb6) [12:44:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [12:45:20] !log tools.cluebotng-monitoring Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373298737 (https://github.com/cluebotng/component-configs/commits/5583d26dcec45d54de2c90673628b22357c4abc5) [12:45:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [12:45:34] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373199788 (https://github.com/cluebotng/component-configs/commits/f66fbfcb1dc21963040e937d86c830cbfd33c4e8) [12:45:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [12:47:02] !log tools.cluebot3 Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373456031 (https://github.com/cluebotng/component-configs/commits/9b75826d2873a10720e1149d8b0e5aecd8df070f) [12:47:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [13:18:32] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 10Pywikibot: pywikibot.org nightly dump is unavailable - https://phabricator.wikimedia.org/T444004 (10Xqt) 03NEW [13:18:34] (03CR) 10Wandji collins: [C:03+2] Add the language-first landing page and API client. [labs/tools/wdaudiolex-fe] - 10https://gerrit.wikimedia.org/r/1365942 (owner: 10Wandji collins) [13:18:48] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 10Pywikibot: pywikibot.org nightly dump is unavailable - https://phabricator.wikimedia.org/T444004#12529817 (10Xqt) p:05Triage→03Medium [13:19:15] (03Merged) 10jenkins-bot: Add the language-first landing page and API client. [labs/tools/wdaudiolex-fe] - 10https://gerrit.wikimedia.org/r/1365942 (owner: 10Wandji collins) [13:33:44] 10Tool-itwiki: Quality assurance 2026 on itWiki repositories (Code of Conduct, CITATION.cff, codemeta.json, Software Heritage) - https://phabricator.wikimedia.org/T444000#12529832 (10valerio.bozzolan) a:03valerio.bozzolan I will give a small sprint on this. [13:39:56] 10Tool-itwiki: Quality assurance 2026 on itWiki repositories (Code of Conduct, CITATION.cff, codemeta.json, Software Heritage) - https://phabricator.wikimedia.org/T444000#12529836 (10valerio.bozzolan) [13:44:26] (03PS1) 10Wandji collins: fix and remove not needed files [labs/tools/wdaudiolex-fe] - 10https://gerrit.wikimedia.org/r/1367444 [13:45:45] (03CR) 10Wandji collins: [C:03+2] fix and remove not needed files [labs/tools/wdaudiolex-fe] - 10https://gerrit.wikimedia.org/r/1367444 (owner: 10Wandji collins) [13:46:29] (03Merged) 10jenkins-bot: fix and remove not needed files [labs/tools/wdaudiolex-fe] - 10https://gerrit.wikimedia.org/r/1367444 (owner: 10Wandji collins) [13:54:32] (03PS1) 10Wandji collins: Doc: document authentication workflow and queries for lexeme pronunciation [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1367448 [13:54:49] (03CR) 10CI reject: [V:04-1] Doc: document authentication workflow and queries for lexeme pronunciation [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1367448 (owner: 10Wandji collins) [14:03:00] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [14:08:26] (03PS2) 10Wandji collins: Doc: document authentication workflow and queries for lexeme pronunciation [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1367448 [14:08:41] (03CR) 10CI reject: [V:04-1] Doc: document authentication workflow and queries for lexeme pronunciation [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1367448 (owner: 10Wandji collins) [14:18:35] (03PS3) 10Wandji collins: Doc: document authentication workflow and queries for lexeme pronunciation [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1367448 [14:18:50] (03CR) 10CI reject: [V:04-1] Doc: document authentication workflow and queries for lexeme pronunciation [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1367448 (owner: 10Wandji collins) [14:42:32] (03PS4) 10Wandji collins: Doc: document authentication workflow and queries for lexeme pronunciation [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1367448 [14:42:48] (03CR) 10CI reject: [V:04-1] Doc: document authentication workflow and queries for lexeme pronunciation [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1367448 (owner: 10Wandji collins) [14:47:01] (03PS5) 10Wandji collins: Doc: document authentication workflow and queries for lexeme pronunciation [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1367448 [14:47:33] (03CR) 10Wandji collins: [C:03+2] Doc: document authentication workflow and queries for lexeme pronunciation [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1367448 (owner: 10Wandji collins) [14:47:50] (03Merged) 10jenkins-bot: Doc: document authentication workflow and queries for lexeme pronunciation [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1367448 (owner: 10Wandji collins) [14:59:24] 10Tools: templatetransclusioncheck displays "no healthy upstream" error - https://phabricator.wikimedia.org/T443148#12529896 (10Adithyak1997) @VVLLAACC Can you check now. [15:31:59] FIRING: BackupCoverageDrop: volume backup coverage dropped on cloudbackup2004:9100 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Backup#BackupCoverageDrop - https://grafana.wikimedia.org/d/fisr2p9/wmcs-backup?viewPanel=21&var-kind=volume&var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DBackupCoverageDrop [15:42:16] 10Cloud-VPS (Project-requests): Request creation of deorph VPS project - https://phabricator.wikimedia.org/T442348#12529922 (10fnegri) I worked on this during itWikiCon with @CristianCantoro, we have a working prototype running on Toolforge at: https://deorph.toolforge.org/ -- at the moment this is using only tw... [16:09:25] (03PS1) 10Wandji collins: Configure deployment files [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1367483 [16:50:03] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 10Pywikibot: unexpected root-owned files in /data/project/pywikibot/public_html - https://phabricator.wikimedia.org/T375279#12530041 (10Xqt) Could someone with root privileges remove these directories: `public_html.old.` please. [17:12:32] 10Cloud-VPS (Project-requests): Request creation of deorph VPS project - https://phabricator.wikimedia.org/T442348#12530082 (10fnegri) a:03fnegri [17:25:45] (03CR) 10Wandji collins: [C:03+2] Configure deployment files [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1367483 (owner: 10Wandji collins) [17:26:01] (03Merged) 10jenkins-bot: Configure deployment files [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1367483 (owner: 10Wandji collins) [18:03:00] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [18:04:30] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 10Pywikibot: pywikibot.org nightly dump is unavailable - https://phabricator.wikimedia.org/T444004#12530103 (10Xqt) [18:15:31] 10Tools: extreg-wos: Reflection is listed, but repo isn't available on Gerrit - https://phabricator.wikimedia.org/T295522#12530123 (10Legoktm) 05Open→03Declined The extreg-wos tool was sunset in December 2023 and won't be receiving any more updates. [18:16:01] 10Tools: extreg-wos.toolforge.org: Do not list empty repos as red - https://phabricator.wikimedia.org/T297196#12530125 (10Legoktm) 05Open→03Declined The extreg-wos tool was sunset in December 2023 and won't be receiving any more updates. [18:16:08] 10Tools: Make table at tool-extreg-wos sortable if possible - https://phabricator.wikimedia.org/T176873#12530127 (10Legoktm) 05Open→03Declined The extreg-wos tool was sunset in December 2023 and won't be receiving any more updates. [18:32:08] 10Tool-coverage: Create a proper frontend for coverage.toolforge.org - https://phabricator.wikimedia.org/T438645#12530147 (10Legoktm) 05Open→03Resolved Satisfied for now, we can add more reports easily if others have ideas. [18:36:33] 10Tool-extjsonuploader, 07Code-Health, 13Patch-Needs-Improvement, 07Test-Coverage: Display extension test coverage in infobox - https://phabricator.wikimedia.org/T220939#12530153 (10Legoktm) https://coverage.toolforge.org/ now has an API, e.g. https://coverage.toolforge.org/api/extensions/AbuseFilter If y... [19:15:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:20:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:31:59] FIRING: BackupCoverageDrop: volume backup coverage dropped on cloudbackup2004:9100 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Backup#BackupCoverageDrop - https://grafana.wikimedia.org/d/fisr2p9/wmcs-backup?viewPanel=21&var-kind=volume&var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DBackupCoverageDrop [20:12:17] 10Data-Services, 06Data-Persistence, 06DBA: Please include page_props on x4 - https://phabricator.wikimedia.org/T443973#12530223 (10Mike_Peel) 05Open→03Resolved a:03Mike_Peel Many thanks @Bugreporter - after a bit of iteration, that approach worked well! I'll mark this as resolved. [20:56:13] (03update) 10danyya: Draft: Resolve T434400 "Dumps ingester crashes because of too many prepared statements (2)" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/109 [20:57:35] (03update) 10danyya: Draft: Resolve T434400 "Dumps ingester crashes because of too many prepared statements (2)" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/109 [20:59:03] 10Data-Services, 06Data-Persistence, 06DBA: Please include page_props on x4 - https://phabricator.wikimedia.org/T443973#12530229 (10JJMC89) 05Resolved→03Invalid [21:04:41] (03update) 10danyya: Draft: Resolve T434400 "Dumps ingester crashes because of too many prepared statements (2)" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/109 [22:03:00] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [22:14:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:18:35] (03update) 10danyya: Draft: Resolve T434400 "Dumps ingester crashes because of too many prepared statements (2)" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/109 [22:24:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:04:03] (03update) 10danyya: Draft: Resolve T434400 "Dumps ingester crashes because of too many prepared statements (2)" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/109 [23:30:54] (03update) 10danyya: Draft: Resolve T434400 "Dumps ingester crashes because of too many prepared statements (2)" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/109 [23:31:59] FIRING: BackupCoverageDrop: volume backup coverage dropped on cloudbackup2004:9100 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Backup#BackupCoverageDrop - https://grafana.wikimedia.org/d/fisr2p9/wmcs-backup?viewPanel=21&var-kind=volume&var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DBackupCoverageDrop [23:36:09] (03PS1) 10Wandji collins: Update script to handle MySQL [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1367523 [23:42:12] (03CR) 10Wandji collins: [C:03+2] Update script to handle MySQL [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1367523 (owner: 10Wandji collins) [23:42:30] (03Merged) 10jenkins-bot: Update script to handle MySQL [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1367523 (owner: 10Wandji collins) [23:43:00] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse