[00:14:49] (03PS1) 10Wandji collins: Copy service template before stopping webservice. [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1367524 [00:18:24] (03CR) 10Wandji collins: [C:03+2] Copy service template before stopping webservice. [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1367524 (owner: 10Wandji collins) [00:18:43] (03Merged) 10jenkins-bot: Copy service template before stopping webservice. [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1367524 (owner: 10Wandji collins) [00:18:53] (03update) 10danyya: Draft: Resolve T434400 "Dumps ingester crashes because of too many prepared statements (2)" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/109 [00:28:01] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [00:39:28] (03PS1) 10Wandji collins: trim command to run install venv [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1367526 [00:44:44] (03CR) 10Wandji collins: [C:03+2] trim command to run install venv [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1367526 (owner: 10Wandji collins) [00:45:02] (03Merged) 10jenkins-bot: trim command to run install venv [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1367526 (owner: 10Wandji collins) [02:14:22] (03update) 10renovatebot: Update pnpm to v12.17.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/41 [02:14:38] (03update) 10renovatebot: Update pnpm to v12.18.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/41 [02:14:54] (03update) 10renovatebot: Update pnpm to v12.17.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/49 [02:14:55] (03update) 10renovatebot: Update pnpm to v12.18.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/49 [02:15:08] (03update) 10renovatebot: Update pnpm to v12.17.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/48 [02:15:11] (03update) 10renovatebot: Update pnpm to v12.18.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/48 [02:16:14] (03update) 10renovatebot: Update pnpm to v12.17.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/78 [02:16:15] (03update) 10renovatebot: Update pnpm to v12.18.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/78 [02:17:06] (03update) 10renovatebot: Update pnpm to v12.17.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/67 [02:17:11] (03update) 10renovatebot: Update pnpm to v12.18.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/67 [03:31:59] FIRING: BackupCoverageDrop: volume backup coverage dropped on cloudbackup2004:9100 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Backup#BackupCoverageDrop - https://grafana.wikimedia.org/d/fisr2p9/wmcs-backup?viewPanel=21&var-kind=volume&var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DBackupCoverageDrop [04:28:01] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [05:13:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:18:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:22:22] 10Tool-techcontribs: add some stats for mediawiki wiki and wikitech - https://phabricator.wikimedia.org/T443775#12530404 (10Chlod) p:05Triage→03Low I'm not against it, though working on Tech Contribs will require access to [Toolforge](https://wikitech.wikimedia.org/wiki/Help:Toolforge) as it hooks into numer... [05:40:24] 10Tool-Attribution-Generator, 10Tool-dataweaver: Att - https://phabricator.wikimedia.org/T444028 (10Mr.Likethat) 03NEW [05:40:34] 10Tool-Attribution-Generator, 10Tool-dataweaver: Att - https://phabricator.wikimedia.org/T444028#12530434 (10Mr.Likethat) a:03Mr.Likethat [06:31:10] (03open) 10snowfire: T431879: Mark new articles with a seedling icon, format age in green; some caching changes; add local-testing mode [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/34 [06:33:46] 10Tool-wikinewsie: New Wikipedia articles should be their own event type on Wikinewsie - https://phabricator.wikimedia.org/T431879#12530449 (10SnowFire) https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/34 for the current proposed change. [07:06:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:10:44] (03approved) 10bodhisattwa: docs: fix errors and improve local setup instructions in README [toolforge-repos/wsindex] - 10https://gitlab.wikimedia.org/toolforge-repos/wsindex/-/merge_requests/8 (owner: 10semajk) [07:11:03] (03merge) 10bodhisattwa: docs: fix errors and improve local setup instructions in README [toolforge-repos/wsindex] - 10https://gitlab.wikimedia.org/toolforge-repos/wsindex/-/merge_requests/8 (owner: 10semajk) [07:16:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:31:59] FIRING: BackupCoverageDrop: volume backup coverage dropped on cloudbackup2004:9100 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Backup#BackupCoverageDrop - https://grafana.wikimedia.org/d/fisr2p9/wmcs-backup?viewPanel=21&var-kind=volume&var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DBackupCoverageDrop [08:05:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:10:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:28:01] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [08:38:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:43:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:36:49] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 10Pywikibot: pywikibot.org nightly dump is unavailable - https://phabricator.wikimedia.org/T444004#12530541 (10Xqt) 05Open→03Resolved a:03Xqt Found: ` tools.pywikibot@tools-bastion-15:~$ toolforge webservice status DEPRECATED: 'php7.4' t... [11:09:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:19:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:26:30] !log tools.cluebot3 Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373456031 (https://github.com/cluebotng/component-configs/commits/9b75826d2873a10720e1149d8b0e5aecd8df070f) [11:26:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [11:26:50] !log tools.cluebotng-trainer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373423759 (https://github.com/cluebotng/component-configs/commits/0bc0cef490c8c2afb008ff906644dc7fb977bdb6) [11:26:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [11:32:00] FIRING: BackupCoverageDrop: volume backup coverage dropped on cloudbackup2004:9100 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Backup#BackupCoverageDrop - https://grafana.wikimedia.org/d/fisr2p9/wmcs-backup?viewPanel=21&var-kind=volume&var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DBackupCoverageDrop [11:53:34] 10Tool-delintbot: Error 503 on Wikitext to lint transformation REST endpoint caused bot process to crash - https://phabricator.wikimedia.org/T444036 (10Redmin) 03NEW [11:59:09] 10Tool-delintbot: Error 503 on Wikitext to lint transformation REST endpoint caused bot process to crash - https://phabricator.wikimedia.org/T444036#12530635 (10Redmin) p:05Triage→03High The last time something like this happened was when that endpoint (and at least another one) was throwing an error 504 ({T... [12:07:39] (03open) 10r4356th: Handle error 503s thrown by the REST API [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/50 (https://phabricator.wikimedia.org/T444036) [12:08:18] (03merge) 10r4356th: Handle error 503s thrown by the REST API [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/50 (https://phabricator.wikimedia.org/T444036) [12:19:53] 10Tool-delintbot, 13Patch-For-Review: Error 503 on Wikitext to lint transformation REST endpoint caused bot process to crash - https://phabricator.wikimedia.org/T444036#12530700 (10Redmin) Just filed {T444039}. [12:28:01] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [12:30:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:35:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:40:02] 10Tool-delintbot: Mention what was fixed in the edit summary - https://phabricator.wikimedia.org/T416068#12530707 (10Redmin) 05Open→03In progress [12:45:10] 14cloud-services-team (Kanban), 06Data-Engineering, 10Dumps-Generation, 10Wikimedia-Portals: dumps.wikimedia.org/other/pageviews/ appears to be stalled at 20210106-140000 - https://phabricator.wikimedia.org/T271561#12530711 (10Superyetkin) 05Resolved→03Open p:05Medium→03High The problem has just re... [12:58:54] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42373423759 (https://github.com/cluebotng/component-configs/commits/0bc0cef490c8c2afb008ff906644dc7fb977bdb6) [12:58:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [13:01:06] 10Toolforge, 06tools-platform-team: [harbor] quota is not configured for new tool - https://phabricator.wikimedia.org/T444043 (10DamianZaremba) 03NEW [13:13:26] 10Tool-athar: Add API Support - https://phabricator.wikimedia.org/T444045 (10Bonnjalal) 03NEW [13:13:55] 10Tool-athar: Add Tags and Campaings To Activities - https://phabricator.wikimedia.org/T444046 (10Bonnjalal) 03NEW [13:16:18] (03update) 10renovatebot: Update dependency jsdom to v31 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/68 [13:16:21] (03open) 10renovatebot: Update dependency jsdom to v31 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/68 [13:56:22] (03open) 10r4356th: Generate detailed edit summaries for all edits [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/51 (https://phabricator.wikimedia.org/T416068) [13:59:26] !log tools.cluebot3 Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42487487035 (https://github.com/cluebotng/component-configs/commits/626a79604f6624870ee9b519d9ab5ba4a75a9d9c) [13:59:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [13:59:29] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42487482308 (https://github.com/cluebotng/component-configs/commits/9dc5d5efa17f56825669f145bfabd6eb84c81180) [13:59:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [13:59:31] !log tools.cluebot-syncer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42487486901 (https://github.com/cluebotng/component-configs/commits/626a79604f6624870ee9b519d9ab5ba4a75a9d9c) [13:59:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [13:59:36] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42487482275 (https://github.com/cluebotng/component-configs/commits/9dc5d5efa17f56825669f145bfabd6eb84c81180) [13:59:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [14:01:24] !log tools.toolforge-functional-runner Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42487486795 (https://github.com/cluebotng/component-configs/commits/626a79604f6624870ee9b519d9ab5ba4a75a9d9c) [14:01:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.toolforge-functional-runner/SAL [14:01:26] !log tools.cluebotng-editsets Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42487486987 (https://github.com/cluebotng/component-configs/commits/626a79604f6624870ee9b519d9ab5ba4a75a9d9c) [14:01:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [14:01:28] !log tools.cluebot-documentation Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42487486856 (https://github.com/cluebotng/component-configs/commits/626a79604f6624870ee9b519d9ab5ba4a75a9d9c) [14:01:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-documentation/SAL [14:02:28] !log tools.cluebotng-staging Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42487482230 (https://github.com/cluebotng/component-configs/commits/9dc5d5efa17f56825669f145bfabd6eb84c81180) [14:02:28] !log tools.cluebotng-trainer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42487486913 (https://github.com/cluebotng/component-configs/commits/626a79604f6624870ee9b519d9ab5ba4a75a9d9c) [14:02:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [14:02:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [14:02:33] !log tools.cluebot3 Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42487541046 (https://github.com/cluebotng/component-configs/commits/e2bb466346d7884db14f89067638f5906d0fe96d) [14:02:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [14:03:07] !log tools.cluebot-syncer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42487569287 (https://github.com/cluebotng/component-configs/commits/1a8ef68cdb6e06e915b1d61d24c3f2defcf942d2) [14:03:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [14:03:19] !log tools.cluebotng-trainer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42487576357 (https://github.com/cluebotng/component-configs/commits/e851b03c8a8f5c65affe9e1655fbb7905114d0f0) [14:03:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [14:03:33] !log tools.cluebotng-monitoring Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42487477066 (https://github.com/cluebotng/component-configs/commits/02ed68cf95c253eadeba979b345d248dfb9477f7) [14:03:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [14:04:30] !log tools.cluebot-documentation Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42487576505 (https://github.com/cluebotng/component-configs/commits/e851b03c8a8f5c65affe9e1655fbb7905114d0f0) [14:04:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-documentation/SAL [14:05:05] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42487576438 (https://github.com/cluebotng/component-configs/commits/e851b03c8a8f5c65affe9e1655fbb7905114d0f0) [14:05:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [14:05:18] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42487535788 (https://github.com/cluebotng/component-configs/commits/036cdff1b35f023142582fcbc7eed1095df13f08) [14:05:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [14:05:20] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42487535872 (https://github.com/cluebotng/component-configs/commits/036cdff1b35f023142582fcbc7eed1095df13f08) [14:05:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [14:05:25] !log tools.cluebot3 Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42487576377 (https://github.com/cluebotng/component-configs/commits/e851b03c8a8f5c65affe9e1655fbb7905114d0f0) [14:05:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [14:05:58] !log tools.cluebotng-staging Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42487576386 (https://github.com/cluebotng/component-configs/commits/e851b03c8a8f5c65affe9e1655fbb7905114d0f0) [14:05:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [14:06:34] !log tools.cluebot-syncer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42487576409 (https://github.com/cluebotng/component-configs/commits/e851b03c8a8f5c65affe9e1655fbb7905114d0f0) [14:06:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [14:09:11] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42487576456 (https://github.com/cluebotng/component-configs/commits/e851b03c8a8f5c65affe9e1655fbb7905114d0f0) [14:09:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [14:10:16] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42487576379 (https://github.com/cluebotng/component-configs/commits/e851b03c8a8f5c65affe9e1655fbb7905114d0f0) [14:10:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [14:11:03] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42487576388 (https://github.com/cluebotng/component-configs/commits/e851b03c8a8f5c65affe9e1655fbb7905114d0f0) [14:11:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [14:27:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:37:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:45:21] (03update) 10r4356th: Generate detailed edit summaries for all edits [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/51 (https://phabricator.wikimedia.org/T416068) [14:48:48] (03update) 10raymond-ndibe: _check_for_matching_build - check the build's builder is current [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/213 (https://phabricator.wikimedia.org/T442364) (owner: 10damian) [14:54:09] (03update) 10r4356th: Generate detailed edit summaries for all edits [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/51 (https://phabricator.wikimedia.org/T416068) [14:55:10] (03update) 10r4356th: Generate detailed edit summaries for all edits [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/51 (https://phabricator.wikimedia.org/T416068) [15:02:00] RESOLVED: BackupCoverageDrop: volume backup coverage dropped on cloudbackup2004:9100 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Backup#BackupCoverageDrop - https://grafana.wikimedia.org/d/fisr2p9/wmcs-backup?viewPanel=21&var-kind=volume&var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DBackupCoverageDrop [15:05:14] 10Tool-inteGraality: Dashboard - inteGraality didn't write in wiki - https://phabricator.wikimedia.org/T444049 (10Bricetofly) 03NEW [15:05:41] (03update) 10r4356th: Generate detailed edit summaries for all edits and make sure replacing center tags around tables does not produce invalid output when the table has no style attribute [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/51 (https://phabricator.wikimedia.org/T416068) [15:07:06] 10Tool-inteGraality: Dashboard - inteGraality didn't write in wiki - https://phabricator.wikimedia.org/T444049#12530883 (10Bricetofly) I worked but with a lot of delay... https://www.wikidata.org/wiki/User:Bricetofly/Streets-of-Biel [15:08:45] (03update) 10r4356th: Generate detailed edit summaries for all edits and make sure replacing center tags around tables does not produce invalid output when the table has no style attribute [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/51 (https://phabricator.wikimedia.org/T416068) [15:10:24] (03update) 10r4356th: Generate detailed edit summaries for all edits and make sure replacing center tags around tables does not produce invalid output when the table has no style attribute [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/51 (https://phabricator.wikimedia.org/T416068) [15:11:59] (03merge) 10r4356th: Generate detailed edit summaries for all edits and make sure replacing center tags around tables does not produce invalid output when the table has no style attribute [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/51 (https://phabricator.wikimedia.org/T416068) [15:12:25] 10Tool-delintbot: Mention what was fixed in the edit summary - https://phabricator.wikimedia.org/T416068#12530887 (10Redmin) 05In progress→03Resolved a:03Redmin [15:13:11] 10Tool-delintbot: Error 503 on Wikitext to lint transformation REST endpoint caused bot process to crash - https://phabricator.wikimedia.org/T444036#12530893 (10Redmin) 05Open→03Resolved a:03Redmin [15:15:07] 10Toolforge, 06tools-platform-team: [components-api] requires re-formatting generated code - https://phabricator.wikimedia.org/T443675#12530896 (10Raymond_Ndibe) p:05Triage→03Low [15:15:37] 10Toolforge, 06tools-platform-team: [harbor] Review default quota - https://phabricator.wikimedia.org/T444050 (10DamianZaremba) 03NEW [15:17:34] 10Toolforge, 06tools-platform-team: [components-api] requires re-formatting generated code - https://phabricator.wikimedia.org/T443675#12530916 (10Raymond_Ndibe) running tox/pre-commit (which is something you have to run anyways after making code changes) handles this for you no? granted it involves typing an... [15:18:37] 10Toolforge, 06tools-platform-team: [harbor] quota is not configured for new tool - https://phabricator.wikimedia.org/T444043#12530918 (10Raymond_Ndibe) Harbor cronjobs were not running before this. This should be resolved now [15:18:58] 10Toolforge, 06tools-platform-team: [harbor] quota is not configured for new tool - https://phabricator.wikimedia.org/T444043#12530919 (10Raymond_Ndibe) 05Open→03Resolved [15:20:00] 10Toolforge, 06tools-platform-team: [harbor] quota is not configured for new tool - https://phabricator.wikimedia.org/T444043#12530920 (10DamianZaremba) Looks good ` tools.cluebot-documentation@tools-bastion-15:~$ toolforge build quota Registry =================== Storage ----------- Available 563.82Mi... [15:22:58] (03update) 10raymond-ndibe: _check_for_matching_build - check the build's builder is current [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/213 (https://phabricator.wikimedia.org/T442364) (owner: 10damian) [15:23:06] 10Toolforge (Quota-requests): Request increased build quota for cluebot3 Toolforge tool - https://phabricator.wikimedia.org/T444051 (10DamianZaremba) 03NEW [15:23:27] 10Toolforge (Quota-requests): Request increased build quota for cluebotng-staging Toolforge tool - https://phabricator.wikimedia.org/T444052 (10DamianZaremba) 03NEW [15:23:30] (03approved) 10raymond-ndibe: Refactor outdated check [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/172 (owner: 10damian) [15:24:38] (03unapproved) 10raymond-ndibe: Refactor outdated check [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/172 (owner: 10damian) [15:25:13] 10Toolforge (Quota-requests): Request increased build quota for cluebot-documentation Toolforge tool - https://phabricator.wikimedia.org/T444053 (10DamianZaremba) 03NEW [15:25:49] 10Toolforge (Quota-requests): Request increased build quota for cluebotng-editsets Toolforge tool - https://phabricator.wikimedia.org/T444054 (10DamianZaremba) 03NEW [15:26:28] 10Toolforge (Quota-requests): Request increased build quota for cluebotng-trainer Toolforge tool - https://phabricator.wikimedia.org/T444055 (10DamianZaremba) 03NEW [15:31:46] 10Tool-delintbot: Document how to operate the bot - https://phabricator.wikimedia.org/T416070#12531004 (10Redmin) [15:32:16] 10Tool-delintbot, 07Documentation: Document how to operate the bot - https://phabricator.wikimedia.org/T416070#12531005 (10Redmin) [15:32:53] 10Toolforge, 06tools-platform-team: [harbor] quota is not configured for new tool - https://phabricator.wikimedia.org/T444043#12531008 (10taavi) >>! In T444043#12530918, @Raymond_Ndibe wrote: > Harbor cronjobs were not running before this. This should be resolved now Why was this? Why is there no alert to... [15:35:29] (03update) 10damian: _check_for_matching_build - check the build's builder is current [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/213 (https://phabricator.wikimedia.org/T442364) [15:36:00] (03update) 10damian: _check_for_matching_build - check the build's builder is current [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/213 (https://phabricator.wikimedia.org/T442364) [15:37:17] (03update) 10raymond-ndibe: _check_for_matching_build - check the build's builder is current [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/213 (https://phabricator.wikimedia.org/T442364) (owner: 10damian) [15:56:29] (03open) 10raymond-ndibe: docs: specify python version for data model generation [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/215 [15:56:35] (03update) 10raymond-ndibe: docs: specify python version for data model generation [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/215 [15:59:53] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42487576388 (https://github.com/cluebotng/component-configs/commits/e851b03c8a8f5c65affe9e1655fbb7905114d0f0) [15:59:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [16:01:50] (03update) 10raymond-ndibe: Refactor outdated check [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/172 (https://phabricator.wikimedia.org/T442364) (owner: 10damian) [16:04:36] (03update) 10raymond-ndibe: Refactor outdated check [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/172 (https://phabricator.wikimedia.org/T442364) (owner: 10damian) [16:09:20] (03update) 10damian: Refactor outdated check [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/172 (https://phabricator.wikimedia.org/T442364) [16:09:32] (03update) 10damian: Refactor outdated check [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/172 (https://phabricator.wikimedia.org/T442364) [16:10:27] (03update) 10raymond-ndibe: Revert "builds-api: use different tags for latest/deprecated" [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1503 (https://phabricator.wikimedia.org/T442364) (owner: 10damian) [16:11:27] (03approved) 10raymond-ndibe: Revert "builds-api: use different tags for latest/deprecated" [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1503 (https://phabricator.wikimedia.org/T442364) (owner: 10damian) [16:12:23] (03update) 10raymond-ndibe: Refactor outdated check [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/172 (https://phabricator.wikimedia.org/T442364) (owner: 10damian) [16:13:04] (03approved) 10raymond-ndibe: Refactor outdated check [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/172 (https://phabricator.wikimedia.org/T442364) (owner: 10damian) [16:15:50] (03update) 10raymond-ndibe: tests: test the parsing of components [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/205 (https://phabricator.wikimedia.org/T362077) [16:25:30] 10Toolforge, 06tools-platform-team: [components-api] requires re-formatting generated code - https://phabricator.wikimedia.org/T443675#12531042 (10DamianZaremba) >>! In T443675#12530916, @Raymond_Ndibe wrote: > running tox/pre-commit (which is something you have to run anyways after making code changes) handle... [16:26:43] (03merge) 10raymond-ndibe: Refactor outdated check [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/172 (https://phabricator.wikimedia.org/T442364) (owner: 10damian) [16:27:20] (03approved) 10raymond-ndibe: _check_for_matching_build - check the build's builder is current [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/213 (https://phabricator.wikimedia.org/T442364) (owner: 10damian) [16:27:32] (03merge) 10raymond-ndibe: _check_for_matching_build - check the build's builder is current [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/213 (https://phabricator.wikimedia.org/T442364) (owner: 10damian) [16:28:01] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [16:28:48] (03update) 10raymond-ndibe: tests: test the parsing of components [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/205 (https://phabricator.wikimedia.org/T362077) [16:30:43] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.238-20261108162741-f1ccddf0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1506 (https://phabricator.wikimedia.org/T442364) [16:32:52] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-api: bump to 0.0.227-20261108162653-77edb3a7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1507 (https://phabricator.wikimedia.org/T442364) [16:32:59] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-api: bump to 0.0.227-20261108162653-77edb3a7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1507 (https://phabricator.wikimedia.org/T442364) [17:02:55] (03approved) 10raymond-ndibe: builds-api: bump to 0.0.227-20261108162653-77edb3a7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1507 (https://phabricator.wikimedia.org/T442364) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:03:03] (03merge) 10raymond-ndibe: builds-api: bump to 0.0.227-20261108162653-77edb3a7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1507 (https://phabricator.wikimedia.org/T442364) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:03:36] (03update) 10raymond-ndibe: components-api: bump to 0.0.238-20261108162741-f1ccddf0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1506 (https://phabricator.wikimedia.org/T442364) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:06:29] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [builds-api] returns incorrect parameter values for use_latest_versions & use_deprecated_versions causing [components-api] to not re-use existing builds - https://phabricator.wikimedia.org/T442364#12531103 (10DamianZaremba) Looks good in production for... [17:25:37] (03approved) 10raymond-ndibe: components-api: bump to 0.0.238-20261108162741-f1ccddf0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1506 (https://phabricator.wikimedia.org/T442364) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:25:50] (03merge) 10raymond-ndibe: components-api: bump to 0.0.238-20261108162741-f1ccddf0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1506 (https://phabricator.wikimedia.org/T442364) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:27:48] !log tools.cluebotng-trainer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42504935665 (https://github.com/cluebotng/component-configs/commits/e7413dba8702990176d7157963daf69127d894fc) [17:27:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [17:29:45] !log tools.cluebotng-trainer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42505001795 (https://github.com/cluebotng/component-configs/commits/3d315d295fbd9ddb713c5f5abe3e6e921e02f904) [17:29:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [17:40:01] 10Toolforge, 06tools-platform-team: [components-api] gets stuck during deployment (?) - https://phabricator.wikimedia.org/T444060 (10DamianZaremba) 03NEW [18:32:15] 10Tool-delintbot: Mention what was fixed in the edit summary - https://phabricator.wikimedia.org/T416068#12531166 (10Redmin) Note: I updated the logic in commit [[ https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/commit/0d73a43660b0d1110ae2a5b719c884e53b3139eb | 0d73a43660b0d1110ae2a5b719c884e53b3139... [18:59:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:09:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:16:52] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/40 [19:56:16] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-etcd05 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436473#12531244 (10Southparkfan) The process was a bit more involved than I had hoped. For future reference, if you wish to replace this host with a ne... [20:13:02] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Replace deployment-etcd05 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436473#12531265 (10Southparkfan) 05In progress→03Resolved [20:27:53] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Replace deployment-zookeeper-3 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436481#12531287 (10Southparkfan) See T428495, T435405 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/1309257 for m... [20:28:01] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [20:38:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:08:01] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [21:19:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:24:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:36:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:41:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:43:01] FIRING: [2x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [21:48:01] RESOLVED: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [22:01:11] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:30:28] (03update) 10danyya: Draft: Resolve T434400 "Dumps ingester crashes because of too many prepared statements (2)" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/109 [22:42:59] (03update) 10raymond-ndibe: tests: test the parsing of components [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/205 (https://phabricator.wikimedia.org/T362077) [22:45:49] (03update) 10raymond-ndibe: tests: test the parsing of components [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/205 (https://phabricator.wikimedia.org/T362077) [22:56:10] (03open) 10raymond-ndibe: models: reject unknown fields in the tool config [repos/cloud/toolforge/components-api] (add_component_parsing_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/216 (https://phabricator.wikimedia.org/T362077) [23:02:44] (03update) 10raymond-ndibe: models: reject unknown fields in the tool config [repos/cloud/toolforge/components-api] (add_component_parsing_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/216 (https://phabricator.wikimedia.org/T362077) [23:27:18] (03update) 10raymond-ndibe: models: use a type statement for the ComponentInfo alias [repos/cloud/toolforge/components-api] (add_component_parsing_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/206 (https://phabricator.wikimedia.org/T362077) [23:27:42] (03update) 10raymond-ndibe: models: use a type statement for the ComponentInfo alias [repos/cloud/toolforge/components-api] (forbid_unknown_config_fields) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/206 (https://phabricator.wikimedia.org/T362077) [23:31:33] (03update) 10raymond-ndibe: config: set component_type in the generated config [repos/cloud/toolforge/components-api] (use_type_statement_for_component_info) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/207 (https://phabricator.wikimedia.org/T362077) [23:31:47] (03update) 10raymond-ndibe: config: warn about components without component_type [repos/cloud/toolforge/components-api] (set_component_type_in_generated_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T362077) [23:36:00] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [builds-api] returns incorrect parameter values for use_latest_versions & use_deprecated_versions causing [components-api] to not re-use existing builds - https://phabricator.wikimedia.org/T442364#12531473 (10Raymond_Ndibe) > Thanks for helping get this... [23:42:39] (03update) 10raymond-ndibe: feat: add webservice component type support [repos/cloud/toolforge/components-api] (warn_on_missing_component_type) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/199 (https://phabricator.wikimedia.org/T362077) [23:52:57] (03update) 10raymond-ndibe: feat: add webservice component type support [repos/cloud/toolforge/components-api] (warn_on_missing_component_type) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/199 (https://phabricator.wikimedia.org/T362077)