[01:54:36] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#11943394 (10bd808) The bots are back. {F83049286,size=full} [01:55:23] 10Beta-Cluster-Infrastructure: Beta cluster down: Error: 502, Backend fetch failed - https://phabricator.wikimedia.org/T426831#11943399 (10bd808) β†’14Duplicate dup:03T420833 [01:55:24] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#11943396 (10bd808) [01:58:22] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#11943402 (10bd808) p:05Triageβ†’03High [02:10:07] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#11943406 (10bd808) The current traffic pattern is full of the "2 requests per /24 network" pattern. Range blocks will need to be vast again to make much... [05:33:01] 10Scap, 10MediaWiki-Release-Tools, 13Patch-For-Review: PHP Fatal error: Cannot redeclare wikidiff2_do_diff() - https://phabricator.wikimedia.org/T340862#11943510 (10Legoktm) a:05Legoktmβ†’03None [05:41:59] (03PS1) 10Vaughn Walters: jjb: [catalyst-daily] Convert to job-template [integration/config] - 10https://gerrit.wikimedia.org/r/1290482 (https://phabricator.wikimedia.org/T426919) [05:57:41] (03PS3) 10Vaughn Walters: jjb: [catalyst-daily] Convert to job-template [integration/config] - 10https://gerrit.wikimedia.org/r/1290482 (https://phabricator.wikimedia.org/T426919) [06:27:31] FIRING: [4x] ProbeDown: Service gerrit2003:29418 has failed probes (tcp_gerrit_ssh_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [06:27:37] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T426932 (10phaultfinder) 03NEW [06:28:52] PROBLEM - Host gerrit2003 is DOWN: PING CRITICAL - Packet loss = 100% [06:30:32] RECOVERY - Host gerrit2003 is UP: PING OK - Packet loss = 0%, RTA = 31.59 ms [06:32:31] RESOLVED: [4x] ProbeDown: Service gerrit2003:29418 has failed probes (tcp_gerrit_ssh_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [06:33:28] PROBLEM - Host contint1002 is DOWN: PING CRITICAL - Packet loss = 100% [06:34:58] RECOVERY - Host contint1002 is UP: PING OK - Packet loss = 0%, RTA = 0.37 ms [06:35:22] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T426932#11943583 (10phaultfinder) [06:37:33] ah hashar I missed your last message yesterday sorry, yeah it was a reboot for the kernel upgrade, it's done now [07:10:09] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T426932#11943607 (10ABran-WMF) 05Openβ†’03Resolved a:03ABran-WMF expected, linked to kernel reboot [07:10:19] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - contint1002:1443 has failed probes (http_integration_wikimedia_org_ip4) - https://phabricator.wikimedia.org/T426932#11943610 (10ABran-WMF) [07:58:45] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: CAPI: Handle invalid names gracefully - https://phabricator.wikimedia.org/T426843#11943710 (10jnuche) 05Openβ†’03Resolved In prod [08:05:13] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#11943731 (10ABran-WMF) [08:05:51] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - contint1002:1443 has failed probes (http_integration_wikimedia_org_ip4) - https://phabricator.wikimedia.org/T426932#11943733 (10A_smart_kitten) (side-note: interesting how @phaultfinder works with the current alert-names -- this was original... [08:10:55] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#11943755 (10ABran-WMF) [08:49:21] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#11943953 (10ABran-WMF) [09:06:01] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06cloud-services-team, 06collaboration-services: webservice-cli package deb gitlab CI job went from 9 minutes to 27 minutes - https://phabricator.wikimedia.org/T426827#11943992 (10LSobanski) [10:35:25] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06cloud-services-team, 06collaboration-services: webservice-cli package deb gitlab CI job went from 9 minutes to 27 minutes - https://phabricator.wikimedia.org/T426827#11944204 (10fgiunchedi) The problem seems to be `fakeroot` and a huge `ulimit -n`,... [10:39:35] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06cloud-services-team, 06collaboration-services: webservice-cli package deb gitlab CI job went from 9 minutes to 27 minutes - https://phabricator.wikimedia.org/T426827#11944222 (10fgiunchedi) To test this theory I changed `webservice-cli` gitlab-ci t... [10:47:54] arnaudb: congrats on the reboots :) [11:02:59] Hola, ΒΏalguien sabe por quΓ© no me funciona "recheck"? [11:03:32] Hi, does anyone know why "recheck" isn't working for me? [11:03:36] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/OAuth/+/1290105 [11:04:05] (I'm KiwiTheMember) [11:05:53] Guest31, maybe it's because you're not yet in https://www.mediawiki.org/wiki/Continuous_integration/Allow_list [11:07:42] Also see https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Comment_commands [11:08:42] ohhhh, ok [11:08:59] xSavitar: can you do it for me? [11:10:05] anyway, this is my first patch, I guess it's too early to ask to be whitelisted [11:10:30] So only if you can send a recheck on my patch [11:10:51] You can propose a patch to add yourself and maybe someone will come around and approve it. [11:11:26] no, I'll wait with it [11:11:38] I trigger the CI jobs for you. [11:11:44] *triggered [11:11:54] Thank you !!!! [11:12:05] And thanks for the patch by the way. [11:12:26] β™₯ [11:16:44] xSavitar: If I can bother you again, I will [11:27:43] Sure, no worries. [12:04:44] (03open) 10kbach: chore: Fix validation duplicates [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/2 (https://phabricator.wikimedia.org/T426825) [12:06:29] (03update) 10kbach: Draft: chore: Fix validation duplicates [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/2 (https://phabricator.wikimedia.org/T426825) [12:12:27] (03update) 10kbach: Draft: chore: Fix validation duplicates [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/2 (https://phabricator.wikimedia.org/T426825) [12:17:09] (03update) 10kbach: chore: Fix validation duplicates [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/2 (https://phabricator.wikimedia.org/T426825) [12:18:37] (03CR) 10Jforrester: "Neatly done." [integration/config] - 10https://gerrit.wikimedia.org/r/1290482 (https://phabricator.wikimedia.org/T426919) (owner: 10Vaughn Walters) [13:08:13] (03open) 10seanleong-wmde: fix: mwscript addsite normalize the arguments different than running addSite.php directly. [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/650 [13:11:40] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T423912#11944807 (10Novem_Linguae) {T426960} is a regression that I just noticed. Maybe caused by this week's train? [13:20:10] 10Beta-Cluster-Infrastructure, 06Traffic, 13Patch-For-Review: No Puppet resources found on instance deployment-cache-upload08 on project deployment-prep - https://phabricator.wikimedia.org/T426822#11944850 (10ssingh) 05Openβ†’03Resolved Should now be resolved; @bd808 already cherry-picked but this has... [13:27:23] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for RenovateBot - https://phabricator.wikimedia.org/T426964 (10RenovateBot) 03NEW [13:27:52] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for RenovateBot - https://phabricator.wikimedia.org/T426964#11944914 (10Count_Count) I confirm that this is indeed my alt account. [13:33:50] just FYI to releng, a request on [[mw:Talk:Phabricator/Help]] that seems like it might've needed actioning (and/or a declination, as applicable) from someone with Phab CLI access was archived without a reply: https://www.mediawiki.org/w/index.php?title=Talk%3APhabricator%2FHelp&oldid=prev&diff=8383796 [14:00:58] 10WikimediaDebug, 10Observability-Metrics: Upgrade XHGui from 0.21.3 to latest (0.23.6) - https://phabricator.wikimedia.org/T426368#11945086 (10colewhite) a:03colewhite [14:13:34] (03update) 10dhardy: PersonalDashboard: Add example pages [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/106 (https://phabricator.wikimedia.org/T424663) [14:52:11] 10Phabricator, 10Phabricator Antivandalism Extension, 06Release-Engineering-Team (Priority Backlog πŸ“₯): Make it more discoverable that a Phab user account triggered AVA countermeasures - https://phabricator.wikimedia.org/T426972 (10Aklapper) 03NEW p:05Triageβ†’03Low [15:12:52] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T423912#11945436 (10hashar) 05Openβ†’03Resolved >>! In T423912#11944807, @Novem_Linguae wrote: > {T426960} is a regression that I jus... [15:16:12] 10Phabricator, 10Phabricator Antivandalism Extension, 06Release-Engineering-Team (Priority Backlog πŸ“₯): Make it more discoverable that a Phab user account triggered AVA countermeasures - https://phabricator.wikimedia.org/T426972#11945463 (10Aklapper) A Conduit call felt nice but after trying a few things (see... [16:10:50] (03CR) 10Hashar: [C:03+2] "I apologize for the delay. I have deployed the job and would deploy the Zuul configuration as soon as it has merged ( I4e07ff36c0061844c6b" [integration/config] - 10https://gerrit.wikimedia.org/r/1288021 (owner: 10Wandji collins) [16:10:54] (03CR) 10Hashar: [C:03+2] zuul: add the tool wdaudiolex-fe for ci testing [integration/config] - 10https://gerrit.wikimedia.org/r/1287509 (https://phabricator.wikimedia.org/T426366) (owner: 10Wandji collins) [16:11:58] (03PS6) 10Jforrester: Zuul: [labs/tools/wdaudiolex-fe] Add custom CI testing [integration/config] - 10https://gerrit.wikimedia.org/r/1287509 (https://phabricator.wikimedia.org/T426366) (owner: 10Wandji collins) [16:12:04] (03CR) 10Jforrester: [C:03+2] "…" [integration/config] - 10https://gerrit.wikimedia.org/r/1287509 (https://phabricator.wikimedia.org/T426366) (owner: 10Wandji collins) [16:12:29] (03CR) 10Hashar: [C:04-2] jjb: Provide labs-tools-wdaudiolex-fe [integration/config] - 10https://gerrit.wikimedia.org/r/1288021 (owner: 10Wandji collins) [16:13:35] (03CR) 10CI reject: [V:04-1] Zuul: [labs/tools/wdaudiolex-fe] Add custom CI testing [integration/config] - 10https://gerrit.wikimedia.org/r/1287509 (https://phabricator.wikimedia.org/T426366) (owner: 10Wandji collins) [16:13:36] (03CR) 10CI reject: [V:04-1] Zuul: [labs/tools/wdaudiolex-fe] Add custom CI testing [integration/config] - 10https://gerrit.wikimedia.org/r/1287509 (https://phabricator.wikimedia.org/T426366) (owner: 10Wandji collins) [16:13:57] hashar: Oops, it's broken? [16:15:31] (03CR) 10Hashar: [C:04-2] "Actually I have had a look at `labs/tools/wdaudiolex-fe`Β which has a tox.ini with:" [integration/config] - 10https://gerrit.wikimedia.org/r/1288021 (owner: 10Wandji collins) [16:19:30] Ha. [16:24:19] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#11945726 (10bd808) >>! In T420833#11943406, @bd808 wrote: > That is 11,118 distinct /24 networks each sending 2 requests in the last ~2 hours. 19,091 n... [16:24:52] (03Abandoned) 10Hashar: jjb: Provide labs-tools-wdaudiolex-fe [integration/config] - 10https://gerrit.wikimedia.org/r/1288021 (owner: 10Wandji collins) [16:27:37] (03PS7) 10Hashar: Zuul: [labs/tools/wdaudiolex-fe] Add Node 24 generic job [integration/config] - 10https://gerrit.wikimedia.org/r/1287509 (https://phabricator.wikimedia.org/T426366) (owner: 10Wandji collins) [16:27:51] (03CR) 10Hashar: [C:03+2] Zuul: [labs/tools/wdaudiolex-fe] Add Node 24 generic job [integration/config] - 10https://gerrit.wikimedia.org/r/1287509 (https://phabricator.wikimedia.org/T426366) (owner: 10Wandji collins) [16:28:47] 10Phabricator, 10Phabricator Antivandalism Extension, 06Release-Engineering-Team (Priority Backlog πŸ“₯): Make it more discoverable that a Phab user account triggered AVA countermeasures - https://phabricator.wikimedia.org/T426972#11945733 (10Aklapper) ...and the attempt via a Transaction, posting that here onl... [16:29:04] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for RenovateBot - https://phabricator.wikimedia.org/T426964#11945735 (10Anoop) added trusted-contributors list https://phabricator.wikimedia.org/feed/7642382135108196699/ , Gitlab account should be activated in few mi... [16:31:00] (03Merged) 10jenkins-bot: Zuul: [labs/tools/wdaudiolex-fe] Add Node 24 generic job [integration/config] - 10https://gerrit.wikimedia.org/r/1287509 (https://phabricator.wikimedia.org/T426366) (owner: 10Wandji collins) [16:33:14] !log Reloaded Zuul to enable Node24 CI job for `labs/tools/wdaudiolex-fe` # T426366 [16:33:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:33:16] T426366: FE - Add Gerrit Code Review workflow for Front-end repository - https://phabricator.wikimedia.org/T426366 [16:36:23] 10Phabricator, 10Phabricator Antivandalism Extension, 06Release-Engineering-Team (Priority Backlog πŸ“₯): Make it more discoverable that a Phab user account triggered AVA countermeasures - https://phabricator.wikimedia.org/T426972#11945745 (10Aklapper) And resulting clean patch: ` diff --git a/src/herald/AntiVa... [16:40:50] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for RenovateBot - https://phabricator.wikimedia.org/T426964#11945747 (10Anoop) 05Openβ†’03Resolved a:03Anoop [16:43:41] 10Continuous-Integration-Config, 10WDAudioLex, 13Patch-For-Review: FE - Add Gerrit Code Review workflow for Front-end repository - https://phabricator.wikimedia.org/T426366#11945751 (10hashar) I have deployed the CI configuration that would enable a job based on Node 24 which runs `npm ci && npm run test`.... [18:51:55] 10Phabricator, 10Phabricator Antivandalism Extension, 06Release-Engineering-Team (Priority Backlog πŸ“₯): Make it more discoverable that a Phab user account triggered AVA countermeasures - https://phabricator.wikimedia.org/T426972#11946124 (10Bugreporter) See also T102576#9949987 [19:49:52] (03approved) 10aghirelli: chore: Fix validation duplicates [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/2 (https://phabricator.wikimedia.org/T426825) (owner: 10kbach) [19:50:17] (03merge) 10aghirelli: chore: Fix validation duplicates [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/2 (https://phabricator.wikimedia.org/T426825) (owner: 10kbach) [20:04:07] (03approved) 10aghirelli: oas3-valid-media-example: Disable rule due to false positive errors generated for valid dates [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/1 (https://phabricator.wikimedia.org/T425952) (owner: 10milazg) [20:04:14] (03merge) 10aghirelli: oas3-valid-media-example: Disable rule due to false positive errors generated for valid dates [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/1 (https://phabricator.wikimedia.org/T425952) (owner: 10milazg) [21:49:55] (03CR) 10Jforrester: [C:03+2] jjb: [catalyst-daily] Convert to job-template [integration/config] - 10https://gerrit.wikimedia.org/r/1290482 (https://phabricator.wikimedia.org/T426919) (owner: 10Vaughn Walters) [21:51:44] (03Merged) 10jenkins-bot: jjb: [catalyst-daily] Convert to job-template [integration/config] - 10https://gerrit.wikimedia.org/r/1290482 (https://phabricator.wikimedia.org/T426919) (owner: 10Vaughn Walters) [22:11:09] (03PS1) 10Matthias Mullie: Add BetaFeatures dependency for MMV [integration/config] - 10https://gerrit.wikimedia.org/r/1290943 (https://phabricator.wikimedia.org/T426225) [22:15:09] (03PS1) 10Jdlrobson: Add BetaFeatures to phan dependencies for MMV [integration/config] - 10https://gerrit.wikimedia.org/r/1290944 [22:15:28] (03PS2) 10Jdlrobson: Add BetaFeatures to phan dependencies for MMV [integration/config] - 10https://gerrit.wikimedia.org/r/1290944 (https://phabricator.wikimedia.org/T426225) [22:16:38] (03PS3) 10Jdlrobson: Add BetaFeatures to phan dependencies for MMV [integration/config] - 10https://gerrit.wikimedia.org/r/1290944 (https://phabricator.wikimedia.org/T426225) [22:17:35] (03Abandoned) 10Matthias Mullie: Add BetaFeatures dependency for MMV [integration/config] - 10https://gerrit.wikimedia.org/r/1290943 (https://phabricator.wikimedia.org/T426225) (owner: 10Matthias Mullie) [22:31:18] (03CR) 10Reedy: "Do you want BetaFeatures available in non phan jobs?" [integration/config] - 10https://gerrit.wikimedia.org/r/1290944 (https://phabricator.wikimedia.org/T426225) (owner: 10Jdlrobson) [22:54:27] (03CR) 10Matthias Mullie: "IIRC think it was already in place there" [integration/config] - 10https://gerrit.wikimedia.org/r/1290944 (https://phabricator.wikimedia.org/T426225) (owner: 10Jdlrobson) [23:20:36] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: zuul2002 (ganeti VM) - up but not reachable - https://phabricator.wikimedia.org/T426693#11946959 (10Dzahn) {P92814} I did all kinds of debugging, checking IP, traceroute, routing table, firewall on the affected machine and the g... [23:21:06] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: zuul2002 (ganeti VM) - up but not reachable - https://phabricator.wikimedia.org/T426693#11946960 (10Dzahn) 05Openβ†’03Resolved