[00:18:47] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission 💀), 10SRE, 10serviceops, and 2 others: replacement for gerrit2001, decom gerrit2001 - https://phabricator.wikimedia.org/T243027 (10Dzahn) [00:44:55] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission 💀), 10Patch-For-Review: Bring up gerrit2002 - https://phabricator.wikimedia.org/T313250 (10Dzahn) [00:46:17] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission 💀), 10Patch-For-Review: Bring up gerrit2002 - https://phabricator.wikimedia.org/T313250 (10Dzahn) https://gerrit-replica-new.wikimedia.org/ is now a working webserver (we got gerrit service to keep running and not flap and fixed a cert issue) [00:47:50] gerrit is currently replicating to new replica gerrit2002. laters [00:59:23] (03PS1) 10Krinkle: wmf-config-wg-vars: Add search concurrency and skip HISTORY file [tools/code-utils] - 10https://gerrit.wikimedia.org/r/820251 [00:59:46] (03CR) 10Krinkle: [C: 03+2] wmf-config-wg-vars: Add search concurrency and skip HISTORY file [tools/code-utils] - 10https://gerrit.wikimedia.org/r/820251 (owner: 10Krinkle) [01:00:26] (03Merged) 10jenkins-bot: wmf-config-wg-vars: Add search concurrency and skip HISTORY file [tools/code-utils] - 10https://gerrit.wikimedia.org/r/820251 (owner: 10Krinkle) [01:43:53] (03PS1) 10Krinkle: wmf-config-wg-vars: Fix 'tests/' skipper [tools/code-utils] - 10https://gerrit.wikimedia.org/r/820253 [01:44:09] (03CR) 10Krinkle: [C: 03+2] wmf-config-wg-vars: Fix 'tests/' skipper [tools/code-utils] - 10https://gerrit.wikimedia.org/r/820253 (owner: 10Krinkle) [01:44:46] (03Merged) 10jenkins-bot: wmf-config-wg-vars: Fix 'tests/' skipper [tools/code-utils] - 10https://gerrit.wikimedia.org/r/820253 (owner: 10Krinkle) [02:38:58] Project beta-scap-sync-world build #62561: 04FAILURE in 4 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/62561/ [02:52:35] Project beta-scap-sync-world build #62562: 04STILL FAILING in 7 min 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/62562/ [02:57:17] Project beta-scap-sync-world build #62563: 04STILL FAILING in 2 min 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/62563/ [03:07:57] Project beta-scap-sync-world build #62564: 04STILL FAILING in 3 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/62564/ [03:19:32] Project beta-scap-sync-world build #62565: 04STILL FAILING in 4 min 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/62565/ [03:26:15] Yippee, build fixed! [03:26:16] Project beta-scap-sync-world build #62566: 09FIXED in 1 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/62566/ [04:24:07] dancy: _joe_ sorry for the ping, but noticing scap-pull on mwmaint still halting on sudo prompt. not urgent, but just want to make sure it's tracked somewhere so it's not me keeping track of it on a post-it note. existing task is fine if there is one or perhaps file one :) [04:40:22] 10Project-Admins: Archive unused #Design-Innovation-Team project tag - https://phabricator.wikimedia.org/T312831 (10Aklapper) [04:40:27] 10Project-Admins: Archive unused #Design-Innovation-Team project tag - https://phabricator.wikimedia.org/T312831 (10Aklapper) 05Open→03Resolved a:03Aklapper Thanks! [05:36:16] <_joe_> Krinkle: you might have noticed we have a huge maintenance ongoing, I have submitted patches two days ago but yesterday I didn't literally have the time to look at them better [07:16:43] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for mfossati - https://phabricator.wikimedia.org/T313812 (10ArielGlenn) Missed you again today but there were also no patches scheduled for deployment. Third time's a charm? :-) [09:19:52] <_joe_> Krinkle: that is fixed now [09:25:16] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for mfossati - https://phabricator.wikimedia.org/T313812 (10mfossati) My sincere apologies @ArielGlenn . I'll reschedule to a reasonable date, as all the engineers of my team are on vacation now. [09:36:41] (03PS1) 10Gehel: Java build for mediawiki/libs/metrics-platform [integration/config] - 10https://gerrit.wikimedia.org/r/820396 [09:40:44] 10Project-Admins: Create project tag for - https://phabricator.wikimedia.org/T314406 (10EChetty) >>! In T314406#8127910, @Ottomata wrote: > @Echetty I'd hope so. Most of that work is described in the original Modern Event Platform epic: https://phabricator.wikimedia.org/T185233 > > (Which I thi... [10:01:13] !log clearing out stuck beta deployment jobs T314378 T72597 [10:01:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:01:16] T314378: Stop triggering `beta-scap-sync-world` on `beta-mediawiki-config-update-eqiad` completion - https://phabricator.wikimedia.org/T314378 [10:01:16] T72597: Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - https://phabricator.wikimedia.org/T72597 [10:03:24] TheresNoTime: re T314378, I wonder if we even need the config-update job at all now that code-update also updates config [10:04:11] taavi: agree! I spent a few minutes yesterday trying to figure out the difference in the two jobs (bar the fact the config job is triggered immediately after a relevant change is merged) [10:04:27] if config changes can wait the ~10 minutes, that'd solve it entirely :) [10:05:52] especially since the config job usually takes longer than 10 minutes till it runs [10:09:07] (03PS1) 10Majavah: zuul: stop triggering beta-mediawiki-config-update-eqiad jobs [integration/config] - 10https://gerrit.wikimedia.org/r/820405 (https://phabricator.wikimedia.org/T314378) [10:09:09] (03PS1) 10Majavah: jjb: remove beta-mediawiki-config-update-eqiad job [integration/config] - 10https://gerrit.wikimedia.org/r/820406 (https://phabricator.wikimedia.org/T314378) [10:09:31] ^ so something like that? [10:19:01] (03CR) 10Samtar: [C: 03+1] "lgtm" [integration/config] - 10https://gerrit.wikimedia.org/r/820405 (https://phabricator.wikimedia.org/T314378) (owner: 10Majavah) [10:19:50] (03CR) 10Samtar: [C: 03+1] "also lgtm" [integration/config] - 10https://gerrit.wikimedia.org/r/820406 (https://phabricator.wikimedia.org/T314378) (owner: 10Majavah) [10:20:23] have some +1s [11:38:12] 10Phabricator (Upstream), 10Upstream: "Call to phutil_nonempty_string() expected null or a string, got: int." when attempting to view Subversion repos - https://phabricator.wikimedia.org/T310936 (10Aklapper) [11:38:39] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream: "Call to phutil_nonempty_string() expected null or a string, got: int." when attempting to view Subversion repos - https://phabricator.wikimedia.org/T310936 (10Aklapper) [12:40:49] (03PS1) 10Jaime Nuche: serial lock: make sure dir for the lock exists [tools/scap] - 10https://gerrit.wikimedia.org/r/820438 [12:43:55] (03CR) 10Jaime Nuche: "Tested on train-dev" [tools/scap] - 10https://gerrit.wikimedia.org/r/820438 (owner: 10Jaime Nuche) [12:44:47] 10Project-Admins: Create User-eakinloose project - https://phabricator.wikimedia.org/T313800 (10EAkinloose) Thanks! [12:45:09] (03CR) 10CI reject: [V: 04-1] serial lock: make sure dir for the lock exists [tools/scap] - 10https://gerrit.wikimedia.org/r/820438 (owner: 10Jaime Nuche) [12:51:27] <_joe_> say I need assistance for setting up CI for a project in gitlab [12:51:31] <_joe_> who do I ask? [13:20:27] _joe_: #wikimedia-gitlab [13:20:51] <_joe_> taavi: heh I went with the jelto option :) [13:57:21] (03PS2) 10Jaime Nuche: serial lock: make sure dir for the lock exists [tools/scap] - 10https://gerrit.wikimedia.org/r/820438 [13:57:52] 10Project-Admins: Create project tag for - https://phabricator.wikimedia.org/T314406 (10Ottomata) Ya we can totally reorg Event-Platform as needed. [14:00:30] (03Abandoned) 10Jbond: WIP: add files for custom image for beaker builds [integration/config] - 10https://gerrit.wikimedia.org/r/812463 (https://phabricator.wikimedia.org/T253635) (owner: 10Jbond) [14:02:26] (03PS3) 10Jaime Nuche: serial lock: make sure dir for the lock exists [tools/scap] - 10https://gerrit.wikimedia.org/r/820438 [14:18:00] (03CR) 10CI reject: [V: 04-1] serial lock: make sure dir for the lock exists [tools/scap] - 10https://gerrit.wikimedia.org/r/820438 (owner: 10Jaime Nuche) [14:21:11] (03PS4) 10Jaime Nuche: serial lock: make sure dir for the lock exists [tools/scap] - 10https://gerrit.wikimedia.org/r/820438 [14:39:55] 10Continuous-Integration-Config, 10ci-test-error: Flake8 5.0.0 release breaking CI jobs - https://phabricator.wikimedia.org/T314577 (10TheresNoTime) [14:40:08] 10Continuous-Integration-Config, 10ci-test-error: Flake8 5.0.0 release breaking CI jobs - https://phabricator.wikimedia.org/T314577 (10TheresNoTime) [14:42:19] 10Continuous-Integration-Config, 10ci-test-error: Flake8 5.0.0 release breaking CI jobs - https://phabricator.wikimedia.org/T314577 (10TheresNoTime) [14:42:30] 10Continuous-Integration-Config, 10ci-test-error: Flake8 5.0.0+ release breaking CI jobs - https://phabricator.wikimedia.org/T314577 (10TheresNoTime) [15:05:24] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [15:13:13] (03CR) 10Ahmon Dancy: [C: 03+2] zuul: stop triggering beta-mediawiki-config-update-eqiad jobs [integration/config] - 10https://gerrit.wikimedia.org/r/820405 (https://phabricator.wikimedia.org/T314378) (owner: 10Majavah) [15:15:16] (03Merged) 10jenkins-bot: zuul: stop triggering beta-mediawiki-config-update-eqiad jobs [integration/config] - 10https://gerrit.wikimedia.org/r/820405 (https://phabricator.wikimedia.org/T314378) (owner: 10Majavah) [15:16:25] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/820405 [15:16:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:16:43] Project beta-mediawiki-config-update-eqiad build #750: 15ABORTED in 2 min 7 sec: https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/750/ [15:17:35] (03CR) 10Ahmon Dancy: [C: 03+2] jjb: remove beta-mediawiki-config-update-eqiad job [integration/config] - 10https://gerrit.wikimedia.org/r/820406 (https://phabricator.wikimedia.org/T314378) (owner: 10Majavah) [15:20:34] (03Merged) 10jenkins-bot: jjb: remove beta-mediawiki-config-update-eqiad job [integration/config] - 10https://gerrit.wikimedia.org/r/820406 (https://phabricator.wikimedia.org/T314378) (owner: 10Majavah) [15:21:20] !log Deleting beta-mediawiki-config-update-eqiad job [15:21:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:25:20] dancy: does ^ need another zuul reload? [15:25:38] no, it's not touching zuul config files [15:25:47] s/zuul/jenkins [15:25:57] thanks dancy, you were faster than me on deploying those [15:28:48] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [15:32:19] TheresNoTime: No Jenkins restart required for the job deletion. [15:32:42] ^^ [15:59:01] 10Release-Engineering-Team, 10Data-Engineering-Radar, 10Gerrit-Privilege-Requests: Requesting membership of the analytics group in gerrit for 'snwachukwu' and 'nokafor' - https://phabricator.wikimedia.org/T314592 (10Milimetric) [16:13:29] 10Phabricator, 10Triagers: Add TheresNoTime to Triagers group - https://phabricator.wikimedia.org/T314595 (10TheresNoTime) [16:20:59] 10Phabricator, 10Triagers, 10User-bd808: Add TheresNoTime to Triagers group - https://phabricator.wikimedia.org/T314595 (10bd808) 05Open→03Resolved a:03bd808 {{Done}} Silencing notifications requires real-time collaboration with someone with shell access to the Phabricator servers, but you now will hav... [18:09:41] 10Phabricator: Run two bulk jobs silently - https://phabricator.wikimedia.org/T314597 (10Aklapper) Done: ` aklapper@phab1001:~$ sudo /srv/phab/phabricator/bin/bulk make-silent --id 3262 Configured job "3262" to run silently. aklapper@phab1001:~$ sudo /srv/phab/phabricator/bin/bulk make-silent --id 3263 Configure... [18:16:14] 10Phabricator: Run two bulk jobs silently - https://phabricator.wikimedia.org/T314597 (10TheresNoTime) Ah but now clicking "Continue" shows me that the job has now been set to silent: {F35398066,size=full} I've completed both of the above jobs now, thank you! As an aside, is there a reason this isn't an option... [19:59:24] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T308076 (10matmarex) >>! In T308076#8124294, @matmarex wrote: > ##### Risky Patch! 🚂🔥 > > * **Change**: https://gerrit.wikimedia.org/... [20:09:32] 10Release-Engineering-Team, 10Scap: Scap backport: Display change title when asking for confirmation to backport - https://phabricator.wikimedia.org/T314612 (10jeena) [20:18:25] 10Release-Engineering-Team, 10Scap: Scap backport: Notify on irc when change has been deployed to mwdebug - https://phabricator.wikimedia.org/T314613 (10jeena) [20:25:03] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T308076 (10matmarex) I can't get a response on IRC, but from where I'm sitting, it looks like we now have wmf.23 everywhere. [20:26:55] 10GitLab (Infrastructure), 10serviceops, 10serviceops-collab, 10Patch-For-Review: Document and test failover for GitLab and GitLab Replica - https://phabricator.wikimedia.org/T296713 (10Arnoldokoth) ` # host gitlab-replica-old.wikimedia.org gitlab-replica-old.wikimedia.org has address 208.80.154.15 gitlab-... [20:29:27] 10GitLab (Infrastructure), 10serviceops, 10serviceops-collab, 10Patch-For-Review: Document and test failover for GitLab and GitLab Replica - https://phabricator.wikimedia.org/T296713 (10Arnoldokoth) @Jelto I managed to switch over the hosts i.e. gitlab1003 and gitlab2002. The current replica is now gitlab2... [21:19:19] (03CR) 10Ahmon Dancy: [C: 03+2] Calendar: branch train 2am UTC [tools/release] - 10https://gerrit.wikimedia.org/r/820241 (owner: 10Thcipriani) [21:20:38] (03Merged) 10jenkins-bot: Calendar: branch train 2am UTC [tools/release] - 10https://gerrit.wikimedia.org/r/820241 (owner: 10Thcipriani) [21:20:58] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission 💀), 10Patch-For-Review: Add gerrit2002 as a replica of gerrit1001 - https://phabricator.wikimedia.org/T313972 (10dancy) Confirming that replication from gerrit1001 to gerrit2002 is working. [21:21:30] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission 💀), 10Patch-For-Review: Add gerrit2002 as a replica of gerrit1001 - https://phabricator.wikimedia.org/T313972 (10dancy) [21:42:11] 10GitLab (Infrastructure), 10serviceops, 10serviceops-collab, 10Patch-For-Review: Document and test failover for GitLab and GitLab Replica - https://phabricator.wikimedia.org/T296713 (10Arnoldokoth) Thanks to @Dzahn the SSL issue is now resolved. I did have a question though, what happens to the old repl... [23:18:58] (03PS1) 10Subramanya Sastry: Hide empty talk page DiscussionTools invitation [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/820578 [23:19:00] (03PS1) 10Subramanya Sastry: Strip p-wrappers around "empty" paragraphs [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/820579 [23:25:50] we are currently switching the gerrit-replica.wikimedia.org to a new machine. DNS switch. everything looks fine, can clone from it etc [23:26:08] in half an hour we'll check again if traffic completely stopped [23:43:16] (03PS2) 10Subramanya Sastry: Strip p-wrappers around "empty" paragraphs [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/820579