[01:14:20] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#12092818 (10Dzahn) added new maintenance window, July 7th, 15:30–16:30 UTC # 08:30–09:30 PDT to Deployment Cale... [01:15:40] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services, 13Patch-For-Review: Update Jenkins hosts from Java 17 to Java 21 - https://phabricator.wikimedia.org/T418109#12092819 (10Dzahn) new maintenance window to switch jenkins to new contint hosts, tomorrow, July 7th, 15:30–16:30 UTC #... [01:44:36] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services, 13Patch-For-Review: Update Jenkins hosts from Java 17 to Java 21 - https://phabricator.wikimedia.org/T418109#12092857 (10Dzahn) runbook v2: https://docs.google.com/document/d/1dR2k4Ea2p67Fj12iZALQh1BwlHe5bKfhp7esg7Tc2VM/edit?usp=... [01:44:45] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#12092858 (10Dzahn) runbook v2 https://docs.google.com/document/d/1dR2k4Ea2p67Fj12iZALQh1BwlHe5bKfhp7esg7Tc2VM/e... [06:13:43] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review, 07User-notice: Sunsetting mirrors.wikimedia.org - https://phabricator.wikimedia.org/T416707#12093097 (10MoritzMuehlenhoff) 05Open→03Resolved a:03MoritzMuehlenhoff mirrors.wikimedia.org has been deregist... [07:19:34] Project mediawiki-core-doxygen build #21427: 04FAILURE in 1 min 29 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/21427/ [07:22:31] FIRING: [4x] ProbeDown: Service gerrit2003:29418 has failed probes (tcp_gerrit_ssh_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [07:22:42] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T431381 (10phaultfinder) 03NEW [07:25:21] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T431381#12093220 (10ABran-WMF) [07:25:25] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T431381#12093224 (10ABran-WMF) [07:27:31] RESOLVED: [4x] ProbeDown: Service gerrit2003:29418 has failed probes (tcp_gerrit_ssh_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [08:32:15] Yippee, build fixed! [08:32:15] Project mediawiki-core-doxygen build #21428: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/21428/ [10:13:31] 06Release-Engineering-Team, 06collaboration-services: Path filtering for gerrit - https://phabricator.wikimedia.org/T431398 (10ABran-WMF) 03NEW [14:38:44] 10Phabricator (phabricator-next), 06Release-Engineering-Team: Deploy Phab/Phorge 2026-07-07 - https://phabricator.wikimedia.org/T431440 (10brennen) 03NEW [14:39:17] (03open) 10brennen: update submodules for 2026-07-07 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/112 (https://phabricator.wikimedia.org/T431440) [14:57:30] (03merge) 10brennen: update submodules for 2026-07-07 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/112 (https://phabricator.wikimedia.org/T431440) [15:05:31] FIRING: [4x] ProbeDown: Service gerrit2003:29418 has failed probes (tcp_gerrit_ssh_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:05:44] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T431448 (10phaultfinder) 03NEW [15:07:17] 10Phabricator (2026-07-07), 06Release-Engineering-Team (Doing 😎): Deploy Phab/Phorge 2026-07-07 - https://phabricator.wikimedia.org/T431440#12095141 (10Aklapper) 05Open→03Resolved a:03brennen Things that got deployed: * {T416964} and {T420522} via https://gitlab.wikimedia.org/repos/phabricator/phabri... [15:10:31] RESOLVED: [4x] ProbeDown: Service gerrit2003:29418 has failed probes (tcp_gerrit_ssh_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:11:31] 10Phabricator, 06Release-Engineering-Team (Doing 😎), 13Patch-For-Review: Remove remaining diffusion.ssh-host entries from Phab config and Puppet - https://phabricator.wikimedia.org/T429367#12095185 (10Aklapper) [15:16:45] 10Phabricator (2026-07-07), 06Release-Engineering-Team (Doing 😎): Port custom code in DiffusionCommitController to use upstream PhorgeRestrictableInteractionInterface - https://phabricator.wikimedia.org/T418578#12095241 (10Aklapper) 05Open→03Resolved This should be fixed now as it was part of today's d... [15:16:48] 10Phabricator (2026-07-07): Hovering over long truncated strings may result in some vertical movement - https://phabricator.wikimedia.org/T420522#12095243 (10Aklapper) This should be fixed now as it was part of today's deployment. [15:16:50] 10Phabricator (2026-07-07), 07CSS, 07Mobile: Simplified patch status comments have inconsistent margin on mobile - https://phabricator.wikimedia.org/T416964#12095244 (10Aklapper) 05Open→03Resolved This should be fixed now as it was part of today's deployment. [15:16:54] 10Phabricator (2026-07-07): Hovering over long truncated strings may result in some vertical movement - https://phabricator.wikimedia.org/T420522#12095247 (10Aklapper) 05Open→03Resolved [15:18:36] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T431448#12095254 (10ABran-WMF) [15:27:41] (03PS1) 10Hslater: Zuul: [mediawiki/extensions/ContentTransfer] Add OOJSPlus [integration/config] - 10https://gerrit.wikimedia.org/r/1308136 [15:29:08] 06Release-Engineering-Team, 06collaboration-services: temp - https://phabricator.wikimedia.org/T431453 (10ABran-WMF) 03NEW [15:32:31] FIRING: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:37:31] FIRING: [4x] ProbeDown: Service gerrit2003:29418 has failed probes (tcp_gerrit_ssh_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:42:31] RESOLVED: [4x] ProbeDown: Service gerrit2003:29418 has failed probes (tcp_gerrit_ssh_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:51:13] (03update) 10aklapper: Draft: Replace deprecated Phabricator Conduit API calls [repos/releng/gitlab-webhooks] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/41 (https://phabricator.wikimedia.org/T430410) [15:51:30] PROBLEM - jenkins_service_running on contint1002 is CRITICAL: PROCS CRITICAL: 0 processes with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [15:54:09] ^ maintenance period [16:16:41] RECOVERY - jenkins_service_running on contint1003 is OK: PROCS OK: 1 process with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [16:21:20] (03update) 10aklapper: Draft: Replace deprecated Phabricator Conduit API calls [repos/releng/gitlab-webhooks] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/41 (https://phabricator.wikimedia.org/T430410) [16:35:31] RECOVERY - jenkins_service_running on contint1002 is OK: PROCS OK: 1 process with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [16:38:31] PROBLEM - jenkins_service_running on contint1002 is CRITICAL: PROCS CRITICAL: 0 processes with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [16:58:26] (03update) 10aklapper: Draft: Replace deprecated Phabricator Conduit API calls [repos/releng/gitlab-webhooks] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/41 (https://phabricator.wikimedia.org/T430410) [16:58:40] Project mediawiki-core-doxygen build #21435: 04STILL FAILING in 32 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/21435/ [17:01:44] (03update) 10aklapper: Draft: Replace deprecated Phabricator Conduit API calls [repos/releng/gitlab-webhooks] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/41 (https://phabricator.wikimedia.org/T430410) [17:05:24] (03update) 10aklapper: Draft: Replace deprecated Phabricator Conduit API calls [repos/releng/gitlab-webhooks] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/41 (https://phabricator.wikimedia.org/T430410) [17:08:15] (03update) 10aklapper: Draft: Replace deprecated Phabricator Conduit API calls [repos/releng/gitlab-webhooks] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/41 (https://phabricator.wikimedia.org/T430410) [17:10:17] (03update) 10aklapper: Replace deprecated Phabricator Conduit API calls [repos/releng/gitlab-webhooks] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/41 (https://phabricator.wikimedia.org/T430410) [17:10:54] dduvall: a couple of your zuul k8s worker nodes crashed and rebooted overnight; did the cluster recover properly? [17:11:09] (well, the nodes didn't crash, the hardware they were running on crashed) [17:16:31] FIRING: [4x] ProbeDown: Service gerrit2003:29418 has failed probes (tcp_gerrit_ssh_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:17:59] !log gerrit: deleted /srv/gerrit/java_pid3571660.hprof [17:18:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:21:25] andrewbogott: i see 10 "active" instances so it seems like it recovered ok [17:21:31] RESOLVED: [4x] ProbeDown: Service gerrit2003:29418 has failed probes (tcp_gerrit_ssh_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:22:32] dduvall: great! For your reference, the two victims were; [17:22:36] kube-2js6t-default-worker-5ff26-rg5n9-nxtmp.zuul [17:22:36] kube-2js6t-default-worker-5ff26-rg5n9-r722f.zuul [17:26:29] Project mediawiki-core-doxygen build #21436: 04STILL FAILING in 27 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/21436/ [17:29:15] andrewbogott: ack. thanks for checking in about it! [17:43:51] Project mediawiki-core-doxygen build #21437: 04STILL FAILING in 17 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/21437/ [17:53:52] (03PS1) 10Hashar: Switch Jenkins from contint1002 to contint1003 [integration/config] - 10https://gerrit.wikimedia.org/r/1308172 (https://phabricator.wikimedia.org/T418521) [17:59:12] (03CR) 10Hashar: [C:03+2] "Deployed and I have approved the new scripts in the script security configuration" [integration/config] - 10https://gerrit.wikimedia.org/r/1308172 (https://phabricator.wikimedia.org/T418521) (owner: 10Hashar) [18:00:53] (03Merged) 10jenkins-bot: Switch Jenkins from contint1002 to contint1003 [integration/config] - 10https://gerrit.wikimedia.org/r/1308172 (https://phabricator.wikimedia.org/T418521) (owner: 10Hashar) [18:05:37] Hi, I don't know if it's known or I'm missing something obvious but this patch fails CI really fast. It looks like it can't even clone the repo. https://gerrit.wikimedia.org/r/c/operations/software/thumbor-plugins/+/1308156 [18:06:29] https://integration.wikimedia.org/ci/job/thumbor-plugins-pipeline-test/352/console [18:07:44] Project mediawiki-core-doxygen build #21438: 04STILL FAILING in 23 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/21438/ [18:08:12] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#12096393 (10Dzahn) https://docs.google.com/document/d/e/2PACX-1vQUiV0n8D2BcUKEFe4I3sxQKoYCc6g5nzrthtt_U-m23QC2-... [18:10:02] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#12096398 (10Dzahn) {F92294292} [18:10:11] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#12096399 (10Dzahn) 05Open→03Resolved [18:10:28] Amir1: gerrit's been unhappy today, i wonder if it was hitting an outage [18:12:10] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services: Ensure PipelineLib works on new Jenkins host - https://phabricator.wikimedia.org/T423968#12096410 (10hashar) 05Open→03Resolved a:03hashar They seem to be working under Ja... [18:13:01] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#12096417 (10Dzahn) {meme, src="antoine-approve"} [18:13:31] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#12096418 (10hashar) Very well done @Dzahn thank you so much for having prepared all that migration and for... [18:13:36] https://integration.wikimedia.org/ci/job/alerts-pipeline-test/3476/console also seems weird - timed out trying to connect to contint1002 [18:22:12] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services: Update Jenkins hosts from Java 17 to Java 21 - https://phabricator.wikimedia.org/T418109#12096450 (10hashar) @Dzahn successfully led the effort to switch Jenkins to a new host that has Java 21 (T418521). We are now running Jenkins... [18:22:25] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services: Update Jenkins hosts from Java 17 to Java 21 - https://phabricator.wikimedia.org/T418109#12096452 (10hashar) 05In progress→03Resolved a:03hashar [18:24:53] oh, right. jenkins switch. sorry, i'm not tracking all the things very effectively today. (cc: Amir1) [18:25:37] brennen: it has been switched to contint1003 [18:41:21] oh sorry, I just got out of meeting [18:41:23] thanks! [18:47:53] now it tries to connect to contint2002.wikimedia.org and failing https://integration.wikimedia.org/ci/job/thumbor-plugins-pipeline-test/355/console [19:01:06] hardcoded somewhere, i wonder? [19:02:17] Antoine just verified those pipeline jobs as working. it was a seperate checkbox specifically for just pipeline-tests [19:02:42] please try to ask him if still reachable [19:05:31] codesearch doesn't return anything [19:10:19] attempting a fix [19:10:34] assumes it's a firewall issue [19:21:57] Amir1: [19:22:02] brennen: I fixed something [19:22:10] but there is another issue behind that one it seems [19:22:27] I fixed firewall issue causing the timeout on git clone. [19:28:44] I think I see the next issue as well. [19:28:53] jenkins-agent user was not in docker group [19:32:07] that do seem important [19:34:06] 10Phabricator Translations Extension, 06translatewiki.net: Phabricator translations have been failing i18n-check for almost a month - https://phabricator.wikimedia.org/T431487 (10Pppery) 03NEW [19:38:20] I think it's fixed. it's running right now. [19:38:27] thanks mutante [19:40:48] #16 18.23 Building wheel for thumbor (pyproject.toml): started [19:41:59] !log contint1003/2003 - add jenkins-agent user to docker group; restart jenkins [19:41:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:43:10] Amir1: fixed the job. at this point it's "just" make: *** [Makefile:46: lint] Error 1 [20:00:49] (03approved) 10addshore: performance: improve cli startup time [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/652 (https://phabricator.wikimedia.org/T431234 https://phabricator.wikimedia.org/T431235) (owner: 10santhosh) [20:16:11] (03open) 10addshore: Override golang-gosec_static_analysis_check to fix ci [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/653 [20:16:36] (03update) 10addshore: Override golang-gosec_static_analysis_check to fix ci [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/653 [20:32:31] FIRING: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:32:39] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T431456#12097120 (10phaultfinder) [20:37:30] (03update) 10addshore: Override golang-gosec_static_analysis_check to fix ci [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/653 [20:37:31] RESOLVED: [4x] ProbeDown: Service gerrit2003:29418 has failed probes (tcp_gerrit_ssh_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:45:40] (03approved) 10addshore: Override golang-gosec_static_analysis_check to fix ci [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/653 [20:45:54] (03merge) 10addshore: Override golang-gosec_static_analysis_check to fix ci [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/653 [20:46:36] (03merge) 10addshore: performance: improve cli startup time [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/652 (https://phabricator.wikimedia.org/T431234 https://phabricator.wikimedia.org/T431235) (owner: 10santhosh) [20:51:12] (03open) 10addshore: Update various go packages [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/654 [20:52:19] (03open) 10addshore: Update shellbox to shellbox:2026-07-03-212105 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/655 [20:53:34] 06Release-Engineering-Team, 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Add RSS extension to PatchDemo - https://phabricator.wikimedia.org/T426130#12097207 (10brennen) 05Open→03In progress [20:53:47] (03open) 10addshore: Update to releng/node20-test-browser:20.20.2 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/656 [20:54:07] (03open) 10addshore: Update to eventgate-wikimedia:v1.31.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/657 [20:54:50] (03open) 10addshore: Update from redis:6.2 to redis:8.6.4 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/658 [21:03:44] (03update) 10addshore: Update various go packages [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/654 [21:04:58] 06Release-Engineering-Team (Doing 😎): Allow configuration of canary and production checks based on deployment target - https://phabricator.wikimedia.org/T428971#12097235 (10bd808) 05Open→03In progress a:03dancy Per weekly sync meeting updates [21:33:04] (03open) 10aghirelli: release: prepare 0.1.0 for npm publish [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/15 (https://phabricator.wikimedia.org/T425627) [21:39:45] (03merge) 10aghirelli: release: prepare 0.1.0 for npm publish [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/15 (https://phabricator.wikimedia.org/T425627) [21:42:21] mutante: "add jenkins-agent user to docker group; restart jenkins" [21:42:23] well done ! [21:43:33] the sole test I did for pipelinelinb jobs was on the dummy change https://gerrit.wikimedia.org/r/c/operations/alerts/+/1308168 [21:50:01] 10GitLab, 06Release-Engineering-Team (Priority Backlog 📥), 07Documentation: [[wikitech:Password and 2FA reset]] should cover GitLab MFA resets - https://phabricator.wikimedia.org/T431415#12097315 (10bd808) This is interesting from [[https://docs.gitlab.com/user/profile/account/two_factor_authentication_troub... [21:50:04] 06Release-Engineering-Team, 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work, 13Patch-For-Review: Add RSS extension to PatchDemo - https://phabricator.wikimedia.org/T426130#12097317 (10brennen) p:05Triage→03Medium [[ https://gitlab.wikimedia.org/repos/test-platform/catalyst/patchdemo/-/merge_requests... [22:13:10] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-OATHAuth, 06Product Safety and Integrity: beta: /enwiki/wiki/Special:AccountSecurity UnexpectedValueException: More than one recovery codes instance was found in the database for this user. - https://phabricator.wikimedia.org/T429278#12097342 (10Reedy) p:... [22:34:30] 10Beta-Cluster-Infrastructure, 10Wikidata: Beta Wikidata uses production Commons - https://phabricator.wikimedia.org/T431507 (10AlexisJazz) 03NEW