[06:35:37] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T406653#11253317 (10ABran-WMF) [06:35:40] 06Release-Engineering-Team, 06collaboration-services, 07Security: ProbeDown - gerrit1003:443 - https://phabricator.wikimedia.org/T406403#11253318 (10ABran-WMF) [06:36:06] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T406653#11253322 (10ABran-WMF) 05Open→03Resolved a:03ABran-WMF [06:52:48] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - gerrit1003:443 - https://phabricator.wikimedia.org/T406653#11253358 (10ABran-WMF) [06:54:00] 10Gerrit, 06collaboration-services: Gerrit: fix allocstalls - https://phabricator.wikimedia.org/T404965#11253360 (10ABran-WMF) [06:54:01] 10Gerrit, 06collaboration-services: gerrit: config tweaks - https://phabricator.wikimedia.org/T406333#11253361 (10ABran-WMF) [06:54:07] 10Gerrit, 06collaboration-services: gerrit: fix allocstalls - https://phabricator.wikimedia.org/T404965#11253362 (10ABran-WMF) [07:27:34] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance deployment-maps-master02 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [07:58:13] (03PS3) 10Hashar: inference-services: Remove the outlink-transformer CI job. [integration/config] - 10https://gerrit.wikimedia.org/r/1187752 (https://phabricator.wikimedia.org/T404294) (owner: 10Bartosz Wójtowicz) [08:02:35] (03CR) 10Hashar: "I have been busy over the last few weeks and I am lagging with my review." [integration/config] - 10https://gerrit.wikimedia.org/r/1187752 (https://phabricator.wikimedia.org/T404294) (owner: 10Bartosz Wójtowicz) [08:26:35] of course some REL1_39 repos are not compatible with node20 cause of wdio > fiber (example: ProofreadPage ) [08:26:37] fun time [08:46:46] 06Release-Engineering-Team, 06collaboration-services, 07Security: ProbeDown - gerrit1003:443 - https://phabricator.wikimedia.org/T406403#11253645 (10ABran-WMF) 05Open→03Resolved Those tweaks helped shed what opened {T406653} and this task: {F66738242} We're able to see here that the maximum connectio... [09:37:35] 10Beta-Cluster-Infrastructure, 06DBA, 10MediaWiki-libs-Rdbms, 07Epic: Enable MariaDB/MySQL's Strict Mode - https://phabricator.wikimedia.org/T108255#11253837 (10Reedy) [09:56:07] (03PS1) 10Hashar: Fix link to task in the motd banner [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1194573 [09:56:57] 10Beta-Cluster-Infrastructure, 06Traffic: Copy the Traffic team on alerts for deployment-cache* hosts - https://phabricator.wikimedia.org/T406650#11253944 (10Ladsgroup) >>! In T406650#11252791, @bd808 wrote: >>>! In T406650#11252612, @bd808 wrote: >>>>! In T406650#11252502, @Aklapper wrote: >>> I don't know th... [09:57:37] (03CR) 10Hashar: [C:03+2] Add a banner for a Gerrit switch over maintenance (031 comment) [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1193017 (https://phabricator.wikimedia.org/T387833) (owner: 10Hashar) [10:00:44] (03CR) 10Tacsipacsi: "Thanks for the patch!" [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1194573 (owner: 10Hashar) [10:19:17] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10ConfirmEdit (CAPTCHA extension), 06Trust and Safety Product Team, and 2 others: ConfirmEdit hCaptcha: Get QUnit tests to run in CI - https://phabricator.wikimedia.org/T406231#11254014 (10kostajh) >>! In T406231#11252420, @thcipriani wrote: > To... [10:22:20] 10Continuous-Integration-Config, 10ConfirmEdit (CAPTCHA extension): ConfirmEdit: Load all sub-extensions in CI - https://phabricator.wikimedia.org/T399593#11254057 (10kostajh) 05Open→03Stalled Going to try for {T398193} instead. [12:10:05] (03CR) 10Hashar: [C:03+2] zuul: Add Hubaishan to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1193561 (owner: 10Zabe) [12:10:54] (03PS3) 10Hashar: Ease configuration of the motd banner [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1194221 [12:18:29] (03Merged) 10jenkins-bot: zuul: Add Hubaishan to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1193561 (owner: 10Zabe) [12:23:43] (03CR) 10Hashar: [C:03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/1193561 (owner: 10Zabe) [12:24:52] (03CR) 10Hashar: Fix link to task in the motd banner (031 comment) [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1194573 (owner: 10Hashar) [12:25:07] (03PS2) 10Hashar: Fix link to task in the motd banner [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1194573 [12:29:11] (03PS4) 10Hashar: Ease configuration of the motd banner [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1194221 [12:30:43] (03CR) 10Hashar: "Rebased and includes fixes by @tacsipacsi@jnet.hu made in parent patch Id1f4f0cd8d527504dac813e76fea32508c6e3bb2" [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1194221 (owner: 10Hashar) [13:00:37] hi folks... `registry.cloud.releng.team` is returning 503s from gitlab [13:00:40] is this expected? [13:00:58] https://gitlab.wikimedia.org/repos/sre/xcheesescore/-/jobs/639853 is one example of impacted jobs [13:02:34] It was mentioned on slack too [13:02:48] I suspect it's not expected :P [13:03:25] also.. are we using a valid domain on the internet (releng.team) internally? [13:04:11] or do we own releng.team? :) [13:05:04] oh we own it, cool [13:08:24] vgutierrez: please file a task in Phabricator tagging #gitlab-ci [GitLab (CI & Job Runners)] [13:08:35] sure [13:08:46] that registry is powered by Reggie and is hosted on our Digital Ocean cluster [13:08:50] afaik [13:08:51] :) [13:10:51] ack, that's T406733 [13:10:52] 10GitLab (CI & Job Runners): registry.cloud.releng.team returning 503s - https://phabricator.wikimedia.org/T406733 (10Vgutierrez) 03NEW [13:10:52] T406733: registry.cloud.releng.team returning 503s - https://phabricator.wikimedia.org/T406733 [13:11:45] jnuche: jelto: do you have any access to the Digital Ocean to check how Reggie is behaving / restart it? [13:14:21] in theory yes, let me login to the cluster and check if I can find anything reggie related. dancy knows more about that probably [13:15:03] definitely yes :] [13:15:11] 10GitLab (CI & Job Runners): registry.cloud.releng.team returning 503s - https://phabricator.wikimedia.org/T406733#11255061 (10hashar) That registry is powered by Reggie and is hosted on our Digital Ocean cluster. [13:15:16] I lack access to the DO cluster, to be fair I never bothered to even check at it :) [13:15:52] hashar: I can access by logging in with my WM google account, you probably can too [13:16:20] this feels like something that should be documented somewhere :-) [13:16:22] I'm also trying to take a look at the cluster, but I'm not really familiar with how to manage it [13:16:43] * hashar knows nothing about Digital Ocean [13:18:04] I found a pod called "reggie-0" in the gitlab-runner namespace [13:18:23] do you have a way to retrieve logs / errors out of it? [13:18:55] 06Project-Admins, 06LPL Hypothesis, 10Projects-Cleanup, 10Wikipedia Subscriptions: Archive Wikipedia Subscriptions - https://phabricator.wikimedia.org/T404331#11255068 (10Nikerabbit) p:05Triage→03Low [13:21:21] I can see some logs, but not a lot, mostly 404s for the xcheesescore for example [13:22:49] but it looks like the reggie pod is just 13m old. I assume the problem is also just present since that time? [13:23:24] jelto: yeah.. it was working as expected ~1h ago [13:23:36] ok.. now it's working [13:24:28] 10Diffusion, 10Phabricator (Upstream), 06Release-Engineering-Team, 07Upstream: Diffusion code view: Per-file change info is slow to load despite caching (diffusion.lastmodifiedquery) - https://phabricator.wikimedia.org/T403215#11255084 (10Aklapper) Trying with directories listed on https://phabricator.wiki... [13:24:35] 10Diffusion, 10Phabricator (Upstream), 06Release-Engineering-Team, 07Performance Issue, 07Upstream: Diffusion code view: Per-file change info is slow to load despite caching (diffusion.lastmodifiedquery) - https://phabricator.wikimedia.org/T403215#11255085 (10Aklapper) [13:24:52] Yes I can see the PUT request with a 201 [13:25:17] 10Diffusion, 10Phabricator (Upstream), 06Release-Engineering-Team, 07Performance Issue, 07Upstream: Diffusion code view: Per-file change info is slow to load despite caching (diffusion.lastmodifiedquery) - https://phabricator.wikimedia.org/T403215#11255089 (10Aklapper) [13:25:21] https://www.irccloud.com/pastebin/jaPN1IoU/ [13:25:36] well, I'm surprised it's working now, it seems that the pod recreation didn't succeed? Something about about the pvc failing to attach: [13:25:43] https://www.irccloud.com/pastebin/NbCmtb90/ [13:27:20] maybe some autoscaling foo and reggie got moved to another node and detaching of the volume took more time [13:27:55] yeah, I can see a line right after that showing a successful attachment [13:33:16] 10GitLab (CI & Job Runners): registry.cloud.releng.team returning 503s - https://phabricator.wikimedia.org/T406733#11255113 (10Jelto) The `reggie-01` pod was moved to another node and the pvc was still stuck on the old node. I guess that happened during automatic scale down of the cluster. ` kubectl -n gitlab-... [13:34:58] jnuche: jelto: thank you for the investigation and fix! :] [13:35:28] kubernetes fixed the issue on its own :D I just looked around [13:35:56] same :) [13:43:26] 10GitLab (CI & Job Runners): registry.cloud.releng.team returning 503s - https://phabricator.wikimedia.org/T406733#11255146 (10Jelto) reggie is quite a critical component of the non-production/dev CI stack. So maybe it's possible to make it a bit more resilient or HA (which is probably tricky because of the pers... [13:43:49] 10GitLab (CI & Job Runners), 06collaboration-services: registry.cloud.releng.team returning 503s - https://phabricator.wikimedia.org/T406733#11255148 (10Jelto) [13:57:25] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 3 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#11255228 (10Jelto) [14:03:55] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 3 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#11255295 (10Jelto) Deployment-wise everything is done, gitlab artifacts and packages use the object stora... [14:19:22] (03CR) 10Arnaudb: [C:03+1] Ease configuration of the motd banner [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1194221 (owner: 10Hashar) [14:19:55] (03CR) 10Arnaudb: [C:03+1] Fix link to task in the motd banner [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1194573 (owner: 10Hashar) [14:27:12] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 3 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#11255415 (10jcrespo) All good to me to close an follow up later, but please let's merge T403946 asap (not... [14:36:58] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Dipanshu - https://phabricator.wikimedia.org/T406742 (10Dipanshu) 03NEW [15:00:04] 06Release-Engineering-Team (Radar), 10Cloud-VPS (Quota-requests): Grant gitlab-runners-staging access to fast-iops volume type and a 4xiops instance flavor - https://phabricator.wikimedia.org/T406271#11255542 (10Andrew) 05Open→03Resolved a:03Andrew I think this is done; @dduvall please reopen if you... [15:01:31] (03update) 10dduvall: jwt: Consider `project_path` and `job_project_path` claims [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/107 [15:02:56] (03CR) 10Tacsipacsi: Fix link to task in the motd banner (031 comment) [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1194573 (owner: 10Hashar) [15:05:47] 10Phabricator, 06Release-Engineering-Team (Radar), 06Traffic: Phabricator videos fail in Firefox ("Range" request gets 503 from Varnish) - https://phabricator.wikimedia.org/T397661#11255569 (10Aklapper) @Vgutierrez: Would you have any shareable opinions how to proceed? Can we do anything on our side here, a... [15:06:18] (03CR) 10Tacsipacsi: Ease configuration of the motd banner (032 comments) [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1194221 (owner: 10Hashar) [15:24:29] 10Continuous-Integration-Config, 06Editing-team, 10VisualEditor, 06MediaWiki-Platform-Team (Radar): In CI explicitly define VisualEditor as a dependency - https://phabricator.wikimedia.org/T406376#11255679 (10Tacsipacsi) >>! In https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Translate/+/1193465/comme... [15:32:51] 10Continuous-Integration-Config, 06Editing-team, 10VisualEditor, 06MediaWiki-Platform-Team (Radar): In CI explicitly define VisualEditor as a dependency - https://phabricator.wikimedia.org/T406376#11255713 (10DLynch) I'd agree that some sort of conditional-definition syntax fills a common need. If you [loo... [16:04:59] FYI jnuche, hashar: IMHO T406755 might be significant enough in terms of user impact to qualify as a train blocker [16:05:10] (but I’ll leave the decision to you and not make it a subtask yet) [16:05:25] T406755: Vector TOC started to show -tags instead of showing the italic text used in the headers - https://phabricator.wikimedia.org/T406755 [16:09:58] Lucas_WMDE: thanks for flagging [16:15:13] 10Phabricator, 06Release-Engineering-Team (Radar), 06Traffic: Phabricator videos fail in Firefox ("Range" request gets 503 from Varnish) - https://phabricator.wikimedia.org/T397661#11255927 (10Vgutierrez) Taking a second look at the curl reproducer, I'm seeing the following behavior: First request using: `... [16:18:12] jnuche: that one with Vector looks like an issue with the Parser indeed [16:18:19] I am flagging content transform team on slack [16:19:12] hashar: ack, thx. Doesn't seem to be affecting other wiktionaries: https://en.wiktionary.org/wiki/sarsaparilla?useskin=vector [16:19:28] rn I'm leaning towards not making it a blocker [16:21:37] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T405678#11255959 (10hashar) [16:23:46] bah, en wiki is group 2. Forget what I said [16:26:19] mmh, nope, it's group 1. So yeah, not affected [16:30:03] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T405678#11256015 (10Aklapper) [16:36:17] 10Phabricator, 06Release-Engineering-Team (Radar), 06Traffic: Phabricator videos fail in Firefox ("Range" request gets 503 from Varnish) - https://phabricator.wikimedia.org/T397661#11256047 (10Vgutierrez) following up on my last comment, the webm file size is 1660261 bytes, so a request asking for a range st... [16:44:52] 10Gerrit, 06collaboration-services: gerrit2003 is trying to backup incrementally 2 million files every hour, clogging backus and filling in available disk space - https://phabricator.wikimedia.org/T406762#11256089 (10jcrespo) [16:45:49] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T405679#11256095 (10Aklapper) a:05Aklapper→03hashar I just sold this train for very good moneys to @hashar, I hope will also be very g... [16:45:52] 10Gerrit, 06collaboration-services: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space - https://phabricator.wikimedia.org/T406762#11256098 (10jcrespo) [16:47:51] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work: Add a zuul tenant config on the zuul scheduler host (zuul1001) - https://phabricator.wikimedia.org/T406384#11256112 (10thcipriani) @dduvall should be able to look at this after 2025-10-15. [16:48:00] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T405679#11256113 (10hashar) {meme, src="antoine-approve"} [16:48:10] andre: ^ =] [16:48:19] I am so happy you haven't come yet to disable {meme} on Phab [16:50:01] 10Gerrit, 06collaboration-services: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space - https://phabricator.wikimedia.org/T406762#11256118 (10jcrespo) [16:50:53] 10Gerrit, 06collaboration-services: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space - https://phabricator.wikimedia.org/T406762#11256121 (10Dzahn) I confirm right now host `gerrit2003` is still defined as a `spare_host`. Based on th... [16:56:21] I keep having connection reset issues with gerrit today, and it prompts me to login every time. Did I miss some announcement, or did y'all finally figure out that I'm really an AI scraper and not a real person? [16:56:57] Ah the task right above maybe? [16:57:25] I think I got logged out this morning [16:58:42] Me too I think but now it wants me to sign in every 5 seconds, then dies [16:59:04] oh joy [16:59:21] that would sounds like an issue with your session maybe [16:59:46] or something got poorly tweaked [17:02:10] Eh bien [17:02:30] Snow day! [17:03:34] Plugin install error: https://gerrit.wikimedia.org/r/plugins/wm-schedule-deployment/static/wm-schedule-deployment.js load error from https://gerrit.wikimedia.org/r/plugins/wm-schedule-deployment/static/wm-schedule-deployment.js [17:03:52] And a bunch of errors in the console. So idk what's going on [17:04:13] that would be because your browser could not download the resource? [17:05:07] Presumably, something like that. Am I getting rate-limited or something? [17:05:12] possibly [17:05:22] that is managed by SRE collab ( #wikimedia-sre-collab ) [17:06:03] Daimona: can you file at as a task flagging #gerrit and #sre-collab [17:06:17] there is agraph showing up something about mod_qos https://grafana.wikimedia.org/d/nX8li17Sk/gerrit-overview-sre-collab?orgId=1&from=now-2d&to=now&timezone=utc&var-node=gerrit1003&var-instance=gerrit.wikimedia.org:443&viewPanel=panel-47 [17:06:26] apparently something got tweaked this morning, I guess by arnaudb [17:06:33] and maybe that is the system that is catching you [17:07:36] Thank you, I was just checking the dashboard indeed. I'll be AFK for the next 30 mins or so, will file an issue later today if no changes [17:09:23] (03update) 10dduvall: jwt: Consider `project_path` and `job_project_path` claims [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/107 [17:10:48] I am off myself, I got family at home [17:11:18] 10Gerrit, 06collaboration-services, 13Patch-For-Review: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space - https://phabricator.wikimedia.org/T406762#11256212 (10jcrespo) p:05Unbreak!→03High No longer UBN once the job has been d... [17:11:56] 10Gerrit, 06collaboration-services, 13Patch-For-Review: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space - https://phabricator.wikimedia.org/T406762#11256219 (10jcrespo) [17:12:27] 10Gerrit, 06collaboration-services, 13Patch-For-Review: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space - https://phabricator.wikimedia.org/T406762#11256220 (10jcrespo) [17:42:13] 06Project-Admins, 06LPL Hypothesis, 10Projects-Cleanup, 10Wikipedia Subscriptions: Archive Wikipedia Subscriptions - https://phabricator.wikimedia.org/T404331#11256374 (10eamedina) [17:45:12] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services, 10function-evaluator, and 4 others: Wikifunctions function orchestrator and evaluator test suites failing on GitLab CI with OOM errors - https://phabricator.wikimedia.org/T399348#11256384 (10Jdforrester-WMF) [17:55:22] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work, 13Patch-For-Review: Set up zuul web on zuul1001 - https://phabricator.wikimedia.org/T405119#11256400 (10Dzahn) ` [zuul1001:~] $ sudo systemctl status zuul-web ● zuul-web.service - zuul-web service Loade... [18:30:29] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10ConfirmEdit (CAPTCHA extension), 06Trust and Safety Product Team, and 3 others: ConfirmEdit hCaptcha: Get QUnit tests to run in CI - https://phabricator.wikimedia.org/T406231#11256522 (10Dreamy_Jazz) [18:34:56] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10ConfirmEdit (CAPTCHA extension), 06Trust and Safety Product Team, and 3 others: ConfirmEdit hCaptcha: Get QUnit tests to run in CI - https://phabricator.wikimedia.org/T406231#11256567 (10Dreamy_Jazz) a:03Dreamy_Jazz A summary of the internal... [18:38:16] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work: buildkit v0.25.1 released - https://phabricator.wikimedia.org/T406772 (10dancy) 03NEW [18:38:41] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work: buildkit v0.25.1 released - https://phabricator.wikimedia.org/T406772#11256604 (10dancy) [18:58:49] 10GitLab (CI & Job Runners), 06serviceops: failed to push docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-2.10.5-py3.11-2025-10-03-192132-3003d4328df66a0086a350fdd2ba1dbd80a235c5: unknown: blob upload invalid - https://phabricator.wikimedia.org/T406392#11256660 (10dancy) >>! In T406... [19:02:08] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work, 13Patch-For-Review: Set up zuul web on zuul1001 - https://phabricator.wikimedia.org/T405119#11256667 (10Dzahn) The zuul-web service and container is running. Port 9000 should be mapped to the host. I just d... [19:04:56] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [YOUR DEVELOPER ACCOUNT USERNAME HERE] - https://phabricator.wikimedia.org/T405600#11256673 (10Pppery) 05Stalled→03Invalid Closing due to lack of response. [19:06:56] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services: Gerrit connection troubles and ERR_CONNECTION_RESET - https://phabricator.wikimedia.org/T406774 (10hashar) 03NEW [19:07:05] Daimona_: filed as https://phabricator.wikimedia.org/T406774 [19:12:57] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services: Gerrit connection troubles and ERR_CONNECTION_RESET - https://phabricator.wikimedia.org/T406774#11256694 (10hashar) p:05Triage→03Unbreak! I forgot to subscribe @ABran-WMF As part of {T406403} there were two changes made this morning: http... [19:13:01] Thanks! [19:25:05] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [YOUR DEVELOPER ACCOUNT USERNAME HERE] - https://phabricator.wikimedia.org/T406776 (10Olatundeawo) 03NEW [19:39:22] tgr_: Daimona_: I could imagine your browser doing way more newconnections/requests than other for some reason and you end up triggering whatever limit exists [19:39:38] or something else, like the pool is almost full and because you do a lot of requests you are hit more often than others [19:39:48] then honestly I have no idea how the mod_qos system works :/ [19:40:07] but I got a patch to make it logging only https://gerrit.wikimedia.org/r/1194723 [19:40:22] I have no idea either. Thanks for the patch though! [19:41:25] At any rate, I sure wouldn't mind to be able to use gerrit and do some work... [19:48:58] I hear you! [19:51:01] More reasons to get gerrit behind the CDN instead of building a separate traffic protection stack for it. Concerns about routing ssh via the CDN are I think the blocker for that still. Breaking all ssh clones of gerrit repos still seems like a poor solution. [19:59:40] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Angelkay - https://phabricator.wikimedia.org/T406776#11256812 (10Pppery) [20:01:28] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-maps-master02 in project deployment-prep - https://phabricator.wikimedia.org/T405888#11256823 (10bd808) 05Open→03Resolved a:03Muehlenhoff Fixed by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1194156. Thanks @Mu... [20:13:56] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Gerrit connection troubles and ERR_CONNECTION_RESET - https://phabricator.wikimedia.org/T406774#11256868 (10hashar) 05Open→03Resolved a:03hashar I have elected to put mod_qos to be in log only mode which is someth... [20:24:46] bd808: yup the CDN brings a lot of benefits for sure but LVS/Gerrit 29418 are a difficiulty [20:25:11] my original idea was to relocate gitiles to say gitiles.wikimedia.org and move that behind the CDN [20:25:40] 10Scap (SpiderPig 🕸️): Text pasted into the SpiderPig patch select bar should be trimmed - https://phabricator.wikimedia.org/T406780 (10Tgr) 03NEW [20:25:43] with the old url on gerrit.wikimedia.org/r/plugins/gitiles to redirect to the new host (and issuing a simple redirect should be much of a trouble) [20:25:51] I guess I will revisit that and push forward with that idea [20:26:09] at least that will split the two softs [20:26:29] I am off! [20:30:03] maintenance-disconnect-full-disks build 743978 integration-agent-docker-1043 (/: 35%, /srv: 100%, /var/lib/docker: 36%): OFFLINE due to disk space [20:35:03] maintenance-disconnect-full-disks build 743979 integration-agent-docker-1043 (/: 35%, /srv: 30%, /var/lib/docker: 35%): RECOVERY disk space OK [20:47:37] (03update) 10dancy: backport.py: Revise relevant dependency selection algorithm [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1010 (https://phabricator.wikimedia.org/T365146 https://phabricator.wikimedia.org/T371611 https://phabricator.wikimedia.org/T388025) [21:03:30] (03update) 10dancy: backport.py: Revise relevant dependency selection algorithm [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1010 (https://phabricator.wikimedia.org/T365146 https://phabricator.wikimedia.org/T371611 https://phabricator.wikimedia.org/T388025) [22:13:10] (03open) 10dduvall: variables: `kokkuri variables env` command [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/145 [22:27:17] (03update) 10dduvall: variables: `kokkuri variables env` command [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/145 [22:27:17] (03update) 10dduvall: variables: `kokkuri variables env` command [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/145 [22:27:20] (03update) 10dduvall: variables: `kokkuri variables env` command [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/145 [22:28:14] (03update) 10dduvall: variables: `kokkuri variables env` command [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/145 [22:28:16] (03update) 10dduvall: variables: `kokkuri variables env` command [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/145 [22:31:54] (03close) 10dduvall: jwt: Consider `project_path` and `job_project_path` claims [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/107 [22:34:34] (03update) 10dduvall: variables: `kokkuri variables env` command [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/145 [22:34:35] (03update) 10dduvall: variables: `kokkuri variables env` command [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/145 [22:34:38] (03update) 10dduvall: variables: `kokkuri variables env` command [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/145 [22:54:55] (03update) 10dduvall: variables: `kokkuri variables env` command [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/145 [22:54:56] (03update) 10dduvall: variables: `kokkuri variables env` command [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/145 [22:54:58] (03update) 10dduvall: variables: `kokkuri variables env` command [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/145