[00:30:25] (03CR) 10Jforrester: "Running `docker-pkg -c dockerfiles/config.yaml --info update --reason "Re-build to add security updates apt repositories" --version 0.2.0-" [integration/config] - 10https://gerrit.wikimedia.org/r/724501 (owner: 10Hashar) [01:15:10] 10MediaWiki-Releasing, 10Documentation, 10MW-1.31-release: Formally EOL REL1_31 - https://phabricator.wikimedia.org/T279858 (10Reedy) [01:15:27] 10MediaWiki-Releasing, 10Documentation, 10MW-1.31-release: Formally EOL REL1_31 - https://phabricator.wikimedia.org/T279858 (10Reedy) [01:17:09] 10MediaWiki-Releasing, 10Documentation, 10MW-1.31-release: Formally EOL REL1_31 - https://phabricator.wikimedia.org/T279858 (10Reedy) a:03Reedy [01:17:34] 10MediaWiki-Releasing, 10Documentation, 10MW-1.31-release: Formally EOL REL1_31 - https://phabricator.wikimedia.org/T279858 (10Reedy) [01:17:42] 10MediaWiki-Releasing, 10Documentation, 10MW-1.31-release: Formally EOL REL1_31 - https://phabricator.wikimedia.org/T279858 (10Reedy) [01:26:24] 10MediaWiki-Releasing, 10Documentation, 10MW-1.31-release: Write and send formal EOL email for REL1_31 - https://phabricator.wikimedia.org/T292036 (10Reedy) [01:28:01] (03PS4) 10Reedy: Zuul: Drop CI support for REL1_31 branch [integration/config] - 10https://gerrit.wikimedia.org/r/683031 (https://phabricator.wikimedia.org/T281294) (owner: 10Jforrester) [01:28:06] (03CR) 10Reedy: "zomg it rebases" [integration/config] - 10https://gerrit.wikimedia.org/r/683031 (https://phabricator.wikimedia.org/T281294) (owner: 10Jforrester) [01:31:09] 10MediaWiki-Releasing, 10Documentation, 10MW-1.31-release: Write and send formal EOL email for REL1_31 - https://phabricator.wikimedia.org/T292036 (10Reedy) [03:10:21] (03PS1) 10Subramanya Sastry: div.catlinks moved outside mw-body-content on arwiki [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/724580 [05:34:33] 10Release-Engineering-Team (Radar), 10Analytics-Radar, 10WikimediaDebug, 10observability, and 4 others: Create a separate 'mwdebug' cluster - https://phabricator.wikimedia.org/T262202 (10jijiki) @Krinkle I agree that we should come up with a complete solution for this. I will close this task and we can con... [05:36:31] 10Release-Engineering-Team (Radar), 10Analytics-Radar, 10WikimediaDebug, 10observability, and 4 others: Create a separate 'mwdebug' cluster - https://phabricator.wikimedia.org/T262202 (10jijiki) 05Open→03Resolved p:05Triage→03Medium [05:43:54] 10Release-Engineering-Team (Radar), 10Analytics-Radar, 10WikimediaDebug, 10observability, and 4 others: Create a separate 'mwdebug' cluster - https://phabricator.wikimedia.org/T262202 (10Joe) For the record, the mwdebug cluster on kubernetes has its own servergroup. [05:46:21] 10Release-Engineering-Team (Next), 10MW-on-K8s, 10serviceops: Provide an mwdebug functionality on kubernetes - https://phabricator.wikimedia.org/T276994 (10jijiki) p:05Triage→03Medium [07:08:23] (03CR) 10Hashar: dockerfiles: rebuild Bullseye images for security repos (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/724501 (owner: 10Hashar) [07:33:28] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review: scap's canary check gives confusing logstash link - https://phabricator.wikimedia.org/T291870 (10hashar) Digging result: Puppet has modules/scap/templates/scap.cfg.erb:60:canary_service: mwdeploy which put in our scap.cfg: `canary_service: mwdeploy` W... [07:39:43] 10Phabricator: HTTP 502 "Next hop connection failed" when trying to upload a file - https://phabricator.wikimedia.org/T290321 (10dcaro) 05Open→03Resolved a:03dcaro This was caused by a very slow internet connection from my laptop. There might be some timeout in the path that breaks too early and makes the... [07:49:56] 10Release-Engineering-Team, 10Scap, 10serviceops, 10Patch-For-Review: scap's canary check gives confusing logstash link - https://phabricator.wikimedia.org/T291870 (10hashar) The servergroup comes from https://gerrit.wikimedia.org/r/c/operations/puppet/+/546448 which set the environment variable in Apache.... [08:10:27] 10Scap, 10Patch-For-Review: Scap error when deploying kartotherian - https://phabricator.wikimedia.org/T291990 (10Jgiannelos) Thanks @dancy [08:19:28] 10Release-Engineering-Team, 10Scap, 10serviceops: scap's canary check gives confusing logstash link - https://phabricator.wikimedia.org/T291870 (10hashar) a:03hashar [08:19:54] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T281166 (10kostajh) [08:37:36] 10Gerrit, 10Wikidata, 10Wikidata-Campsite, 10wdwb-tech: wikidata-dev instances causing git "Internal error during upload-pack" every 5 minutes - https://phabricator.wikimedia.org/T287459 (10Lucas_Werkmeister_WMDE) The wb-reconcile instance is already gone. The fedprops-euspecies instance still exists, but... [09:37:29] (03CR) 10Hashar: [C: 03+1] Zuul: Drop CI support for REL1_31 branch [integration/config] - 10https://gerrit.wikimedia.org/r/683031 (https://phabricator.wikimedia.org/T281294) (owner: 10Jforrester) [09:39:24] 10Gerrit, 10Wikidata, 10Wikidata-Campsite, 10wdwb-tech: wikidata-dev instances causing git "Internal error during upload-pack" every 5 minutes - https://phabricator.wikimedia.org/T287459 (10hashar) 05Open→03Resolved a:03hashar It was still an issue last time I checked but I have been unable to reprod... [09:51:20] 10Deployments, 10Sustainability (Incident Followup): Local private files on deployment host should be backed up somewhere - https://phabricator.wikimedia.org/T69818 (10jcrespo) This is the contents on backups for deploy1002 as of yesterday (under NDA, because I don't know if the file names are sensitive): (Go... [09:53:38] 10Deployments, 10Sustainability (Incident Followup): Local private files on deployment host should be backed up somewhere - https://phabricator.wikimedia.org/T69818 (10jcrespo) @Krinkle before closing this, could you ask me to restore a file at some point in time (within the last 2 months, and we don't have to... [10:35:20] Hi releng-team. After the lastest gitlab roll out, I can't seem to find CI Lint anymore in the UI. For example, https://gitlab.wikimedia.org/gmodena/platform-airflow-dags/-/ci/lint now gives me a 404. Is it a known/expected thing? [10:40:51] 10Scap, 10Patch-For-Review: Scap error when deploying kartotherian - https://phabricator.wikimedia.org/T291990 (10Jgiannelos) I tried to run another scap deployment but i am getting the same error. From what I understand, the problem must be on the maps node side (scap target) where the installed package is sc... [10:47:06] (03CR) 10Majavah: [C: 04-1] fix: template.py Python 3 fallout (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/724527 (https://phabricator.wikimedia.org/T291990) (owner: 10Ahmon Dancy) [11:14:47] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10Performance-Team, 10SRE, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10jijiki) >>! In T290536#7383383, @akosiaris wrote: >>>! In T290536#7383272, @jijiki wrote: > > > That's currently my preferred w... [11:34:56] 10Scap, 10Patch-For-Review: Scap error when deploying kartotherian - https://phabricator.wikimedia.org/T291990 (10Jgiannelos) Just a heads up, this is currently blocking us from pushing a couple of changes to kartotherian to test our prod environments in k8s which is currently our main task. [11:40:22] 10Scap, 10Patch-For-Review: Scap error when deploying kartotherian - https://phabricator.wikimedia.org/T291990 (10Jgiannelos) p:05Triage→03High [11:41:11] 10Scap, 10Patch-For-Review: Scap error when deploying kartotherian - https://phabricator.wikimedia.org/T291990 (10Jgiannelos) p:05High→03Triage [11:42:22] 10Scap, 10serviceops, 10Patch-For-Review: Scap error when deploying kartotherian - https://phabricator.wikimedia.org/T291990 (10jijiki) >>! In T291990#7387469, @Jgiannelos wrote: > Just a heads up, this is currently blocking us from pushing a couple of changes to kartotherian to test our prod environments in... [11:42:40] 10Scap, 10serviceops, 10Patch-For-Review: Scap error when deploying kartotherian - https://phabricator.wikimedia.org/T291990 (10jijiki) p:05Triage→03High [12:12:58] 10Phabricator: Cover dashboards and panels in weekly Phab changes email - https://phabricator.wikimedia.org/T292062 (10Aklapper) p:05Triage→03Low [12:18:08] 10Phabricator: Cover dashboards and panels in weekly Phab changes email - https://phabricator.wikimedia.org/T292062 (10Aklapper) Latest dashboard changes: ` SELECT CONCAT("https://phabricator.wikimedia.org/dashboard/view/", d.id) AS dashboard, d.phid AS dashboardPHID, u.userName AS author, d.viewPolicy AS viewP... [12:34:36] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T281166 (10kostajh) [13:26:19] gmodena: hrm - that path loads for me both as an admin and if i impersonate your user account. [14:06:20] brennen mmm... I don't get 404s anymore either :| [14:06:53] gmodena: odd. i don't see anything obvious in the top-level application log, but i can dig a bit more. [14:19:21] 10Release-Engineering-Team (Next), 10wikimedia.biterg.io, 10GitLab (Integrations): Bitergia gitlab read access for metrics - https://phabricator.wikimedia.org/T290247 (10brennen) [14:19:39] 10Release-Engineering-Team (Done by Wed 06 Oct), 10Infrastructure-Foundations, 10Security-Team, 10CAS-SSO, and 2 others: Open gitlab.wikimedia.org to all users with Wikimedia developer accounts - https://phabricator.wikimedia.org/T288162 (10brennen) 05Open→03Resolved a:03brennen The instance is now o... [14:50:27] 10MediaWiki-Releasing, 10Security: Write and send pre-release announcements for MediaWiki 1.31.16/1.35.4/1.36.2 - https://phabricator.wikimedia.org/T285407 (10Reedy) https://lists.wikimedia.org/hyperkitty/list/mediawiki-announce@lists.wikimedia.org/thread/RXMBJ3IZPSHLPFPDKGR5MXU3EONHRXE7/ [14:50:34] 10MediaWiki-Releasing, 10Security: Write and send pre-release announcements for MediaWiki 1.31.16/1.35.4/1.36.2 - https://phabricator.wikimedia.org/T285407 (10Reedy) [14:51:59] 10MediaWiki-Releasing, 10Documentation, 10MW-1.31-release: Write and send formal EOL email for REL1_31 - https://phabricator.wikimedia.org/T292036 (10Reedy) [14:52:37] 10MediaWiki-Releasing, 10Documentation, 10MW-1.31-release: Write and send formal EOL email for REL1_31 - https://phabricator.wikimedia.org/T292036 (10Reedy) [14:53:02] 10MediaWiki-Releasing, 10Documentation, 10MW-1.31-release: Write and send formal EOL email for REL1_31 - https://phabricator.wikimedia.org/T292036 (10Reedy) [14:53:24] 10MediaWiki-Releasing, 10Documentation, 10MW-1.31-release: Write and send formal EOL email for REL1_31 - https://phabricator.wikimedia.org/T292036 (10Reedy) [14:54:24] 10MediaWiki-Releasing, 10Documentation, 10MW-1.31-release: Write and send formal EOL email for REL1_31 - https://phabricator.wikimedia.org/T292036 (10Reedy) 05Open→03Resolved [14:54:30] 10MediaWiki-Releasing, 10Documentation, 10MW-1.31-release: Formally EOL REL1_31 - https://phabricator.wikimedia.org/T279858 (10Reedy) [14:55:16] 10MediaWiki-Releasing, 10Documentation, 10MW-1.31-release: Formally EOL REL1_31 - https://phabricator.wikimedia.org/T279858 (10Reedy) 05Open→03In progress [14:55:20] 10Project-Admins: Archive MSSQL project - https://phabricator.wikimedia.org/T230583 (10Reedy) [14:55:24] 10Continuous-Integration-Infrastructure, 10MW-1.31-release, 10Patch-For-Review: Drop CI for REL1_31 branch once it's EOL - https://phabricator.wikimedia.org/T281294 (10Reedy) [14:55:28] 10Project-Admins: Archive Oracle Database project - https://phabricator.wikimedia.org/T230582 (10Reedy) [14:55:53] 10MediaWiki-Releasing, 10Documentation, 10MW-1.31-release: Formally EOL REL1_31 - https://phabricator.wikimedia.org/T279858 (10Reedy) [15:03:02] !log gitlab: pausing shared runners while working through access-level implications [15:03:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:07:28] (03CR) 10Ahmon Dancy: fix: template.py Python 3 fallout (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/724527 (https://phabricator.wikimedia.org/T291990) (owner: 10Ahmon Dancy) [15:37:35] 10Project-Admins: Create project tag for - https://phabricator.wikimedia.org/T292081 (10Madalina) [15:42:34] (03PS2) 10Hashar: Revert "dockerfiles: [civicrm] Don't provide the Wikimedia fork of civicrm-buildkit" [integration/config] - 10https://gerrit.wikimedia.org/r/724215 (https://phabricator.wikimedia.org/T277500) (owner: 10Ejegg) [15:43:41] (03CR) 10Hashar: [C: 03+2] "Amended to make it a revert of 5188def and to attach this change to T277500. I will rebuild the image and update the job." [integration/config] - 10https://gerrit.wikimedia.org/r/724215 (https://phabricator.wikimedia.org/T277500) (owner: 10Ejegg) [15:45:38] (03Merged) 10jenkins-bot: Revert "dockerfiles: [civicrm] Don't provide the Wikimedia fork of civicrm-buildkit" [integration/config] - 10https://gerrit.wikimedia.org/r/724215 (https://phabricator.wikimedia.org/T277500) (owner: 10Ejegg) [15:45:52] (03PS2) 10Ahmon Dancy: fix: template.py Python 3 fallout [tools/scap] - 10https://gerrit.wikimedia.org/r/724527 (https://phabricator.wikimedia.org/T291990) [15:48:02] ejegg: I finally reached your civicrm buildkit change! :) [15:51:08] 10Release-Engineering-Team, 10Scap, 10serviceops, 10Patch-For-Review: scap's canary check gives confusing logstash link - https://phabricator.wikimedia.org/T291870 (10Krinkle) If I understand correctly, the now-removed approach involved a single Logstash link that would enumerate all the canaries. The cana... [15:51:48] (03CR) 10Hashar: [C: 03+1] "Thcipriani typically run that script iirc so will let him give the final call." [tools/release] - 10https://gerrit.wikimedia.org/r/722885 (owner: 10Jdrewniak) [15:52:38] !log Successfully published image docker-registry.discovery.wmnet/releng/civicrm:0.2.2 # T277500 [15:52:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:52:42] T277500: Update Fundraising tech CI image to use upstream buildkit, no symlink for civicrm - https://phabricator.wikimedia.org/T277500 [15:57:50] (03PS1) 10Hashar: jjb: use proper path for Civicrm buildkit [integration/config] - 10https://gerrit.wikimedia.org/r/724772 (https://phabricator.wikimedia.org/T277500) [16:00:28] (03CR) 10Ahmon Dancy: Added tox.ini with initial tests (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/724134 (owner: 10Ahmon Dancy) [16:03:47] thanks hashar ! [16:03:55] (03CR) 10Arlolra: [C: 03+2] div.catlinks moved outside mw-body-content on arwiki [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/724580 (owner: 10Subramanya Sastry) [16:04:01] ejegg: something is running at https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm-docker/6110/console [16:04:13] it has reached phpunit, so I am guessing the buildkit is working ;) [16:05:22] yep, looks good for now, hashar. I guess maybe later we can move the buildkit clone into the civi setup script :) [16:08:04] (03CR) 10Hashar: [C: 03+2] "Thanks! And that shellcheck wrap up in tox.ini is a great system!" [tools/train-dev] - 10https://gerrit.wikimedia.org/r/724134 (owner: 10Ahmon Dancy) [16:08:30] (03Merged) 10jenkins-bot: Added tox.ini with initial tests [tools/train-dev] - 10https://gerrit.wikimedia.org/r/724134 (owner: 10Ahmon Dancy) [16:08:39] Thanks hashar! [16:09:20] (03CR) 10Hashar: [C: 03+2] "I did a recheck on https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/724582 and it seems to work fine now." [integration/config] - 10https://gerrit.wikimedia.org/r/724772 (https://phabricator.wikimedia.org/T277500) (owner: 10Hashar) [16:09:51] ejegg: I think the original idea was that when one send a patch to civicrm-buildkit repo in Gerrit, we ensure that the job works [16:10:12] (03CR) 10Ahmon Dancy: Access train-dev git server instead of gerrit (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/723267 (owner: 10Ahmon Dancy) [16:10:24] now that you get access on the upstream repo, I guess it is simpler to use composer to fetch it or eventually a git submodule [16:10:30] and archive our civicrm-buildkit repo [16:10:52] it passed! [16:10:56] dancy: you are welcome ! [16:11:02] (03Merged) 10jenkins-bot: jjb: use proper path for Civicrm buildkit [integration/config] - 10https://gerrit.wikimedia.org/r/724772 (https://phabricator.wikimedia.org/T277500) (owner: 10Hashar) [16:11:34] 10Release-Engineering-Team (Radar), 10Infrastructure-Foundations, 10CAS-SSO, 10GitLab (Auth & Access): Attempting to login to gitlab.wikimedia.org sometimes results in CAS 500 Internal Server Error - https://phabricator.wikimedia.org/T291964 (10jbond) it seems like the error page sometimes gets blocked by... [16:11:54] (03CR) 10Ahmon Dancy: Added tox.ini with initial tests (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/724134 (owner: 10Ahmon Dancy) [16:11:56] dancy: funnily ebernhardson has added `shellcheck` to the releng/tox* image which end up stick to whatever version is provided by Debian. Shipping it via shellcheck-py is nice (though I usually don't really like shipping binaries this way) [16:13:48] ejegg: i am subscribed to the task ( https://phabricator.wikimedia.org/T277500 ), then our timezones don't nicely align but releng people should be able to assist with docker image build / Jenkins job deployment [16:14:10] 10MediaWiki-Releasing, 10Documentation: Formally EOL REL1_31 - https://phabricator.wikimedia.org/T279858 (10Reedy) p:05Triage→03Medium [16:14:59] 10Release-Engineering-Team (Radar), 10Infrastructure-Foundations, 10CAS-SSO, 10GitLab (Auth & Access): Attempting to login to gitlab.wikimedia.org sometimes results in CAS 500 Internal Server Error - https://phabricator.wikimedia.org/T291964 (10jbond) noting here that @RhinosF1 also reported this issue via... [16:15:44] 10Continuous-Integration-Infrastructure, 10Composer, 10Patch-For-Review: Re-build CI containers with Composer 2.0 - https://phabricator.wikimedia.org/T279857 (10Reedy) [16:15:54] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Drop CI for REL1_31 branch once it's EOL - https://phabricator.wikimedia.org/T281294 (10Reedy) 05Open→03In progress p:05Triage→03Medium [16:24:22] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10MediaWiki Train Development Environment: Add CI to mediawiki/tools/train-dev - https://phabricator.wikimedia.org/T259586 (10dancy) 05Open→03Resolved [16:25:10] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10MediaWiki Train Development Environment: Add CI to mediawiki/tools/train-dev - https://phabricator.wikimedia.org/T259586 (10dancy) [16:25:40] dancy: a job well done! [16:25:55] Thx. Should have done it long ago! [16:26:10] but train-dev needed some evolution first. [16:31:23] brennen ack. I can let you know if this behaviour happens again. [16:32:24] brennen btw, I've been working with Gitlab for some PoC/exploratory work and the experience has been *delightful*. I'm a huge fan already :) [16:32:43] gmodena: glad to hear it. :) [16:33:59] the auth stuff has been sort of finicky and there are some corners of things that aren't very well documented, but _in general_ it's been a fairly pleasant system to work on. [16:39:21] (03CR) 10Thcipriani: [C: 03+2] "works for me if it works for @Jdrewniak" [tools/release] - 10https://gerrit.wikimedia.org/r/722885 (owner: 10Jdrewniak) [16:41:14] (03Merged) 10jenkins-bot: deployments-calendar: change portals deploy to a US friendly time. [tools/release] - 10https://gerrit.wikimedia.org/r/722885 (owner: 10Jdrewniak) [16:43:54] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: CiviCRM CI tests failing due to half-resolution of T277500 - https://phabricator.wikimedia.org/T291897 (10hashar) 05Open→03Resolved https://gerrit.wikimedia.org/r/c/integra... [17:00:51] 10Release-Engineering-Team, 10Security-Team, 10GitLab (CI & Job Runners), 10User-brennen: Limit GitLab shared runners to trusted contributors - https://phabricator.wikimedia.org/T292094 (10brennen) [17:39:27] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T281166 (10Legoktm) Someone who isn't me needs to figure out what to do with {T280806} otherwise a bunch of bots are going to break when it rolls out... [17:43:31] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T281166 (10jeena) @Legoktm just want to double check whether the above should be a blocker to rolling the train forward today? [17:44:01] 10Release-Engineering-Team, 10Security-Team, 10GitLab (CI & Job Runners), 10User-brennen: Limit GitLab shared runners to trusted contributors - https://phabricator.wikimedia.org/T292094 (10brennen) [17:48:43] 10Release-Engineering-Team (Done by Wed 06 Oct), 10Security-Team, 10GitLab (CI & Job Runners), 10Patch-For-Review, and 2 others: Limit GitLab shared runners to images from Wikimedia Docker registry - https://phabricator.wikimedia.org/T291978 (10Legoktm) Is this intended to be a security limitation or perfo... [17:51:01] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T281166 (10Legoktm) Personally, I think so. But I've already reverted this twice and already said last week I wasn't going to again, someone else need... [17:55:51] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10Platform Engineering Roadmap, 10User-Daniel: Convert static mediawiki configuration to form more suitable for containers - https://phabricator.wikimedia.org/T263166 (10mmodell) >>! In T263166#7370542, @Legoktm wrote: > Broadly: > * SRE-level stuff, like ser... [18:01:36] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T281166 (10jeena) [18:02:43] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T281166 (10jeena) [18:04:43] 10Release-Engineering-Team (Done by Wed 06 Oct), 10Security-Team, 10GitLab (CI & Job Runners), 10Patch-For-Review, and 2 others: Limit GitLab shared runners to images from Wikimedia Docker registry - https://phabricator.wikimedia.org/T291978 (10brennen) > At that point, what's the difference in allowing up... [18:06:25] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T281166 (10thcipriani) >>! In T281166#7388882, @Legoktm wrote: > Someone who isn't me needs to figure out what to do with {T280806} otherwise a bunch... [18:09:33] 10Release-Engineering-Team (Done by Wed 06 Oct), 10Security-Team, 10GitLab (CI & Job Runners), 10Patch-For-Review, and 2 others: Limit GitLab shared runners to images from Wikimedia Docker registry - https://phabricator.wikimedia.org/T291978 (10mmodell) I think that many of the upstream docker images shoul... [18:16:02] Hey all, apparently there are no actual deployers available for todays backport window [18:16:17] (03PS1) 10Jdlrobson: Remove Niharika from available deployers [tools/release] - 10https://gerrit.wikimedia.org/r/724810 [18:16:23] oh ^ was about to ask if we should do that [18:16:47] is there a way we can do better at making sure the deployers assigned to the window available? [18:18:10] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T281166 (10Jdlrobson) [18:18:59] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T281166 (10Jdlrobson) T292071 is a train blocker now, (following the revert on T292030). Patch is ready, it just needs backporting (https://gerrit.wi... [18:19:46] 10Release-Engineering-Team (Doing), 10Security-Team, 10GitLab (CI & Job Runners), 10User-brennen: Limit GitLab shared runners to trusted contributors - https://phabricator.wikimedia.org/T292094 (10brennen) [18:20:03] 10Release-Engineering-Team (Doing), 10Security-Team, 10GitLab (CI & Job Runners), 10User-brennen: Limit GitLab shared runners to trusted contributors - https://phabricator.wikimedia.org/T292094 (10brennen) a:03brennen [18:20:18] ottomata: email them nicely and ask? [18:20:53] I can help with a deploy if needed. [18:21:04] dancy: see -operations [18:21:07] ok [18:22:24] thank you. [18:22:24] RhinosF1 email releng? or who should we ask? [18:34:37] ottomata: the deployers [18:34:48] Martin is nearly always available [18:34:54] Might be a one off [18:54:03] 10Release-Engineering-Team (Done by Wed 06 Oct), 10Security-Team, 10GitLab (CI & Job Runners), 10Patch-For-Review, and 2 others: Limit GitLab shared runners to images from Wikimedia Docker registry - https://phabricator.wikimedia.org/T291978 (10sbassett) >>! In T291978#7388954, @brennen wrote: > I think ma... [19:01:24] 10Release-Engineering-Team (Done by Fri 03 Sep), 10GitLab, 10Security Team AppSec, 10Security-Team, and 2 others: Create Security Team group within gitlab.wikimedia.org - https://phabricator.wikimedia.org/T289292 (10sbassett) [19:24:42] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10Platform Engineering Roadmap, 10User-Daniel: Convert static mediawiki configuration to form more suitable for containers - https://phabricator.wikimedia.org/T263166 (10daniel) I'm currently doing an exploration of how we can make more flexible (and at the s... [19:34:02] 10Release-Engineering-Team (Doing), 10Security-Team, 10GitLab (CI & Job Runners), 10User-brennen: Limit GitLab shared runners to trusted contributors - https://phabricator.wikimedia.org/T292094 (10sbassett) Hey @brennen - The #security-team will have a chat about this at our upcoming clinic but just some... [19:47:46] 10Phabricator: Update Herald (H260) to include upcoming CommTech sprint milestones (5573, 5574, 5575, 5576) - https://phabricator.wikimedia.org/T292112 (10ldelench_wmf) [20:05:25] 10Release-Engineering-Team (Doing), 10Security-Team, 10GitLab (CI & Job Runners), 10User-brennen: Limit GitLab shared runners to trusted contributors - https://phabricator.wikimedia.org/T292094 (10brennen) [20:09:22] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T281166 (10Jdlrobson) [21:36:02] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T281166 (10jeena) [21:38:04] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T281166 (10jeena) I filed T292126 as a train blocker. If it should not block the train please correct me. [21:43:08] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [21:49:22] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T281166 (10Legoktm) >>! In T281166#7389640, @jeena wrote: > I filed T292126 as a train blocker. If it should not block the train please correct me. I... [21:58:44] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T281166 (10Legoktm) [21:59:46] the postmerge queue in zuul looks sad. Oldest job in the queue is idle for 34 minutes apparently waiting on an excutor [22:00:34] postmerge is deprioritized over test I believe [22:01:04] and where my docker artifacts get built :(( [22:07:02] The scap run in the beta-scap-sync-world job has bad config. It is trying to query deployment-logstash03.deployment-prep.eqiad1.wikimedia.cloud for canary errors. That host was replaced by deployment-logstash04.deployment-prep.eqiad1.wikimedia.cloud. [22:08:49] "logstash.svc.deployment-prep.eqiad1.wikimedia.cloud" may be an even better target for the scap config. That appears to be a custom DNS entry pointing to all 3 logstash instances inside the beta cluster. [22:14:19] dduvall, jeena: have there been any discussions about making a separate zuul queue for pipeline-publish jobs? [22:14:38] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [22:15:15] sitting behind a bunch of code coverage publish jobs waiting for a deployable artifact is frustrating [22:15:47] bd808: oy. sounds frustrating :/ [22:16:03] we haven't discussed it but it's certainly possible [22:18:11] if you're up for writing a zuul layout patch, i can review. otherwise, file a task and we'll triage it for our next sprint (we're doing those now!) [22:18:29] sprints! madness :) [22:18:39] it's a new concept i know [22:19:08] we used to do flails. now we do sprints [22:19:16] I'll take a look and try one or the other. I don't have anyone to sword fight with while I wait for the compiler :) [22:19:23] 10Release-Engineering-Team (Done by Wed 06 Oct), 10Security-Team, 10GitLab (CI & Job Runners), 10Patch-For-Review, and 2 others: Limit GitLab shared runners to images from Wikimedia Docker registry - https://phabricator.wikimedia.org/T291978 (10Legoktm) I'm not really sure my questions got answered, specif... [22:20:14] :) [22:25:11] 10Release-Engineering-Team (Done by Wed 06 Oct), 10Security-Team, 10GitLab (CI & Job Runners), 10Patch-For-Review, and 2 others: Limit GitLab shared runners to images from Wikimedia Docker registry - https://phabricator.wikimedia.org/T291978 (10Legoktm) Thinking a bit more... I don't know if GitLab support... [22:28:00] 10Release-Engineering-Team (Doing), 10Security-Team, 10GitLab (CI & Job Runners), 10User-brennen: Limit GitLab shared runners to trusted contributors - https://phabricator.wikimedia.org/T292094 (10Legoktm) This seems like an absolute necessity unfortunately, note that we already have this in place for Gerr... [22:37:41] 10Release-Engineering-Team (Doing), 10Security-Team, 10GitLab (CI & Job Runners), 10User-brennen: Limit GitLab shared runners to trusted contributors - https://phabricator.wikimedia.org/T292094 (10mmodell) > Mark all new users as "External" and have a process for unchecking the box. In some ways this may b... [22:40:05] 10Continuous-Integration-Infrastructure, 10Zuul: Separate zuul queue for pipelinelib publish jobs - https://phabricator.wikimedia.org/T292130 (10bd808) [22:40:17] 10Release-Engineering-Team (Done by Wed 06 Oct), 10Security-Team, 10GitLab (CI & Job Runners), 10Patch-For-Review, and 2 others: Limit GitLab shared runners to images from Wikimedia Docker registry - https://phabricator.wikimedia.org/T291978 (10brennen) To answer the original question: > Will we be allowe... [22:41:09] 10Continuous-Integration-Infrastructure, 10Zuul: Separate zuul queue for pipelinelib publish jobs - https://phabricator.wikimedia.org/T292130 (10bd808) I looked briefly at zuul/layout.yaml to see if I knew enough to attempt a patch for this. It turns out that I do not. ;) [22:49:27] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10Platform Engineering Roadmap, 10User-Daniel: Convert static mediawiki configuration to form more suitable for containers - https://phabricator.wikimedia.org/T263166 (10mmodell) >>! In T263166#7389233, @daniel wrote: > I plan to submit a more detailed propos... [23:31:42] 10Release-Engineering-Team (Doing), 10Security-Team, 10GitLab (CI & Job Runners), 10User-brennen: Limit GitLab shared runners to trusted contributors - https://phabricator.wikimedia.org/T292094 (10brennen) @thcipriani found something that looks like one potential hack-around for disabling CI: [[https://doc... [23:34:38] 10Release-Engineering-Team (Doing), 10Security-Team, 10GitLab (CI & Job Runners), 10User-brennen: Limit GitLab shared runners to trusted contributors - https://phabricator.wikimedia.org/T292094 (10mmodell) > If there’s an error or the request times out, the pipeline is accepted. That default sure seems ba...