[00:03:04] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10GitLab (Initialization), 10User-brennen: SSH Access of Git data in GitLab - https://phabricator.wikimedia.org/T276148 (10Sergey.Trofimovsky.SF) Gitlab takes a bit of an opposite approach with this. Gitlab server manages its own user key datab... [01:13:00] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10serviceops: Replace production deployment servers and update them to Buster - https://phabricator.wikimedia.org/T265963 (10Dzahn) 05Open→03Resolved [01:13:03] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) [05:56:10] 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops: legoktm can't build CI docker images without using root because he's no longer in contint-admins - https://phabricator.wikimedia.org/T275731 (10Joe) >>! In T275731#6871524, @hashar wrote: > I would rather have cherry picked people that knows about... [06:45:05] 10Release-Engineering-Team-TODO, 10DNS, 10SRE, 10Traffic, and 3 others: DNS for GitLab - https://phabricator.wikimedia.org/T276170 (10JMeybohm) p:05Triage→03Medium [07:14:40] 10Gerrit, 10Release-Engineering-Team: Can't delete weird ref using git in Gerrit - https://phabricator.wikimedia.org/T275946 (10Legoktm) > I am entitled to ignore that issue and delete the reference directly from disk. This sounds reasonable to me. We should also adjust the ACL to make sure it's not possible... [08:19:37] (03CR) 10Gehel: [C: 04-1] [DNM] Replace the java-codehealth-patch by adding step in the standard job (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/666869 (https://phabricator.wikimedia.org/T264873) (owner: 10Gehel) [08:52:41] (03PS33) 10Awight: Parallelism as a command object; parallelize short PHPUnits [integration/quibble] - 10https://gerrit.wikimedia.org/r/587885 (https://phabricator.wikimedia.org/T235449) [08:54:56] (03CR) 10Awight: "PS 33: manual rebase" [integration/quibble] - 10https://gerrit.wikimedia.org/r/587885 (https://phabricator.wikimedia.org/T235449) (owner: 10Awight) [08:56:01] (03CR) 10jerkins-bot: [V: 04-1] Parallelism as a command object; parallelize short PHPUnits [integration/quibble] - 10https://gerrit.wikimedia.org/r/587885 (https://phabricator.wikimedia.org/T235449) (owner: 10Awight) [09:27:20] (03PS6) 10Gehel: [DNM] Replace the java-codehealth-patch by adding step in the standard job [integration/config] - 10https://gerrit.wikimedia.org/r/666869 (https://phabricator.wikimedia.org/T264873) [09:28:22] (03PS7) 10Gehel: Replace the java-codehealth-patch by adding step in the standard job [integration/config] - 10https://gerrit.wikimedia.org/r/666869 (https://phabricator.wikimedia.org/T264873) [10:04:00] 00:00:16.177 [FATAL tini (7)] exec /src/mvnw failed: No such file or directory [10:04:02] of course ;D [10:07:25] hashar: which project? [10:08:32] (03PS1) 10Hashar: Add mvn wrapper [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/667833 [10:08:39] gehel: ^^ :) [10:08:51] I have deployed your patch to the java8 job for gearman-java [10:09:03] to try out what would happen when a repo is not configured / setup for sonar:sonar ;) [10:09:16] (03CR) 10Gehel: [C: 04-1] Add mvn wrapper (031 comment) [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/667833 (owner: 10Hashar) [10:09:21] so yeah hmm mvn -N io.takari:maven:0.7.7:wrapper [10:09:24] git commit -a [10:10:23] (03CR) 10Gehel: [C: 04-1] Add mvn wrapper (031 comment) [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/667833 (owner: 10Hashar) [10:10:44] minor improvement needed :) [10:12:14] (03CR) 10Hashar: Add mvn wrapper (031 comment) [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/667833 (owner: 10Hashar) [10:12:19] (03PS2) 10Hashar: Add mvn wrapper [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/667833 [10:12:24] gehel: fait ;D [10:12:47] (03CR) 10Gehel: [C: 03+1] "LGTM" [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/667833 (owner: 10Hashar) [10:14:40] gehel: and for xgboost the job in the gate pipeline did SUCCESS so the maven cache got saved ;) [10:14:49] \o/ [10:17:27] (03CR) 10Hashar: "recheck after adding sonar:sonar to the jobs ( https://gerrit.wikimedia.org/r/c/integration/config/+/666869 )" [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/667833 (owner: 10Hashar) [10:18:23] (03CR) 10Hashar: Replace the java-codehealth-patch by adding step in the standard job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/666869 (https://phabricator.wikimedia.org/T264873) (owner: 10Gehel) [10:18:52] gehel: there is a misleading comment at https://gerrit.wikimedia.org/r/c/integration/config/+/666869/7/jjb/job-templates.yaml about replacing the docker image variable in the java8 job when one wants to use java11 [10:19:10] instead we should use the child template '{name}-maven-java11-docker' [10:19:20] which is exactly the same as the java8 but with the docker image var already overwritten ;) [10:21:06] (03CR) 10jerkins-bot: [V: 04-1] Add mvn wrapper [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/667833 (owner: 10Hashar) [10:22:04] (03CR) 10Gehel: Replace the java-codehealth-patch by adding step in the standard job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/666869 (https://phabricator.wikimedia.org/T264873) (owner: 10Gehel) [10:22:06] (03PS8) 10Gehel: Replace the java-codehealth-patch by adding step in the standard job [integration/config] - 10https://gerrit.wikimedia.org/r/666869 (https://phabricator.wikimedia.org/T264873) [10:23:10] hashar: comment updated [10:23:19] not sure if that's what you meant [10:26:12] gehel: looks good [10:26:23] there is some gotcha with the entrypoint being reused which I have to look at [10:26:42] * gehel needs to get Oscar for diner and speech therapy in a few... [10:26:52] diner at 11am !: ] [10:27:07] still 15' before I have to run [10:27:11] file file, je passe un coup de polish sur ton change pas d'inquiétude [10:27:14] school stops at 11:50 [10:27:32] j'ai pas compris "entrypoint being reused" [10:27:55] pour gearman-java on définit maven_path: ./run-with-gearman.sh [10:28:09] qui n'est pas dans l'image releng/maven-java11 utilisée pour sonar:sonar [10:28:16] donc ça plante ;) [10:28:36] ofc... je t'avoue que j'ai joyeusement ignoré gearman [10:28:42] oui ;) [10:28:56] For a minute i thought it was just me being tired, but no you guys are in fact speaking another language lol [10:29:05] si tout est standardisé avec mvnw je pense qu'on peut juste l'hardcoder [10:29:23] Zppix: yeah switched to french unexpectedly 8) [10:29:36] Zppix: yeah, that happens sometimes... [10:29:47] You had me for a second heh [10:30:04] gearman is the only one not able to run directly mvnw [10:30:09] (that I know of) [10:30:39] yeah ;-\ [10:32:50] the testsuite should take care of spawning the gearman daemon [10:32:57] but I have zero clue how to achieve that ;] [10:33:48] (03CR) 10Hashar: [C: 03+2] "I have rolled back the jobs update so we no more use sonar:sonar but it is not a big deal." [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/667833 (owner: 10Hashar) [10:36:38] (03Merged) 10jenkins-bot: Add mvn wrapper [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/667833 (owner: 10Hashar) [11:00:03] (03PS9) 10Hashar: Replace the java-codehealth-patch by adding step in the standard job [integration/config] - 10https://gerrit.wikimedia.org/r/666869 (https://phabricator.wikimedia.org/T264873) (owner: 10Gehel) [11:01:57] (03CR) 10Hashar: [C: 03+1] "PS9: since gearman-java has a custom entrypoint, we needed a way to also set the entrypoint for sonar:sonar. I have done that by adding " [integration/config] - 10https://gerrit.wikimedia.org/r/666869 (https://phabricator.wikimedia.org/T264873) (owner: 10Gehel) [11:04:47] hashar: if gearman provides mvnw, I don't think we need to have the maven sonar path customizable [11:05:01] yeah that as well [11:05:06] and just require every repos to have mvnw [11:05:34] and then we can remove maven from the CI image ;] [11:50:50] That's the goal ! [12:14:00] 10MediaWiki-Codesniffer, 10phan, 10MediaWiki-Core-Testing: Use sniff or phan rule to detect violations of Stable interface policy (@internal, @stable etc.) - https://phabricator.wikimedia.org/T209394 (10Daimona) >>! In T209394#6871970, @Krinkle wrote: > See the policy for details, but an initial version coul... [12:18:33] 10Beta-Cluster-Infrastructure, 10Discovery-Search (Current work): Search broken on beta cluster wikis - https://phabricator.wikimedia.org/T275975 (10Xqt) Probably the following log helps for Investigation: headers -- ` VERBOSE pywiki:logging.py:101 headers= {'Date': 'Mon, 01 Mar 2021 09:02:51 GMT'... [12:32:33] 10Beta-Cluster-Infrastructure, 10Discovery-Search (Current work): Search broken on beta cluster wikis - https://phabricator.wikimedia.org/T275975 (10LucasWerkmeister) I tried looking for details on logstash-beta.wmflabs.org but that doesn’t seem to have any events at all (T233134, I guess?). [14:05:39] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.33 deployment blockers - https://phabricator.wikimedia.org/T274937 (10LarsWirzenius) Train is at group0. [14:08:28] (03CR) 10Gehel: Replace the java-codehealth-patch by adding step in the standard job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/666869 (https://phabricator.wikimedia.org/T264873) (owner: 10Gehel) [14:14:17] 10Beta-Cluster-Infrastructure, 10Discovery-Search (Current work): Search broken on beta cluster wikis - https://phabricator.wikimedia.org/T275975 (10Xqt) [14:24:33] gehel: lets merge your change as-is and drop the maven thing after? ;) [14:25:54] isn't this already sufficient: https://gerrit.wikimedia.org/r/c/integration/gearman-java/+/667833/ [14:26:15] Oh, it's still missing the sonar configuration [14:26:40] Meeting in 5' [14:29:34] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10GitLab (Initialization), 10User-brennen: SSH Access of Git data in GitLab - https://phabricator.wikimedia.org/T276148 (10thcipriani) [14:31:10] (03PS1) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/667873 [14:33:57] (03PS3) 10Hashar: Add rename-project plugin @ a880148 [software/gerrit] (deploy/wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/630548 (https://phabricator.wikimedia.org/T201953) [14:34:34] (03CR) 10jerkins-bot: [V: 04-1] Jenkins job validation (DO NOT SUBMIT) [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/667873 (owner: 10Hashar) [14:34:45] (03CR) 10Hashar: "I should push that eventually but renaming a repository would need some written run book I guess. Maybe I should rebuild it again." [software/gerrit] (deploy/wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/630548 (https://phabricator.wikimedia.org/T201953) (owner: 10Hashar) [14:35:47] gehel: ah yeah indeed gearman-java fails with "00:03:08.109 [ERROR] SonarQube server [http://localhost:9000] can not be reached" ;] [14:41:24] 10Release-Engineering-Team-TODO, 10DNS, 10SRE, 10Traffic, and 3 others: DNS for GitLab - https://phabricator.wikimedia.org/T276170 (10wkandek) https://gerrit.wikimedia.org/r/c/operations/puppet/+/667731 [14:56:28] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10SRE, 10GitLab (Initialization), 10User-brennen: SSH Access of Git data in GitLab - https://phabricator.wikimedia.org/T276148 (10thcipriani) Updated the ticket with details about AuthorizedKeysCommand, and removed the proposed solution (here... [14:57:55] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Analytics, 10Better Use Of Data, and 5 others: EventLogging PHP EventServiceClient should use EventBus->send(). - https://phabricator.wikimedia.org/T272863 (10Mholloway) 05Open→03Resolved a:03Mholloway [15:01:41] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10SRE, 10GitLab (Initialization), 10User-brennen: SSH Access of Git data in GitLab - https://phabricator.wikimedia.org/T276148 (10wkandek) A second sshd daemon for port 29418 concentrates all configuration for gitlab/ssh access in a separate... [15:29:11] (03PS1) 10Cory Massaro: Revert "Zuul: [mediawiki/services/function-schemata] Use bespoke pipeline jobs" [integration/config] - 10https://gerrit.wikimedia.org/r/667812 [15:51:46] James_F: legoktm: forrestbot and gerritreviewerbot both unsub'd from mediawiki-commits last night due to gmail thinking the list is spam [15:51:48] SMTP error from remote mail server after end of data: [15:51:50] 550-5.7.1 [208.80.154.21 12] Our system has detected that this message is [15:51:52] 550-5.7.1 likely unsolicited mail. To reduce the amount of spam sent to Gmail, [15:51:54] 550-5.7.1 this message has been blocked. Please visit [15:51:56] 550-5.7.1 https://support.google.com/mail/?p=UnsolicitedMessageError [15:51:58] 550 5.7.1 for more information. v191si9750579qka.355 - gsmtp [15:52:05] (unsub'd because of bounces due to gmail thinking the commit list is spam, that is) [15:54:27] greg-g: mediawiki-commits is still a thing? Last time I looked at https://lists.wikimedia.org/pipermail/mediawiki-commits/ and it was empty [15:54:32] I guess it is now on google groups? [15:57:47] well, I get -owner mail for it :) and presumably it wouldn't be getting bounces if it wasn't sending emails [15:58:07] maybe it was due to the monthly password reminder email? timing is suspect for that [15:58:18] might be [15:58:43] at least the mailman archive is empty [15:58:49] so I guess we more or less decomissioned it [15:59:14] or we broke it somehow [15:59:49] I'll wait until one of the people involved with those two projects responds :) [16:01:22] and from some old task, the reviewer bot (which adds folks as reviewer in Gerrit) is/was apparently based on receiving emails [16:01:22] bah [16:13:30] hashar: well this is weird, the setting "Archive messages?" in mailman for that list is set to No [16:27:20] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10SRE, 10GitLab (Initialization), 10User-brennen: SSH Access of Git data in GitLab - https://phabricator.wikimedia.org/T276148 (10MoritzMuehlenhoff) >>! In T276148#6874923, @wkandek wrote: > A second sshd daemon for port 29418 concentrates al... [16:34:12] 10Gerrit, 10Release-Engineering-Team: Can't delete weird ref using git in Gerrit - https://phabricator.wikimedia.org/T275946 (10hashar) With plain cgit, here is a script that prepares an upstream repository and a work one. Creates a `refs/master` reference in upstream then attempt to delete it from the work re... [16:34:19] 10Gerrit, 10Release-Engineering-Team: Can't delete weird ref using git in Gerrit - https://phabricator.wikimedia.org/T275946 (10hashar) p:05Triage→03Low [16:39:12] 10Beta-Cluster-Infrastructure: [beta cluster] UnderflowException: Ran out of captcha images - https://phabricator.wikimedia.org/T276176 (10Majavah) [16:41:17] 10Beta-Cluster-Infrastructure, 10Discovery-Search (Current work): Search broken on beta cluster wikis - https://phabricator.wikimedia.org/T275975 (10Tgr) Looking at CirrusSearch.log on beta fluorine, it seems ElasticSearch is unreachable (requests fail with 502 Bad Gateway). Which is indeed the case: ` tgr@de... [16:43:13] 10Beta-Cluster-Infrastructure, 10Discovery-Search (Current work): Search broken on beta cluster wikis - https://phabricator.wikimedia.org/T275975 (10EBernhardson) This will be fixed soonish, looks like a problem with bad elasticsearch plugin release that contains multiple versions of the same plugin, along wit... [16:44:21] 10Beta-Cluster-Infrastructure, 10Discovery-Search (Current work): Search broken on beta cluster wikis - https://phabricator.wikimedia.org/T275975 (10Tgr) ` tgr@deployment-elastic05:~$ sudo systemctl status elasticsearch ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elastics... [16:44:41] (03PS34) 10Awight: Parallelism as a command object [integration/quibble] - 10https://gerrit.wikimedia.org/r/587885 (https://phabricator.wikimedia.org/T235449) [16:44:43] (03PS9) 10Awight: Split extension and skin npm and composer tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/587888 [16:44:45] (03PS3) 10Awight: Split core npm and composer tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/588087 [16:45:32] 10Beta-Cluster-Infrastructure, 10Discovery-Search (Current work): Search broken on beta cluster wikis - https://phabricator.wikimedia.org/T275975 (10dcausse) It's related to T274204 [16:45:33] 10Beta-Cluster-Infrastructure, 10Discovery-Search (Current work): Search broken on beta cluster wikis - https://phabricator.wikimedia.org/T275975 (10dcausse) [16:47:39] (03CR) 10jerkins-bot: [V: 04-1] Split extension and skin npm and composer tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/587888 (owner: 10Awight) [16:48:05] (03CR) 10jerkins-bot: [V: 04-1] Split core npm and composer tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/588087 (owner: 10Awight) [16:48:35] (03CR) 10jerkins-bot: [V: 04-1] Parallelism as a command object [integration/quibble] - 10https://gerrit.wikimedia.org/r/587885 (https://phabricator.wikimedia.org/T235449) (owner: 10Awight) [16:52:01] Strange! I have no idea why my change would cause a dependency error without touching tox.ini or *requirements.txt. > ERROR: Package u'quibble' requires a different Python: 2.7.16 not in '>=3.5' [17:05:45] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10MW-on-K8s, 10Release Pipeline, 10serviceops-radar, and 2 others: Ensure MW PrivateSettings exist on releases server for production MW builds - https://phabricator.wikimedia.org/T276145 (10thcipriani) [17:06:57] 10Gerrit, 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)): Can't delete weird ref using git in Gerrit - https://phabricator.wikimedia.org/T275946 (10thcipriani) [17:10:05] 10Deployments, 10Release-Engineering-Team: L10n cache files building up on backup deploy hosts - https://phabricator.wikimedia.org/T275826 (10hashar) [17:10:07] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)): scap clean -delete does not have permission to delete old l10n caches - https://phabricator.wikimedia.org/T260146 (10hashar) [17:12:05] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO: L10n cache files building up on backup deploy hosts - https://phabricator.wikimedia.org/T275826 (10thcipriani) p:05Triage→03Low [17:12:56] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)): L10n cache files building up on backup deploy hosts - https://phabricator.wikimedia.org/T275826 (10thcipriani) a:03dancy [17:14:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO, 10serviceops, 10Patch-For-Review: Apache on doc1001 does not see updated PHP files for hours/days after deployment - https://phabricator.wikimedia.org/T275468 (10thcipriani) [17:14:27] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10SRE, 10GitLab (Initialization), 10User-brennen: SSH Access of Git data in GitLab - https://phabricator.wikimedia.org/T276148 (10Legoktm) I remember having to help people in `#mediawiki` whose network blocked very high ports like 29418. Mayb... [17:15:17] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO, 10serviceops, 10Patch-For-Review: Apache on doc1001 does not see updated PHP files for hours/days after deployment - https://phabricator.wikimedia.org/T275468 (10thcipriani) p:05Triage→03Medium [17:16:13] 10Release-Engineering-Team-TODO: deploy-1002.devtools.eqiad.wmflabs fails puppet - https://phabricator.wikimedia.org/T275186 (10thcipriani) p:05Triage→03Low [17:17:15] 10Release-Engineering-Team-TODO, 10GitLab, 10GitLab-Test, 10User-brennen: gitlab-test: undefined method `user_cas_omniauth_authorize_path' with omniauth_auto_sign_in_with_provider - https://phabricator.wikimedia.org/T274954 (10thcipriani) [17:17:35] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10GitLab, 10GitLab-Test, 10User-brennen: gitlab-test: undefined method `user_cas_omniauth_authorize_path' with omniauth_auto_sign_in_with_provider - https://phabricator.wikimedia.org/T274954 (10thcipriani) [17:19:27] 10Release-Engineering-Team-TODO, 10FR-Docker, 10Fundraising-Backlog, 10dev-images, and 2 others: dev-images +2 rights and Docker registry credentials for FR-Tech - https://phabricator.wikimedia.org/T274303 (10thcipriani) [17:20:30] 10Release-Engineering-Team-TODO, 10FR-Docker, 10Fundraising-Backlog, 10Gerrit-Privilege-Requests, and 3 others: dev-images +2 rights and Docker registry credentials for FR-Tech - https://phabricator.wikimedia.org/T274303 (10thcipriani) We can take care of the Gerrit request. There's probably also an associ... [17:21:26] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10SRE, 10GitLab (Initialization), 10User-brennen: SSH Access of Git data in GitLab - https://phabricator.wikimedia.org/T276148 (10brennen) > I remember having to help people in #mediawiki whose network blocked very high ports like 29418. Mayb... [17:22:01] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO: deploy mediawiki/tools/releases on deployment server with scap - https://phabricator.wikimedia.org/T274255 (10thcipriani) [17:22:33] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO: deploy mediawiki/tools/releases on deployment server with scap - https://phabricator.wikimedia.org/T274255 (10thcipriani) p:05Triage→03Low [17:24:53] 10Phabricator, 10Project-Admins, 10Diffusion-Repository-Administrators: Modify projects Wikimedia-VE-Tech and related ACL to use for WikiSP - https://phabricator.wikimedia.org/T273344 (10thcipriani) >>! In T273344#6827463, @Aklapper wrote: > Ping - could #release-engineering-team please provide input whether... [17:25:50] 10Deployments, 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Sustainability (Incident Followup), 10User-brennen: Avoid unfinished train deploys over holidays, weekends, or other stretches of no-deploy days - https://phabricator.wikimedia.org/T260401 (10thcipriani) a:03thcipriani [17:26:02] 10Deployments, 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Sustainability (Incident Followup), 10User-brennen: Avoid unfinished train deploys over holidays, weekends, or other stretches of no-deploy days - https://phabricator.wikimedia.org/T260401 (10thcipriani) p:05Triage→03Medium [17:26:43] 10Release-Engineering-Team-TODO, 10Scap: Automate scap patch - https://phabricator.wikimedia.org/T245185 (10thcipriani) [17:27:33] 10Release-Engineering-Team-TODO, 10Scap: Automate scap prep - https://phabricator.wikimedia.org/T245184 (10thcipriani) [17:29:16] 10Phabricator, 10Project-Admins, 10Diffusion-Repository-Administrators: Modify projects Wikimedia-VE-Tech and related ACL to use for WikiSP - https://phabricator.wikimedia.org/T273344 (10mmodell) FWIW the acl* naming convention is not necessary going forward. We can use the lock icon {icon lock color=red} to... [17:32:27] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10SRE: debian-glue jobs ignored error messages about libeatmydata.so in LD_PRELOAD - https://phabricator.wikimedia.org/T240430 (10LarsWirzenius) From other contexts, I *think* this needs to be fixed in Debian's own tooling. The impact is th... [17:36:19] 10Release-Engineering-Team-TODO, 10FR-Docker, 10Fundraising-Backlog, 10Gerrit-Privilege-Requests, and 3 others: dev-images +2 rights and Docker registry credentials for FR-Tech - https://phabricator.wikimedia.org/T274303 (10AndyRussG) >>! In T274303#6875436, @thcipriani wrote: > We can take care of the Ger... [17:37:16] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO, 10VisualEditor, 10Editing-team (Tracking): Firefox CI tests keep failing in VE with Firefox 68 - https://phabricator.wikimedia.org/T240955 (10thcipriani) >>! In T240955#6670656, @Jdforrester-WMF wrote: > Still unfixed. :-( Talking about thi... [17:37:34] 10Release-Engineering-Team, 10Scap: SCAP python error on successful deploy - https://phabricator.wikimedia.org/T233644 (10LarsWirzenius) I've not noticed this in any of the train deployments, I think. Certainly not in today's train deployment. Does this still happen? [17:38:30] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10SRE: debian-glue jobs ignored error messages about libeatmydata.so in LD_PRELOAD - https://phabricator.wikimedia.org/T240430 (10thcipriani) [17:38:58] greg-g: ahhhhh thanks >.< [17:39:08] I wonder if I can disable the spam filter [17:39:11] hashar: mediawiki-commits gets mail but has no archives because it would be so large [17:39:18] or maybe we move off of gmail [17:47:13] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10SRE, 10GitLab (Initialization), 10User-brennen: SSH Access of Git data in GitLab - https://phabricator.wikimedia.org/T276148 (10Sergey.Trofimovsky.SF) Allow me to stress that the SSH port for Gitlab is a long term choice. Whatever is decide... [17:47:21] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO, 10VisualEditor, 10Editing-team (Tracking): Firefox CI tests keep failing in VE with Firefox 68 - https://phabricator.wikimedia.org/T240955 (10hashar) If [[https://bugzilla.mozilla.org/show_bug.cgi?id=1464690 | bugzilla 1464690]] is actually... [17:52:04] fixed both hopefully [17:53:56] 10Release-Engineering-Team, 10Scap: SCAP python error on successful deploy - https://phabricator.wikimedia.org/T233644 (10Gilles) 05Open→03Invalid Hasn't happened to me since, I think, but I rarely deploy mediawiki things these days. Closing it on the assumption that it magically went away, we can always r... [18:16:05] legoktm: amount of mails would explain why there are no archives. Thanks! [18:16:15] off for dinner / bed etc ;) [18:16:23] o/ have a good evening [18:46:00] 10Gerrit, 10Release-Engineering-Team-TODO, 10Upstream: Can't delete weird ref using git in Gerrit - https://phabricator.wikimedia.org/T275946 (10thcipriani) a:05thcipriani→03None Noting here that I deleted this ref -- hopefully this unblocks folks: ` gerrit2@gerrit1001:/srv/gerrit/git/mediawiki/extensio... [18:50:11] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.33 deployment blockers - https://phabricator.wikimedia.org/T274937 (10thcipriani) [18:52:47] 10phan, 10MediaWiki-Core-Testing: Use sniff or phan rule to detect violations of Stable interface policy (@internal, @stable etc.) - https://phabricator.wikimedia.org/T209394 (10Daimona) >>! In T209394#6871970, @Krinkle wrote: > See the policy for details, but an initial version could be just to warn if a meth... [19:02:09] 10phan, 10MediaWiki-Core-Testing: Use sniff or phan rule to detect violations of Stable interface policy (@internal, @stable etc.) - https://phabricator.wikimedia.org/T209394 (10Krinkle) >>! In T209394#6874363, @Daimona wrote: > Phan has several issue types for this, […] Long story short, phan assumes that the... [19:17:45] 10Gerrit, 10Release-Engineering-Team-TODO, 10Upstream: Can't delete weird ref using git in Gerrit - https://phabricator.wikimedia.org/T275946 (10Legoktm) >>! In T275946#6875886, @thcipriani wrote: > Noting here that I deleted this ref -- hopefully this unblocks folks: Looks like it is, thanks! [20:06:48] 10phan, 10MediaWiki-Core-Testing, 10Patch-For-Review: Use sniff or phan rule to detect violations of Stable interface policy (@internal, @stable etc.) - https://phabricator.wikimedia.org/T209394 (10Daimona) >>! In T209394#6875989, @Krinkle wrote: >>>! In T209394#6874363, @Daimona wrote: >> Phan has several i... [20:10:30] 10phan, 10MediaWiki-Core-Testing, 10Patch-For-Review: Use sniff or phan rule to detect violations of Stable interface policy (@internal, @stable etc.) - https://phabricator.wikimedia.org/T209394 (10Daimona) >>! In T209394#6876161, @Daimona wrote: > most of which seem to be caused by HookRunner/HookContainer... [21:37:57] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10SRE, 10Traffic, 10GitLab (Initialization), and 2 others: forward external traffic to gitlab VMs (was: Port map of how Gitlab is accessed) - https://phabricator.wikimedia.org/T276144 (10Dzahn) I merged and deployed [[ https://gerrit.wikimedi... [21:40:49] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10SRE, 10Traffic, 10GitLab (Initialization), and 2 others: forward external traffic to gitlab VMs (was: Port map of how Gitlab is accessed) - https://phabricator.wikimedia.org/T276144 (10Dzahn) Now this is either resolved (if we go unencrypte... [22:01:30] 10Beta-Cluster-Infrastructure: [beta cluster] UnderflowException: Ran out of captcha images - https://phabricator.wikimedia.org/T276176 (10Tgr) Let's just disable captchas on beta for the time being. If there's too much spam we can revert (but wikitech does not have captchas and is not on fire). [22:22:32] !log Set `profile::mediawiki::mcrouter_wancache::use_onhost_memcached: true` manaully in Horizon for deployment-mediawiki-07 (TODO: Move to cloud/eqiad1 in operations/puppet.git). [22:22:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:22:55] !log Run `sudo systemctl restart memcached` on deployment-mediawiki-07 [22:22:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:00:24] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10GitLab (Initialization), 10User-brennen: Remove Speed & Function blockers for GitLab work - https://phabricator.wikimedia.org/T274458 (10thcipriani) [23:00:42] 10Gerrit, 10GitLab (Initialization), 10User-brennen: Provide Speed & Function rough numbers for our current Gerrit web traffic - https://phabricator.wikimedia.org/T275564 (10thcipriani) 05Open→03Resolved >>! In T275564#6873176, @Sergey.Trofimovsky.SF wrote: > Thank you, guys! That should be enough to sta... [23:15:16] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO, 10VisualEditor, 10Editing-team (Tracking): Firefox CI tests keep failing in VE with Firefox 68 - https://phabricator.wikimedia.org/T240955 (10hashar) [23:32:59] 10Beta-Cluster-Infrastructure: [beta cluster] UnderflowException: Ran out of captcha images - https://phabricator.wikimedia.org/T276176 (10Tgr) Ugh. Are code updates broken on beta as well? The patch doesn't seem to arrive. [23:40:28] 10Beta-Cluster-Infrastructure: [beta cluster] UnderflowException: Ran out of captcha images - https://phabricator.wikimedia.org/T276176 (10Reedy) >>! In T276176#6877052, @Tgr wrote: > Ugh. Are code updates broken on beta as well? The patch doesn't seem to arrive. Often! I've just unstuck CI so they should appe... [23:43:06] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10serviceops: Replace production deployment servers and update them to Buster - https://phabricator.wikimedia.org/T265963 (10Dzahn) [23:43:25] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) [23:45:25] 10Project-Admins, 10Collection: Decide what to do with open OCG tasks in Phabricator - https://phabricator.wikimedia.org/T161312 (10Aklapper) 05Open→03Resolved a:03Aklapper done