[04:37:49] 10Release-Engineering-Team, 10MediaWiki-User-management, 10User-DannyS712: User rights validation is sometimes malfunctioning (with FlaggedRevs only?) - https://phabricator.wikimedia.org/T234743 (10Xaosflux) @DougWeller, I'm a bit lost in your comment. On enwiki at least, accepting pending changes is part of... [04:50:51] James_F: Thanks for the code review :) [07:38:08] 10Beta-Cluster-Infrastructure, 10MediaWiki-Core-Testing, 10MediaWiki-User-login-and-signup, 10MediaWiki-extensions-CentralAuth, and 3 others: Login to at least en.wikipedia.beta.wmflabs.org and commons.wikimedia.beta.wmflabs.org sometimes fails with `There seems to... - https://phabricator.wikimedia.org/T243123 [07:59:07] 10Beta-Cluster-Infrastructure, 10Wikidata, 10User-Ladsgroup, 10Wikidata-Campsite (Wikidata-Campsite-Iteration-∞): InvalidArgumentException during Wikibase initialization on Beta Wikidata - https://phabricator.wikimedia.org/T245540 (10Tarrow) Just to add to the documentation of this incident: did this only... [08:01:36] 10Beta-Cluster-Infrastructure, 10Wikidata, 10User-Ladsgroup, 10Wikidata-Campsite (Wikidata-Campsite-Iteration-∞): InvalidArgumentException during Wikibase initialization on Beta Wikidata - https://phabricator.wikimedia.org/T245540 (10WMDE-leszek) > What was the impact of this breakage? All of beta wikidata... [08:13:59] 10Release-Engineering-Team, 10MediaWiki-User-management, 10User-DannyS712: User rights validation is sometimes malfunctioning (with FlaggedRevs only?) - https://phabricator.wikimedia.org/T234743 (10DougWeller) You can't have OS without being a sysop. I apologise for confusing you. As you say, I should obviou... [08:39:08] 10Release-Engineering-Team, 10MediaWiki-User-management, 10User-DannyS712: User rights validation is sometimes malfunctioning (with FlaggedRevs only?) - https://phabricator.wikimedia.org/T234743 (10JEumerus) I believe what Doug Weller is trying to say is that enwiki oversighters are all admins and admins on... [09:11:52] 10Release-Engineering-Team, 10MediaWiki-User-management, 10User-DannyS712: User rights validation is sometimes malfunctioning (with FlaggedRevs only?) - https://phabricator.wikimedia.org/T234743 (10Zache) >>! In T234743#5896089, @JEumerus wrote: > I believe what Doug Weller is trying to say is that enwiki ov... [09:59:16] (03PS1) 10Tarrow: Add daily job to run npm audit on Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/573235 [09:59:40] (03PS2) 10Tarrow: Add daily job to run npm audit on Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/573235 (https://phabricator.wikimedia.org/T244001) [10:01:58] o/ I made a bit of a strawman patch for running `npm audit` daily on Wikibase. Would be awesome if someone had a moment to take a look and see if it's a) doing what I think and b) follows the best practices for specifying jenkins jobs. Thanks! [10:25:54] 10Release-Engineering-Team, 10Gerrit-Privilege-Requests, 10Discovery-Search (Current work): Add mstyles to the gerrit group wikidata-query - https://phabricator.wikimedia.org/T243496 (10dcausse) 05Openβ†’03Resolved a:03dcausse [10:34:05] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Release Pipeline, 10User-zeljkofilipin: Write Pipelinelib tutorial - https://phabricator.wikimedia.org/T240913 (10zeljkofilipin) [10:37:09] 10Beta-Cluster-Infrastructure, 10MediaWiki-Core-Testing, 10MediaWiki-User-login-and-signup, 10MediaWiki-extensions-CentralAuth, and 3 others: Login to at least en.wikipedia.beta.wmflabs.org and commons.wikimedia.beta.wmflabs.org sometimes fails with `There seems to... - https://phabricator.wikimedia.org/T243123 [10:56:06] 10Beta-Cluster-Infrastructure, 10Wikidata, 10User-Ladsgroup, 10Wikidata-Campsite (Wikidata-Campsite-Iteration-∞): InvalidArgumentException during Wikibase initialization on Beta Wikidata - https://phabricator.wikimedia.org/T245540 (10Lucas_Werkmeister_WMDE) 05Openβ†’03Resolved [11:29:14] 10Gerrit: Gerrit account got messed up - https://phabricator.wikimedia.org/T245545 (10Aklapper) > I tried changing it... didn't help. Did you manage to change the email address on wikitech.wikimedia.org? [11:38:09] 10Release-Engineering-Team-TODO, 10Scap, 10serviceops: Deploy scap 3.13.0-1 - https://phabricator.wikimedia.org/T245530 (10LarsWirzenius) The changes are on gerrit, release is tagged, the rest waits on SRE to build package and install it on servers. Is my understanding, but it's my first time, so I might ha... [12:04:04] 10Release-Engineering-Team-TODO, 10Scap, 10serviceops: Deploy scap 3.13.0-1 - https://phabricator.wikimedia.org/T245530 (10jijiki) p:05Triageβ†’03Medium a:03jijiki [12:15:25] 10Release-Engineering-Team, 10MediaWiki-User-management, 10User-DannyS712: User rights validation is sometimes malfunctioning (with FlaggedRevs only?) - https://phabricator.wikimedia.org/T234743 (10Xaosflux) OK, so I think it is confirmed that this has nothing at all to do with OS, which you absolutely can b... [12:55:39] (03PS2) 10Thiemo Kreuz (WMDE): Release v30.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/573011 (https://phabricator.wikimedia.org/T245309) (owner: 10Jforrester) [12:56:12] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] Release v30.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/573011 (https://phabricator.wikimedia.org/T245309) (owner: 10Jforrester) [13:03:43] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Add an integration smoke test to scap - https://phabricator.wikimedia.org/T245614 (10LarsWirzenius) [13:10:28] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Release, 10Train Deployments: 1.35.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T233867 (10Aklapper) I know this task is closed, but `T245553` (core's `` not working at all) seems quite a regression, and... [13:20:22] 10Gerrit: Gerrit account got messed up - https://phabricator.wikimedia.org/T245545 (10Soda) >>! In T245545#5896618, @Aklapper wrote: >> I tried changing it... didn't help. > Did you manage to change the email address on wikitech.wikimedia.org? I did change it to my old personal email address, waited for an hour... [13:36:48] (03CR) 10Daimona Eaytoy: [C: 03+1] "Belatedly +1ing, looks much cleaner now, and thanks for the added test cases!" (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/571023 (owner: 10Umherirrender) [13:38:54] (03CR) 10Daimona Eaytoy: [C: 03+1] "Looks good. It'd be great if we could improve gen-changelog.sh to include similar additions, but it's fine for now." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/573011 (https://phabricator.wikimedia.org/T245309) (owner: 10Jforrester) [14:28:34] (03PS1) 10Lars Wirzenius: Add: an acceptance (or integration) test suite for scap [tools/scap] - 10https://gerrit.wikimedia.org/r/573298 [14:32:09] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Release, 10Train Deployments: 1.35.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T233867 (10zeljkofilipin) @Aklapper should T245553 be blocker for this week's train (T233868), if it's a regression? [14:34:35] (03CR) 10Lars Wirzenius: "I added a smoke test level integration test for scap. It's stupidly simple (runs "scap version"), but has already found code bug (now fixe" [tools/scap] - 10https://gerrit.wikimedia.org/r/573298 (owner: 10Lars Wirzenius) [14:36:26] (03PS2) 10Lars Wirzenius: Add: an acceptance (or integration) test suite for scap [tools/scap] - 10https://gerrit.wikimedia.org/r/573298 (https://phabricator.wikimedia.org/T245614) [14:51:43] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Patch-For-Review: Tag a new release of scap - https://phabricator.wikimedia.org/T243358 (10LarsWirzenius) [15:05:26] hm, hiya [15:06:03] is there a way to roll out changes in ProductionServices.php slowly to justt certain wikis? [15:06:28] I need to change a service port, but would like to deploy it to just a few live spots before I do the whole thing [15:06:30] to make sure everything works [15:06:34] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Release Pipeline, 10User-zeljkofilipin: Write Pipelinelib tutorial - https://phabricator.wikimedia.org/T240913 (10LarsWirzenius) @dduvall Thanks, I'll read them through and comment. Hunt me down with a torch... [15:25:49] re https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/573307 [15:28:43] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Wikimedia-Logstash, and 2 others: Look into shoving gerrit logs into logstash - https://phabricator.wikimedia.org/T141324 (10Aklapper) [16:09:53] ottomata: the only way we have of targeting certain wikis is with the version of MediaWiki+Extensions :\ [16:10:11] hm [16:10:15] you could probably feature flag this for certain wikis? [16:10:29] that'd require code train deploy? [16:10:44] yeah, although config changes don't ride the train [16:10:45] i guess i could add a new Productionservice entry, and then configure certain wikis to use that one for certain streams [16:11:13] don't know how messy that'd be, but that's likely your best option [16:11:51] that is, I see folks targeting certain wikis with configuration settings in CommonSettings pretty frequently. [16:12:00] yes, that i can do i think [16:12:03] hm [16:12:07] thanks [16:12:10] a bit annoying [16:12:12] might do that [16:12:17] better to be safe here [16:13:06] er, initialisesettings that is [16:19:32] (03CR) 10Jforrester: "> Patch Set 2: Code-Review+1" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/573011 (https://phabricator.wikimedia.org/T245309) (owner: 10Jforrester) [16:19:46] (03CR) 10Jforrester: [C: 03+1] Release v30.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/573011 (https://phabricator.wikimedia.org/T245309) (owner: 10Jforrester) [16:32:46] (03CR) 10Daimona Eaytoy: [C: 03+2] "> > Patch Set 2: Code-Review+1" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/573011 (https://phabricator.wikimedia.org/T245309) (owner: 10Jforrester) [16:33:22] (03Merged) 10jenkins-bot: Release v30.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/573011 (https://phabricator.wikimedia.org/T245309) (owner: 10Jforrester) [16:35:40] 10MediaWiki-Codesniffer, 10Patch-For-Review: Release mediawiki-codesniffer 30.0 - https://phabricator.wikimedia.org/T245309 (10Jdforrester-WMF) 5ac5bde72c8fc7267c5fc2dc3f6e0d45770bf924 has been tagged. [16:37:53] 10MediaWiki-Codesniffer, 10Patch-For-Review: Release mediawiki-codesniffer 30.0 - https://phabricator.wikimedia.org/T245309 (10Jdforrester-WMF) 05Openβ†’03Resolved a:03Jdforrester-WMF Done. [16:38:13] (03CR) 10Brennen Bearnes: [V: 03+2 C: 03+2] "This builds locally. Let's take another crack at publishing." [releng/dev-images] - 10https://gerrit.wikimedia.org/r/572488 (https://phabricator.wikimedia.org/T245304) (owner: 10Kosta Harlan) [16:41:54] James_F: evidently missed your question yesterday - i think the above ^ will publish, but i'm going to find out now. [16:42:20] Good luck. [16:45:10] !log Updating dev-images docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/572488 [16:45:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:23:45] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Analytics: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10Jdforrester-WMF) For clarity, this needs the... [17:45:45] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Analytics: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10Ottomata) Ah I don't think we were aware ther... [17:47:52] PROBLEM - Parsoid on deployment-parsoid09 is CRITICAL: connect to address 172.16.5.63 and port 8000: Connection refused [17:47:52] PROBLEM - Parsoid on deployment-mediawiki-parsoid10 is CRITICAL: connect to address 172.16.0.141 and port 8000: Connection refused [17:53:43] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Improve Basic Kubernetes Knowledge within RelEng - https://phabricator.wikimedia.org/T245653 (10jeena) [17:57:11] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Clean old wmf branches in Gerrit - https://phabricator.wikimedia.org/T244368 (10thcipriani) a:03thcipriani [17:57:44] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Test upgrade of Gerrit to 2.16.13 on the ganeti test instance - https://phabricator.wikimedia.org/T243981 (10thcipriani) a:03thcipriani Plan to work on this with @dduvall very soon. [18:01:09] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10dev-images, 10docker-pkg, 10User-brennen: dev/stretch-php72-fpm-apache2-xdebug deploy fails from contint1001 - https://phabricator.wikimedia.org/T245304 (10brennen) 05Openβ†’03Resolved Published. [18:01:41] 10Gerrit, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Gerrit account got messed up - https://phabricator.wikimedia.org/T245545 (10thcipriani) a:03thcipriani [18:10:04] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Zuul, 10User-DannyS712: Remote call on integration-agent-docker-1001 failed - https://phabricator.wikimedia.org/T245054 (10Jdforrester-WMF) Proposed: Re-pool, re-start, re-build, give up? [18:13:43] (03CR) 1020after4: [C: 03+1] "Any reason not to merge this? it looks good to me." [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) (owner: 10Effie Mouzeli) [18:15:47] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Docker, 10MW-1.35-notes (1.35.0-wmf.18; 2020-02-04), and 4 others: Provide docker-compose based development / CI environment in MediaWiki core - https://phabricator.wikimedia.org/T238224 (10zeljkof... [18:19:04] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10User-brennen: train-deploy-notes Jenkins job fails in conjunction with branch.py - https://phabricator.wikimedia.org/T243330 (10dduvall) a:05mmodellβ†’03dduvall Stealing this from @mmodell since I... [18:20:01] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Zuul, 10Patch-For-Review: Convert zuul to use scap - https://phabricator.wikimedia.org/T215458 (10mmodell) 05Openβ†’03Stalled bloc... [18:20:03] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 (10mmodell) [18:23:18] 10Project-Admins, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Containers, 10MediaWiki-Docker, 10Release Pipeline: Re-purposing MediaWiki-Docker and friends - https://phabricator.wikimedia.org/T245167 (10Jdforrester-WMF) p:05Triageβ†’03Medium [18:28:52] (03CR) 10Jforrester: [C: 03+1] "LGTM, but I've never written a patch in this repo, just used its results, so…" [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) (owner: 10Effie Mouzeli) [18:32:02] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Get an IDE to run tests against services running in kubernetes - https://phabricator.wikimedia.org/T245656 (10jeena) [18:33:10] 10Project-Admins, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Containers, 10MediaWiki-Docker, 10Release Pipeline: Re-purposing MediaWiki-Docker and friends - https://phabricator.wikimedia.org/T245167 (10Jdforrester-WMF) p:05Mediumβ†’03High [18:35:28] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Get an IDE to run tests against services running in kubernetes - https://phabricator.wikimedia.org/T245656 (10thcipriani) p:05Triageβ†’03Medium [18:35:32] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Improve Basic Kubernetes Knowledge within RelEng - https://phabricator.wikimedia.org/T245653 (10thcipriani) p:05Triageβ†’03Medium [18:36:02] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Patch-For-Review: Add an integration smoke test to scap - https://phabricator.wikimedia.org/T245614 (10LarsWirzenius) p:05Triageβ†’03Medium [18:36:07] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Patch-For-Review: Tag a new release of scap - https://phabricator.wikimedia.org/T243358 (10LarsWirzenius) p:05Triageβ†’03High [18:36:13] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Arrange PGP keysigning at All Hands 2020 - https://phabricator.wikimedia.org/T242340 (10LarsWirzenius) p:05Triageβ†’03Medium [18:50:53] 10Gerrit, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Gerrit account got messed up - https://phabricator.wikimedia.org/T245545 (10thcipriani) p:05Triageβ†’03High [18:51:01] 10Release-Engineering-Team (Other / Uncategorized), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Releng March priorities vs resources Chart - https://phabricator.wikimedia.org/T242237 (10thcipriani) [18:51:03] 10Continuous-Integration-Infrastructure, 10Jenkins: .mp4 build artifacts not viewable due to CSP - https://phabricator.wikimedia.org/T245658 (10Daimona) [18:51:37] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Zuul, 10User-DannyS712: Remote call on integration-agent-docker-1001 failed - https://phabricator.wikimedia.org/T245054 (10thcipriani) a:03thcipriani Optimistically, could just be the agent on that machi... [18:56:33] (03PS2) 10Umherirrender: Ignore self-closing doc comments /***/ [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/571016 (https://phabricator.wikimedia.org/T243721) [18:58:18] !log unblocking beta-scap-eqiad job (stuck for 18 hrs) [18:58:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:01:54] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Zuul, 10User-DannyS712: Remote call on integration-agent-docker-1001 failed - https://phabricator.wikimedia.org/T245054 (10thcipriani) I brought the agent back online, it ran at least one job ok: https://i... [19:13:58] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T233868 (10Jdforrester-WMF) [19:16:02] PROBLEM - Free space - all mounts on deployment-snapshot01 is CRITICAL: CRITICAL: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)deployment-prep.deployment-snapshot01.diskspace.root.byte_percentfree (<10.00%) [19:16:57] what is the problem with space? [19:17:07] I'm not there, nothing's running on it... [19:18:35] keeps doing it after 24-48 hours [19:18:45] most of the space in use is the l10n cache [19:20:58] RECOVERY - Free space - all mounts on deployment-snapshot01 is OK: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found) [19:21:30] >_< [19:24:29] Is this because of the new scap code? [19:24:45] We're now building both CDB and "flat" PHP files for i18n. [19:24:51] So the size demands will be higher… [19:25:20] But we don't use pre-release scap on CI, do we? [19:28:00] there's only 19G allocated for / and 14G used [19:28:08] so I guess the rest fills up with /srv/mediawiki [19:28:25] the bulk used now is 7.4G /srv/mediawiki, 5G /var [19:28:54] there's only measly megabytes anywhere I could clean up [19:29:07] apergos: Nuke /srv/mediawiki/php-master/cache/l10n [19:29:17] It'll recreate it next sync, and work for a couple of days [19:29:20] I can but will we have to do that once a day? [19:29:31] meh [19:29:36] I've done it two or three times now, and filed a bug [19:29:47] ah link me? I'll subscribe [19:29:50] Not much we can do to get more space without rebuilding it [19:30:12] T245402 [19:30:13] T245402: deployment-snapshot01.deployment-prep.eqiad.wmflabs keeps running out of space - https://phabricator.wikimedia.org/T245402 [19:30:21] 5.5g in that one directory, gez [19:30:23] +e [19:30:35] It's the .~tmp~ dirs that seem to mess it up [19:33:04] I could add manually a once every day cron to remove that dir for now... [19:34:54] done, will note on ticket (and put the ticket number in the crontab too :-P) [19:36:03] FWIW, we do have the PHP flat files in beta right now [19:36:05] 10Beta-Cluster-Infrastructure: deployment-snapshot01.deployment-prep.eqiad.wmflabs keeps running out of space - https://phabricator.wikimedia.org/T245402 (10ArielGlenn) I've added manually a root cron job over there to remove wholesale all of /srv/mediawiki/php-master/cache/l10n once a day. But it's not the grea... [19:36:08] i.e., scap next [19:36:21] which likely cause the l10n to take more space for the moment [19:36:36] (although when it first ran it wasn't taking much more space) [19:36:48] the problem is we have >1000 extensions to build l10n from [19:36:54] 9G or whatevr the ticket said is a lot of space [19:36:55] 10Project-Admins, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Docker-Hub-MediaWiki, 10MediaWiki-Containers, 10Release Pipeline: Re-purposing MediaWiki-Docker and friends - https://phabricator.wikimedia.org/T245167 (10Jdforrester-WMF) 05Openβ†’03Resolved Done. #MediaWiki-Docker is now t... [19:37:12] the .~tmp~ dirs get created by rsync delay-updates [19:37:27] yeah, it's like 3G in production with 189 extensions [19:37:34] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Docker, 10Goal, and 5 others: Provide docker-compose based development / CI environment in MediaWiki core - https://phabricator.wikimedia.org/T238224 (10Jdforrester-WMF) [19:38:00] so the order of magnitude more extensions taking 5G isn't too surprising. The .~tmp~ thing is frustrating :\ [19:38:07] it sure is [19:38:16] 10Release-Engineering-Team (Pipeline), 10MediaWiki-Docker, 10serviceops: Clarify and document our docker image building process and policies. - https://phabricator.wikimedia.org/T216234 (10Jdforrester-WMF) This is about the MediaWiki-Docker and production pipeline images, not the Docker Hub image. [19:39:10] is there a way to just clean those up i particular? [19:41:08] so those will get cleaned up, but only if scap is successful [19:41:37] when you pass --delay-updates to rsync, rsync will make a new file inside .~tmp~ and then move all those files into place at the last moment [19:41:44] and then remove the directory [19:41:53] and so if it fails naturally it leaves all that behind [19:41:56] yep [19:42:27] so those directories are fine to remove if scap isn't in progress [19:42:45] but you (or the cron job) has to know that [19:43:24] and of course scap isn't in progress now but it is a second later when the cron job starts removing things... [19:46:40] 10Beta-Cluster-Infrastructure: deployment-snapshot01.deployment-prep.eqiad.wmflabs keeps running out of space - https://phabricator.wikimedia.org/T245402 (10thcipriani) We're running the next version of scap in beta currently which generates PHP l10n cache files in addition to the normal JSON + CDB files (T99740... [19:57:05] it's too bad I have to have all 1k extensions on there when I don't need them :-/ [20:01:00] ls -lh /srv/mediawiki-staging/php-master/extensions/ | wc -l -> 1014 :( [20:01:08] gah [20:17:03] We theoretically could immediately make part II of the scap changes (but not release) to only use flat files? [20:17:22] That's moving a bit faster than I think we'd hoped, but… [20:29:24] PROBLEM - Host deployment-puppetmaster03 is DOWN: CRITICAL - Host Unreachable (172.16.4.91) [20:29:32] PROBLEM - Host deployment-puppetdb02 is DOWN: CRITICAL - Host Unreachable (172.16.4.104) [20:34:22] (03CR) 10Jforrester: [C: 03+1] Add: an acceptance (or integration) test suite for scap [tools/scap] - 10https://gerrit.wikimedia.org/r/573298 (https://phabricator.wikimedia.org/T245614) (owner: 10Lars Wirzenius) [20:37:37] (03CR) 10Thcipriani: "Overall, seems good to me." (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) (owner: 10Effie Mouzeli) [20:55:40] (03CR) 1020after4: [C: 03+1] Add: an acceptance (or integration) test suite for scap [tools/scap] - 10https://gerrit.wikimedia.org/r/573298 (https://phabricator.wikimedia.org/T245614) (owner: 10Lars Wirzenius) [21:07:00] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review, 10Technical-Debt: Investigate whether GD is still needed on appservers - https://phabricator.wikimedia.org/T227734 (10Jdforrester-WMF) I propose that we Decline this, given that it's alrea... [21:08:41] 10MediaWiki-Codesniffer, 10Upstream: Enable Squiz.ControlStructures.ForLoopDeclaration/ForEachLoopDeclaration sniff - https://phabricator.wikimedia.org/T214332 (10Umherirrender) https://github.com/squizlabs/PHP_CodeSniffer/releases/tag/3.5.4 contains one fix: > Squiz.ControlStructures.ForLoopDeclaration now ha... [21:10:23] (03PS1) 10Umherirrender: Update squizlabs/php_codesniffer to 3.5.4 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/573382 [21:13:24] (03CR) 10Jforrester: "Fancy." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/573382 (owner: 10Umherirrender) [21:30:53] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Zuul, 10User-DannyS712: Remote call on integration-agent-docker-1001 failed - https://phabricator.wikimedia.org/T245054 (10thcipriani) 05Openβ†’03Resolved Calling this resolved. I killed the `java -jar s... [21:34:22] thcipriani: Hurrah. [21:36:13] (03PS2) 10Jforrester: Revert "Zuul: Temporarily exclude FlaggedRevs from VE's dependencies" [integration/config] - 10https://gerrit.wikimedia.org/r/571397 [21:36:17] (03CR) 10Jforrester: [C: 03+2] Revert "Zuul: Temporarily exclude FlaggedRevs from VE's dependencies" [integration/config] - 10https://gerrit.wikimedia.org/r/571397 (owner: 10Jforrester) [21:37:16] (03Merged) 10jenkins-bot: Revert "Zuul: Temporarily exclude FlaggedRevs from VE's dependencies" [integration/config] - 10https://gerrit.wikimedia.org/r/571397 (owner: 10Jforrester) [21:37:57] thcipriani: Also, is there a runbook you used to kill it? Hint hint. [21:44:53] twentyafterfour: Feel like it's time to land https://gerrit.wikimedia.org/r/c/mediawiki/tools/release/+/563492 or should we wait more? [22:05:50] James_F: land it [22:10:01] (03CR) 1020after4: [C: 03+2] Drop make-wmf-branch entirely [tools/release] - 10https://gerrit.wikimedia.org/r/563492 (https://phabricator.wikimedia.org/T222829) (owner: 10Jforrester) [22:10:30] (03Merged) 10jenkins-bot: Drop make-wmf-branch entirely [tools/release] - 10https://gerrit.wikimedia.org/r/563492 (https://phabricator.wikimedia.org/T222829) (owner: 10Jforrester) [22:38:11] twentyafterfour: Whee. [22:38:32] twentyafterfour: Did you want me to draft a wikitech-l e-mail for you? [22:38:55] (03PS3) 10Jforrester: Drop merge-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/563498 [22:39:05] (03PS3) 10Jforrester: make-release: Alpha-sort list of production extensions [tools/release] - 10https://gerrit.wikimedia.org/r/563499 [22:39:18] (03CR) 10Jforrester: [C: 03+2] make-release: Alpha-sort list of production extensions [tools/release] - 10https://gerrit.wikimedia.org/r/563499 (owner: 10Jforrester) [22:39:52] (03Merged) 10jenkins-bot: make-release: Alpha-sort list of production extensions [tools/release] - 10https://gerrit.wikimedia.org/r/563499 (owner: 10Jforrester) [23:12:46] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Release-Tools, 10MediaWiki-Releasing (Workflow Improvements), 10Patch-For-Review: merge branch.py and make-wmf-branch - https://phabricator.wikimedia.org/T222829 (10Jdforrester-WMF) Pr... [23:52:14] 10Release-Engineering-Team-TODO, 10Operations, 10SRE-Access-Requests: Should 'doc' machines (i.e. doc1001) have contint-roots as a group? - https://phabricator.wikimedia.org/T245691 (10Jdforrester-WMF) [23:52:30] 10Release-Engineering-Team-TODO, 10Operations, 10SRE-Access-Requests: Should 'doc' machines (i.e. doc1001) have contint-roots as a group? - https://phabricator.wikimedia.org/T245691 (10Jdforrester-WMF) Created following chatting with @greg earlier.