[01:15:42] 10Phabricator, 10Developer-Advocacy (Apr-Jun 2019): Re-evaluate our use of Phabricator Conpherence chat - https://phabricator.wikimedia.org/T127640 (10revi) No consensus at stewards-l, one suggesting Mailing list and other one (in conphrence) suggested use of somewhat chat-like system contradicts, so the best... [03:27:34] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10VPS-project-libraryupgrader: LibraryUpgrader CI normalisation tasks, June 2019 - https://phabricator.wikimedia.org/T225325 (10Legoktm) [04:32:41] (03PS1) 10Legoktm: Unbreak clone URLs for test dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/516064 [04:32:43] (03PS1) 10Legoktm: Create a separate list of phan dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/516065 (https://phabricator.wikimedia.org/T225112) [04:32:45] (03PS1) 10Legoktm: Drop dependencies from normal map added for phan [integration/config] - 10https://gerrit.wikimedia.org/r/516066 (https://phabricator.wikimedia.org/T225112) [04:34:21] (03CR) 10Legoktm: [C: 03+2] Unbreak clone URLs for test dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/516064 (owner: 10Legoktm) [04:34:39] (03CR) 10Legoktm: [C: 03+2] Create a separate list of phan dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/516065 (https://phabricator.wikimedia.org/T225112) (owner: 10Legoktm) [04:34:53] (03CR) 10Legoktm: [C: 03+2] Drop dependencies from normal map added for phan [integration/config] - 10https://gerrit.wikimedia.org/r/516066 (https://phabricator.wikimedia.org/T225112) (owner: 10Legoktm) [04:35:54] (03Merged) 10jenkins-bot: Unbreak clone URLs for test dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/516064 (owner: 10Legoktm) [04:36:12] (03Merged) 10jenkins-bot: Create a separate list of phan dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/516065 (https://phabricator.wikimedia.org/T225112) (owner: 10Legoktm) [04:36:32] (03Merged) 10jenkins-bot: Drop dependencies from normal map added for phan [integration/config] - 10https://gerrit.wikimedia.org/r/516066 (https://phabricator.wikimedia.org/T225112) (owner: 10Legoktm) [04:38:01] !log deploying https://gerrit.wikimedia.org/r/516065 https://gerrit.wikimedia.org/r/516066 [04:38:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [05:32:08] (03CR) 10Jforrester: "Fancy." [integration/config] - 10https://gerrit.wikimedia.org/r/516065 (https://phabricator.wikimedia.org/T225112) (owner: 10Legoktm) [05:56:04] legoktm: I'm not sure if splitting the phan dependency tree has a measurable improvement? [05:56:17] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<11.11%) [06:11:09] 10Project-Admins: Project for VVIT WikiConnect's Annual Plan (2019-2020) - https://phabricator.wikimedia.org/T225392 (10KCVelaga) [06:12:25] PROBLEM - Disk space on contint1001 is CRITICAL: DISK CRITICAL - free space: / 2086 MB (4% inode=61%): https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space [06:12:33] Hmm. Is that related? [06:16:00] when is that new disk space arriving? [06:56:19] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:24:49] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): Old cloudvirt (with Intel Xeon) are twice slower than new ones (Intel Sky Lake) - https://phabricator.wikimedia.org/T223971 (10Wurgl) Could this difference be caused by prevention of those design flaws called Spectre & Meltdown? [07:25:26] 10Release-Engineering-Team (Kanban), 10MediaViewer, 10Multimedia, 10Browser-Tests, and 3 others: MediaViewer selenium tests are failing with "Misconfigured -- Unsupported OS/browser/version/device combo" because our target MacOS has aged out - https://phabricator.wikimedia.org/T214389 (10simon04) [07:38:12] 10Phabricator, 10Mail, 10Operations, 10Regression: "Phabricator monthly statistics" email on wikitech-l@ missing for May 2019 - https://phabricator.wikimedia.org/T224804 (10ArielGlenn) p:05Triage→03Normal I see that @MoritzMuehlenhoff installed bsd-mailx manually on the box to fix future runs; this sh... [13:18:32] (03CR) 10Krinkle: [C: 03+1] jjb: Provide php72-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/515289 (owner: 10Jforrester) [13:18:35] (03PS2) 10Krinkle: jjb: Provide php72-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/515289 (owner: 10Jforrester) [13:34:44] (03CR) 10Krinkle: [C: 04-1] "I think the pub removal was unintentional." (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/515289 (owner: 10Jforrester) [13:34:59] (03PS3) 10Krinkle: jjb: Provide php72-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/515289 (owner: 10Jforrester) [13:36:49] (03CR) 10Krinkle: [C: 03+2] "INFO:jenkins_jobs.builder:Number of jobs generated: 1" [integration/config] - 10https://gerrit.wikimedia.org/r/515289 (owner: 10Jforrester) [13:39:27] (03Merged) 10jenkins-bot: jjb: Provide php72-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/515289 (owner: 10Jforrester) [13:42:55] (03PS2) 10Krinkle: jjb: Provide job_mwext-php72-phan-docker and mwskin-php72-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/515290 (owner: 10Jforrester) [13:43:34] (03PS3) 10Krinkle: jjb: Provide job_mwext-php72-phan-docker and mwskin-php72-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/515290 (owner: 10Jforrester) [13:51:08] (03CR) 10Krinkle: [C: 03+2] "INFO:jenkins_jobs.cli.subcommand.update:Updating jobs in ['config/'] (['mwext-php70-phan-docker'])" [integration/config] - 10https://gerrit.wikimedia.org/r/515290 (owner: 10Jforrester) [13:51:16] (03PS2) 10Krinkle: jjb: Provide mediawiki-core-php72-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/515291 (owner: 10Jforrester) [13:53:32] (03Merged) 10jenkins-bot: jjb: Provide job_mwext-php72-phan-docker and mwskin-php72-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/515290 (owner: 10Jforrester) [13:54:29] (03PS3) 10Krinkle: jjb: Provide mediawiki-core-php72-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/515291 (owner: 10Jforrester) [14:10:20] (03CR) 10Krinkle: [C: 03+2] jjb: Provide mediawiki-core-php72-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/515291 (owner: 10Jforrester) [14:10:47] (03CR) 10Krinkle: [C: 03+2] "INFO:jenkins_jobs.cli.subcommand.update:Updating jobs in ['config/'] (['mediawiki-core-php72-phan-docker'])" [integration/config] - 10https://gerrit.wikimedia.org/r/515291 (owner: 10Jforrester) [14:12:41] (03Merged) 10jenkins-bot: jjb: Provide mediawiki-core-php72-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/515291 (owner: 10Jforrester) [14:15:38] 10Release-Engineering-Team (Backlog), 10Quibble (marble): CI Quibble jobs should be feed with git repos and dependencies - https://phabricator.wikimedia.org/T211704 (10awight) T211702, T211703, and this task all seem slightly at odds with each other. If we clone everything before running quibble, then we lose... [14:51:31] James_F: It had some impact, though not as large as I wanted I think. The biggest change was pulling Wikibase out of the MobileFrontend dependencies, which would be pretty significant for repos that recursively depend upon MF. [15:17:37] 10Release-Engineering-Team, 10MediaWiki-extensions-General, 10Wikimedia-production-error (Shared Build Failure): Fix flapping tests in gated extensions as of 2019-06-06 - https://phabricator.wikimedia.org/T225162 (10Krinkle) [15:17:40] 10Release-Engineering-Team, 10MediaWiki-extensions-General, 10Wikimedia-production-error (Shared Build Failure): Fix flapping tests in gated extensions as of 2019-06-06 - https://phabricator.wikimedia.org/T225162 (10Krinkle) [15:18:49] 10Release-Engineering-Team, 10MediaWiki-extensions-General: Fix flapping tests in gated extensions as of 2019-06-06 - https://phabricator.wikimedia.org/T225162 (10Krinkle) [15:26:08] * James_F nods. [15:26:43] (03PS2) 10Jforrester: layout: Switch quibble jobs from php70-phan-docker to php72-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/515292 [15:39:19] 10Diffusion, 10Release-Engineering-Team (Kanban), 10Operations, 10User-zeljkofilipin: Cannot connect to vcs@git-ssh.wikimedia.org (since move from phab1001 to phab1003) - https://phabricator.wikimedia.org/T224677 (10ArielGlenn) @mmodell you get farther than I do. I've checked the db and see the right key i... [15:41:09] 10Release-Engineering-Team, 10Scap, 10serviceops, 10PHP 7.2 support, 10User-jijiki: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10thcipriani) >>! In T224857#5243133, @Joe wrote: > Regarding rollbacks: I guess that we can add a switch that, just for rol... [15:43:15] 10Diffusion, 10Release-Engineering-Team (Kanban), 10Operations, 10User-zeljkofilipin: Cannot connect to vcs@git-ssh.wikimedia.org (since move from phab1001 to phab1003) - https://phabricator.wikimedia.org/T224677 (10mmodell) @arielGlenn: The only thing left to do that I can think of is to run the git sshd... [15:45:58] 10Diffusion, 10Release-Engineering-Team (Kanban), 10Operations, 10User-zeljkofilipin: Cannot connect to vcs@git-ssh.wikimedia.org (since move from phab1001 to phab1003) - https://phabricator.wikimedia.org/T224677 (10ArielGlenn) Awesome, I'll be around if it's not ridiculous o'clock for me. There's a presen... [15:49:20] (03CR) 10Krinkle: [C: 03+2] layout: Switch quibble jobs from php70-phan-docker to php72-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/515292 (owner: 10Jforrester) [15:52:20] (03CR) 10jerkins-bot: [V: 04-1] layout: Switch quibble jobs from php70-phan-docker to php72-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/515292 (owner: 10Jforrester) [15:52:31] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10VPS-project-libraryupgrader: LibraryUpgrader CI normalisation tasks, June 2019 - https://phabricator.wikimedia.org/T225325 (10Jdforrester-WMF) [15:58:33] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10VPS-project-libraryupgrader: LibraryUpgrader CI normalisation tasks, June 2019 - https://phabricator.wikimedia.org/T225325 (10Jdforrester-WMF) [16:04:30] PROBLEM - Free space - all mounts on deployment-mwmaint01 is CRITICAL: CRITICAL: deployment-prep.deployment-mwmaint01.diskspace.root.byte_percentfree (<11.11%) [16:09:27] RECOVERY - Free space - all mounts on deployment-mwmaint01 is OK: OK: All targets OK [16:19:24] James_F, Krinkle: opinions on Popups having two package.json files? https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/Popups/+/master/.storybook/package.json - should libraryupgrader be recursively looking through the repo tree for this stuff or just deal with the root and ask maintainers to only have one package.json file? [16:21:44] legoktm: Hmm. [16:22:39] In cases like this, I think it's fine for LU to ignore it and make it the responsibility of the maintainers. [16:26:27] legoktm: yeah, ignore for now. It's not used in CI afaik, either. [16:26:42] Once it is, it'll be past the Node 10 upgrade, and thus not needing to be separate anymore. [16:26:59] (03PS1) 10Legoktm: Add GeoData phan dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/516298 [16:27:11] sounds good, thanks [16:29:53] (03CR) 10Legoktm: [C: 03+2] Add GeoData phan dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/516298 (owner: 10Legoktm) [16:30:07] (03CR) 10Jforrester: [C: 03+2] "Oh, right." [integration/config] - 10https://gerrit.wikimedia.org/r/516298 (owner: 10Legoktm) [16:30:59] 10Diffusion, 10Release-Engineering-Team (Kanban), 10Operations, 10User-zeljkofilipin: Cannot connect to vcs@git-ssh.wikimedia.org (since move from phab1001 to phab1003) - https://phabricator.wikimedia.org/T224677 (10ArielGlenn) Just going to leave this here. https://bugs.debian.org/cgi-bin/bugreport.cgi?bu... [16:31:27] (03Merged) 10jenkins-bot: Add GeoData phan dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/516298 (owner: 10Legoktm) [16:31:57] !log deployed https://gerrit.wikimedia.org/r/516298 [16:31:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:32:52] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: New phan dependencies significantly slowed down CI tests - https://phabricator.wikimedia.org/T225112 (10Legoktm) >>! In T225112#5241339, @Umherirrender wrote: > To get a overview about previous added dependency for phan - visit https://g... [16:36:10] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10serviceops, 10Epic: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Legoktm) Instead of splitting this per-wiki, I'd rather see similar config settings grouped togethe... [16:40:23] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10serviceops, 10Epic: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10ArielGlenn) I would be overwhelmed by a directory with 84 little config files in it. If we're talk... [16:55:19] (03PS3) 10Jforrester: layout: Switch quibble jobs from php70-phan-docker to php72-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/515292 [16:55:55] (03PS1) 10Legoktm: [Refreshed] Add phan dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/516300 [16:57:29] (03CR) 10Jforrester: [C: 03+2] [Refreshed] Add phan dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/516300 (owner: 10Legoktm) [16:59:05] (03Merged) 10jenkins-bot: [Refreshed] Add phan dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/516300 (owner: 10Legoktm) [16:59:08] legoktm: OK for me to deploy 515292 or do you want me to hold off whilst LU does its thing? [16:59:53] !log Deployed https://gerrit.wikimedia.org/r/516300 [16:59:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:03:36] the LU abbreviation keeps confusing me with LocalizationUpdate [17:05:09] ^ [17:05:20] We like reusing names and abbreviations [17:09:42] (03CR) 10Krinkle: [C: 03+2] layout: Switch quibble jobs from php70-phan-docker to php72-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/515292 (owner: 10Jforrester) [17:11:20] (03CR) 10jerkins-bot: [V: 04-1] layout: Switch quibble jobs from php70-phan-docker to php72-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/515292 (owner: 10Jforrester) [17:11:47] James_F: In the multi-megabyte output I assume there is an error somewhere, but Chrome is unable to render it, so... good luck :D [17:15:27] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10CirrusSearch, 10MediaWiki-extensions-Scribunto, and 4 others: Add a PHPUnit group to skip test on gated CI runs - https://phabricator.wikimedia.org/T225068 (10Jdforrester-WMF) OK, let's move this forward with a decision; the "blacklist" model,... [17:18:28] James_F: can we run php72 in parallel with php70 for 2-3 days and then drop php70? that way it's trivial to identify regressions vs outright failures [17:18:29] Krinkle: I couldn't see anything in reading it, but yeah, it's not ideal. [17:18:41] legoktm: 'Cos CI is currently running too fast? ;-) Sure. [17:19:05] (A few *days*? Not hours?) [17:19:22] the phan jobs are pretty fast, relatively speaking [17:19:45] "too fast" I love the sound of future us saying that [17:19:59] greg-g: happy to use a new abbreviation if someone comes up with one :) [17:20:23] LibUp. [17:20:28] LocUp. [17:20:33] Obviously. ;-) [17:24:06] (03PS4) 10Jforrester: layout: Drop quibble experimental php70-phan-docker jobs; php72 suffices [integration/config] - 10https://gerrit.wikimedia.org/r/515292 [17:24:08] (03PS1) 10Jforrester: layout: Add quibble experimental mwext-php72-phan-docker, alongside php70 [integration/config] - 10https://gerrit.wikimedia.org/r/516301 [17:24:29] Given that the first set are experimental-only, I'm going to be… "surprised" if they break things. [17:25:36] 10Diffusion, 10Release-Engineering-Team (Kanban), 10Operations, 10User-zeljkofilipin: Cannot connect to vcs@git-ssh.wikimedia.org (since move from phab1001 to phab1003) - https://phabricator.wikimedia.org/T224677 (10mmodell) > openssh-server: SSH AuthorizedKeysCommand hangs when output is too large Ah ha!... [17:25:47] (03CR) 10Jforrester: [C: 03+2] layout: Add quibble experimental mwext-php72-phan-docker, alongside php70 [integration/config] - 10https://gerrit.wikimedia.org/r/516301 (owner: 10Jforrester) [17:27:18] (03CR) 10jerkins-bot: [V: 04-1] layout: Add quibble experimental mwext-php72-phan-docker, alongside php70 [integration/config] - 10https://gerrit.wikimedia.org/r/516301 (owner: 10Jforrester) [17:27:24] * James_F sighs. [17:28:19] LibUp sounds good [17:29:32] So… the error is "FAILURE: Job mwext-php72-phan-docker not defined" [17:29:45] Which is entertaining. [17:33:53] lol [17:34:00] um [17:34:07] did you push the job via jjb? [17:34:12] It uses it 813 times, and reports it not existing once. [17:34:25] some awkward typo? [17:34:26] No, I didn't merge that code. [17:34:57] I assumed Krinkle did when he merged it. [17:35:18] Krinkle: Do you leave the jjb fun for me? [17:36:46] I deployed it, I thought I did anyway [17:37:09] I pasted the commands and the jobs created [17:37:12] was it not created today? [17:37:35] https://github.com/wikimedia/integration-config/commits/master [17:37:38] INFO:jenkins_jobs.cli.subcommand.update:Number of jobs updated: 1 [17:37:43] ah, it's not mediawiki-core-php72-phan-docker [17:37:47] Merged but not built. [17:37:56] It's mwext-php72-phan-docker that your commit is enabling [17:38:00] that indeed doesn't exist yet [17:38:06] and doesn't have a merged commit for it either afaik [17:38:07] !log Pushed new mwext-php72-phan-docker with jjb [17:38:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:38:18] (03CR) 10Jforrester: [C: 03+2] "…" [integration/config] - 10https://gerrit.wikimedia.org/r/516301 (owner: 10Jforrester) [17:38:41] Also, yes, the 5MB output file with "error" at the end isn't great. :-) [17:38:53] nvm, https://gerrit.wikimedia.org/r/#/c/integration/config/+/515290/ does indeed specify it, and I did deply it [17:38:54] weird [17:39:10] * James_F shrugs. [17:39:55] (03Merged) 10jenkins-bot: layout: Add quibble experimental mwext-php72-phan-docker, alongside php70 [integration/config] - 10https://gerrit.wikimedia.org/r/516301 (owner: 10Jforrester) [17:41:51] !log Reloading zuul to Add experimental mwext-php72-phan-docker jobs to quibble template [17:41:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:44:08] (03PS1) 10Legoktm: Set zuul parameters for PHP 7.2 variants of phan jobs too [integration/config] - 10https://gerrit.wikimedia.org/r/516306 [17:44:10] James_F: ^^ [17:45:39] Oh, good point. [17:45:51] legoktm: is the PHP_BIN thing still used? I'd expect that to be dead by ow. [17:45:52] now* [17:46:00] (as far as you know, that is) [17:46:00] (03CR) 10Jforrester: [C: 03+2] Set zuul parameters for PHP 7.2 variants of phan jobs too [integration/config] - 10https://gerrit.wikimedia.org/r/516306 (owner: 10Legoktm) [17:46:07] I'm 99% sure its totally unused [17:46:59] unless there's some legacy non-docker job lying around that's still running? [17:47:02] Hmm. https://integration.wikimedia.org/ci/job/mwext-php72-phan-docker/1/console failed but it doesn't say why… [17:47:14] legoktm: There probably is. It's tech debt all the way down. :-( [17:47:18] James_F: it was 13:42:26 /tmp/jenkins3868979676313044773.sh: line 3: THING_SUBNAME: unbound variable [17:47:29] which my patch will fix [17:47:32] (03Merged) 10jenkins-bot: Set zuul parameters for PHP 7.2 variants of phan jobs too [integration/config] - 10https://gerrit.wikimedia.org/r/516306 (owner: 10Legoktm) [17:48:37] !log Reloading zuul to set zuul parameters for PHP 7.2 variants of phan jobs too [17:48:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:52:08] OK, works for UrlShortner. Do people have thoughts as to which repos will be stress-tests for phan on 72 vs. 70? :-) [17:52:31] AF? [17:52:39] CN? [17:52:49] MW Core? ;p [17:55:14] Core comes later. :-) [17:56:56] James_F: core, Scribunto, Wikibase, Linter, and MassMessage are my usual targets. The first three have basically all the weirdest/most complex code possible, and the last two are my own projects :) [17:57:20] Kk. [17:58:35] Running AF/CN/Scrib/WB/Linter/MM. [17:59:02] for core specifically, https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/master/.phan/internal_stubs/ were generated for php7.0, I don't know if we need to regenerate them for 7.2 [17:59:08] oooh wikibase. good call [17:59:25] I probably should have written down how I generated them because I've mostly forgotten now [18:00:32] AbuseFilter, CentralNotice, Scribunto, Linter, and MassMessage pass on php70 and php72; Wikibase fails in php72 but not php70. :-( [18:00:52] hmmm funky [18:01:13] 10:59:21 [InvalidArgumentException] [18:01:13] 10:59:21 Package phan/phan at version 0.8 has a PHP requirement incompatible with your PHP version (7.2.19) [18:01:14] ouch [18:01:20] * James_F sighs. [18:01:30] if only we could give 7.0 the boot [18:01:35] That's the plan. [18:01:45] you know me. I want 7.0 and hhvm gone gone gone [18:01:54] Not just you. :-) [18:01:58] :-D [18:02:07] James_F: oh bleh, Wikibase is still on an old phan version. [18:02:12] Given I wrote the patch for MW dropping HHVM support… [18:02:20] Matej was working on the upgrade patch IIRC [18:02:25] legoktm: Is it time to just upgrade them and disable what breaks/ [18:02:34] (It is.) [18:03:02] who's the point person for wb these days [18:03:22] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/494176 [18:03:54] ohsigh [18:04:02] legoktm: Shall we just mute the current warnings and merge? [18:04:23] The maintainers can fix and re-enable on their own time without holding up the rest of the world. [18:05:32] there's a bunch of weird PhanRedefinedInheritedInterface errors [18:05:41] something is wrong with their configures paths I think. [18:07:33] 10Continuous-Integration-Config, 10Release-Engineering-Team: Remove PHP_BIN from WMF CI job configuration - https://phabricator.wikimedia.org/T225451 (10Krinkle) [18:08:25] 10Continuous-Integration-Config, 10Release-Engineering-Team: Remove PHP_BIN from WMF CI job configuration - https://phabricator.wikimedia.org/T225451 (10Legoktm) ` [13:46:07] I'm 99% sure its totally unused [13:46:59] unless there's some legacy non-docker job lying around that's still runni... [18:09:00] legoktm: Hmm, integration/config is not in code search? [18:09:31] https://gerrit.wikimedia.org/r/c/labs/codesearch/+/494720 [18:10:07] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Operations, 10SRE-Access-Requests: Request: add awight to contint-docker - https://phabricator.wikimedia.org/T223262 (10greg) If you really want :) Approved. [18:10:43] Ta. [18:49:12] (03CR) 10Jforrester: [C: 03+2] "As this is experimental-only, this doesn't break anything." [integration/config] - 10https://gerrit.wikimedia.org/r/515292 (owner: 10Jforrester) [18:50:57] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban): Run phan on PHP 7.2, not PHP 7.0 - https://phabricator.wikimedia.org/T225456 (10Jdforrester-WMF) [18:51:31] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban): Run phan on PHP 7.2, not PHP 7.0 - https://phabricator.wikimedia.org/T225456 (10Jdforrester-WMF) [18:51:33] 10Continuous-Integration-Config, 10phan: Prepare phan-docker on CI for phan2.0.0 - https://phabricator.wikimedia.org/T223847 (10Jdforrester-WMF) [18:52:19] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban): Run phan on PHP 7.2, not PHP 7.0 - https://phabricator.wikimedia.org/T225456 (10Jdforrester-WMF) [18:53:22] 10Continuous-Integration-Config: Move all CI generic tasks from PHP70 to PHP72 - https://phabricator.wikimedia.org/T225457 (10Jdforrester-WMF) [18:53:28] o/ [18:53:41] i'm getting no space left on device errors when in a docker build job [18:53:42] https://integration.wikimedia.org/ci/blue/organizations/jenkins/service-pipeline-test-and-publish/detail/service-pipeline-test-and-publish/242/pipeline/ [18:53:50] 10Continuous-Integration-Config: Move all CI generic tasks from PHP70 to PHP72 - https://phabricator.wikimedia.org/T225457 (10Jdforrester-WMF) [18:53:54] 10Continuous-Integration-Config, 10Fresnel, 10Performance-Team: Switch Fresnel for MediaWiki jobs from php70 to php72 - https://phabricator.wikimedia.org/T220866 (10Jdforrester-WMF) [18:54:01] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban): Run phan on PHP 7.2, not PHP 7.0 - https://phabricator.wikimedia.org/T225456 (10Jdforrester-WMF) [18:54:03] 10Continuous-Integration-Config: Move all CI generic tasks from PHP70 to PHP72 - https://phabricator.wikimedia.org/T225457 (10Jdforrester-WMF) [18:54:10] the heck is that [18:54:22] a new jenkins UI? [18:54:28] oh ya blue ocean [18:54:35] there's a link in all the jobs to use it [18:54:39] 10Continuous-Integration-Config, 10Parsoid-PHP: Parsoid/PHP post-merge doc publishing task needs updating to use PHP 7.2 - https://phabricator.wikimedia.org/T219319 (10Jdforrester-WMF) [18:54:41] 10Continuous-Integration-Config: Move all CI generic tasks from PHP70 to PHP72 - https://phabricator.wikimedia.org/T225457 (10Jdforrester-WMF) [18:54:56] here [18:54:56] https://integration.wikimedia.org/ci/job/trigger-service-pipeline-test-and-publish/243/ [18:55:22] ah sorry no [18:55:22] this on [18:55:23] e [18:55:23] https://integration.wikimedia.org/ci/job/service-pipeline-test-and-publish/241/ [18:55:35] Krenair: Not "new", but only used by the pipeline jobs so you won't see it much. [18:55:58] during npm install [18:55:58] npm WARN tar ENOSPC: no space left on device, write [18:56:02] * James_F nods. [18:56:09] Running in Durability level: MAX_SURVIVABILITY [18:56:37] so how do you tell which instance ran this thing [18:57:08] /dev/md0 46G 44G 0 100% / [18:57:12] Krenair: it's on contint1001 [18:57:20] oh [18:57:35] ottomata: so uh, contint1001 is out of disk space [18:58:05] thcipriani: marxarelli|afk: ^^ [18:58:12] did we have a cleanup command/script? [18:58:58] We do. [18:59:08] * James_F looks. [18:59:14] * thcipriani looks [18:59:24] * thcipriani runs cleanup script [18:59:28] * James_F backs off and leaves to the actual expert. [18:59:49] thcipriani: for future reference, where is said script? :) [18:59:52] We really need to do T207707 [18:59:53] T207707: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 [19:01:25] RECOVERY - Disk space on contint1001 is OK: DISK OK https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space [19:01:46] !log cleaning contint1001 docker images using https://phabricator.wikimedia.org/P8604 [19:01:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:02:22] ^ legoktm is the script I've been using. Keeps all "latest" to keep docker-pkg happy, removes all else [19:03:03] I really should move a bunch of things over to use the latest docker images; we've got things in a mix. [19:03:11] IIRC docker-pkg now has a prune command that does roughly the same thing [19:03:12] thcipriani: ty [19:03:14] Which really doesn't help with space. [19:03:34] ottomata: so there's a bit more space now if you re try the job [19:03:36] (but I don't know if the version of docker-pkg with prune is what's deployed or not) [19:03:40] k rtrying [19:04:14] (03CR) 10jerkins-bot: [V: 04-1] layout: Drop quibble experimental php70-phan-docker jobs; php72 suffices [integration/config] - 10https://gerrit.wikimedia.org/r/515292 (owner: 10Jforrester) [19:04:39] (03CR) 10Jforrester: [C: 03+2] "(Manually aborted as the tox job got stuck due to full disc; re-trying.)" [integration/config] - 10https://gerrit.wikimedia.org/r/515292 (owner: 10Jforrester) [19:04:41] thcipriani: docker-pkg --help shows prune on contint1001 [19:06:07] (03CR) 10Jforrester: [C: 04-1] "This will break (at least) Wikibase, sadly, until T217018 is fixed. :-(" [integration/config] - 10https://gerrit.wikimedia.org/r/515293 (owner: 10Jforrester) [19:07:55] legoktm: ack. Should probably just cron that to run. Shouldn't be disruptive to run whenever, I think. [19:08:15] * thcipriani makes note to self [19:08:25] worked this time, thank you all! [19:10:34] !log Fix "setup quibble mw-install" collapsible section pattern in Jenkins administration, so that it doesn't match the "PHP 7.1+" warning from Phan. The pattern is meant to detect the first line of mw/maint/install.php output [19:10:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:14:09] Was phan-on-php70 just upgraded? It's warning on core commits for something that hasn't changed recently. [19:14:28] (also, shouldn't that be upgraded via composer.json per-repo, if not already?) [19:16:00] Krinkle: The docker image? No. [19:17:34] Looks like it comes from the docker image, but it installs at run-time via PHAN_VERSION [19:17:47] mediawiki-phan on 7.2 is 0.2.0 added on Friday in 59d6549c5e127acf07d6951215b7c352aba25b57 and not (yet) used anywhere except experimental jobs; before that it was 7.0 via 0.1.15 last touched a7fbbd518ff5ba3b18527ab1fbd7d2c5bd0bff57 in mid-May. [19:18:25] Oh, the version of phan? Erm. I dunno. [19:18:29] which in turn comes from composer.json#extra.phan in mediawiki/core (absent) with fallback to vendor/mediawiki/mediawiki-phan-config/composer.json [19:18:35] Right. [19:18:39] so is properly decided within the repo, good good [19:18:52] but apparently upgrading it doesn't have it run on thhe repo so doesn't catch regressions. [19:18:56] Indirection magic. It's what CIruns. [19:18:59] It only runs on files that were changed -_- [19:19:07] Ah. [19:19:22] And when you're bumping it in composerjson, no php files are changed so... [19:19:23] ugh [19:19:28] Yeah… [19:19:40] E_TOOMUCHMAGIC. [19:20:03] What do we usually do when Phan is whining about new rules for existing code? Do we disable by default and then post-upgrade have a commit that proposes to enable it once it passes on core? [19:20:20] Looks like PhanInfiniteRecursion is being a bit too smart. [19:20:27] at https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/514370/ [19:20:57] Krinkle: It's what we do with other CI tools, so… yes/ [19:21:21] I vaguely recall that the mw-phan-config repo has a CI job that makes it run over all of core [19:21:23] but maybe that's phpcs [19:23:29] (03PS5) 10Jforrester: layout: Drop quibble experimental php70-phan-docker jobs; php72 suffices [integration/config] - 10https://gerrit.wikimedia.org/r/515292 [19:24:15] Looks like we don't have such a job [19:24:16] https://gerrit.wikimedia.org/r/#/c/mediawiki/tools/phan/+/509582/ [19:24:26] These just upgrade phan and then get released and then upgraded by LU. [19:24:29] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews: Code Stewardship Review: OAuth extension - https://phabricator.wikimedia.org/T224919 (10Anomie) I found this after seeing [[https://www.mediawiki.org/w/index.php?diff=3261645&oldid=3261642]] and searching, as the relevant project tags don't seem... [19:25:03] s/LU/LibUp/ [19:25:14] And yes. [19:31:51] CI jobs running out of headroom? [19:32:21] Doesn't seem to be deadlocked on a particular executor, it's just not assigning new jobs. [19:32:50] Also "Your Jenkins data directory /var/lib/jenkins (AKA JENKINS_HOME) is almost full. You should act on it before it gets completely full." [19:34:03] oh [19:34:09] you need to restart jenkins [19:34:13] after a OOS [19:34:15] i think [19:34:32] at least i've seen #jenkins-infra having to do that. [19:35:31] Kk. [19:37:21] !log Restarting jenkins with `sudo service restart jenkins` on contint1001 due to out-of-space issue earlier. [19:37:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:37:53] Belay that, apparently I'm not enough of a sudoer. :-) [19:38:06] thcipriani: Could you kick jenkins? [19:38:51] hrm, sure [19:39:44] !log restarting jenkins [19:39:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:41:41] looks to be back, jobs seem to be running [19:42:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Operations, 10SRE-Access-Requests: Request: add awight to contint-docker - https://phabricator.wikimedia.org/T223262 (10awight) >>! In T223262#5247693, @greg wrote: > If you really want :) Approved. #masochism not found [19:43:28] (03CR) 10Jforrester: [C: 03+2] "…" [integration/config] - 10https://gerrit.wikimedia.org/r/515292 (owner: 10Jforrester) [19:48:00] 10Release-Engineering-Team (Backlog), 10MediaWiki-Release-Tools, 10Core Platform Team (Release Automation (TEC1)), 10Core Platform Team Kanban (Contractor - Ready), 10MediaWiki-Releasing (Workflow Improvements): Bot to upgrade the different bits of mw.o for tarba... - https://phabricator.wikimedia.org/T217964 [19:48:27] (03Merged) 10jenkins-bot: layout: Drop quibble experimental php70-phan-docker jobs; php72 suffices [integration/config] - 10https://gerrit.wikimedia.org/r/515292 (owner: 10Jforrester) [19:50:06] !log Restarting zuul to drop quibble experimental php70-phan-docker jobs; php72 suffices [19:50:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:55:22] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews: Code Stewardship Review: OAuth extension - https://phabricator.wikimedia.org/T224919 (10Krinkle) >>! In T224919#5247965, @Anomie wrote: > {{citation needed}}. Tests on recent patches such as https://gerrit.wikimedia.org/r/c/mediawiki/extensions/O... [19:56:49] Krinkle: yeah, we don't have good ahead of time CI set up yet. I usually test it locally. If there's a rule that's problematic, disable it in the repo, file a bug in #phan, and usually one of us will look at it a bit more closely, and decide whether to take it upstream and/or globally disable it [20:13:12] Well, that's not helpful [20:13:15] Scribunto now failing with [20:13:15] 21:11:58 30 | WARNING | Deprecated global $wgContLang used [20:13:15] 21:11:58 | | (MediaWiki.Usage.DeprecatedGlobalVariables.Deprecated$wgContLang) [20:13:16] et [20:13:17] c [20:15:10] Reedy: Yeah, that's because mwcs hasn't been updated in that repo but also has. Hundreds of repos are affected. [20:15:16] 10Release-Engineering-Team, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Security: CentralNotice setting a surprising content security policy in production when using &banner= URL parameter - https://phabricator.wikimedia.org/T225261 (10Krenair) [20:17:46] 10Release-Engineering-Team (Kanban), 10Scap: php-1.33.0-wmf.23/cache/l10n isn't cleaned up in prod - https://phabricator.wikimedia.org/T224637 (10thcipriani) p:05Triage→03Normal a:03thcipriani Guessing this is the aborted `clean` from https://tools.wmflabs.org/sal/log/AWq2FV6OOwpQ-3Pk_Reb I think this s... [20:25:39] 10Release-Engineering-Team, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Security: CentralNotice setting a surprising content security policy in production when using &banner= URL parameter - https://phabricator.wikimedia.org/T225261 (10Krinkle) I'll also point out that `banner=` are subj... [20:30:53] James_F: Regarding slow test jobs, especially the shared gate and the Ciruss tests we identified, is there an epic task yet for e.g. cutting e.g. 5 minutes off with the idea to do so based on the slowest tests we find? [20:30:56] Closest task I found was https://phabricator.wikimedia.org/T93556 [20:35:01] Krinkle: T221434 [20:35:01] T221434: Ensure we're testing appropriately and not over-testing across Wikimedia-deployed code - https://phabricator.wikimedia.org/T221434 [20:35:17] 10Continuous-Integration-Config, 10MediaWiki-extensions-Translate: Translate coverage publish step fails with: Incorrect whitelist config, no code coverage will be generated. - https://phabricator.wikimedia.org/T221913 (10awight) I can confirm the work that's already been done on this ticket. As Thiemo pointe... [20:35:25] Krinkle: But not a specific time goal task yet. [20:35:55] I don't want us to sacrifice code coverage on the altar of achieve 4m 59s over 5m 10s or whatever. [20:40:27] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews: Code Stewardship Review: OAuth extension - https://phabricator.wikimedia.org/T224919 (10Anomie) >>! In T224919#5248030, @Krinkle wrote: >>>! In T224919#5247965, @Anomie wrote: >> {{citation needed}}. Tests on recent patches such as https://gerrit... [20:49:39] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews: Code Stewardship Review: OAuth extension - https://phabricator.wikimedia.org/T224919 (10Krinkle) Yeah, OAuth's PHPUnit test classes weren't failing because it didn't cover or integrate with any core or extension code that depended on the PHP feat... [20:54:00] James_F: I see. Yeah, my motivation isn't about reducing test coverage. I suppose it is related in that it reduces run-time as well, but my thinking is more to generally focus on quality of unit tests. [20:54:15] We've done sprints like this before and generally did not involve reduction in code coverage. [20:54:40] I'm not convinced at this point that we need to make those kinds of "hard" choices yet. [20:54:43] Krinkle: Totally, I'm just nervous about writing the top-level task with a target before we get a sense of where the big wins are. [20:55:37] The two pieces of work to split int/unit and to skip isolated internal tests except for patches to the owning repo will likely be major savings. [20:55:46] (He says, optimistically.) [20:57:00] Yeah, but that also seems significantly harder and more work than to improve a few tests to not perform unit-test-like iterations and variants over a test case that is effectively an integration test. E.g. best practice I follow is generally to cover a user story once or twice with an integration test, and then to perform some hot loops for many input/output variants at a lower level. [20:57:21] When unit tests are well-written and only take a few seconds in total, it might not be worth weeks of work to try and carefully run a subset of them. [20:57:35] Input welcome on T225068 ;-) [20:57:36] T225068: Add a PHPUnit group to skip test on gated CI runs - https://phabricator.wikimedia.org/T225068 [20:58:24] OK. I've added those two tasks to my note pad. Will circle back in a month or so. [20:58:46] Adding a new group definition in MW core, slapping it on a few major tests, and tweaking quibble to not run them except once doesn't feel like "weeks of work". [20:58:49] I'd like to set 10 minutes as a goal for the shared gate job, and then consider these two and mine as a way to achieve it. [20:58:49] * James_F nods. [20:59:04] Yeah, 5–10-ish is the objective. [20:59:15] I'd prefer for the target to be 5 and us be OK with only achieving 10. [20:59:30] But as I said, I'd reticent to commit right now. [21:00:04] Reedy: MWCS is now "smart" and if you bump the required MW version, then it starts warning about new stuff, because it can [21:00:59] Yeah, I think it's important to acknowledge though that this kind of grouping and disabling will reduce test coverage effectively. Given the shared run-time that is MW, there is no such thing as a purely isolated unit test, except for code outside core in extensions that only uses classes from the same repo and not interact with anything from core, vendor, or other extensions. [21:01:22] Of course, we may want to decide for speed reasons to lower our integration confidence and disable some we think aren't worth running in the gate. But it's not risk free. [21:02:38] Krinkle: Even then, it's trivially possible to poison other people's code by fiddling with globals. [21:04:19] "code outside core in extensions that only uses [..]" implies it does not use globals, e.g. injected parameters. Such quality code may be rare, but it's not non-existent. [21:05:15] It's as close to non-existent as makes no difference. :-0 [21:07:57] (03PS6) 10Jforrester: Stop running PHP55 tests anywhere. [integration/config] - 10https://gerrit.wikimedia.org/r/514031 (https://phabricator.wikimedia.org/T224907) [21:10:14] (03CR) 10jerkins-bot: [V: 04-1] Stop running PHP55 tests anywhere. [integration/config] - 10https://gerrit.wikimedia.org/r/514031 (https://phabricator.wikimedia.org/T224907) (owner: 10Jforrester) [21:13:22] * James_F sighs. No idea what random bit of python that's from. [21:28:16] 10Continuous-Integration-Config, 10phan: Prepare phan-docker on CI for phan2.0.0 - https://phabricator.wikimedia.org/T223847 (10Jdforrester-WMF) Next step is actually moving to phan 2.x? Would that be this task or a different one? If the latter, is this now Resolved? [21:41:23] https://gitenterprise.me/2019/06/10/gerrithub-io-is-moving-to-gerrit-v3-0/ [21:46:23] 10Continuous-Integration-Infrastructure, 10Zuul: Depends-On should not work across branches - https://phabricator.wikimedia.org/T210685 (10matmarex) This just caused an issue in VisualEditor and MobileFrontend because a core change was backported to REL1_33 without its dependencies in the extensions (marked wi... [21:58:39] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Wikidata-Campsite, 10MW-1.33-release: Class undefined: Wikibase\DataModel\Entity\ItemId in WikibaseLib.entitytypes.php on line 35 - https://phabricator.wikimedia.org/T225364 (10matmarex) [21:58:41] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10AbuseFilter, 10CX-deployments, and 2 others: mediawiki/vendor REL1_* no longer ship dependencies for wmf extensions that are not in the mediawiki tarball - https://phabricator.wikimedia.org/T189560 (10matmarex) [21:59:40] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10AbuseFilter, 10CX-deployments, and 2 others: mediawiki/vendor REL1_* no longer ship dependencies for wmf extensions that are not in the mediawiki tarball - https://phabricator.wikimedia.org/T189560 (10matmarex) [22:05:53] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews: Code Stewardship Review: OAuth extension - https://phabricator.wikimedia.org/T224919 (10Tgr) I have been thinking about filing something like this for a while. Given the importance of OAuth (the majority of external tools rely on it, and it's a b... [22:35:10] legoktm: Also CirrusSearch: https://codesearch.wmflabs.org/search/?q=enable_class_alias_support&i=nope&files=&repos= [22:38:25] greg-g: What's the rule on deploys to fix broken stuff? [22:39:02] (03CR) 10Brennen Bearnes: [V: 03+2 C: 03+2] "Seems very sensible, tested and working." [releng/local-charts] - 10https://gerrit.wikimedia.org/r/514637 (owner: 10Jeena Huneidi) [22:39:03] Reedy asking for deployment rules? :) [22:39:13] Reedy i think it's UBN fixes. [22:40:01] i think otto asked some where [22:40:29] I mean, SRE have been deploying today anyway :P [22:40:35] Reedy see back scroll of #wikimedia-sre [22:41:03] There's only back scroll if you were present for the conversation [22:41:11] oh, you wern't in that channel [22:41:13] Reedy no deploys unless ubn/emergency [23:24:48] Reedy: sorry, UBN! and as minimal as possible