[01:43:24] 10Continuous-Integration-Infrastructure, 10Confirmation prompt for rollback action, 10Browser-Tests: Fix flaky Selenium test: Rollback without confirmation should perform rollback via GET request without asking the user to confirm - https://phabricator.wikimedia.org/T226144 (10kostajh) [02:15:36] 10Continuous-Integration-Infrastructure, 10Confirmation prompt for rollback action, 10Browser-Tests, 10Patch-For-Review: Fix flaky Selenium test: Rollback without confirmation should perform rollback via GET request without asking the user to confirm - https://phabricator.wikimedia.org/T226144 (10kostajh) [04:07:17] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<22.22%) [06:47:18] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:29:02] (03PS1) 10Hashar: Add extension-quibble to gated repositories [integration/config] - 10https://gerrit.wikimedia.org/r/517982 [07:30:29] (03CR) 10jerkins-bot: [V: 04-1] Add extension-quibble to gated repositories [integration/config] - 10https://gerrit.wikimedia.org/r/517982 (owner: 10Hashar) [07:31:09] (03PS2) 10Hashar: Add extension-quibble to gated repositories [integration/config] - 10https://gerrit.wikimedia.org/r/517982 [07:31:44] (03CR) 10Hashar: "PS1 just add the test" [integration/config] - 10https://gerrit.wikimedia.org/r/517982 (owner: 10Hashar) [07:32:35] (03CR) 10jerkins-bot: [V: 04-1] Add extension-quibble to gated repositories [integration/config] - 10https://gerrit.wikimedia.org/r/517982 (owner: 10Hashar) [07:39:35] (03PS3) 10Hashar: Add extension-quibble to gated repositories [integration/config] - 10https://gerrit.wikimedia.org/r/517982 [07:42:35] (03CR) 10Hashar: [C: 03+2] Add extension-quibble to gated repositories [integration/config] - 10https://gerrit.wikimedia.org/r/517982 (owner: 10Hashar) [07:44:02] (03Merged) 10jenkins-bot: Add extension-quibble to gated repositories [integration/config] - 10https://gerrit.wikimedia.org/r/517982 (owner: 10Hashar) [08:00:07] (03PS1) 10Pwirth: Add test for mediawiki/extensions/OOJSPlus [integration/config] - 10https://gerrit.wikimedia.org/r/517985 [08:03:04] (03PS2) 10Pwirth: Add test for mediawiki/extensions/OOJSPlus [integration/config] - 10https://gerrit.wikimedia.org/r/517985 [08:08:57] 10Continuous-Integration-Config, 10fundraising-tech-ops: Phan job fails on CI for mediawiki/core fundraising/REL1_31 - https://phabricator.wikimedia.org/T226156 (10hashar) [08:12:44] (03PS1) 10Hashar: Skip php7.2 phan analysis for fundraising/REL1_31 [integration/config] - 10https://gerrit.wikimedia.org/r/517988 (https://phabricator.wikimedia.org/T226156) [08:16:24] (03PS1) 10Mainframe98: Archive the FirstSteps extension [integration/config] - 10https://gerrit.wikimedia.org/r/517993 (https://phabricator.wikimedia.org/T226149) [08:17:22] (03PS1) 10Hashar: zuul: drop filters for REL1_27 to REL1_29 [integration/config] - 10https://gerrit.wikimedia.org/r/517995 [08:22:26] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Jenkins: Upgrade jenkins instances to 2.171.1 - https://phabricator.wikimedia.org/T226159 (10hashar) [08:25:36] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Jenkins: Upgrade jenkins instances to 2.171.1 - https://phabricator.wikimedia.org/T226159 (10hashar) Guten tag @MoritzMuehlenhoff , would you be able to update the Jenkins Debian package on our apt.wikimedia.org repository please... [08:25:45] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Jenkins: Upgrade jenkins instances to 2.171.1 - https://phabricator.wikimedia.org/T226159 (10hashar) p:05Triage→03Normal [08:28:21] (03CR) 10Hashar: [C: 03+2] Add test for mediawiki/extensions/OOJSPlus [integration/config] - 10https://gerrit.wikimedia.org/r/517985 (owner: 10Pwirth) [08:29:48] (03Merged) 10jenkins-bot: Add test for mediawiki/extensions/OOJSPlus [integration/config] - 10https://gerrit.wikimedia.org/r/517985 (owner: 10Pwirth) [08:31:22] (03CR) 10Hashar: [C: 03+2] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/517985 (owner: 10Pwirth) [09:02:50] (03CR) 10Hashar: [C: 03+2] Archive the FirstSteps extension [integration/config] - 10https://gerrit.wikimedia.org/r/517993 (https://phabricator.wikimedia.org/T226149) (owner: 10Mainframe98) [09:04:19] (03Merged) 10jenkins-bot: Archive the FirstSteps extension [integration/config] - 10https://gerrit.wikimedia.org/r/517993 (https://phabricator.wikimedia.org/T226149) (owner: 10Mainframe98) [09:04:21] (03CR) 10Hashar: [C: 03+2] zuul: drop filters for REL1_27 to REL1_29 [integration/config] - 10https://gerrit.wikimedia.org/r/517995 (owner: 10Hashar) [09:05:25] (03CR) 10Hashar: [C: 03+2] Skip php7.2 phan analysis for fundraising/REL1_31 [integration/config] - 10https://gerrit.wikimedia.org/r/517988 (https://phabricator.wikimedia.org/T226156) (owner: 10Hashar) [09:05:50] (03Merged) 10jenkins-bot: zuul: drop filters for REL1_27 to REL1_29 [integration/config] - 10https://gerrit.wikimedia.org/r/517995 (owner: 10Hashar) [09:06:49] (03Merged) 10jenkins-bot: Skip php7.2 phan analysis for fundraising/REL1_31 [integration/config] - 10https://gerrit.wikimedia.org/r/517988 (https://phabricator.wikimedia.org/T226156) (owner: 10Hashar) [09:07:04] Question - I was planning to switch on a feature flag in next SWAT, but with group1 still on wmf.8, that's not the greatest idea... [09:07:05] I could backport a few patches to wmf.8 (in an extension) - just not sure whether that'd be ok (IDK what procedure is now that train is delayed) [09:08:08] matthiasmullie: I think group1 is on wmf.10 since Tuesday? [09:08:25] Sorry! must have been a rollback? [09:08:39] yes a revert [09:08:44] yesterday, job queue issues [09:08:57] both 0 and 1 reverted in the end [09:09:19] yeah [09:09:32] apergos: Thanks for the news, I'll read SAL & things [09:12:32] Wow, nothing in the SAL--it must have been an emergency... [09:13:41] ? [09:13:43] It's in the SAL [09:13:49] a lot of of job queue jobs weren't working right [09:14:07] matthiasmullie: Why would you necessarily need to backport to .8? [09:14:26] I dunno if/when the train is likely to move forward, I don't think the cause of the breakage has been identified yet [09:15:00] next SWAT was scheduled to enable new feature, which - in .8 - has a few bugs still [09:15:06] Reedy: gotcha, the grep side of my brain isn't working yet [09:15:15] (03CR) 10Hashar: [C: 03+1] jjb: Migrate docker-setup-mwext-for-phan from php70 to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/517769 (owner: 10Jforrester) [09:15:29] (03CR) 10Hashar: [C: 03+1] jjb: Migrate wikibase-{client|repo}-docker from php70 to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/517768 (owner: 10Jforrester) [09:15:29] matthiasmullie: If they're bugfix backports... Rather than the whole feature [09:15:33] How many patches/files touched? [09:16:50] 5 patches, 10-ish files (1 of which PHP) [09:17:26] (and an i18n change) [09:21:09] hmm, i18n requires a full scap [09:21:42] I mean, that's not unreasonable, but will take most of the SWAT window [09:21:49] So could depend what else is scheduled [09:22:01] a lot has been scheduled [09:22:11] wouldn't want to do that during swat - should probably find another window then [09:22:23] we *could* postpone, but feature has already been announced, and the team is on offsite next week (could still deploy then, but little harder to follow up) [09:23:03] wouldn't be the end of the world if we'd had to postpone things [09:23:24] but if doing it today wouldn't hurt anyone else, might as well :p [09:29:54] Those patches bring .8 upto the same as .10 basically? [09:31:00] about half the way there (there are other - unrelated - in .10 patches that wouldn't have to be backported) [09:31:48] That all doesn't sound unreasonable :) [09:32:52] the train is on US windows, so might not be any action till much later today [09:33:57] I suggest I pull my patches from next SWAT window, then use the hour after SWAT (so in 1:30) to do some backports to extensions/MediaInfo and wmf-config, and scap - sounds good? [09:42:32] Sounds reasonable to me, yeag [09:42:43] Can always start a bit early if swat gets done early :) [09:43:40] oh [09:43:45] I messed up timezones [09:43:54] SWAT is still 1:15 out, right? [09:44:05] I could even do things now, before swat... [09:46:52] alright - I'll start merging patches [13:08:06] PROBLEM - puppet last run on contint1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[jenkins] [13:34:03] hashar: would you mind checking to see if https://gerrit.wikimedia.org/r/c/integration/config/+/517681 was deployed? [13:34:33] From looking at https://integration.wikimedia.org/ci/job/mwext-codehealth-patch/666/console (09:24:39) it doesn't seem to have been [13:44:12] kostajh: checking :D [13:44:20] the docker container should have [13:44:33] and yeah they have [13:45:30] kostajh: the jobs are apparently up to date [13:45:37] hmm [13:48:34] --env-file /dev/fd/63 docker-registry.wikimedia.org/releng/quibble-coverage:0.0.31-5 [13:49:24] https://github.com/wikimedia/integration-config/blob/master/dockerfiles/quibble-coverage/changelog [13:49:31] So it should be running a newer version than yours... [13:49:48] Yeah was just looking at that, thanks Reedy. I'll see if my change got dropped [13:51:15] I'm confused though, since mwext-phpunit-coverage.sh looks to have my changes in master [13:52:01] running the image locally, I see the updated script too [13:53:22] oh, I think I have an idea. [13:56:00] silly mistake on my part :( [13:56:17] (03PS1) 10Hashar: Stop running wmf-quibble gating job with php7.0/php7.1 [integration/config] - 10https://gerrit.wikimedia.org/r/518033 [13:59:46] (03PS1) 10Kosta Harlan: Codehealth: Fix logic for checking if HTML coverage was generated [integration/config] - 10https://gerrit.wikimedia.org/r/518034 [14:08:44] (03PS2) 10Hashar: Stop running wmf-quibble gating job with php7.0/php7.1 [integration/config] - 10https://gerrit.wikimedia.org/r/518033 [14:08:46] (03PS1) 10Hashar: Ensure test has proper coverage [integration/config] - 10https://gerrit.wikimedia.org/r/518035 [14:09:34] (03CR) 10Hashar: [C: 03+2] Codehealth: Fix logic for checking if HTML coverage was generated [integration/config] - 10https://gerrit.wikimedia.org/r/518034 (owner: 10Kosta Harlan) [14:09:35] :] [14:09:53] hashar: cheers [14:10:31] we shuld probably move those script to composer plugins/packages etc [14:11:04] (03Merged) 10jenkins-bot: Codehealth: Fix logic for checking if HTML coverage was generated [integration/config] - 10https://gerrit.wikimedia.org/r/518034 (owner: 10Kosta Harlan) [14:16:04] !log building some container for codehealth pipeline https://gerrit.wikimedia.org/r/518034 [14:16:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:25:07] (03PS1) 10Kosta Harlan: Bump coverage jobs to use latest quibble-coverage image [integration/config] - 10https://gerrit.wikimedia.org/r/518037 [14:25:21] hashar: if you prefer that I just bump the codehealth job instead of all of them, let me know ^ [14:25:33] hiya [14:25:38] i used to have project creation rights in gerrit [14:25:41] can i get them back? :) [14:27:45] Some how a bug in gerrit 3.0 went unnoticed heh [14:28:02] * paladox fixes that in https://gerrit-review.googlesource.com/c/gerrit/+/228732 [14:30:06] (03CR) 10Thcipriani: [V: 03+2 C: 03+2] Merge remote-tracking branch 'upstream/v2.15.14' into wmf/stable-2.15 [software/gerrit] (wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/516727 (owner: 10Paladox) [14:30:15] \o/ thcipriani [14:34:22] (03PS1) 10Thcipriani: Gerrit v2.15.14 [software/gerrit] (deploy/wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/518039 [14:34:26] (03CR) 10Hashar: [C: 03+2] Ensure test has proper coverage [integration/config] - 10https://gerrit.wikimedia.org/r/518035 (owner: 10Hashar) [14:34:42] (03CR) 10Hashar: [C: 03+2] Stop running wmf-quibble gating job with php7.0/php7.1 [integration/config] - 10https://gerrit.wikimedia.org/r/518033 (owner: 10Hashar) [14:34:52] (03CR) 10Paladox: [C: 03+2] Gerrit v2.15.14 [software/gerrit] (deploy/wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/518039 (owner: 10Thcipriani) [14:35:43] (03Merged) 10jenkins-bot: Ensure test has proper coverage [integration/config] - 10https://gerrit.wikimedia.org/r/518035 (owner: 10Hashar) [14:36:28] (03Merged) 10jenkins-bot: Stop running wmf-quibble gating job with php7.0/php7.1 [integration/config] - 10https://gerrit.wikimedia.org/r/518033 (owner: 10Hashar) [14:37:25] 10Release-Engineering-Team, 10Developer Productivity, 10local-charts: Create an interface for the local-charts ecosystem - https://phabricator.wikimedia.org/T224939 (10kostajh) > @thcipriani: at least on a mac with homebrew, it's as simple as brew install python I dunno, for me that's just where the problem... [14:44:41] !log Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/518033/ [14:44:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:53:28] 10Release-Engineering-Team (Watching / External), 10Scoring-platform-team (Current), 10Wikimedia-Incident: [Spike] Write reports about why Ext:ORES is helping cause server 500s and write tasks to fix - https://phabricator.wikimedia.org/T181010 (10Halfak) [14:56:02] 00:00:02.268 rsync: failed to set times on "/cache/.": Operation not permitted (1) [14:56:02] 00:04:50.945 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.2] [14:56:03] bah [14:56:16] almost 5 minutes just to transfer the cache :\ [14:57:02] 10Release-Engineering-Team, 10Release Pipeline: PipelineBot should be voting - https://phabricator.wikimedia.org/T226191 (10thcipriani) [14:57:34] 10Release-Engineering-Team, 10Release Pipeline: PipelineBot should be voting - https://phabricator.wikimedia.org/T226191 (10thcipriani) p:05Triage→03Normal [15:00:25] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10serviceops: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10thcipriani) >>! In T207707#5239428, @hashar wrote: > The new disks can be shown as sdc and s... [15:06:46] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10serviceops: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10hashar) From a discussion with @mmodell, we might need an extra partition soonish as well.... [15:09:16] !log Nuking castor caches for Quibble jobs integration-castor03 : rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/*quibble*/{composer,npm} [15:09:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:09:47] I still have to restart Jenkins on integration.wikimedia.org :( [15:09:52] will do that later this evening [15:11:55] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Jenkins: Upgrade jenkins instances to 2.171.1 - https://phabricator.wikimedia.org/T226159 (10hashar) I have upgraded Jenkins on release hosts and restarted it. contint1001 still has to be upgraded/restarted. I was looking for a... [15:20:49] (03CR) 10Jforrester: Bump coverage jobs to use latest quibble-coverage image (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/518037 (owner: 10Kosta Harlan) [15:26:23] (03CR) 10Jforrester: "In Ied75a40eb7f I'm proposing removing the complexity of these branch filters by just making them positive matches rather than negative qu" [integration/config] - 10https://gerrit.wikimedia.org/r/517995 (owner: 10Hashar) [15:29:51] (03CR) 10Jforrester: "Oops, forgot about this branch. Thanks!" [integration/config] - 10https://gerrit.wikimedia.org/r/517988 (https://phabricator.wikimedia.org/T226156) (owner: 10Hashar) [15:33:59] (03PS2) 10Kosta Harlan: Bump coverage jobs to use latest quibble-coverage image [integration/config] - 10https://gerrit.wikimedia.org/r/518037 [16:02:41] (03PS2) 10Thcipriani: Update go-playground validator [blubber] - 10https://gerrit.wikimedia.org/r/516671 (https://phabricator.wikimedia.org/T215319) [16:03:01] (03CR) 10Thcipriani: Update go-playground validator (031 comment) [blubber] - 10https://gerrit.wikimedia.org/r/516671 (https://phabricator.wikimedia.org/T215319) (owner: 10Thcipriani) [16:07:39] (03CR) 10Jforrester: [C: 03+1] Update go-playground validator [blubber] - 10https://gerrit.wikimedia.org/r/516671 (https://phabricator.wikimedia.org/T215319) (owner: 10Thcipriani) [16:15:57] (03CR) 10Thcipriani: [C: 03+2] Update go-playground validator [blubber] - 10https://gerrit.wikimedia.org/r/516671 (https://phabricator.wikimedia.org/T215319) (owner: 10Thcipriani) [16:21:28] (03Merged) 10jenkins-bot: Update go-playground validator [blubber] - 10https://gerrit.wikimedia.org/r/516671 (https://phabricator.wikimedia.org/T215319) (owner: 10Thcipriani) [16:22:25] 10Release-Engineering-Team-TODO, 10Scap, 10serviceops, 10User-jijiki: Deploy scap 3.10.0-1 - https://phabricator.wikimedia.org/T224915 (10jijiki) [16:23:33] 10Release-Engineering-Team-TODO, 10Scap, 10serviceops, 10User-jijiki: Deploy scap 3.10.0-1 - https://phabricator.wikimedia.org/T224915 (10jijiki) @thcipriani I have packaged the newer version and uploaded to stretch-wikimedia, I will upgrade the servers soon, time permits. [16:24:20] (03CR) 10PipelineBot: "pipeline-dashboard: blubber-pipeline-publish" [blubber] - 10https://gerrit.wikimedia.org/r/516671 (https://phabricator.wikimedia.org/T215319) (owner: 10Thcipriani) [16:24:22] (03CR) 10PipelineBot: "pipeline-dashboard: blubber-pipeline-publish" [blubber] - 10https://gerrit.wikimedia.org/r/516671 (https://phabricator.wikimedia.org/T215319) (owner: 10Thcipriani) [16:24:24] (03CR) 10jenkins-bot: Update go-playground validator [blubber] - 10https://gerrit.wikimedia.org/r/516671 (https://phabricator.wikimedia.org/T215319) (owner: 10Thcipriani) [16:40:09] (03CR) 10Krinkle: [WIP] layout: Skip PHP70 and PHP71 jobs for MW master; simplify regexes (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/516838 (owner: 10Jforrester) [16:42:32] (03CR) 10Jforrester: [WIP] layout: Skip PHP70 and PHP71 jobs for MW master; simplify regexes (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/516838 (owner: 10Jforrester) [16:42:36] (03CR) 10Jforrester: [C: 04-1] [WIP] layout: Skip PHP70 and PHP71 jobs for MW master; simplify regexes [integration/config] - 10https://gerrit.wikimedia.org/r/516838 (owner: 10Jforrester) [16:49:25] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO, 10Cloud-VPS (Quota-requests), 10cloud-services-team (Kanban): Request new Flavor for integration Cloud VPS project - https://phabricator.wikimedia.org/T225025 (10Andrew) 05Open→03Resolved I added a new flavor named 'mediumram' to... [17:05:33] (03CR) 10Jforrester: [C: 03+2] Bump coverage jobs to use latest quibble-coverage image [integration/config] - 10https://gerrit.wikimedia.org/r/518037 (owner: 10Kosta Harlan) [17:08:27] (03Merged) 10jenkins-bot: Bump coverage jobs to use latest quibble-coverage image [integration/config] - 10https://gerrit.wikimedia.org/r/518037 (owner: 10Kosta Harlan) [17:10:35] 10Release-Engineering-Team-TODO, 10Scap, 10serviceops, 10User-jijiki: Deploy scap 3.10.0-1 - https://phabricator.wikimedia.org/T224915 (10thcipriani) >>! In T224915#5271635, @jijiki wrote: > @thcipriani I have packaged the newer version and uploaded to stretch-wikimedia, I will upgrade the servers soon, ti... [17:12:01] !log Updated JJB definitions for all six coverage jobs [17:12:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:13:29] Bah, too many damn magic commands. [17:13:51] James_F: you got it eventually :) thx for deploying [17:15:17] (03PS2) 10Jforrester: Revert "layout: Temporarily make all seccheck jobs non-voting" [integration/config] - 10https://gerrit.wikimedia.org/r/517765 (https://phabricator.wikimedia.org/T226064) [17:15:22] (03CR) 10Jforrester: [C: 03+2] Revert "layout: Temporarily make all seccheck jobs non-voting" [integration/config] - 10https://gerrit.wikimedia.org/r/517765 (https://phabricator.wikimedia.org/T226064) (owner: 10Jforrester) [17:16:41] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: mwext-php70-phan-seccheck-docker blocking merges - https://phabricator.wikimedia.org/T226064 (10Jdforrester-WMF) >>! In T226064#5267543, @Legoktm wrote: > It wasn't a local patch, it was just that environment variables... [17:17:47] (03Merged) 10jenkins-bot: Revert "layout: Temporarily make all seccheck jobs non-voting" [integration/config] - 10https://gerrit.wikimedia.org/r/517765 (https://phabricator.wikimedia.org/T226064) (owner: 10Jforrester) [17:24:59] James_F: do you mind +2'ing https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/517677? [17:29:18] (03PS2) 10Jforrester: [WIP] layout: Skip PHP70 and PHP71 jobs for MW master [integration/config] - 10https://gerrit.wikimedia.org/r/516838 (https://phabricator.wikimedia.org/T225628) [17:29:20] (03PS1) 10Jforrester: layout: Replace negative branch matches of quibble jobs with positive ones [integration/config] - 10https://gerrit.wikimedia.org/r/518088 [17:29:41] kostajh: Aha, it did pass. Wonderful. [17:29:41] (03CR) 10Smalyshev: [V: 03+2 C: 03+2] Cleanup perms for wikidata/query [wikidata/query/rdf] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/512978 (owner: 10Paladox) [17:30:49] (03CR) 10jerkins-bot: [V: 04-1] [WIP] layout: Skip PHP70 and PHP71 jobs for MW master [integration/config] - 10https://gerrit.wikimedia.org/r/516838 (https://phabricator.wikimedia.org/T225628) (owner: 10Jforrester) [17:30:51] (03CR) 10jerkins-bot: [V: 04-1] layout: Replace negative branch matches of quibble jobs with positive ones [integration/config] - 10https://gerrit.wikimedia.org/r/518088 (owner: 10Jforrester) [17:31:03] (03CR) 10Jforrester: [C: 04-2] "To be really clear. :-)" [integration/config] - 10https://gerrit.wikimedia.org/r/516838 (https://phabricator.wikimedia.org/T225628) (owner: 10Jforrester) [17:31:22] (03CR) 10Jforrester: [C: 03+2] jjb: Migrate wikibase-{client|repo}-docker from php70 to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/517768 (owner: 10Jforrester) [17:33:36] (03Merged) 10jenkins-bot: jjb: Migrate wikibase-{client|repo}-docker from php70 to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/517768 (owner: 10Jforrester) [17:35:24] !log Updated JJB definitions for wikibase-{client|repo}-docker to use php72 not php70. [17:35:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:39:55] (03CR) 10Jforrester: [C: 03+2] jjb: Migrate docker-setup-mwext-for-phan from php70 to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/517769 (owner: 10Jforrester) [17:42:57] (03Merged) 10jenkins-bot: jjb: Migrate docker-setup-mwext-for-phan from php70 to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/517769 (owner: 10Jforrester) [17:45:30] !log Updated JJB definitions for mwext-php7\d-phan(-seccheck)?-docker for new docker-setup-mwext-for-phan macro. [17:45:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:51:17] 10Release-Engineering-Team (Kanban), 10Security-Team, 10phan-taint-check-plugin: Phan-taint-check-plugin not available for PHP > 7.0 - https://phabricator.wikimedia.org/T207344 (10Jdforrester-WMF) Oh, and all that code has been written into [[https://gerrit.wikimedia.org/r/q/project:mediawiki%252Ftools%252Fp... [17:53:23] 10Release-Engineering-Team (Kanban), 10Security-Team, 10phan-taint-check-plugin: Phan-taint-check-plugin not available for PHP > 7.0 - https://phabricator.wikimedia.org/T207344 (10Daimona) Yes, that's because intermediate versions have some incompatibilities. The first working version is the one above anyway... [18:00:46] 10Beta-Cluster-Infrastructure, 10Scoring-platform-team: Grant Andy Craze deployment access for beta cluster - https://phabricator.wikimedia.org/T226207 (10Halfak) [18:01:07] 10Beta-Cluster-Infrastructure, 10Scoring-platform-team: Grant Andy Craze deployment access for beta cluster - https://phabricator.wikimedia.org/T226207 (10Halfak) He is https://wikitech.wikimedia.org/wiki/User:Accraze [18:01:23] 10Beta-Cluster-Infrastructure, 10Scoring-platform-team: Grant Andy Craze deployment access for beta cluster - https://phabricator.wikimedia.org/T226207 (10Halfak) AKA: @ACraze [18:12:34] RECOVERY - puppet last run on contint1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:13:55] ^^ i fixed it [18:14:15] the jenkins package was being held and puppet does not consider it being present [18:58:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Jenkins: Upgrade jenkins instances to 2.171.1 - https://phabricator.wikimedia.org/T226159 (10hashar) 05Open→03Resolved a:03hashar [18:58:47] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Jenkins: Upgrade jenkins instances to 2.171.1 - https://phabricator.wikimedia.org/T226159 (10hashar) https://debmonitor.wikimedia.org/packages/jenkins shows all hosts are up-to-date. [19:23:26] (03PS5) 10Kosta Harlan: Add MediaWiki core to codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/514642 (https://phabricator.wikimedia.org/T224989) [19:24:59] (03PS6) 10Kosta Harlan: Add MediaWiki core to codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/514642 (https://phabricator.wikimedia.org/T224989) [19:27:38] James_F: ^ if you want to try today, I'm around for the next hour [19:31:49] Yeah, was just looking. Let's do it. [19:32:21] (03CR) 10Jforrester: [C: 03+2] Add MediaWiki core to codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/514642 (https://phabricator.wikimedia.org/T224989) (owner: 10Kosta Harlan) [19:34:38] (03CR) 10jerkins-bot: [V: 04-1] Add MediaWiki core to codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/514642 (https://phabricator.wikimedia.org/T224989) (owner: 10Kosta Harlan) [19:35:19] * kostajh looks [19:35:36] Oh, it needs splitting into docker / jjb / layout things probably [19:35:42] * kostajh scrolls through thousands of lines [19:38:07] James_F: not sure I follow [19:38:29] https://gerrit.wikimedia.org/r/c/integration/config/+/514642/6/zuul/layout.yaml#1540 is this line the problem? [19:39:34] You're adding jjb jobs to layout that don't "exist" until after this patch lands. And the jjb jobs refer to a docker image version that doesn't exist until after this patch lands. [19:39:53] If we split into three patches on top of each other I think it'll be happy. [19:41:52] The error (buried in the output) is `FAILURE: Job mwcore-codehealth-master-non-voting not defined`. [19:42:06] (And `FAILURE: Job mwcore-codehealth-patch not defined`.) [19:44:55] James_F: OK, I see. I'll update [19:48:24] (03PS1) 10Kosta Harlan: Codehealth: New image which supports analyzing MediaWiki core [integration/config] - 10https://gerrit.wikimedia.org/r/518120 (https://phabricator.wikimedia.org/T224989) [19:48:33] (03Abandoned) 10Kosta Harlan: Add MediaWiki core to codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/514642 (https://phabricator.wikimedia.org/T224989) (owner: 10Kosta Harlan) [19:51:55] James_F: btw I renamed `mwcore-codehealth` to `mediawiki-core-codehealth` as it seems a bit more consistent with the rest of the projects [19:52:48] (03PS1) 10Kosta Harlan: Codehealth: JJB config for mediawiki-core [integration/config] - 10https://gerrit.wikimedia.org/r/518121 (https://phabricator.wikimedia.org/T224989) [19:53:19] Oh. I just wrote patches to rename in the other direction. :-) [19:54:02] I like "mwcore". It's clear and short and that's a good thing. [19:54:08] James_F: fine by me [19:54:19] James_F: need me to do anything or will you update the patches? [19:54:44] !log Pushing new java8-sonar-scanner:0.4.7 image [19:54:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:54:55] I'll fiddle. [19:58:55] (03PS2) 10Jforrester: Codehealth: JJB config for mediawiki-core [integration/config] - 10https://gerrit.wikimedia.org/r/518121 (https://phabricator.wikimedia.org/T224989) (owner: 10Kosta Harlan) [19:58:57] (03PS2) 10Jforrester: Codehealth: Zuul layout to analyze mediawiki-core [integration/config] - 10https://gerrit.wikimedia.org/r/518122 (owner: 10Kosta Harlan) [20:01:13] (03CR) 10Jforrester: [C: 03+2] Codehealth: JJB config for mediawiki-core [integration/config] - 10https://gerrit.wikimedia.org/r/518121 (https://phabricator.wikimedia.org/T224989) (owner: 10Kosta Harlan) [20:01:22] (03CR) 10Jforrester: [C: 03+2] Codehealth: Zuul layout to analyze mediawiki-core [integration/config] - 10https://gerrit.wikimedia.org/r/518122 (owner: 10Kosta Harlan) [20:01:38] (03CR) 10Jforrester: [C: 03+1] "Let's double-check first." [integration/config] - 10https://gerrit.wikimedia.org/r/518122 (owner: 10Kosta Harlan) [20:03:18] James_F: are you creating a manual build with params? [20:03:24] (03Merged) 10jenkins-bot: Codehealth: JJB config for mediawiki-core [integration/config] - 10https://gerrit.wikimedia.org/r/518121 (https://phabricator.wikimedia.org/T224989) (owner: 10Kosta Harlan) [20:03:26] (03CR) 10jerkins-bot: [V: 04-1] Codehealth: Zuul layout to analyze mediawiki-core [integration/config] - 10https://gerrit.wikimedia.org/r/518122 (owner: 10Kosta Harlan) [20:04:17] !log Pushed new jjb mwcore-codehealth-* jobs and updated mwext-codehealth-* ones [20:04:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:04:29] (03CR) 10Jforrester: [C: 03+2] Codehealth: Zuul layout to analyze mediawiki-core [integration/config] - 10https://gerrit.wikimedia.org/r/518122 (owner: 10Kosta Harlan) [20:06:02] (03Merged) 10jenkins-bot: Codehealth: Zuul layout to analyze mediawiki-core [integration/config] - 10https://gerrit.wikimedia.org/r/518122 (owner: 10Kosta Harlan) [20:06:44] !log Add new mwcore-codehealth-* config to Zuul [20:06:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:07:05] (03CR) 10Jforrester: [C: 03+2] layout: Drop out-of-date comment about php7 vs. hhvm [integration/config] - 10https://gerrit.wikimedia.org/r/518123 (owner: 10Jforrester) [20:07:16] 10Beta-Cluster-Infrastructure, 10Scoring-platform-team: Grant Andy Craze deployment access for beta cluster - https://phabricator.wikimedia.org/T226207 (10Reedy) I've added him as a member to `deployment-prep`... Does he need to be a project admin too? To answer the question... @Halfak can you deploy ORES on... [20:08:23] 10Beta-Cluster-Infrastructure, 10Scoring-platform-team: Grant Andy Craze deployment access for beta cluster - https://phabricator.wikimedia.org/T226207 (10Halfak) Seems like I don't need those rights. So I should be OK. I imagine we gave others those rights in the past so that we could clean up scap direct... [20:09:10] `13:07:30 cp: cannot create directory '/workspace/cache/sonar': No such file or directory` [20:09:17] (03Merged) 10jenkins-bot: layout: Drop out-of-date comment about php7 vs. hhvm [integration/config] - 10https://gerrit.wikimedia.org/r/518123 (owner: 10Jforrester) [20:09:29] James_F: nothing in /workspace/src ? [20:09:41] https://integration.wikimedia.org/ci/job/mwcore-codehealth-patch/1/console [20:09:45] Oh, are we loading core in an odd place? [20:10:10] (03PS1) 10Kosta Harlan: Codehealth: Make mwcore-codehealth-master synchronous [integration/config] - 10https://gerrit.wikimedia.org/r/518137 (https://phabricator.wikimedia.org/T224989) [20:10:17] 10Beta-Cluster-Infrastructure, 10Scoring-platform-team: Grant Andy Craze deployment access for beta cluster - https://phabricator.wikimedia.org/T226207 (10Reedy) 05Open→03Resolved a:03Reedy >>! In T226207#5272262, @Halfak wrote: > I imagine we gave others those rights in the past so that we could clean u... [20:10:37] James_F: not sure, looking [20:10:55] (03CR) 10Jforrester: [C: 03+2] Codehealth: Make mwcore-codehealth-master synchronous [integration/config] - 10https://gerrit.wikimedia.org/r/518137 (https://phabricator.wikimedia.org/T224989) (owner: 10Kosta Harlan) [20:11:39] James_F: could you have a look to see what's at $(pwd)/src in the workspace? [20:13:05] (03Merged) 10jenkins-bot: Codehealth: Make mwcore-codehealth-master synchronous [integration/config] - 10https://gerrit.wikimedia.org/r/518137 (https://phabricator.wikimedia.org/T224989) (owner: 10Kosta Harlan) [20:15:29] James_F: oh right. The mwext-codehealth job uses quibble which checks out the code [20:17:08] Yeah, !!merge : *job_quibble [20:17:08] James_F: so I guess we'd want scm: config ? [20:17:58] James_F: or use quibble and skip all stages [20:18:09] Maybe? I'm out of my depth on the intricacies here, sorry. [20:18:20] James_F: testing something out [20:20:31] 10Beta-Cluster-Infrastructure, 10User-DannyS712: Enwiki beta cluster: Interface Admin rights request - https://phabricator.wikimedia.org/T224809 (10SQL) 05Open→03Resolved a:03SQL Done. [20:22:46] James_F: I'll have a patch up soon that uses `--skip=all` and possibly one other refinement on that. should do what we want [20:25:09] Cool. [20:25:43] Ideally the mwcore-codehealth* and the mwext-codehealth* config was shared as much as possible. [20:25:59] (03PS1) 10Kosta Harlan: Codehealth: mwcore, checkout code with quibble [integration/config] - 10https://gerrit.wikimedia.org/r/518139 (https://phabricator.wikimedia.org/T224989) [20:26:15] (Eventually.) [20:27:29] James_F: yeah, someday :) [20:28:14] Shall we give ^^ a go? [20:29:21] James_F: sure! [20:29:35] (03CR) 10Jforrester: [C: 03+2] Codehealth: mwcore, checkout code with quibble [integration/config] - 10https://gerrit.wikimedia.org/r/518139 (https://phabricator.wikimedia.org/T224989) (owner: 10Kosta Harlan) [20:29:38] James_F: I tested it locally so... [20:29:49] * James_F grins. [20:29:55] Worked On My Machine™. [20:30:00] greater than zero chance it will work [20:30:13] Which is more than doing nothing. [20:31:52] (03Merged) 10jenkins-bot: Codehealth: mwcore, checkout code with quibble [integration/config] - 10https://gerrit.wikimedia.org/r/518139 (https://phabricator.wikimedia.org/T224989) (owner: 10Kosta Harlan) [20:31:57] James_F: I do have to walk out the door in a few minutes but I'll be back in 45 minutes or so. I can trigger a quick check once it's deployed [20:32:11] It's deployed now. [20:32:52] Re-trigged on https://gerrit.wikimedia.org/r/c/mediawiki/core/+/505784 [20:33:40] oh dear [20:33:51] I guess we need to install the dependencies so that MediaWiki can install. annoying [20:34:01] one more patch coming [20:35:15] (03PS1) 10Kosta Harlan: Codehealth: Install dependencies for mwcore-codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/518140 [20:35:17] James_F: ^ should do it [20:35:28] I'll experiment more with quibble and skipping MW installation at a later time [20:35:31] back in a bit [20:38:09] 10Beta-Cluster-Infrastructure, 10Scoring-platform-team: Grant Andy Craze deployment access for beta cluster - https://phabricator.wikimedia.org/T226207 (10Halfak) Makes sense. Thanks for your help. [20:38:16] (03CR) 10Jforrester: [C: 03+2] Codehealth: Install dependencies for mwcore-codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/518140 (owner: 10Kosta Harlan) [20:38:18] (03Merged) 10jenkins-bot: Codehealth: Install dependencies for mwcore-codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/518140 (owner: 10Kosta Harlan) [20:49:59] Well, back to failing on `cp: cannot create directory '/workspace/cache/sonar': No such file or directory`; I guess we need to add a volume mount? [20:50:33] Not sure. Will look in a bit [21:00:38] (03CR) 10Hashar: "I eventually spend most of the day trying several filtering and eventually gave up in despair. I "just" wanted to skip some Quibble jobs " [integration/config] - 10https://gerrit.wikimedia.org/r/516838 (https://phabricator.wikimedia.org/T225628) (owner: 10Jforrester) [21:01:41] (03CR) 10Jforrester: [C: 04-2] "> Patch Set 2:" [integration/config] - 10https://gerrit.wikimedia.org/r/516838 (https://phabricator.wikimedia.org/T225628) (owner: 10Jforrester) [21:16:25] James_F: back. Looking at it [21:23:45] (03PS1) 10Kosta Harlan: Codehealth: Make workspace cache dir if it doesn't exist [integration/config] - 10https://gerrit.wikimedia.org/r/518150 [21:24:00] James_F: ^ [21:24:09] Yeah. [21:24:31] Not a great fix, but I need to better understand what needs to happen for castor to reuse the cache [21:24:37] (03CR) 10Jforrester: [C: 03+2] Codehealth: Make workspace cache dir if it doesn't exist [integration/config] - 10https://gerrit.wikimedia.org/r/518150 (owner: 10Kosta Harlan) [21:25:28] umm [21:25:48] kostajh: why is the builder in jjb not creating /cache? [21:26:01] it shouldn't be /workspace/cache unless something weird is happening [21:26:07] (03Merged) 10jenkins-bot: Codehealth: Make workspace cache dir if it doesn't exist [integration/config] - 10https://gerrit.wikimedia.org/r/518150 (owner: 10Kosta Harlan) [21:26:17] kostajh: also there's a bug that castor is broken for jobs that don't run in the gate queue [21:26:41] legoktm: not sure, I'll have to look a bit later. kids just got home :) [21:27:05] https://phabricator.wikimedia.org/T189077 [21:28:49] legoktm: thanks. I'll ping you on the next patch for castor + sonar [21:29:00] ok :) [21:29:11] I'm a bit behind on reviewing everything rn so I need to take a look soon anyways [21:29:38] !log Pushing new java8-sonar-scanner:0.4.8 image [21:29:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:30:39] (03PS1) 10Jforrester: jjb: [CodeHealth] Use java8-sonar-scanner:0.4.8 [integration/config] - 10https://gerrit.wikimedia.org/r/518152 [21:30:59] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/518152 (owner: 10Jforrester) [21:33:18] (03Merged) 10jenkins-bot: jjb: [CodeHealth] Use java8-sonar-scanner:0.4.8 [integration/config] - 10https://gerrit.wikimedia.org/r/518152 (owner: 10Jforrester) [21:43:17] 10Beta-Cluster-Infrastructure, 10Icinga, 10Wikidata, 10Wikidata Tech Debt Exploration: Create an alarm for wikidata beta dispatching - https://phabricator.wikimedia.org/T203641 (10Addshore) [21:43:42] 10Beta-Cluster-Infrastructure, 10Icinga, 10Wikidata, 10Wikidata Tech Debt Exploration: Create an alarm for wikidata beta dispatching - https://phabricator.wikimedia.org/T203641 (10Addshore) p:05Triage→03Normal [21:54:53] James_F: based on legoktm's comments, I'm going to remove the poor attempts at integrating with castor, since it doesn't seem like it would work anyway [21:56:12] long-term we need to fix castor because it has some other problems but I filed that a year ago and we haven't even come up with a workaround so... [21:57:15] (03PS1) 10Kosta Harlan: Codehealth: Remove castor integration attempts [integration/config] - 10https://gerrit.wikimedia.org/r/518156 [21:57:33] Looks like user nobody doesn't have the permissions? [21:58:18] Also, we can bump back to 0.4.7 rather than do a new release. Much hackier. :-) [21:58:29] James_F: yeah, but even if it did, it doesn't seem like the cache would get (re)used per legoktm [21:59:21] James_F: I'm afraid 0.4.7 has `cp -a /cache/sonar /workspace/cache/sonar`, which is what caused the error for core [21:59:24] not for mwext [21:59:30] so a new release it must be [21:59:33] Oh, sure. [21:59:40] (03CR) 10Jforrester: [C: 03+2] Codehealth: Remove castor integration attempts [integration/config] - 10https://gerrit.wikimedia.org/r/518156 (owner: 10Kosta Harlan) [22:02:00] (03Merged) 10jenkins-bot: Codehealth: Remove castor integration attempts [integration/config] - 10https://gerrit.wikimedia.org/r/518156 (owner: 10Kosta Harlan) [22:03:22] !log Pushing new java8-sonar-scanner:0.4.9 image. [22:03:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:08:43] James_F: looking away from IRC for a bit but please ping me when the jobs are updated and I'll run a recheck [22:11:40] kostajh: Done. [22:19:05] James_F: thanks. as an aside, 8 minutes for the build doesn't seem terrible, and if we can remove the MW install and composer/npm install steps it will probably be more like 4-5 minutes [22:19:55] * James_F nods. [23:15:44] (03CR) 10Brennen Bearnes: [C: 03+1] "Probably these could be separate patches, but I think they look fine. Tested the `helm init` change." [releng/local-charts] - 10https://gerrit.wikimedia.org/r/517780 (owner: 10Jeena Huneidi) [23:55:30] (03PS2) 10Jforrester: jjb: Rename mediawiki-{core-code|phpunit}-coverage-(patch-)?docker for consistency, pt. 1 [integration/config] - 10https://gerrit.wikimedia.org/r/518124 [23:55:37] (03CR) 10Jforrester: [C: 03+2] jjb: Rename mediawiki-{core-code|phpunit}-coverage-(patch-)?docker for consistency, pt. 1 [integration/config] - 10https://gerrit.wikimedia.org/r/518124 (owner: 10Jforrester) [23:55:43] (03PS2) 10Jforrester: layout: Switch from mediawiki-phpunit-coverage-patch-docker to mwcore-phpunit-coverage-patch [integration/config] - 10https://gerrit.wikimedia.org/r/518125 [23:57:51] (03Merged) 10jenkins-bot: jjb: Rename mediawiki-{core-code|phpunit}-coverage-(patch-)?docker for consistency, pt. 1 [integration/config] - 10https://gerrit.wikimedia.org/r/518124 (owner: 10Jforrester) [23:58:44] (03CR) 10Jforrester: [C: 03+2] layout: Switch from mediawiki-phpunit-coverage-patch-docker to mwcore-phpunit-coverage-patch [integration/config] - 10https://gerrit.wikimedia.org/r/518125 (owner: 10Jforrester) [23:58:52] (03PS2) 10Jforrester: jjb: Rename mediawiki-{core-code|phpunit}-coverage-(patch-)?docker for consistency, pt. 2 [integration/config] - 10https://gerrit.wikimedia.org/r/518126