[06:21:05] Project beta-update-databases-eqiad build #37623: 04FAILURE in 1 min 5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/37623/ [07:21:10] Yippee, build fixed! [07:21:10] Project beta-update-databases-eqiad build #37624: 09FIXED in 1 min 10 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/37624/ [08:05:32] 10Deployments, 10Release-Engineering-Team, 10VisualEditor, 10Wikimedia-Logstash, and 4 others: Logstash discards messages from MediaWiki if they contain uncommon keys in the $context array - https://phabricator.wikimedia.org/T234564 (10fgiunchedi) Since the specific issue is resolved I'm for resolving this... [08:11:23] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910): Some All-Users.git references have been force updated - https://phabricator.wikimedia.org/T236344 (10hashar) [08:14:18] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10Wikimedia-Incident: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10hashar) I also found four `All-Users.git` reference that got force updated :-\ T236344 [08:16:03] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910): Some All-Users.git references have been force updated - https://phabricator.wikimedia.org/T236344 (10hashar) [08:18:32] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910): Some All-Users.git references are outdated after gerrit1001 migration - https://phabricator.wikimedia.org/T236344 (10hashar) [08:23:01] (03PS15) 10Awight: Commands can expand recursively [integration/quibble] - 10https://gerrit.wikimedia.org/r/519776 [08:33:38] 10Gerrit: SSH login to gerrit no longer works for me (simetrical/Aryeh Gregor) - https://phabricator.wikimedia.org/T236191 (10hashar) I do not see any connection attempt to Gerrit 29418 with login `simetrical`. Could it be that your local ssh config route the request through a bastion or some similar oddity? T... [08:43:03] 10Gerrit: SSH login to gerrit no longer works for me (simetrical/Aryeh Gregor) - https://phabricator.wikimedia.org/T236191 (10hashar) p:05Triage→03Normal [08:46:54] (03PS1) 10Awight: Enable Book Referencing for browser tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/545755 (https://phabricator.wikimedia.org/T236255) [09:07:39] (03CR) 10Hashar: [C: 04-1] "Technically, yes that would work but that cause a lot of problems in the long run." [integration/quibble] - 10https://gerrit.wikimedia.org/r/545755 (https://phabricator.wikimedia.org/T236255) (owner: 10Awight) [09:24:21] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10Move-Files-To-Commons, 10TCB-Team, and 5 others: Test WikiTextEditor class with browser tests. - https://phabricator.wikimedia.org/T190829 (10hashar) Great to see this task has now been resolved. Well done... [09:36:52] (03CR) 10Hashar: [C: 04-1] Upgrade PostBuildScript and Update jjb to 2.0.3 (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/544907 (https://phabricator.wikimedia.org/T188398) (owner: 10Hashar) [09:37:00] (03PS7) 10Hashar: Upgrade PostBuildScript and Update jjb to 2.0.3 [integration/config] - 10https://gerrit.wikimedia.org/r/544907 (https://phabricator.wikimedia.org/T188398) [09:53:17] (03CR) 10Awight: "> Technically, yes that would work but that cause a lot of problems" [integration/quibble] - 10https://gerrit.wikimedia.org/r/545755 (https://phabricator.wikimedia.org/T236255) (owner: 10Awight) [09:53:20] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10MediaWiki-Vagrant, 10User-zeljkofilipin: `npm run selenium` fails on MW Vagrant (SyntaxError: Unexpected identifier; due to Node 6) - https://phabricator.wikimedia.org/T236282 (1... [09:53:22] (03Abandoned) 10Awight: Enable Book Referencing for browser tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/545755 (https://phabricator.wikimedia.org/T236255) (owner: 10Awight) [09:53:45] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10MediaWiki-Vagrant, 10User-zeljkofilipin: `npm run selenium` fails on MW Vagrant (SyntaxError: Unexpected identifier; due to Node 6) - https://phabricator.wikimedia.org/T236282 (1... [09:55:32] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Testing, 10Browser-Tests, 10Patch-For-Review, 10User-zeljkofilipin: Run browser tests in parallel - https://phabricator.wikimedia.org/T226869 (10zeljkofilipin) [09:59:13] (03CR) 10Hashar: [C: 03+2] Upgrade PostBuildScript and Update jjb to 2.0.3 [integration/config] - 10https://gerrit.wikimedia.org/r/544907 (https://phabricator.wikimedia.org/T188398) (owner: 10Hashar) [10:00:49] !log Reconfiguring most Jenkins jobs due to PostBuildScript plugin change https://gerrit.wikimedia.org/r/#/c/integration/config/+/544907/ [10:00:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:02:16] (03Merged) 10jenkins-bot: Upgrade PostBuildScript and Update jjb to 2.0.3 [integration/config] - 10https://gerrit.wikimedia.org/r/544907 (https://phabricator.wikimedia.org/T188398) (owner: 10Hashar) [10:04:43] !log Updated integration/jenkins-job-builder.git to 2.0.3 # T188398 [10:04:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:06:21] hashar: :O, we should run a reindex on accounts [10:08:04] https://grafana.wikimedia.org/d/Bw2mQ3iWz/gerrit-javamelody?panelId=12&fullscreen&orgId=1 [10:11:58] hashar: hey, following up regarding gerrit/zuul prometheus exporter mappings. anything to add or are we good go go? [10:12:25] shdubsh: oh that, I haven't looked at it yet :-\\\ [10:13:23] paladox: aren't account reindexed anyway? And for a few user references I have seen, we actually have outdated ones :-\\\ [10:13:51] paladox: some are back to an arbitrary point from October 1st . Same issue we had to fight with after the migration [10:13:54] Yup, but if you push a commit to the repo [10:13:55] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201907), 10Patch-For-Review: Investigate gerrit session expiration - https://phabricator.wikimedia.org/T222472 (10LarsWirzenius) I too had to log in today. Debian buster, FF 60. No reboot. [10:13:59] It dosent auto reindex [10:14:15] paladox: also that grafan gerrit-javamelody I am not sure it got updated. Might still point to cobalt? [10:14:24] It’s updated :) [10:14:28] cool [10:14:31] It’s using gerrit.wikimedia.org [10:14:38] Not any specific hostname S [10:14:42] *hostnames [10:16:23] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201907), 10Patch-For-Review: Investigate gerrit session expiration - https://phabricator.wikimedia.org/T222472 (10Paladox) Did you all click the remember me box? I think we are going to experiment with the webses... [10:17:46] hashar: fair enough, please when you have a chance, thanks! [10:21:27] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201907), 10Patch-For-Review: Investigate gerrit session expiration - https://phabricator.wikimedia.org/T222472 (10Ladsgroup) >>! In T222472#5602195, @Paladox wrote: > Did you all click the remember me box? Of cou... [10:22:35] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201907), 10Patch-For-Review: Investigate gerrit session expiration - https://phabricator.wikimedia.org/T222472 (10Mainframe98) >>! In T222472#5602206, @Ladsgroup wrote: >>>! In T222472#5602195, @Paladox wrote: >>... [10:22:40] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Jenkins, 10Technical-Debt: Align jenkins-job-builder with upstream - https://phabricator.wikimedia.org/T236036 (10hashar) [10:23:57] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201907), 10Patch-For-Review: Investigate gerrit session expiration - https://phabricator.wikimedia.org/T222472 (10LarsWirzenius) I have made a point of clicking the remember-me box every login while collecting dat... [10:24:20] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Jenkins, 10Technical-Debt: Align jenkins-job-builder with upstream - https://phabricator.wikimedia.org/T236036 (10hashar) `integration/config.git` `jenkins-jobs-requirements.txt` now uses 2.0.3 I have updated `integrati... [10:25:02] shdubsh: is the aim to eventually replace statsd / graphite entirely in favor of just Prometheus ? ;) [10:25:18] hashar: yes [11:45:39] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201910), 10Release, 10Train Deployments: 1.35.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T233851 (10LarsWirzenius) [12:00:35] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201907), 10Patch-For-Review: Investigate gerrit session expiration - https://phabricator.wikimedia.org/T222472 (10Paladox) @Ladsgroup sometimes users may forget to press it. It’s easily forgettable. [12:00:41] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201910), 10Release, 10Train Deployments: 1.35.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T233851 (10LarsWirzenius) [12:30:07] (03PS1) 10Hashar: jjb: git plugin does not default to shallow [integration/config] - 10https://gerrit.wikimedia.org/r/545828 [12:33:57] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10User-zeljkofilipin: ERROR webdriver: Request failed due to Error: session not created: Chrome version must be between 70 and 73 - https://phabricator.wikimedia.org/T234610 (10zeljk... [12:34:57] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10User-zeljkofilipin: ERROR webdriver: Request failed due to Error: session not created: Chrome version must be between 70 and 73 - https://phabricator.wikimedia.org/T234610 (10zeljk... [12:42:43] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10User-zeljkofilipin: ERROR webdriver: Request failed due to Error: session not created: Chrome version must be between 70 and 73 - https://phabricator.wikimedia.org/T234610 (10hasha... [12:43:01] zeljkof: chromedriver=v73 that is fun ;) [12:43:18] that means anyone running the selenium tests locally with a more recent version of chromium ends up broken [12:43:36] (also gotta switch the CI containers to buster to get the latest chromium) [12:43:46] at least for the CI containers using chromium [12:43:52] which arguably is a different task [12:44:04] hashar: no, it means anybody using chromedriver from package.json and a recent version of chrome will have all tests failins [12:44:06] failing [12:44:21] that is what I have just said! [12:44:24] anyway, I've updated the docs, it would work now [12:44:41] hashar: ah, sorry, I've read chromium as chromedriver [12:44:43] we should drop that binary providing package [12:45:00] OR have package.json to also provide chromium at some fixed version [12:45:43] -then I am really not a fan of those node modules providing binaries - [12:46:43] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201910), 10Release, 10Train Deployments: 1.35.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T233851 (10LarsWirzenius) [12:47:23] (03CR) 10Hashar: [C: 03+2] "INFO:jenkins_jobs.builder:Number of jobs generated: 1" [integration/config] - 10https://gerrit.wikimedia.org/r/545828 (owner: 10Hashar) [12:47:27] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201910), 10Release, 10Train Deployments: 1.35.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T233851 (10LarsWirzenius) [12:49:50] (03Merged) 10jenkins-bot: jjb: git plugin does not default to shallow [integration/config] - 10https://gerrit.wikimedia.org/r/545828 (owner: 10Hashar) [12:50:59] hashar: Yeah, I worry about running CI on buster when prod is on stretch. [12:51:11] yeah [12:51:16] we should stick mediawiki to stretch [12:51:19] but [12:51:35] the selenium test, we might be fine running them on buster [12:51:38] THOUGH [12:51:45] We should ideally pull Chrome from live rather than a package. [12:51:52] that means the selenium tests wouold no more highlight potential issues with the stretch php7.2 packages :-\\\\ [12:51:54] bah [12:52:00] so yeah [12:52:08] this way devs can pin chromium as well [12:52:34] Rather than wait for Debian, it'd be nice to find out that all of the site is broken in the new version of Chrome automatically rather than from manual testing. [12:52:55] months and months ago, some service project was willing to use puppeteer with the raw binary from whatever on the internet. Eventually we had it to skip downloading the binary and use the Debian package instead [12:52:56] Running a dozen versions of Chrome and of Firefox and so on will be a major undertaking, however. [12:53:07] Bandwidth/load? [12:53:09] but in the case of mediawiki / selenium tests that is different. We don't really care about using the Debian packaged version [12:53:17] and should just use whatever is recent [12:53:21] Yeah, we care more about the experience for end-users. [12:53:27] yeah [12:53:53] and Debian apparently only maintains latest chromium for their current distro : buster [12:54:08] so on the task I say to drop chromedriver=v73 pinning [12:54:15] but most probably we should also pin Chrome :] [12:54:24] the devil is figuring out how to ship it at install phase [12:54:28] and obviously have that cached [12:54:32] We really want to pull Chrome from sid, not buster. ;-) [12:55:08] maybe [13:14:07] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10MediaWiki-Vagrant, and 2 others: `npm run selenium` fails on MW Vagrant (SyntaxError: Unexpected identifier; due to Node 6) - https://phabricator.wikimedia.org/T236282 (10zeljkofil... [13:15:17] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10MediaWiki-Vagrant, and 2 others: `npm run selenium` fails on MW Vagrant (SyntaxError: Unexpected identifier; due to Node 6) - https://phabricator.wikimedia.org/T236282 (10zeljkofil... [13:15:53] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10MediaWiki-Vagrant, 10Patch-For-Review: `npm run selenium` fails on MW Vagrant (SyntaxError: Unexpected identifier; due to Node 6) - https://phabricator.wikimedia.org/T236282 (10z... [13:21:12] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing: ERROR webdriver: Request failed due to Error: session not created: Chrome version must be between 70 and 73 - https://phabricator.wikimedia.org/T234610 (10zeljkofilipin) [13:21:26] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing: ERROR webdriver: Request failed due to Error: session not created: Chrome version must be between 70 and 73 - https://phabricator.wikimedia.org/T234610 (10zeljkofilipin) a:05zeljkofi... [13:23:30] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201910), 10Release, 10Train Deployments: 1.35.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T233851 (10LarsWirzenius) Deployed to all wikis. Logstash didn't explode immediately, but keeping an eye on it. [13:25:10] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10MediaWiki-Vagrant, 10Patch-For-Review: `npm run selenium` fails on MW Vagrant (SyntaxError: Unexpected identifier; due to Node 6) - https://phabricator.wikimedia.org/T236282 (10h... [13:27:35] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10MediaWiki-Vagrant, 10Patch-For-Review: `npm run selenium` fails on MW Vagrant (SyntaxError: Unexpected identifier; due to Node 6) - https://phabricator.wikimedia.org/T236282 (10h... [13:32:10] (03PS1) 10Hashar: jjb: lintr-docker-non-voting missed zuul parameters [integration/config] - 10https://gerrit.wikimedia.org/r/545840 [13:37:39] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10Jenkins, 10Technical-Debt: Align jenkins-job-builder with upstream - https://phabricator.wikimedia.org/T236036 (10Jdforrester-WMF) Resolved? [13:42:07] (03CR) 10Hashar: [C: 03+2] "Diff is as expected. And https://integration.wikimedia.org/ci/job/lintr-docker-non-voting/ now shows the parameters \o/" [integration/config] - 10https://gerrit.wikimedia.org/r/545840 (owner: 10Hashar) [13:43:24] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10Jenkins, 10Technical-Debt: Align jenkins-job-builder with upstream - https://phabricator.wikimedia.org/T236036 (10hashar) 2.0.10 introduces a depth parameter for git shallow clone and it always explicitly set wi... [13:44:34] (03Merged) 10jenkins-bot: jjb: lintr-docker-non-voting missed zuul parameters [integration/config] - 10https://gerrit.wikimedia.org/r/545840 (owner: 10Hashar) [13:51:53] (03PS1) 10Pwirth: Remove Archived extension BlueSpiceUserPreferences. T197898 [integration/config] - 10https://gerrit.wikimedia.org/r/545853 [13:52:57] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10Jenkins, 10Technical-Debt: Align jenkins-job-builder with upstream - https://phabricator.wikimedia.org/T236036 (10hashar) 2.3.0 has AnsiColor to always set the colormap to `xterm`. 4cce3d6a78fbe4e1de39299f77acf... [13:55:42] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10Jenkins, 10Technical-Debt: Align jenkins-job-builder with upstream - https://phabricator.wikimedia.org/T236036 (10hashar) 2.4.0 adds some extra XML parameters for the copyartifact property. That only affects be... [14:03:44] (03CR) 10Jforrester: [C: 03+2] Remove Archived extension BlueSpiceUserPreferences. T197898 [integration/config] - 10https://gerrit.wikimedia.org/r/545853 (owner: 10Pwirth) [14:05:14] (03Merged) 10jenkins-bot: Remove Archived extension BlueSpiceUserPreferences. T197898 [integration/config] - 10https://gerrit.wikimedia.org/r/545853 (owner: 10Pwirth) [14:05:39] !log Zuul: Drop BlueSpiceUserPreferences from CI, archived. [14:05:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:10:15] (03PS1) 10Pwirth: Remove archived extension BlueSpiceBlog. T231853 [integration/config] - 10https://gerrit.wikimedia.org/r/545858 [14:11:14] (03CR) 10Jforrester: [C: 03+2] Remove archived extension BlueSpiceBlog. T231853 [integration/config] - 10https://gerrit.wikimedia.org/r/545858 (owner: 10Pwirth) [14:12:55] (03Merged) 10jenkins-bot: Remove archived extension BlueSpiceBlog. T231853 [integration/config] - 10https://gerrit.wikimedia.org/r/545858 (owner: 10Pwirth) [14:15:55] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO: Repositories dashboard - https://phabricator.wikimedia.org/T112259 (10mmodell) Sort of related: {T235874} [14:16:59] !log Zuul: Drop BlueSpiceBlog from CI, archived. T231853 [14:17:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:17:04] T231853: Archive "BlueSpiceBlog" extension - https://phabricator.wikimedia.org/T231853 [14:19:51] 10Deployments, 10Release-Engineering-Team, 10serviceops, 10Performance-Team (Radar): Cache of wmf-config/InitialiseSettings often 1 step behind - https://phabricator.wikimedia.org/T236104 (10Krinkle) [14:21:00] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10Jenkins, 10Technical-Debt: Align jenkins-job-builder with upstream - https://phabricator.wikimedia.org/T236036 (10hashar) 2.9.0 makes the `--output` related warning to only show once ( e07ad13ad0173a4f5e6ab899cf... [14:27:36] (03PS1) 10Hashar: Upgrade jjb to 2.9.1 [integration/config] - 10https://gerrit.wikimedia.org/r/545861 (https://phabricator.wikimedia.org/T236036) [14:29:34] (03CR) 10jerkins-bot: [V: 04-1] Upgrade jjb to 2.9.1 [integration/config] - 10https://gerrit.wikimedia.org/r/545861 (https://phabricator.wikimedia.org/T236036) (owner: 10Hashar) [14:31:19] NOOOO [14:31:54] VersionConflict: (PyYAML 5.1.2 (/src/.tox/zuul_tests/lib/python2.7/site-packages), Requirement.parse('PyYAML<4,>=3.10.0')) [14:38:07] (03CR) 1020after4: "Not sure who else could review this code. I've tested it in local kibana development environment and the change works." [releng/phatality] - 10https://gerrit.wikimedia.org/r/545593 (https://phabricator.wikimedia.org/T235945) (owner: 1020after4) [14:45:58] (03PS2) 10Hashar: Upgrade jjb to 2.9.1 [integration/config] - 10https://gerrit.wikimedia.org/r/545861 (https://phabricator.wikimedia.org/T236036) [14:46:00] (03PS1) 10Hashar: Contrain PyYAML<4 [integration/config] - 10https://gerrit.wikimedia.org/r/545868 (https://phabricator.wikimedia.org/T236036) [14:46:55] (03PS2) 1020after4: Build the url field from server + url [releng/phatality] - 10https://gerrit.wikimedia.org/r/545593 (https://phabricator.wikimedia.org/T235945) [14:50:08] (03PS2) 10Hashar: Constrain PyYAML<4 [integration/config] - 10https://gerrit.wikimedia.org/r/545868 (https://phabricator.wikimedia.org/T236036) [14:50:32] (03CR) 10Hashar: [C: 03+2] Constrain PyYAML<4 [integration/config] - 10https://gerrit.wikimedia.org/r/545868 (https://phabricator.wikimedia.org/T236036) (owner: 10Hashar) [14:51:58] (03CR) 10Hashar: "The jjb diff is huge :/ Maybe I can instead have a change per jjb version to show it is noop / reduce the resulting diff. That is what I " [integration/config] - 10https://gerrit.wikimedia.org/r/545861 (https://phabricator.wikimedia.org/T236036) (owner: 10Hashar) [14:52:00] (03CR) 10jerkins-bot: [V: 04-1] Constrain PyYAML<4 [integration/config] - 10https://gerrit.wikimedia.org/r/545868 (https://phabricator.wikimedia.org/T236036) (owner: 10Hashar) [14:57:23] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10Browser-Tests, and 2 others: Selenium tests failing for PluggableAuth plugins - https://phabricator.wikimedia.org/T236030 (10Krinkle) [15:08:58] 10Deployments, 10Release-Engineering-Team, 10serviceops, 10Performance-Team (Radar): Cache of wmf-config/InitialiseSettings often 1 step behind - https://phabricator.wikimedia.org/T236104 (10Krinkle) I don't know off-hand whether the mtime comes from the deployment server (set when the last git command sav... [15:19:14] (03PS3) 10Hashar: Contrain PyYAML<4 [integration/config] - 10https://gerrit.wikimedia.org/r/545868 (https://phabricator.wikimedia.org/T236036) [15:19:46] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10local-charts: `make install` fails with `Error: caskroom/cask was moved. Tap homebrew/cask-cask instead.` - https://phabricator.wikimedia.org/T236393 (10zeljkofilipin) [15:20:02] (03CR) 10Hashar: "PyYAML 3.x does not emit a warning about yaml.load() usage. That in turn caused utils/zuul-layout-validate.sh to fail when grep can not fi" [integration/config] - 10https://gerrit.wikimedia.org/r/545868 (https://phabricator.wikimedia.org/T236036) (owner: 10Hashar) [15:20:14] (03PS3) 10Hashar: Upgrade jjb to 2.9.1 [integration/config] - 10https://gerrit.wikimedia.org/r/545861 (https://phabricator.wikimedia.org/T236036) [15:20:42] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10Jenkins, 10Patch-For-Review, 10Technical-Debt: Align jenkins-job-builder with upstream - https://phabricator.wikimedia.org/T236036 (10hashar) Almost ready to reach latest version available :-] [15:25:59] howdy releng, is there a way to check out what version of PHP we are running in production across all wikimedia projects? [15:27:27] jgleeson: Special:Version [15:27:43] PHP 7.2.22-1+0~20190902.26+debian9~1.gbpd64eb7+wmf1 (fpm-fcgi) [15:27:52] thanks Reedy! [15:48:21] Government to table a GE motion on Monday! [15:49:27] (03PS1) 10Zfilipin: Fix `make install` on macOS [releng/local-charts] - 10https://gerrit.wikimedia.org/r/545881 (https://phabricator.wikimedia.org/T236393) [15:54:09] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10local-charts, 10Patch-For-Review, 10User-zeljkofilipin: `make install` fails with `Error: caskroom/cask was moved. Tap homebrew/cask-cask instead.` - https://phabricator.wikimedia.org/T236393 (10zeljkofilipin) [15:54:14] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10local-charts, 10Patch-For-Review, 10User-zeljkofilipin: `make install` fails with `Error: caskroom/cask was moved. Tap homebrew/cask-cask instead.` - https://phabricator.wikimedia.org/T236393 (10zeljkofilipin)... [15:55:45] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10local-charts, 10Patch-For-Review, 10User-zeljkofilipin: `make install` fails with `Error: caskroom/cask was moved. Tap homebrew/cask-cask instead.` - https://phabricator.wikimedia.org/T236393 (10zeljkofilipin)... [15:57:29] 10Phabricator, 10Release-Engineering-Team (Development services), 10Operations, 10hardware-requests, 10serviceops: The phabricator server, WMF7426, was given to us temporarily, we would like to make it permanent - https://phabricator.wikimedia.org/T232887 (10mark) I'm a bit confused; as far as I know the... [16:02:50] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10local-charts, 10User-zeljkofilipin: `make install` fails with `“minikube-darwin-amd64” cannot be opened because the developer cannot be verified.` - https://phabricator.wikimedia.org/T236402 (10zeljkofilipin) [16:05:38] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10local-charts, 10User-zeljkofilipin: `make install` fails with `“minikube-darwin-amd64” cannot be opened because the developer cannot be verified.` - https://phabricator.wikimedia.org/T236402 (10zeljkofilipin) [16:07:54] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10local-charts, 10User-zeljkofilipin: `make install` fails with `“minikube-darwin-amd64” cannot be opened because the developer cannot be verified.` - https://phabricator.wikimedia.org/T236402 (10zeljkofilipin) [16:35:58] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10local-charts, 10User-zeljkofilipin: `make install` fails with `“minikube-darwin-amd64” cannot be opened because the developer cannot be verified.` - https://phabricator.wikimedia.org/T236402 (10zeljkofilipin) [16:39:15] thcipriani https://gerrit.wikimedia.org/r/monitoring?part=graph&graph=usedMemory looks better! [16:39:43] unused memory is wasted memory [16:40:05] except when you need the extra one :) [16:40:34] Reedy it's not reached 32 yet :P [16:40:45] where as if you look to your left, it almost reached 20 [16:41:57] https://gerrit.wikimedia.org/r/monitoring?part=graph&graph=activeThreads&period=semaine [16:42:01] has been low since monday! [16:42:06] which is awesome! [16:43:28] well monday evening but still monday :P [16:55:36] (03CR) 10Krinkle: Build the url field from server + url (031 comment) [releng/phatality] - 10https://gerrit.wikimedia.org/r/545593 (https://phabricator.wikimedia.org/T235945) (owner: 1020after4) [17:10:39] 10Release-Engineering-Team (Deployment services), 10Security-Team, 10Wikimedia-Extension-setup, 10Wikimedia-Site-requests, 10Wikimedia-extension-review-queue: Deploy WebAuthn to Wikimedia Wikis - https://phabricator.wikimedia.org/T227242 (10Aklapper) If this about https://www.mediawiki.org/wiki/Extension... [17:27:53] (03PS4) 10Hashar: Constrain PyYAML<4 [integration/config] - 10https://gerrit.wikimedia.org/r/545868 (https://phabricator.wikimedia.org/T236036) [17:28:13] (03CR) 10Hashar: [C: 03+2] Constrain PyYAML<4 [integration/config] - 10https://gerrit.wikimedia.org/r/545868 (https://phabricator.wikimedia.org/T236036) (owner: 10Hashar) [17:29:53] (03Merged) 10jenkins-bot: Constrain PyYAML<4 [integration/config] - 10https://gerrit.wikimedia.org/r/545868 (https://phabricator.wikimedia.org/T236036) (owner: 10Hashar) [17:47:53] PROBLEM - Parsoid on deployment-parsoid09 is CRITICAL: connect to address 172.16.5.63 and port 8000: Connection refused [17:47:53] PROBLEM - Parsoid on deployment-mediawiki-parsoid10 is CRITICAL: connect to address 172.16.0.141 and port 8000: Connection refused [17:54:47] 10Continuous-Integration-Infrastructure, 10Quibble: Install wikidiff2 into mediawiki-quibble-api-testing-vendor-mysql-php72-docker - https://phabricator.wikimedia.org/T236333 (10Pchelolo) [19:08:40] 10Continuous-Integration-Config, 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (201910), 10Scap, and 5 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10ArielGlenn) Hey @Jdforrester-WMF I'm looking around at... [19:21:11] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910): Some All-Users.git references are outdated after gerrit1001 migration - https://phabricator.wikimedia.org/T236344 (10thcipriani) Rerunning on all `refs/*` showed a few inconsistencies. For `All-Users.git` t... [19:21:26] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910): Some All-Users.git references are outdated after gerrit1001 migration - https://phabricator.wikimedia.org/T236344 (10thcipriani) p:05Triage→03High a:03thcipriani [19:44:31] Project beta-scap-eqiad build #272618: 04FAILURE in 4.6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/272618/ [19:56:32] Yippee, build fixed! [19:56:33] Project beta-scap-eqiad build #272619: 09FIXED in 2 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/272619/ [20:23:41] (03PS3) 1020after4: Build the url field from server + url [releng/phatality] - 10https://gerrit.wikimedia.org/r/545593 (https://phabricator.wikimedia.org/T235945) [20:25:36] (03CR) 1020after4: Build the url field from server + url (031 comment) [releng/phatality] - 10https://gerrit.wikimedia.org/r/545593 (https://phabricator.wikimedia.org/T235945) (owner: 1020after4) [21:53:43] Hey team! Wondering if there's an idea of where https://phabricator.wikimedia.org/T236333 is on your roadmap? [21:59:53] 10Gerrit, 10Release-Engineering-Team, 10Operations, 10Wikimedia Design Style Guide: Automatic pickup of Gerrit clone master doesn't happen (due to git-lfs not installed on production misc) - https://phabricator.wikimedia.org/T235677 (10Volker_E) We've reverted Git LFS for now in https://github.com/wikimedi... [22:03:47] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO, 10Quibble: Install wikidiff2 into mediawiki-quibble-api-testing-vendor-mysql-php72-docker - https://phabricator.wikimedia.org/T236333 (10brennen) [22:13:50] PROBLEM - Check systemd state on contint1001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [22:14:24] Project beta-code-update-eqiad build #269225: 04FAILURE in 1 min 24 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/269225/ [22:15:04] RECOVERY - Check systemd state on contint1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [22:24:24] Yippee, build fixed! [22:24:24] Project beta-code-update-eqiad build #269226: 09FIXED in 1 min 23 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/269226/ [22:26:48] (03PS1) 10Awight: Drop HHVM support [integration/quibble] - 10https://gerrit.wikimedia.org/r/545968 [22:31:41] (03CR) 1020after4: [C: 03+1] "lgtm" [releng/local-charts] - 10https://gerrit.wikimedia.org/r/538365 (owner: 10Jeena Huneidi) [22:37:03] (03PS2) 10Awight: Drop HHVM support [integration/quibble] - 10https://gerrit.wikimedia.org/r/545968 [23:00:17] (03PS1) 10Awight: Chronometer emits folding markers [integration/quibble] - 10https://gerrit.wikimedia.org/r/545972 [23:01:30] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910): Some All-Users.git references are outdated after gerrit1001 migration - https://phabricator.wikimedia.org/T236344 (10thcipriani) 05Open→03Resolved I've updated the All-Users.git refs that were out of da... [23:01:49] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10Wikimedia-Incident: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10thcipriani) [23:02:19] 10Gerrit, 10Release-Engineering-Team, 10Operations, 10Wikimedia Design Style Guide: Automatic pickup of Gerrit clone master doesn't happen (due to git-lfs not installed on production misc) - https://phabricator.wikimedia.org/T235677 (10mmodell) So I think this should really be deployed with scap. That woul... [23:07:00] (03PS16) 10Awight: Commands can expand recursively [integration/quibble] - 10https://gerrit.wikimedia.org/r/519776 [23:08:02] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10Wikimedia-Incident: git review notes split brain - https://phabricator.wikimedia.org/T236443 (10thcipriani) [23:08:26] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10Wikimedia-Incident: git review notes split brain - https://phabricator.wikimedia.org/T236443 (10thcipriani) p:05Triage→03Normal [23:10:57] nikkinikk: re: T236333, don't know offhand. added to our TODO board, did a bit of digging and mostly just realized i need to learn where that stuff is defined. [23:10:58] T236333: Install wikidiff2 into mediawiki-quibble-api-testing-vendor-mysql-php72-docker - https://phabricator.wikimedia.org/T236333 [23:20:01] brennen: you can work backwards from the jjb job definition in integration/config to find the container [23:20:06] er container image rather [23:20:16] Project beta-update-databases-eqiad build #37640: 04FAILURE in 15 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/37640/ [23:28:53] thcipriani: ah, right. so in this case i think it's just releng/quibble-stretch-php72 [23:29:13] sounds right [23:29:16] :) [23:39:10] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO, 10Quibble: Install wikidiff2 into mediawiki-quibble-api-testing-vendor-mysql-php72-docker - https://phabricator.wikimedia.org/T236333 (10brennen) a:03brennen Unless I'm mistaken, this should just take adding a package in the list her... [23:41:48] (03PS1) 10Brennen Bearnes: quibble-stretch-php72: add wikidiff2 [integration/config] - 10https://gerrit.wikimedia.org/r/545978 (https://phabricator.wikimedia.org/T236333) [23:42:46] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO, 10Quibble, 10Patch-For-Review: Install wikidiff2 into mediawiki-quibble-api-testing-vendor-mysql-php72-docker - https://phabricator.wikimedia.org/T236333 (10brennen) It's hitting end-of-workday here and I'm a bit fried, will pick up... [23:44:21] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing: ERROR webdriver: Request failed due to Error: session not created: Chrome version must be between 70 and 73 - https://phabricator.wikimedia.org/T234610 (10Krinkle) I added to `package...