[08:16:49] (03PS2) 10Hashar: Update composer from 1.10.5 to 1.10.17 [integration/config] - 10https://gerrit.wikimedia.org/r/640574 (owner: 10Reedy) [08:19:11] (03PS3) 10Hashar: Update composer from 1.10.5 to 1.10.17 [integration/config] - 10https://gerrit.wikimedia.org/r/640574 (owner: 10Reedy) [08:20:05] (03CR) 10Hashar: [C: 03+2] "I have added the missing changelog entry for composer-php70 and bumped the changelog of all child images with:" [integration/config] - 10https://gerrit.wikimedia.org/r/640574 (owner: 10Reedy) [08:21:09] (03Merged) 10jenkins-bot: Update composer from 1.10.5 to 1.10.17 [integration/config] - 10https://gerrit.wikimedia.org/r/640574 (owner: 10Reedy) [08:29:06] !log Building containers for composer update from 1.10.5 .. 1.10.17 : https://gerrit.wikimedia.org/r/c/integration/config/+/640574 [08:29:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:35:11] 10Continuous-Integration-Config, 10Test-Coverage: Allow `check coverage` on more repos - https://phabricator.wikimedia.org/T267781 (10Reedy) [09:59:38] 10Continuous-Integration-Config, 10Test-Coverage: Allow `check coverage` on more repos - https://phabricator.wikimedia.org/T267781 (10Reedy) [10:07:18] hashar: ^ Any idea how much of a PITA that would be to implement? [10:07:55] zthe coverage thing? no idea [10:08:13] iirc the idea is we run a coverage command for HEAD^ and HEAD [10:08:28] then compare the resulting coverage reports (probably in clover format which is XML [10:09:51] the versions of coverage for mw core/exts and "everything else" seem to be quite different [10:10:43] so I am wrong :`\ [10:13:56] (03CR) 10Hashar: [C: 03+2] "That is good enough for now, can be enhanced later on" [integration/config] - 10https://gerrit.wikimedia.org/r/623599 (owner: 10Hashar) [10:14:38] https://github.com/wikimedia/mediawiki-tools-phpunit-patch-coverage I guess is what you're thinking of.. [10:14:41] * Reedy looks furhter [10:15:14] (03Merged) 10jenkins-bot: Script to find image updates in jjb [integration/config] - 10https://gerrit.wikimedia.org/r/623599 (owner: 10Hashar) [10:34:45] 10Continuous-Integration-Config, 10Test-Coverage: Run proactive coverage checking on more repos - https://phabricator.wikimedia.org/T267781 (10Reedy) [10:35:54] 10Continuous-Integration-Config, 10Test-Coverage: Run proactive coverage checking on more repos - https://phabricator.wikimedia.org/T267781 (10Reedy) [10:51:37] is it a known issue that Beta is down again? [11:05:10] 10Gerrit: Reviewer-bot does not add reviewers when leaving initial WIP mode - https://phabricator.wikimedia.org/T267787 (10WMDE-Fisch) [11:14:30] !log Unarchiving https://github.com/wikimedia/mediawiki-services-mathjax and renaming it to https://github.com/wikimedia/mediawiki-services-mathjax-attic # T249908 [11:14:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:14:36] T249908: Move math related repositories from GitHub(?) to gerrit - https://phabricator.wikimedia.org/T249908 [11:14:53] Lucas_WMDE: probably not :-\ [11:18:46] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Math, and 3 others: Move math related repositories from GitHub(?) to gerrit - https://phabricator.wikimedia.org/T249908 (10hashar) I have renamed the legacy repo... [11:19:10] !log Rename https://github.com/wikimedia/mediawiki-services-mathjax-attic back to https://github.com/wikimedia/mediawiki-services-mathjax and keeping it unarchived for Gerrit to replicate to it # T249908 [11:19:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:21:02] Lucas_WMDE: will look into it [11:21:13] thanks [11:21:22] there was a probably-related task, i can dig it up i na few minutes [11:21:25] (in meeting sry) [11:22:09] Lucas_WMDE: yeah so Varnish got upgraded from 5 to 6 on production but that did not happen on beta. So it the main cache is broken ( https://phabricator.wikimedia.org/T267561 ) [11:24:53] hashar: the task I remembered was https://phabricator.wikimedia.org/T267578, sounds like it’s the same issue [11:25:12] yeah that is the same more or less [11:25:13] (Tyler had fixed that one two days ago, I wondered if it had relapsed) [11:25:16] oh [11:25:17] yeah [11:27:46] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO, 10Operations, 10Traffic: Puppet disabled in beta cluster varnish deployment-cache-text06 - https://phabricator.wikimedia.org/T267578 (10hashar) 05Openβ†’03Declined In T267561 , the varnish related packages have been upgraded to the ones for... [11:27:50] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic: Beta needs to be upgraded to Varnish 6 - https://phabricator.wikimedia.org/T267561 (10hashar) [11:27:57] Lucas_WMDE: so that is related and I have closed it in favor of the latest task https://phabricator.wikimedia.org/T267561 [11:29:19] ok [11:46:04] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic: Beta needs to be upgraded to Varnish 6 - https://phabricator.wikimedia.org/T267561 (10hashar) modules/profile/manifests/cache/varnish/frontend.pp line 87 invokes `confd::file` which initalizes the confd module. It has: ` name=modules/confd/manifests/ini... [12:02:57] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic: Beta needs to be upgraded to Varnish 6 - https://phabricator.wikimedia.org/T267561 (10hashar) I have moved the `confd::srv_dns: deployment-prep.eqiad.wmflabs` setting to a new puppet prefix `deployment-cache` at https://horizon.wikimedia.org/project/pre... [12:03:04] Lucas_WMDE: I gave it a try and unbroke one thing , but that is stuck at some later stage :-\ [12:03:22] it is unlikely to be fixed before next week though. Alost everyone is away [12:10:17] ok :/ thanks for trying [13:09:53] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic: Beta needs to be upgraded to Varnish 6 - https://phabricator.wikimedia.org/T267561 (10ArielGlenn) I was able to get further along by doing things manually as root on one of the instances, deployment-cache-text06. I looked at the systemd unit file, /li... [14:14:30] 10Beta-Cluster-Infrastructure: Beta cluster not working (Error: 502, Next Hop Connection Failed) - https://phabricator.wikimedia.org/T267796 (10AlexisJazz) [14:16:17] 10Beta-Cluster-Infrastructure: Beta cluster not working (Error: 502, Next Hop Connection Failed) - https://phabricator.wikimedia.org/T267796 (10ArielGlenn) The varnishes in beta are broken right now, see T267561 [14:26:38] 10Gerrit, 10Wikimedia-GitHub, 10Patch-For-Review: mediawiki/extensions/WSOAuth Github and Gerrit repo have diverged - https://phabricator.wikimedia.org/T263955 (10Xxmarijnw) I was not asking for permission, but rather wanted to make sure it was an OK thing to do from a DevOps perspective. I created a new pa... [14:28:11] 10Beta-Cluster-Infrastructure: Beta cluster not working (Error: 502, Next Hop Connection Failed) - https://phabricator.wikimedia.org/T267796 (10AlexisJazz) [14:28:49] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic: Beta needs to be upgraded to Varnish 6 - https://phabricator.wikimedia.org/T267561 (10AlexisJazz) [14:30:20] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic: Beta needs to be upgraded to Varnish 6 - https://phabricator.wikimedia.org/T267561 (10hashar) Ariel pointed /usr/lib/x86_64-linux-gnu/varnish/vmods/libvmod_re2.so is the wrong version. One this task or another related one, I remembered someone marke... [14:30:40] 10Phabricator, 10Tools: Publicly log account bans made using the phab-ban tool - https://phabricator.wikimedia.org/T200856 (10Majavah) [14:31:37] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic: Beta needs to be upgraded to Varnish 6 - https://phabricator.wikimedia.org/T267561 (10hashar) ` name="$ sudo systemctl status varnish-frontend" ● varnish-frontend.service - varnish-frontend (Varnish HTTP Accelerator) Loaded: loaded (/lib/systemd/syst... [14:32:47] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic: Beta needs to be upgraded to Varnish 6 - https://phabricator.wikimedia.org/T267561 (10AlexisJazz) Hey it works now! [14:33:03] !log Beta cluster is available again. The frontend cache had a mixup between varnish 5 running and config being for varnish 6. The package got upgraded and it seems to serve traffic now. # T267561 [14:33:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:33:07] T267561: Beta needs to be upgraded to Varnish 6 - https://phabricator.wikimedia.org/T267561 [14:33:07] Lucas_WMDE: beta cluster is up [14:33:43] <3 [14:34:33] yup , seems to be working [14:34:36] including edits [14:45:05] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic, 10User-Ryasmeen: Beta needs to be upgraded to Varnish 6 - https://phabricator.wikimedia.org/T267561 (10hashar) 05Openβ†’03Resolved I have enabled puppet again and ran it. Something I spotted: ` --- /lib/systemd/system/confd.service 2020-11-06 20:04:... [14:46:05] Lucas_WMDE: magic. apergos and I looked at it on and off and spammed commands to fix stuff as they broke [14:46:27] seems like the issue was a faulty upgrade of debian package from varnish 5 to 6 [14:48:39] ok so folks, deployment-prep has the latest icu now (libicu63) [14:49:03] I doubt you should see anything goofy but just in case [14:50:28] i see [14:51:00] and if in regular use you see nothing interesting, that too is useful info [14:51:10] monday the upgrade is planned on production :-) [19:46:05] (03PS3) 10Thiemo Kreuz (WMDE): Add test for index error in AssignmentInReturnSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/606157 [19:50:13] (03CR) 10jerkins-bot: [V: 04-1] Add test for index error in AssignmentInReturnSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/606157 (owner: 10Thiemo Kreuz (WMDE)) [20:12:14] 10Release-Engineering-Team-TODO, 10Release, 10Train Deployments: 1.36.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T263184 (10Krinkle) Consider this train blocked until the cause behind T267668 is either fixed or reverted in master. I've prepared a revert at 10Continuous-Integration-Config, 10Test-Coverage: Run proactive coverage checking on more repos - https://phabricator.wikimedia.org/T267781 (10Reedy) I guess some overlap with {T265407}, though that is for skins, and this is (primarily) filed for libraries..