[00:33:51] 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Release: Tag Scap release 1.36.0 and ask serviceops to deploy it to production - https://phabricator.wikimedia.org/T266676 (10DannyS712) [00:35:11] 10MediaWiki-Releasing, 10Security: Release MediaWiki 1.31.9/1.34.3/1.35.0 - https://phabricator.wikimedia.org/T256334 (10DannyS712) >>! In T256334#6499324, @Reedy wrote: >>>! In T256334#6499264, @Jdforrester-WMF wrote: >> Resolved, surely? > > Ish, kinda, not completely. > > One task to open up (after I've h... [00:47:05] 10Release-Engineering-Team, 10Product-Infrastructure-Team-Backlog, 10Recommendation-API, 10Services, and 2 others: Migrate recommendation-api to kubernetes - https://phabricator.wikimedia.org/T241230 (10bmansurov) @akosiaris Thanks for reviewing my patchsets. I was wondering if you've seen my last comment.... [07:51:55] (03CR) 10Hashar: [V: 03+2 C: 03+2] Add metrics-reporter-jmx plugin [software/gerrit] (deploy/wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/640177 (https://phabricator.wikimedia.org/T184086) (owner: 10Hashar) [07:53:52] (03PS2) 10Hashar: Add metrics-reporter-prometheus plugin [software/gerrit] (deploy/wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/640174 (https://phabricator.wikimedia.org/T184086) [07:54:02] (03CR) 10Hashar: [V: 03+2 C: 03+2] Add metrics-reporter-prometheus plugin [software/gerrit] (deploy/wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/640174 (https://phabricator.wikimedia.org/T184086) (owner: 10Hashar) [09:28:58] 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T263182 (10Aklapper) #Regression in ar.wp in T267614 has also been mentioned (CC @cscott). [10:41:36] 10Phabricator, 10Mail: Phab email account verification not received - https://phabricator.wikimedia.org/T267590 (10Aklapper) Not sure I performed the same steps trying to reproduce, hence listing them here: * In the web browser, in Phabricator, I go to `Settings > Email Addresses`. I click "Add New Address".... [10:54:05] 10Beta-Cluster-Infrastructure, 10DBA, 10Operations, 10serviceops: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 (10ArielGlenn) I will updating to icu63 in deployment-prep, with Moritz looking on. This will likely happen later today, and I'll post updates about the prog... [11:09:51] (03CR) 10Lars Wirzenius: [C: 03+2] Allow default l10n language to be set by SCAP_MW_LANG environment var [tools/scap] - 10https://gerrit.wikimedia.org/r/640231 (owner: 10Ahmon Dancy) [11:12:10] (03Merged) 10jenkins-bot: Allow default l10n language to be set by SCAP_MW_LANG environment var [tools/scap] - 10https://gerrit.wikimedia.org/r/640231 (owner: 10Ahmon Dancy) [11:12:40] (03CR) 10Lars Wirzenius: "> Patch Set 2: Code-Review-1" (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/634929 (owner: 10Lars Wirzenius) [11:32:55] 10MediaWiki-Releasing, 10Security: Obtain CVEs for 1.31.9/1.34.3/1.35.0 security releases - https://phabricator.wikimedia.org/T256341 (10Reedy) 05Open→03Resolved p:05Triage→03Medium a:03Reedy [11:32:58] 10MediaWiki-Releasing, 10Security: Release MediaWiki 1.31.9/1.34.3/1.35.0 - https://phabricator.wikimedia.org/T256334 (10Reedy) [11:33:03] 10MediaWiki-Releasing, 10Security: Obtain CVEs for 1.31.9/1.34.3/1.35.0 security releases - https://phabricator.wikimedia.org/T256341 (10Reedy) [11:33:29] 10MediaWiki-Releasing, 10Security: Release MediaWiki 1.31.9/1.34.3/1.35.0 - https://phabricator.wikimedia.org/T256334 (10Reedy) 05Open→03Resolved p:05Triage→03Medium a:03Reedy >>! In T256334#6615203, @DannyS712 wrote: >>>! In T256334#6499324, @Reedy wrote: >>>>! In T256334#6499264, @Jdforrester-WMF w... [11:41:44] (03PS3) 10Lars Wirzenius: build scap.deb in Docker [tools/scap] - 10https://gerrit.wikimedia.org/r/634929 [11:59:19] hasharLunch / awight are we meeting today? [11:59:30] kostajh: Planning to! [11:59:43] k [12:06:55] (03PS27) 10Awight: Dockerfile launches external apache [integration/quibble] - 10https://gerrit.wikimedia.org/r/516729 (https://phabricator.wikimedia.org/T225218) (owner: 10Kosta Harlan) [12:32:23] (03PS28) 10Awight: Dockerfile launches external apache [integration/quibble] - 10https://gerrit.wikimedia.org/r/516729 (https://phabricator.wikimedia.org/T225218) (owner: 10Kosta Harlan) [12:41:02] (03PS29) 10Awight: Dockerfile launches external apache [integration/quibble] - 10https://gerrit.wikimedia.org/r/516729 (https://phabricator.wikimedia.org/T225218) (owner: 10Kosta Harlan) [12:56:57] (03CR) 10Kosta Harlan: [C: 03+1] "Adam and I did some interactive debugging. It works! The follow up is that Adam will slim down the configuration files to either include j" [integration/quibble] - 10https://gerrit.wikimedia.org/r/516729 (https://phabricator.wikimedia.org/T225218) (owner: 10Kosta Harlan) [12:57:29] awight: does api-testing work for you with that patch as well? [13:03:46] kostajh: It did at one point, let me check again... [13:25:24] kostajh: a whole lot of "connection refused" [13:27:10] yeah, that's what I see as well [13:27:27] I think I may know why, one sec [13:29:19] oh, maybe I don't. awight I assumed it was because of a missing rewrite rule for rest.php [13:29:46] (side note: would be nice if we followed the convention of serving from a /w directory as that makes the configuration more aligned with production but that's a bigger change) [13:30:58] kostajh: ah yeah. It's got to be the router, right? https://gerrit.wikimedia.org/r/c/integration/quibble/+/638706/1/quibble/backend.py [13:31:06] ah [13:31:23] so uh how does that work under Apache? :-) [13:31:25] awight: in core, we have tests/api-testing/.api-testing-quibble.json [13:31:32] and that has hardcoded port 9412 [13:31:38] ^ [13:31:44] hiiii [13:31:51] sorry was having lunch with kids :-\ [13:32:10] hashar: :-) That's the best! Sorry to hear that they don't have school, though? [13:32:22] port 9412 is arbitrary. We took that port to differentiate it from the Apache 80 port at the time we ran CI jobs directly on the jenkins host [13:32:35] my oldest kid is at home yeah :) [13:33:04] for api-testing, the nodejs module has support to add configuration in some json file indeed [13:33:05] I'm secretly looking forward to when schools are closed again [13:33:11] hehe [13:33:31] at least in France, they are trying to avoid closing schools [13:33:53] cause that further delay learning, and affect the most the least favorised kids :-\ [13:34:38] still about quibble, the router.php is really just for the php built in server [13:34:57] when using an external server such as Apache, there should be some mod_rewrite rules here and there [13:35:21] nice [13:35:35] I think the issue is that we need to have both 9412 and 9413 supported [13:35:51] (03CR) 10Hashar: [C: 03+1] "+1 on the general approach ;]" [integration/quibble] - 10https://gerrit.wikimedia.org/r/516729 (https://phabricator.wikimedia.org/T225218) (owner: 10Kosta Harlan) [13:35:54] so maybe api-testing-quibble.json in core should remove the port, and quibble in ApiTesting in command.py should pass in the active port [13:36:07] sounds good yeah [13:36:23] maybe api-testing supports an env variable to recognizes it [13:36:55] (03PS1) 10Hoo man: Make is_in_docker also work if run under podman [integration/quibble] - 10https://gerrit.wikimedia.org/r/640413 [13:37:06] doh [13:37:24] WE HAVE A NEW COMER! :] [13:37:46] Send them the T-shirt <3 [13:38:12] (03CR) 10jerkins-bot: [V: 04-1] Make is_in_docker also work if run under podman [integration/quibble] - 10https://gerrit.wikimedia.org/r/640413 (owner: 10Hoo man) [13:44:20] kostajh: Want to split this, u can patch core and I'll pass in the port number from quibble? [13:47:02] kostajh: I'm so confused. What reads API_TESTING_CONFIG_FILE? [13:48:11] How does it know the server URL in the first place... but not the port? I guess lots of hardcoding? [13:49:45] awight: I'll have to look a bit later, sorry! [13:49:47] codesearch seems worse lately [13:49:53] kostajh: +1 no worries [13:51:37] found it, https://gerrit.wikimedia.org/g/mediawiki/tools/api-testing [13:53:28] (03CR) 10Hashar: Make is_in_docker also work if run under podman (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/640413 (owner: 10Hoo man) [13:55:02] okay, for simplicity I'm just going to rewrite this file. We can tweak to pass the URL through the environment, later. If we did, we might as well replace everything in this file with environment. [13:55:23] (03PS2) 10Hashar: Make is_in_docker also work if run under podman [integration/quibble] - 10https://gerrit.wikimedia.org/r/640413 (owner: 10Hoo man) [14:22:56] (03PS30) 10Awight: Dockerfile launches external apache [integration/quibble] - 10https://gerrit.wikimedia.org/r/516729 (https://phabricator.wikimedia.org/T225218) (owner: 10Kosta Harlan) [14:22:59] (03PS1) 10Awight: Adapt API URL to the current webserver [integration/quibble] - 10https://gerrit.wikimedia.org/r/640432 (https://phabricator.wikimedia.org/T225218) [14:27:23] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10serviceops, 10Patch-For-Review: Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10MoritzMuehlenhoff) Before we start reimaging, let's also merge https://gerrit.wikimed... [14:30:28] 10Beta-Cluster-Infrastructure, 10DBA, 10Operations, 10serviceops: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 (10Reedy) >>! In T264991#6615421, @Marostegui wrote: > Thank you! What's the expected impact of `updateCollation.php`? Many many categorylinks rows being up... [14:31:53] (03CR) 10jerkins-bot: [V: 04-1] Adapt API URL to the current webserver [integration/quibble] - 10https://gerrit.wikimedia.org/r/640432 (https://phabricator.wikimedia.org/T225218) (owner: 10Awight) [14:31:53] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Operations, 10serviceops, 10Patch-For-Review: Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10MoritzMuehlenhoff) [14:32:42] (03CR) 10jerkins-bot: [V: 04-1] Dockerfile launches external apache [integration/quibble] - 10https://gerrit.wikimedia.org/r/516729 (https://phabricator.wikimedia.org/T225218) (owner: 10Kosta Harlan) [14:34:15] (03CR) 10Hashar: [C: 03+2] "Thank you Hoo man!" [integration/quibble] - 10https://gerrit.wikimedia.org/r/640413 (owner: 10Hoo man) [14:34:37] 10Beta-Cluster-Infrastructure, 10DBA, 10Operations, 10serviceops: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 (10jijiki) >>! In T264991#6615421, @Marostegui wrote: > Thank you! What's the expected impact of `updateCollation.php`? We will run on one wiki and see how... [14:34:42] (03PS2) 10Awight: Adapt API URL to the current webserver [integration/quibble] - 10https://gerrit.wikimedia.org/r/640432 (https://phabricator.wikimedia.org/T225218) [14:34:44] hmm, it didn't quite work her. [14:34:44] (03PS31) 10Awight: Dockerfile launches external apache [integration/quibble] - 10https://gerrit.wikimedia.org/r/516729 (https://phabricator.wikimedia.org/T225218) (owner: 10Kosta Harlan) [14:34:47] (03PS1) 10Awight: Typo in Dockerfile [integration/quibble] - 10https://gerrit.wikimedia.org/r/640433 [14:34:48] *yet [14:40:50] 10Beta-Cluster-Infrastructure, 10DBA, 10Operations, 10serviceops: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 (10Marostegui) Thank you @Reedy! [14:43:39] (03CR) 10jerkins-bot: [V: 04-1] Dockerfile launches external apache [integration/quibble] - 10https://gerrit.wikimedia.org/r/516729 (https://phabricator.wikimedia.org/T225218) (owner: 10Kosta Harlan) [14:43:52] (03CR) 10jerkins-bot: [V: 04-1] Adapt API URL to the current webserver [integration/quibble] - 10https://gerrit.wikimedia.org/r/640432 (https://phabricator.wikimedia.org/T225218) (owner: 10Awight) [14:58:58] (03Merged) 10jenkins-bot: Make is_in_docker also work if run under podman [integration/quibble] - 10https://gerrit.wikimedia.org/r/640413 (owner: 10Hoo man) [14:59:19] (03CR) 10Kosta Harlan: [C: 03+2] Typo in Dockerfile [integration/quibble] - 10https://gerrit.wikimedia.org/r/640433 (owner: 10Awight) [15:00:19] (03PS3) 10Awight: Adapt API URL to the current webserver [integration/quibble] - 10https://gerrit.wikimedia.org/r/640432 (https://phabricator.wikimedia.org/T225218) [15:00:21] (03PS32) 10Awight: Dockerfile launches external apache [integration/quibble] - 10https://gerrit.wikimedia.org/r/516729 (https://phabricator.wikimedia.org/T225218) (owner: 10Kosta Harlan) [15:05:24] Great, now we can see that there are missing rewrite rules. [15:09:00] Not so clear what to do... https://www.mediawiki.org/wiki/RESTBase/Installation#Apache [15:09:26] kostajh: in case you have any apache conf sitting around? ^ [15:10:25] eh this is not restbase, is it. [15:24:39] (03Merged) 10jenkins-bot: Typo in Dockerfile [integration/quibble] - 10https://gerrit.wikimedia.org/r/640433 (owner: 10Awight) [15:35:59] kostajh: Giving up for the day, but I found that the api-testing job fails with "SyntaxError: Unexpected token < in JSON at position 0", so I assume a bad response from Apache. But here's the crazy part: if I run the docker image with --skip-zuul, tests pass. Adding --skip-zuul again, it fails. [15:36:43] whew, sounds weird [15:41:35] 10Beta-Cluster-Infrastructure, 10DBA, 10Operations, 10serviceops: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 (10ArielGlenn) Upgrade plan on deployment-prep: [] add profile::mediawiki::php::icu63: true to hiera for deployment-prep project prefix; this will only have... [15:42:15] note that I plan to start the update of deployment-prep to libicu63 (new php packages, new libxml as well) over the next hour [15:42:24] unless someone stops me! https://phabricator.wikimedia.org/T264991#6616409 [15:42:34] 10Beta-Cluster-Infrastructure, 10DBA, 10Operations, 10serviceops: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 (10MoritzMuehlenhoff) Sounds good! [15:52:09] Danger: There was an error submitting the form. Please try again. <--- get when trying to add an entry to the project puppet hiera settings for deployment-prep [15:52:13] retried, same [15:55:15] third time it took it, just bad luck the first two times i guess [15:55:25] (03CR) 10Ahmon Dancy: build scap.deb in Docker (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/634929 (owner: 10Lars Wirzenius) [15:59:58] (03CR) 10Ahmon Dancy: build scap.deb in Docker (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/634929 (owner: 10Lars Wirzenius) [16:07:02] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO, 10Operations, 10Traffic: Puppet disabled in beta cluster varnish deployment-cache-text06 - https://phabricator.wikimedia.org/T267578 (10jijiki) p:05Triage→03Medium [16:07:28] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic: Beta needs to be upgraded to Varnish 6 - https://phabricator.wikimedia.org/T267561 (10jijiki) p:05Triage→03Medium [16:08:33] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO, 10Operations, 10Traffic: Beta cluster seems to be extremely slow for logged in user during page navigation - https://phabricator.wikimedia.org/T267435 (10jijiki) p:05Triage→03Medium [16:09:57] (03CR) 10Ahmon Dancy: build scap.deb in Docker (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/634929 (owner: 10Lars Wirzenius) [16:17:13] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO, 10Operations, 10Traffic: Puppet disabled in beta cluster varnish deployment-cache-text06 - https://phabricator.wikimedia.org/T267578 (10thcipriani) Even with puppet disabled, packages were upgraded which broke this again. I set the packages to... [16:19:21] 10Beta-Cluster-Infrastructure, 10DBA, 10Operations, 10serviceops: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 (10ArielGlenn) Note that I'm running cumin 'O{project:deployment-prep name:^deployment-mediawiki-[0-9]+$ } or O{project:deployment-prep name:^deployment... [16:30:50] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Add prometheus exporter to Gerrit - https://phabricator.wikimedia.org/T184086 (10hashar) [16:31:47] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Add prometheus exporter to Gerrit - https://phabricator.wikimedia.org/T184086 (10fgiunchedi) The patch is live, unfortunately due to how our Prometheus puppetization works it means we'r... [16:45:58] (03PS2) 10Ahmon Dancy: Use SCAP_MW_LANG=en to speed up check script [tools/train-dev] - 10https://gerrit.wikimedia.org/r/640253 [16:57:12] 2020-11-10T16:50:01Z INFO git.cmd: git diff --abbrev=40 --full-index -M --raw --no-color [16:57:13] 2020-11-10T16:50:01Z ERROR sync-upstream: Local diffs detected. Commit your changes! [16:58:34] starts: 2020-11-06T20:10:01Z ERROR sync-upstream: Local diffs detected. Commit your changes! [17:06:59] change in [17:07:08] diff --git a/modules/confd/manifests/init.pp b/modules/confd/manifests/init.pp [17:07:16] class confd( ... [17:07:24] - Stdlib::Fqdn $srv_dns = $::domain, [17:07:34] + String $srv_dns = $::domain, [17:07:41] do we know who was working on this? [17:12:07] 10Beta-Cluster-Infrastructure, 10DBA, 10Operations, 10serviceops: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 (10ArielGlenn) Can't proceed at the moment, puppet sync to deployment-prep has been broken since Nov 6. Log excerpts from the earliest error: ` 2020-11-06T20... [17:22:41] apergos: That looks like something Tyler was working on. [17:22:46] * dancy digs [17:23:41] thanks for looking! [17:26:32] https://phabricator.wikimedia.org/T267439 [17:28:49] huh [17:30:52] I have askd jbond42 to have a look, hmm [17:38:34] im curious did the cxhange here https://phabricator.wikimedia.org/T267439#6609779 make any difference? [17:40:06] dancy or thcipriani...^^ [17:40:45] I'll have to defer to Tyler on that topic. [17:43:08] he around today? [17:43:16] the problem as i see it is that you attempt to configure the srv_dns value to an empty string in horizone using confd::srv_dns. which would explain the change from Stdlib::fqdn to String as an empty string is not a valid Stdlib::Fqnd. however the thing that is confusing me is that with my change i https://phabricator.wikimedia.org/rOPUPc871b021a44108bed44cd044f5177b075ecb322c i derectly pass [17:43:22] the srv_dns value to confd via ... [17:43:25] ... profile::conftool::state and afaik passing a paramter diretly should take precedence over the one you have in horizon [17:44:09] He is around as far as I know. That said, I haven't seen him send any messages today. I have a meeting scheduled with him in a few hours. [17:44:30] currently the puppet policy in production dosn;t allow overriding the srv_dns paramter other then via automatic paramter lookup (which isnt supported in production, but used extensivly in cloud) [17:45:20] i can probably make a nother local change which should get you back to the behviour before my change and if this is blocking [17:45:30] and look for a better soultion to apply to production [17:45:33] ..later [17:47:27] i.e. i would just update the template (modules/confd/templates/initscripts/confd.systemd.erb) to replace `<%= @srv_dns %>` to a static string [17:47:45] the main thing right now for me is the uncommited local change [17:48:49] im gonna go out on a limb and say that is almost definetly from thcipriani see: https://phabricator.wikimedia.org/T267439#6609779 [17:49:00] yeah that's what dancy said [17:49:51] I don't think Tyler would mind resetting that file so that you can make progress, especially if what you're trying to do fixes what he was trying to fix. [17:50:07] well no, I'm just trying to test the new icu [17:50:13] completely unrelated, unfortunately [17:51:26] hrm. [17:51:41] apergos: im about 90% sure that the local change dosn;t fix the underlingin issues thcipriani was trying to fix and can probably be reverted without making things anyworse [17:52:12] well I'll do it and leave a note on the my task, and cc him I guess [17:52:20] hopefully it doesn't make things worse [17:52:32] * dancy crosses fingers [17:52:49] yes i have also added my self to the task and will make a comment [17:54:08] ok great [17:54:37] 10Beta-Cluster-Infrastructure, 10DBA, 10Operations, 10serviceops: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 (10ArielGlenn) This is apparently from T267439 After some discussion with jbond and dancy in irc, I am going to revert that and hope I'm not making the varn... [17:54:53] I have reverted, now we wait for a sync and see what happens [17:54:59] ack [17:55:26] thank you both for spending time with me on this [17:55:35] Happy to (sorta) help. [17:56:05] heh [17:56:09] 5 minutes til the next sync [18:01:15] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10User-Ryasmeen: MediaWiki beta varnish is down. - https://phabricator.wikimedia.org/T267439 (10jbond) >>! In T267439#6609779, @thcipriani wrote: > Noticed that puppet stopped running on beta's `deployment-cache-text06` due to c871b021a44108bed44cd04... [18:01:18] sync passed [18:01:21] looked good! [18:02:04] cool and i have added a coment to the task [18:02:09] I see it :-) [18:02:16] thanks again all! [18:02:22] np [18:03:30] 10Beta-Cluster-Infrastructure, 10DBA, 10Operations, 10serviceops: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 (10ArielGlenn) Puppet sync back to working. Back on track to continue with the update in deployment-prep. [18:15:59] 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T263182 (10brennen) I missed T267614 in the shuffle yesterday afternoon. After discussion on IRC: We'll backport @cscott's fix shortl... [18:18:56] volans: regarding the cumin integration tests, if we can get them to run in kubernetes, we can run them in CI as it is now. for users running them locally, in that case they'd need to at least download helm and minikube in order to run them. [18:20:55] not sure if that's something you want to pursue, but it is an option anyway [18:38:39] 10Beta-Cluster-Infrastructure, 10DBA, 10Operations, 10serviceops: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 (10ArielGlenn) Update in deployment-prep is now complete, assuming I did not miss any hosts. ` root@deployment-cumin:~# cumin 'O{project:deployment-prep... [18:40:25] !log deployment-prep: switched deployment-prep to libicu63, see T264991 [18:40:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:40:28] T264991: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 [18:41:01] Test away, folks. I'll be doing some dump testing over the next few days (yeah I know, vacation etc but eh) [18:42:53] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic: Beta needs to be upgraded to Varnish 6 - https://phabricator.wikimedia.org/T267561 (10Ottomata) I just tried to fix by installing varnish 6, but clearly (and obviously) it isn't that simple. ` $ sudo apt-get remove varnish varnish-modules libvmod-netma... [19:31:49] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic: Beta needs to be upgraded to Varnish 6 - https://phabricator.wikimedia.org/T267561 (10ArielGlenn) I guess that T267439 might be related. [19:32:53] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10MW-on-K8s: Experiment with generating json config - https://phabricator.wikimedia.org/T267057 (10Krinkle) See {T223602} for the previous work in this area where myself and James have analyzed most all of... [19:32:59] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10MW-on-K8s, 10Performance-Team (Radar): Experiment with generating json config - https://phabricator.wikimedia.org/T267057 (10Krinkle) [20:25:08] (03CR) 10Thcipriani: [C: 03+2] make-release: Stop branching the Petition extension [tools/release] - 10https://gerrit.wikimedia.org/r/639888 (https://phabricator.wikimedia.org/T267450) (owner: 10Jforrester) [20:26:15] (03Merged) 10jenkins-bot: make-release: Stop branching the Petition extension [tools/release] - 10https://gerrit.wikimedia.org/r/639888 (https://phabricator.wikimedia.org/T267450) (owner: 10Jforrester) [20:39:52] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible, 10User-zeljkofilipin: Beta Cluster is down - https://phabricator.wikimedia.org/T267551 (10thcipriani) This one might not be a varnish problem. I see php-fpm complaining on `deployment-mediawiki-07`: ` PHP Fatal error: Uncaught RuntimeException: Re... [20:40:12] 10Release-Engineering-Team (Deployment services), 10VPS-project-codesearch, 10MediaWiki-Release-Tools, 10Patch-For-Review: Stop branching the Petition repo for production (and thus showing up in CodeSearch) - https://phabricator.wikimedia.org/T267450 (10Jdforrester-WMF) 05Open→03Resolved [20:42:12] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible, 10User-Ryasmeen, 10User-zeljkofilipin: Beta Cluster is down - https://phabricator.wikimedia.org/T267551 (10thcipriani) 05Open→03Resolved a:03thcipriani Restarting php7.2-fpm on `deployment-mediawiki-07` fixed this one. [21:10:24] 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T263182 (10brennen) 05Open→03Resolved [21:25:48] 10Release-Engineering-Team (Pipeline), 10Release Pipeline: blubber: Execute python module as RUN command of dockerfile - https://phabricator.wikimedia.org/T267685 (10jeena) [21:43:14] thcipriani are you around? There might be a complication at T256367 since your SUL account wasn't created by WMF office IT as far as I can tell - https://phabricator.wikimedia.org/T256367#6617813 [21:43:14] T256367: WMF-NDA access for DannyS712 - https://phabricator.wikimedia.org/T256367 [21:58:42] * thcipriani reads [22:00:13] DannyS712: heh, I would be running quite a long con to get you NDA access :P [22:00:37] what other proof can I offer? [22:00:45] (03PS1) 10Dduvall: experimental: flense command for aggregating blubber LLB graphs [blubber] - 10https://gerrit.wikimedia.org/r/640562 [22:01:31] No idea - maybe have another manager confirm, or someone from the IT team confirm onwiki (have they done so in the past?) [22:01:54] also, error on the beta cluster: [22:01:54] [X6sNIqwQBHcAAB@uG@YAAAAU] /rpc/RunSingleJob.php PHP Fatal Error from line 84 of /srv/mediawiki/php-master/includes/libs/redis/RedisConnectionPool.php: Uncaught RuntimeException: RedisConnectionPool requires a Redis client library. See https://www.mediawiki.org/wiki/Redis#Setup in [22:01:54] /srv/mediawiki/php-master/includes/libs/redis/RedisConnectionPool.php:84 [22:01:54] Seeing a bunch of these [22:01:58] (03CR) 10jerkins-bot: [V: 04-1] experimental: flense command for aggregating blubber LLB graphs [blubber] - 10https://gerrit.wikimedia.org/r/640562 (owner: 10Dduvall) [22:02:02] (03PS2) 10Dduvall: experimental: flense command for aggregating blubber LLB graphs [blubber] - 10https://gerrit.wikimedia.org/r/640562 [22:03:12] (03CR) 10jerkins-bot: [V: 04-1] experimental: flense command for aggregating blubber LLB graphs [blubber] - 10https://gerrit.wikimedia.org/r/640562 (owner: 10Dduvall) [22:03:57] 10LibUp, 10MW-1.31-release, 10Patch-For-Review: Bump PHPCS version in REL1_31 extensions - https://phabricator.wikimedia.org/T236070 (10Reedy) p:05High→03Medium Bumping down to Medium as I've done all the remaining skins/extensions that are tarballed [22:04:54] apergos: see DannyS712's comment.... did php-redis get removed? [22:05:39] I saw that same error in beta a bit ago, but php-redis was still there [22:05:42] * thcipriani finds task [22:06:32] https://phabricator.wikimedia.org/T267551 [22:08:22] thcipriani see https://logstash-beta.wmflabs.org/goto/ebd35768e53ca6079adce9e4826d4570 for the errors - still occuring [22:09:18] Reedy: no but it wasn't in the upgrade list [22:09:21] let me do that now [22:11:39] someone has already, or it is already the latest version [22:11:41] I came across this because I was trying to get https://meta.wikimedia.beta.wmflabs.org/wiki/Special:Gadgets to update after my edit to the definitions, and it won't - Krinkle do you know how to get it to refresh? [22:12:01] purge the definition page [22:12:06] I did [22:13:28] * Purge the definitions cache, for example if MediaWiki:Gadgets-definition [22:13:28] * was edited. [22:13:34] Presumably cache is just unhappy [22:13:54] !log restarting php7.2-fpm on deployment-jobrunner03 and deployment-parsoid11 [22:13:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:14:12] those were the two hosts still throwing that error from logstash [22:14:19] both had php-redis installed [22:14:41] all of them should have restarted with the update but who knows [22:15:07] seem to not be reporting that error after the kick, must've still needed a restart [22:15:11] computers suck [22:15:15] my check that the library was available was only from php cli, not the web [22:15:15] ^ [22:15:26] yeah they do [22:15:33] feel free to note that on uh [22:15:40] https://phabricator.wikimedia.org/T264991 [22:15:51] might be of interest for when the upgrade happens in prod [22:19:53] ok so I lied, php-redis was in my update list and it was installed at the same time as everything else [22:19:59] just rechecked the command run, and also the logs [22:20:05] so not even that a little bit. [22:20:40] varnish is still a thing, thcipriani, you might have seen the comments on the task [22:20:56] had to revert your local fix bcause it broke puppet sync too [22:22:09] apergos: just reading that, local fix revert is fine, sorry for leaving a mess, this is related to: https://phabricator.wikimedia.org/T267578 -- which I have no idea about other than the new varnish keeps getting installed and breaking beta [22:23:14] I think ottomata (?) might have tried to upgrade to varnish 6 and hope that would help [22:24:08] there's three different tickets floating around about it [22:24:10] https://phabricator.wikimedia.org/T267561 [22:24:20] that's the last (?) one [22:24:51] linked to that one just now [22:24:57] no idea here either, kina hope someone who knows varnish can step in and rescue things [22:26:21] anyways, anyone who wants, please play with things with an eye to collation etc, with the new libicu63 [22:27:52] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic: Beta needs to be upgraded to Varnish 6 - https://phabricator.wikimedia.org/T267561 (10thcipriani) [22:27:56] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO, 10Operations, 10Traffic: Puppet disabled in beta cluster varnish deployment-cache-text06 - https://phabricator.wikimedia.org/T267578 (10thcipriani) [22:28:02] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic: Beta needs to be upgraded to Varnish 6 - https://phabricator.wikimedia.org/T267561 (10thcipriani) [22:28:03] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible, 10User-Ryasmeen, 10User-zeljkofilipin: Beta Cluster is down - https://phabricator.wikimedia.org/T267551 (10thcipriani) [22:30:17] https://phabricator.wikimedia.org/T267439 here's the last one of those. meh [22:32:18] anyways, off for the night, hope everything stays working [22:38:55] turns out the issue with gadgets was my forgetting to add a | [22:45:05] 10MediaWiki-Codesniffer, 10PHP 8.0 support, 10User-DannyS712: Create sniff for `match` errors in PHP 8.0 - https://phabricator.wikimedia.org/T267689 (10DannyS712) [22:46:30] 10MediaWiki-Codesniffer, 10PHP 8.0 support, 10User-DannyS712: Create sniff for `match` errors in PHP 8.0 - https://phabricator.wikimedia.org/T267689 (10DannyS712) [23:18:08] (03PS1) 10Brennen Bearnes: WIP: Add xdebug build script to stretch [releng/dev-images] - 10https://gerrit.wikimedia.org/r/640567 [23:19:02] !log Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/640226 and https://gerrit.wikimedia.org/r/c/performance/coal/+/640227 in beta; should only affect deployment-webperf11. [23:19:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:19:27] (03CR) 10Jeena Huneidi: [C: 03+2] "Merging this since there have been no new reviews and we've made many of the suggested changes. We will wait to merge the core patch until" [releng/dev-images] - 10https://gerrit.wikimedia.org/r/622575 (https://phabricator.wikimedia.org/T245444) (owner: 10Brennen Bearnes) [23:19:50] I may have some time next to look at Varnish if this goes OK. [23:22:03] (03CR) 10Jeena Huneidi: [V: 03+2 C: 03+2] Split php-fpm and apache2 containers [releng/dev-images] - 10https://gerrit.wikimedia.org/r/622575 (https://phabricator.wikimedia.org/T245444) (owner: 10Brennen Bearnes)