[00:15:39] (03PS1) 10Krinkle: zuul: Try to convert a 'branch' filter to 'skip-if' [integration/config] - 10https://gerrit.wikimedia.org/r/499682 [00:17:07] (03CR) 10jerkins-bot: [V: 04-1] zuul: Try to convert a 'branch' filter to 'skip-if' [integration/config] - 10https://gerrit.wikimedia.org/r/499682 (owner: 10Krinkle) [01:09:01] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.33.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T206677 (10Lucas_Werkmeister_WMDE) [01:47:23] perhaps known, https://integration.wikimedia.org/zuul/ is empty, https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/AdvancedSearch/+/499661/ didn't run CI tests, and a `recheck` comment doesn't seem to have triggered anything [03:00:15] 10Beta-Cluster-Infrastructure, 10User-DannyS712: Beta Cluster: Rights Request - https://phabricator.wikimedia.org/T219475 (10DannyS712) [03:02:04] 10Beta-Cluster-Infrastructure, 10User-DannyS712: Requesting Pending Changes Reviewer on enwiki beta - https://phabricator.wikimedia.org/T188873 (10DannyS712) a:03DannyS712 [03:05:47] 10Beta-Cluster-Infrastructure, 10User-DannyS712: Requesting Pending Changes Reviewer on enwiki beta - https://phabricator.wikimedia.org/T188873 (10DannyS712) a:05DannyS712→03None Oops, I didn't see the second part of the request (the protection). I have added the pending changes reviewer right (https://en.... [05:06:54] phabricator down again? [05:07:02] greg-g? [07:05:45] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<20.00%) [07:15:48] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [08:02:58] (03PS1) 10Legoktm: [SecurePoll] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/499717 [08:03:28] (03CR) 10Legoktm: [C: 03+2] [SecurePoll] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/499717 (owner: 10Legoktm) [08:05:26] (03Merged) 10jenkins-bot: [SecurePoll] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/499717 (owner: 10Legoktm) [08:06:58] Krinkle: zuul: Minor clean up custom job settings is undeployed? [08:07:49] oh, it's a practical no-op [08:08:42] !log deployed https://gerrit.wikimedia.org/r/c/integration/config/+/499539 (no-op) and https://gerrit.wikimedia.org/r/499717 (SecurePoll phan) [08:08:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:08:44] legoktm: I might have forgot a deploy yesterday [08:09:04] didn't see anything from you :) [08:09:38] good! [08:09:51] also I have a question for you regarding quibble and filtering out stages to run / skip :) [08:10:09] currently if one does: quibble --run phpunit qunit mediawiki/extensionsBoilerPlate [08:10:15] mediawiki/extensionsBoilerPlate wil lbe considered as a run stage [08:10:20] because --run has nargs='*' [08:10:56] so I am pondering between having a comma separated list of args: --run phpunit,qunit but that comes with a dirty hack in argparse [08:10:57] or [08:11:25] use multiple one each taking a single stage: --run phpunit --run qunit [08:11:47] the later is straightforward in argparse. just nargs=1 action='append' [08:13:46] legoktm: ^^ random thought about quibble :D [08:14:40] hashar: comma separated seems the easiest [08:14:54] yeah my thought [08:14:57] well I don't know what the dirty hack would look like [08:14:59] at the price of some hack in th ecode [08:15:19] but even if if we had to do 'foo' in list.split(',') I'd be okay with that [08:15:35] (03CR) 10Hashar: "A big note: the jobs settings are applied serially and whenever one match, it will override value that might previously have been set. Ty" [integration/config] - 10https://gerrit.wikimedia.org/r/499539 (owner: 10Krinkle) [08:16:28] legoktm: turns out I already have the hack ready ubut it is late for you to review it anyway ;] [08:16:35] https://gerrit.wikimedia.org/r/#/c/integration/quibble/+/496125/5/quibble/cmd.py [08:16:44] i am adding you as a reviewer, there is no urgency for this anyway [09:37:50] (03CR) 10Hashar: [C: 03+2] "Sorry it took me a while to come to it. Note that 10_env_mw_install_path.php no more applies as far as I know. We can clean that up late" (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/494804 (owner: 10Krinkle) [09:38:43] (03Merged) 10jenkins-bot: mediawiki.d: Improve docs about dev settings and combine env sections [integration/quibble] - 10https://gerrit.wikimedia.org/r/494804 (owner: 10Krinkle) [09:40:30] (03CR) 10jenkins-bot: mediawiki.d: Improve docs about dev settings and combine env sections [integration/quibble] - 10https://gerrit.wikimedia.org/r/494804 (owner: 10Krinkle) [09:43:25] (03PS4) 10Hashar: mediawiki.d: Merge into one file [integration/quibble] - 10https://gerrit.wikimedia.org/r/494805 (owner: 10Krinkle) [09:49:09] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews, 10Graphoid, 10Operations, and 2 others: graphoid: Code stewardship request - https://phabricator.wikimedia.org/T211881 (10dr0ptp4kt) Okay, this has been sitting in draft for too long, so I'm going to provide this simply so that we have it her... [10:27:28] (03CR) 10Hashar: [C: 03+2] "That is arguably easier to follow/understand. Thank you!" [integration/quibble] - 10https://gerrit.wikimedia.org/r/494805 (owner: 10Krinkle) [10:28:04] (03Merged) 10jenkins-bot: mediawiki.d: Merge into one file [integration/quibble] - 10https://gerrit.wikimedia.org/r/494805 (owner: 10Krinkle) [10:28:30] (03CR) 10jenkins-bot: mediawiki.d: Merge into one file [integration/quibble] - 10https://gerrit.wikimedia.org/r/494805 (owner: 10Krinkle) [10:45:05] !log Tagged Quibble 0.0.30 6ddc6d508cb554e6443ff72648da3ea8a3253fff [10:45:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:50:15] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.33.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T206677 (10Lucas_Werkmeister_WMDE) [10:57:51] 10Release-Engineering-Team, 10MinervaNeue, 10Readers-Web-Backlog: MinervaNeue CI tests frequently fail on bad certificate or request timeout - https://phabricator.wikimedia.org/T219394 (10phuedx) A data point (may or mayn't be related): SauceLabs had an incident yesterday: https://status.us-west-1.saucelabs.... [11:29:13] 10Release-Engineering-Team, 10MinervaNeue, 10Readers-Web-Backlog: MinervaNeue CI tests frequently fail on bad certificate or request timeout - https://phabricator.wikimedia.org/T219394 (10phuedx) I kicked off a build and it passed: https://integration.wikimedia.org/ci/view/Reading-Web/job/selenium-MinervaNeu... [11:32:18] anyone around in this tz and care to look at https://phabricator.wikimedia.org/T219450 ? [11:32:25] being reported in -operations: [11:32:38] (01:29:53 μμ) yannf: This is a very serious bug. Now Main Namespace pages are now locked for everybody, including admins. Please fix ASAP. [11:32:47] hashar: any thoughts? (since you're here, sorry) [11:36:35] apergos: I suspect poking Daniel K might be better with it being seemingly his area of changes over the years [11:36:44] oh, he's in the right tz [11:36:48] rriiiiggghhhttt [11:36:51] Poked him in -core [11:37:10] although there were some structured-data-wikibase-thing commons deployments in the last days that might have... [11:37:16] who had those? I'll look [11:38:14] 18:43, 27 March 2019 (UTC) reported on wiki, could have been earlier [11:39:21] 19:00–21:00 UTC # 12:00–14:00 PDT 21:00–23:00 UTC+2 this was the train, later [11:39:50] Reedy: ^^ [11:39:58] o_0 [11:39:58] Hmm [11:40:04] I think it’s more likely due to a config change than the train deployment [11:40:09] see my comment on the task [11:40:46] Which config change then? :P [11:41:31] there was 'Use new WBCS on Commons too' earlier but it was reverted, also earlier [11:42:17] https://commons.wikimedia.org/wiki/Commons:Village_pump#%22wikitext%22_content_is_not_allowed_on_page_%E2%80%A6_in_slot_%22Main%22 here's the first report of the issue (that I know of) [11:42:42] presumably https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/499531, though it’s not directly linked in the SAL entry [11:43:11] (bleh, that’s a change where I uploaded a follow-up, which means it won’t revert cleanly :/ ) [11:43:32] sold, I buy that [11:44:06] mar 27 6:50 pm post-merge, is that utc? when did it make it around? [11:44:55] 10Gerrit, 10Release-Engineering-Team (Kanban), 10GitHub-Mirrors: Puppet repo not being updated on github - https://phabricator.wikimedia.org/T219264 (10Reedy) 05Resolved→03Open Doesn't look to be resolved to me... https://github.com/wikimedia/mediawiki-extensions-WikimediaMaintenance/commits/master Num... [11:45:11] oh, that's local time, so mar 27 4:50 utc maybe [11:46:07] at least 7 changes with "Commons" in them on the same day [11:46:25] this would have been noticed pretty soon after deployment I'd say [11:53:18] is anyone currently on the Commons issue? [11:53:25] otherwise I can try to revert the config and test it on mwdebug [11:53:33] Feel free [11:53:40] ok [11:54:01] (06:55:12 μμ) logmsgbot: !log jforrester@deploy1001 Synchronized wmf-config/InitialiseSettings.php: T214075 SDC: Enable Wikidata federation on Commons (duration: 00m 57s) (my time so 16:55 utc yesterday) [11:54:02] T214075: Enable federated access to entities and properties from Wikidata to Commons - https://phabricator.wikimedia.org/T214075 [11:54:12] please do and thanks [13:03:54] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10MediaWiki-General-or-Unknown, 10Patch-For-Review, 10User-zeljkofilipin: `npm install` fails for mediawiki/core with EPEERINVALID when running on Node 11 - https://phabricator.wikimedia.org/T210506 (10zeljkofilipin) a:03zeljkofilipin [13:11:29] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10MediaWiki-General-or-Unknown, 10Patch-For-Review, 10User-zeljkofilipin: `npm install` fails for mediawiki/core with EPEERINVALID when running on Node 11 - https://phabricator.wikimedia.org/T210506 (10hashar) This task got forked at s... [13:16:51] 10Gerrit, 10Release-Engineering-Team (Kanban), 10GitHub-Mirrors: Puppet repo not being updated on github - https://phabricator.wikimedia.org/T219264 (10hashar) 05Open→03Resolved The replication does work that extension got replicated to github: ` [2019-03-28 12:45:32,563] [19650510] Replication to git@gi... [13:21:11] 10Release-Engineering-Team, 10Developer Productivity, 10Epic: Add Windows installation Documentation and Install Script for local-charts repo - https://phabricator.wikimedia.org/T219438 (10hashar) Note that you can get a 90 days Windows VM directly from Microsoft: https://developer.microsoft.com/en-us/micros... [13:23:04] 10Release-Engineering-Team, 10Developer Productivity, 10Epic: Add Windows installation Documentation and Install Script for local-charts repo - https://phabricator.wikimedia.org/T219438 (10Addshore) >>! In T219438#5065571, @hashar wrote: > Note that you can get a 90 days Windows VM directly from Microsoft: h... [13:54:04] 10Gerrit, 10Release-Engineering-Team (Kanban), 10GitHub-Mirrors: Puppet repo not being updated on github - https://phabricator.wikimedia.org/T219264 (10Reedy) >>! In T219264#5065559, @hashar wrote: > The replication does work that extension got replicated to github: > ` > [2019-03-28 12:45:32,563] [19650510]... [13:56:25] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Quibble (marble): Quibble should clone repositories in parallel - https://phabricator.wikimedia.org/T211701 (10kostajh) @hashar thanks for adding this feature! Have you considered making the default number of workers 8/16/whatever? [14:00:38] 10Phabricator, 10MobileFrontend: Overlap between (Status, Priority, Story Points and Visible To) in phabricator mobile version - https://phabricator.wikimedia.org/T219503 (10alanajjar) [14:11:30] 10Release-Engineering-Team, 10MinervaNeue, 10Readers-Web-Backlog: MinervaNeue CI tests frequently fail on bad certificate or request timeout - https://phabricator.wikimedia.org/T219394 (10Niedzielski) 05Open→03Resolved a:03Niedzielski Time heals all wounds. [14:12:44] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10MediaWiki-General-or-Unknown, 10Patch-For-Review, 10User-zeljkofilipin: `npm install` fails for mediawiki/core with EPEERINVALID when running on Node 11 - https://phabricator.wikimedia.org/T210506 (10zeljkofilipin) There might be two... [14:22:55] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10MediaWiki-General-or-Unknown, 10Patch-For-Review, 10User-zeljkofilipin: `npm install` fails for mediawiki/core with EPEERINVALID when running on Node 11 - https://phabricator.wikimedia.org/T210506 (10hashar) Seems it is just that fib... [14:29:47] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10MediaWiki-General-or-Unknown, 10Patch-For-Review, 10User-zeljkofilipin: `npm install` fails for mediawiki/core with EPEERINVALID when running on Node 11 - https://phabricator.wikimedia.org/T210506 (10hashar) Also https://github.com/l... [14:30:53] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10MediaWiki-General-or-Unknown, 10Patch-For-Review, 10User-zeljkofilipin: `npm install` fails for mediawiki/core with EPEERINVALID when running on Node 11 - https://phabricator.wikimedia.org/T210506 (10zeljkofilipin) | mediawiki/core |... [14:52:52] 10Release-Engineering-Team, 10Developer Productivity, 10Epic: Add Windows installation Documentation and Install Script for local-charts repo - https://phabricator.wikimedia.org/T219438 (10brennen) > Could run into issue going down that route, as in order to run linux containers you'd need another VM? Yeah... [15:00:50] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Quibble (marble): Quibble should clone repositories in parallel - https://phabricator.wikimedia.org/T211701 (10hashar) It definitely should defaults to some higher number of workers. Exact value left to be defined. I have let it default to 1 for now... [15:02:13] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Quibble (marble): Quibble should clone repositories in parallel - https://phabricator.wikimedia.org/T211701 (10kostajh) > So I am playing it safe :] very sensible :) [15:03:48] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10Documentation: Improve documentation on Docker-based development environments for new developers - https://phabricator.wikimedia.org/T217614 (10brennen) @srodlund Thanks! We've let this drift a bit since I filed this task, but hopefully will g... [15:04:04] greg-g: & co, i cant find the call link D: [15:04:18] addshore: https://meet.google.com/smp-tgoh-uim?authuser=0 [15:04:21] :D [15:04:33] addshore: see also -pipeline, the official IRC channel :) [15:04:38] oh yes [15:05:06] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10Documentation: Improve documentation on Docker-based development environments for new developers - https://phabricator.wikimedia.org/T217614 (10egardner) Agreed – since we last chatted about this I've spent a lot of time in the MW Docker Dev en... [15:14:15] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10MediaWiki-General-or-Unknown, 10Patch-For-Review, 10User-zeljkofilipin: `npm install` fails for mediawiki/core with EPEERINVALID when running on Node 11 - https://phabricator.wikimedia.org/T210506 (10zeljkofilipin) Blocked by T215562... [15:14:26] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10MediaWiki-General-or-Unknown, 10Patch-For-Review, 10User-zeljkofilipin: `npm install` fails for mediawiki/core with EPEERINVALID when running on Node 11 - https://phabricator.wikimedia.org/T210506 (10zeljkofilipin) a:05zeljkofilipin... [15:14:39] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown, 10Patch-For-Review, 10User-zeljkofilipin: `npm install` fails for mediawiki/core with EPEERINVALID when running on Node 11 - https://phabricator.wikimedia.org/T210506 (10zeljkofilipin) [15:18:44] 10Continuous-Integration-Config, 10Patch-For-Review, 10Upstream, 10User-zeljkofilipin: npm 6 consistently fails with "Z_DATA_ERROR: invalid distance too far back" on some repos - https://phabricator.wikimedia.org/T215562 (10zeljkofilipin) [15:29:41] 10Release-Engineering-Team (Kanban), 10Code-Health-Metrics, 10Patch-For-Review, 10User-zeljkofilipin: Generate baseline analysis of all extensions - https://phabricator.wikimedia.org/T219156 (10zeljkofilipin) I have a fresh copy of all repos from Gerrit, using [[ https://github.com/zeljkofilipin/gerrit/blo... [15:52:46] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.33.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T206677 (10Daimona) [15:58:55] 10Phabricator: Overlap between (Status, Priority, Story Points and Visible To) in phabricator mobile version - https://phabricator.wikimedia.org/T219503 (10JJMC89) [16:06:49] 10Phabricator (Upstream), 10Upstream: Changing task title from empty to non-empty shows as "created this task" in history - https://phabricator.wikimedia.org/T209449 (10epriestley) The upstream expectation is that you can not remove a task title, and can not create a task with no title. You are always supposed... [16:10:30] PROBLEM - Host deployment-db03 is DOWN: CRITICAL - Host Unreachable (172.16.5.23) [16:36:25] 10Phabricator (Upstream), 10Upstream: Don't offer "Show Hidden Columns" when there are no hidden columns (Manage Board) - https://phabricator.wikimedia.org/T90779 (10epriestley) I think there are two possible changes we could make here based on the task title. It's not entirely clear to me which is being sugge... [16:40:03] wow addshore that's an amazing find (re: edit check on entities from foreign repos) [16:40:10] 10Phabricator (Upstream), 10Mobile, 10Upstream: Overlap between (Status, Priority, Story Points and Visible To) in phabricator mobile version - https://phabricator.wikimedia.org/T219503 (10Aklapper) Confirming, also happens in current upstream [16:40:33] apergos: meh, its just another stupid thing that is such an odd case, but also not [16:40:52] the stupid thing is, we never spotted it in development, as the default item namespace is 120, but in production on wikidata it is 0 [16:41:10] in testwikidata i's 120 still? [16:41:10] so it is not a case that came up all through development, only on beta, test and prod. Shame we didn't catch it in beta or test [16:41:18] yep [16:41:23] nope, testwikidata it is also 0 [16:41:25] I have 0 on my local instance [16:41:42] vagrant? ;) [16:41:43] because I was trying to mimic wd prod as much as possible [16:41:49] aaaah [16:41:49] nah right on the laptop :-) [16:41:56] for dumps :-D [16:42:02] :) [16:42:11] i clearly blame you for not spotting it then ;) [16:42:14] hehe [16:42:21] hey, I didn't :-D :-D [16:42:41] daily browser tests targeting beta could have caught this [16:42:45] or targetting test [16:42:50] both [16:43:00] yes beta s often in an unhappy state [16:43:09] maybe that would push priorites around having it be less unhappy [16:43:27] I enjoy our daily browser tests for wikidata, but it is a shame most of our browser tests are still in ruby, and those ones are not in a good state for beta [16:43:35] mmm gotcha [16:43:56] one of the big issues with wikidata, is nothing, at all, is like prod, not even test [16:43:58] hey did you want to comment on the incident report with any other action items or thoughts? [16:44:07] some of this seems good to capture there [16:44:17] Yes, I'll add some more comments there toward the end of ths day [16:44:19] Of course, browser tests flake twenty times for every three that they run. [16:44:22] yeah well apparently testcommons isn't so much like commons either [16:44:25] both bad news [16:44:34] (03PS3) 10Ejegg: DonationInterface tests run on PHP7 and MediaWiki 1.31 [integration/config] - 10https://gerrit.wikimedia.org/r/499334 [16:44:42] apergos: No, it's almost identical. The problem is no-one thought to test editing NS0. [16:44:45] its close enough for the really big things [16:44:54] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10Epic, 10Patch-For-Review, 10User-zeljkofilipin: Add MacOS installation Documentation and Install Script for local-charts repo - https://phabricator.wikimedia.org/T219437 (10zeljkofilipin) [16:44:57] well I'm told that file display isn't working [16:45:02] we can see if turning it on literally makes everything explode :D [16:45:04] ask yannf about it [16:45:15] Oh, yeah, obviously I wasn't going to fuck around with Swift config for a test wiki. [16:45:20] perhaps it was never hooked up to some of the services like swift etc? [16:45:23] :P [16:45:34] Swift is dark and full of terrors. [16:45:40] giving it some backing store seems kind of needed [16:45:56] Why? The binary storage part is the one thing we're definitely not touching. :-) [16:45:58] example: in beta I do not have a dumps nfs server on which to write files [16:46:06] I do have a local flesystem with the right path and etc [16:46:17] 10Phabricator (Upstream), 10Upstream: Actions not showing in chronological order in "grouped" task actions - https://phabricator.wikimedia.org/T88186 (10epriestley) In the general case, this is expected behavior. Consider these actions: - Change title. - Add a comment. - Close task. We reorder these ac... [16:46:19] so just have... something [16:46:24] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10Patch-For-Review, 10User-zeljkofilipin: Upgrade webdriverio to version 5 - https://phabricator.wikimedia.org/T213268 (10zeljkofilipin) a:03zeljkofilipin [16:46:54] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10Patch-For-Review, 10User-zeljkofilipin: Upgrade webdriverio to version 5 - https://phabricator.wikimedia.org/T213268 (10zeljkofilipin) p:05Triage→03Normal [16:47:19] anyways who would expect it to break editing in some unrelated namespace [16:47:31] but that's the thing about comprehensive tests, they test the crap you think is unrelated [16:47:36] Well, indeed. :-( But still, my fault. [16:47:45] well fault schmault [16:47:47] MW doesn't have any wikitext editing integration tests. [16:47:56] it's all about making things better for the next time [16:48:16] I looked at it for a few hours last year and decided it was too hard to get it working in a state that people would merge. [16:48:19] * James_F nods. [16:48:27] some stuff is hard [16:48:42] Yeah, and MW unit tests are very hard. [16:48:45] I got bit on that earlier this week (something not testable, code merged in core -> broke the thing) [16:49:06] Well, the unit tests are easy, the integration tests that run are not. [16:49:42] right [16:52:02] We have unit tests for edit /stashing/ (which is new), but not editing. [16:52:18] Similarly, we have unit tests for API-based uploads, which is new-ish, but not regular ones. [16:52:37] little by little [16:53:32] fwiw I think that the mcr+sdc+wikibase stuff is very complex and these tests, should they or even subsets of them get written, are going to save us many times over [16:57:23] hi rel-eng folks! [16:57:53] Anyone want to take a look at this patch to update donationInterface tests to use the most recent LTS version of Mediawiki? [16:57:56] # Quibble jobs should not be run on fundraising/REL1_31. [16:57:56] derp [16:58:00] https://gerrit.wikimedia.org/r/499334 [16:58:22] should also update it to use PHP7.0 [16:58:30] to match the current payments-wiki environment [16:59:00] ejegg: You're no longer running 1.27 anywhere? [16:59:12] James_F: just upgraded on Monday [16:59:15] Nice! [16:59:18] well, upgraded one cluster [16:59:20] 70 not 72? :-( [16:59:25] (03PS1) 10Daimona Eaytoy: Enable Phan for ProofreadPage [integration/config] - 10https://gerrit.wikimedia.org/r/499819 [17:00:27] James_F: yeah, looks like debian stretch is sticking with 7.0 [17:00:33] then buster jumps right to 7.3 [17:00:53] 7.0 went EOL in Jan 2019. We're moving production to 7.2. [17:01:33] James_F: in-house packaged version? [17:01:36] Yes. [17:01:50] "We" == Service Ops. [17:01:57] 10Continuous-Integration-Config, 10Operations, 10Patch-For-Review, 10Upstream, 10User-zeljkofilipin: npm 6 consistently fails with "Z_DATA_ERROR: invalid distance too far back" on some repos - https://phabricator.wikimedia.org/T215562 (10Krinkle) a:03MoritzMuehlenhoff [17:01:59] * apergos peeks in [17:02:06] I'll see if the fr-tech-ops folks would be interested in that [17:02:14] but for now we're running 7.0 [17:03:10] James_F: that's for stretch, right? [17:04:05] (03CR) 10Jforrester: [C: 03+1] DonationInterface tests run on PHP7 and MediaWiki 1.31 [integration/config] - 10https://gerrit.wikimedia.org/r/499334 (owner: 10Ejegg) [17:04:14] thanks James_F [17:04:15] ejegg: Yes, stretch+7.2 [17:04:29] cool cool, will relay the info to Jeff_Green [17:16:13] 10Continuous-Integration-Config, 10Operations, 10Patch-For-Review, 10User-zeljkofilipin: npm 6 consistently fails with "Z_DATA_ERROR: invalid distance too far back" on some repos - https://phabricator.wikimedia.org/T215562 (10Krinkle) It seems we've found the culprit. The problem is indeed the zlib1g libra... [17:16:16] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10Epic, 10Patch-For-Review, 10User-zeljkofilipin: Add MacOS installation Documentation and Install Script for local-charts repo - https://phabricator.wikimedia.org/T219437 (10zeljkofilipin) Why is this tagged #epic? Copied from parent task? [17:18:33] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10Patch-For-Review, 10User-zeljkofilipin: Add MacOS installation Documentation and Install Script for local-charts repo - https://phabricator.wikimedia.org/T219437 (10jeena) Probably...I just made a subtask and didn't realize that would happen.... [17:19:05] 10Release-Engineering-Team, 10Developer Productivity: Add Windows installation Documentation and Install Script for local-charts repo - https://phabricator.wikimedia.org/T219438 (10jeena) [17:19:08] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10Patch-For-Review, 10User-zeljkofilipin: Add MacOS installation Documentation and Install Script for local-charts repo - https://phabricator.wikimedia.org/T219437 (10jeena) [17:19:50] 10Release-Engineering-Team, 10Developer Productivity: Add Windows installation Documentation and Install Script for local-charts repo - https://phabricator.wikimedia.org/T219438 (10jeena) I might ask my brother if I can use his windows machine to try it out. [17:20:30] 10Beta-Cluster-Infrastructure, 10User-DannyS712: Beta Cluster: Rights Request - https://phabricator.wikimedia.org/T219475 (10Krenair) 05Open→03Resolved a:03Krenair [17:20:48] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10local-charts, 10Patch-For-Review: Automate LocalSettings.php creation for local-charts - https://phabricator.wikimedia.org/T217869 (10jeena) [17:21:17] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10Patch-For-Review, 10User-zeljkofilipin: Add MacOS installation Documentation and Install Script for local-charts repo - https://phabricator.wikimedia.org/T219437 (10greg) >>! In T219437#5066723, @jeena wrote: > Probably...I just made a subtas... [17:22:46] 10Beta-Cluster-Infrastructure, 10User-DannyS712: Requesting Pending Changes Reviewer on enwiki beta - https://phabricator.wikimedia.org/T188873 (10Krenair) 05Open→03Resolved a:03DannyS712 removed protection [17:23:56] !log deployment-prep T219087 beginning master switch [17:24:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:24:04] T219087: Get rid of deployment-db0[34] - https://phabricator.wikimedia.org/T219087 [17:30:33] marxarelli: FYI, new train blocker. :-( [17:30:38] Filing now. [17:31:04] k. thanks [17:31:20] hm, this isn't working :/ [17:32:18] ok [17:33:57] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search, 10MediaWiki-Search, and 2 others: Browsing to Special:Search is broken on wmf.23, redirects to Special:Search&ns0=1 not Special:Search?ns0=1 - https://phabricator.wikimedia.org/T219539 (10Jdforrester-WMF) p:05Triage→03Unbreak! [17:34:18] I think that did it [17:34:29] now I just have to update MW config [17:36:26] !log forking plugins/quota from upstream [17:36:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:44:53] why is MW not using the new master [17:45:29] 10Release-Engineering-Team, 10MinervaNeue, 10Readers-Web-Backlog: MinervaNeue CI tests frequently fail on bad certificate or request timeout - https://phabricator.wikimedia.org/T219394 (10Jdlrobson) [17:48:00] it still thinks db04 is the master [17:48:03] why [17:54:18] ah because it has to be first on the list [17:54:23] perhaps [17:55:38] Project beta-scap-eqiad build #243109: 04FAILURE in 4 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/243109/ [17:55:51] ^ me [18:01:39] no its still doing it [18:02:12] could it be because of opcache? Dosen't php need restarting? [18:04:13] I tried restarting php7.2-fpm and apache [18:04:14] no luck [18:06:49] Yippee, build fixed! [18:06:49] Project beta-scap-eqiad build #243110: 09FIXED in 9 min 49 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/243110/ [18:14:31] (03CR) 10Thcipriani: [C: 03+2] Switch change-propagation to the pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/496387 (https://phabricator.wikimedia.org/T213193) (owner: 10Alexandros Kosiaris) [18:15:08] why does it think the host is localhost? [18:16:19] (03Merged) 10jenkins-bot: Switch change-propagation to the pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/496387 (https://phabricator.wikimedia.org/T213193) (owner: 10Alexandros Kosiaris) [18:18:29] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.33.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T206677 (10Jdforrester-WMF) [18:18:40] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.33.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T206677 (10Jdforrester-WMF) [18:19:10] something is not right with php [18:19:13] marxarelli: My train blockers now fixed and deployed. There is still T219514 open though. [18:19:16] T219514: Variables old_wikitext and new_wikitext are blank in Page namespace - https://phabricator.wikimedia.org/T219514 [18:19:24] it thinks our wgLBFactoryConf is class LBFactorySimple [18:19:43] and it looks like it has default settings for stuff, wtf [18:20:19] Project beta-update-databases-eqiad build #32758: 04FAILURE in 18 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32758/ [18:20:56] !log reload zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/496387/ [18:20:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:21:03] I'm not sure I trust mwrepl on deploy01 [18:22:22] based on mwscript eval.php they know the master is supposed to be -db05 [18:22:30] James_F: k [18:22:34] and yet they persist with trying to talk to the old master [18:22:54] marxarelli: and james is about to deploy a backport fixing that one too [18:23:05] was just going to ask [18:23:07] great! [18:23:20] looks like the backport failed some tests https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ProofreadPage/+/499801 [18:23:24] :/ [18:23:38] marxarelli: Yeah, I'm force-merging to deploy and proper-merging the test fix. [18:23:47] ah, just merged [18:23:58] James_F: great! thank you [18:26:22] interesting, if I try to open connection to the proper master I get unknown error [18:26:26] helpful [18:30:05] maybe I'm doing it wrong [18:30:07] but still [18:32:45] weird [18:32:51] removing the old master has done the trick [18:33:25] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.33.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T206677 (10Jdforrester-WMF) [18:36:45] marxarelli: And that's all blockers marked fixed (except one that was fixed before the train was cut but is still being worked on, or something). [18:37:49] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown, 10Patch-For-Review, 10User-zeljkofilipin: `npm install` fails for mediawiki/core with EPEERINVALID when running on Node 11 - https://phabricator.wikimedia.org/T210506 (10zeljkofilipin) Node 6 end of life is next month ([[ https://github.com/n... [18:37:53] James_F: you're amazing :) thank you! [18:39:36] Happy to help. [18:49:17] !log shut off deployment-db04 instance per T219087 [18:49:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:49:20] T219087: Get rid of deployment-db0[34] - https://phabricator.wikimedia.org/T219087 [18:49:53] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Get rid of deployment-db0[34] - https://phabricator.wikimedia.org/T219087 (10Krenair) a:03Krenair Just got to wait for deletion time now. Will probably give it a couple of weeks. [18:53:11] PROBLEM - Host deployment-db04 is DOWN: CRITICAL - Host Unreachable (172.16.5.5) [18:55:43] 10Project-Admins: Rename #Wikimedia-production-error tag or "Report Application Error" form to something less generic? - https://phabricator.wikimedia.org/T216795 (10Krinkle) I've updated the appearance of the "Application Error" form. | Before| After |--|-- | {F28500060} | {F28500010} Main changes: * It prom... [19:10:46] Hmm, https://gerrit.wikimedia.org/r/q/(branch:wmf%252F1.33.0-wmf.21+OR+branch:wmf%252F1.33.0-wmf.22+OR+branch:wmf%252F1.33.0-wmf.23)+(reviewedby:jforrester+OR+owner:jforrester) is quite long, I suppose. [19:21:58] Yippee, build fixed! [19:21:59] Project beta-update-databases-eqiad build #32759: 09FIXED in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32759/ [19:32:06] * paladox https://phabricator.wikimedia.org/T219300 is finally done (added ssh support/docs), will merge later today! [19:40:28] (03Abandoned) 10Paladox: Fix blocking users [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/497429 (owner: 10Paladox) [19:42:48] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.33.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T206677 (10dduvall) 05Open→03Resolved [20:16:13] (03Abandoned) 10Paladox: Modify access rules [All-Users] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/499258 (owner: 10Paladox) [20:31:40] 10Phabricator, 10Multimedia: Notify multimedia-team@lists.wikimedia.org when a task is UBN and tagged with Multimedia - https://phabricator.wikimedia.org/T219553 (10MarkTraceur) [20:39:01] 10Phabricator, 10Multimedia: Notify multimedia-team@lists.wikimedia.org when a task is UBN and tagged with Multimedia - https://phabricator.wikimedia.org/T219553 (10greg) It's not letting me add the mailing list.... I paste in multimedia-team@lists.wikimedia.org and then it disappears when I click away. It sa... [20:45:31] 10Phabricator, 10Multimedia: Notify multimedia-team@lists.wikimedia.org when a task is UBN and tagged with Multimedia - https://phabricator.wikimedia.org/T219553 (10MarkTraceur) Boy, we should really clean up that project's membership. I'd be fine putting in the current members of the team, but it'd be Yet An... [20:59:56] 10Phabricator, 10Multimedia: Notify multimedia-team@lists.wikimedia.org when a task is UBN and tagged with Multimedia - https://phabricator.wikimedia.org/T219553 (10Aklapper) >>! In T219553#5067591, @greg wrote: > It's not letting me add the mailing list I've created a `multimedia-team-list` mailing list user... [21:03:06] 10Phabricator, 10Multimedia: Notify multimedia-team@lists.wikimedia.org when a task is UBN and tagged with Multimedia - https://phabricator.wikimedia.org/T219553 (10Aklapper) 05Open→03Resolved a:03Aklapper H315 created [21:03:50] (03PS2) 10Krinkle: zuul: Try to convert a 'branch' filter to 'skip-if' [integration/config] - 10https://gerrit.wikimedia.org/r/499682 [21:03:53] (03PS3) 10Krinkle: zuul: Try to convert a 'branch' filter to 'skip-if' [integration/config] - 10https://gerrit.wikimedia.org/r/499682 [21:05:49] (03CR) 10jerkins-bot: [V: 04-1] zuul: Try to convert a 'branch' filter to 'skip-if' [integration/config] - 10https://gerrit.wikimedia.org/r/499682 (owner: 10Krinkle) [21:09:45] 10Phabricator, 10Multimedia: Notify multimedia-team@lists.wikimedia.org when a task is UBN and tagged with Multimedia - https://phabricator.wikimedia.org/T219553 (10greg) >>! In T219553#5067698, @Aklapper wrote: >>>! In T219553#5067591, @greg wrote: >> It's not letting me add the mailing list > > I've created... [21:14:45] (03PS4) 10Krinkle: zuul: Try to convert a 'branch' filter to 'skip-if' [integration/config] - 10https://gerrit.wikimedia.org/r/499682 [21:16:12] (03CR) 10jerkins-bot: [V: 04-1] zuul: Try to convert a 'branch' filter to 'skip-if' [integration/config] - 10https://gerrit.wikimedia.org/r/499682 (owner: 10Krinkle) [21:21:46] My gerrit dashboard isn't loading (but other people's seem to be): https://gerrit.wikimedia.org/r/#/q/owner:thalia.e.chan%2540googlemail.com+status:open - any idea what might be up? [21:23:27] just hangs on "Working ..." when I click that link, but my dashboard is working fine. very odd :/ [21:27:18] hmm [21:27:27] gerrit is not loading for me [21:29:57] Agreed. It was working for me when Tchanders first reported but it's failing now. [21:31:05] Looks like they're discussing it in #wikimedia-operations [21:31:33] (03PS1) 10Dduvall: WIP Allow configuration of pipeline [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/499918 [21:41:37] PROBLEM - puppet last run on contint2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_jenkins CI slave scripts] [21:43:03] ACKNOWLEDGEMENT - puppet last run on contint2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_jenkins CI slave scripts] cole_white gerrit fallout [21:52:49] Project beta-code-update-eqiad build #240596: 04FAILURE in 17 min: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/240596/ [21:54:08] Project beta-code-update-eqiad build #240597: 04STILL FAILING in 1 min 18 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/240597/ [21:54:09] Tchanders and edsanders it should be fixed now [21:54:13] at least works for me. [21:55:34] Yippee, build fixed! [21:55:35] Project beta-code-update-eqiad build #240598: 09FIXED in 1 min 26 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/240598/ [21:55:54] Yes, me too [22:02:43] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [22:03:50] PROBLEM - Puppet errors on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [3.0] [22:29:46] 10Project-Admins: Rename #Wikimedia-production-error tag or "Report Application Error" form to something less generic? - https://phabricator.wikimedia.org/T216795 (10Jdforrester-WMF) Looks good! [23:53:51] RECOVERY - Puppet errors on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [2.0]