[00:51:34] 10LibUp: Update .gitattributes to export-ignore all dot files/folders - https://phabricator.wikimedia.org/T274613 (10Reedy) [03:34:29] (03CR) 10Krinkle: [C: 03+2] Update URL of VisualEditor demo [integration/docroot] - 10https://gerrit.wikimedia.org/r/663665 (https://phabricator.wikimedia.org/T274222) (owner: 10Bartosz Dziewoński) [03:35:28] (03Merged) 10jenkins-bot: Update URL of VisualEditor demo [integration/docroot] - 10https://gerrit.wikimedia.org/r/663665 (https://phabricator.wikimedia.org/T274222) (owner: 10Bartosz Dziewoński) [04:39:52] 10Beta-Cluster-Infrastructure: Several import sources are gone on zh beta cluster - https://phabricator.wikimedia.org/T274620 (10WhitePhosphorus) [07:24:08] hey Amir1 I had just gotten to the same conclusion about the rdbms patch most likely to be responsible [07:24:16] for the new trainblocker, that is [07:24:44] what do you think about the best way forward? we can't really revert it because of the wanobjectcache changes that rely on it [07:24:57] ugh [07:25:11] I don't have proof of course, just an intution [07:25:30] can we revert all of them? [07:25:37] uh. [07:27:08] I think it means reverting this config change: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/658372/ but also the code changes that made that possible, and probably anything applied on top of them [07:27:10] I was thinking of reverting them all on wmf.30 only [07:27:21] this is messy enough to make me uncomfortable [07:27:22] and if situation improves, reverting them all on master [07:27:32] but let's make a list [07:28:55] I'm also not sure what having the key format suddenly change, is going to do with respect to operations that will go looking for an existing key and it won't be there because they ask for a key usign the wrong format, after the revert [07:46:39] I think after looking at stuff a bit, it's not as bad as I feared: the one config change and the rbms patch might be the sole things that need to be rolled back. still looknig though [07:50:11] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/652572/7/includes/libs/objectcache/wancache/WANObjectCache.php this one might have to go back in addition [07:51:24] it depends, if that's not used when 'coalesceKeys' is non_global, then maybe we are ok [07:54:04] I'm slightly confused, how WAN changes are related to LB changes? they are (at least on paper) completely separate components of mw [07:57:14] https://phabricator.wikimedia.org/T252564 this is the connection (pun intended) [07:57:45] see here https://phabricator.wikimedia.org/T252564#6212081 and further [08:00:46] ugh [08:08:57] but it might be just the config chang that has to go back, along with the one rdbms patch and done [08:09:13] I would love someone who understands the code better than me to look at it though [08:15:26] Project mediawiki-core-doxygen-docker build #22584: 04FAILURE in 10 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/22584/ [08:16:21] hmm, https://phabricator.wikimedia.org/T252564#6805689 it means wmf.27 have it too [08:21:27] yes it does, several months went by before the underlying issue was found and resolved [08:23:07] I have pinged daniel in our channel via element, perhaps he can weigh in when he is available (other people are in a bad tz right now), or perhaps you know enough about this code to be sure if the reverts are ok (I don't, sadly) [08:24:33] (03PS1) 10Hashar: jjb: update mediawiki jobs to Quibble 0.0.46 [integration/config] - 10https://gerrit.wikimedia.org/r/663782 (https://phabricator.wikimedia.org/T274590) [08:24:58] I don't know the codebase well enough to help :( [08:27:09] ok. we shall see what daniel says, or anyone else in that channel perhaps [08:27:39] and I have posted the equivalent on the task (a sort of plea for help) [08:29:27] worst case aaron sees it later and can do-the-needful [08:32:38] Amir1: apergos: in my experience, the memcached / wan object cache / multidatacenter stuff is all for Aaron and T.imo [08:32:54] I have already added him on the task [08:32:58] and krinkle is already on it [08:33:22] and touching it by trial and errors would surely lead to smoe kind of complicated cache corruption, perf regression or plain good outage (FUD dont quote me I dont know really) [08:33:55] that is the area of mediawiki I am happy to claim my total utter incompetency and 100% blindly rely on the people who knows [08:33:58] heh [08:34:37] my expectation is that Aaron will work on it later this evening (his morning) [08:34:59] and we should get some patch / analysis / proposal which we can land monday evening I guess [08:35:36] or maybe disabling the config change is enough and that can potentially be done today [08:36:38] we'll see. I can't do much now but wait [09:00:04] Project mwcore-phpunit-coverage-master build #1212: 04STILL FAILING in 6 hr 0 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1212/ [09:16:47] 10Release-Engineering-Team (Local Dev), 10dev-images, 10docker-pkg, 10serviceops, and 2 others: docker-pkg: "certificate verify failed: unable to get local issuer certificate" for docker-registry.discovery.wmnet when publishing dev-images from contint2001 - https://phabricator.wikimedia.org/T274306 (10JMeyb... [09:18:10] Yippee, build fixed! [09:18:11] Project mediawiki-core-doxygen-docker build #22585: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/22585/ [09:21:20] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)): Regenerate a gpg key for Antoine and add to releng-secrets - https://phabricator.wikimedia.org/T274277 (10hashar) I can't edit though apparently cause one of the key is expired. But that can be solved via another venue. Will look at acquiring some... [11:23:14] why has a beta-mediawiki-config-update-eqiad job been queued for over ten hours now? [11:54:52] beta cluster ci jobs are stuck and might need to be touched to get them running again [12:05:27] looking [12:05:57] !log canceled one beta-scap-eqiad job per https://w.wiki/J5$ [12:05:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:06:34] that should take care of the issue, hopefully [12:19:37] (03PS2) 10Lars Wirzenius: make tests fail under Python3 to verify CI catches it [tools/scap] - 10https://gerrit.wikimedia.org/r/663617 [12:25:44] (03CR) 10jerkins-bot: [V: 04-1] make tests fail under Python3 to verify CI catches it [tools/scap] - 10https://gerrit.wikimedia.org/r/663617 (owner: 10Lars Wirzenius) [12:48:12] (03PS3) 10Lars Wirzenius: make tests fail under Python3 to verify CI catches it [tools/scap] - 10https://gerrit.wikimedia.org/r/663617 [12:50:41] (03CR) 10jerkins-bot: [V: 04-1] make tests fail under Python3 to verify CI catches it [tools/scap] - 10https://gerrit.wikimedia.org/r/663617 (owner: 10Lars Wirzenius) [12:53:06] (03PS4) 10Lars Wirzenius: make tests fail under Python3 to verify CI catches it [tools/scap] - 10https://gerrit.wikimedia.org/r/663617 [12:55:28] (03CR) 10jerkins-bot: [V: 04-1] make tests fail under Python3 to verify CI catches it [tools/scap] - 10https://gerrit.wikimedia.org/r/663617 (owner: 10Lars Wirzenius) [12:57:55] (03PS5) 10Lars Wirzenius: make tests fail under Python3 to verify CI catches it [tools/scap] - 10https://gerrit.wikimedia.org/r/663617 [13:00:25] (03CR) 10jerkins-bot: [V: 04-1] make tests fail under Python3 to verify CI catches it [tools/scap] - 10https://gerrit.wikimedia.org/r/663617 (owner: 10Lars Wirzenius) [13:02:00] (03PS6) 10Lars Wirzenius: make tests fail under Python3 to verify CI catches it [tools/scap] - 10https://gerrit.wikimedia.org/r/663617 [13:03:23] (03CR) 10jerkins-bot: [V: 04-1] make tests fail under Python3 to verify CI catches it [tools/scap] - 10https://gerrit.wikimedia.org/r/663617 (owner: 10Lars Wirzenius) [13:05:58] (03PS7) 10Lars Wirzenius: make tests fail under Python3 to verify CI catches it [tools/scap] - 10https://gerrit.wikimedia.org/r/663617 [13:06:20] (03CR) 10jerkins-bot: [V: 04-1] make tests fail under Python3 to verify CI catches it [tools/scap] - 10https://gerrit.wikimedia.org/r/663617 (owner: 10Lars Wirzenius) [13:12:24] (03Abandoned) 10Lars Wirzenius: make tests fail under Python3 to verify CI catches it [tools/scap] - 10https://gerrit.wikimedia.org/r/663617 (owner: 10Lars Wirzenius) [13:14:13] Majavah: that beta job is bugged :/ [13:14:33] Majavah: but eventually that get resolved [13:15:18] liw: hello :) I looked at pgp-public-keys script to generate the missing sigs. And some how I do not show up in the list :-\ [13:15:37] and thanks for the makefile merge! [13:16:01] oh [13:16:11] and I did not show up cause antoine.asc in my copy was outdated / expired [13:16:15] so it got skipped [13:16:17] that works now! [14:28:41] (03CR) 10Hashar: ci-common: Add a bypass for the ci-src-setup script (034 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/663202 (https://phabricator.wikimedia.org/T274347) (owner: 10David Caro) [14:29:06] (03PS4) 10Hashar: ci-common: Add a bypass for the ci-src-setup script [integration/config] - 10https://gerrit.wikimedia.org/r/663202 (https://phabricator.wikimedia.org/T274347) (owner: 10David Caro) [14:31:14] (03CR) 10David Caro: [C: 03+1] ci-common: Add a bypass for the ci-src-setup script [integration/config] - 10https://gerrit.wikimedia.org/r/663202 (https://phabricator.wikimedia.org/T274347) (owner: 10David Caro) [14:34:06] hashar: which caching task? I don't see it on our team board and don't see it stand out in personal notifications [14:56:04] Krinkle: good morning! That is was apergos and Amir1 discussion early this morning [14:56:33] it's yesterday's and still today's ubn [14:56:34] it's the train blocker [14:56:38] for the train blocker https://phabricator.wikimedia.org/T274589 [14:57:17] you might also know the code well enough to make the call, I didn't and daniel doesn't know that piece of it either [14:57:23] there is some indication of the cause being a patch T252564 [14:57:23] T252564: Let WANObjectCache store "sister keys" on the same backend as the main value key - https://phabricator.wikimedia.org/T252564 [14:57:28] Okay. That one is not memc/wan cache related though. [14:57:34] Right that one is [14:57:36] well that's not the cause [14:57:40] and that WANObjectCache sounds like something for Aaron or Timo :] [14:57:44] but it exposed a bug that required the rdbms lfix [14:58:27] right so we're dealing with an rdbms bug that maybe been introduced to fix a caching issue [14:58:29] and if the fix goes away then we'll have the previous bug as described in the bug linked in my comment https://phabricator.wikimedia.org/T274589#6825678 [14:58:40] unless we at minimum revert the config change... [14:59:01] and that's where I stop and go "uh. side effects? and what else might be impacted?" and I look around for someone who knows better [14:59:06] * apergos shuts up now [15:00:01] What is our hypothesis for t rdbms bug? Are comment store insert queries generally leading to connection loss? [15:00:21] I don't see the link between the task and the waitFor bug fix [15:01:06] (03PS1) 10Lars Wirzenius: make tests fail under Python3 to verify CI catches it [tools/scap] - 10https://gerrit.wikimedia.org/r/663839 [15:03:00] the hypothesis as I understand it is: it was probably introduced in wmf.28; it is very likely not randomly connections being dropped but something in a code path; all jobs of a certain type fail consistently. and as Amir says, The connection is gone error always show up in the select statement of CommentStore->insert() nowhere else. [15:03:25] so then it was a matter of scrying through rdbms changes in wmf.28 to see what might be likely, as a candidate for revert [15:03:35] but do I know this to be at fault? nope. it is a guess. [15:04:08] there were not many of these errors in the last 12 hours (well last time I looked - 12 hours, which was a few hours ago), so that's not so helpful either [15:04:15] like, 20. [15:04:23] Okay I see. Might be good to also look for commits in comment store or upload or job handling. [15:04:46] that's a good point [15:04:46] But I see how you get to this commit now and I can't rule it out indeed [15:05:04] (03CR) 10jerkins-bot: [V: 04-1] make tests fail under Python3 to verify CI catches it [tools/scap] - 10https://gerrit.wikimedia.org/r/663839 (owner: 10Lars Wirzenius) [15:05:16] both Amir and I came to it through a similar process I guess, since when I went to post about it on the task I saw he had already done so :-D [15:05:43] The "MySQL gone away" though comes from native, so there isn't something we can do wrong to make that appear falsely, eg it is not a string we generate afaik [15:05:55] But it's possible we're inducing it somehow [15:06:30] We've seen this for years at a low rate usually on slow queries [15:06:56] But it's possible we're eg waiting forever for replication until the connection times out [15:07:14] I'll need Aaron to debug that. The commit is too messy with unrelated changes. [15:07:55] I added him to the task, though I don't know if he gets timely notifications from that [15:08:16] he and I don't overlap much tz wise [15:08:39] Thanks Amir1 for confirming it is a regression and a serious one affecting all publish jobs. I would have discarded it as low rate connection issue [15:09:32] We should rollback commons while we fix this [15:09:50] Since were presumably breaking stashed uploads now [15:10:09] twentyafterfour: [15:11:48] yes, One thing is that I was looking into is that might be exactly the query before insert comment (somewhere else) might be closing the connection [15:15:49] (03PS2) 10Lars Wirzenius: make tests fail under Python3 to verify CI catches it [tools/scap] - 10https://gerrit.wikimedia.org/r/663839 [15:15:59] I have looked at the list of commits for wmf.28 from here https://www.mediawiki.org/wiki/MediaWiki_1.36/wmf.28 and not found any likely suspects for job/upload/comment [15:17:14] if that is not the best place to look, please let me know [15:20:15] (03CR) 10jerkins-bot: [V: 04-1] make tests fail under Python3 to verify CI catches it [tools/scap] - 10https://gerrit.wikimedia.org/r/663839 (owner: 10Lars Wirzenius) [15:26:30] (03PS3) 10Lars Wirzenius: make tests fail under Python3 to verify CI catches it [tools/scap] - 10https://gerrit.wikimedia.org/r/663839 [15:26:52] (03CR) 10jerkins-bot: [V: 04-1] make tests fail under Python3 to verify CI catches it [tools/scap] - 10https://gerrit.wikimedia.org/r/663839 (owner: 10Lars Wirzenius) [15:35:34] (03CR) 10Hashar: "I had some chat with David about it. I would like to get set -u back again , cause I suspect we might well have some job invoking the scr" [integration/config] - 10https://gerrit.wikimedia.org/r/663202 (https://phabricator.wikimedia.org/T274347) (owner: 10David Caro) [15:36:55] (03CR) 10Hashar: [C: 04-1] ci-common: Add a bypass for the ci-src-setup script [integration/config] - 10https://gerrit.wikimedia.org/r/663202 (https://phabricator.wikimedia.org/T274347) (owner: 10David Caro) [15:37:19] apergos: yup, that's a great way to do it [15:37:36] ok cool [15:43:12] (03PS4) 10Lars Wirzenius: make tests fail under Python3 to verify CI catches it [tools/scap] - 10https://gerrit.wikimedia.org/r/663839 [15:45:08] 10Scap, 10Release Pipeline (Blubber): Blubber, pip, and python3.5 - https://phabricator.wikimedia.org/T274660 (10LarsWirzenius) [15:47:10] (03CR) 10jerkins-bot: [V: 04-1] make tests fail under Python3 to verify CI catches it [tools/scap] - 10https://gerrit.wikimedia.org/r/663839 (owner: 10Lars Wirzenius) [16:01:18] 10Gerrit, 10observability: Enhance Gerrit logs ingested by Logstash - https://phabricator.wikimedia.org/T274661 (10hashar) [16:03:23] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Wikimedia-Logstash, and 2 others: Look into shoving gerrit logs into logstash - https://phabricator.wikimedia.org/T141324 (10hashar) 05Open→03Resolved a:03hashar This was do... [16:56:46] 10Gerrit, 10observability: Enhance Gerrit logs ingested by Logstash - https://phabricator.wikimedia.org/T274661 (10hashar) It has been suggested to use ECS. We run Gerrit 3.2 and 3.3 will change the timestamp format. From http://gerrit-documentation.storage.googleapis.com/Documentation/3.3.1/logs.html: > For... [16:57:57] Krinkle: where's the best place to reach AaronSchulz to see if he can look at this issue? [16:58:06] if you know [16:59:44] -perf I suppose [17:01:01] ah does he watch irc? excellent [17:01:37] although if he watches irc, then he's watching here too :-) [17:19:06] !log Publishing from dev-images docker-pkg files on primary contint for fr-tech images [17:19:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:22:08] 10Release-Engineering-Team (Local Dev), 10dev-images, 10docker-pkg, 10serviceops, 10User-brennen: docker-pkg: "certificate verify failed: unable to get local issuer certificate" for docker-registry.discovery.wmnet when publishing dev-images from contint2001 - https://phabricator.wikimedia.org/T274306 (10b... [17:36:11] 10Project-Admins, 10Analytics, 10Analytics-Visualization: Archive #Analytics-Visualization (which seems to be about Limn)? - https://phabricator.wikimedia.org/T274647 (10DannyS712) [18:22:53] (03CR) 10Dduvall: python: upgrade pip before installing requirements (031 comment) [blubber] - 10https://gerrit.wikimedia.org/r/663348 (https://phabricator.wikimedia.org/T274435) (owner: 10BryanDavis) [18:33:26] 10Project-Admins: Create three Phab Projects for Machine Learning: Lift Wing, Pilot Flag, Test Grounds - https://phabricator.wikimedia.org/T264774 (10calbon) None yet, but soon [19:05:29] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [19:06:21] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [19:23:30] 10LibUp: libup does not remove php5 and inc file extensions from some .phpcs.xml files - https://phabricator.wikimedia.org/T273996 (10Umherirrender) 05Open→03Resolved a:03Umherirrender [19:24:23] 10LibUp: libup fail to run eslint --fix on repos which does not config eslint in existing Gruntfile.js - https://phabricator.wikimedia.org/T273993 (10Umherirrender) 05Open→03Resolved a:03Legoktm [19:44:43] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T271344 (10mmodell) [19:47:57] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T271344 (10mmodell) [19:48:58] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T271344 (10mmodell) [19:52:25] 10Gerrit, 10observability: Enhance Gerrit logs ingested by Logstash - https://phabricator.wikimedia.org/T274661 (10colewhite) a:03colewhite ECS migration task: T234565 [19:52:43] 10Gerrit, 10observability: Enhance Gerrit logs ingested by Logstash - https://phabricator.wikimedia.org/T274661 (10colewhite) p:05Triage→03Medium [20:08:54] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1353.eqiad.wmnet'] ` an... [20:09:40] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1358.eqiad.wmnet'] ` an... [20:31:16] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:31:24] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [21:00:05] Project mwcore-phpunit-coverage-master build #1213: 04STILL FAILING in 6 hr 0 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1213/ [21:03:16] 10LibUp: libup gets confused when npm "dependencies" have security issues (as opposed to "devDependencies") - https://phabricator.wikimedia.org/T263712 (10Umherirrender) Seems to affect also some other repos: https://gerrit.wikimedia.org/r/c/mediawiki/skins/MinervaNeue/+/663719 Json from npm audit is: ` { "... [21:08:16] 10Beta-Cluster-Infrastructure, 10Wikimedia-Logstash, 10observability, 10User-DannyS712: Logstash beta is not getting any events - https://phabricator.wikimedia.org/T274593 (10colewhite) 05Open→03Resolved a:03colewhite Restarted logstash on deployment-logstash03 and logs appear to be flowing again. [21:08:18] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.31 deployment blockers - https://phabricator.wikimedia.org/T271345 (10colewhite) [21:09:38] 10LibUp: libup fail to run eslint --fix on repos which does not config eslint in existing Gruntfile.js - https://phabricator.wikimedia.org/T273993 (10Umherirrender) ORES now has a patch - https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ORES/+/663886 [21:12:36] 10LibUp, 10FR-Smashpig, 10Fundraising-Backlog, 10Fundraising Sprint Corrugated super slide, 10MW-1.36-notes (1.36.0-wmf.26; 2021-01-12): LibUp times out trying to update mediawiki/extensions/DonationInterface because of unsatisfiable dependencies - https://phabricator.wikimedia.org/T271180 (10Umherirrende... [21:14:31] 10LibUp: libup does not update eslint to the version of approved version - https://phabricator.wikimedia.org/T264326 (10Umherirrender) Maybe fixed by eslint update with depth 10 in https://gerrit.wikimedia.org/r/c/labs/libraryupgrader/+/658730 [21:15:17] 10LibUp: libup should set warnings to off in .eslintrc.json when using maxWarnings: 0 - https://phabricator.wikimedia.org/T263922 (10Umherirrender) [21:15:48] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1357.eqiad.wmnet'] ` an... [21:21:36] (03PS1) 10Prod: Start branching WatchSubpages [tools/release] - 10https://gerrit.wikimedia.org/r/663892 (https://phabricator.wikimedia.org/T237809) [21:27:48] 10LibUp: libup should reorder grunt command to run eslint/jsonlint before banana - https://phabricator.wikimedia.org/T274683 (10Umherirrender) [21:28:09] 10Phabricator, 10Project-Admins, 10Release-Engineering-Team, 10Diffusion-Repository-Administrators: Modify projects Wikimedia-VE-Tech and related ACL to use for WikiSP - https://phabricator.wikimedia.org/T273344 (10Aklapper) Ping - could #release-engineering-team please provide input whether we are fine wi... [21:28:36] 10Continuous-Integration-Config, 10LibUp, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: LibraryUpgrader CI normalisation tasks, June/July 2019 - https://phabricator.wikimedia.org/T225325 (10Umherirrender) 05Open→03Resolved a:03Umherirrender Moved last bullet poi... [21:29:57] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [21:31:32] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1356.eqiad.wmnet'] ` an... [21:37:21] 10LibUp: libup should remove exclude of vendor/node_modules from .phpcs.xml - https://phabricator.wikimedia.org/T274684 (10Umherirrender) [21:40:54] 10Gerrit, 10git-protocol-v2: Gerrit upload-pack send ALL references causing massive network I/O on common operations - https://phabricator.wikimedia.org/T103990 (10hashar) [21:55:21] 10LibUp: libup should reorder grunt command to run eslint/jsonlint before banana - https://phabricator.wikimedia.org/T274683 (10Legoktm) I already wrote the grunt parser stuff for this https://gerrit.wikimedia.org/r/plugins/gitiles/labs/libraryupgrader/+/refs/heads/master/libup/grunt.py#203 so adding a fixer sho... [22:09:07] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [libs/IDLeDOM] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/663899 [22:09:09] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [libs/IDLeDOM] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/663899 (owner: 10QChris) [22:09:16] (03PS1) 10QChris: Import done. Revoke import grants [libs/IDLeDOM] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/663900 [22:09:18] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [libs/IDLeDOM] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/663900 (owner: 10QChris) [22:15:42] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by legoktm on cumin1001.eq... [22:16:37] (03CR) 10BryanDavis: [C: 04-1] python: upgrade pip before installing requirements (031 comment) [blubber] - 10https://gerrit.wikimedia.org/r/663348 (https://phabricator.wikimedia.org/T274435) (owner: 10BryanDavis) [22:18:25] (03CR) 10Dduvall: python: upgrade pip before installing requirements (031 comment) [blubber] - 10https://gerrit.wikimedia.org/r/663348 (https://phabricator.wikimedia.org/T274435) (owner: 10BryanDavis) [22:20:34] 10Gerrit: Can't `git pull` mediawiki/core from Gerrit: "fatal: the remote end hung up unexpectedly" - https://phabricator.wikimedia.org/T263293 (10hashar) I found the three errors in Gerrit and they are the same `SshChannelNotFoundException: Received SSH_MSG_CHANNEL_WINDOW_ADJUST on unassigned channel 0 (last as... [22:31:32] (03PS1) 10Dduvall: Export container process output from run step [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/663905 [22:33:59] (03PS2) 10Dduvall: Export container process output from run step [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/663905 [22:40:19] 10LibUp: Duplicate/redundant entries in libraryupgrader commit messages - https://phabricator.wikimedia.org/T228157 (10Umherirrender) I have checked out the parent of the mention commit and the json from `npm audit` only contains one action for grunt (now for the latest 1.3.0) ` { "actions": [ ... "... [22:45:37] 10LibUp: libup fails to update eslint-config-wikimedia to 0.17.0 due to old eslint - https://phabricator.wikimedia.org/T261520 (10Umherirrender) 05Open→03Resolved Yes, would also say it is fixed by https://gerrit.wikimedia.org/r/c/labs/libraryupgrader/+/658730 [22:48:45] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Wikimedia-Logstash, 10observability: logstash-beta.wmflabs.org does not receive any mediawiki events - https://phabricator.wikimedia.org/T233134 (10DannyS712) can this be closed now? Its happened a number of times since this was originally report... [22:51:38] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1348.eqiad.wmnet'] ` an... [23:01:25] (03PS2) 10BryanDavis: python: upgrade pip before installing requirements [blubber] - 10https://gerrit.wikimedia.org/r/663348 (https://phabricator.wikimedia.org/T274435) [23:02:22] (03CR) 10BryanDavis: python: upgrade pip before installing requirements (031 comment) [blubber] - 10https://gerrit.wikimedia.org/r/663348 (https://phabricator.wikimedia.org/T274435) (owner: 10BryanDavis) [23:04:16] (03CR) 10Dduvall: [C: 03+2] "<3 the new commit msg. Looks good!" [blubber] - 10https://gerrit.wikimedia.org/r/663348 (https://phabricator.wikimedia.org/T274435) (owner: 10BryanDavis) [23:05:09] bd808: if you're around to verify i can deploy ^ [23:05:37] marxarelli: I'm here and ready [23:05:44] k [23:06:25] just waiting on the post-merge version bump [23:07:08] (03Merged) 10jenkins-bot: python: upgrade pip before installing requirements [blubber] - 10https://gerrit.wikimedia.org/r/663348 (https://phabricator.wikimedia.org/T274435) (owner: 10BryanDavis) [23:11:50] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Wikimedia-Logstash, 10observability: logstash-beta.wmflabs.org does not receive any mediawiki events - https://phabricator.wikimedia.org/T233134 (10Krinkle) I don't know if the issue we saw this week and last month are the same as the one detaile... [23:16:00] 10Continuous-Integration-Config, 10serviceops: Add tox CI to operations/software/benchmw - https://phabricator.wikimedia.org/T274686 (10Legoktm) [23:18:05] bd808: deployed to staging [23:18:55] you can hit it with some test data from the deployment server, e.g. `curl -H 'content-type: application/yaml' --data-binary @- https://staging.svc.eqiad.wmnet:4666/v1/foo` [23:19:07] or anywhere on the prod network i guess [23:19:34] ok. let me give that a shot and see what my generated file looks like. [23:19:45] k [23:20:19] i just checked it against this data [23:20:23] https://www.irccloud.com/pastebin/xEmRzYi6/ [23:20:47] "Error fetching paste" [23:21:01] huh [23:21:12] never mind, that was my noscript in the way :) [23:21:14] * marxarelli shakes fist at irccloud [23:21:24] * marxarelli nods [23:23:39] marxarelli: my generated py3 file looks right [23:23:52] `RUN python3 "-m" "easy_install" "pip" && python3 "-m" "pip" "install" "-U" "setuptools" "wheel" "tox" "pip"` [23:24:05] cool. i'll roll it to eqiad/codfw then [23:29:27] bd808: deployed [23:31:26] thanks marxarelli! [23:31:43] np! thanks for the patch [23:42:43] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1281.eqiad.wmnet', 'mw12... [23:50:06] 10Phabricator, 10Diffusion-Repository-Administrators: Request for adding Kizule in #Repository-Admins - https://phabricator.wikimedia.org/T250119 (10Aklapper) > But, as far as I know, we are using Diffusion mainly to mirror gerrit repositories. For the records, there are 310 canonical repos in Diffusion (not...