[00:00:58] <paladox>	 though, gerrit 2.16 polygerrit ui gains a searchbar on mobile too :)
[00:01:04] <paladox>	 i think
[00:01:06] * paladox checks
[00:01:09] <mooeypoo>	 Is it shared to everyone, or is it individual dashboards?
[00:01:29] <paladox>	 oh
[00:01:39] <mooeypoo>	 I am only asking so that if it's shared/public, I need to pay more attention to my spelling of Anti Harassment Tools
[00:01:40] <mooeypoo>	 :p
[00:01:45] <paladox>	 heh
[00:01:58] <paladox>	 mooeypoo it's shared i think
[00:02:02] <paladox>	 like gwtui
[00:02:06] <mooeypoo>	 Cool, good to know
[00:02:14] <paladox>	 also the searchbar for mobile is in 3.0.
[00:02:58] <paladox>	 https://gerrit-review.googlesource.com/c/gerrit/+/221732
[00:03:47] <mooeypoo>	 What I found hard in mobile is responding to reviews; it is practically impossible in the old UI, I'm assuming/hoping that's fixed in the new
[00:04:21] <paladox>	 mooeypoo you can try to see if it works for you on https://gerrit.git.wmflabs.org/r/c/testing/test/+/2201
[00:04:36] <paladox>	 also, i've integrated a recheck button && zuul ui into gerrit!
[00:04:56] <mooeypoo>	 \o/
[00:05:42] <paladox>	 mooeypoo so no more refreshing either see https://phabzilla.wmflabs.org/file/data/myqcktnfep6yozxyjshz/PHID-FILE-hx4mb6fibwffwebgvcfh/Screenshot_2019-03-13_at_00.27.37.png
[00:07:05] <mooeypoo>	 oooh neat
[00:10:42] <paladox>	 mooeypoo and the old ui has been removed now :) (gerrit 3.0 was released last month)
[00:32:00] <wikibugs>	 10Release-Engineering-Team (Watching / External), 10Front-end-Standards-Group, 10Librarization, 10MinervaNeue, and 2 others: Move MobileFrontend/Minerva's svg_check.sh script into a reusable, separate library - https://phabricator.wikimedia.org/T179361 (10Volker_E) Repeating what I've said before, coming f...
[01:57:02] <wikibugs>	 (03PS1) 10Jeena Huneidi: Change services to use port 80 within cluster [releng/local-charts] - 10https://gerrit.wikimedia.org/r/514637
[01:58:47] <wikibugs>	 (03PS2) 10Jeena Huneidi: Change services to use port 80 within cluster [releng/local-charts] - 10https://gerrit.wikimedia.org/r/514637
[02:01:21] <wikibugs>	 (03PS3) 10Jeena Huneidi: Change services to use port 80 within cluster [releng/local-charts] - 10https://gerrit.wikimedia.org/r/514637
[02:05:27] <wikibugs>	 (03PS4) 10Jeena Huneidi: Change services to use port 80 within cluster [releng/local-charts] - 10https://gerrit.wikimedia.org/r/514637
[02:25:37] <wikibugs>	 10Phabricator (Upstream), 10Upstream: "Choose subtype" box does not disappear after choosing an option when creating new subtask with "Persistent Chat" option in Conpherence enabled - https://phabricator.wikimedia.org/T220897 (10epriestley)   - This should be fixed upstream by <https://secure.phabricator.com/D...
[03:21:18] <wikibugs>	 (03PS3) 10Kosta Harlan: Publish coverage reports from extension-codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/514016
[03:44:42] <wikibugs>	 (03PS1) 10Kosta Harlan: (wip) Run MediaWiki core through codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/514642 (https://phabricator.wikimedia.org/T224989)
[03:48:26] <wikibugs>	 (03PS2) 10Kosta Harlan: (wip) Run MediaWiki core through codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/514642 (https://phabricator.wikimedia.org/T224989)
[05:13:58] <wikibugs>	 10Diffusion: git operations against diffusion showing server-side logging failure messages - https://phabricator.wikimedia.org/T225176 (10bd808)
[05:52:17] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<55.56%)
[06:36:56] <wikibugs>	 10Continuous-Integration-Config, 10Growth-Team, 10GrowthExperiments, 10Patch-For-Review: Potentially fragile test in GrowthExperiments - https://phabricator.wikimedia.org/T225165 (10hashar)
[06:57:17] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK
[07:08:20] <wikibugs>	 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): labtestvirt2003: test different power management / CPU setups for faster kvm - https://phabricator.wikimedia.org/T225067 (10hashar)
[07:28:01] <wikibugs>	 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): labtestvirt2003: test different power management / CPU setups for faster kvm - https://phabricator.wikimedia.org/T225067 (10hashar) `stress-ng --cpu 1 --cpu-ops=400000` <- turns out that 400k is way too many operations, I am not sure why...
[07:48:39] <wikibugs>	 (03CR) 10Hashar: [C: 03+1] Vendor dependencies as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513503 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight)
[07:53:29] * awight warms hands over oil drum fire
[07:55:25] <wikibugs>	 (03PS1) 10Ema: Modify access rules [debs/varnish4] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/514662
[07:58:09] <Amir1>	 zeljkof hashar: morning. I want to delete all of ruby tests in wikibase and wikibase lexeme https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/514494 and https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikibaseLexeme/+/514497 but jenkins fails because it tries to run them. I removed as much as possible in https://gerrit.wikimedia.org/r/c/integration/config/+/514547 and
[07:58:10] <Amir1>	 https://gerrit.wikimedia.org/r/c/integration/config/+/514507 but it seems it's not enough. Can you check what's missing here? The config is confusing because lots of named "selenium" which I don't know if they refer to the nodejs browser tests or ruby :[
[08:03:48] <awight>	 Amir1: maybe the integration/config changes last night weren't deployed?  I'm not sure how to check that.
[08:05:03] <Amir1>	 awight: I'm considering it too, if that's case, still only ha.shar and zeljko.f can help :( 
[08:06:56] <awight>	 AFAIK, anything called "mwselenium" is referring to the ruby tests.
[08:07:37] <awight>	 webdriver.io (node) browser tests are run in quibble-*-docker, if and only if the tests/selenium directory is present.
[08:08:13] <awight>	 For example, you can see in the logs that quibble ran your node browser tests: https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/52151/console
[08:08:48] <awight>	 Mysterious af that mwselenium* is still running in that other job.
[08:09:29] <Amir1>	 thanks. Let me dig deep into the config
[08:09:48] <awight>	 I have deployment-prep access, maybe I can read the live config.
[08:10:29] <awight>	 ah no it won't be there... but in Jenkins I think
[08:10:39] <Amir1>	 yeah...
[08:12:29] <hashar>	 its is on contint1001
[08:13:13] <hashar>	 Amir1: f35a615284f90ab8d751d3444b874c6ad3116dd7 / I764972711843645afd00e196a3bedd17730b4cbe was not deployed
[08:13:25] <hashar>	 it removes the job mwselenium-quibble-docker for Wikibase
[08:13:40] <hashar>	 !log Reloading Zuul for I764972711843645afd00e196a3bedd17730b4cbe  which drops  mwselenium-quibble-docker from Wikibase
[08:13:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[08:13:48] <hashar>	 Amir1: reloaded
[08:13:58] <Amir1>	 thanks! let me try again
[08:14:05] <hashar>	 +2 on what awight said hehe
[08:14:46] <awight>	 I still don't understand the glue which reads zuul/layout.yaml and triggers Jenkins jobs...  I guess that is Zuul?  Need to learn one day.
[08:15:13] * awight larns https://www.mediawiki.org/wiki/Continuous_integration/Zuul
[08:15:44] <Amir1>	 hashar: I don't know if you're aware but yesterday gate-and-submit was really slow
[08:16:53] <awight>	 Amir1: I found something we can use next time.  https://integration.wikimedia.org/zuul/ has a field "Last reconfigured"
[08:17:12] <Amir1>	 nice!
[08:17:59] <awight>	 Unfortunately, it seems the https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL can deceive us
[08:18:14] <awight>	 Might be worth automating that 
[08:18:19] <awight>	 ...that !log one day
[08:18:34] <wikibugs>	 (03PS1) 10Hashar: Inherit from operations/debs instead of All-Projects [debs/varnish4] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/514664
[08:19:26] <wikibugs>	 (03Abandoned) 10Ema: Modify access rules [debs/varnish4] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/514662 (owner: 10Ema)
[08:19:56] <wikibugs>	 (03CR) 10Hashar: [V: 03+2 C: 03+2] "That saves ema!" [debs/varnish4] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/514664 (owner: 10Hashar)
[08:20:30] <wikibugs>	 10Continuous-Integration-Infrastructure: RelEng SAL automation for Zuul deployments - https://phabricator.wikimedia.org/T225182 (10awight)
[08:20:47] <wikibugs>	 (03PS1) 10Ladsgroup: zuul: Drop mwselenium from WikibaseLexeme [integration/config] - 10https://gerrit.wikimedia.org/r/514665 (https://phabricator.wikimedia.org/T224301)
[08:21:03] <Amir1>	 hashar: Does it make sense: https://gerrit.wikimedia.org/r/#/c/integration/config/+/514665 ?
[08:21:06] <hashar>	 Amir1: yeah because folks do force merge patches which slow it down.
[08:21:26] <hashar>	 and there are bunch of ongoing issues with the jobs running wayyy toooo maaaanyyyyy teeeesttts
[08:21:26] <Amir1>	 oh :(
[08:21:33] <hashar>	 + some instances being randomly slow
[08:21:48] <hashar>	 and flappy tests
[08:21:55] <awight>	 Amir1: hey good news, your tests don't include mwselenium* now!
[08:21:56] <Amir1>	 hashar: I hope this helps: https://phabricator.wikimedia.org/T87781
[08:22:16] <Amir1>	 awight: yeah, wikibase is done now, only wikibase lexeme left
[08:22:18] <awight>	 Amir1: Wouldn't we still have to run everything to gate?
[08:22:38] <Amir1>	 awight: it makes "everything" faster :P
[08:22:59] <hashar>	 depends
[08:23:00] <hashar>	 well
[08:23:00] <Amir1>	 at least a nice proportion of tests (25% of core tests twice as fast)
[08:23:24] <hashar>	 an example is that WikibaseLexeme has bunch of browser tests that take a while to run (a few minutes)
[08:23:45] <hashar>	 and those tests are run for any repo being gated together (which is like ~ 30 repos)
[08:23:55] <hashar>	 and that slow down the whole thing
[08:24:08] <hashar>	 also CirrusSearch is similar and has a serie of test that takes roughly 2 minutes
[08:24:27] <Amir1>	 I see, so something like https://phabricator.wikimedia.org/T225068 would help?
[08:25:04] <hashar>	 Split mediawiki tests into unit and integration tests, yeah that helps a bit. Though frmo the hackathon experiment that just saves a few dozen of seconds or so
[08:25:10] <awight>	 I'm not caught up on the parallelization discussion, but sharing a random thought anyway--Maybe browser tests clients can be run in parallel against a single server?
[08:25:14] <dcausse>	 hashar: what would be an acceptable duration? I can dig into the tests and optimize them
[08:25:24] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] zuul: Drop mwselenium from WikibaseLexeme [integration/config] - 10https://gerrit.wikimedia.org/r/514665 (https://phabricator.wikimedia.org/T224301) (owner: 10Ladsgroup)
[08:25:30] <dcausse>	 (e.g. for CirrusSearch unit tests) 
[08:25:43] <hashar>	 dcausse: sub miliseconds? ;D
[08:25:48] <dcausse>	 meh :)
[08:25:51] <hashar>	 well
[08:26:05] <hashar>	 given it is java, I would even expect to get the test results BEFORE running the test command
[08:26:09] <hashar>	 I am sure maven has a solution
[08:26:15] <hashar>	 no more seriously I don't know
[08:26:24] <hashar>	 I should probably first fill a task
[08:26:31] <awight>	 It's unfortunate that so many tests rely on the database, I think a lot of things could be mocked because we don't care about testing the actual SQL.
[08:26:48] <dcausse>	 ok it's been a long since I looked at the fixture we test there might things I can remove, that cannot hurt anyways 
[08:26:58] <wikibugs>	 (03Merged) 10jenkins-bot: zuul: Drop mwselenium from WikibaseLexeme [integration/config] - 10https://gerrit.wikimedia.org/r/514665 (https://phabricator.wikimedia.org/T224301) (owner: 10Ladsgroup)
[08:27:01] <hashar>	 and yesterday we had a task filled to optimize which PHPUnit tests we are running. So that the CirrusSearch test would NOT be run when sending a change for hmm Citoid?
[08:27:56] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/514665 (https://phabricator.wikimedia.org/T224301) (owner: 10Ladsgroup)
[08:27:57] <dcausse>	 CirrusSearch could be run for core changes but I see no reason to run them when Citoid changes
[08:28:06] <Amir1>	 hashar: thanks!
[08:31:16] <awight>	 dcausse: I think there's value in gating extensions against each other.  We just ran into an issue where AdvancedSearch was *not* a gated extension, and discovered its tests had been broken in 5 different ways by unrelated repos.  I'd like those unrelated patches to fail before breaking important stuff...
[08:31:19] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "Thanks for the test refactoring!" (033 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/513406 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight)
[08:31:56] <awight>	 dcausse: On the other hand, maybe a daily or hourly gate build is good enough, if we had a mechanism to get the right people's attention when things break?
[08:32:11] <hashar>	 Amir1: dcausse awight https://phabricator.wikimedia.org/T225068 "Add a PHPUnit group to skip test on gated CI runs"
[08:32:21] <hashar>	 which is  roughly about being able to skip some tests in some scenario
[08:32:50] <dcausse>	 awight: I agree 
[08:33:02] <awight>	 But will those tests still get run at some point, to prevent cross-extension/core incompatibilities?
[08:33:54] <dcausse>	 I like the idea of having "build the universe" jobs run hourly
[08:34:05] <dcausse>	 like we have for some browser tests against beta
[08:35:13] <hashar>	 yeah definitely a fan of oit as well
[08:35:51] <hashar>	 then some tests take ages are only for a singe extension
[08:36:23] <hashar>	 scribunto for example has some tests for the lua engine and we know that they are not going to be affected by changes to other extensions
[08:37:10] <wikibugs>	 10Release-Engineering-Team, 10MediaWiki-Core-Testing: Ensure we're testing appropriately and not over-testing across Wikimedia-deployed code - https://phabricator.wikimedia.org/T221434 (10hashar) See also {T225068}
[08:39:03] <dcausse>	 hashar: basically you're still undecided on this whitelist vs blacklist approach?
[08:41:36] <hashar>	 dcausse: me ? I have no idea how to fix it up / optimize the run so that is why CI just blindly runs everything :]
[08:41:56] <hashar>	 I wrote the reply yesterday, hopefully some people will catch up today and come with other ideas
[08:43:11] <dcausse>	 ok, happy help to help and flag all cirrus tests once we have something
[08:43:43] <hashar>	 dcausse: https://phabricator.wikimedia.org/T225184 ;D
[08:43:55] <hashar>	 if there are some low hanging fruit to make it faster, that would benefit anyone I guess
[08:44:17] <hashar>	 and for the flagging of those tests, yeah that is pending a few more discussions
[08:44:19] <dcausse>	 sure will work on it today
[08:45:37] <hashar>	 awight: for Quibble, good news I finally got the intent of the chain of changes and I like it. I am just struggling to process the very first change which does the zuul clone.  I guess I need a beer and to work from the beach ; )
[08:46:06] <hashar>	 awight: one sure thing, the code will be nicer once all those changes land!
[08:50:40] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Backlog), 10Striker, 10Tools: Add Gerrit support to Striker (toolsadmin) - https://phabricator.wikimedia.org/T224676 (10Siridech)
[08:58:00] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Backlog), 10Striker, 10Tools: Add Gerrit support to Striker (toolsadmin) - https://phabricator.wikimedia.org/T224676 (10LucasWerkmeister)
[09:09:07] <Amir1>	 One weird issue: T223995 this only happens in my localhost and production (I can get it to reproduce easily) but not in beta cluster, I couldn't get it to fatal in beta. The configs are the same
[09:09:08] <stashbot>	 T223995: BadMethodCallException wbgetentities when getting Lexeme subentities (forms, senses) - https://phabricator.wikimedia.org/T223995
[09:09:33] <Amir1>	 that's one hell of a bug
[09:22:07] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10hashar)
[09:22:10] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%): - https://phabricator.wikimedia.org/T219850 (10hashar) 05Open→03Resolved I cleaned up some images yesterday:  2019-06-05 19:57  <hashar>  cont...
[09:22:30] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-mwmaint01 is CRITICAL: CRITICAL: deployment-prep.deployment-mwmaint01.diskspace.root.byte_percentfree (<11.11%)
[09:24:52] <wikibugs>	 (03CR) 10Awight: Extension and skin submodule update as command (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/513406 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight)
[09:26:56] <awight>	 hashar: Beer and beach is quite appropriate for most difficult questions :-)  I would be fine with shuffling the code around to leave the zuul clone similar to what it does now--the only problem I'd like to solve in that patch is how to collect the necessary parameters into an object, ahead of time.
[09:27:40] <awight>	 One alternative would be to move the os.environ stuff into the calling function, I'm not sure which is nicer honestly.
[09:32:30] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-mwmaint01 is OK: OK: All targets OK
[09:37:59] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10hashar) The new disks can be shown as sdc and sdd.  Currently I think we have 3 RAID 1 arrays, with LVM on t...
[09:39:11] <hashar>	 awight: yes I got the intent, it is right. I am just puzzled by your approach :]   But I am sure it will tick in this afternoon and I might well just +2 the patch
[09:39:26] <hashar>	 I guess I just need to digest your change
[09:39:35] <awight>	 I caught you on an easy day, then ;-)
[09:39:37] <wikibugs>	 (03CR) 10Awight: Extension and skin submodule update as command (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/513406 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight)
[09:39:52] <awight>	 My approach is probably terrible... I just didn't know what else to do!
[09:40:20] <awight>	 Also, apologies for my bad habit of starting long patch chains with the most challenging piece :-/
[09:43:54] <wikibugs>	 (03PS3) 10Awight: Extension and skin submodule update as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513406 (https://phabricator.wikimedia.org/T223752)
[09:44:34] <awight>	 I think I see how to undo my zuul butchery
[09:47:21] <wikibugs>	 10Continuous-Integration-Config, 10Librarization, 10Performance-Team, 10RunningStat, 10patch-welcome: Publish Doxygen for RunningStat library - https://phabricator.wikimedia.org/T185724 (10aaron) p:05Normal→03Lowest
[09:51:50] <hashar>	 awight:  dont worry ;:]
[09:51:59] <hashar>	 I just need to spend some calm time on the first patch of the serie
[10:02:29] <awight>	 Now I remember the other reason I did this weird build_params thing.  I wanted to avoid making the ZuulCloneCommand into a fragile repetition of a long list of params.
[10:02:35] <awight>	 But lemme try it this way just to see...
[10:10:24] <wikibugs>	 10MediaWiki-Codesniffer, 10MediaWiki-General-or-Unknown, 10Performance-Team, 10Performance: Using fully-qualified function calls is faster - https://phabricator.wikimedia.org/T200629 (10Krinkle) Without some evidence that it would help with latencies, this is difficult to act on. E.g. a local patch and run...
[10:10:34] <wikibugs>	 10MediaWiki-Codesniffer, 10MediaWiki-General-or-Unknown, 10Performance-Team (Radar): Using fully-qualified function calls is faster - https://phabricator.wikimedia.org/T200629 (10Krinkle)
[10:30:52] <wikibugs>	 (03PS2) 10Awight: Extract zuul clone into a command object [integration/quibble] - 10https://gerrit.wikimedia.org/r/513377 (https://phabricator.wikimedia.org/T223752)
[10:32:30] <wikibugs>	 (03CR) 10Awight: "PS2: @hashar, thanks for the nudge to get rid of `build_params`.  I like the parameter wiring much better now, it's completely explicit an" [integration/quibble] - 10https://gerrit.wikimedia.org/r/513377 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight)
[10:33:47] <wikibugs>	 10Continuous-Integration-Infrastructure: CI monitoring to detect flapping tests, especially in unrelated gated extensions - https://phabricator.wikimedia.org/T225193 (10awight)
[10:38:20] <wikibugs>	 10Continuous-Integration-Config, 10Fresnel, 10Performance-Team: For the Fresnel job, distinguish system failure from assert failure - https://phabricator.wikimedia.org/T216574 (10Krinkle) 05Open→03Stalled Blocked on Zuul supporting the other build status that Jenkins supports already (e.g. not just SUCCE...
[11:27:25] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T220733 (10Krinkle)
[11:37:36] <wikibugs>	 10Continuous-Integration-Config, 10Fresnel, 10Performance-Team: For the Fresnel job, distinguish system failure from assert failure - https://phabricator.wikimedia.org/T216574 (10hashar) Indeed Zuul only considers `SUCCESS`, we are definitely not going to add support for another build status in our current f...
[11:49:19] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T220733 (10Krinkle)
[12:36:54] <wikibugs>	 10MediaWiki-Releasing, 10MW-1.27-release: Formally announce EOL of MW 1.27 and 1.30 - https://phabricator.wikimedia.org/T225201 (10Reedy)
[12:37:08] <wikibugs>	 10MediaWiki-Releasing, 10MW-1.27-release: Formally announce EOL of MW 1.27 and 1.30 - https://phabricator.wikimedia.org/T225201 (10Reedy) p:05Triage→03High a:03Reedy
[12:37:50] <wikibugs>	 10MediaWiki-Releasing, 10MW-1.27-release: Formally announce EOL of MW 1.27 and 1.30 - https://phabricator.wikimedia.org/T225201 (10Reedy)
[12:38:36] <wikibugs>	 10MediaWiki-Releasing, 10MW-1.27-release: Formally announce EOL of MW 1.27 and 1.30 - https://phabricator.wikimedia.org/T225201 (10Reedy)
[12:38:51] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Documentation, 10Security: EOL REL1_27 and REL1_30 onwiki - https://phabricator.wikimedia.org/T224913 (10Reedy)
[12:39:17] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Documentation, 10Security: EOL REL1_27 and REL1_30 onwiki - https://phabricator.wikimedia.org/T224913 (10Reedy)
[12:39:25] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Documentation: EOL REL1_27 and REL1_30 onwiki - https://phabricator.wikimedia.org/T224913 (10Reedy)
[13:08:39] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T220733 (10Reedy)
[13:09:02] <wikibugs>	 (03PS2) 10Thiemo Kreuz (WMDE): Add FileImporter to gated extensions [integration/config] - 10https://gerrit.wikimedia.org/r/514065 (https://phabricator.wikimedia.org/T222802) (owner: 10Jkroll)
[13:09:58] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T220733 (10dcausse)
[13:11:55] <wikibugs>	 (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] "The previously mentioned, possibly expensive test is currently being worked on via I7af6fe4 (T223017). Even without this patch being merge" [integration/config] - 10https://gerrit.wikimedia.org/r/514065 (https://phabricator.wikimedia.org/T222802) (owner: 10Jkroll)
[13:17:13] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T220733 (10Reedy)
[13:19:32] <wikibugs>	 (03CR) 10WMDE-Fisch: [C: 03+1] Add FileImporter to gated extensions [integration/config] - 10https://gerrit.wikimedia.org/r/514065 (https://phabricator.wikimedia.org/T222802) (owner: 10Jkroll)
[13:23:22] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-Documentation, 10Regression, 10Upstream: Doxygen search.php no longer works on doc.wikimedia.org - https://phabricator.wikimedia.org/T218233 (10MoritzMuehlenhoff) There's two angles to address:  1. Getting this fixed interna...
[13:26:09] <wikibugs>	 10Deployments, 10Operations: Enable scap to roll back broken changes to MediaWiki - https://phabricator.wikimedia.org/T225207 (10Lucas_Werkmeister_WMDE)
[13:28:00] <wikibugs>	 (03PS4) 10Awight: Extension and skin submodule update as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513406 (https://phabricator.wikimedia.org/T223752)
[13:28:02] <wikibugs>	 (03PS3) 10Awight: Composer dependencies as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513498 (https://phabricator.wikimedia.org/T223752)
[13:28:04] <wikibugs>	 (03PS2) 10Awight: Vendor dependencies as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513503 (https://phabricator.wikimedia.org/T223752)
[13:28:06] <wikibugs>	 (03PS2) 10Awight: Composer and npm test as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513504 (https://phabricator.wikimedia.org/T223752)
[13:28:10] <wikibugs>	 (03PS2) 10Awight: Npm install as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513510 (https://phabricator.wikimedia.org/T223752)
[13:28:12] <wikibugs>	 (03PS2) 10Awight: phpunit tests as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513513 (https://phabricator.wikimedia.org/T223752)
[13:28:14] <wikibugs>	 (03PS2) 10Awight: Core npm composer test as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513515 (https://phabricator.wikimedia.org/T223752)
[13:28:16] <wikibugs>	 (03PS2) 10Awight: browser tests as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513518 (https://phabricator.wikimedia.org/T223752)
[13:28:18] <wikibugs>	 (03PS2) 10Awight: MediaWiki installation... as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513520 (https://phabricator.wikimedia.org/T223752)
[13:28:20] <wikibugs>	 (03PS9) 10Awight: Separate planning and execution phases [integration/quibble] - 10https://gerrit.wikimedia.org/r/511749 (https://phabricator.wikimedia.org/T223752)
[13:37:31] <wikibugs>	 10Release-Engineering-Team (Kanban), 10MediaWiki-Release-Tools, 10Core Platform Team (Release Automation (TEC1)), 10Core Platform Team Kanban (Contractor - Ready), and 2 others: Release notes changelog automation for tarball - https://phabricator.wikimedia.org/T217958 (10MarkAHershberger) a:05MarkAHershbe...
[13:58:25] <wikibugs>	 (03PS2) 10Michael Große: Add sonar CodeHealth checks to Wikibase repos [integration/config] - 10https://gerrit.wikimedia.org/r/511066 (https://phabricator.wikimedia.org/T216630)
[13:58:27] <wikibugs>	 (03PS1) 10Michael Große: Remove coverage ci step for extensions with codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/514717
[14:01:24] <wikibugs>	 10Release-Engineering-Team (Kanban), 10MediaWiki-Release-Tools, 10Core Platform Team (Release Automation (TEC1)), 10Core Platform Team Kanban (Contractor - Doing), and 2 others: Make release scripts initiatable via Jenkins - https://phabricator.wikimedia.org/T217839 (10MarkAHershberger) 05Declined→03Ope...
[14:06:15] <wikibugs>	 (03CR) 10Kosta Harlan: [C: 03+1] Add sonar CodeHealth checks to Wikibase repos [integration/config] - 10https://gerrit.wikimedia.org/r/511066 (https://phabricator.wikimedia.org/T216630) (owner: 10Michael Große)
[14:07:39] <wikibugs>	 10Release-Engineering-Team (Watching / External), 10Operations, 10SRE-Access-Requests, 10Patch-For-Review, 10User-jijiki: Requesting access to deployment for Christoph Jauera (WMDE-Fisch) - https://phabricator.wikimedia.org/T211014 (10Ladsgroup) I just added @WMDE-Fisch to [[https://gerrit.wikimedia.org/...
[14:09:50] <wikibugs>	 (03CR) 10WMDE-leszek: [C: 03+1] Add sonar CodeHealth checks to Wikibase repos [integration/config] - 10https://gerrit.wikimedia.org/r/511066 (https://phabricator.wikimedia.org/T216630) (owner: 10Michael Große)
[14:30:04] <wikibugs>	 (03PS1) 10Ladsgroup: zuul: Bring back Wikibase ruby tests [integration/config] - 10https://gerrit.wikimedia.org/r/514733 (https://phabricator.wikimedia.org/T224301)
[14:33:24] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "Sounds good so! :]" [integration/config] - 10https://gerrit.wikimedia.org/r/514065 (https://phabricator.wikimedia.org/T222802) (owner: 10Jkroll)
[14:34:49] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] zuul: Bring back Wikibase ruby tests [integration/config] - 10https://gerrit.wikimedia.org/r/514733 (https://phabricator.wikimedia.org/T224301) (owner: 10Ladsgroup)
[14:35:35] <wikibugs>	 (03Merged) 10jenkins-bot: Add FileImporter to gated extensions [integration/config] - 10https://gerrit.wikimedia.org/r/514065 (https://phabricator.wikimedia.org/T222802) (owner: 10Jkroll)
[14:36:24] <wikibugs>	 (03Merged) 10jenkins-bot: zuul: Bring back Wikibase ruby tests [integration/config] - 10https://gerrit.wikimedia.org/r/514733 (https://phabricator.wikimedia.org/T224301) (owner: 10Ladsgroup)
[14:36:58] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Quibble: Consider httpd for quibble instead of php built-in server - https://phabricator.wikimedia.org/T225218 (10kostajh)
[14:39:22] <wikibugs>	 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: New phan dependencies significantly slowed down CI tests - https://phabricator.wikimedia.org/T225112 (10Ottomata) Can we revert this until this is fixed? There seem to be other problems than just slow CI; CentralNotice tests are failing...
[14:54:55] <wikibugs>	 10Deployments, 10Release-Engineering-Team (Long-Lived-Branches): Static asset time on disk - https://phabricator.wikimedia.org/T140921 (10Jdforrester-WMF) > With this I believe we can safely remove branches 8 days after they were last used for a wiki.  Hurrah.  > And quite possibly are already doing so.  We st...
[14:57:13] <wikibugs>	 10MediaWiki-Releasing, 10MW-1.27-release: Formally announce EOL of MW 1.27 and 1.30 - https://phabricator.wikimedia.org/T225201 (10Reedy)
[14:57:57] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Kanban), 10Mail, 10Operations, and 2 others: Phabricator email comments not posted - https://phabricator.wikimedia.org/T224752 (10fsero) merged and applied
[14:59:23] <shinken-wm>	 PROBLEM - Host webperformance is DOWN: CRITICAL - Host Unreachable (172.16.3.26)
[15:05:39] <shinken-wm>	 RECOVERY - Host webperformance is UP: PING OK - Packet loss = 0%, RTA = 0.64 ms
[15:12:21] <wikibugs>	 10Release-Engineering-Team (Kanban), 10MediaWiki-Release-Tools, 10Core Platform Team (Release Automation (TEC1)), 10Core Platform Team Kanban (Contractor - Ready), and 2 others: Bot to upgrade the different bits of mw.o for tarball releases - https://phabricator.wikimedia.org/T217964 (10MarkAHershberger) a...
[15:12:23] <hashar>	 !log reloading Zuul for I8d6ea5b5491fe50d7144926ef8bc69d4704d4fab T224301
[15:12:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[15:12:26] <stashbot>	 T224301: Clean up ruby test infrastructure - https://phabricator.wikimedia.org/T224301
[15:12:27] <icinga-wm>	 PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 57.14% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[15:12:36] <hashar>	 !log reloading zuul for I044bedd41091b57a1a188ea243c4769a6e21babb T222802
[15:12:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[15:12:39] <stashbot>	 T222802: Add FileImporter to gated extensions - https://phabricator.wikimedia.org/T222802
[15:12:52] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/514733 (https://phabricator.wikimedia.org/T224301) (owner: 10Ladsgroup)
[15:12:56] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/514065 (https://phabricator.wikimedia.org/T222802) (owner: 10Jkroll)
[15:13:22] <wikibugs>	 (03CR) 10Ladsgroup: "> Patch Set 1:" [integration/config] - 10https://gerrit.wikimedia.org/r/514733 (https://phabricator.wikimedia.org/T224301) (owner: 10Ladsgroup)
[15:14:17] <Lucas_WMDE>	 I think there’s a job stuck in gate-and-submit?
[15:14:26] <Lucas_WMDE>	 https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/52253/console hasn’t had any new console output since 14:52:25 UTC
[15:15:55] <wikibugs>	 10Diffusion, 10Release-Engineering-Team (Backlog), 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Gerrit repo scoring/ores/editquality not mirroing - https://phabricator.wikimedia.org/T224996 (10Halfak) I've just tried to pull new changes from https://gerrit.wikimedia.org/r/s...
[15:18:24] <Lucas_WMDE>	 other jobs on integration-slave-docker-1054 are still running, though
[15:18:28] <Lucas_WMDE>	 weird
[15:20:34] <Lucas_WMDE>	 okay, the job is now “red” in https://integration.wikimedia.org/ci/computer/integration-slave-docker-1054/, whatever that means
[15:20:42] <Lucas_WMDE>	 (in the build executor status section)
[15:23:11] <Lucas_WMDE>	 can anyone help me with this? it’s blocking the whole gate-and-submit queue
[15:23:38] <Lucas_WMDE>	 (though so far none of the jobs queued behind it have finished building)
[15:24:52] <Reedy>	 looks to have stalle
[15:24:57] <Reedy>	 I'll kill it
[15:25:01] <Lucas_WMDE>	 thanks
[15:29:22] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Kanban), 10Mail, 10Operations, and 2 others: Phabricator email comments not posted - https://phabricator.wikimedia.org/T224752 (10mmodell) testing reply via email.
[15:30:47] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Kanban), 10Mail, 10Operations, and 2 others: Phabricator email comments not posted - https://phabricator.wikimedia.org/T224752 (10mmodell) 05Open→03Resolved a:03mmodell Working! Thanks @fsero
[15:44:51] <icinga-wm>	 RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[15:51:35] <wikibugs>	 10MediaWiki-Releasing, 10Security: Write and send release announcements for 1.27.6/1.30.2/1.31.2/1.32.2 security releases - https://phabricator.wikimedia.org/T205042 (10Reedy) 05Open→03Resolved
[15:51:39] <wikibugs>	 10MediaWiki-Releasing, 10Security: Write and send release announcements for 1.27.6/1.30.2/1.31.2/1.32.2 security releases - https://phabricator.wikimedia.org/T205042 (10Reedy)
[15:52:05] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Documentation: EOL REL1_27 and REL1_30 onwiki - https://phabricator.wikimedia.org/T224913 (10Reedy)
[15:52:08] <wikibugs>	 10MediaWiki-Releasing, 10MW-1.27-release: Formally announce EOL of MW 1.27 and 1.30 - https://phabricator.wikimedia.org/T225201 (10Reedy) 05Open→03Resolved
[15:58:29] <wikibugs>	 10MediaWiki-Releasing, 10Security: Release MediaWiki 1.27.6/1.30.2/1.31.2/1.32.2 - https://phabricator.wikimedia.org/T205039 (10Reedy)
[15:59:35] <wikibugs>	 10MediaWiki-Releasing, 10Security: Release MediaWiki 1.27.6/1.30.2/1.31.2/1.32.2 - https://phabricator.wikimedia.org/T205039 (10Reedy)
[16:02:13] <wikibugs>	 10MediaWiki-Releasing, 10Security: Obtain CVEs for 1.27.6/1.30.2/1.31.2/1.32.2 security releases - https://phabricator.wikimedia.org/T205048 (10Reedy)
[16:02:52] <wikibugs>	 10MediaWiki-Releasing, 10Documentation, 10Security: Update CVEs and publish them - https://phabricator.wikimedia.org/T225149 (10Reedy)
[16:02:57] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Documentation, 10Security: Update MediaWiki.org links and versions - https://phabricator.wikimedia.org/T224912 (10Reedy)
[16:03:21] <wikibugs>	 10MediaWiki-Releasing, 10Security: Tag MW 1.27.6/1.30.2/1.31.2/1.32.2 - https://phabricator.wikimedia.org/T205047 (10Reedy)
[16:03:28] <wikibugs>	 10MediaWiki-Releasing, 10Security: Update HISTORY in master after 1.27.6/1.30.2/1.31.2/1.32.2 - https://phabricator.wikimedia.org/T205046 (10Reedy)
[16:03:32] <wikibugs>	 10MediaWiki-Releasing, 10Security: Update onwiki release notes for 1.27.6/1.30.2/1.31.2/1.32.2 - https://phabricator.wikimedia.org/T205044 (10Reedy)
[16:05:16] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T220733 (10zeljkofilipin) 05Open→03Resolved
[16:06:23] <James_F>	 Obviously the zuul pipeline is going to be very full for most of the rest of the day.
[16:07:29] <Lucas_WMDE>	 oh good, because it was so empty before
[16:07:33] <icinga-wm>	 PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 42.86% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[16:07:44] * Lucas_WMDE is partially to blame for how full it was already
[16:15:24] <James_F>	 Lucas_WMDE: But now we can blame Security! ;-)
[16:22:28] <wikibugs>	 10Continuous-Integration-Infrastructure: Consider having a top-level jenkins CI job for each commit, so they can be manually killed swiftly rather than one-by-one - https://phabricator.wikimedia.org/T225231 (10Jdforrester-WMF)
[16:22:33] <wikibugs>	 10Continuous-Integration-Config, 10Code-Health-Metrics: Enable codehealth pipeline for more extensions - https://phabricator.wikimedia.org/T224990 (10kostajh) >  It still results in a little duplication in that we generate the coverage twice on postmerge. Maybe there's a way to run the cover-extensions-publish...
[16:24:15] <wikibugs>	 10Release-Engineering-Team (Long-Lived-Branches), 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Spike, 10Wikimedia-Incident: Spike: Plan reforms of the CentralNotice deployment branch - https://phabricator.wikimedia.org/T136904 (10greg) This is done, yes?
[16:25:25] <wikibugs>	 10Release-Engineering-Team (Long-Lived-Branches), 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Spike, 10Wikimedia-Incident: Spike: Plan reforms of the CentralNotice deployment branch - https://phabricator.wikimedia.org/T136904 (10AndyRussG) >>! In T136904#5240957, @greg wrote: > This is...
[16:27:24] <wikibugs>	 10Deployments, 10Release-Engineering-Team (Backlog): Remove "php-" from wiki version numbers - https://phabricator.wikimedia.org/T63733 (10greg)
[16:27:52] <wikibugs>	 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: New phan dependencies significantly slowed down CI tests - https://phabricator.wikimedia.org/T225112 (10Legoktm) >>! In T225112#5240503, @Ottomata wrote: > Can we revert this until this is fixed? There seem to be other problems than just...
[16:28:11] <wikibugs>	 10Release-Engineering-Team (Long-Lived-Branches), 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Spike, 10Wikimedia-Incident: Spike: Plan reforms of the CentralNotice deployment branch - https://phabricator.wikimedia.org/T136904 (10greg) 05Open→03Resolved a:03thcipriani
[16:47:26] <wikibugs>	 10Continuous-Integration-Infrastructure: Have a way to kill jobs from the zuul queue before they start - https://phabricator.wikimedia.org/T225233 (10Reedy)
[16:47:45] <wikibugs>	 10Project-Admins: Create a project for Wikilink - https://phabricator.wikimedia.org/T225065 (10MGChecker) I think this project name could be cause of confusion, because internal links on Wikipedia are often referred to as wikilinks as well. People will probably be inclined to add this project if they experience...
[16:50:01] <wikibugs>	 10Continuous-Integration-Config, 10Growth-Team, 10StructuredDiscussions, 10Patch-For-Review, 10User-kostajh: Fix Flow random test failures - https://phabricator.wikimedia.org/T208988 (10kostajh) I just came across {T150430}, and am wondering if that is what is happening in some of these cases.
[16:50:27] <icinga-wm>	 RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[16:51:14] <James_F>	 Reedy: You made things so bad that icinga is mad with us.
[16:51:23] <Reedy>	 haha
[16:51:44] <Reedy>	 #lifegoals
[16:51:54] <Reedy>	 I haven't even pushed like 30 of the patches
[16:52:57] <paladox>	 lol
[16:53:14] <James_F>	 We could make things much easier by supporting fewer branches. ;-)
[16:53:30] <James_F>	 (Or by doing more frequent, smaller security releases, but that's more work.)
[16:55:16] <Reedy>	 I'm dropping 2 branches in one go
[16:55:19] <Reedy>	 So bleugh to you :P
[16:55:39] <Reedy>	 dropping? have dropped
[16:55:41] <Reedy>	 whatever
[16:56:02] <Reedy>	 The biggest problem here is the thing of all the mw stuff being in one submit queue
[17:01:07] <Reedy>	 I'm gonna be eating soon... so if jenkins catches up and people see it running gate and submit for the 1.27/1.30 patches, please kill as muchas possible
[17:03:01] <wikibugs>	 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: New phan dependencies significantly slowed down CI tests - https://phabricator.wikimedia.org/T225112 (10Ottomata) It is blocking quite a few EventBus merges.  Not sure how to get CentralNotice maintainers involved.
[17:10:27] <wikibugs>	 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: New phan dependencies significantly slowed down CI tests - https://phabricator.wikimedia.org/T225112 (10Legoktm) >>! In T225112#5241107, @Ottomata wrote: > It is blocking quite a few EventBus merges.  Not sure how to get CentralNotice ma...
[17:11:00] <apergos>	 30 patches? wat
[17:11:25] <Reedy>	 there's 3 more branches to push patches to (1.31, 1.32, 1.33)
[17:11:42] <apergos>	 ohmys
[17:11:58] <Reedy>	 so like 50-60 in total
[17:15:22] <apergos>	 ow ow oooowwwwww
[17:18:28] <Reedy>	 I'm fixing the two ci failures (one whitespace, one php doc) issues in the other three branches locally
[17:19:29] <apergos>	 I do not envy you
[17:19:56] <Reedy>	 we've had worse :)
[17:20:13] <apergos>	 war stories over beer, next all hands
[17:20:37] <Reedy>	 lol
[17:23:57] <Reedy>	 it's nearly onto the REL1_27 and REL1_30 patches in gate and submit
[17:24:01] <Reedy>	 please kill P
[17:24:19] <apergos>	 I am not the jenkins job killer, dunno who is
[17:25:47] <Reedy>	 awww
[17:26:53] <apergos>	 sounds like a hashar project to me (though there is no hashar around just now)
[17:27:19] <Reedy>	 anyone with ciadmin...
[17:27:22] <Reedy>	 Lego, James maybe...
[17:27:25] <Reedy>	 most of releng
[17:27:45] <legoktm>	 hi
[17:28:28] <Reedy>	 legoktm: Want to shoot some of the CI gate and submit jobs for 1.27/1.30?
[17:28:34] <Reedy>	 Should be starting soon
[17:28:43] <Reedy>	 pointless them running etc
[17:30:11] <legoktm>	 have you just been killing them from inside jenkins?
[17:30:36] <Reedy>	 yeah, wasn't really sure of any other way
[17:31:04] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Front-end-Standards-Group, 10MediaWiki-extensions-General, 10Core Platform Team Backlog (Designing), 10Services (designing): Decide whether we want the package-lock.json to commit or ignore - https://phabricator.wikimedia.org/T179229 (10kostajh) > OK, yes, this...
[17:31:52] <wikibugs>	 10MediaWiki-Releasing, 10Security: Release MediaWiki 1.27.6/1.30.2/1.31.2/1.32.2 - https://phabricator.wikimedia.org/T205039 (10Jdforrester-WMF)
[17:31:55] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Documentation, 10Security: Update MediaWiki.org links and versions - https://phabricator.wikimedia.org/T224912 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF
[17:36:23] <wikibugs>	 10Project-Admins: Create a project for Wikilink - https://phabricator.wikimedia.org/T225065 (10Samwalton9) Fair concern - quite happy for this to be called `Wikilink-Tool` or something that makes this clearer.
[17:40:39] <wikibugs>	 (03PS3) 10Kosta Harlan: (wip) Run MediaWiki core through codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/514642 (https://phabricator.wikimedia.org/T224989)
[17:47:51] <wikibugs>	 10MediaWiki-Releasing, 10Security: Release MediaWiki 1.27.6/1.30.2/1.31.2/1.32.2 - https://phabricator.wikimedia.org/T205039 (10Jdforrester-WMF)
[17:47:54] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Documentation: EOL REL1_27 and REL1_30 onwiki - https://phabricator.wikimedia.org/T224913 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF
[17:48:25] <wikibugs>	 (03PS1) 10Ottomata: Revert "[EventBus] Add dependency" [integration/config] - 10https://gerrit.wikimedia.org/r/514841 (https://phabricator.wikimedia.org/T225195)
[17:48:34] <wikibugs>	 10MediaWiki-Releasing, 10Security: Update onwiki release notes for 1.27.6/1.30.2/1.31.2/1.32.2 - https://phabricator.wikimedia.org/T205044 (10Jdforrester-WMF) Done 1.27 and 1.30; waiting on 1.31 and 1.32 to land.
[17:48:39] <wikibugs>	 10MediaWiki-Releasing, 10Security: Update onwiki release notes for 1.27.6/1.30.2/1.31.2/1.32.2 - https://phabricator.wikimedia.org/T205044 (10Jdforrester-WMF) a:03Jdforrester-WMF
[17:49:59] <legoktm>	 killed the jobs that started so far
[17:50:23] <Reedy>	 <3
[17:52:40] <apergos>	 is there an easy way to tell the branch from the zuul monitoring page?
[17:53:22] <apergos>	 (I mean this, it's where I watch normally https://integration.wikimedia.org/zuul/)
[17:53:32] <James_F>	 apergos: No. :-(
[17:53:39] <apergos>	 :-(
[17:53:52] <James_F>	 (Ideally we'd put non-master commits in their own queue, as they can't conflict.)
[17:54:11] <apergos>	 can't they conflict with each other in the same branch?
[17:54:25] <apergos>	 (but I'm all or moving them into another queue anyways!)
[17:54:34] <James_F>	 Sorry, yes, I meant that they can't conflict with master.
[17:54:41] <apergos>	 ah right
[17:54:54] <James_F>	 REL1_31 patches don't care if master works at all, and v.v.
[17:55:16] <apergos>	 anything to make them move along faster (or die sooner :-D)
[17:56:28] <James_F>	 We wouldn't need to segregate them into their own pipeline like we did for "gate-and-submit-swat" (which should really be called "gate-and-submit-production"); that is so that we could give it priority.
[17:56:53] <James_F>	 But we're generally not resource-starved so much as needing to keep the stack manageable and having it fall over with flaky tests.
[17:58:47] <apergos>	 well, anything that can keep merge time down in prod would be a blessing for sure
[17:58:59] * James_F nods.
[18:03:13] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] Revert "[EventBus] Add dependency" [integration/config] - 10https://gerrit.wikimedia.org/r/514841 (https://phabricator.wikimedia.org/T225195) (owner: 10Ottomata)
[18:04:48] <Reedy>	 514777,1 should be the last one
[18:05:11] <wikibugs>	 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: New phan dependencies significantly slowed down CI tests - https://phabricator.wikimedia.org/T225112 (10Umherirrender) To get a overview about previous added dependency for phan - visit https://gerrit.wikimedia.org/r/#/q/status:merged+pr...
[18:05:18] <wikibugs>	 (03Merged) 10jenkins-bot: Revert "[EventBus] Add dependency" [integration/config] - 10https://gerrit.wikimedia.org/r/514841 (https://phabricator.wikimedia.org/T225195) (owner: 10Ottomata)
[18:05:21] <wikibugs>	 10Continuous-Integration-Config, 10CirrusSearch, 10MediaWiki-extensions-Scribunto, 10Move-Files-To-Commons, and 2 others: Add a PHPUnit group to skip test on gated CI runs - https://phabricator.wikimedia.org/T225068 (10debt)
[18:05:42] <wikibugs>	 (03CR) 10Ottomata: "Thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/514841 (https://phabricator.wikimedia.org/T225195) (owner: 10Ottomata)
[18:06:32] <James_F>	 !log Reloading Zuul to drop EventBus dependency on CentralNotice for now T225195
[18:06:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[18:06:34] <stashbot>	 T225195: EventBus jobs failing heavily because of CentralNotice and WikibaseRepo - https://phabricator.wikimedia.org/T225195
[18:11:39] <Reedy>	 great
[18:11:43] <Reedy>	 Now stuff can just be left to merge <3
[18:12:16] <apergos>	 \o/
[18:12:48] <legoktm>	 James_F: er, that's just going to cause phan to fail
[18:13:07] <Reedy>	 and i've fixed the fails in the other branches. so can c+2 them all in one go when CI has done msater
[18:19:00] <James_F>	 legoktm: Most things will break phan. Including breathing.
[18:19:54] <James_F>	 Once ottomata has got CN and EB working on the unit test level with each other, we can re-do it.
[18:20:14] <James_F>	 But I'm waiting for gerrit to confirm that phan actually fails. Half the time it doesn't.
[18:20:39] <ottomata>	 is there a way I can run those tests together locally?
[18:20:51] <ottomata>	 i don't have a lot of mw/phpunit fu
[18:20:51] <wikibugs>	 10Continuous-Integration-Config, 10Operations: Fix operations/puppet.git "rebase hell" - https://phabricator.wikimedia.org/T224033 (10hashar) I am assuming you will be able to talk about this during the SRE offsite next week?
[18:22:18] <wikibugs>	 10Project-Admins: Create a project for Wikilink - https://phabricator.wikimedia.org/T225065 (10Aklapper) @Samwalton9: Feel free to rename via "Edit Details" on https://phabricator.wikimedia.org/project/manage/4082/
[18:22:41] <wikibugs>	 10Continuous-Integration-Config, 10OOUI: Speed up oojs/ui CI job/tests - https://phabricator.wikimedia.org/T189055 (10Volker_E) @hashar Any more things to do here/to leave this open?
[18:22:49] <wikibugs>	 10Continuous-Integration-Infrastructure: Consider having a top-level jenkins CI job for each commit, so they can be manually killed swiftly rather than one-by-one - https://phabricator.wikimedia.org/T225231 (10hashar) Hmm but why? :-]  I am not sure I get the use case for killing jobs?
[18:27:23] <wikibugs>	 10Continuous-Integration-Config, 10OOUI: Speed up oojs/ui CI job/tests - https://phabricator.wikimedia.org/T189055 (10hashar) 05Open→03Resolved a:03hashar Yup that followed up an incident which overloaded CI when "too many" oojs patches got send at the same time. But yeah it is faster now.  Eventually th...
[18:39:02] <wikibugs>	 (03PS1) 10Hashar: PATH env variable is no more needed for zuul cloner [integration/quibble] - 10https://gerrit.wikimedia.org/r/514848
[18:40:26] <wikibugs>	 10Release-Engineering-Team (Backlog), 10MediaWiki-Release-Tools, 10Core Platform Team (Release Automation (TEC1)), 10Core Platform Team Kanban (Contractor - Ready), 10MediaWiki-Releasing (Workflow Improvements): Bot to upgrade the different bits of mw.o for tarba... - https://phabricator.wikimedia.org/T217964
[18:46:53] * paladox ressurects https://gerrit-review.googlesource.com/c/gerrit/+/106713
[18:48:11] <paladox>	 apparently alot of changes since that was done making it a merge conflict :P
[18:49:09] <wikibugs>	 (03CR) 10Hashar: "I am going to rebase this change https://gerrit.wikimedia.org/r/#/c/integration/quibble/+/514848/" (034 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/513377 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight)
[19:01:33] <wikibugs>	 (03Abandoned) 10Hashar: PATH env variable is no more needed for zuul cloner [integration/quibble] - 10https://gerrit.wikimedia.org/r/514848 (owner: 10Hashar)
[19:01:44] <wikibugs>	 (03CR) 10Hashar: Extract zuul clone into a command object (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/513377 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight)
[19:10:27] <wikibugs>	 10Continuous-Integration-Infrastructure: Consider having a top-level jenkins CI job for each commit, so they can be manually killed swiftly rather than one-by-one - https://phabricator.wikimedia.org/T225231 (10Jdforrester-WMF) >>! In T225231#5241400, @hashar wrote: > Hmm but why? :-]  I am not sure I get the use...
[19:11:18] <icinga-wm>	 PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[19:22:33] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "Thank you to have dropped the confusing build_params. PS2 is nicer to me now." [integration/quibble] - 10https://gerrit.wikimedia.org/r/513377 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight)
[19:25:11] <wikibugs>	 (03CR) 10Hashar: Extension and skin submodule update as command (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/513406 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight)
[19:28:01] <wikibugs>	 (03Merged) 10jenkins-bot: Extract zuul clone into a command object [integration/quibble] - 10https://gerrit.wikimedia.org/r/513377 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight)
[19:29:29] <wikibugs>	 (03CR) 10Jforrester: [C: 04-1] Minor copyediting (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/510246 (owner: 10Awight)
[19:29:31] <andrewbogott>	 Reedy: I'm trying to upgrade mediawiki on wikitech-static and composer is telling me "    - The requested package cssjanus/cssjanus 1.2.0 is satisfiable by cssjanus/cssjanus[v1.2.0] but these conflict with your requirements or minimum-stability."
[19:29:38] <andrewbogott>	 any suggestions for where I should look next?
[19:30:09] <James_F>	 andrewbogott: Do you have a local copy of mediawiki/vendor.git or is it built locally?
[19:30:22] <wikibugs>	 (03CR) 10Awight: "> I went with a slight refactoring, but the thin wrapper is nice in" (032 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/513377 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight)
[19:30:42] <andrewbogott>	 James_F: looking
[19:31:06] <James_F>	 (I don't know wikitech-static's setup.)
[19:31:22] <andrewbogott>	 I don't seem to have a local copy.  But, I recently did a 'git clean' in w/vendor
[19:31:28] <andrewbogott>	 so maybe I had it before that :/
[19:32:07] <andrewbogott>	 some rudimentary docs are at https://wikitech.wikimedia.org/wiki/Wikitech-static#What_is_wikitech-static_running?
[19:32:10] <James_F>	 Nah, it sounds like it's a normal local built copy, that's fine.
[19:32:15] <andrewbogott>	 ok
[19:32:33] <James_F>	 What version of MW are you running?
[19:32:43] <andrewbogott>	 I'm trying to upgrade from 1.32 to 1.33
[19:32:44] <James_F>	 We bumped cssjanus recently.
[19:33:08] <andrewbogott>	 following https://wikitech.wikimedia.org/wiki/Wikitech-static#Manual_updates
[19:33:48] <James_F>	 It's possible we back-ported something to 1.33 rc.0 (rc -1?) that requires cssjanus 1.3 instead of 1.2?
[19:34:27] <James_F>	 Oh, right, we back-ported ddd4e4dc047 to 1.33 in https://gerrit.wikimedia.org/r/c/mediawiki/core/+/509561
[19:34:31] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "Tested, seems to work appropriately.  Huge thanks for the test refactoring and splitting the original one!" (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/513406 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight)
[19:35:51] * James_F locally pulls a new REL1_33 branch.
[19:39:14] <James_F>	 Hmm. Composer install works fine (as expected).
[19:39:35] <andrewbogott>	 I probably have a weird selection of components here
[19:40:16] <wikibugs>	 (03CR) 10Awight: Minor copyediting (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/510246 (owner: 10Awight)
[19:40:29] <andrewbogott>	 can I get more details from composer?  so far all I get is:
[19:40:32] <andrewbogott>	 https://www.irccloud.com/pastebin/aepFsFlP/
[19:40:36] <James_F>	 OK, I'll roll back to 1.32.1 and then manually pull up to 1_33.
[19:40:48] <andrewbogott>	 thanks for digging!
[19:41:11] <James_F>	 No, that Just Works™ for me. :-(
[19:41:30] <James_F>	 I'd recommend `rm -rf vendor; rm composer.lock; composer install`.
[19:41:43] <andrewbogott>	 'k
[19:41:47] <James_F>	 Which is a bit nuclear, but should fix any composer gremlins.
[19:42:04] <James_F>	 That error is decidedly unhelpful, isn't it?
[19:42:20] <andrewbogott>	 btw, docs and usage statements always say that 'composer' is a command
[19:42:28] <andrewbogott>	 but I don't have that.  It's the same as ./composer.phar right?
[19:42:34] <James_F>	 Oh, sorry, yes.
[19:42:38] <icinga-wm>	 RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[19:42:50] <James_F>	 Installing composer in your path is common but we don't do that in prod/prod-like environments.
[19:42:58] <andrewbogott>	 'k
[19:43:05] <James_F>	 Allows us to run different versions of composer in different places, I guess?
[19:43:42] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Extension and skin submodule update as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513406 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight)
[19:45:25] <wikibugs>	 (03CR) 10Jforrester: [C: 04-1] Minor copyediting (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/510246 (owner: 10Awight)
[19:45:29] <andrewbogott>	 huh, ./composer.phar install gave me an out of memory error
[19:45:32] <andrewbogott>	 that's not encouraging
[19:45:45] * andrewbogott tries it again
[19:46:37] <legoktm>	 andrewbogott: I know composer is a PITA, but make sure you pass --no-dev to composer install, otherwise it'll install other random crap too
[19:46:47] <andrewbogott>	 ok
[19:47:05] <James_F>	 legoktm: Oh, yes, I'll update the instructions.
[19:47:21] <andrewbogott>	 https://www.irccloud.com/pastebin/p25wWMND/
[19:47:41] <James_F>	 Ah, my advice made it worse? Great.
[19:48:32] <andrewbogott>	 I wiped out /vendor and tried again, now I'm back to just the one error about cssjanus
[19:49:15] <James_F>	 Hmm. REL1_33 has vendor.git as a sub-module.
[19:49:33] <James_F>	 andrewbogott: What does the git log in vendor show?
[19:50:00] <andrewbogott>	 seems normal?  Latest patch is 
[19:50:02] <andrewbogott>	 https://www.irccloud.com/pastebin/0sLFi8UG/
[19:50:42] <James_F>	 Oh!
[19:50:45] <James_F>	 That's definitely wrong.
[19:51:35] <James_F>	 It should be f880b00.
[19:52:27] <James_F>	 Try `rm -rf vendor/; git submodule update vendor`
[19:52:51] <wikibugs>	 (03CR) 10jenkins-bot: Extract zuul clone into a command object [integration/quibble] - 10https://gerrit.wikimedia.org/r/513377 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight)
[19:54:22] <halfak>	 o/ twentyafterfour 
[19:54:25] <halfak>	 I'm looking at https://phabricator.wikimedia.org/T224996
[19:54:37] <halfak>	 Got a few minutes to talk about it?
[19:54:50] <andrewbogott>	 James_F: ok, now I have f880b009d59381a526bbfb0a4d73cb542076f1a8
[19:54:55] <andrewbogott>	 should I try composer update again?
[19:55:03] <James_F>	 andrewbogott: Yes.
[19:55:15] <James_F>	 I get:
[19:55:21] <James_F>	 https://www.irccloud.com/pastebin/XXn8o25c/
[19:56:31] <andrewbogott>	 https://www.irccloud.com/pastebin/9rnE29yR/
[19:56:41] <andrewbogott>	 Sorry about all the sad pastes
[19:57:05] <James_F>	 Hmm.
[19:57:14] <wikibugs>	 (03PS5) 10Hashar: Extension and skin submodule update as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513406 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight)
[19:57:15] <andrewbogott>	 Is this something like me running the composer command in the wrong dir?
[19:57:17] <James_F>	 Is your local composer.json file in MW dirty?
[19:57:27] <James_F>	 `git status` is clean?
[19:57:33] <James_F>	 This is very odd.
[19:58:40] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Browser-Tests, 10Code-Health: Automate identifying flaky tests - https://phabricator.wikimedia.org/T224673 (10kostajh) +17,000 one week later. https://gerrit.wikimedia.org/r/q/recheck,17000
[20:00:07] <andrewbogott>	 git status shows a diff in 'ConfirmEdit' let me see if I can resolve that
[20:01:29] <James_F>	 That won't be affecting composer though.
[20:07:15] <wikibugs>	 10Deployments, 10Release-Engineering-Team (Backlog): Take heat off day before the weekly branch-cut? - https://phabricator.wikimedia.org/T118212 (10greg)
[20:07:30] <wikibugs>	 10Deployments, 10Release-Engineering-Team (Long-Lived-Branches): Take heat off day before the weekly branch-cut? - https://phabricator.wikimedia.org/T118212 (10greg)
[20:08:53] <andrewbogott>	 James_F: I tried rolling back to 1_32 and I'm still getting the same complaint from composer
[20:09:23] <andrewbogott>	 hang on, this might be a disk space issue
[20:09:25] <andrewbogott>	 or related to one
[20:11:45] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Developer Productivity, 10Browser-Tests: Proposal to move browser based tests (Selenium and QUnit) to a non-voting pipeline - https://phabricator.wikimedia.org/T225248 (10kostajh)
[20:16:27] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Developer Productivity, 10Browser-Tests: Proposal to move browser based tests (Selenium and QUnit) to a non-voting pipeline - https://phabricator.wikimedia.org/T225248 (10kostajh)
[20:18:36] <andrewbogott>	 James_F: ok, disk space issue resolved but still getting the same failure.  Here's a clue, though: I rolled back to 1_31 and it worked OK, then upgraded to 1_32 and got that error
[20:18:45] <andrewbogott>	 so the issue appeared in 32 on in 33
[20:18:46] <andrewbogott>	 probably
[20:19:21] <paladox>	 andrewbogott do you have a composer.lock file?
[20:19:46] <andrewbogott>	 paladox: yes
[20:19:47] <wikibugs>	 10Deployments, 10Release-Engineering-Team (Backlog): Static asset time on disk - https://phabricator.wikimedia.org/T140921 (10greg) Neat!
[20:19:54] <wikibugs>	 10Deployments, 10Release-Engineering-Team (Long-Lived-Branches): Static asset time on disk - https://phabricator.wikimedia.org/T140921 (10greg)
[20:19:55] <paladox>	 andrewbogott i would remove it
[20:20:01] <paladox>	 then do "composer install"
[20:20:06] <andrewbogott>	 I've been, it doesn't seem to matter
[20:20:08] <paladox>	 which will regenerate the lock file
[20:20:12] <paladox>	 oh
[20:22:55] <bd808>	 andrewbogott: what's the php version there? `php -v`
[20:23:08] <andrewbogott>	 7.0.33
[20:23:15] <paladox>	 in composer.json does it have '"cssjanus/cssjanus": "1.3.0"'
[20:23:31] <bd808>	 ok, that should work. 
[20:23:48] <andrewbogott>	 is there a way to have composer tell me /what/ the dependency issue is?
[20:24:56] <bd808>	 andrewbogott: if you add -v, -vv, or -vvv you'll get a lot more output
[20:25:10] <bd808>	 -vvv is probably waaay too much
[20:26:15] <andrewbogott>	 It says 'deferring duplicate' for several things, including "Deferring duplicate cssjanus/cssjanus"
[20:26:18] <andrewbogott>	 if that's anything
[20:27:06] <andrewbogott>	 ok, so…composer.json:		"cssjanus/cssjanus": "1.3.0",
[20:27:10] <bd808>	 that means that several things are trying to include the package... which could be the trigger for an unresolvable set of versions
[20:27:19] <andrewbogott>	 and and
[20:27:20] <andrewbogott>	 merge-plugin/extensions-templatestyles-composer.json requires cssjanus/cssjanus (== 1.2.0.0)
[20:27:23] <andrewbogott>	 so that's the conflict
[20:27:35] <paladox>	 i guess that's why
[20:27:57] <bd808>	 yup. has that been updated in templatestyles master?
[20:28:12] <apergos>	 is that new?
[20:28:20] <andrewbogott>	 how does this work on other wikis?  Is templatestyles a weird thing to use?
[20:28:29] <andrewbogott>	 I'm not committed to any particular set of extensions on this thing
[20:28:48] <bd808>	 the main page uses templatestyles
[20:28:54] <paladox>	 i doin't see cssjanus in that extension
[20:29:03] <bd808>	 and yes its on other wikis, but in prod we use a hand built vendor dir
[20:29:09] <paladox>	 https://github.com/wikimedia/mediawiki-extensions-TemplateStyles/commit/0f75a1238baf6f98892e21d1566dcd5b8c402023
[20:29:18] <paladox>	 andrewbogott ^^
[20:29:25] <bd808>	 51086738529dbc442fb9e3d0e83e98e1e933cb00 Remove cssjanus dependancy
[20:29:58] <bd808>	 sounds like you may just need to `git pull` that extension?
[20:30:16] <andrewbogott>	 I've been doing
[20:30:16] <andrewbogott>	 git submodule update --init --recursive
[20:30:21] <andrewbogott>	 which surely does that
[20:30:21] <bd808>	 looks like it should be fixed in master and the REL1_32 branch
[20:30:43] <andrewbogott>	 huh, it's on master
[20:30:46] <bd808>	 only if you have setup all the extensions as submodules
[20:30:53] <andrewbogott>	 yeah
[20:30:58] <andrewbogott>	 apparently this one isn't
[20:31:01] <andrewbogott>	 ok, that may do it, trying again
[20:31:10] <apergos>	 yeah I have the version with the dependency removed indeed
[20:31:26] <apergos>	 and I did get all the extensions as submodules
[20:31:50] <apergos>	 otherwise you are in cleanup hell every time you have to update
[20:31:50] <twentyafterfour>	 halfak: sorry I just saw your ping
[20:32:00] <halfak>	 Hey!  
[20:32:07] <halfak>	 Got a few minutes to talk now?
[20:32:13] <twentyafterfour>	 sure
[20:32:21] <halfak>	 I can't seem to figure out what's going on. :|  
[20:32:58] <twentyafterfour>	 I can't either. Something is wrong with permissions in gerrit but I set everything correctly as far as I can see
[20:33:07] <halfak>	 The "authorization error" in the error message I got when cloning from phab is weird. 
[20:33:24] <halfak>	 You'd think that would work. 
[20:34:06] <twentyafterfour>	 halfak: phabricator doesn't yet support lfs 
[20:34:32] <twentyafterfour>	 technically it does support lfs but our install doesn't have it set up 
[20:38:53] <twentyafterfour>	 I spent some time working on lfs support but we need a storage back-end for it to support large files ... 
[20:39:43] <twentyafterfour>	 that task is T182085 but it's on the back-burner currently
[20:39:44] <stashbot>	 T182085: Connect Phabricator to swift for storage of git-lfs and file uploads. - https://phabricator.wikimedia.org/T182085
[20:40:05] <wikibugs>	 10Continuous-Integration-Infrastructure: Consider having a top-level jenkins CI job for each commit, so they can be manually killed swiftly rather than one-by-one - https://phabricator.wikimedia.org/T225231 (10Reedy) Well, I C+2,V+2'd,submit 20-25 patches total (REL1_27 and REL1_30), because we knew the patches...
[20:40:27] <halfak>	 twentyafterfour, Oh I see!  So that cloning error is unrelated.  Hmm
[20:40:45] <twentyafterfour>	 yeah ...
[20:42:01] <twentyafterfour>	 so when this worked, how did you access lfs? 
[20:42:33] <twentyafterfour>	 Can you just pull from github and then push directly to gerrit, at least for now to get deployments unblocked?
[20:45:57] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] Extension and skin submodule update as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513406 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight)
[20:46:38] <halfak>	 twentyafterfour, good question.  I've been doing that with LFS.  Let me try with regular git. 
[20:47:35] <halfak>	 " ! [remote rejected] master -> master (commit 5d2dec8: invalid committer)"
[20:47:52] <twentyafterfour>	 that's the error I'm getting from replication
[20:48:06] <halfak>	 "url = https://gerrit.wikimedia.org/r/scoring/ores/editquality"
[20:48:10] <halfak>	 If that matters
[20:48:28] <wikibugs>	 (03Merged) 10jenkins-bot: Extension and skin submodule update as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513406 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight)
[20:49:10] <halfak>	 twentyafterfour, I'm able to successfully do "git lfs push gerrit master" FWIW
[20:49:22] <halfak>	 But not "git push gerrit master"
[20:49:37] <twentyafterfour>	 weird!
[20:50:47] <halfak>	 Right!  I mean, what could have changed? 
[20:51:00] * halfak double-checks that everything else is working. 
[20:51:09] <twentyafterfour>	 I think there is a commit that gerrit doesn't like 
[20:51:18] <twentyafterfour>	 what does 5d2dec8 look like
[20:52:13] <wikibugs>	 10Continuous-Integration-Infrastructure: mediawiki-config (and others?) should ride gate-and-submit-swat not gate-and-submit - https://phabricator.wikimedia.org/T225252 (10Jdforrester-WMF)
[20:52:21] <halfak>	 https://phabricator.wikimedia.org/rOEQ5d2dec886e8a3a804cb3b75cae1ee7480b234896
[20:52:24] <halfak>	 Merging a pull request. 
[20:53:02] <halfak>	 Some config and some LFS assets (models)
[20:53:04] <halfak>	 Nothing special
[20:53:36] <twentyafterfour>	 halfak: look at the committer 
[20:53:40] <twentyafterfour>	 it's github
[20:53:47] <twentyafterfour>	 gerrit doesn't recognize the committer 
[20:54:07] <twentyafterfour>	 but even when I enable "forge committer identity" permission it still has that error
[20:54:32] <twentyafterfour>	 I just added forge committer for the research-ores group, can you try again to push it to gerrit?
[20:54:33] <icinga-wm>	 PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 53.33% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[20:54:39] <halfak>	 "Authored by Groceryheist on Tue, Jun 4, 03:16."
[20:55:11] <halfak>	 Could it be that groceryheist needs to be trusted?
[20:55:47] <halfak>	 Strange that we don't have any commits since April 28th though
[20:56:12] <halfak>	 *April 23rd
[20:57:32] <twentyafterfour>	 halfak: there were some other issues that were also breaking replication which were resolved yesterday so that's probably the original reason things broke 
[20:57:54] <twentyafterfour>	 this is a separate problem that popped up after we resolved the other replication problems. 
[20:58:13] <twentyafterfour>	 The issue is that gerrit doesn't like the committer (not author) to be "noreply@github.com"
[20:59:38] <halfak>	 Aha.  Hmm.  Any suggestions? 
[21:00:02] <twentyafterfour>	 I don't know, gerrit has a permission specifically for this but it doesn't seem to be having any affect 
[21:00:14] <twentyafterfour>	 or effect even 
[21:00:48] <halfak>	 gerrit is so emotionless ;) 
[21:01:04] <twentyafterfour>	 do you still get the same error if you try to push now? 
[21:01:42] <halfak>	 It worked!
[21:02:13] * halfak pulls into our deploy repo
[21:02:18] <halfak>	 Yup. Woorking here too
[21:02:58] <halfak>	 twentyafterfour, should we fix anything in groceryheist's config?
[21:03:14] <groceryheist>	 hola
[21:04:09] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Developer Productivity, 10Browser-Tests: Proposal to move browser based tests (Selenium and QUnit) to a non-voting pipeline - https://phabricator.wikimedia.org/T225248 (10SBisson) In the last week, Kosta measured about 200 times when "recheck" was used on a patch. W...
[21:04:15] <halfak>	 o/ groceryheist.  We had a weird thing in gerrit because of how github sets the "committer" for the pull requests you merged. 
[21:04:31] <halfak>	 Luckily twentyafterfour was able to solve the issue, but we might need to fix some github config. 
[21:04:47] <groceryheist>	 ok
[21:06:33] <halfak>	 See https://phabricator.wikimedia.org/rOEQ5d2dec886e8a3a804cb3b75cae1ee7480b234896
[21:06:45] <halfak>	 Apparently the "committer" is set to github noreply. 
[21:07:20] <halfak>	 Do you have anything here? https://phabricator.wikimedia.org/settings/user/Groceryheist/page/email/
[21:07:47] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Wikimedia-Site-requests, 10MW-1.34-notes (1.34.0-wmf.7; 2019-05-28): Change Squid references in Wikimedia configuration files - https://phabricator.wikimedia.org/T104148 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF
[21:09:12] <halfak>	 groceryheist, ^ 
[21:09:33] <halfak>	 Oh wwiat.  That probably doesn't matter
[21:10:37] <halfak>	 twentyafterfour, note the committer here https://phabricator.wikimedia.org/rOEQ373b4c143ec4591eaed474a57ad06333af19fc16  (It's also github noreply)
[21:10:54] <halfak>	 And here: https://phabricator.wikimedia.org/rOEQc8f1aa825ab009833fd0c3948ae442123db07b90
[21:11:12] <paladox>	 it would likley be due to a user hiding there email in github
[21:13:58] <halfak>	 But it looks like it didn't cause problems in the past. 
[21:14:10] <halfak>	 groceryheist, looks like there's nothing for you to do.  Sorry for the false alarm. 
[21:18:21] <halfak>	 twentyafterfour, shall we resolve https://phabricator.wikimedia.org/T224996 or is there more you think we should do?
[21:18:43] <wikibugs>	 (03CR) 10jenkins-bot: Extension and skin submodule update as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513406 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight)
[21:18:47] <wikibugs>	 10Zuul: Extract zuul-cloner into standalone library - https://phabricator.wikimedia.org/T225254 (10awight)
[22:00:08] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Developer Productivity, 10Browser-Tests: Proposal to move browser based tests (Selenium and QUnit) to a non-voting pipeline - https://phabricator.wikimedia.org/T225248 (10awight) I like this proposal, but I'm concerned that non-voting tests will become meaningless,...
[22:10:40] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Developer Productivity, 10Browser-Tests: Proposal to move browser based tests (Selenium and QUnit) to a non-voting pipeline - https://phabricator.wikimedia.org/T225248 (10Legoktm) QUnit tests are not browser tests, those are selenium tests.  If tests are flaky, then...
[22:16:54] <wikibugs>	 10Continuous-Integration-Infrastructure: mediawiki-config (and others?) should ride gate-and-submit-swat not gate-and-submit - https://phabricator.wikimedia.org/T225252 (10thcipriani) > gate-and-submit-swat is for immediate-to-prod code. Yes, mw-config is in its own queue, but it should also have priority…  FWIW...
[22:20:05] <icinga-wm>	 RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[22:25:05] <James_F>	 thcipriani: Oh, I was told -swat had higher priority and that was why we made its own pipeline. Couldn't we just make a queue for wmf/ branches inside gate-and-submit?
[22:25:53] <thcipriani>	 IIRC that has something to do with the fact that each of these are dependant pipelines
[22:26:19] <thcipriani>	 something something: sharing tests means they share a queue in the pipeline, regardless of branch
[22:27:11] <thcipriani>	 hash<tab><tab> is the person who explained it to me once upon a time, my explanation is obviously lacking in nuance :)
[22:28:54] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Developer Productivity, 10Browser-Tests: Proposal to move browser based tests (Selenium and QUnit) to a non-voting pipeline - https://phabricator.wikimedia.org/T225248 (10Jrbranaa) I am in favor of leaving them as voting, especially if the primary driver to changing...
[22:29:36] <thcipriani>	 wmf/ branches used to get queued behind master: which didn't make any sense; hence the -swat pipeline, but it was difficult to move mw-config, since it only used the master branch (rather than wmf branches), but that was OK since it got its own queue.
[22:32:31] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Browser-Tests, 10Code-Health: Automate identifying flaky tests - https://phabricator.wikimedia.org/T224673 (10Jrbranaa) Do we have a sense of what is causing these rechecks?  @zeljkofilipin, have any insight into this.  I know we've discussed it a little in the past...
[22:52:49] * James_F nods.
[23:00:41] <wikibugs>	 10Release-Engineering-Team, 10MediaWiki-extensions-CentralNotice, 10Security: CentralNotice setting a surprising content security policy in production - https://phabricator.wikimedia.org/T225261 (10Jdforrester-WMF)
[23:05:37] <wikibugs>	 10Release-Engineering-Team, 10MediaWiki-extensions-CentralNotice, 10Security: CentralNotice setting a surprising content security policy in production - https://phabricator.wikimedia.org/T225261 (10Jdforrester-WMF) If nothing else, `*.wikimedia.org` is pretty bad. But mostly this is surprising that this is h...
[23:06:19] <wikibugs>	 10Release-Engineering-Team, 10MediaWiki-extensions-CentralNotice, 10Security: CentralNotice setting a surprising content security policy in production - https://phabricator.wikimedia.org/T225261 (10Jdforrester-WMF) `16:03:55 <RoanKattouw> James_F: I can find no reason why this wouldn't have been working this...
[23:13:21] <icinga-wm>	 PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[23:15:21] <wikibugs>	 10Release-Engineering-Team, 10MediaWiki-extensions-CentralNotice, 10Security: CentralNotice setting a surprising content security policy in production - https://phabricator.wikimedia.org/T225261 (10Catrope) The code from CentralNotice that does this: https://github.com/wikimedia/mediawiki-extensions-CentralN...
[23:20:35] <icinga-wm>	 RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[23:24:37] <wikibugs>	 10Release-Engineering-Team, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Security: CentralNotice setting a surprising content security policy in production - https://phabricator.wikimedia.org/T225261 (10AndyRussG) Indeed this should be fixed... The point of this was to warn people testing...
[23:46:37] <wikibugs>	 10Release-Engineering-Team, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Security: CentralNotice setting a surprising content security policy in production - https://phabricator.wikimedia.org/T225261 (10Jdforrester-WMF) Specifically, this was found by @notconfusing on https://pl.wikipedia...