[01:07:49] 10Phabricator, 10Developer-Advocacy, 10Epic: (Semi)automatically close Phabricator tickets with status "stalled" after a while - https://phabricator.wikimedia.org/T252522 (10Aklapper) [01:20:02] Project beta-update-databases-eqiad build #43657: 04FAILURE in 1.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/43657/ [02:20:02] Project beta-update-databases-eqiad build #43658: 04STILL FAILING in 1.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/43658/ [03:20:01] Project beta-update-databases-eqiad build #43659: 04STILL FAILING in 1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/43659/ [04:20:02] Project beta-update-databases-eqiad build #43660: 04STILL FAILING in 1.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/43660/ [05:20:05] Project beta-update-databases-eqiad build #43661: 04STILL FAILING in 4.6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/43661/ [06:18:29] 10Gerrit, 10Patch-For-Review: Use WMF Gerrit's logo as favicon - https://phabricator.wikimedia.org/T257218 (10Tgr) >>! In T257218#6316493, @Krinkle wrote: > I'd recommend filling in the internal blanks with white pixels to make it stand out more in typical browser tabs That might work well in night mode, not... [06:20:02] Project beta-update-databases-eqiad build #43662: 04STILL FAILING in 1.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/43662/ [06:33:54] (03CR) 10Jeena Huneidi: "Nice! Will do some testing tomorrow" (031 comment) [blubber] - 10https://gerrit.wikimedia.org/r/613701 (https://phabricator.wikimedia.org/T254629) (owner: 10Ahmon Dancy) [06:37:12] (03CR) 10Jeena Huneidi: "> Patch Set 2:" (031 comment) [blubber] - 10https://gerrit.wikimedia.org/r/613701 (https://phabricator.wikimedia.org/T254629) (owner: 10Ahmon Dancy) [07:04:58] 10MediaWiki-Codesniffer: Create PHPCS/ESLint rule to disallow `if else` blocks with an unnamed block likely intended as `else` - https://phabricator.wikimedia.org/T186939 (10thiemowmde) Could be as easy as checking for a `}` token directly followed by `{` (skipping optional whitespace). Is there any valid PHP sy... [07:20:02] Project beta-update-databases-eqiad build #43663: 04STILL FAILING in 1.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/43663/ [08:20:02] Project beta-update-databases-eqiad build #43664: 04STILL FAILING in 1.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/43664/ [08:23:30] https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php72-docker/51610/consoleFull [08:23:35] 08:08:24 rsync: failed to set times on "/cache/.": Operation not permitted (1) [08:23:35] 08:51:57 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.2] [08:23:41] the castor rsync took ~40 minutes [08:32:45] PROBLEM - Free space - all mounts on integration-agent-docker-1003 is CRITICAL: CRITICAL: integration.integration-agent-docker-1003.diskspace._srv.byte_percentfree (<11.11%) [08:42:47] RECOVERY - Free space - all mounts on integration-agent-docker-1003 is OK: OK: All targets OK [09:20:02] Project beta-update-databases-eqiad build #43665: 04STILL FAILING in 1.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/43665/ [09:26:51] 10Phabricator, 10Quality-and-Test-Engineering-Team (QTE), 10Patch-For-Review, 10User-Vidhi-Mody, 10User-zeljkofilipin: Upgrade WebdriverIO in the phab-deployment repository - https://phabricator.wikimedia.org/T255471 (10zeljkofilipin) [09:26:53] 10Phabricator, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Quality-and-Test-Engineering-Team (QTE), 10Patch-For-Review, and 2 others: Error in pushing code to Gerrit for the phab-deployment repository - https://phabricator.wikimedia.org/T255761 (10zeljk... [09:27:58] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Wikimedia-Logstash, 10observability, and 2 others: Regression test for Logstash filters - https://phabricator.wikimedia.org/T251869 (10fgiunchedi) 05Open→... [09:29:47] 10Phabricator, 10Quality-and-Test-Engineering-Team (QTE), 10Patch-For-Review, 10User-Vidhi-Mody, 10User-zeljkofilipin: Upgrade WebdriverIO in the phab-deployment repository - https://phabricator.wikimedia.org/T255471 (10zeljkofilipin) [09:33:04] 10Phabricator, 10Quality-and-Test-Engineering-Team (QTE), 10Patch-For-Review, 10User-Vidhi-Mody, 10User-zeljkofilipin: Upgrade WebdriverIO in the phab-deployment repository - https://phabricator.wikimedia.org/T255471 (10zeljkofilipin) [09:35:18] 10Phabricator, 10Quality-and-Test-Engineering-Team (QTE), 10Patch-For-Review, 10User-Vidhi-Mody, 10User-zeljkofilipin: Upgrade WebdriverIO in the phab-deployment repository - https://phabricator.wikimedia.org/T255471 (10zeljkofilipin) a:03Vidhi-Mody [10:20:01] Project beta-update-databases-eqiad build #43666: 04STILL FAILING in 1.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/43666/ [10:30:29] maintenance-disconnect-full-disks build 196080 integration-agent-docker-1001 (/srv: 98%): OFFLINE due to disk space [10:32:23] PROBLEM - Free space - all mounts on integration-agent-docker-1001 is CRITICAL: CRITICAL: integration.integration-agent-docker-1001.diskspace._srv.byte_percentfree (<22.22%) [10:42:23] RECOVERY - Free space - all mounts on integration-agent-docker-1001 is OK: OK: All targets OK [10:55:45] maintenance-disconnect-full-disks build 196085 integration-agent-docker-1001: OFFLINE due to disk space [10:55:45] maintenance-disconnect-full-disks build 196085 integration-agent-docker-1012 (/srv: 96%): OFFLINE due to disk space [10:59:43] PROBLEM - Free space - all mounts on integration-agent-docker-1012 is CRITICAL: CRITICAL: integration.integration-agent-docker-1012.diskspace._srv.byte_percentfree (<30.00%) [11:05:11] PROBLEM - Free space - all mounts on integration-agent-docker-1007 is CRITICAL: CRITICAL: integration.integration-agent-docker-1007.diskspace._srv.byte_percentfree (<11.11%) [11:09:44] RECOVERY - Free space - all mounts on integration-agent-docker-1012 is OK: OK: All targets OK [11:15:12] RECOVERY - Free space - all mounts on integration-agent-docker-1007 is OK: OK: All targets OK [11:20:02] Project beta-update-databases-eqiad build #43667: 04STILL FAILING in 1.6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/43667/ [11:20:32] maintenance-disconnect-full-disks build 196090 integration-agent-docker-1001: OFFLINE due to disk space [11:20:32] maintenance-disconnect-full-disks build 196090 integration-agent-docker-1012: OFFLINE due to disk space [11:22:14] 10Project-Admins, 10Operations: Rename #Operations Phab project to #WMF-SRE (or so) - https://phabricator.wikimedia.org/T258305 (10Joe) p:05Triage→03Low [11:45:27] maintenance-disconnect-full-disks build 196095 integration-agent-docker-1001: OFFLINE due to disk space [11:45:27] maintenance-disconnect-full-disks build 196095 integration-agent-docker-1012: OFFLINE due to disk space [12:00:14] Does the Blubber CD pipeline allow for pre-processing of some sort? In order to build our images we need to prepare things, e.g. download dependencies, build java binaries, populate sqlite databases, and what not. Is such a prepare-build stage allowed (or even supported)? [12:04:46] UnhandledPromiseRejectionWarning: Error: ENOSPC: no space left on device, mkdir '/workspace/log/rawSeleniumVideoGrabs' [12:10:28] maintenance-disconnect-full-disks build 196100 integration-agent-docker-1001: OFFLINE due to disk space [12:10:28] maintenance-disconnect-full-disks build 196100 integration-agent-docker-1012: OFFLINE due to disk space [12:20:02] Project beta-update-databases-eqiad build #43668: 04STILL FAILING in 1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/43668/ [12:25:11] 10Release-Engineering-Team, 10Analytics, 10Product-Analytics, 10Repository-Admins: Create a repository and user for Product Analytics Oozie jobs - https://phabricator.wikimedia.org/T230743 (10mpopov) a:05mpopov→03elukey @elukey: Is it okay if I assign you for the user creation part? (The repository par... [12:26:46] 10Release-Engineering-Team, 10Analytics, 10Product-Analytics, 10Repository-Admins: Create a repository and user for Product Analytics Oozie jobs - https://phabricator.wikimedia.org/T230743 (10mpopov) [12:27:26] 10Release-Engineering-Team, 10Analytics, 10Product-Analytics, 10Repository-Admins: Create a repository and user for Product Analytics Oozie jobs - https://phabricator.wikimedia.org/T230743 (10mpopov) [12:35:25] maintenance-disconnect-full-disks build 196105 integration-agent-docker-1001: OFFLINE due to disk space [12:35:26] maintenance-disconnect-full-disks build 196105 integration-agent-docker-1012: OFFLINE due to disk space [12:43:25] PROBLEM - Free space - all mounts on integration-agent-docker-1011 is CRITICAL: CRITICAL: integration.integration-agent-docker-1011.diskspace._srv.byte_percentfree (<11.11%) [12:53:28] RECOVERY - Free space - all mounts on integration-agent-docker-1011 is OK: OK: All targets OK [13:00:32] maintenance-disconnect-full-disks build 196110 integration-agent-docker-1001: OFFLINE due to disk space [13:00:33] maintenance-disconnect-full-disks build 196110 integration-agent-docker-1012: OFFLINE due to disk space [13:20:02] Project beta-update-databases-eqiad build #43669: 04STILL FAILING in 1.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/43669/ [13:21:04] 10Continuous-Integration-Config, 10phan, 10MediaWiki-Core-Testing, 10MW-1.31-release, 10Patch-For-Review: Decide on future of running Phan tests on release branches - https://phabricator.wikimedia.org/T226945 (10Aklapper) [13:25:26] maintenance-disconnect-full-disks build 196115 integration-agent-docker-1001: OFFLINE due to disk space [13:25:27] maintenance-disconnect-full-disks build 196115 integration-agent-docker-1012: OFFLINE due to disk space [13:26:26] 10Project-Admins: Create a #Wikimedia-User-Group-China group project - https://phabricator.wikimedia.org/T172163 (10Aklapper) @Shizhao: Looking at activity on https://phabricator.wikimedia.org/project/profile/2988/ this project tag does not seem to be much used. Is this project tag still active? [13:50:25] maintenance-disconnect-full-disks build 196120 integration-agent-docker-1001: OFFLINE due to disk space [13:50:26] maintenance-disconnect-full-disks build 196120 integration-agent-docker-1012: OFFLINE due to disk space [13:52:31] 10Phabricator, 10Quality-and-Test-Engineering-Team (QTE), 10Patch-For-Review, 10User-Vidhi-Mody, 10User-zeljkofilipin: Upgrade WebdriverIO in the phab-deployment repository - https://phabricator.wikimedia.org/T255471 (10zeljkofilipin) [14:01:57] 10Release-Engineering-Team, 10Analytics, 10Product-Analytics, 10Repository-Admins: Create a repository and user for Product Analytics Oozie jobs - https://phabricator.wikimedia.org/T230743 (10elukey) it is yes, I'll bring this up during the next SRE meeting! Thanks for the patience :) [14:15:39] maintenance-disconnect-full-disks build 196125 integration-agent-docker-1001: OFFLINE due to disk space [14:15:39] maintenance-disconnect-full-disks build 196125 integration-agent-docker-1012: OFFLINE due to disk space [14:21:56] Yippee, build fixed! [14:21:56] Project beta-update-databases-eqiad build #43670: 09FIXED in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/43670/ [14:35:10] what's the process of adding a new extension to production wikis? [14:35:28] James_F, would you be able to point me at something re the above? [14:37:00] liw: Most of it is on https://wikitech.wikimedia.org/wiki/How_to_deploy_code [14:37:06] >Case 1c: new submodule (extension, skin, etc.) [14:37:07] etc [14:37:17] Or just see my patches to do it and copy them. [14:37:23] liw: Which extension? [14:37:25] Also, https://www.mediawiki.org/wiki/Writing_an_extension_for_deployment [14:37:34] Yeah, that too. [14:38:30] James_F, Reedy, question triggered by https://gerrit.wikimedia.org/r/c/mediawiki/tools/release/+/614804 [14:38:49] should there be a Phabricator ticket for adding a new extension? [14:38:52] liw: Essentially: 1) Be happy for it to go into prod. 2) Add to tools-releases list. [wait 'til both trains have the new extension submodule] 3) Add to prod config for i18n. 4) Add some IS and CS config to load it. 5) Start enabling it, ideally on Beta Cluster first. [14:38:55] liw: Yes. [14:39:13] Which T258423 is. [14:39:26] The checklist there is about right. [14:39:41] Indeed, it's better than I'd do off the top of my head. [14:40:14] The task says Tyler's agreed for the branch cut to happen now. [14:40:20] T258423: Deploy OAuthRateLimiter extension to Wikimedia Production - https://phabricator.wikimedia.org/T258423 [14:40:25] maintenance-disconnect-full-disks build 196130 integration-agent-docker-1001: OFFLINE due to disk space [14:40:26] maintenance-disconnect-full-disks build 196130 integration-agent-docker-1012: OFFLINE due to disk space [14:40:49] James_F, you're also a reviewer for the change, do you want to CR+2 or shall I? [14:40:59] https://gerrit.wikimedia.org/g/mediawiki/extensions/OAuthRateLimiter/+/refs/heads/master looks fine to me [14:41:11] liw: Go ahead, I'm just a poseur, you're the real deal. ;-) [14:41:27] James_F, I think you mistake me for you [14:41:37] (03CR) 10Lars Wirzenius: [C: 03+2] "LGTM" [tools/release] - 10https://gerrit.wikimedia.org/r/614804 (https://phabricator.wikimedia.org/T258423) (owner: 10Ppchelko) [14:42:28] (03Merged) 10jenkins-bot: Start branching OAuthRateLimiter extension for production [tools/release] - 10https://gerrit.wikimedia.org/r/614804 (https://phabricator.wikimedia.org/T258423) (owner: 10Ppchelko) [14:47:33] James_F, Reedy, thanks, that's dealt with (ping Pchelolo) [14:48:06] thank you! did I make it into this week branch cut? [14:52:30] Pchelolo, I think that happened automatically at 02 UTC, so about twelve hours ago :( [14:53:10] ok, thank you. It doesn't really matter, we're not in a rush [14:53:21] not as if there's any code that actually does anything yet merged ;P [14:53:58] Reedy: there's test code that asserts that true is true [14:54:07] that's pretty important in my opinion [14:54:12] heh [14:54:37] if we the basics aren't right... [15:02:41] (03PS1) 10MSantos: Revert "jjb: Codehealth pipeline NodeJS configuration" [integration/config] - 10https://gerrit.wikimedia.org/r/615193 [15:02:54] (03CR) 10jerkins-bot: [V: 04-1] Revert "jjb: Codehealth pipeline NodeJS configuration" [integration/config] - 10https://gerrit.wikimedia.org/r/615193 (owner: 10MSantos) [15:05:28] maintenance-disconnect-full-disks build 196135 integration-agent-docker-1001: OFFLINE due to disk space [15:05:28] maintenance-disconnect-full-disks build 196135 integration-agent-docker-1012: OFFLINE due to disk space [15:24:22] (03PS3) 10Thcipriani: Prevent container leak if docker build fails [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/614851 (https://phabricator.wikimedia.org/T235680) (owner: 10Ahmon Dancy) [15:30:31] maintenance-disconnect-full-disks build 196140 integration-agent-docker-1001: OFFLINE due to disk space [15:30:32] maintenance-disconnect-full-disks build 196140 integration-agent-docker-1012: OFFLINE due to disk space [15:40:26] maintenance-disconnect-full-disks build 196142 integration-agent-docker-1011 (/srv: 100%): OFFLINE due to disk space [15:47:44] !log intergration-agent-docker-1001 back online [15:48:15] (03PS3) 10Ahmon Dancy: Fix handling of indirect 'copies' references [blubber] - 10https://gerrit.wikimedia.org/r/613701 (https://phabricator.wikimedia.org/T254629) [15:48:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:49:23] (03CR) 10Ahmon Dancy: "> Patch Set 2:" (031 comment) [blubber] - 10https://gerrit.wikimedia.org/r/613701 (https://phabricator.wikimedia.org/T254629) (owner: 10Ahmon Dancy) [15:55:26] maintenance-disconnect-full-disks build 196145 integration-agent-docker-1011: OFFLINE due to disk space [15:55:26] maintenance-disconnect-full-disks build 196145 integration-agent-docker-1012: OFFLINE due to disk space [16:17:56] Hi dear releng. I have a question: do we have a capability to deploy a fork of a composer package? We only want to have it temporarily, we are working on getting the code merged upstream, but it's a very slow-moving process [16:19:00] I'm not proposing we do it, just wanna know if we have the capability and what's involved [16:20:37] maintenance-disconnect-full-disks build 196150 integration-agent-docker-1011: OFFLINE due to disk space [16:20:37] maintenance-disconnect-full-disks build 196150 integration-agent-docker-1012: OFFLINE due to disk space [16:28:33] Pchelolo: Yeah, it's not too difficult [16:29:04] You can mess around with repositories for a package... Or publish to packagist over on another name, and remporarily replace etc [16:29:50] yeah, with repositories we've added github for testing for now, but I don't think it's a great idea to deploy stuff from github directly [16:30:05] do we have our own composer repo hosted somewhere? [16:30:19] 10Phabricator, 10Security-Team, 10Security: Everyone can view a restricted file - https://phabricator.wikimedia.org/T258239 (10sbassett) p:05Triage→03Low [16:32:03] You mean a registry? No [16:32:18] yeah, registry [16:33:12] I mean, most composer packages via composer are basically deployed from github anyway [16:33:30] just for WMF prod, we just use the mediawiki/vendor repo for our sanitisation/monitoring of it [16:33:40] mmm, I see. [16:34:12] Pchelolo: how big is the patch? you can just edit mediawiki/vendor directly if it's small [16:34:30] it's pretty big. couple hundred lines [16:34:56] link? [16:35:14] https://github.com/thephpleague/oauth2-server/pull/1122 [16:35:14] composer also lets us depend on specific git branches/versions, which we can do in mw/vendor [16:36:08] thank you. ok, so there's options, that's good. I am mostly doing some very preliminary research here, I still hope we get this merged upstream soon enough to not need any of it [16:36:45] see https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/vendor/+/de222557e70af529c3fca71103361ef8b4b84520%5E%21/#F0 for elasticsearch [16:37:12] saved me digging htat out :P [16:37:58] thank you a lot. that's very helpful [16:45:35] maintenance-disconnect-full-disks build 196155 integration-agent-docker-1011: OFFLINE due to disk space [16:45:35] maintenance-disconnect-full-disks build 196155 integration-agent-docker-1012: OFFLINE due to disk space [16:51:50] Hey all, I have a quick train-related question for the RelEng team if anyone's around [16:53:09] EricGardner: Probably easiest to just ask your question :) [16:54:39] Ha, ok. I'm on the Structured Data team. We have a patch for the MediaInfo extension which has been merged, but never got tagged for a train deployment: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikibaseMediaInfo/+/610918/ [16:54:48] phab ticket: https://phabricator.wikimedia.org/T251940 [16:55:17] What do you mean tagged? [16:55:20] It's listed as being in wmf/1.36.0-wmf.1 [16:55:38] So should be on all WMF wikis by the end of the week... [16:55:55] And as it was merged last night, that sounds as expected [16:56:10] ok, great! sorry, I was looking on the relevant Phab ticket and didn't see a tag with the release [16:56:23] That's only done for backports, I think [16:56:40] Rather than stuff that just rides the train more normally [16:57:21] cool. What's the best way to check for this in the future? I was looking at the deployments page but I only see info for this week there [16:57:36] I'm still a bit confused about where to look for some of this information [16:57:51] So as of a few weeks ago, the train branching time is now fixed [16:58:58] EricGardner: Gerrit has an "included in" button that shows you which branches the patch is in. Then check https://versions.toolforge.org/ for the status of the train [16:59:04] Apparently 12 hours ago https://github.com/wikimedia/mediawiki/commit/4f21a6c9320185e6c4e8d397d64282648c0f00a1 [16:59:20] So, anything that's merged into master before the branch point, will go out with the train that week [17:00:26] in the past it seemed like most of our tickets got a tag in phabricator indicating that they'd made the branch cut. Some non-developers on our team were asking why this didn't get one [17:01:28] sounds like going forward I should just point folks to the roadmap [17:01:29] oh hm [17:01:36] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO, 10Code-Stewardship-Reviews: deployment-prep: Code stewardship request - https://phabricator.wikimedia.org/T215217 (10Mholloway) [17:01:38] 10Beta-Cluster-Infrastructure: deployment-puppetmaster04: git-sync-upstream is failing with a merge conflict since 2020-07-17T08:50:01Z - https://phabricator.wikimedia.org/T258451 (10Mholloway) [17:02:04] Is it only done for certain repos? [17:02:15] https://github.com/wikimedia/mediawiki/commit/4f21a6c9320185e6c4e8d397d64282648c0f00a1 [17:02:34] releasetaggerbot should've tagged that task [17:03:23] > Exception: No PHID found for slug #mw1.36.0-wmf.2! [17:03:39] lol [17:04:18] James_F: ^^ [17:04:52] does this mean ReleaseTaggerBot messed up here? [17:05:27] no, James created the project wrong [17:06:00] I fixed the hashtags on https://phabricator.wikimedia.org/project/manage/4885/ [17:06:22] EricGardner: but yes, the bot should've tagged your task. thanks for pointing that out [17:07:12] just triggered a run now [17:09:37] I think it lost all the previous changes unfortunately [17:10:27] maintenance-disconnect-full-disks build 196160 integration-agent-docker-1011: OFFLINE due to disk space [17:10:28] maintenance-disconnect-full-disks build 196160 integration-agent-docker-1012: OFFLINE due to disk space [17:33:17] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments: 1.36.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T257970 (10thcipriani) p:05Triage→03Medium a:03LarsWirzenius [17:34:49] (03CR) 10Brennen Bearnes: "This change is ready for review." [integration/config] - 10https://gerrit.wikimedia.org/r/614879 (https://phabricator.wikimedia.org/T258448) (owner: 10Brennen Bearnes) [17:35:32] maintenance-disconnect-full-disks build 196165 integration-agent-docker-1011: OFFLINE due to disk space [17:35:32] maintenance-disconnect-full-disks build 196165 integration-agent-docker-1012: OFFLINE due to disk space [17:43:46] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: HTTP CRITICAL: HTTP/1.1 302 Found - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 716 bytes in 0.117 second response time [17:52:32] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments: 1.36.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T257971 (10thcipriani) a:03brennen [17:52:50] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments: 1.36.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T257971 (10thcipriani) p:05Triage→03Medium [17:56:49] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Jenkins, 10User-brennen: maintenance-disconnect-full-disks: Bring nodes back online when disk space iss... - https://phabricator.wikimedia.org/T258515 [18:00:26] maintenance-disconnect-full-disks build 196170 integration-agent-docker-1011: OFFLINE due to disk space [18:00:27] maintenance-disconnect-full-disks build 196170 integration-agent-docker-1012: OFFLINE due to disk space [18:01:33] (03PS2) 10Brennen Bearnes: DNM: maintenance-disconnect-full-disks: bring fixed nodes online [integration/config] - 10https://gerrit.wikimedia.org/r/614899 (https://phabricator.wikimedia.org/T258515) [18:01:37] (03PS2) 10Brennen Bearnes: WIP: maintenance-disconnect-full-disks: remove old workspaces [integration/config] - 10https://gerrit.wikimedia.org/r/614879 (https://phabricator.wikimedia.org/T258448) [18:02:54] 10Beta-Cluster-Infrastructure, 10Operations: deployment-puppetmaster04: git-sync-upstream is failing with a merge conflict since 2020-07-17T08:50:01Z - https://phabricator.wikimedia.org/T258451 (10Krinkle) /cc @elukey @jbond per . [18:06:06] EricGardner: yeah the "included in" indicator on Gerrit is authorotative. The changelogs linked from https://www.mediawiki.org/wiki/MediaWiki_1.35/Roadmap are also a good one (bit easier to use). The phab tags are best-effort and not always as obvious or correct, but most of the time are :) [18:09:13] longma: I have a fix for a pretty annoying regression from wmf.40 https://phabricator.wikimedia.org/T257766 . Is it OK for me to deploy that now-ish, shortly before the train, or would you prefer I wait until the evening backport window? [18:09:37] Thanks – I didn't know about "included in" in Gerrit. There is a lot hiding in the gerrit UI that I'm still not familiar with [18:10:19] RoanKattouw: I am in the middle of deploying to testwikis but I'll let you know when I'm done [18:15:24] !log iterating on test of maintenance-disconnect-full-disks for T258515 [18:15:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:15:27] T258515: maintenance-disconnect-full-disks: Bring nodes back online when disk space issues are fixed - https://phabricator.wikimedia.org/T258515 [18:25:14] maintenance-disconnect-full-disks build 196176 integration-agent-docker-1011 (/: 18%): RECOVERY disk space OK [18:30:11] maintenance-disconnect-full-disks build 196178 integration-agent-docker-1012 (): RECOVERY disk space OK [18:36:22] RoanKattouw: I've finished the deploy so you can go ahead with your fix when ready [18:36:29] Thanks! [18:41:41] maintenance-disconnect-full-disks build 196182 integration-agent-docker-1004 (/: 18%, /srv: 7%, /var/lib/docker: 33%): OFFLINE due to disk space [18:42:05] maintenance-disconnect-full-disks build 196183 integration-agent-docker-1004 (/: 18%, /srv: 7%, /var/lib/docker: 33%): RECOVERY disk space OK [18:42:11] ^ intentional. [18:44:39] (03PS3) 10Ahmon Dancy: DNM: maintenance-disconnect-full-disks: bring fixed nodes online [integration/config] - 10https://gerrit.wikimedia.org/r/614899 (https://phabricator.wikimedia.org/T258515) (owner: 10Brennen Bearnes) [18:45:43] (03CR) 10Brennen Bearnes: "This change is ready for review." [integration/config] - 10https://gerrit.wikimedia.org/r/614899 (https://phabricator.wikimedia.org/T258515) (owner: 10Brennen Bearnes) [18:46:00] (03PS4) 10Ahmon Dancy: maintenance-disconnect-full-disks: bring fixed nodes online [integration/config] - 10https://gerrit.wikimedia.org/r/614899 (https://phabricator.wikimedia.org/T258515) (owner: 10Brennen Bearnes) [19:09:40] (03CR) 10Brennen Bearnes: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/614899 (https://phabricator.wikimedia.org/T258515) (owner: 10Brennen Bearnes) [19:11:26] (03CR) 10Kosta Harlan: [C: 03+1] Revert "jjb: Codehealth pipeline NodeJS configuration" [integration/config] - 10https://gerrit.wikimedia.org/r/615193 (owner: 10MSantos) [19:12:57] (03CR) 10Brennen Bearnes: [C: 03+2] "Paired with Ahmon here, merging as currently deployed. Will be monitoring for the remainder of today." [integration/config] - 10https://gerrit.wikimedia.org/r/614899 (https://phabricator.wikimedia.org/T258515) (owner: 10Brennen Bearnes) [19:14:04] (03Merged) 10jenkins-bot: maintenance-disconnect-full-disks: bring fixed nodes online [integration/config] - 10https://gerrit.wikimedia.org/r/614899 (https://phabricator.wikimedia.org/T258515) (owner: 10Brennen Bearnes) [19:18:23] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Jenkins, and 2 others: maintenance-disconnect-full-disks: Bring nodes back online... - https://phabricator.wikimedia.org/T258515 [19:24:50] how come https://gerrit.wikimedia.org/r/c/mediawiki/tools/cli/+/614002 and child patches aren't merging? Gerrit just shows "ready to submit" [19:55:36] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Jenkins, and 2 others: maintenance-disconnect-full-disks: Bring nodes back online... - https://phabricator.wikimedia.org/T258515 [20:10:30] 10Phabricator: Phab doesn't respect "Request Desktop Website" - https://phabricator.wikimedia.org/T258529 (10Reedy) [20:14:41] 10Release-Engineering-Team, 10Platform Team, 10Platform Team Initiatives (API Gateway): Deploy OAuthRateLimiter extension to Wikimedia Production - https://phabricator.wikimedia.org/T258423 (10eprodromou) [20:15:18] 10Release-Engineering-Team, 10Platform Team Initiatives (API Gateway), 10Platform Team Workboards (Green): Deploy OAuthRateLimiter extension to Wikimedia Production - https://phabricator.wikimedia.org/T258423 (10eprodromou) [20:30:06] (03PS7) 10Jeena Huneidi: [DNM] This is a test [blubber-doc/example/helloworldoid] (kubernetes-tutorial) - 10https://gerrit.wikimedia.org/r/612410 [20:30:42] (03CR) 10jerkins-bot: [V: 04-1] [DNM] This is a test [blubber-doc/example/helloworldoid] (kubernetes-tutorial) - 10https://gerrit.wikimedia.org/r/612410 (owner: 10Jeena Huneidi) [20:34:51] 10Phabricator, 10Quality-and-Test-Engineering-Team (QTE), 10User-Vidhi-Mody: CI not running for phabricator/deployment - https://phabricator.wikimedia.org/T258531 (10Vidhi-Mody) p:05Triage→03Medium [20:35:24] 10Phabricator, 10Quality-and-Test-Engineering-Team (QTE), 10User-Vidhi-Mody: CI not running for phabricator/deployment - https://phabricator.wikimedia.org/T258531 (10Vidhi-Mody) [20:35:26] 10Phabricator, 10Quality-and-Test-Engineering-Team (QTE), 10Patch-For-Review, 10User-Vidhi-Mody, 10User-zeljkofilipin: Upgrade WebdriverIO in the phab-deployment repository - https://phabricator.wikimedia.org/T255471 (10Vidhi-Mody) [20:36:11] (03PS8) 10Jeena Huneidi: [DNM] This is a test [blubber-doc/example/helloworldoid] (kubernetes-tutorial) - 10https://gerrit.wikimedia.org/r/612410 [20:37:35] 10Continuous-Integration-Config, 10Phabricator, 10Quality-and-Test-Engineering-Team (QTE), 10User-Vidhi-Mody: CI not running for phabricator/deployment - https://phabricator.wikimedia.org/T258531 (10Reedy) [20:40:27] PROBLEM - Free space - all mounts on integration-agent-docker-1011 is CRITICAL: CRITICAL: integration.integration-agent-docker-1011.diskspace._srv.byte_percentfree (<22.22%) [20:50:24] RECOVERY - Free space - all mounts on integration-agent-docker-1011 is OK: OK: All targets OK [20:55:24] 10MediaWiki-Releasing, 10MW-1.35-notes, 10MW-1.35-release, 10Platform Team Workboards (Initiatives): Release MW 1.35 - https://phabricator.wikimedia.org/T256375 (10eprodromou) [21:04:11] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10Release Pipeline, 10Patch-For-Review, 10User-dancy: Release pipeline is creating/not cleaning intermediate dangling images - https://phabricator.wikimedia.org/T235680 (10dancy) [21:05:16] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10Release Pipeline, 10Patch-For-Review, 10User-dancy: Failed docker build leaves dangling container - https://phabricator.wikimedia.org/T235680 (10dancy) [21:07:43] 10Release-Engineering-Team (Onboarding), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release Pipeline (Blubber), 10User-dancy: Blubber: Allow more than one layer of copying - https://phabricator.wikimedia.org/T254629 (10dancy) [21:16:40] !log Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/615269 in beta puppet; should only affect deployment-webperf11. [21:16:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:26:27] PROBLEM - Free space - all mounts on integration-agent-docker-1011 is CRITICAL: CRITICAL: integration.integration-agent-docker-1011.diskspace._srv.byte_percentfree (<11.11%) [21:30:30] maintenance-disconnect-full-disks build 196217 integration-agent-docker-1011 (/: 18%, /srv: 99%, /var/lib/docker: 51%): OFFLINE due to disk space [21:35:42] maintenance-disconnect-full-disks build 196218 integration-agent-docker-1011 (/: 18%, /srv: 42%, /var/lib/docker: 49%): RECOVERY disk space OK [21:40:42] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [software/envoyproxy/ratelimiter] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/615306 [21:40:45] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [software/envoyproxy/ratelimiter] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/615306 (owner: 10QChris) [21:41:27] RECOVERY - Free space - all mounts on integration-agent-docker-1011 is OK: OK: All targets OK [21:41:38] (03PS1) 10QChris: Import done. Revoke import grants [software/envoyproxy/ratelimiter] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/615307 [21:41:41] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [software/envoyproxy/ratelimiter] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/615307 (owner: 10QChris) [22:23:12] 10Phabricator: Phab doesn't respect "Request Desktop Website" browser setting - https://phabricator.wikimedia.org/T258529 (10Aklapper) [22:32:07] 10Phabricator, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Developer Productivity: Enhance phabricator project metrics and reporting - https://phabricator.wikimedia.org/T255703 (10mmodell) [22:42:43] 10Phabricator, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Developer Productivity: Enhance phabricator project metrics and reporting - https://phabricator.wikimedia.org/T255703 (10mmodell) [22:43:22] 10Phabricator, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Developer Productivity: Enhance phabricator project metrics and reporting - https://phabricator.wikimedia.org/T255703 (10mmodell) Here is a v0.1 UI mock: {F31943608} [22:43:36] (03CR) 10Thcipriani: [C: 03+2] Prevent container leak if docker build fails [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/614851 (https://phabricator.wikimedia.org/T235680) (owner: 10Ahmon Dancy) [22:44:09] (03Merged) 10jenkins-bot: Prevent container leak if docker build fails [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/614851 (https://phabricator.wikimedia.org/T235680) (owner: 10Ahmon Dancy) [22:47:44] 10Phabricator, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Developer Productivity: Enhance phabricator project metrics and reporting - https://phabricator.wikimedia.org/T255703 (10mmodell) [22:48:16] (03PS1) 10Thcipriani: DNM: Jenkins job fiddling [blubber] - 10https://gerrit.wikimedia.org/r/615309 [22:51:01] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review: Pipelinelib: support passing environment variables to docker images - https://phabricator.wikimedia.org/T257526 (10jeena) 05Open→03Resolved Verified... [22:51:56] (03Abandoned) 10Thcipriani: DNM: Jenkins job fiddling [blubber] - 10https://gerrit.wikimedia.org/r/615309 (owner: 10Thcipriani) [23:20:00] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Striker, 10Tools: Add Gerrit support to Striker (toolsadmin) - https://phabricator.wikimedia.org/T224676 (10bd808) 05Open→03Stalled Marking as stalled and moving back to needs discussion because of the pot... [23:20:04] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO: Stop using Differential for code review - https://phabricator.wikimedia.org/T191182 (10bd808) [23:36:01] PROBLEM - Free space - all mounts on deployment-snapshot01 is CRITICAL: CRITICAL: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)deployment-prep.deployment-snapshot01.diskspace.root.byte_percentfree (<10.00%) [23:46:02] RECOVERY - Free space - all mounts on deployment-snapshot01 is OK: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)