[08:00:06] Project mwcore-phpunit-coverage-master build #517: 04FAILURE in 5 hr 0 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/517/ [08:14:57] Well https://en.wikipedia.beta.wmflabs.org/wiki/Earth doesn't look good :S [08:35:05] huh, weird. beta has less logging enabled than prod. I would have assumed it would have everything in prod plus more [08:35:50] Maybe accidental regression from 07b99aea4a, unclear [09:58:02] 10Release-Engineering-Team (CI & Testing services), 10MediaWiki-extensions-Lingo, 10ci-test-error: extension/Lingo pipeline fails with unexpected UNSTABLE status - https://phabricator.wikimedia.org/T246625 (10Peter.ovchyn) [10:14:57] 10Beta-Cluster-Infrastructure, 10Chinese-Sites: Request for sysop right on zh Beta Cluster for Sunny00217 - https://phabricator.wikimedia.org/T246201 (10Sunny00217) >>! In T246201#5920584, @zhuyifei1999 wrote: > I don't see [[https://zh.wikipedia.beta.wmflabs.org/wiki/Special:%E7%94%A8%E6%88%B7%E5%88%97%E8%A1%... [10:16:23] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10MediaWiki-Docker, 10Developer Productivity, and 3 others: Command-line wrapper for interacting with core's docker-compose stack - https://phabricator.wikimedia.org/T246111 (10kostajh) > My only concern would be scope creep as far as making what... [10:18:28] 10MediaWiki-Codesniffer: Auto-fix misses to add a required newline to single-line if() - https://phabricator.wikimedia.org/T246630 (10thiemowmde) [10:26:24] 10Project-Admins, 10Community-Tech: Create a Phabricator project tag for "Commons deletion notification bot" - https://phabricator.wikimedia.org/T229759 (10Aklapper) 05Open→03Declined Thanks for the docs update! As I cannot find out which tasks are about this project, I'm going to decline this... [10:31:48] 10MediaWiki-Codesniffer: Bogus MediaWiki.WhiteSpace.SpaceyParenthesis auto-fix when there is whitespace at the end of the line - https://phabricator.wikimedia.org/T246636 (10thiemowmde) [10:39:33] 10Release-Engineering-Team, 10Operations, 10serviceops: mcrouter proxies and scap proxies - https://phabricator.wikimedia.org/T245841 (10Joe) >>! In T245841#5922561, @jijiki wrote: >>>! In T245841#5919699, @Joe wrote: >> >> What would having all scap proxies also be mcrouter proxies change in terms of the s... [10:51:50] 10Scap, 10Patch-For-Review: Send scap log directly to logstash via syslog input - https://phabricator.wikimedia.org/T86969 (10fgiunchedi) Patch has been deployed, however I overlooked adding `@cee` and will followup with a fix [11:40:18] 10Phabricator, 10Fundraising-Backlog: Please adapt Phabricator ACLs for "acl*WMF-FR" - https://phabricator.wikimedia.org/T246648 (10Aklapper) Thanks for filing this, Moritz. The problem is that currently the only people with "Can Edit" capability are @DStrine and `@atgo`. @DStrine: On an related note, could y... [12:12:35] 10Beta-Cluster-Infrastructure: Commons Beta-Cluster pages frequently don't save - https://phabricator.wikimedia.org/T246650 (10AlexisJazz) [12:24:20] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Release Pipeline: Review pipelinelib reference documentation - https://phabricator.wikimedia.org/T246253 (10LarsWirzenius) 05Open→03Resolved a:03LarsWirzenius I've read through the current version of the reference sub-page. It looks ready now.... [12:24:22] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Release Pipeline: Document pipelinelib - https://phabricator.wikimedia.org/T239423 (10LarsWirzenius) [13:47:22] 10Beta-Cluster-Infrastructure, 10Chinese-Sites: Request for sysop right on zh Beta Cluster for Sunny00217 - https://phabricator.wikimedia.org/T246201 (10Sunny00217) >>! In T246201#5925821, @Sunny00217 wrote: >>>! In T246201#5919535, @Aklapper wrote: >> @Sunny00217: Hi, which specific right are you requesting?... [13:54:32] 10MediaWiki-Codesniffer: Add potentially problematic assertEquals( 1 ) to the PHPUnitAssertEquals sniff - https://phabricator.wikimedia.org/T246662 (10thiemowmde) [14:12:58] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:15:05] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:15:14] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:20:09] (03PS1) 10Thiemo Kreuz (WMDE): Add problematic values 1 and 1.0 to PHPUnitAssertEqualsSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/576045 (https://phabricator.wikimedia.org/T246662) [14:24:14] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Draft proposal for re-implementing pipelinelib for non-jenkins - https://phabricator.wikimedia.org/T246665 (10LarsWirzenius) [14:24:41] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Draft proposal for re-implementing pipelinelib for non-jenkins - https://phabricator.wikimedia.org/T246665 (10LarsWirzenius) [14:24:43] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Release Pipeline: Document pipelinelib - https://phabricator.wikimedia.org/T239423 (10LarsWirzenius) [14:24:58] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Draft proposal for re-implementing pipelinelib for non-jenkins - https://phabricator.wikimedia.org/T246665 (10LarsWirzenius) p:05Triage→03High [14:26:56] 10Beta-Cluster-Infrastructure: deployment-mediawiki-07 socket timeout 2020-03-01 15:14-16:09 UTC - https://phabricator.wikimedia.org/T246577 (10AlexisJazz) 05Resolved→03Open Just getting timeouts now. [15:05:31] hello! i may have asked this before but don't remember [15:05:41] is there a way to scap sync-file to specific wikis or groups? [15:07:47] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 92770 bytes in 1.119 second response time [15:09:43] 10Diffusion, 10Phabricator, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Toolforge, and 2 others: Mirroring Diffusion repositories to GitHub seems to be broken - https://phabricator.wikimedia.org/T242898 (10mmodell) Ok thanks to @thcipriani's script, I got all of the `git@github.com/toolfor... [15:09:56] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 93235 bytes in 0.875 second response time [15:10:04] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 51716 bytes in 1.005 second response time [15:13:15] 10Diffusion, 10Phabricator, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Toolforge, and 2 others: Create a tool (conduit client) for managing diffusion repository URLs - https://phabricator.wikimedia.org/T246249 (10mmodell) 05Open→03Resolved [15:13:20] 10Diffusion, 10Phabricator, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Toolforge, and 2 others: Mirroring Diffusion repositories to GitHub seems to be broken - https://phabricator.wikimedia.org/T242898 (10mmodell) [15:21:40] 10Phabricator, 10Fundraising-Backlog: Please adapt Phabricator ACLs for "acl*WMF-FR" - https://phabricator.wikimedia.org/T246648 (10DStrine) I have removed the user and added the link to the form in the description. I've had a few conversations with Phabricator admins in the past about this setup. As far as I... [15:26:45] 10MediaWiki-Codesniffer: Treat assertEmpty() as problematic in PHPUnitAssertEquals - https://phabricator.wikimedia.org/T246674 (10Krinkle) [15:32:58] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Release, 10Train Deployments: 1.35.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T233870 (10Lucas_Werkmeister_WMDE) [15:43:27] 10Continuous-Integration-Config, 10Code-Health-Metrics, 10Patch-For-Review: Report results from SonarCloud to Gerrit - https://phabricator.wikimedia.org/T217008 (10kostajh) > Manually adding would be a decent option if we can generate HTML (which I don't think is the case), otherwise it would probably look p... [15:47:35] 10Beta-Cluster-Infrastructure: deployment-mediawiki-07 socket timeout 2020-03-01 15:14-16:09 UTC - https://phabricator.wikimedia.org/T246577 (10Zoranzoki21) I can't reproduce this. [15:52:13] 10Beta-Cluster-Infrastructure: deployment-mediawiki-07 socket timeout 2020-03-01 15:14-16:09 UTC - https://phabricator.wikimedia.org/T246577 (10AlexisJazz) >>! In T246577#5933010, @Zoranzoki21 wrote: > I can't reproduce this. Now it works again. Maybe leave this open for a day or so, see if remains up this time. [15:56:38] 10MediaWiki-Codesniffer: Treat assertEmpty() as problematic in PHPUnitAssertEquals - https://phabricator.wikimedia.org/T246674 (10thiemowmde) In the past, I liked PHP's `empty()` exactly because it is so relaxed. Whenever I saw something like `if ( !empty( $var ) )`, I was sure no unwanted empty or empty-ish val... [16:02:18] 10Beta-Cluster-Infrastructure: deployment-mediawiki-07 socket timeout 2020-03-01/02 - https://phabricator.wikimedia.org/T246577 (10RhinosF1) [16:07:13] Anyone from Releng have thoughts on https://phabricator.wikimedia.org/T246577 [16:25:04] James_F: if you have a sec, just want to make sure i'm not doing config wrong here [16:25:07] https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/576066 [16:45:24] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Security-Team, 10Patch-For-Review, and 2 others: Wikimedia deployers audit - https://phabricator.wikimedia.org/T237696 (10chasemp) a:05sbassett→03chasemp I'll review the current and merge if t... [16:45:35] 10Phabricator, 10Fundraising-Backlog: Please adapt Phabricator ACLs for "acl*WMF-FR" - https://phabricator.wikimedia.org/T246648 (10Aklapper) Thanks! @Dstrine: https://phabricator.wikimedia.org/project/edit/1070/ still says that only one single active individual (you) can change anything, and I consider that a... [16:48:29] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Zuul, 10User-DannyS712: Remote call on integration-agent-docker-1001 failed - https://phabricator.wikimedia.org/T245054 (10thcipriani) Documented at https://www.mediawiki.org/wiki/Continuous_integration/Je... [16:50:37] 10Project-Admins: Clarify scope of #Office-IT project tag in Phabricator (Not actively used by team, hence archive it) - https://phabricator.wikimedia.org/T242292 (10Aklapper) 05Open→03Resolved `T218491` has been resolved, which means there are no issues left which have #Office-IT as their only associated ac... [16:52:11] 10Beta-Cluster-Infrastructure: deployment-mediawiki-07 socket timeout 2020-03-01/02 - https://phabricator.wikimedia.org/T246577 (10AlexisJazz) aaaaand it broke again. [16:55:05] 10Beta-Cluster-Infrastructure: deployment-mediawiki-07 socket timeout 2020-03-01/02 - https://phabricator.wikimedia.org/T246577 (10RhinosF1) If you're in europe, it's a wikimedia-wide outage. [16:56:39] 10Beta-Cluster-Infrastructure: deployment-mediawiki-07 socket timeout 2020-03-01/02 - https://phabricator.wikimedia.org/T246577 (10RhinosF1) Services should be back now :) [17:05:10] 10Beta-Cluster-Infrastructure: deployment-mediawiki-07 socket timeout 2020-03-01/02 - https://phabricator.wikimedia.org/T246577 (10AlexisJazz) >>! In T246577#5933401, @RhinosF1 wrote: > If you're in europe, it's a wikimedia-wide outage. There are many people in Europe you know! ;) Including me. Alright, let's h... [17:08:38] 10Release-Engineering-Team-TODO, 10Wikimedia-General-or-Unknown, 10Wikimedia-Site-requests, 10Patch-For-Review: Investigate whether wmf-config's configuration cache is still needed - https://phabricator.wikimedia.org/T169821 (10Krinkle) It looks like about 1/3rd of the time to generate the config-cache (ge... [17:24:58] 10Beta-Cluster-Infrastructure: deployment-mediawiki-07 socket timeout 2020-03-01/02 - https://phabricator.wikimedia.org/T246577 (10thcipriani) p:05Triage→03Medium >>! In T246577#5933058, @RhinosF1 wrote: >>>! In T246577#5933032, @AlexisJazz wrote: >>>>! In T246577#5933010, @Zoranzoki21 wrote: >>> I can't rep... [17:29:03] (03CR) 10Thcipriani: [C: 03+2] Add: an acceptance (or integration) test suite for scap [tools/scap] - 10https://gerrit.wikimedia.org/r/573298 (https://phabricator.wikimedia.org/T245614) (owner: 10Lars Wirzenius) [17:32:58] (03Merged) 10jenkins-bot: Add: an acceptance (or integration) test suite for scap [tools/scap] - 10https://gerrit.wikimedia.org/r/573298 (https://phabricator.wikimedia.org/T245614) (owner: 10Lars Wirzenius) [17:34:49] 10Beta-Cluster-Infrastructure: deployment-mediawiki-07 socket timeout 2020-03-01/02 - https://phabricator.wikimedia.org/T246577 (10RhinosF1) >>! In T246577#5933479, @AlexisJazz wrote: >>>! In T246577#5933401, @RhinosF1 wrote: >> If you're in europe, it's a wikimedia-wide outage. > > There are many people in Eur... [17:37:53] Thanks for looking at the beta cluster issues thcipriani [17:38:44] I remember alerts going off last week for a few days but not all week [17:38:48] * thcipriani doffs cap [17:39:19] So it chaging with James_F mid week could be possible [17:39:43] yeah, that's not all I saw in logs, but that was pretty prevalent in the logs :\ [17:46:50] 10Beta-Cluster-Infrastructure: deployment-mediawiki-07 socket timeout 2020-03-01/02 - https://phabricator.wikimedia.org/T246577 (10AlexisJazz) Here's another odd thing that may or may not be related: https://commons.wikimedia.beta.wmflabs.org/wiki/File:6f2dbfewe.jpg Sure, I'm a butcher as I'm trying to create a... [17:47:52] PROBLEM - Parsoid on deployment-parsoid09 is CRITICAL: connect to address 172.16.5.63 and port 8000: Connection refused [17:47:52] PROBLEM - Parsoid on deployment-mediawiki-parsoid10 is CRITICAL: connect to address 172.16.0.141 and port 8000: Connection refused [18:04:12] thcipriani: Fix is to change (master of) scap to stop generating the CBD file generation step? [18:04:34] But that's non-trivial. I looked at ripping it out quickly last week and decided I didn't know scap well enough. [18:11:39] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Toolforge, and 3 others: Add CI checks for golang admission controllers - https://phabricator.wikimedia.org/T236203 (10Jdforrester-WMF) Sorry, this ran i... [18:12:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Toolforge, and 3 others: Add CI checks for golang admission controllers - https://phabricator.wikimedia.org/T236203 (10Jdforrester-WMF) a:05Jdforrester-... [18:12:42] (03Abandoned) 10Jforrester: [WIP] jjb: Add a golang jjb defintion for `golint` and `go test` [integration/config] - 10https://gerrit.wikimedia.org/r/552865 (https://phabricator.wikimedia.org/T236203) (owner: 10Jforrester) [18:13:18] James_F: oh, is this because it's loading both? [18:13:50] thcipriani: Yeah, and the Beta boxes have more limited disk than prod. [18:14:02] (And are building 1000 extensions' worth, not 200.) [18:14:12] oh the disk space thing you mean? [18:14:43] Oh, this is a memory space issue. Ignore me. [18:14:47] I was just poking about some OOM errors for php-fpm on deployment-mediawiki-07, yep [18:15:01] Beta config should mean it never even sees the CDB files, let alone reads them now. [18:15:25] OOMing on the flat files is surprising; that'd be something for Timo/Aaron as the experts though. [18:15:51] James_F: what’s going on with files as well on commons beta? https://phabricator.wikimedia.org/T246577#5933733 [18:17:27] No idea. People writing bad scripts can make MW break, news at 11. [18:18:13] James_F: that should be another task then [18:18:15] ?? [18:18:24] Or just Invalid'ed, yes. [18:19:58] 10Beta-Cluster-Infrastructure: deployment-mediawiki-07 socket timeout 2020-03-01/02 - https://phabricator.wikimedia.org/T246577 (10RhinosF1) >>! In T246577#5933733, @AlexisJazz wrote: > Here's another odd thing that may or may not be related: https://commons.wikimedia.beta.wmflabs.org/wiki/File:6f2dbfewe.jpg Pro... [18:20:32] James_F: Can I put 18:17:27 No idea. People writing bad scripts can make MW break, news at 11. On Bash.wmflabs.org [18:20:37] (03CR) 10Brennen Bearnes: [V: 03+2 C: 03+2] "Verified build locally." [releng/dev-images] - 10https://gerrit.wikimedia.org/r/574983 (https://phabricator.wikimedia.org/T246202) (owner: 10Kosta Harlan) [18:22:20] RhinosF1: Go for it. [18:22:43] 10Beta-Cluster-Infrastructure, 10Commons, 10MediaWiki-File-management: I can duplicate my files (identitcal revisions) on betacommons - https://phabricator.wikimedia.org/T246695 (10AlexisJazz) [18:22:55] !log Updating dev-images docker-pkg files on contint1001 for T246202 [18:22:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:22:57] T246202: Add tideways-xhprof to PHP image - https://phabricator.wikimedia.org/T246202 [18:23:20] 10Beta-Cluster-Infrastructure, 10MediaWiki-File-management: I can duplicate my files (identitcal revisions) on betacommons - https://phabricator.wikimedia.org/T246695 (10AlexisJazz) [18:23:41] James_F: https://bash.toolforge.org/quip/AXCcfRAt0fjmsHBaoqIy [18:23:56] And looks like they filed a seperate bug [18:24:41] Good. [18:25:20] 10Beta-Cluster-Infrastructure, 10MediaWiki-File-management: I can duplicate my files (identical revisions) on betacommons by (un)deleting them - https://phabricator.wikimedia.org/T246695 (10AlexisJazz) [18:25:29] * RhinosF1 likes bash. That must be one of my faves. [18:27:58] You have some good ones [18:28:15] The price of being around for too long, clearly. :-) [18:28:50] Yippee, build fixed! [18:28:51] Project mwcore-phpunit-coverage-master build #518: 09FIXED in 3 hr 28 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/518/ [18:32:02] James_F: you and Reedy I think must be in my top 3 [18:32:17] Reedy’s “wonderfuk” quote is up there [18:32:30] * James_F will endeavour to say fewer bitter sarcastic comments. [18:32:45] James_F: We shall revoke your passport then [18:33:23] Reedy: "Fewer" will still leave me in my rightful place the upper echelons of the British establishment by count of devastating comments. [18:34:34] Developers are fun on IRC, probably a lot better in real life but I can’t really call anyone I know who is technically a “developer” because i end up doing 90% of their work or teaching them. The fun of being best in a Computer Science Class. [18:35:13] Or fixing their computer. That's always fun. [18:35:28] That’s my family [18:36:22] God would probably struggle to find out what they manage to do [18:36:34] And they never try the basics first [18:37:37] Being a computer user is as different from being a developer/sre as being a car driver is from being an automotive engineer/master mechanic. [18:38:01] Yup. I try to explain that I design Ferraris and am no good at fixing your Prius, but it doesn't seem to convince. [18:38:02] Expecting most folks to think about computers the way "we" do is a bad way to think about things [18:39:05] True [18:39:17] +100 to saying no to family/friends tech support. But also +1 to helping them find a good service & support company to work with [18:39:42] My grandad can break a computer pressing the power button. I think twice I refused to support him [18:39:55] Once for a virus after he fell for a scam [18:40:01] I forced my whole family over to Macs in the early noughties and it's been a huge blessing. [18:40:14] lol [18:40:22] James_F how did you manage that? :P [18:40:32] And another when he broke it after ignlring me trying to patch his hard drive to stop it failing [18:40:42] * paladox forced his family to have iphones :) [18:41:24] Most of mine do anyway [18:42:05] My mum has a chromebook which is hard for her to break. Although she just can’t use it and doesn’t understand its use case. She chose that though. [18:45:38] I have an "expired" chrombook that Google doesn't let attach to their services anymore. I won't be buying another now that I understand that forced obsolescence [18:47:15] It’s like 6/7 years after first release at least I think [18:48:49] https://support.google.com/chrome/a/answer/6220366?hl=en -- "Auto Update Expiration" seems to be in part based on the contract negotiated by the hardware vendor with G [18:49:14] "Google will publish a model’s AUE date on this page after its release, giving buyers time to make purchase decisions. Please check the AUE date when making a purchasing decision." -- so yeah [18:51:38] Yeah [18:52:06] Hmm [18:52:14] My chromebook must be scrap now too [18:55:51] Reedy: find out [18:56:09] I'll have to go dig it out [18:56:20] Ha [18:56:32] Probably if it’s been hidden away for years [18:59:08] More moving house [18:59:17] Ah [18:59:18] It's in one of three places. It won't be hard to find ;) [18:59:28] I bet i have all sorts of nonsense [18:59:43] I have loads of broken lightning cables [19:02:21] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10BlueSpice: BlueSpice related tests fails on gate-and-submit-1.31 in quibble-composer tests - https://phabricator.wikimedia.org/T235807 (10thcipriani) [19:03:17] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10BlueSpice: BlueSpice related tests fails on gate-and-submit-1.31 in quibble-composer tests - https://phabricator.wikimedia.org/T235807 (10Jdforrester-WMF) >>! In T235807#5677386, @Osnard wrote: > How about changing... [19:05:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Toolforge, and 3 others: Add CI checks for golang admission controllers - https://phabricator.wikimedia.org/T236203 (10dduvall) @Bstorm, we've been toili... [19:07:45] (03CR) 10Jforrester: "My main concern here is that we lose fallback-to-master behaviour for novel branches (e.g. if someone created a REL_9_99 branch right now)" [integration/quibble] - 10https://gerrit.wikimedia.org/r/550087 (owner: 10Robert Vogel) [19:08:46] (03CR) 10Jforrester: Add pattern based fallback for branches (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/550087 (owner: 10Robert Vogel) [19:09:08] (03PS3) 10Jforrester: Add pattern based fallback for branches [integration/quibble] - 10https://gerrit.wikimedia.org/r/550087 (owner: 10Robert Vogel) [19:12:14] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Security-Team, 10Patch-For-Review, and 2 others: Wikimedia deployers audit - https://phabricator.wikimedia.org/T237696 (10chasemp) a:05chasemp→03sbassett {P10586, lines=20} back to scott for... [19:12:27] 10Beta-Cluster-Infrastructure, 10JavaScript: Sync pages (gadgets, in CSS and/or JS) from production wikis - https://phabricator.wikimedia.org/T51779 (10cscott) This has always been a desire of the Parsoid team: Beta doesn't have enough content to make it a worthwhile testing target for us. Ideally we'd sync "... [19:13:01] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Security-Team, 10Security, 10user-sbassett: Wikimedia deployers audit - https://phabricator.wikimedia.org/T237696 (10chasemp) [19:16:07] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Release Pipeline (Blubber): Implement golang directives for Blubber - https://phabricator.wikimedia.org/T246700 (10dduvall) [19:19:42] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Security-Team, 10Security, 10user-sbassett: Wikimedia deployers audit - https://phabricator.wikimedia.org/T237696 (10sbassett) 05Open→03Resolved @chasemp - LGTM, thanks for the deploy! I'll... [19:39:20] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10ChangeProp, 10Operations, and 5 others: Migrate cpjobqueue to kubernetes - https://phabricator.wikimedia.org/T220399 (10Pchelolo) [19:53:57] RhinosF1: bd808 Seems I might be ok for like 18 months... It's a chromebook 2! [19:54:17] Pretty cool [20:14:10] 10Continuous-Integration-Infrastructure: beta-mediawiki-config-update-eqiad was stuck on 2020-02-19 - https://phabricator.wikimedia.org/T245770 (10Zoranzoki21) @Jdforrester-WMF Ok. But this constantly happens (each 1-2 days). [20:43:17] (03PS1) 10Brennen Bearnes: add delete-wmf-branches [tools/release] - 10https://gerrit.wikimedia.org/r/576136 (https://phabricator.wikimedia.org/T244368) [20:45:03] (03PS2) 10Brennen Bearnes: add delete-wmf-branches [tools/release] - 10https://gerrit.wikimedia.org/r/576136 (https://phabricator.wikimedia.org/T244368) [20:46:21] 10Release-Engineering-Team-TODO, 10Scap, 10MediaWiki-Internationalization, 10Performance-Team, 10Patch-For-Review: Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740 (10Krinkle) > @thcipriani mentioned this in {T246577} I'm blocking prod roll out... [20:52:49] (03CR) 10Krinkle: add delete-wmf-branches (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/576136 (https://phabricator.wikimedia.org/T244368) (owner: 10Brennen Bearnes) [21:03:57] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:05:36] James_F: ^ same as the issues recently? [21:06:05] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:06:14] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:08:50] hm [21:09:44] (03PS3) 10Brennen Bearnes: add delete-wmf-branches [tools/release] - 10https://gerrit.wikimedia.org/r/576136 (https://phabricator.wikimedia.org/T244368) [21:11:41] 10Diffusion, 10Phabricator, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Toolforge, and 2 others: Mirroring Diffusion repositories to GitHub seems to be broken - https://phabricator.wikimedia.org/T242898 (10mmodell) 05Open→03Resolved [21:12:25] Krenair: see scrollback and https://phabricator.wikimedia.org/T246577 [21:13:03] I know [21:14:53] !log restarted php7.2-fpm on deployment-mediawiki-07 [21:14:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:15:57] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 93237 bytes in 1.050 second response time [21:16:03] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 51716 bytes in 1.005 second response time [21:16:57] Krenair: that worked [21:17:15] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10MediaWiki-Docker, 10Developer Productivity, and 3 others: Command-line wrapper for interacting with core's docker-compose stack - https://phabricator.wikimedia.org/T246111 (10mmodell) @kostajh: I'd like to collaborate on this as I already made... [21:18:47] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 92770 bytes in 1.090 second response time [21:21:26] 10Beta-Cluster-Infrastructure: deployment-mediawiki-07 socket timeout 2020-03-01/02 - https://phabricator.wikimedia.org/T246577 (10Krenair) FYI beta broke at 21:03:57ish today and I restored it around 21:14:53ish. ` Mar 2 20:38:20 deployment-mediawiki-07 php7.2-fpm: PHP Fatal error: Allowed memory size of 6920... [21:21:36] James_F: i'm getting a lot of HTTP errors in Jenkins while trying to merge https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/575229/ - is that a known problem? [21:29:56] 10Phabricator, 10Fundraising-Backlog: Please adapt Phabricator ACLs for "acl*WMF-FR" - https://phabricator.wikimedia.org/T246648 (10DStrine) 05Open→03Resolved [21:43:03] Jdlrobson: Not sure what was causing those issues; think Packagist may be having troubles again. [22:06:23] James_F: also did you make any changes to logos recently that may have broken the logo on the beta cluster? Am investigating this now [22:14:25] James_F: answered my own question - https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/570379/ [22:46:14] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Other / Uncategorized), 10Release-Engineering-Team-TODO, 10Desktop Improvements, and 2 others: Automatically display production content on Beta Cluster wikis on desktop (via an API request) - https://phabricator.wikimedia.org/T207508 (10Jdlrobson)... [22:46:22] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Other / Uncategorized), 10Release-Engineering-Team-TODO, 10Desktop Improvements, and 3 others: Automatically display production content on Beta Cluster wikis on desktop (via an API request) - https://phabricator.wikimedia.org/T207508 (10Jdlrobson) [22:55:39] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Docker, 10Developer Productivity, and 3 others: Command-line wrapper for interacting with core's docker-compose stack - https://phabricator.wikimedia.org/T246111 (10thcipriani) [23:05:58] 10Release-Engineering-Team-TODO, 10Scap, 10Wikimedia-Incident: Add a quick rollback command to scap that skips all unnecessary steps - https://phabricator.wikimedia.org/T244544 (10thcipriani) [23:33:21] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO: Please install 'npx' node package runner on CI nodes - https://phabricator.wikimedia.org/T246400 (10thcipriani) [23:54:38] 10Deployments, 10Release-Engineering-Team: Make "scap lock" easy to use and adopt as standard practice - https://phabricator.wikimedia.org/T234407 (10Krinkle) 05Open→03Resolved a:03Krinkle The reason I couldn't find the parameters is because I was using the help command incorrectly. ` $ scap help lock u...