[00:25:13] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Wikimedia-General-or-Unknown, 10PHP 7.4 support: Make Wikimedia Production MediaWiki compatible with PHP 7.4 - https://phabricator.wikimedia.org/T247658 (10Reedy) [00:52:40] (03PS1) 10Krinkle: doc: Add more luasandbox links [integration/docroot] - 10https://gerrit.wikimedia.org/r/592796 (https://phabricator.wikimedia.org/T186302) [01:04:17] 10MediaWiki-Codesniffer, 10MediaWiki-General, 10MW-1.35-notes (1.35.0-wmf.26; 2020-03-31), 10MW-1.35-release, and 2 others: Fix all functions that have required parameters following optional ones - https://phabricator.wikimedia.org/T248078 (10Reedy) >>! In T248078#6080103, @DannyS712 wrote: >>>! In T248078... [01:05:29] (03PS1) 10Krinkle: doc: Add entry for Fresnel [integration/docroot] - 10https://gerrit.wikimedia.org/r/592798 (https://phabricator.wikimedia.org/T251213) [01:09:41] (03PS8) 10DannyS712: Move `SuperfluousVariadicArgComment` out of the FunctionCommentSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/592286 (https://phabricator.wikimedia.org/T250958) [02:38:53] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:38:55] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:43:43] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 92795 bytes in 1.198 second response time [02:43:48] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 92491 bytes in 1.054 second response time [07:01:20] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] "You just wrote another comment that doesn't fit on my laptop screen any more. I'm sorry if I made you invest so much time here." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/591600 (https://phabricator.wikimedia.org/T250765) (owner: 10DannyS712) [07:30:38] 10Release-Engineering-Team-TODO, 10Scap, 10MediaWiki-Internationalization, 10Performance-Team, 10Patch-For-Review: Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740 (10Joe) >>! In T99740#6085845, @Krinkle wrote: > I think 5% is huge! As part of T... [07:56:10] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10serviceops, and 3 others: Cache of wmf-config/InitialiseSettings often 1 step behind - https://phabricator.wikimedia.org/T236104 (10KartikMistry) @Jdforrester-WMF https://wikitech.w... [08:13:12] Were there any recent upgrades that might have affected browser tests, or maybe trouble in the cluster? We're suddenly seeing 100% failure for any patch in mw-ext-TwoColConflict. Looks like the error is "[SEVERE]: Timed out receiving message from renderer" [08:43:36] awight: that sounds strange, did you create a phab task? [08:44:31] 10Release-Engineering-Team-TODO, 10Scap, 10MediaWiki-Internationalization, 10Performance-Team, 10Patch-For-Review: Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740 (10Joe) Assuming we'll be ok with restarting php-fpm at every release, I reduced... [08:47:42] zeljkof: Not yet, but I'll do that shortly. I was exhausting some of the easy things to check, first... Entasking now! [08:48:05] awight: please do, so it doesn't get lost :) [08:48:24] it's unlikely only one repository is affected, so it's probably failing everywhere [08:48:48] I mean, if it's a CI problem, not repository problem :) [09:18:31] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10serviceops, and 3 others: Cache of wmf-config/InitialiseSettings often 1 step behind - https://phabricator.wikimedia.org/T236104 (10Lucas_Werkmeister_WMDE) I’ve removed it again. [10:00:41] !log canceled one beta-scap-eqiad job per https://w.wiki/J5$ [10:00:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:00:48] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T249962 (10LarsWirzenius) Staring the branch cut. [10:07:39] 10Release-Engineering-Team-TODO, 10Scap, 10MediaWiki-Internationalization, 10Performance-Team, 10Patch-For-Review: Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740 (10Joe) First, the results of the real traffic test. These are averages over 10 m... [10:32:24] 10Phabricator: Make sure anti-vandalism features are up to snuff - https://phabricator.wikimedia.org/T84 (10Aklapper) [11:17:10] 10MediaWiki-Codesniffer: MediaWiki.Commenting.DocComment.SyntaxAlignedDocStar issue - https://phabricator.wikimedia.org/T251248 (10Reedy) [11:54:14] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:57] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:55:45] 10Phabricator, 10TCB-Team, 10WMDE-QWERTY-Sprint-2020-04-15: Spaces request for "Technical Wishes Private" - https://phabricator.wikimedia.org/T248104 (10Tobi_WMDE_SW) Thanks a lot @Aklapper ! [11:55:53] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:44:06] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 52135 bytes in 1.282 second response time [12:44:47] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 92489 bytes in 0.915 second response time [12:45:44] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 92795 bytes in 1.062 second response time [12:52:54] 10Release-Engineering-Team, 10Wikidata, 10Documentation: Update links to doc.wikimedia.org - https://phabricator.wikimedia.org/T251259 (10Addshore) [12:53:25] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T249962 (10LarsWirzenius) Deplolyed to testwikis, and ran over the EU SWAT window. Logstash looks calms after half an hour. [12:54:42] 10Release-Engineering-Team, 10Wikidata, 10Documentation: Update links to doc.wikimedia.org - https://phabricator.wikimedia.org/T251259 (10Addshore) CCing #release-engineering-team as I expect we can create a set of redirects from the old path to avoid needing to try to fix the links everywhere. I had a littl... [13:07:12] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T249962 (10LarsWirzenius) Promoted to group0. [13:10:41] PROBLEM - Parsoid on deployment-parsoid11 is CRITICAL: connect to address 172.16.1.115 and port 8000: Connection refused [13:17:52] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Scap: Rename `scap sync` to `scap sync-world` - https://phabricator.wikimedia.org/T250302 (10LarsWirzenius) a:03LarsWirzenius [13:26:18] 10Release-Engineering-Team, 10Wikidata, 10Documentation: Update links to doc.wikimedia.org - https://phabricator.wikimedia.org/T251259 (10Samantha_Alipio_WMDE) In looking through Wikiba.se, there are currently no links to doc.wikimedia.org nor any broken links related to this ticket. [13:32:47] (03PS9) 10Thiemo Kreuz (WMDE): Full rewrite of the UnusedUseStatementSniff, 4x faster [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/592415 [13:37:17] (03PS1) 10Thiemo Kreuz (WMDE): Don't run parallel-lint on test files [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/592947 [13:38:18] (03CR) 10jerkins-bot: [V: 04-1] Full rewrite of the UnusedUseStatementSniff, 4x faster [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/592415 (owner: 10Thiemo Kreuz (WMDE)) [13:48:29] <_joe_> hashar: how do I create the github mirror of a new gerrit repo? [13:48:45] <_joe_> I don't remember if I just need to create the project as empty on github [13:48:52] _joe_: hi, create the repository on github replacing slashes with dashes [13:49:03] <_joe_> hashar: sweet, thanks [13:49:09] so that operations/software/foo becomes https://github.com/wikimedia/operations-software-foo [13:49:18] and eventually Gerrit will sync to it after some time [13:49:21] <_joe_> we need to remember to set new repos as public :P [13:49:27] or when something touches the git repo on gerrit [13:52:27] or you can force-replication [13:54:22] 10Continuous-Integration-Config, 10TCB-Team, 10Two-Column-Edit-Conflict-Merge, 10WMDE-QWERTY-Team: Add TwoColConflict to the "gated" extensions list in core - https://phabricator.wikimedia.org/T251276 (10thiemowmde) [13:54:48] i think 20.after4 is planning a script to do the magic to create the github and diffusion mirrors and to start the replication [13:58:00] 10Release-Engineering-Team-TODO, 10Scap, 10MediaWiki-Internationalization, 10Performance-Team, 10Patch-For-Review: Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740 (10Joe) a:05Joe→03None Just to be clearer: we achieved a much larger improvem... [14:01:56] _joe_: added to the repo creation doc https://www.mediawiki.org/wiki/Git/Creating_new_repositories#Step_7:_Create_a_repository_in_Gitub ;D [14:03:10] <_joe_> I'm not doing the diffusion mirror, I thought there was no reason with gitiles [14:03:18] <_joe_> also, didn't we have to turn it off? [14:03:29] <_joe_> if it's still relevant I'll do that as well [14:19:20] _joe_: Differential is the Phabricator code review system, and yeah we got rid of that one [14:19:32] Diffusion is the repo hosting / browser (same as gitiles) [14:19:34] 10Release-Engineering-Team-TODO, 10Scap, 10MediaWiki-Internationalization, 10Performance-Team, 10Patch-For-Review: Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740 (10Joe) Some more data: Rendering the enwiki Barack Obama page, with concurrency... [14:20:03] when commits are in Diffusion, that also means Phabricator knows about them and would recognize them when writing comments [14:24:47] (03CR) 10Daimona Eaytoy: "> Yes, you are right, I use "premature optimization" in a wider," [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/591600 (https://phabricator.wikimedia.org/T250765) (owner: 10DannyS712) [14:31:54] (03PS10) 10Thiemo Kreuz (WMDE): Full rewrite of the UnusedUseStatementSniff, 4x faster [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/592415 [14:32:16] <_joe_> hashar: then why have gitiles :P [14:32:42] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+2] Code cleanup: Remove unneeded parenthesis around additions [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/592748 (owner: 10Umherirrender) [14:33:02] _joe_: yeah that is a bazaar :D [14:33:31] (03Merged) 10jenkins-bot: Code cleanup: Remove unneeded parenthesis around additions [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/592748 (owner: 10Umherirrender) [14:34:02] <_joe_> hashar: I mean, I don't think it's impossible to make phabricator know about commits from gerrit, and we actually do [14:34:31] _joe_: yeah that is done by recplicating the repos from Gerrit to Diffusion [14:34:56] <_joe_> you mean the plugin that shows the gerrit changes in the tasks? [14:35:12] (03PS4) 10Thiemo Kreuz (WMDE): Set ignoreSpacingBeforeAssignments to false for OperatorSpacing [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/592751 (https://phabricator.wikimedia.org/T232265) (owner: 10Umherirrender) [14:36:34] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] "It looks like some tab-expansion is going on, therefor the different test results. I hope it's a stable change I made in patch set 4." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/592751 (https://phabricator.wikimedia.org/T232265) (owner: 10Umherirrender) [14:37:02] (03CR) 10jerkins-bot: [V: 04-1] Full rewrite of the UnusedUseStatementSniff, 4x faster [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/592415 (owner: 10Thiemo Kreuz (WMDE)) [14:37:19] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+2] Ignore anon classes when looking for return in FunctionCommentSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/592464 (owner: 10Umherirrender) [14:37:51] (03Merged) 10jenkins-bot: Ignore anon classes when looking for return in FunctionCommentSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/592464 (owner: 10Umherirrender) [14:38:09] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] "I was not aware this is a thing. I can't think of a use, but maybe we are missing something? Let's give this some time. No need to rush." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/592746 (owner: 10Umherirrender) [14:40:01] (03CR) 10jerkins-bot: [V: 04-1] Set ignoreSpacingBeforeAssignments to false for OperatorSpacing [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/592751 (https://phabricator.wikimedia.org/T232265) (owner: 10Umherirrender) [14:40:49] <_joe_> hashar: interestingly, I can create repos in gerrit but not in diffusion, so I'd need help :P [14:49:48] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace._srv.byte_percentfree (<10.00%) [14:51:45] _joe_: which name? [14:54:59] <_joe_> operations/debs/helm3 [14:55:29] 10phan-taint-check-plugin: Understand what's going on with ChangeTags::modifyDisplayQuery - https://phabricator.wikimedia.org/T250689 (10Daimona) > What's going on? Is it a false positive? Yes. > Why did it disappear for core but not extensions? Is the fix at r588096 faulty? Don't care. > If it's a false posi... [14:56:15] PROBLEM - Free space - all mounts on deployment-deploy02 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy02.diskspace._srv.byte_percentfree (<11.11%) [14:59:37] 10Continuous-Integration-Infrastructure (Slipway), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Wikidata, and 2 others: Create docker based CI job to build the wikidata-query-gui - https://phabricator.wikimedia.org/T210286 (10Addshore) [14:59:50] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [15:00:26] (03PS11) 10Thiemo Kreuz (WMDE): Full rewrite of the UnusedUseStatementSniff, 4x faster [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/592415 [15:00:56] 10Continuous-Integration-Infrastructure (Slipway), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Wikidata, and 2 others: Create docker based CI job to build the wikidata-query-gui - https://phabricator.wikimedia.org/T210286 (10Addshore) p:05Medium→03Low [15:02:51] (03CR) 10Thiemo Kreuz (WMDE): "Oh dear, that was a bit painful to figure out. But it was worth it! Turns out I was running into an edge-case that existed in the old impl" (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/592415 (owner: 10Thiemo Kreuz (WMDE)) [15:03:28] (03CR) 10Daimona Eaytoy: [C: 03+1] Don't run parallel-lint on test files [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/592947 (owner: 10Thiemo Kreuz (WMDE)) [15:04:05] (03PS3) 10Umherirrender: Code cleanup: Remove calls to File::recordMetric [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/592746 [15:04:35] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] "Oh dear. The tokenizers in different PHP versions return different whitespace tokens! How to fix this? The easiest fix is probably to just" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/592751 (https://phabricator.wikimedia.org/T232265) (owner: 10Umherirrender) [15:09:08] (03CR) 10Umherirrender: "The metric does not hurt, but it is not record in other sniffs in the repo and that makes it useless here." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/592746 (owner: 10Umherirrender) [15:15:08] what defines the scap targets in this file: [15:15:23] [deploy1001:/srv/deployment/design/style-guide/scap] $ cat targets [15:15:23] bromine.eqiad.wmnet [15:15:24] vega.codfw.wmnet [15:15:33] these need to be updated [15:16:31] do i manually edit that file, is it in the deployment repo or is it scap::target in puppet ( but i dont see it) [15:21:45] alright. https://gerrit.wikimedia.org/r/#/c/design/style-guide/+/592980 [15:22:00] but that is a special setup that syncs from github. i will mail Volker [15:27:24] <_joe_> mutante: the scap targets are in the deploy repo usually [15:27:40] <_joe_> a mistake that bites us regularly [15:40:34] (03PS3) 10Umherirrender: Move fix of typo annotations to FunctionAnnotationsSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/592277 [15:51:16] (03CR) 10Umherirrender: "Rebased" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/592277 (owner: 10Umherirrender) [16:12:57] 10Release-Engineering-Team-TODO, 10Scap, 10MediaWiki-Internationalization, 10Performance-Team, 10Patch-For-Review: Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740 (10Ladsgroup) >>! In T99740#6089198, @Joe wrote: > Some more data: > > Rendering... [16:26:51] (03CR) 10Krinkle: [C: 03+2] doc: Add more luasandbox links [integration/docroot] - 10https://gerrit.wikimedia.org/r/592796 (https://phabricator.wikimedia.org/T186302) (owner: 10Krinkle) [16:26:51] (03Merged) 10jenkins-bot: doc: Add more luasandbox links [integration/docroot] - 10https://gerrit.wikimedia.org/r/592796 (https://phabricator.wikimedia.org/T186302) (owner: 10Krinkle) [16:26:51] (03PS2) 10Krinkle: doc: Add entry for Fresnel [integration/docroot] - 10https://gerrit.wikimedia.org/r/592798 (https://phabricator.wikimedia.org/T251213) [16:26:51] (03CR) 10Krinkle: [C: 03+2] doc: Add entry for Fresnel [integration/docroot] - 10https://gerrit.wikimedia.org/r/592798 (https://phabricator.wikimedia.org/T251213) (owner: 10Krinkle) [16:26:51] (03Merged) 10jenkins-bot: doc: Add entry for Fresnel [integration/docroot] - 10https://gerrit.wikimedia.org/r/592798 (https://phabricator.wikimedia.org/T251213) (owner: 10Krinkle) [16:37:16] RECOVERY - Free space - all mounts on deployment-deploy02 is OK: OK: All targets OK [16:47:00] 10Phabricator, 10CommRel-Specialists-Support (Apr-Jun-2020): Please add Szymon to CRS' private space - https://phabricator.wikimedia.org/T251299 (10Elitre) [16:49:22] (03PS1) 10Jforrester: make-release: Make the branch commit explain why, not what [tools/release] - 10https://gerrit.wikimedia.org/r/593003 [16:56:54] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:00:12] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:00:56] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:01:27] (03CR) 10Lars Wirzenius: [C: 03+2] "Nicely justified commit message!" [tools/release] - 10https://gerrit.wikimedia.org/r/593003 (owner: 10Jforrester) [17:02:35] * James_F grins at liw. [17:02:49] I only noticed after two lines, and decided to keep with it. [17:03:55] (03Merged) 10jenkins-bot: make-release: Make the branch commit explain why, not what [tools/release] - 10https://gerrit.wikimedia.org/r/593003 (owner: 10Jforrester) [17:04:05] James_F, :) [17:04:50] oooh, nice! [17:06:18] It's a lot easier to do in English than other languages, given the ridiculous breadth of the argot and range of the verbiage, which provides for the perfect bon mot. ;-) [17:25:26] James_F: is andre__ out? if so, do you think you'd have a few minutes today for T251198 ? [17:25:27] T251198: Please create project tag 'SRE-OnFire-Incident-Docs' - https://phabricator.wikimedia.org/T251198 [17:25:57] cdanis: Don't know, but sure. [17:27:00] 10Phabricator: @Phabricator_maintenance is sending email notifications - https://phabricator.wikimedia.org/T216867 (10MBinder_WMF) I know it's been a while since this was visited, but I wonder if there are plans to address this? The vast majority of batch edits that I want silenced I can request, but the ones t... [17:27:29] cdanis: Done: https://phabricator.wikimedia.org/project/board/4754/ [17:27:49] <3 thanks! [17:27:52] 10Project-Admins: Please create project tag 'SRE-OnFire-Incident-Docs' - https://phabricator.wikimedia.org/T251198 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF Done: https://phabricator.wikimedia.org/project/board/4754/ [17:30:06] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 52104 bytes in 0.986 second response time [17:30:49] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 92506 bytes in 0.985 second response time [17:31:42] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 92811 bytes in 1.013 second response time [17:35:59] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T249962 (10Jdforrester-WMF) [18:03:23] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T249962 (10Reedy) [18:03:43] 10Phabricator: @Phabricator_maintenance is sending email notifications - https://phabricator.wikimedia.org/T216867 (10Aklapper) Probably low prio because nowadays folks with CLI access can turn batch edits silent, hence I //assume// that `@Phabricator_maintenance` isn't used anymore. [18:16:17] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T249962 (10brennen) [18:18:22] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/reCaptcha] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/593027 (https://phabricator.wikimedia.org/T250641) [18:18:36] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/extensions/reCaptcha] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/593027 (https://phabricator.wikimedia.org/T250641) (owner: 10Jforrester) [18:19:41] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/reCaptcha] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/593027 (https://phabricator.wikimedia.org/T250641) (owner: 10Jforrester) [18:20:26] !log Zuul: [mediawiki/extensions/reCaptcha] Mark as archived T250641 [18:20:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:20:29] T250641: Archive the reCaptcha extension - https://phabricator.wikimedia.org/T250641 [18:55:21] (03PS1) 10Krinkle: [WIP] Draft Jenkins job for running tests inside a sub-VM [integration/config] - 10https://gerrit.wikimedia.org/r/593034 (https://phabricator.wikimedia.org/T250808) [19:00:40] (03PS2) 10Krinkle: [WIP] Draft Jenkins job for running tests inside a sub-VM [integration/config] - 10https://gerrit.wikimedia.org/r/593034 (https://phabricator.wikimedia.org/T250808) [19:06:38] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T249962 (10brennen) [19:39:45] PROBLEM - Free space - all mounts on deployment-logstash2 is CRITICAL: CRITICAL: deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (No valid datapoints found) deployment-prep.deployment-logstash2.diskspace._var_lib_elasticsearch.byte_percentfree (No valid datapoints found)deployment-prep.deployment-logstash2.diskspace.root.byte_percentfree (<100.00%) [19:43:14] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10serviceops, and 3 others: Cache of wmf-config/InitialiseSettings often 1 step behind - https://phabricator.wikimedia.org/T236104 (10Jdforrester-WMF) Thank you! [20:08:25] 10Phabricator: @Phabricator_maintenance is sending email notifications - https://phabricator.wikimedia.org/T216867 (10MBinder_WMF) That makes sense. If it's possible to train me on CLI access, I'd be game and it would help me and my teams. [20:19:56] 10Gerrit, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Fundraising-Backlog, 10Fundraising Sprint Heaven is a page on Wikipedia, 10Fundraising Sprint I 2020: Reset fr-tech drush repo, create sub-repo for vendor - https://phabricator.wikimedia.org/T249547 (10DStrine) [20:49:06] 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO, 10MediaWiki-extensions-CodeReview, 10Wikimedia-Site-requests, 10Technical-Debt: Undeploy CodeReview - https://phabricator.wikimedia.org/T116948 (10CCicalese_WMF) [20:49:15] 10Release-Engineering-Team, 10Operations, 10Core Platform Team Workboards (Clinic Duty Team), 10Performance Issue, 10Wikimedia-database-error: WikiPage::updateCategoryCounts causing replication lag due to long-running writes on commonswiki - https://phabricator.wikimedia.org/T240405 (10CCicalese_WMF) 05... [20:52:22] 10Phabricator: @Phabricator_maintenance is sending email notifications - https://phabricator.wikimedia.org/T216867 (10Aklapper) @Mbinder_WMF: Currently CLI access is defined by `phabricator-admin` in [puppet's data.yaml](https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/admin/d... [20:59:14] 10phan-taint-check-plugin: ERROR: Error: Undefined constant 'ast\AST_LIST' - https://phabricator.wikimedia.org/T251346 (10cscott) [21:24:19] PROBLEM - Free space - all mounts on deployment-deploy02 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy02.diskspace._srv.byte_percentfree (<11.11%) [21:29:17] RECOVERY - Free space - all mounts on deployment-deploy02 is OK: OK: All targets OK [21:34:22] 10Phabricator: @Phabricator_maintenance is sending email notifications - https://phabricator.wikimedia.org/T216867 (10MBinder_WMF) Done: {T251349} Thanks! [23:01:49] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace._srv.byte_percentfree (<10.00%) [23:11:49] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK