[00:18:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: Move CI instances to use ceph in WMCS - https://phabricator.wikimedia.org/T260916 (10bd808) [00:22:32] 10Project-Admins: Create project for my skin (Cosmos) - https://phabricator.wikimedia.org/T264540 (10Universal_Omega) @Aklapper https://github.com/Universal-Omega/MediaWiki-Cosmos-skin/ will be shut down shortly now that I learned Gerrit, so yes Wikimedia phabricator will be used. [00:36:16] (03PS10) 10DannyS712: Add FinalPrivateSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/628209 (https://phabricator.wikimedia.org/T263188) [00:41:14] 10MediaWiki-Codesniffer, 10Patch-For-Review, 10User-DannyS712: Allow @noinspection in FunctionAnnotationsSniff - https://phabricator.wikimedia.org/T263390 (10DannyS712) a:03DannyS712 [00:41:16] (03CR) 10DannyS712: "This change is ready for review." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/632010 (https://phabricator.wikimedia.org/T263390) (owner: 10DannyS712) [00:43:04] 10MediaWiki-Codesniffer, 10Patch-For-Review, 10User-DannyS712: Add sniff to disallow specific tags with no text after them - https://phabricator.wikimedia.org/T252914 (10DannyS712) @thiemowmde @Krinkle would you mind taking a look at the patch I sent? [01:02:53] 10MediaWiki-Codesniffer: phpcs should error on unclosed @codingStandardsIgnoreStart annotations - https://phabricator.wikimedia.org/T127241 (10DannyS712) Hmm, looking into trying to write a sniff for it, but I couldn't tell if the sniff would be able to identify the tags because they would just disable the sniff... [01:08:05] 10MediaWiki-Codesniffer, 10VPS-project-codesearch, 10User-DannyS712: Add codesniffer to codesearch - https://phabricator.wikimedia.org/T264557 (10DannyS712) [01:58:39] 10MediaWiki-Codesniffer, 10Patch-For-Review, 10User-DannyS712: phpcs should error on unclosed @codingStandardsIgnoreStart annotations - https://phabricator.wikimedia.org/T127241 (10DannyS712) a:03DannyS712 [01:59:01] (03CR) 10DannyS712: "This change is ready for review." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/632011 (https://phabricator.wikimedia.org/T127241) (owner: 10DannyS712) [02:03:57] (03CR) 10DannyS712: [C: 03+1] handle backslash-prefixed is_null usage [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/631989 (owner: 10MMoll) [02:05:34] 10MediaWiki-Codesniffer, 10Patch-For-Review: Create a sniff to enforce alphabetical order of literal arrays - https://phabricator.wikimedia.org/T247813 (10DannyS712) Given that this new sniff does nothing unless manually applied with `@require-sorted-array`, unless there are any objections soon I plan to +2 th... [02:06:30] (03PS16) 10DannyS712: Add a sniff for `$wg*` variables that aren't globals [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/593617 (https://phabricator.wikimedia.org/T251443) [02:08:05] (03CR) 10DannyS712: Add PropertyDocumentationSniff to validate @var on class properties (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/619304 (owner: 10Umherirrender) [04:09:12] * DannyS712 is poking around on the beta cluster [04:18:23] 10Beta-Cluster-Infrastructure, 10User-DannyS712: Beta cluster: stat failed for PagedTiffHandler/extension.json - https://phabricator.wikimedia.org/T264563 (10DannyS712) [04:18:39] * DannyS712 is unable to poke around on the beta cluster, per ^ [04:51:40] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10lmata) Hello @mmodel, I do not see myself on this list for acl*Project-Admins Is there anything additional I should do on my end? Maybe I have missed some steps. please advise, th... [05:06:44] 10MediaWiki-Codesniffer, 10VPS-project-codesearch, 10Patch-For-Review, 10User-DannyS712: Add codesniffer to codesearch - https://phabricator.wikimedia.org/T264557 (10Legoktm) 05Open→03Resolved a:03DannyS712 [05:41:45] gitlab test instance down? [05:42:20] Failed to write to log, write /var/log/gitlab/gitlab-shell/gitlab-shell.log: no space left on device [05:47:54] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Platform Engineering, 10User-DannyS712: [Beta cluster] Table 'centralauth.securepoll_lists' doesn't exist - https://phabricator.wikimedia.org/T264477 (10LSobanski) #DBA team doesn't manage the beta cluster. Adding the #release-engineering-team wh... [05:48:50] brennen: ^ gitlab test is down (when you're awake, I hope this doesn't wake you) [06:57:14] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+2] "I never used this tag, but it looks like it is well documented, as well as pretty self-explanatory." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/632010 (https://phabricator.wikimedia.org/T263390) (owner: 10DannyS712) [06:58:13] (03Merged) 10jenkins-bot: FunctionAnnotationsSniff: allow @noinspection [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/632010 (https://phabricator.wikimedia.org/T263390) (owner: 10DannyS712) [07:00:15] 10MediaWiki-Codesniffer, 10User-DannyS712: Allow @noinspection in FunctionAnnotationsSniff - https://phabricator.wikimedia.org/T263390 (10DannyS712) 05Open→03Resolved [07:23:37] (03PS1) 10AndyRussG: Create base Buster image [releng/dev-images] - 10https://gerrit.wikimedia.org/r/632171 (https://phabricator.wikimedia.org/T262976) [07:36:43] (03PS1) 10AndyRussG: Create fundraising-buster-php73-apache2-xdebug image [releng/dev-images] - 10https://gerrit.wikimedia.org/r/632173 (https://phabricator.wikimedia.org/T262976) [08:00:04] Project mwcore-phpunit-coverage-master build #951: 04FAILURE in 5 hr 0 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/951/ [09:02:19] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] Move encoding and extensions arg to ruleset.xml (032 comments) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/614285 (https://phabricator.wikimedia.org/T200956) (owner: 10Umherirrender) [09:23:06] Project beta-update-databases-eqiad build #45313: 04FAILURE in 0.87 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/45313/ [09:23:50] Project beta-code-update-eqiad build #315032: 04FAILURE in 44 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/315032/ [09:23:51] Project beta-scap-eqiad build #319590: 04FAILURE in 0.87 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319590/ [09:29:51] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T263177 (10hashar) [09:32:15] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T263177 (10hashar) [09:33:13] Project beta-code-update-eqiad build #315033: 04STILL FAILING in 12 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/315033/ [09:33:48] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T263177 (10hashar) [09:43:13] Project beta-code-update-eqiad build #315034: 04STILL FAILING in 12 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/315034/ [09:53:12] Project beta-code-update-eqiad build #315035: 04STILL FAILING in 11 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/315035/ [09:58:56] 10Release-Engineering-Team, 10Wikimedia-GitHub: Add Discord webhook to wikimedia GitHub for tracking repository changes. - https://phabricator.wikimedia.org/T264559 (10MarcoAurelio) I will change the default branch to master as that seems uncontroversial. Regarding the webhook, I do not think I have done anyth... [10:01:58] 10Release-Engineering-Team, 10Wikimedia-GitHub: Add Discord webhook to wikimedia GitHub for tracking repository changes. - https://phabricator.wikimedia.org/T264559 (10MarcoAurelio) Default branch changed to `master`. [10:03:12] Project beta-code-update-eqiad build #315036: 04STILL FAILING in 11 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/315036/ [10:13:13] Project beta-code-update-eqiad build #315037: 04STILL FAILING in 12 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/315037/ [10:20:06] Project beta-update-databases-eqiad build #45314: 04STILL FAILING in 5.6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/45314/ [10:23:12] Project beta-code-update-eqiad build #315038: 04STILL FAILING in 11 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/315038/ [10:26:03] hashar: it looks the CI jobs for beta are failing for various reasons [10:26:24] https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319590/console <-- looks like a coding error in an extension [10:26:41] https://integration.wikimedia.org/ci/view/Beta/job/beta-code-update-eqiad/315038/console <-- seems like CI is unable to fetch WSArrays [10:26:42] 00:00:10.125 fatal: clone of 'https://gerrit.wikimedia.org/r/mediawiki/extensions/WSArrays' into submodule path '/srv/mediawiki-staging/php-master/extensions/WSArrays' failed [10:26:42] 00:00:10.126 Failed to clone 'WSArrays' a second time, aborting [10:26:42] 00:00:10.127 INFO:mwextupdate:Exit code: 1 [10:26:43] :-( [10:27:01] et [10:27:02] stderr: Fatal error: Uncaught Exception: Unable to open file /srv/mediawiki-staging/php-master/extensions/PagedTiffHandler/extension.json: filemtime(): stat failed for /srv/mediawiki-staging/php-master/extensions/PagedTiffHandler/extension.json in /srv/mediawiki-staging/php-master/includes/registration/ExtensionRegistry.php:176 [10:27:09] fatal: remote error: mediawiki/extensions/WSArrays unavailable [10:27:21] hmm [10:27:26] maybe that extension got deleted but is still registered in mediawiki/extensions.git [10:27:35] let me see [10:27:42] but we don't delete gerrit repos usually [10:27:55] true [10:28:15] but this one is definitely not there anymore [10:28:21] * hashar digs in log [10:28:39] hashar: it was https://phabricator.wikimedia.org/diffusion/EWSA/manage/uris/ [10:28:43] it existed [10:28:47] not sure why it got deleted [10:30:21] yup, it's in mediawiki/extensions.git [10:30:27] I'll remove it from there then [10:30:41] hmm [10:31:08] there is still a repository on the disk at /srv/gerrit/git/mediawiki/extensions/WSArrays.git [10:33:12] Project beta-code-update-eqiad build #315039: 04STILL FAILING in 12 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/315039/ [10:33:15] Project beta-scap-eqiad build #319591: 04STILL FAILING in 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319591/ [10:33:17] Removed at https://gerrit.wikimedia.org/r/c/mediawiki/extensions/+/632022 [10:33:37] that would help [10:33:42] and apparently the repo is empty [10:33:46] yup [10:33:54] it got its-phabricator disabled a few days ago but I don't see how it could be related [10:34:20] oh [10:34:22] the repo was deleted from gerrit [10:34:24] right? [10:34:25] and it is marked hidden [10:34:29] if so, by whom? [10:34:52] because I cannot see it at https://gerrit.wikimedia.org/r/admin/repos/q/filter:WSArrays [10:35:02] will look at it after lunch [10:35:20] ok, this should fix one of the beta-scap issues [10:35:33] changed to hidden by Xxmarijnw on Mon Sep 28 16:07:02 2020 [10:35:39] not sure why it would only surface now [10:35:40] :/ [10:35:50] hashar: can you unhide? [10:35:56] We don't hide repos on gerrit [10:36:02] yeah [10:36:07] there's no point either [10:36:07] I don't have access to the repository [10:36:15] not admin? [10:36:17] Gerrit can't load it for some reason, I would have to dig into it eventually [10:36:20] I am :] [10:36:27] I thought so [10:36:29] weird [10:36:36] yeah [10:36:43] will try to index it later after lunch [10:36:44] maybe via SSH [10:36:46] gotta eat :] [10:36:48] sure [10:36:51] bon appetit [10:36:51] bbl [10:36:53] merci! [10:36:56] :) [10:40:22] I also don't see recent changes at https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/PagedTiffHandler/+/refs/heads/master that would cause the second PHP error [10:44:09] Yippee, build fixed! [10:44:09] Project beta-code-update-eqiad build #315040: 09FIXED in 1 min 8 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/315040/ [10:44:11] Project beta-scap-eqiad build #319592: 04STILL FAILING in 0.9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319592/ [10:44:25] Yay, one fixed [10:44:42] Submodule 'modules/DataTables' (https://github.com/DataTables/DataTables.git) registered for path 'PagesList/modules/DataTables' [10:44:44] Ugh [10:44:49] That doesn't sound right [10:45:20] Ah, submodule [10:49:10] 10Beta-Cluster-Infrastructure, 10Scap: beta-scap-eqiad failing since 2/3 days ago - https://phabricator.wikimedia.org/T264593 (10MarcoAurelio) [10:49:21] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Scap: beta-scap-eqiad failing since 2/3 days ago - https://phabricator.wikimedia.org/T264593 (10MarcoAurelio) [10:49:33] filed ^ [10:49:37] so we don't forget [10:53:52] Project beta-scap-eqiad build #319593: 04STILL FAILING in 2.9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319593/ [11:02:27] Project beta-scap-eqiad build #319594: 04STILL FAILING in 0.92 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319594/ [11:03:41] 10Beta-Cluster-Infrastructure, 10User-DannyS712: Beta cluster: stat failed for PagedTiffHandler/extension.json - https://phabricator.wikimedia.org/T264563 (10Urbanecm) p:05Triage→03Unbreak! Apparently breaks deployment, https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319594/console, triaging appro... [11:03:49] Project beta-scap-eqiad build #319595: 04STILL FAILING in 0.91 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319595/ [11:08:30] Project beta-scap-eqiad build #319596: 04STILL FAILING in 0.91 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319596/ [11:09:42] HitomiAkane: B&C window is ongoing, would you mind joining -operations? [11:10:21] sure [11:10:55] 10Beta-Cluster-Infrastructure, 10User-DannyS712: Beta cluster: stat failed for PagedTiffHandler/extension.json - https://phabricator.wikimedia.org/T264563 (10Urbanecm) For some reason, many folders in /srv/mediawiki-staging/php-master/extensions are empty, even they should have some content. git status shows a... [11:13:49] Project beta-scap-eqiad build #319597: 04STILL FAILING in 0.85 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319597/ [11:17:48] 10Gerrit: mediawiki/extensions/WSArrays unreachable in Gerrit - https://phabricator.wikimedia.org/T264596 (10hashar) [11:20:01] Project beta-update-databases-eqiad build #45315: 04STILL FAILING in 0.99 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/45315/ [11:23:50] Project beta-scap-eqiad build #319598: 04STILL FAILING in 0.95 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319598/ [11:24:45] (03PS3) 10Umherirrender: Add PropertyDocumentationSniff to validate @var on class properties [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/619304 [11:30:07] !log Run cd /srv/mediawiki-staging/php-master/extensions/$1 && git reset HEAD * && git checkout -- * for all extensions at beta (T264563) [11:30:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:30:09] T264563: Beta cluster: stat failed for PagedTiffHandler/extension.json - https://phabricator.wikimedia.org/T264563 [11:30:55] 10Beta-Cluster-Infrastructure, 10User-DannyS712, 10User-Urbanecm: Beta cluster: stat failed for PagedTiffHandler/extension.json - https://phabricator.wikimedia.org/T264563 (10Urbanecm) 05Open→03Resolved a:03Urbanecm Fixed this by doing the following: ` urbanecm@deployment-deploy01:/srv/mediawiki-stagi... [11:31:47] !log Restart build of beta-scap-eqiad to sync beta with master branches [11:31:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:32:16] (03PS4) 10Umherirrender: Add PropertyDocumentationSniff to validate @var on class properties [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/619304 [11:32:46] (03CR) 10Umherirrender: "Fixed a comment and also include the lowercase primitives from Ic116319f95a5f02116bd122d650752498c5cb909 also her" (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/619304 (owner: 10Umherirrender) [11:33:41] sorry Urbanecm I was disconnected, as for https://tr.wikipedia.org/wiki/%C3%96zel:GrupHaklar%C4%B1Listesi the right was moved to sysop [11:33:51] HitomiAkane: yup, I synced that, because you wasn't responding :/ [11:33:53] is this correct [11:34:14] sorry i'm having a slow connection today my fault [11:34:24] can happen to everyone :) [11:34:39] 10Gerrit: mediawiki/extensions/WSArrays unreachable in Gerrit - https://phabricator.wikimedia.org/T264596 (10hashar) 05Open→03Resolved a:03hashar I have hacked the repository on the server to remove the state=hidden. Made it read only instead. [11:34:50] Thank you for your patience [11:35:31] so now the task to be closed as resolved ? right? [11:35:59] Grmblblb [11:36:46] Urbanecm: on beta cluster bunch of files are now owned by root under /srv/mediawiki-staging/php-master/extensions/PagedTiffHandler :) [11:36:48] fixing them [11:37:17] hashar: grr [11:37:22] !log deployment-prep: sudo chown -R jenkins-deploy:wikidev /srv/mediawiki-staging/php-master/extensions/PagedTiffHandler [11:37:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:37:31] well hmm [11:37:36] Should've used sudo -u jenkins-deploy [11:37:48] yeah bunch of files are owned by root now ;] [11:37:51] I think it's many extensions :/ [11:38:28] yeah dont worry [11:38:45] !log deployment-prep: sudo chown -R jenkins-deploy:wikidev /srv/mediawiki-staging/php-master/extensions [11:38:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:38:56] Thanks hashar [11:39:04] At least I fixed the missing extensions part :D [11:39:05] !log deployment-prep sudo chown jenkins-deploy:wikidev /srv/mediawiki-staging/php-master/.git/index [11:39:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:39:19] \o/ [11:39:51] Urbanecm: yeah someone has put WSArrays as a hidden extension [11:40:06] which broke stuff here and there, notably that makes it no more readable over https:// without authentication :] [11:40:20] Oh [11:40:22] possibly mediawiki/extensions.git should have a deployment-prep branch that only registers the extension we need [11:40:37] (and I made WSArrays read-only , though probably we can just delete it) [11:41:05] Why delete it, someone may need it [11:48:12] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Scap: beta-scap-eqiad failing since 2/3 days ago - https://phabricator.wikimedia.org/T264593 (10DannyS712) Is this still failing? Might be a duplicate of {T264563} [11:48:15] Yippee, build fixed! [11:48:16] Project beta-scap-eqiad build #319599: 09FIXED in 16 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319599/ [11:54:08] \o/ [11:54:42] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Scap, 10User-Urbanecm: beta-scap-eqiad failing since 2/3 days ago - https://phabricator.wikimedia.org/T264593 (10Urbanecm) 05Open→03Resolved a:03Urbanecm Fixed by fixing the task Danny mentioned. [12:17:32] 10Release-Engineering-Team (Logspam), 10GeoData, 10Discovery-Search (Current work), 10MW-1.36-notes (1.36.0-wmf.10; 2020-09-22), 10Wikimedia-production-error: Elastica\Exception\ResponseException from line 56 of includes/Searcher.php: - https://phabricator.wikimedia.org/T258641 (10Gehel) 05Open→03Reso... [12:21:26] Yippee, build fixed! [12:21:27] Project beta-update-databases-eqiad build #45316: 09FIXED in 1 min 26 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/45316/ [13:24:59] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Operations: Migrate Gerrit to profile::java - https://phabricator.wikimedia.org/T264182 (10MoritzMuehlenhoff) >>! In T264182#6511478, @hashar wrote: > We need the `dbg` package in... [13:42:27] 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Scap, 10Wikimedia-Site-requests: Move scap plugins from mediawiki-config repo to scap repo - https://phabricator.wikimedia.org/T248490 (10LarsWirzenius) [13:42:40] 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Scap, 10EngProd-Virtual-Hackathon, 10Python3-Porting: Port scap to Python 3.4 - https://phabricator.wikimedia.org/T246025 (10LarsWirzenius) [13:42:54] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Scap, 10MediaWiki Train Development Environment, and 3 others: Create a development & testing environment for scap + train deployments - https://phabricator.wikimedia.org/T255695 (10LarsWirze... [13:43:01] 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)): Learn Go - https://phabricator.wikimedia.org/T234543 (10LarsWirzenius) [13:45:31] (03CR) 10Lars Wirzenius: virt-install.sh: add --force option (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/631940 (owner: 10Ahmon Dancy) [13:46:25] (03CR) 10Lars Wirzenius: [V: 03+2 C: 03+2] Add ControlPath so connection sharing works [tools/train-dev] - 10https://gerrit.wikimedia.org/r/631293 (owner: 10Ahmon Dancy) [13:48:10] (03CR) 10Lars Wirzenius: [C: 03+2] Make sync-world work when there are no proxies [tools/scap] - 10https://gerrit.wikimedia.org/r/631548 (owner: 10Ahmon Dancy) [13:49:01] (03CR) 10Lars Wirzenius: [C: 03+2] Allow scap command path to be overriden via environment var [tools/scap] - 10https://gerrit.wikimedia.org/r/631549 (owner: 10Ahmon Dancy) [13:50:33] (03Merged) 10jenkins-bot: Make sync-world work when there are no proxies [tools/scap] - 10https://gerrit.wikimedia.org/r/631548 (owner: 10Ahmon Dancy) [13:50:49] (03Merged) 10jenkins-bot: Allow scap command path to be overriden via environment var [tools/scap] - 10https://gerrit.wikimedia.org/r/631549 (owner: 10Ahmon Dancy) [13:53:20] (03CR) 10Lars Wirzenius: [C: 03+2] php_fpm: add feature flag to always restart [tools/scap] - 10https://gerrit.wikimedia.org/r/631776 (https://phabricator.wikimedia.org/T264362) (owner: 10Thcipriani) [13:53:50] 10Beta-Cluster-Infrastructure: deployment-snapshot01.deployment-prep.eqiad.wmflabs keeps running out of space - https://phabricator.wikimedia.org/T245402 (10ArielGlenn) Just a note that the swift patchset https://gerrit.wikimedia.org/r/c/operations/puppet/+/601429 was merged and that cherry-pick silently and aut... [13:55:12] (03Merged) 10jenkins-bot: php_fpm: add feature flag to always restart [tools/scap] - 10https://gerrit.wikimedia.org/r/631776 (https://phabricator.wikimedia.org/T264362) (owner: 10Thcipriani) [14:02:38] 10Continuous-Integration-Config, 10Operations, 10SRE-tools, 10Patch-For-Review: Add shell scripts CI validations - https://phabricator.wikimedia.org/T148494 (10jbond) I think this has been added via https://gerrit.wikimedia.org/r/c/operations/puppet/+/602693 can anyone confirm if we are still missing anyth... [14:05:03] 10Release-Engineering-Team, 10Wikimedia-GitHub: Add Discord webhook to wikimedia GitHub for tracking repository changes. - https://phabricator.wikimedia.org/T264559 (10Universal_Omega) >>! In T264559#6516617, @MarcoAurelio wrote: > I will change the default branch to master as that seems uncontroversial. Regar... [14:05:21] 10Release-Engineering-Team, 10Wikimedia-GitHub: Add Discord webhook to wikimedia GitHub for tracking repository changes. - https://phabricator.wikimedia.org/T264559 (10Universal_Omega) >>! In T264559#6516622, @MarcoAurelio wrote: > Default branch changed to `master`. Thank you! [14:52:09] (03CR) 10Ahmon Dancy: virt-install.sh: add --force option (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/631940 (owner: 10Ahmon Dancy) [14:56:22] thcipriani and/or Krenair: I don't want this widely announced but we have a partial VM backup system in the works. We don't have storage capacity for the whole cloud so I'm looking for volunteers to add to the opt-out list. Are there things you know of in deployment-prep (or elsewhere) that are simple enough to recreate that backups are largely useless? [14:57:11] I'd presume in deployment-prep... only really the db master needs backing up [14:57:45] Reedy is an optimist [14:58:13] I mean, if that's right and everything else is properly puppetized that would rule [14:58:41] For some things I am ;P [15:02:07] snapshot02 does not need to be backed up, I can get a new instance going (if nothing else is broken) relatively easily [15:02:17] in deployent-prep [15:04:55] apergos: thanks, adding ^.*snapshot02.*$ [15:05:05] 👍 [15:10:31] (03PS1) 10Ahmon Dancy: train-dev.yml: Set up /home/debian/.config/libvirt/libvirt.conf [tools/train-dev] - 10https://gerrit.wikimedia.org/r/632255 [15:13:23] (03PS4) 10Ahmon Dancy: virt-install.sh: add --force option [tools/train-dev] - 10https://gerrit.wikimedia.org/r/631940 [15:14:42] (03CR) 10Ahmon Dancy: "Depends on 632255" [tools/train-dev] - 10https://gerrit.wikimedia.org/r/631940 (owner: 10Ahmon Dancy) [15:16:03] 10Beta-Cluster-Infrastructure: deployment-snapshot01.deployment-prep.eqiad.wmflabs keeps running out of space - https://phabricator.wikimedia.org/T245402 (10dpifke) Apologies for not being around to help with this last week. I was moving, thus out of office and with limited internet connectivity. In the future... [15:21:41] 10Project-Admins, 10Discovery-Search: Do not dump WikibaseCirrusSearch codebase tasks into some #Discovery-Search team project tag only - https://phabricator.wikimedia.org/T263725 (10CBogen) Guillaume will follow up with Andre. [15:32:17] andrewbogott: all of the integration-agent-docker-* hosts are very much cattle at this point; the one there that would probably require fiddling to get back would be the puppetmaster [15:32:42] for deployment-prep: the puppetmaster and the db are the ones that jump to mind as being hard to restore [15:33:36] thcipriani: I have '"^integration-agent.*$"' already added to my ignore list, so that should cover that project i think [15:33:46] +1 [15:34:07] For deployment-prep, it sounds like I need to re-learn how to write negative regexps :) [15:34:12] (!?) [15:34:16] heh [15:34:37] '^deployment-docker-.*$' should be fine to skip for sure, if that's useful [15:36:32] what about deployment-elastic07? Those are also pretty big [15:37:01] The search data can definitely be re-generated [15:37:29] cool [15:37:48] And if there's no search in beta for a while... It doesn't feel the biggest issue out there [15:38:18] the other big ones are deployment-ms-be06 [15:38:22] no idea what that is [15:38:44] and I guess deployment-aqs03 [15:39:08] media storage backend [15:39:10] swift storage backed [15:39:11] so uploads [15:39:31] AQS is analytics? [15:39:40] sounds right [15:40:02] ok, so the swift stuff is probably pretty important :) [15:40:35] ah, aqs == Analytics Query Service + Cassanrda [15:40:56] which would likely also be hard to restore [15:42:20] so far I have [15:42:21] https://gerrit.wikimedia.org/r/c/operations/puppet/+/632253/2/modules/profile/templates/wmcs/backy2/wmcs_backup_instances.yaml.erb [15:43:13] greg-g: gitlab-test should be back up shortly; disk was filling with prometheus data. [15:43:19] disabled that and restarted. [15:43:47] !log restarting gitlab-test after disabling prometheus metrics [15:43:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:45:28] brennen: thank you! [15:47:28] Project beta-scap-eqiad build #319624: 04FAILURE in 3 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319624/ [15:49:39] >ssh: connect to host deployment-mediawiki-07.deployment-prep.eqiad.wmflabs port 22: No route to host [15:49:42] looks like deployment-mediawiki-07 is shutdown? [15:49:50] according to horizon [15:50:14] (03CR) 10Lars Wirzenius: [V: 03+2 C: 03+2] train-dev.yml: Set up /home/debian/.config/libvirt/libvirt.conf [tools/train-dev] - 10https://gerrit.wikimedia.org/r/632255 (owner: 10Ahmon Dancy) [15:50:27] (03CR) 10Lars Wirzenius: [V: 03+2 C: 03+2] virt-install.sh: add --force option [tools/train-dev] - 10https://gerrit.wikimedia.org/r/631940 (owner: 10Ahmon Dancy) [15:50:33] (03PS8) 10Ahmon Dancy: Add app servers to train-dev [tools/train-dev] - 10https://gerrit.wikimedia.org/r/631509 (https://phabricator.wikimedia.org/T260674) [15:50:35] !log start deployment-mediawiki-07 [15:50:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:51:12] (03CR) 10Lars Wirzenius: [V: 03+2 C: 03+2] virt-install.sh: add --force option (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/631940 (owner: 10Ahmon Dancy) [15:51:16] thank you thcipriani [15:53:28] Reedy: I just moved a bunch of deployment-prep things to ceph; everything should be back now [15:54:23] migration details at https://lists.wikimedia.org/pipermail/cloud-announce/2020-September/000319.html [15:57:21] Yippee, build fixed! [15:57:21] Project beta-scap-eqiad build #319625: 09FIXED in 3 min 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319625/ [15:59:43] andrewbogott: I did presume it might've been that :) [16:00:49] I figured :) I think the deployment-prep bits are moved now so my script is on to confusing other people [16:47:17] (03PS9) 10Ahmon Dancy: Add app servers to train-dev [tools/train-dev] - 10https://gerrit.wikimedia.org/r/631509 (https://phabricator.wikimedia.org/T260674) [17:14:31] thanks for fixing WSArrays repo hashar :-) [17:20:18] 10Continuous-Integration-Config, 10Operations, 10SRE-tools, 10Patch-For-Review: Add shell scripts CI validations - https://phabricator.wikimedia.org/T148494 (10hashar) >>! In T148494#6517546, @jbond wrote: > I think this has been added via https://gerrit.wikimedia.org/r/c/operations/puppet/+/602693 can any... [17:33:11] 10MediaWiki-Codesniffer, 10User-DannyS712: Sniff for wrong order of PHPUnit assert parameters - https://phabricator.wikimedia.org/T253910 (10DannyS712) @thiemowmde @Krinkle would you be willing to take a look at https://gerrit.wikimedia.org/r/c/mediawiki/tools/codesniffer/+/599482/ please? [17:38:39] 10MediaWiki-Codesniffer: How to handle the nullable operator on multi type lists in code - https://phabricator.wikimedia.org/T250734 (10DannyS712) Sounds good to me, would probably need to go in FunctionCommentSniff with the handling of nullable parameters if there is no standard upstream [18:28:06] 10Project-Admins: Create project for my skin (Cosmos) - https://phabricator.wikimedia.org/T264540 (10Universal_Omega) @Aklapper the original repository is now Archived. [18:32:02] (03Abandoned) 10Hashar: helm-linter: Update envoy to 1.15.1 [integration/config] - 10https://gerrit.wikimedia.org/r/631736 (https://phabricator.wikimedia.org/T264157) (owner: 10Hashar) [18:34:57] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Scap, 10Performance-Team (Radar), 10User-Ladsgroup: Add Scap support for static-array format of LCStore - https://phabricator.wikimedia.org/T105683 (10Krinkle) 05Open→03Resolved [18:35:04] 10Release-Engineering-Team-TODO, 10Scap, 10MediaWiki-Internationalization, 10Performance-Team, 10Patch-For-Review: Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740 (10Krinkle) [18:35:51] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T263177 (10hashar) >>! In T263177#6514497, @Tacsipacsi wrote: > Since all wikis are on wmf.10 currently, does this mean... [19:05:40] once blockers are resolved, is wmf11 or wmf12 (skipping wfm11) or both (one after the other in quick succession) being deployed this week? [19:22:47] 10Gerrit: Cannot see my own changesets - https://phabricator.wikimedia.org/T264657 (10Amd989) [19:28:23] subbu|lunch: that first sentence fragment is a doozy :) [19:39:20] Yippee, build fixed! [19:39:21] Project mwcore-phpunit-coverage-master build #952: 09FIXED in 4 hr 39 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/952/ [19:46:20] greg-g, ok :) [19:47:00] Project beta-scap-eqiad build #319648: 04FAILURE in 3 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319648/ [19:57:22] Project beta-scap-eqiad build #319649: 04STILL FAILING in 3 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319649/ [20:01:59] 10LibUp, 10Page-Previews, 10Readers-Web-Backlog (Tracking): Update nvm for libup to match CI - https://phabricator.wikimedia.org/T264328 (10Jdlrobson) [20:07:15] Project beta-scap-eqiad build #319650: 04STILL FAILING in 3 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319650/ [20:07:15] 10Gerrit: Cannot see my own changesets - https://phabricator.wikimedia.org/T264657 (10Aklapper) 05Open→03Stalled Hi @Amd989, thanks for taking the time to report this and welcome to Wikimedia Phabricator! This seems to be about https://gerrit.wikimedia.org/r/c/632286 (please always follow https://www.mediaw... [20:14:15] 10Project-Admins: Create project for my skin (Cosmos) - https://phabricator.wikimedia.org/T264540 (10Aklapper) 05Open→03Resolved a:03Aklapper Thanks! Requested public project #Cosmos has been created: https://phabricator.wikimedia.org/project/view/5017/ Interested people are welcome to join the project as... [20:17:35] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Scap, 10Sustainability (Incident Followup): Make scap skip restarting php-fpm when using --force - https://phabricator.wikimedia.org/T243009 (10Krinkle) >>! In T253673#6509807, @Krinkle wrot... [20:25:50] The last Puppet run was at Tue Sep 29 13:26:28 UTC 2020 (9058 minutes ago). [20:25:52] Great [20:27:05] Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Function lookup() did not find a value for the name 'profile::mediawiki::mcrouter_wancache::use_onhost_memcached' (file: /etc/puppet/modules/profile/manifests/mediawiki/mcrouter_wancache.pp, line: 4) on node deployment-mediawiki-07.deployment-prep.eqiad.wmflabs [20:27:27] Yippee, build fixed! [20:27:27] Project beta-scap-eqiad build #319651: 09FIXED in 4 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319651/ [20:36:21] Project beta-scap-eqiad build #319653: 04FAILURE in 2 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319653/ [20:41:00] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [extensions/LDAPSyncAll] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/632316 [20:41:02] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [extensions/LDAPSyncAll] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/632316 (owner: 10QChris) [20:41:06] (03PS1) 10QChris: Import done. Revoke import grants [extensions/LDAPSyncAll] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/632317 [20:41:08] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [extensions/LDAPSyncAll] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/632317 (owner: 10QChris) [20:45:54] !log Fixed puppet for deployment-mediawiki-07: s/memcache/memcached/ [20:45:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:47:14] Project beta-scap-eqiad build #319654: 04STILL FAILING in 3 min 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319654/ [20:48:53] .... [20:50:38] Urbanecm: well that one is weird [20:51:22] hauskatze: well it matches what i see locally at the very least [20:51:25] trying to fix that :D [20:57:26] Yippee, build fixed! [20:57:27] Project beta-scap-eqiad build #319655: 09FIXED in 3 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319655/ [20:58:13] 10Release-Engineering-Team-TODO, 10Release, 10Train Deployments: 1.36.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T263178 (10hashar) [20:58:17] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T263177 (10hashar) [21:00:01] !log Run puppet at deployment-mwmaint01 and deployment-mediawiki-07 [21:00:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:00:12] !log Run keyholder arm at deployment-cumin [21:00:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:01:45] 10Release-Engineering-Team-TODO, 10Release, 10Train Deployments: 1.36.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T263178 (10hashar) Last week we could not deploy the scheduled update 1.36.0-wmf.11 (T263177) and its status is still pending resolution of {T264370}. We do not yet have a p... [21:06:46] Project beta-scap-eqiad build #319656: 04FAILURE in 2 min 52 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319656/ [21:07:19] Here we go again [21:07:43] hauskatze ...and the memories of mamma mia pop into my head [21:07:59] mamma mía, here we go again [21:09:21] my my, how can I resist fix you [21:11:14] Blue since the day we partedbeta-scap-eqiad failed [21:17:29] Yippee, build fixed! [21:17:29] Project beta-scap-eqiad build #319657: 09FIXED in 3 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/319657/ [22:07:00] (03PS5) 10Nikki Nikkhoui: Experimental WVUI pipeline update [integration/config] - 10https://gerrit.wikimedia.org/r/631864 [22:21:28] 10Beta-Cluster-Infrastructure: Beta cluster has reached its quota - https://phabricator.wikimedia.org/T257118 (10Urbanecm) [22:22:48] 10Beta-Cluster-Infrastructure: Beta cluster has reached its quota - https://phabricator.wikimedia.org/T257118 (10Urbanecm) [22:29:21] !log deleted deployment-imagescaler01 and deployment-imagescaler02 (T257118) [22:29:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:29:23] T257118: Beta cluster has reached its quota - https://phabricator.wikimedia.org/T257118 [22:31:06] !log deleted deployment-mailman01 (T257118) [22:31:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:31:41] 10Beta-Cluster-Infrastructure: Beta cluster has reached its quota - https://phabricator.wikimedia.org/T257118 (10Krinkle) [22:32:30] 10Beta-Cluster-Infrastructure: Beta cluster has reached its quota - https://phabricator.wikimedia.org/T257118 (10Krinkle) > deployment-poolcounter06 Marking as needed on behalf of Platform Engineering team. This is the primary and only poolcounter node, and is indeed being used per [LabsServices](https://gerri... [23:18:27] (03CR) 10BryanDavis: "Bump. It would be nice to have this merged and deployed at https://blubberoid.wikimedia.org so I can continue work on converting the dev e" [blubber] - 10https://gerrit.wikimedia.org/r/629877 (owner: 10BryanDavis) [23:34:06] 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO, 10Code-Stewardship-Reviews: Code Stewardship Review: OAuth extension - https://phabricator.wikimedia.org/T224919 (10WDoranWMF) @Jrbranaa Platform have done quite a bit on this I'm marking us as owners unless there is an objection and... [23:36:03] 10Release-Engineering-Team (Code Health), 10MediaWiki-extensions-ShortUrl, 10Platform Engineering, 10Code-Stewardship-Reviews: Code Stewardship Review: ShortUrl Extension - https://phabricator.wikimedia.org/T187045 (10WDoranWMF) @Krinkle is there a possibility to integrate this with urlshortener or vice ve... [23:39:08] 10Release-Engineering-Team (Code Health), 10MediaWiki-extensions-ShortUrl, 10Platform Engineering, 10Code-Stewardship-Reviews: Code Stewardship Review: ShortUrl Extension - https://phabricator.wikimedia.org/T187045 (10Ladsgroup) >>! In T187045#6519795, @WDoranWMF wrote: > @Krinkle is there a possibility to... [23:45:54] 10Beta-Cluster-Infrastructure: Beta cluster has reached its quota - https://phabricator.wikimedia.org/T257118 (10Ladsgroup) [23:48:38] 10Release-Engineering-Team (Code Health), 10MediaWiki-extensions-ShortUrl, 10Platform Engineering, 10Code-Stewardship-Reviews: Code Stewardship Review: ShortUrl Extension - https://phabricator.wikimedia.org/T187045 (10Krinkle) I think the people involved here generally agree that there is no need to provid... [23:52:01] 10Beta-Cluster-Infrastructure: Beta cluster has reached its quota - https://phabricator.wikimedia.org/T257118 (10Krinkle)