[00:20:37] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201911): mwgate-composer-php72-docker missing php-intl - https://phabricator.wikimedia.org/T236907 (10Reedy) [00:20:39] 10Release-Engineering-Team-TODO (201911), 10MediaWiki-General, 10MediaWiki-Page-Diffs, 10wikidiff2, and 3 others: DifferenceEngineTest and ApiComparePagesTest failures when wikidiff2 is installed - https://phabricator.wikimedia.org/T237049 (10Reedy) 05Open→03Resolved a:03Anomie Ok, cool. So its just... [00:40:33] (03PS1) 10Brennen Bearnes: stretch-php72-fpm-apache: allow cache to expire [releng/dev-images] - 10https://gerrit.wikimedia.org/r/548941 [00:55:01] 10MediaWiki-Releasing, 10Core Platform Team, 10MW-1.34-notes, 10MW-1.34-release: Release MW 1.34 - https://phabricator.wikimedia.org/T232023 (10Krinkle) >>! From **task description**: > This is a task for the mechanics of doing the release. […] tasks that you believe should block the release […] add them... [00:55:11] 10MediaWiki-Releasing, 10Core Platform Team, 10MW-1.34-notes, 10MW-1.34-release: Release MW 1.34 - https://phabricator.wikimedia.org/T232023 (10Krinkle) [01:03:03] 10Gerrit, 10Release-Engineering-Team, 10Operations, 10Wikimedia Design Style Guide (Wikimedia Design Style Guide v1.1): Automatic pickup of Gerrit clone master doesn't happen due to missing git-lfs – new deployment env - https://phabricator.wikimedia.org/T235677 (10Volker_E) [01:44:28] (03CR) 10Jeena Huneidi: [C: 03+1] stretch-php72-fpm-apache: allow cache to expire [releng/dev-images] - 10https://gerrit.wikimedia.org/r/548941 (owner: 10Brennen Bearnes) [02:37:05] (03PS1) 10Zoranzoki21: Remove labs/tools/discordwiki from zuul/layout.yaml [integration/config] - 10https://gerrit.wikimedia.org/r/548946 (https://phabricator.wikimedia.org/T237491) [03:26:38] 10Gerrit, 10Gerrit-Privilege-Requests, 10Pywikibot, 10User-MarcoAurelio: Give access to l10n-bot-watcher group - https://phabricator.wikimedia.org/T237263 (10abi_) Thank you! [08:47:34] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201911), 10Jenkins, 10Upstream: JJB is too slow - https://phabricator.wikimedia.org/T236746 (10hashar) a:03hashar [08:47:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201911), 10Jenkins, 10Upstream: JJB is too slow - https://phabricator.wikimedia.org/T236746 (10hashar) 05Stalled→03Open [08:48:33] (03PS1) 10Hashar: Upgrade jjb to 3.1.0 [integration/config] - 10https://gerrit.wikimedia.org/r/548964 (https://phabricator.wikimedia.org/T236746) [08:57:32] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Growth-Team, and 3 others: Missing XML files cause "Publish Performance test result report" - https://phabricator.wikimedia.org/T164296 (10hashar) 05Open→03Resolved a:03hasha... [09:02:49] !log Jenkins: upgrade Monitoring plugin (javamelody) to 1.80.0 + misc updates [09:02:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:04:03] !log Restarted CI Jenkins [09:04:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:10:51] (03CR) 10Hashar: [C: 03+2] Update RESTBase link [integration/docroot] - 10https://gerrit.wikimedia.org/r/547846 (https://phabricator.wikimedia.org/T227766) (owner: 10Ammarpad) [09:11:34] (03Merged) 10jenkins-bot: Update RESTBase link [integration/docroot] - 10https://gerrit.wikimedia.org/r/547846 (https://phabricator.wikimedia.org/T227766) (owner: 10Ammarpad) [09:14:29] Yippee, build fixed! [09:14:29] Project beta-code-update-eqiad build #270957: 09FIXED in 1 min 28 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/270957/ [09:17:29] (03CR) 10Hashar: [C: 03+2] "https://doc.wikimedia.org/ updated :)" [integration/docroot] - 10https://gerrit.wikimedia.org/r/547846 (https://phabricator.wikimedia.org/T227766) (owner: 10Ammarpad) [09:18:08] 10Continuous-Integration-Config, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201911), 10RESTBase, and 3 others: Remove link to defunct https://rest.wikimedia.org/ on https://doc.wikimedia.org/ - https://phabricator.wikimedia.org/T227766 (10hashar) 05Open→03Resolve... [09:45:23] 10Release-Engineering-Team-TODO (201911), 10MediaWiki-General, 10MediaWiki-Page-Diffs, 10wikidiff2, and 3 others: DifferenceEngineTest and ApiComparePagesTest failures when wikidiff2 is installed - https://phabricator.wikimedia.org/T237049 (10hashar) 05Resolved→03Open That fix it for `master` but the r... [09:45:25] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201911): mwgate-composer-php72-docker missing php-intl - https://phabricator.wikimedia.org/T236907 (10hashar) [10:00:29] 10Differential, 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10GitHub-Mirrors: Create and maintain somehow a list of repos mastered in GitHub (and in Differential) - https://phabricator.wikimedia.org/T237470 (10hashar) Github had a feature to declare a rep... [10:03:00] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10Wikimedia-Incident: git review notes split brain - https://phabricator.wikimedia.org/T236443 (10hashar) Well done Tyler thank you :] [10:07:08] 10Gerrit, 10Release-Engineering-Team, 10serviceops: Gerrit crashed due to out of Heap - https://phabricator.wikimedia.org/T225166 (10hashar) 05Open→03Resolved The heap size has been grown from 20G to 32G ( https://gerrit.wikimedia.org/r/545381 ) We have a new server which is way more powerful We did a... [10:16:01] 10Differential, 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10GitHub-Mirrors: Create and maintain somehow a list of repos mastered in GitHub (and in Differential) - https://phabricator.wikimedia.org/T237470 (10Aklapper) This is basically the inverted vers... [10:16:22] 10Differential, 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10GitHub-Mirrors: Create and maintain somehow a list of repos mastered in GitHub (and in Differential) - https://phabricator.wikimedia.org/T237470 (10Aklapper) [10:17:42] 10Gerrit, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Scap: `scap clean` failure - https://phabricator.wikimedia.org/T218783 (10hashar) The error now more shows up since the cleaning has been disabled behind a feature flag by https://gerrit.wikimedia.org/r/#/c/opera... [10:30:25] 10Gerrit, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Security-Team: wikimedia/security gerrit requests - https://phabricator.wikimedia.org/T237449 (10Aklapper) [10:38:36] (03CR) 10Zfilipin: [C: 03+1] Upgrade jjb to 3.1.0 [integration/config] - 10https://gerrit.wikimedia.org/r/548964 (https://phabricator.wikimedia.org/T236746) (owner: 10Hashar) [10:47:40] 10Project-Admins, 10Research: create tags for Research - https://phabricator.wikimedia.org/T228350 (10Aklapper) >>! In T228350#5637500, @leila wrote: > Looking at the list of subprojects for Research I'm wondering: do you think it's possible to also make address-knowledge-gap and knowledge-integrity subproject... [11:31:22] !log Deleted https://github.com/wikimedia/labs-tools-discordwiki | T237491 [11:31:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:31:25] T237491: Delete labs/tools/discordwiki on Gerrrit - https://phabricator.wikimedia.org/T237491 [11:41:34] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10MediaWiki-Core-Testing, 10Code-Health: Quibble + PHPUnit: Run databaseless and database tests in parallel - https://phabricator.wikimedia.org/T235449 (10kostajh) > As a stopgap, the log format can be configured to... [12:46:25] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10Performance-Team, and 2 others: Release new version of wdio-mediawiki - https://phabricator.wikimedia.org/T236927 (10zeljkofilipin) @Krinkle I've merged the commit. You have usual... [12:54:11] (03CR) 10Hashar: [C: 04-1] Remove labs/tools/discordwiki from zuul/layout.yaml (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/548946 (https://phabricator.wikimedia.org/T237491) (owner: 10Zoranzoki21) [13:29:30] PROBLEM - Host deployment-maps04 is DOWN: CRITICAL - Host Unreachable (172.16.4.10) [13:39:33] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10MW-1.35-notes (1.35.0-wmf.5; 2019-11-05), and 3 others: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [13:41:32] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10MW-1.35-notes (1.35.0-wmf.5; 2019-11-05), and 3 others: Repositories with Selenium tests should not use mediawiki code directly - https://phabricator.wikimedia.org/T199113 (10zeljk... [13:41:57] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10MW-1.35-notes (1.35.0-wmf.5; 2019-11-05), and 3 others: Repositories with Selenium tests should not use mediawiki code directly - https://phabricator.wikimedia.org/T199113 (10zeljk... [13:42:14] (03Abandoned) 10Zfilipin: WIP npm ERR! missing script: selenium-daily WIP [integration/config] - 10https://gerrit.wikimedia.org/r/546679 (https://phabricator.wikimedia.org/T199113) (owner: 10Zfilipin) [13:43:00] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10MW-1.35-notes (1.35.0-wmf.5; 2019-11-05), and 3 others: Repositories with Selenium tests should not use mediawiki code directly - https://phabricator.wikimedia.org/T199113 (10zeljk... [13:44:44] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10MW-1.35-notes (1.35.0-wmf.5; 2019-11-05), and 3 others: Repositories with Selenium tests should not use mediawiki code directly - https://phabricator.wikimedia.org/T199113 (10zeljk... [13:46:10] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10MW-1.35-notes (1.35.0-wmf.5; 2019-11-05), and 3 others: Repositories with Selenium tests should not use mediawiki code directly - https://phabricator.wikimedia.org/T199113 (10zeljk... [13:47:13] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10MW-1.35-notes (1.35.0-wmf.5; 2019-11-05), and 3 others: Repositories with Selenium tests should not use mediawiki code directly - https://phabricator.wikimedia.org/T199113 (10zeljk... [13:47:36] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Q1 Selenium framework improvements - https://phabricator.wikimedia.org/T198389 (10zeljkofilipin) [13:47:38] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10MW-1.35-notes (1.35.0-wmf.5; 2019-11-05), and 3 others: Repositories with Selenium tests should not use mediawiki code directly - https://phabricator.wikimedia.org/T199113 (10zeljk... [14:02:18] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201911), 10Patch-For-Review: Investigate Docker slowness between 18.06.2 and 18.09.7 - https://phabricator.wikimedia.org/T236675 (10hashar) I guess I can do some profiling using `pe... [14:05:23] (03CR) 10Jforrester: [C: 03+2] "Wow, yes." [integration/config] - 10https://gerrit.wikimedia.org/r/548964 (https://phabricator.wikimedia.org/T236746) (owner: 10Hashar) [14:06:45] (03Merged) 10jenkins-bot: Upgrade jjb to 3.1.0 [integration/config] - 10https://gerrit.wikimedia.org/r/548964 (https://phabricator.wikimedia.org/T236746) (owner: 10Hashar) [14:07:03] (03PS1) 10Jforrester: Re-apply "jjb: Bump all jobs for new php71/72/73 image cascade" [integration/config] - 10https://gerrit.wikimedia.org/r/549086 (https://phabricator.wikimedia.org/T236907) [14:07:12] (03CR) 10jerkins-bot: [V: 04-1] Re-apply "jjb: Bump all jobs for new php71/72/73 image cascade" [integration/config] - 10https://gerrit.wikimedia.org/r/549086 (https://phabricator.wikimedia.org/T236907) (owner: 10Jforrester) [14:09:05] (03PS2) 10Jforrester: Re-apply "jjb: Bump all jobs for new php71/72/73 image cascade" [integration/config] - 10https://gerrit.wikimedia.org/r/549086 (https://phabricator.wikimedia.org/T236907) [14:12:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201911), 10Test-Coverage: mwcore-phpunit-coverage-master times out after 5 hours - https://phabricator.wikimedia.org/T232706 (10hashar) All the builds since last capture: | Status... [14:22:15] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201911): Evaluate Airflow's suitability for CI - https://phabricator.wikimedia.org/T237450 (10Ottomata) Ah interesting, good to know. Can Argo Events also produce external events, perhaps via Kafka? Something like that might make integ... [14:23:07] Hi Release Engineering! I am the (relatively) new Sr. Data Strategist on the Advancement team working with Fundraising, and I am hoping for help troubleshooting a new private gerrit repo we have recently set up. I recently committed a few files to the master branched, pushed for review, and received a confirmation gerrit confirmation URL - which errors out [14:23:59] I have the URL, screen shot of my terminal window, and the change ID for additional info - please let me know what's needed [14:26:06] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201911), 10Jenkins, 10Upstream: JJB is too slow - https://phabricator.wikimedia.org/T236746 (10hashar) 05Open→03Resolved Before 3.1.0: ` $ time ./utils/jjb-diff.sh real 1m25,6... [14:30:12] Eyener: hi, what error do you get (and the url please) [14:33:49] Hi paladox! Here is the URL which errors out when I attempt to load it (I'm logged in): https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/analytics/+/546757 [14:34:27] Hmm [14:34:40] No other errors in the terminal or while executing git commands [14:34:59] I think the change either didn’t make it into the repo or it’s gone missing some how... [14:35:12] I see no errors in Java melody that would explain it [14:35:34] cc thcipriani ^ [14:35:52] Would you like the change ID? [14:35:56] Eyener: can you repush please? [14:36:10] Yeh, I suggest that and try repushing [14:37:33] eyener: hold on repushing just for a sec, please [14:37:51] okay [14:38:10] thanks [14:38:41] Oh! [14:38:42] I carn’t seem to view the repo [14:40:25] Thcipriani: I think it must be because he’s not in the group that is the owner of that repo [14:40:26] But then again I dunno how he managed to push if he’s not [14:40:33] paladox: it would be likely that you don't have permissions. [14:41:15] eyener: I'm assuming you'reErin Yener on gerrit, correct? [14:41:59] Yes that's me [14:42:11] hrm, that should put you in the group that owns this repo. [14:42:17] Yeh [14:42:48] this is the only change I see for that repo. I suspect there's something funny with permissions here. [14:43:17] FWIW, I can see the change on disk and it looks fine, but not through the web UI (and I'm an admin...so that's strange :)) [14:43:39] Ha well - it's somewhere! [14:43:42] * paladox came to that conclusion after i couldn’t find the repo [14:44:55] thcipriani: see if granting read to that repo fixes it, or do you not see the change too? [14:45:50] I can see the repo, but not the change, lemme try that. my suspicion is that because it inherits from wikimedia/fundraising and eyener is not in the "fundraising" group (only "fundraising/analytics") that might be the issue. [14:46:08] (Read to that group) [14:47:12] 10Release-Engineering-Team, 10Wikimedia Design Style Guide, 10Patch-For-Review, 10User-Ladsgroup: Use `git lfs` for large binary files of Design Style Guide - https://phabricator.wikimedia.org/T235013 (10ArielGlenn) Cons for git-lfs: it doesn't support ssh, only http(s), although apparently there's an auth... [14:47:55] eyener: I fiddled with perms a bit, can you see it now? https://gerrit.wikimedia.org/r/546757 [14:49:38] Yes! Thank you thcipriani! [14:49:48] eyener: great :) [14:50:20] paladox: granting read to wikimedia-fundraising-analytics (on the repo they own) seems to have done the trick ;) [14:50:50] Great. Will the others in the repo be good to go now too? [14:54:15] yeah, I think so, lemme know if anything else seems amiss [15:00:09] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201907), 10Patch-For-Review: Investigate gerrit session expiration - https://phabricator.wikimedia.org/T222472 (10Aklapper) >>! In T222472#5634681, @Mainframe98 wrote: > I was too, until today. I had to log in aga... [15:16:51] 10Release-Engineering-Team, 10Wikimedia Design Style Guide, 10Patch-For-Review, 10User-Ladsgroup: Use `git lfs` for large binary files of Design Style Guide - https://phabricator.wikimedia.org/T235013 (10akosiaris) >>! In T235013#5580957, @Dzahn wrote: > @Ladsgroup git-lfs is not installed on the prod serv... [15:35:40] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201911): Evaluate Airflow's suitability for CI - https://phabricator.wikimedia.org/T237450 (10LarsWirzenius) We may have to write a small adaptor, but other than that, yes. [15:39:58] https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/ looks like it may need attention [15:40:32] !log Upgraded mediawiki-quibble-composer-mysql-php72-docker as a test flight for T236907 T237049 [15:40:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:40:36] T236907: mwgate-composer-php72-docker missing php-intl - https://phabricator.wikimedia.org/T236907 [15:40:36] T237049: DifferenceEngineTest and ApiComparePagesTest failures when wikidiff2 is installed - https://phabricator.wikimedia.org/T237049 [15:42:42] mdholloway: hmm, seen that before [16:06:17] (03CR) 10Jforrester: [C: 03+2] "Deployed, and all seems well (he says…)." [integration/config] - 10https://gerrit.wikimedia.org/r/549086 (https://phabricator.wikimedia.org/T236907) (owner: 10Jforrester) [16:07:07] (03Merged) 10jenkins-bot: Re-apply "jjb: Bump all jobs for new php71/72/73 image cascade" [integration/config] - 10https://gerrit.wikimedia.org/r/549086 (https://phabricator.wikimedia.org/T236907) (owner: 10Jforrester) [16:07:21] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201911), 10Patch-For-Review: mwgate-composer-php72-docker missing php-intl - https://phabricator.wikimedia.org/T236907 (10Jdforrester-WMF) [16:07:23] 10Release-Engineering-Team-TODO (201911), 10MediaWiki-General, 10MediaWiki-Page-Diffs, 10wikidiff2, and 5 others: DifferenceEngineTest and ApiComparePagesTest failures when wikidiff2 is installed - https://phabricator.wikimedia.org/T237049 (10Jdforrester-WMF) 05Open→03Resolved >>! In T237049#5639509, @... [16:07:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201911), 10Patch-For-Review: mwgate-composer-php72-docker missing php-intl - https://phabricator.wikimedia.org/T236907 (10Jdforrester-WMF) 05Open→03Resolved Deployed. [16:07:47] 10Release-Engineering-Team (Deployment services), 10Security-Team, 10Wikimedia-Extension-setup, 10Wikimedia-Site-requests, and 2 others: Deploy WebAuthn to Wikimedia Wikis - https://phabricator.wikimedia.org/T227242 (10Jdforrester-WMF) [16:19:01] 10Release-Engineering-Team-TODO (201911): Upgrade PHP 7.2.x CI images to PHP 7.2.24 to align with production - https://phabricator.wikimedia.org/T237275 (10Jdforrester-WMF) a:03Jdforrester-WMF [16:19:47] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201911), 10Patch-For-Review: Investigate Docker slowness between 18.06.2 and 18.09.7 - https://phabricator.wikimedia.org/T236675 (10Jdforrester-WMF) a:03hashar [16:19:49] 10Release-Engineering-Team-TODO (201911), 10Code-Review-Workgroup: Define code review metrics - https://phabricator.wikimedia.org/T229510 (10Jdforrester-WMF) a:03Jdforrester-WMF [16:20:55] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201911), 10Quibble: Create an integration test running Quibble with mediawiki/core - https://phabricator.wikimedia.org/T235118 (10Jdforrester-WMF) [16:21:09] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (201911), 10Developer Productivity, 10RESTBase, and 3 others: Move restbase chart from local-charts to deployment-charts repository - https://phabricator.wikimedia.org/T228910 (10Jdforrester-WMF) [16:21:16] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (201911), 10Developer Productivity, 10local-charts, and 2 others: Add pipeline config and publish step for restbase dev docker image - https://phabricator.wikimedia.org/T234580 (10Jdforrester-WMF) [16:21:23] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (201911), 10Developer Productivity, 10local-charts, 10Patch-For-Review: Update local-charts repository to use restbase chart from deployment-charts repo - https://phabricator.wikimedia.org/T228915 (10Jdforrester-WMF) [16:21:35] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201911), 10Documentation, 10User-MModell: Make PHD run on the backup phabricator server (phab2001, currently) - https://phabricator.wikimedia.org/T232883 (10Jdforrester-WMF) [16:21:43] 10Deployments, 10Release-Engineering-Team-TODO (201911), 10User-MModell: Automate the recurring management of wikitech:Deployments and phab:#train_deployments - https://phabricator.wikimedia.org/T114488 (10Jdforrester-WMF) [16:21:52] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201911), 10MediaWiki-Core-Testing, 10Performance-Team, and 2 others: Release new version of wdio-mediawiki - https://phabricator.wikimedia.org/T236927 (10Jdforrester-WMF) [16:22:02] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201911), 10International-Developer-Events, 10Wikimedia-Technical-Conference-2019, and 2 others: Wikimedia Technical Conference 2019 Session: System level testing: patterns an... - https://phabricator.wikimedia.org/T234635 [16:23:06] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team-TODO (201911), 10Zuul: Zuul cancels all changes when a change is manually merged - https://phabricator.wikimedia.org/T203846 (10Jdforrester-WMF) [16:23:17] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201911), 10Zuul, 10Documentation: Document Zuul problems caused by force merge - https://phabricator.wikimedia.org/T225955 (10Jdforrester-WMF) [16:23:26] 10Phabricator (Search), 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201911), 10User-MModell: Test out the Phabricator 'ferrit' search engine. - https://phabricator.wikimedia.org/T230787 (10Jdforrester-WMF) [16:23:38] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201911): Upgrade Gerrit to 2.15.17 - https://phabricator.wikimedia.org/T229110 (10Jdforrester-WMF) [16:24:18] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201911), 10Browser-Tests, and 2 others: Selenium tests failing for PluggableAuth plugins - https://phabricator.wikimedia.org/T236030 (10Jdforrester-WMF) [16:24:21] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201911), 10MediaWiki-Core-Testing: ERROR webdriver: Request failed due to Error: session not created: Chrome version must be between 70 and 73 - https://phabricator.wikimedia.org/T234610 (10Jdforrester-WMF) [16:24:23] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (201911), 10local-charts, 10Documentation, 10Patch-For-Review: Document how to contribute to local-charts - https://phabricator.wikimedia.org/T236587 (10Jdforrester-WMF) [16:24:25] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201911), 10Toolforge, and 2 others: Add CI checks for golang admission controllers - https://phabricator.wikimedia.org/T236203 (10Jdforrester-WMF) [16:24:27] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201911), 10Patch-For-Review, 10User-zeljkofilipin: Upgrade webdriverio to version 5 for all repositories - https://phabricator.wikimedia.org/T234314 (10Jdforrester-WMF) [16:24:29] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201911), 10wikimedia.biterg.io, 10Upstream: Gerrit replica does not support the 'query' command - https://phabricator.wikimedia.org/T235251 (10Jdforrester-WMF) [16:24:31] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201911), 10observability, 10serviceops-radar: Gerrit: Alert when deadlock occur ahead of Gerrit being unresponsive - https://phabricator.wikimedia.org/T230138 (10Jdforrester-WMF) [16:24:33] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201911), 10User-MModell: Make sure elasticsearch 6 is supported in phabricator - https://phabricator.wikimedia.org/T181393 (10Jdforrester-WMF) [16:24:36] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201911), 10MobileFrontend, 10Readers-Web-Backlog (Tracking): Javascript test failures on REL1_31 / REL1_32 / REL1_33 - https://phabricator.wikimedia.org/T230454 (10Jdforrester-WMF) [16:24:39] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201911), 10Ruby, 10User-zeljkofilipin: Mark mediawiki_api and mediawiki_selenium Ruby gems as deprecated - https://phabricator.wikimedia.org/T228160 (10Jdforrester-WMF) [16:24:41] 10Release-Engineering-Team-TODO (201911), 10User-greg, 10Wikimedia-extension-review-queue: Investigate and make improvements to the extension review process - https://phabricator.wikimedia.org/T195244 (10Jdforrester-WMF) [16:24:43] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201911), 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Update files required by Selenium in core, extensions and skins - https://phabricator.wikimedia.org/T210726 (10Jdforrester-WMF) [16:24:48] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201911), 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Selenium tests should be easier to run - https://phabricator.wikimedia.org/T182691 (10Jdforrester-WMF) [16:24:51] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (201911): gate-and-submit-wmf might have redundant quibble jobs (Oct 2019) - https://phabricator.wikimedia.org/T234381 (10Jdforrester-WMF) [16:24:53] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201911), 10MediaWiki-Core-Testing, 10Documentation, and 2 others: Blog posts about new Selenium framework features - https://phabricator.wikimedia.org/T191982 (10Jdforrester-WMF) [16:24:56] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201911), 10MediaWiki-Core-Testing, 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10Jdforrester... [16:24:59] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201911), 10Developer-Advocacy, 10wikimedia.biterg.io: biterg.io Gerrit crawling probably stresses the server too much - https://phabricator.wikimedia.org/T234328 (10Jdforrester-WMF) [16:28:10] 10Project-Admins, 10Research: create tags for Research - https://phabricator.wikimedia.org/T228350 (10leila) Let's do it, please. Thanks! [16:47:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201911): Evaluate Airflow's suitability for CI - https://phabricator.wikimedia.org/T237450 (10dduvall) >>! In T237450#5640522, @Ottomata wrote: > Ah interesting, good to know. > > Can Argo Events also produce external events, perhaps vi... [16:48:18] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201911): Upgrade Gerrit to 2.15.17 - https://phabricator.wikimedia.org/T229110 (10Paladox) I think we're planning to go with 2.16 instead of doing this then 2.16. Since a tentative date has been decided. (Also this... [16:50:32] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201911), 10MediaWiki-Core-Testing, 10Performance-Team, and 2 others: Release new version of wdio-mediawiki - https://phabricator.wikimedia.org/T236927 (10Krinkle) 05Open→03Resolved Done. [16:50:35] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201911), 10Patch-For-Review, 10User-zeljkofilipin: Upgrade webdriverio to version 5 for all repositories - https://phabricator.wikimedia.org/T234314 (10Krinkle) [17:05:39] new report from script running against beta commons: ""Waiting for cawiki: 130084.95 seconds lagged" [17:05:56] reporting host: deployment-mediawiki-09 [17:06:09] no such report 5 minutes ago (and there are 6 script-based edits to prove it) [17:06:50] but that lag is longer than 36 hours, wonder what's going on [17:08:11] Could be round-robin issues with one server very behind? [17:10:09] https://wikidata.beta.wmflabs.org/wiki/Special:DispatchStats has “90 days, 8 hours and 4 minutes” for all wikis o_O [17:10:21] not sure if that’s relevant though [17:10:48] Ouch. [17:11:07] “waiting for cawiki” just sounds more like dispatch lag than replication lag to me [17:11:56] Aren't cron/maintenance jobs for beta in a strange state? [17:12:06] Oh, wait, didn't we deleted one of the Beta wikis recently-ish? Maybe that's it? [17:12:51] https://github.com/wikimedia/operations-mediawiki-config/commits/master/dblists/all-labs.dblist [17:12:55] Zero back in July [17:13:05] Which is more than 90 days? [17:13:22] Ah, no, it was cswiki getting added, never mind. [17:16:00] reedy@deployment-deploy01:~$ /usr/local/bin/mwscript extensions/Wikibase/repo/maintenance/dispatchChanges.php --wiki wikidatawiki [17:16:00] 17:15:48 Starting loop for unlimited passes or 540 seconds [17:16:00] 17:15:51 ERROR: LogicException: Domain 'afwiki' is not recognized. in /srv/mediawiki-staging/php-master/includes/jobqueue/JobQueueGroup.php:196 [17:16:24] Numerous of those... [17:16:27] * hauskater waiting to test that WebAuthn thing [17:17:10] hauskater: Do you have a device you can use? :P [17:17:31] Reedy: let me know and I can search in my drawer [17:17:35] or call Q [17:17:37] :P [17:17:48] You basically need the latest and greatest yubikey [17:17:56] Or some mobile phones can do it AIUI [17:18:00] {"errors":[{"code":"maxlag","text":"Waiting for cawiki: 130084.95 seconds lagged.","data":{"host":"cawiki","lag":130084.95,"type":"wikibase-dispatching","dispatchLag":7805097} etc [17:18:22] so now I should parse the maxlag return and make sure it's not wikibase-dispatching and let the script ignore those? ,eh [17:18:24] *meh [17:19:05] * apergos edits script some more [17:19:16] I feel like teaching our scripts to ignore that is not great [17:20:01] oh heh [17:20:04] Lucas_WMDE: The stale-est and median don't exist on beta [17:20:08] I can't "just ignore it" [17:20:09] So I guess that's kinda expected [17:20:22] Why is it trying to dispatch to those? [17:20:23] well Reedy believe it or not, you'll need first to wait for jenkins to find a suitable executor: https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/ [17:20:24] I can either not request maxlag info from the api at request time [17:20:30] or I can get the error (and no other response) [17:20:36] last successful Scap for beta ca. 12:00 UTC [17:20:37] * apergos taps fingers in annoyance [17:20:38] hauskater: easy to fix [17:20:49] Reedy: do not manually deploy [17:20:50] :P [17:20:55] I wasn't going to [17:20:59] You just kill that job in jenkins [17:21:01] And the next works [17:21:05] lol [17:21:27] sort of "you're fired, next please" [17:21:37] heh [17:21:51] code update is now running after 5 hours of not [17:22:08] yup [17:23:05] That might help. [17:24:02] It is building now [17:24:39] * apergos looks in bemusedly [17:24:42] *on [17:26:00] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata-Campsite: beta wikidata trying to dispatch to wikis that don't exist - https://phabricator.wikimedia.org/T237551 (10Reedy) [17:26:04] wow 50 Bucks for a Yubico key [17:26:11] geez really? ouch [17:26:29] the cheapest apergos [17:26:45] yowza [17:26:48] as far as I could see [17:27:03] There's a $45 one [17:27:06] And one goes upto £70 [17:27:09] *$70 [17:27:14] Yup [17:27:21] Well actually, there's a $20 one [17:27:29] that's more like it [17:27:43] Some trascend encrypted usbs costs thousands of dollars so I guess this is cheaper ;) [17:27:49] *transcend [17:27:57] Reedy clicked the wrong currency :P [17:28:06] But you need the ~$45-50 ones for webauthn [17:28:14] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata-Campsite: beta wikidata trying to dispatch to wikis that don't exist - https://phabricator.wikimedia.org/T237551 (10Lucas_Werkmeister_WMDE) [17:28:17] £70 is expensive :P [17:28:26] Wikipedia, the free encyclopedia where only rich can log in [17:28:37] zing! [17:28:52] https://solokeys.com/ [17:28:54] free as in freedom, not free as in beer [17:29:19] I'm waiting for their SOMU which should support webauthn [17:29:31] $35 https://www.crowdsupply.com/solokeys/somu [17:30:04] huh [17:30:21] bookmarked [17:30:37] I do have some of their first gen ones already [17:30:50] But https://github.com/solokeys/solo is kinda cool to have :) [17:32:03] going to check that out later (also now bookmarked) [17:33:26] I think I'm going to save the money for https://www.astonmartin.com/en-us/models/db11 instead :P [17:35:16] lol @ car [17:36:05] paladox: with customised plates: Wiki-Up :P [17:36:11] lololol [17:36:25] or Brexit [17:36:59] well that'll be funny :P [17:37:19] hauskater use the lib dems reference :P [17:37:30] brexit bol*** ? [17:37:40] yup [17:37:46] well bol** to brexit [17:38:11] Reedy: Hmm. I registered my YubiKey but it won't let me login. [17:38:21] James_F: Do you have the right type? :P [17:38:36] I don't know? Firefox seemed to pass it through to the form… [17:38:44] ccccccknjiurhctrdnfkcbtndcufckfngvegdgfubhhd [17:38:49] Oops. [17:38:59] lol [17:39:06] xD [17:39:24] "Verification failed." isn't very helpful. [17:39:29] hee hee [17:39:29] I'm getting the same [17:39:32] ccccccljuuntvrvuekceiignhgljuietjbfvtuctgkgb [17:39:38] We have our first contestant on the "Get locked out of your account testing the new WebAuthn" [17:39:38] lolol [17:39:43] * apergos collects up all the tokens [17:39:47] * James_F grins. [17:39:53] * hauskater looks logstash [17:40:04] I've got whatever the USB-C YubiCo key is. [17:40:15] You need a yubikey 5 [17:40:21] But as you've enrolled, you should have the right device [17:40:53] * Reedy files a bug [17:41:19] It's helpfully unlabelled except for the 2D barcode. [17:42:06] twentyafterfour: andrewbogott and I have a "pretty please can this ride the train today?" request for https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/OpenStackManager/+/549124/ [17:42:35] we can SWAT it later too, but riding the train is environmentally conscious :) [17:43:02] !log deployment-prep dropped and rebuilt MachineVision extension tables [17:43:02] Just merge it and it'll magically blow up for twentyafterfour and not you. ;-) [17:43:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:43:08] bd808: no problem at all. +2'd already [17:43:44] Reedy: If you're shelled into Beta Cluster, can you clear my WebAuthn config? ;-) [17:43:51] yeah I can, lol [17:43:57] Ta. [17:44:04] thanks twentyafterfour :) Maybe I can buy you a beverage in Atlanta next week? [17:44:48] Saw nothing in Logstash other than CentralAuthVerbose entries [17:44:48] bd808: I won't be at the tech conf unfortunately but I'll be there on the 15th at some point [17:44:52] Done [17:45:16] a whole sub-economy of bartering, where 'bevcoin' is used to represent value [17:45:37] twentyafterfour: *nod* I figured you would be rolling in for the offsite. I don't fly out until the 16th, so there is beverage hope! [17:46:01] liw: it is a well established tradition to offer via irc and never pay up in person ;) [17:46:33] bd808: cool I'll def. try to catch up with you before you leave, at least to give you a flying pig t-shirt [17:46:47] omg! yes! [17:46:48] bd808, my bank will be mortified to hear my collateral is no good [17:48:47] lots of logspam re. Failed executing job: cirrusSearchElasticaWrite [18:05:39] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201911), 10Documentation, 10User-MModell: Make PHD run on the backup phabricator server (phab2001, currently) - https://phabricator.wikimedia.org/T232883 (10mmodell) 05Open→03Resolved [18:05:42] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10DBA, and 2 others: Prepare a disaster recovery plan for failing over Phabricator - https://phabricator.wikimedia.org/T190572 (10mmodell) [18:06:25] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201911), 10Documentation, 10User-MModell: Make PHD run on the backup phabricator server (phab2001, currently) - https://phabricator.wikimedia.org/T232883 (10mmodell) 05Resolved→03Open [18:06:27] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10DBA, and 2 others: Prepare a disaster recovery plan for failing over Phabricator - https://phabricator.wikimedia.org/T190572 (10mmodell) [18:07:54] welp. I have a typo in my 3000 image files on beta. guess I need to script up something to fix it :-/ [18:08:02] just a template but it irks me [18:19:57] Reedy: maybe a maintenance script to disableWebAuthForUser.php can be created mirroring the features disableOATHForUser.php has, just in case? [18:20:27] hauskater: Same one would work [18:20:35] nice [18:20:36] or should [18:21:52] Let me verify [18:25:05] Maybe rename it [18:26:17] reedy@deployment-deploy01:/srv/mediawiki-staging$ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=metawiki Reedy [18:26:17] OATHAuth disabled for Reedy. [18:54:29] tomorrow I'll be running a script to fix the template typo in the 3k bulk beta commons uploads; I'll announce here before it goes. [19:30:24] 10Release-Engineering-Team, 10Wikimedia Design Style Guide, 10Patch-For-Review, 10User-Ladsgroup: Use `git lfs` for large binary files of Design Style Guide - https://phabricator.wikimedia.org/T235013 (10Dzahn) Just wanted to add something i recently noticed while upgrading the Gerrit replica. 44% of the... [19:31:45] 10Release-Engineering-Team, 10Wikimedia Design Style Guide, 10Patch-For-Review, 10User-Ladsgroup: Use `git lfs` for large binary files of Design Style Guide - https://phabricator.wikimedia.org/T235013 (10mmodell) >>! In T235013#5641942, @Dzahn wrote: > But when i checked now again i see it has been fixed... [19:35:57] 10Release-Engineering-Team, 10Wikimedia Design Style Guide, 10Patch-For-Review, 10User-Ladsgroup: Use `git lfs` for large binary files of Design Style Guide - https://phabricator.wikimedia.org/T235013 (10mmodell) Much appreciation to @akosiaris and @ArielGlenn for further clarity on this! You've cleared u... [19:46:54] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Fresnel, 10Performance-Team, 10Jenkins: Jenkins console section matching breaks output HTML rendering - https://phabricator.wikimedia.org/T236222 (10Krinkle) [19:47:02] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Fresnel, 10Performance-Team, 10Jenkins: Jenkins console section matching breaks output HTML rendering - https://phabricator.wikimedia.org/T236222 (10Krinkle) [19:47:08] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Fresnel, 10Performance-Team, 10Jenkins: Jenkins console section matching breaks output HTML rendering - https://phabricator.wikimedia.org/T236222 (10Krinkle) Blocked on the above being released and deployed in CI. [20:01:47] * paladox just learned whos his labour mp for my area [20:03:31] (03CR) 10Brennen Bearnes: [V: 03+2 C: 03+2] stretch-php72-fpm-apache: allow cache to expire [releng/dev-images] - 10https://gerrit.wikimedia.org/r/548941 (owner: 10Brennen Bearnes) [20:15:46] 10Release-Engineering-Team, 10Wikimedia Design Style Guide, 10Patch-For-Review, 10User-Ladsgroup: Use `git lfs` for large binary files of Design Style Guide - https://phabricator.wikimedia.org/T235013 (10Dzahn) >>! In T235013#5641952, @mmodell wrote: > Interesting. Actually 1001 and 2001 are different. I... [20:17:00] 10Continuous-Integration-Infrastructure, 10Core Platform Team, 10MediaWiki-Installer, 10Regression: MediaWiki web installer does not show extension when their dependency is missing - https://phabricator.wikimedia.org/T220514 (10Krinkle) [20:31:20] !log Updating dev-images docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/c/releng/dev-images/+/548941 [20:31:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:32:44] ^ cc: longma - next published mw dev image after that finishese should work correctly with sshfs mounts. [20:56:01] (03CR) 10Jforrester: [C: 03+1] Add a sniff for methods deprecated in PHPUnit 8 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/547958 (https://phabricator.wikimedia.org/T192167) (owner: 10Daimona Eaytoy) [21:00:28] 10Release-Engineering-Team, 10Wikimedia Design Style Guide, 10Patch-For-Review, 10User-Ladsgroup: Use `git lfs` for large binary files of Design Style Guide - https://phabricator.wikimedia.org/T235013 (10Paladox) So i'm not entirly sure how gerrit2001 managed to grow like this. I wonder if we should delete... [21:34:29] 10Phabricator, 10Project-Admins: Convert TimedMediaHandler pseudo-subprojects into proper subprojects - https://phabricator.wikimedia.org/T237188 (10Jdforrester-WMF) Sure. We should probably also make a sub-project for -VideoJS to split out from -Player and rename the latter to KalturaPlayer, but that can be d... [21:38:57] 10Release-Engineering-Team, 10Wikimedia Design Style Guide, 10Patch-For-Review, 10User-Ladsgroup: Use `git lfs` for large binary files of Design Style Guide - https://phabricator.wikimedia.org/T235013 (10thcipriani) Most of that 30GB is in packfiles. Those packfiles all seem to contain the same objects: `... [21:49:23] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 10Elasticsearch, 10Discovery-Search (Current work): Warning: Deprecated field [inline] used, expected [source] instead - https://phabricator.wikimedia.org/T237000 (10EBernhardson) Deployed same configuration settings as we set in prod to the beta cluster. Will... [22:19:33] (03PS1) 10Jeena Huneidi: Fix permissions issue mounting volumes [releng/local-charts] - 10https://gerrit.wikimedia.org/r/549217 [22:46:55] !log Fix closing pattern of "setup quibble" Console Section in Jenkins to not break on composertest jobs. [22:46:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:59:00] 10Beta-Cluster-Infrastructure, 10Growth-Team, 10StructuredDiscussions: [betalabs-regression] Cannot add/update info on Structured discussion boards - https://phabricator.wikimedia.org/T234242 (10ssastry) @Catrope do you know what is going on here? [23:24:56] (03CR) 10MaxSem: Add a sniff for methods deprecated in PHPUnit 8 (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/547958 (https://phabricator.wikimedia.org/T192167) (owner: 10Daimona Eaytoy) [23:52:34] PROBLEM - Puppet staleness on deployment-restbase02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [43200.0]