[01:32:15] PROBLEM - Free space - all mounts on deployment-deploy02 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy02.diskspace._srv.byte_percentfree (<11.11%) [01:42:15] RECOVERY - Free space - all mounts on deployment-deploy02 is OK: OK: All targets OK [01:42:48] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace._srv.byte_percentfree (<10.00%) [03:27:48] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [08:02:47] 10Beta-Cluster-Infrastructure, 10AbuseFilter, 10User-DannyS712: Unable to create abuse filter on the beta cluster - https://phabricator.wikimedia.org/T245386 (10DannyS712) [08:03:04] 10Beta-Cluster-Infrastructure, 10AbuseFilter, 10User-DannyS712: Unable to create abuse filter on the beta cluster - https://phabricator.wikimedia.org/T245386 (10DannyS712) [08:16:22] Project beta-scap-eqiad build #288544: 04FAILURE in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/288544/ [08:19:00] PROBLEM - Free space - all mounts on deployment-snapshot01 is CRITICAL: CRITICAL: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)deployment-prep.deployment-snapshot01.diskspace.root.byte_percentfree (<60.00%) [08:19:33] bleugh, that didn't last long [08:23:18] what's causing the out of space stuff? [08:23:30] I've done nothing over there recently so it ain't me [08:23:49] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace._srv.byte_percentfree (<10.00%) [08:23:51] Project beta-scap-eqiad build #288545: 04STILL FAILING in 5 min 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/288545/ [08:24:19] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Community-Tech, 10Expiring-Watchlist-Items, 10TCB-Team: Create `watchlist_expiry` table in production after wmf.19 is available - https://phabricator.wikimedia.org/T244631 (10Marostegui) [08:29:44] Project beta-scap-eqiad build #288546: 04STILL FAILING in 4 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/288546/ [08:30:51] 10Continuous-Integration-Infrastructure (Slipway), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Wikidata, and 2 others: Migrate wikidata-query-gui-build to Docker containers - https://phabricator.wikimedia.org/T210286 (10Addshore) Right now the goal would still be... [08:33:59] 10Project-Admins, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Containers, 10MediaWiki-Docker, 10Release Pipeline: Re-purposing MediaWiki-Docker and friends - https://phabricator.wikimedia.org/T245167 (10Addshore) >>! In T245167#5881817, @dbarratt wrote: > Perhaps move the exist... [08:38:52] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [08:39:03] Project beta-scap-eqiad build #288547: 04STILL FAILING in 4 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/288547/ [08:43:57] (03PS2) 10Kosta Harlan: Build XDebug from source to work around pear/pecl errors [releng/dev-images] - 10https://gerrit.wikimedia.org/r/572488 (https://phabricator.wikimedia.org/T245304) [08:48:54] Project beta-scap-eqiad build #288548: 04STILL FAILING in 4 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/288548/ [08:59:03] Project beta-scap-eqiad build #288549: 04STILL FAILING in 4 min 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/288549/ [09:09:06] Project beta-scap-eqiad build #288550: 04STILL FAILING in 4 min 36 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/288550/ [09:22:30] 10Beta-Cluster-Infrastructure: deployment-snapshot01.deployment-prep.eqiad.wmflabs keeps running out of space - https://phabricator.wikimedia.org/T245402 (10Reedy) [09:23:15] PROBLEM - Free space - all mounts on deployment-deploy02 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy02.diskspace._srv.byte_percentfree (<11.11%) [09:26:02] Project beta-scap-eqiad build #288551: 04STILL FAILING in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/288551/ [09:33:13] RECOVERY - Free space - all mounts on deployment-deploy02 is OK: OK: All targets OK [09:34:01] RECOVERY - Free space - all mounts on deployment-snapshot01 is OK: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found) [09:38:07] Yippee, build fixed! [09:38:07] Project beta-scap-eqiad build #288552: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/288552/ [09:57:41] (03CR) 10Kosta Harlan: "Not necessarily for this patch, but is there anything more we can do to get the image size down from the current 448M?" [releng/dev-images] - 10https://gerrit.wikimedia.org/r/572488 (https://phabricator.wikimedia.org/T245304) (owner: 10Kosta Harlan) [12:39:52] (03PS1) 10MarcoAurelio: [Teahouse] Archive extension [integration/config] - 10https://gerrit.wikimedia.org/r/572653 (https://phabricator.wikimedia.org/T176349) [12:39:53] 10MediaWiki-Codesniffer: Small improve on autofix of MediaWiki.Usage.IsNullSniff possible to avoid unneeded parenthesis - https://phabricator.wikimedia.org/T242780 (10thiemowmde) [12:39:54] (03CR) 10Daimona Eaytoy: Detect missing & for ParamNameNoMatch error code and autofix (032 comments) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/571023 (owner: 10Umherirrender) [12:39:54] 10Beta-Cluster-Infrastructure, 10AbuseFilter, 10User-DannyS712: Unable to create abuse filter on the beta cluster - https://phabricator.wikimedia.org/T245386 (10Daimona) [12:39:54] 10Beta-Cluster-Infrastructure, 10MediaWiki-Core-Testing, 10MediaWiki-User-login-and-signup, 10MediaWiki-extensions-CentralAuth, and 3 others: Login to at least en.wikipedia.beta.wmflabs.org and commons.wikimedia.beta.wmflabs.org sometimes fails with `There seems to... - https://phabricator.wikimedia.org/T243123 [12:39:54] 10Beta-Cluster-Infrastructure, 10AbuseFilter, 10User-DannyS712: Unable to create abuse filter on the beta cluster - https://phabricator.wikimedia.org/T245386 (10Daimona) That's because token validation is failing. You get the same issue when you try to log in, hence closing as duplicate (same root cause). [12:39:56] 10Continuous-Integration-Infrastructure, 10Operations, 10Packaging, 10puppet-compiler, 10User-jbond: PCC always has an ERROR when compiling for servers with profile::redis::slave - https://phabricator.wikimedia.org/T228266 (10jbond) [12:39:57] !log extension-Teahouse@github: Mirror deleted # T176349 [12:39:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:39:57] T176349: [Task] Decide if the Teahouse extension can be archived - https://phabricator.wikimedia.org/T176349 [12:39:57] !log extension-Teahouse@phab: Mirror deactivated # T176349 [12:39:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:30:17] (03PS1) 10Lars Wirzenius: Fix: 'version' subcommand function should return 0, not None [tools/scap] - 10https://gerrit.wikimedia.org/r/572666 [15:56:35] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Tag a new release of scap - https://phabricator.wikimedia.org/T243358 (10LarsWirzenius) While preparing to do this, I wrote a little, simplistic integration test, which runs "scap version". Don't laugh: it already found a bug. It seems "scap version" has... [16:05:24] 10Continuous-Integration-Config, 10VideoCutTool: Set up continuous integration for VideoCutTool - https://phabricator.wikimedia.org/T244079 (10Gopavasanth) 05Open→03Resolved [16:13:23] 10Phabricator, 10Release-Engineering-Team: change "default" column on Parsoid workboard - https://phabricator.wikimedia.org/T245308 (10MBinder_WMF) Yea, FWIW, I've found Move to Column does not work consistently, and generally advise against using it when at all possible. Sorry I'm late to that suggestion. :-/ [17:34:46] 10Beta-Cluster-Infrastructure, 10MediaWiki-Core-Testing, 10MediaWiki-User-login-and-signup, 10MediaWiki-extensions-CentralAuth, and 3 others: Login to at least en.wikipedia.beta.wmflabs.org and commons.wikimedia.beta.wmflabs.org sometimes fails with `There seems to... - https://phabricator.wikimedia.org/T243123 [17:47:43] 10Beta-Cluster-Infrastructure, 10MediaWiki-Core-Testing, 10MediaWiki-User-login-and-signup, 10MediaWiki-extensions-CentralAuth, and 3 others: Login to at least en.wikipedia.beta.wmflabs.org and commons.wikimedia.beta.wmflabs.org sometimes fails with `There seems to... - https://phabricator.wikimedia.org/T243123 [17:47:52] PROBLEM - Parsoid on deployment-parsoid09 is CRITICAL: connect to address 172.16.5.63 and port 8000: Connection refused [17:47:53] PROBLEM - Parsoid on deployment-mediawiki-parsoid10 is CRITICAL: connect to address 172.16.0.141 and port 8000: Connection refused [17:48:43] (03PS2) 10Thiemo Kreuz (WMDE): Optimize MultipleEmptyLines sniff for peformance (again) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/571003 [18:03:58] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+2] "I finished my follow up patch Ic4113d2 and would love to get another review for that. I was able to tremendously reduce the complexity of " [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/570955 (owner: 10Umherirrender) [18:04:31] (03Merged) 10jenkins-bot: Detect multiple empty lines after single line comment or php open [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/570955 (owner: 10Umherirrender) [18:43:19] (03PS3) 10Thiemo Kreuz (WMDE): Optimize MultipleEmptyLines sniff for peformance (again) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/571003 [19:04:20] 10Project-Admins, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Containers, 10MediaWiki-Docker, 10Release Pipeline: Re-purposing MediaWiki-Docker and friends - https://phabricator.wikimedia.org/T245167 (10dbarratt) >>! In T245167#5881935, @Jdforrester-WMF wrote: > That'd be fine... [20:02:43] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Scap, 10Performance-Team (Radar), 10User-Ladsgroup: Add Scap support for static-array format of LCStore - https://phabricator.wikimedia.org/T105683 (10thcipriani) This change is currently live i... [20:20:00] 10Release-Engineering-Team-TODO, 10Scap, 10MediaWiki-Internationalization, 10MW-1.35-notes (1.35.0-wmf.11; 2019-12-17), and 2 others: Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740 (10Krinkle) [20:24:22] 10Release-Engineering-Team-TODO, 10Scap, 10MediaWiki-Internationalization, 10Patch-For-Review, 10Performance-Team (Radar): Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740 (10Krinkle) [20:24:34] 10Release-Engineering-Team-TODO, 10Scap, 10MediaWiki-Internationalization, 10Patch-For-Review, 10Performance-Team (Radar): Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740 (10Krinkle) [20:24:58] 10Release-Engineering-Team-TODO, 10Scap, 10MediaWiki-Internationalization, 10Patch-For-Review, 10Performance-Team (Radar): Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740 (10Krinkle) [20:25:04] (03CR) 10Krinkle: [C: 03+2] Optimize MultipleEmptyLines sniff for peformance (again) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/571003 (owner: 10Thiemo Kreuz (WMDE)) [20:25:37] (03Merged) 10jenkins-bot: Optimize MultipleEmptyLines sniff for peformance (again) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/571003 (owner: 10Thiemo Kreuz (WMDE)) [20:25:41] 10Release-Engineering-Team-TODO, 10Scap, 10MediaWiki-Internationalization, 10Patch-For-Review, 10Performance-Team (Radar): Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740 (10Krinkle) [20:29:24] PROBLEM - Host deployment-puppetmaster03 is DOWN: CRITICAL - Host Unreachable (172.16.4.91) [20:29:34] PROBLEM - Host deployment-puppetdb02 is DOWN: CRITICAL - Host Unreachable (172.16.4.104) [22:37:45] 10Phabricator, 10Regression: Use of monospaced font URL descriptions no longer display correctly - https://phabricator.wikimedia.org/T245472 (10MarcoAurelio) [22:38:10] 10Phabricator, 10Regression: Use of monospaced font URL descriptions no longer display correctly - https://phabricator.wikimedia.org/T245472 (10MarcoAurelio) [22:38:24] 10Phabricator, 10Regression: [Regression] Use of monospaced font URL descriptions no longer display correctly - https://phabricator.wikimedia.org/T245472 (10MarcoAurelio)