[05:43:47] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<50.00%) [06:58:46] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [08:23:11] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] Add my private email address to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/500178 (owner: 10Lucas Werkmeister (WMDE)) [16:20:47] RECOVERY - Disk space on contint1001 is OK: DISK OK [16:22:59] (03CR) 10D3r1ck01: [C: 03+1] "Just a fashion thing, but can be merged as-is." (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/500178 (owner: 10Lucas Werkmeister (WMDE)) [17:22:41] 10Phabricator-Bot-Requests, 10WMSE-Bug-Reporting-and-Translation-2019: Show active projects first on subproject page, then archived projects - https://phabricator.wikimedia.org/T218041 (10Aklapper) @mmodell: Thanks! Is that one-liner change something to propose to upstream? [19:09:41] 10Beta-Cluster-Infrastructure, 10Analytics, 10EventBus: PHP fatal error submitting block on Beta - https://phabricator.wikimedia.org/T219737 (10MarcoAurelio) I see something at https://kibana4.wmflabs.org/goto/3a38e83ee655fbc97beca6c53679d431 Catchable fatal error: Argument 3 passed to EventFactory::creat... [19:14:39] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.33.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T206678 (10Krenair) [19:22:49] Krenair: not sure why 498393 got manually merged - jenkins seemed not happy with it? [19:23:58] hauskatze, looks like it was submitted before jenkins gave a negative vote [19:24:04] yes [19:24:24] shall we revert that? [19:24:32] or wait until investigation? [19:24:38] I prefer to wait tbh [19:24:48] maybe it can be fixed w/o reverting the whole thing [19:24:52] I think it's a pretty simple fix tbh hauskatze [19:24:57] just missing a ' = null' in the new function def [19:25:04] I'll write a patch, one sec [19:25:15] {thumbs-up} [19:28:04] btw I think new blocks are broken, amending existing blocks may be ok [19:28:14] still a blocker even for group0 [19:28:22] bad enough having this on master [19:29:39] this new blocks are 'funny', so to speak [19:29:58] ? [19:30:13] they're enabled on Meta for some weeks I'm yet to find a case where I need to issue a partial block instead of a sitewide one [19:30:35] or you mean 'new blocks' as of 'blocks issued as of today' in Beta? [19:30:36] oh you're talking about partial blocks [19:30:40] that wasn't what I was referring to [19:30:43] I mean [19:30:47] right, gj hauskatze [19:30:53] self-trout [19:31:00] If you make a new block [19:31:06] i.e. block someone who is not currently blocked [19:31:09] vs. [19:31:13] If you amend an existing block [19:31:18] i.e. block someone who IS currently blocked [19:31:19] Ah, I see [19:31:41] In the amending case, this error will probably not happen [19:35:54] This is unusual [19:35:56] Krenair: tested myself [19:36:12] Normally with software bugs, the normal case is fine, it's edge cases that break [19:36:15] a block on an non-blocked IP returns a page like those when the site is down [19:36:23] 10Phabricator, 10Release-Engineering-Team, 10Operations, 10Wikimedia-Incident: Analyze and amend (if necessary) workflow of user reporting and detecting large regressions/outages - https://phabricator.wikimedia.org/T219589 (10Yann) Hi, There should be a clear way for users and bug reporters to indicate how... [19:36:27] "Our servers are currently under maintenance or experiencing a technical problem. Please try again in a few minutes." [19:36:45] and the trace: PHP fatal error: [19:36:45] Argument 3 passed to EventFactory::createUserBlockChangeEvent() must be an instance of Block, null given [19:36:50] yeah [19:37:02] checking logstash [19:38:57] yep, it's the same [19:39:15] although checking fatalmonitor I also see a very frequent error that might be worth reporting too [19:39:34] "[{exception_id}] {exception_url} ErrorException from line 141 of /srv/mediawiki/php-master/includes/site/SiteList.php: PHP Warning: Array key should be either a string or an integer" [19:43:43] lots of things could be calling that [19:49:38] 10Beta-Cluster-Infrastructure, 10Wikimedia-production-error: PHP Warning: Array key should be either a string or an integer - https://phabricator.wikimedia.org/T219738 (10MarcoAurelio) [19:49:43] filed, just in case [19:49:48] eventbus is also mentioned [19:52:16] 10Phabricator: Form 46 imagery is broken - https://phabricator.wikimedia.org/T219739 (10MarcoAurelio) [19:52:41] 10Phabricator: Form 46 imagery is broken - https://phabricator.wikimedia.org/T219739 (10MarcoAurelio) Last modified by @krinkle [20:03:27] 10Beta-Cluster-Infrastructure, 10Wikimedia-production-error: PHP Warning: Array key should be either a string or an integer - https://phabricator.wikimedia.org/T219738 (10MarcoAurelio) While the example I took was from enwiki beta, by far the wiki with most error messages logged is wikidatawiki beta (`XKEb6qwQ... [20:48:28] (03PS1) 10MarcoAurelio: Whitelist user Nikitrain on CI [integration/config] - 10https://gerrit.wikimedia.org/r/500360 [20:49:54] 10Beta-Cluster-Infrastructure, 10Wikimedia-production-error: PHP Warning: Array key should be either a string or an integer - https://phabricator.wikimedia.org/T219738 (10Bawolff) I think there might be more than one thing going wrong here, but primarily: 4f72d439 added some code `$wikiRef = WikiMap::getWiki... [20:51:36] (03PS2) 10MarcoAurelio: Whitelist user Nikitrain on CI [integration/config] - 10https://gerrit.wikimedia.org/r/500360 [20:53:20] 10Beta-Cluster-Infrastructure, 10Wikimedia-production-error: PHP Warning: Array key should be either a string or an integer - https://phabricator.wikimedia.org/T219738 (10Bawolff) "EventFactory.php" line 144 also seems kind of wrong. Its passing the wiki name to self::getDomain, but the docs for self::getDomai... [20:53:52] 10Beta-Cluster-Infrastructure, 10Analytics, 10EventBus, 10Wikimedia-production-error: PHP Warning: Array key should be either a string or an integer - https://phabricator.wikimedia.org/T219738 (10Bawolff) [20:54:45] 10Beta-Cluster-Infrastructure, 10Analytics, 10EventBus, 10Wikimedia-production-error: PHP Warning: Array key should be either a string or an integer - https://phabricator.wikimedia.org/T219738 (10MarcoAurelio) 4f72d439 is #eventbus and is causing T219737 as well - ping @Pchelolo too. [20:58:07] (03CR) 10Zoranzoki21: [C: 03+1] Whitelist user Nikitrain on CI [integration/config] - 10https://gerrit.wikimedia.org/r/500360 (owner: 10MarcoAurelio)