[23:19:54] bd808: any thoughts about how to deal with Setup a secondary index on ElasticSearch, either on the production Phab host or elsewhere, depending on complexity [23:20:12] argh I hate linux clipboards [23:20:25] heh. what's the real question? [23:20:42] how to deal with T151428 that is [23:20:43] T151428: The UdpSocket to 127.0.0.1:10514 has been closed and can not be written to anymore - https://phabricator.wikimedia.org/T151428 [23:21:01] leave it? [23:21:09] makes it impossible to test logstash changes on vagrant [23:21:17] what's lost? a couple of wfdebug messages? [23:21:27] oh because it's fatal [23:21:31] hmmm [23:21:34] it throws an exception on every request [23:22:05] why do we not see that in prod? [23:22:05] just subclass SyslogUdpHandler to not close sockets? [23:22:26] probably because it happens after the socket is closed so there is nowhere to log [23:22:27] I... think we already subclass it [23:22:52] or maybe the destruction order on shutown is different [23:23:21] or we are not logging on debug level so there is no logging issued from __destruct [23:23:43] that's more likely although it would blow up beta [23:24:07] we use MediaWiki\Logger\Monolog\SyslogHandler in prod [23:24:24] which is a subclass of Monolog\Handler\SyslogUdpHandle [23:24:28] *r [23:24:41] is there any logging that we are not doing via the UDP handler? [23:24:58] I mean if this does happen on prod would it show uop? [23:25:02] *up* [23:25:28] hmmm... not sure [23:25:37] * bd808 looks at config [23:25:50] well, yes [23:26:12] if it was an unhandled fatal then HHVM would catch it and log via its syslog channels [23:27:16] tgr: you have a rare talent for finding weird edge cases ;) [23:27:54] I don't think it's an edge case, it will affect anyone using the elk role on vagrant [23:28:03] well, it might [23:28:13] I'm pretty sure that's an edge case :) [23:28:13] object destruction order not being deterministic [23:30:15] we could just do some subclassing to swallow those errors on vagrant [23:30:20] so the socket is closed when the handler is closed. I guess we need to track down where that happens. It probably a registered shutdown handler somewhere [23:31:11] the socket is closed in UdpSocket::__destruct, and LBFactory::__destruct tries to write it [23:31:33] ugh [23:31:44] arguably it's wrong to trigger anything that includes logging from __destruct [23:32:28] logging in destructors is always bad mojo [23:32:39] it's also arguable that a logger should not throw exceptions when something tries to use it after it destructed [23:32:49] so not sure which way to take it [23:32:58] argue that upstream with MOnolog :) [23:34:13] so you'd prefer fixing LBFactory? [23:35:03] We can fix it in SyslogHandler but that's just one handler out of many possible [23:35:29] SyslogHandler::write() would be the place to swallow the fatal [23:35:39] I can make a patch for that [23:36:09] but it will come up for someone else using a different handler at some point [23:38:13] I can write the patch, just wasn't sure which way to go [23:38:48] since the same is true for logging from __destruct, it will probably just happen with something else next [23:39:30] so either way it might grow into a bigger change and I want to know I am doing it in the right direction :) [23:39:43] I'm writing one for it in SyslogHandler [23:41:36] tgr: do you think we should bug the world by telling the php error logger that we caught an exception or just ignore the problem? [23:42:33] not sure [23:42:48] if there is a very silent way of telling the word then yes [23:43:34] I think the options are use error_log() or just look the other way [23:43:37] since it's probably not an error but if there is an error with closing sockets and someone is looking for it then hiding it would be bad [23:46:47] error_log seems to be meant for scary errors [23:47:06] the couldn't even get logging set up kind [23:47:34] trigger_error() then [23:47:53] that will break vagrant since people tend to set it to E_ALL [23:48:17] you are telling me that its already broken... [23:48:31] well, keep it broken [23:48:41] and yet you are the only person to report it... [23:48:46] I think the least bad option is to just document it somewhere as a potential pitfall and then look the other way [23:48:56] which makes me think this is an edge case with certain things setup [23:49:43] works for me. logging is best effort only anyway IMO [23:51:56] do many people use the elk role? [23:52:18] the change to LBFactory::close is fairly recent IIRC (a month or so) [23:55:40] tgr: oh! we wrap things in prod to catch shit like this -- https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/logging.php#L315-L331 [23:56:13] https://github.com/Seldaek/monolog/blob/master/src/Monolog/Handler/WhatFailureGroupHandler.php [23:56:38] that's probably the right way to fix this in mw-vagrant too [23:58:23] that looks nice [23:58:31] OK, I'll do that