[17:30:30] <legoktm>	 James_F: thanks :)
[17:36:46] <James_F>	 legoktm: You want those all back-ported, right?
[17:38:09] <legoktm>	 James_F: yes, but probably after they run through the train and don't cause problems. I have one more EditPage deprecation patch I need to write, and then I'll write up release notes for everything at once
[17:39:10] <James_F>	 Sure. Cool.
[19:01:46] <no_justification>	 anomie: Similar issue to yesterday: Notice: Undefined index: rc_logid in /srv/mediawiki/php-1.30.0-wmf.19/includes/changes/RecentChange.php on line 349
[19:03:29] <no_justification>	 James_F: Notice: Undefined variable: wgUser in /srv/mediawiki/php-1.30.0-wmf.19/extensions/VisualEditor/ApiVisualEditor.php on line 302
[19:04:55] <Reedy>	 undefined global? lol
[19:06:05] <Reedy>	 Never mind it doesn't need to use a global
[19:06:07] * Reedy will fix
[19:06:46] <Reedy>	 $user is already defined
[19:07:37] <no_justification>	 Also, tons of Warning: Attempted to serialize unserializable builtin class Closure$ChangesListSpecialPage::__construct#29;1583 in /srv/mediawiki/php-1.30.0-wmf.19/includes/libs/objectcache/MultiWriteBagOStuff.php on line 187
[19:07:42] <no_justification>	 Various #s though
[19:07:52] <Reedy>	 https://gerrit.wikimedia.org/r/379596
[19:07:58] <anomie>	 no_justification: Similar error, but it looks like the cause is going to be different. I'll probably need to dig up a backtrace for that one if I'm going to figure it out.
[19:09:12] <no_justification>	 Here's the full entry from logstash: https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2017.09.21/hhvm?id=AV6lzbMqTqnoMLVn4XiA&_g=h@bfc149c
[19:12:54] <no_justification>	 Relatedly to unserialize bug: Warning: Attempted to serialize unserializable builtin class Closure$ORES\Hooks::makeApplicableCallback;1272 in /srv/mediawiki/php-1.30.0-wmf.19/includes/libs/objectcache/MultiWriteBagOStuff.php on line 187
[19:13:27] <no_justification>	 Reedy: Ugh, while you're in ApiVisualEditor....
[19:13:54] <no_justification>	 Grr where'd it go
[19:13:58] <no_justification>	 Undefined $content on some line
[19:14:57] <no_justification>	 Notice: Undefined variable: content in /srv/mediawiki/php-1.30.0-wmf.19/extensions/VisualEditor/ApiVisualEditor.php on line 571
[19:18:19] <anomie>	 no_justification: For the serialize ones, the bug is either that the special page is being serialized or that when they did Iec2d82f6a they didn't take the serialization of the special page into account.
[19:18:30] * no_justification nods
[19:20:19] <anomie>	 no_justification: Hmm, no backtrace in that kibana link :( Fortunately the message is also in error.log, which does have a backtrace (:
[19:25:05] <anomie>	 no_justification: The rc_logid one looks like it's coming from Flow, specifically Flow\Data\Listener\RecentChangesListener::onAfterInsert() is leaving that field out when it manually creates a RecentChange object to save. It only happens if a 'RecentChange_save' hook function adds tags to the RC entry.
[19:32:08] <anomie>	 no_justification: Looking at the backtraces for the serialize ones in error.log, it looks like it's more likely that they're somehow putting their filters (including the closures) into the ParserOutput, so when it tries to save the ParserOutput to the parser cache that happens.
[19:32:35] <no_justification>	 ouch
[19:32:42] <no_justification>	 Yeah you can't cache a closure
[19:35:00] <legoktm>	 o.O why are the filters going into ParserOutput...
[19:35:41] <anomie>	 That I have no idea. I couldn't manage to quickly track it down through static analysis and don't want to spend more time on it.
[20:47:11] <James_F>	 no_justification, Reedy: Whoops. Thanks for fixing that VE issue.
[20:47:23] <no_justification>	 I got the first one, second is harder
[20:47:27] <no_justification>	 The undefined $content one
[20:47:33] <no_justification>	 I looked, but that function is like 21909120912 lines long
[20:47:39] <no_justification>	 I couldn't follow all the branches
[20:47:50] <no_justification>	 (er Reedy got, I deployed)
[20:47:56] <James_F>	 Oh, hmm.
[20:49:25] <James_F>	 no_justification: Can you file a task? Will probably need to get Ed/Bartosz to look at that.
[20:53:51] <no_justification>	 T176440 filed
[20:53:52] <stashbot>	 T176440: Undefined variable: content in ApiVisualEditor - https://phabricator.wikimedia.org/T176440
[20:54:34] <no_justification>	 James_F: It's a few-off entries, not a ton. So no need to drop everything, must be an edge case
[20:54:59] <no_justification>	 (or a simple kludge fix of just unconditionally set it to '' or something, but that's your call)
[20:56:26] <no_justification>	 James_F: Also just spotted, but I dunno how to describe it other than "Incompatible constraints": https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2017.09.21/parsoid?id=AV6mLwTBTqnoMLVn6rDi&_g=h@5ff62f2
[20:58:29] <James_F>	 no_justification: Yeah, thanks. Will pass the Parsoid one to the Parsing team.
[21:01:51] <subbu>	 James_F, no_justification, we'll just quiet those .. something that has been there since day one .. we've fixed a bunch since then, and the current resolution is the best we can do for the most part .. so no point clogging up production logs with it.
[21:02:15] <no_justification>	 Okie dokie, whatever you think's the best course of action.
[21:02:33] <no_justification>	 We could also adjust logging levels -- so you can still have them to play whack a mole with but most people won't notice them
[21:02:37] <no_justification>	 Like INFO level or something
[21:03:45] <subbu>	 those come out as warnings from parsoid .. so, we have to adjust those on our end .. unless you have other filtering magic on the logstash end.
[21:07:22] <no_justification>	 I've got them coming from wtp*
[21:07:31] <no_justification>	 See that example I linked ^^
[21:09:13] <subbu>	 ya, that makes sense .. those are parsoid servers ... so, are you saying you can throttle those in logstash directly?
[21:18:31] <no_justification>	 I could filter them out of dashboards yes, but I'm aiming for "proper logging" of stuff in a way that A) the right people see it, and B) it doesn't look like an issue to me
[21:18:47] <no_justification>	 For things like this, where it's not a /big/ problem and doesn't cause any operational issues with the sites
[21:30:57] <subbu>	 makes sense. maybe we should just look at hiding those on the parsoid end but enable those warnings on parser tests, so we still have opportunities for playing whack-a-mole. i think that is what we'll do.
[21:31:57] <subbu>	 thanks for flagging this .. had meant to do this for a while, but didn't get around to it ... but today is a good day to tackle it since i already pushed another log noise suppression patch earlier today.
[21:32:55] <no_justification>	 No worries, first time I'd seen it :)
[21:34:57] <no_justification>	 anomie, legoktm: Hate to nag, but any thoughts on that Closure ending up in the pcache?
[22:42:20] <legoktm>	 no_justification: I can look later today
[22:42:28] <no_justification>	 thx