[02:52:34] <tgr>	 no_justification: is there a way to get clone/download stats for a WMF git repo?
[02:53:47] <no_justification>	 Not really, no. That'd be cool though!
[02:53:57] <no_justification>	 We have download stats from Extension/Skin Distributor
[02:54:40] <tgr>	 yeah, and theoretically from Github although that seems hard to get
[02:55:33] <tgr>	 gerrit does not have tarballs, right?
[02:56:04] <no_justification>	 Nope
[02:56:12] <no_justification>	 Now you've never sniped me into seeing if I can write a plugin to extract this :p
[03:01:08] <tgr>	 no_justification: https://phabricator.wikimedia.org/T186848
[03:01:33] <no_justification>	 There's already a task for tarball downloads of MW
[03:01:38] <no_justification>	 Somewhere
[03:14:21] <no_justification>	 tgr: UploadPackMetricsHook.java
[03:14:23] <no_justification>	 Bam
[03:14:23] <no_justification>	 Sounds like an in!
[03:17:59] <tgr>	 cool
[03:28:30] <tgr>	 gitiles is awesome, btw
[03:28:42] <tgr>	 faster than github, probably
[03:30:41] <no_justification>	 :)
[03:30:43] <no_justification>	 I love gitiles
[03:32:25] <no_justification>	 Without even digging further, here's the top of the file....
[03:32:26] <no_justification>	 public class UploadPackMetricsHook implements PostUploadHook {
[03:32:26] <no_justification>	   enum Operation {
[03:32:27] <no_justification>	     CLONE,
[03:32:27] <no_justification>	     FETCH;
[03:32:27] <no_justification>	   }
[03:32:33] <no_justification>	 I think we can get allllll we want :)
[03:33:30] <no_justification>	 I wonder if these are all included in some of the metrics plugins
[03:37:33] <no_justification>	 tgr: Sooooo, https://gerrit.wikimedia.org/r/Documentation/dev-plugins.html#metrics. Currently there's 3 plugins for metrics (elasticsearch, graphite, jmx)....I'm curious /what/ all metrics it outputs
[03:37:50] <no_justification>	 I could probably whip up a quick and dirty one that dumps it to json or something
[03:44:35] <no_justification>	 I also discovered a fun plugin point: https://gerrit.wikimedia.org/r/Documentation/dev-plugins.html#included-in
[03:44:45] <no_justification>	 That could be useful for the "is it deployed" question
[05:25:23] <tgr>	 no_justification: having group0/1/2 in "included in" would be awesome
[05:27:21] <tgr>	 https://github.com/gerrit-review/gerrit/tree/7d35ff3fbeddc22fd2e0b42ff3f971598bba0c25/java/com/google/gerrit/metrics/proc seems like the place for default metrics but it's mostly low-level stuff
[05:28:11] <no_justification>	 Well there's a lot of higher level metrics too
[05:28:17] <no_justification>	 I just gotta play with metrics exporting
[05:28:21] <no_justification>	 (there isn't a default)
[09:08:48] <paladox>	 no_justification: and we could just include above the file list in polygerrit what group we are on with mw 
[09:08:49] <paladox>	 Heh
[15:19:42] <anomie>	 no_justification: The TL;DR for your division by zero log message is that the new version of wikimedia/running-stat in wmf.20 (change d0f42c24ced) broke PHP serialization compatibility in a non-obvious way, so when an instance of PSquare cached in wmf.17 is unserialized in wmf.20 it's missing critical data in its private fields. Details in T186839#3958490.
[15:19:43] <stashbot>	 T186839: Divide by zero in PRSquare  - https://phabricator.wikimedia.org/T186839
[15:25:15] <no_justification>	 Ah. So just ignore and it'll go away when we finish
[15:25:25] <no_justification>	 That's fine, it wasn't super spamny
[17:36:57] <no_justification>	 anomie: Comment stuff.... Catchable fatal error: Argument 3 passed to CommentStore::createComment() must be an instance of array, string given in /srv/mediawiki/php-1.31.0-wmf.20/includes/filerepo/file/LocalFile.php on line 2512
[17:38:57] <anomie>	 hmm
[17:44:57] <anomie>	 no_justification: https://gerrit.wikimedia.org/r/409391
[17:48:03] <no_justification>	 +2'd, will backport & deploy thx
[17:48:06] <no_justification>	 Any thoughts on [{exception_id}] {exception_url} BadMethodCallException from line 906 of /srv/mediawiki/php-1.31.0-wmf.20/includes/Revision.php: Call to a member function getContent() on a non-object (null)
[17:48:09] <no_justification>	 Just spotted.
[17:48:13] <no_justification>	 Man, wmf.20 is noisy!
[17:53:01] <anomie>	 No idea on that one, unless something is unserializing a Revision object somehow. DanielK_WMDE or addshore might have insight.
[18:02:54] <anomie>	 ... Ok, that actually is it. AbuseFilter is serializing whole Article objects for some of its logged parameters.
[18:04:19] <anomie>	 (and $article->mPage->mLastRevision is a Revision object)
[18:04:32] <anomie>	 no_justification: ^
[18:05:15] <no_justification>	 Ew
[18:16:22] <no_justification>	 addshore: [{exception_id}] {exception_url} Wikibase\Lib\Store\RevisionedUnresolvedRedirectException from line 103 of /srv/mediawiki/php-1.31.0-wmf.20/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php: Unresolved redirect to Q33072118
[18:25:07] <addshore>	 Let me grab my laptop
[18:25:52] <addshore>	 no_justification: how much is that happening?
[18:26:01] <no_justification>	 I just saw the one
[18:26:12] <no_justification>	 But I'm paying close attention since wmf.20 been so wobbly
[18:27:09] <addshore>	 no_justification: do you have the whole stacktrace?>
[18:27:14] <no_justification>	 One moment
[18:27:40] <addshore>	 It doesn't look too concerning
[18:29:53] <no_justification>	 https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2018.02.09/mediawiki#AWF7vfmQqJPajYV1St_v
[18:30:27] <addshore>	 "Oh no. Something went very wrong. Its not just that I couldn't find your document, I couldn't even try. The index was missing, or the type. Go check out Elasticsearch, something isn't quite right here. :D"
[18:30:57] <no_justification>	 Grrr
[18:31:00] <no_justification>	 I'll pastebin
[18:31:30] <no_justification>	 https://phabricator.wikimedia.org/P6676
[18:33:04] <addshore>	 no_justification: looks similar to https://phabricator.wikimedia.org/T172443
[18:33:17] <no_justification>	 Good enough for me
[18:33:28] <no_justification>	 You ain't worried? I ain't worried
[18:33:35] <addshore>	 Yeh, I wouldnt worry too much about it, but perhaps just add the stack to that ticket
[18:37:06] <no_justification>	 anomie: divide by zero still happening, even after group2 rollout
[18:38:09] <no_justification>	 Unrelated: Notice: Undefined index: wiki in /srv/mediawiki/php-1.31.0-wmf.20/extensions/AbuseFilter/includes/api/ApiQueryAbuseLog.php on line 143
[18:40:57] <anomie>	 no_justification: It'll happen until all APC-cached instances of PSquare from wmf.17 fall out of cache. I don't know how long that will take.
[18:41:07] <anomie>	 (I think it's the APC cache, anyway)
[18:41:18] <legoktm>	 can we just change the cache key?
[18:43:58] <anomie>	 That could probably be done. As far as I can tell the only point to the whole thing is that it's using PSquare to try to figure out the average time for an {{#invoke:}} so it can log ones that are much longer.
[18:47:38] <legoktm>	 [10:47:26] <wikibugs> (PS1) Legoktm: Invalidate slow function call cache to avoid warnings [extensions/Scribunto] - https://gerrit.wikimedia.org/r/409411 (https://phabricator.wikimedia.org/T186839)
[19:04:31] <no_justification>	 Ah, it's no longer divide by zero
[19:04:33] <no_justification>	 I just noticed
[19:04:40] <no_justification>	 Now it's undefined index 0
[19:07:52] <anomie>	 The "undefined index 0" was happening before too. It's another result of the same unserialization thing.
[19:08:24] <anomie>	 no_justification: Anyway, https://gerrit.wikimedia.org/r/#/c/409411/ should be good to backport now.
[19:17:24] <no_justification>	 Yay errors going away
[19:42:48] <no_justification>	 The AbuseFilter one is the most common now
[20:00:40] <legoktm>	 no_justification: https://gerrit.wikimedia.org/r/409427 for AbuseFilter
[20:00:51] <no_justification>	 ty <3
[20:06:25] <legoktm>	 no_justification: https://phabricator.wikimedia.org/T186914#3959277 gerritbot didn't post a comment?
[20:07:04] <no_justification>	 Weird.
[20:08:08] <paladox>	 maybe it's in the log the error
[20:08:37] <paladox>	 i migrated the plugin to use the newer api. (not all though)
[20:29:37] <no_justification>	 legoktm: Got the cherry pick ready, when the CN deploy is done can do yours
[20:29:45] <legoktm>	 :)
[20:39:46] <James_F>	 no_justification: And https://gerrit.wikimedia.org/r/#/c/409438/ for one of the Structured Discussion ones.
[20:40:24] <James_F>	 https://gerrit.wikimedia.org/r/#/q/status:open+branch:wmf/1.31.0-wmf.20+-label:Code-Review-1
[20:41:16] <no_justification>	 Ty sir
[20:41:44] <paladox>	 heh, you just found a bug in polygerrit.
[20:41:59] <no_justification>	 James_F: some point when it's not Friday and I'm not doing last minute deploys we should talk about hoe to use that data I sent last night 😝
[20:42:04] <James_F>	 :-D
[20:42:06] * no_justification grins evilly 
[20:42:36] <James_F>	 DELETE * FROM user_preferences WHERE up_key = skin;
[20:42:39] <James_F>	 Simple.
[20:44:41] <no_justification>	 better: TRUNCATE TABLE user_preferences;
[20:44:46] <no_justification>	 Oh, I ran my script dry-run
[20:45:08] <James_F>	 How well does TRUNCATE do with secondary DBs though?
[20:45:08] <no_justification>	 It's wayyyyy too aggressive.... --hidden probably shouldn't be dropped most of the time
[20:45:17] * James_F nods.
[20:45:26] <no_justification>	 Also: --unknown I think I only check core prefs.
[20:45:29] <no_justification>	 Which would suck
[20:45:31] <James_F>	 Ha.
[20:45:32] <no_justification>	 For every extension
[20:45:35] <James_F>	 Details.
[20:45:42] <no_justification>	 I'm thinking instead: just give it a key/value
[20:45:46] <no_justification>	 "Delete all skins that have X"
[20:45:48] <James_F>	 Known-bad?
[20:46:00] <no_justification>	 We still have folks with the hidden HHVM pref, for one ;-)
[20:46:00] <James_F>	 Hmm. Might work for the first few big ones.
[20:46:03] <no_justification>	 I discovered this last night
[20:46:08] <no_justification>	 Like 10k users on testwiki with it
[20:46:09] <James_F>	 Well, dropping the HHVM pref is fine.
[20:46:19] <no_justification>	 (also: testwiki has 10k users with prefs?)
[20:46:24] <James_F>	 It's not like it's going to start being followed.
[20:46:32] <no_justification>	 The user preferences table is *fascinating*
[20:46:50] <James_F>	 ISTR we were auto-setting HHVM for all visitors to testwiki for a bit.
[20:47:06] <no_justification>	 Very possible, I saw a bunch of stuff I don't even remember....
[20:47:12] <no_justification>	 Like, skin names I'd never heard of
[20:47:22] <no_justification>	 All kinds of fun shit
[20:48:09] <no_justification>	 I spit it out in that huge JSON, but we could probably massage it into some nicer format
[20:48:13] <no_justification>	 So we can slice & dice
[20:48:28] * James_F grins.
[20:51:23] <no_justification>	 Heh, users with nostalgia, simple or myskin still set :p
[21:39:37] <tgr>	 anomie: can you do an emergency code review? https://gerrit.wikimedia.org/r/#/c/409449/
[21:41:34] <anomie>	 tgr: That does more than just ignore a comment at the end.
[21:42:01] <anomie>	 If there's a correct opening div and a closing div anywhere in the document, it'll hit.
[21:43:08] <tgr>	 is that a problem? the opening still needs to be at the beginning
[21:43:14] <tgr>	 a regexp seemed scarier
[21:49:07] <anomie>	 '/^' . preg_quote( $start, '/' ) . '(.*)' . preg_quote( $end, '/' ) . '((?>\s*<!--.*?-->)*\s*)$/s' should about do it, I think. Replace with '$1$2'. Or just match the substr( $text, $endPos - $startLen + $endLen ) against '/^(?>\s*<!--.*?-->)*\s*$/s' to validate it's just comments.
[21:51:55] <tgr>	 well, if it's not just comments, not unwrapping will still break mobilefrontend
[21:52:03] <tgr>	 but yeah, should be good enough for now
[21:57:35] <tgr>	 anomie: updated
[21:57:59] <anomie>	 tgr: If it's sane enough to do so, it'd be nice to add a test that actually runs a ParserOutput through the parser cache and then unwraps.
[21:58:58] <tgr>	 I'd rather do that in a followup once production's unbroken
[22:01:10] <anomie>	 Anyway, looks good. Go ahead and +2, I'm off.
[22:07:46] <tgr>	 thx
[23:15:06] <no_justification>	 legoktm: Soooo, I know this is an HHVM bug really, but I saw it spike today with wmf.20
[23:15:13] <no_justification>	 Warning: Unable to record MySQL stats with: EXPLAIN /* MediaWiki\Linter\Database::getTotalsEstimate */ SELECT * FROM `linter` WHERE linter_cat = '5'
[23:15:26] <no_justification>	 I haven't seen it for linter before today (the 5 is variable, obviously)
[23:15:49] <legoktm>	 we just switched from SELECT COUNT(*) to using $dbr->estimateRowCount() which does EXPLAIN SELECT COUNT (*)
[23:18:32] <legoktm>	 no_justification: should we ask DBAs what to do about the warning?
[23:20:35] <no_justification>	 No :(
[23:20:40] <no_justification>	 I ran the warning down ages ago
[23:20:44] <no_justification>	 It's a supposedly-fixed HHVM bug
[23:20:48] <no_justification>	 But it still happens
[23:20:52] <no_justification>	 Nothing to do but ignore it
[23:20:56] <no_justification>	 Was more fyi
[23:22:38] <legoktm>	 ok :)