[08:54:17] <_joe_> cdanis: no, and I remember that spike [08:55:19] <_joe_> but given it was over when I noticed it, and that it was mostly driven by the 95th percentile (so some very slow requests), I didn't investigate further [12:05:24] _joe_: interesting. worker thread starvation isn't a root cause but it is interesting [12:46:58] _joe_: have we ever attempted to make use of the USDT available in php? https://www.php.net/manual/en/features.dtrace.dtrace.php [12:47:28] eBPF features are available and robust in buster [12:59:37] <_joe_> cdanis: nope [12:59:50] <_joe_> and well, we'll have to move forward to buster first [13:00:10] i think they're in stretch kernels, but the userspace tools prepackaged there are quite old [13:01:11] <_joe_> cdanis: one interesting thing to try would be to get a php backtrace of each process [13:01:23] it would be amazing [13:01:39] <_joe_> something we had with HHVM [13:01:53] <_joe_> well there it was enough to do perf record [13:02:52] it would be amazing to have something fast enough to leave running all the time, that just snapshotted stack traces of all running workers, and then aggregated them [13:03:15] <_joe_> yeah well, we kinda-have it with excimer [13:03:30] <_joe_> we do aggregate execution samples already [13:04:09] but excimer profiling is too expensive to do all the time, right? [13:04:14] <_joe_> no [13:04:29] <_joe_> but it's a reporting system, not something you can inspect at runtime [13:06:43] a tool we could run during an outage would be great [15:04:33] anyone know what causes an #invoke call to show up as text on a page, instead of being evaluated? parse timeout? [15:05:41] ah

#invoke:Navbox with collapsible groups [15:06:06] Post‐expand include size: 2078469/2097152 bytes [15:06:08] yeah ok [15:08:47] I'm sure this is happening for a bunch of the other pandemic articles -- I noticed this on https://en.wikipedia.org/wiki/2020_coronavirus_pandemic_in_Italy [15:15:55] Yeah, it's expected that editors fix their content when that happens. [15:16:14] (Rather than refusing to save.) [15:18:13] James_F: what does the 2MB limit refer to? [15:19:04] cdanis: Raw (pre-parsed but post-expanded) wikitext. [15:19:22] So '''Foo''' not Foo [15:19:23] ah okay, that was my hunch after checking the HTML size, thanks [15:19:33] (HTML size is 1.2Mbyte or so) [15:19:38] It's a limiter to stop the Parser being fed way too much. [15:20:24] is there anything tracking the post-expand size distribution of articles? [15:20:26] 2MiB of wikitext or 10k templates, whichever limit you hit first. [15:21:20] I don't think so; the HTML representation is lighter than or the same as the wikitext weight except for tables and media files, so it'd be hard to have a meaningful page that was problematic. [15:23:52] also happening on https://en.wikipedia.org/wiki/2020_coronavirus_pandemic [15:24:04] Yeah. :-( [15:24:25] looks like it's mostly affecting the navboxes, which isn't great, but isn't terrible [15:24:47] this is on some sort of fuzzy boundary for me between being a content issue and a tech issue [15:24:59] Totally. [15:25:44] breaking things into sub-articles is probably part of the answer? but I also expect the top-level article to be a huge ball of wikitext no matter what. [15:25:54] Yes. And yes. :-( [15:26:27] is there an appropriate next step here? noting the issue on their talk pages perhaps? [15:26:36] Leave it to them. [15:26:40] ok [15:26:43] This is a common issue on "hot" articles. [15:27:01] I was guessing/hoping there was enough community expertise around this [15:28:27] I've flagged it to some enwiki sysops. [15:28:58] do you happen to know if there's any bots tracking the occurrence of this / any data we provide re: such? it seems like a not-uncommon maintenance task [15:29:01] thanks :) [15:30:43] No; maybe there should be a tracking category? Hmm.