[00:05:47] dpifke: around for another 45min or so if you need anything [00:13:35] Do you have time to take a look at one or both XHGui patches now that upstream CI has given them the OK? [00:19:50] dpifke: ack, Iv'e reviewed those meanwhile, just need to do some local testing tomorrow. [00:20:01] Cool, thanks. [00:20:05] any config patches to deploy in the next hour? [00:20:47] How about https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/620142? [00:21:10] yep, sounds good to me. [00:21:14] And possibly https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/620139? [00:21:16] wanna do it now? [00:21:58] Sure, let me login to Kibana etc. [00:26:52] OK, all set. [00:29:18] ack [01:33:02] alright, I'm off. o/ [07:15:52] Krinkle: looks good :) [07:32:23] This is great: Mozilla regression tests against large Wikipedia pages like the Obama page: https://bugzilla.mozilla.org/show_bug.cgi?id=1659166 [13:03:37] phedenskog: the piechart plugin should be working now [18:08:47] FYI, we're growing the ram on the X002 (ArcLamp) hosts, which is going to require rebooting each. We're starting with webperf2002 in codfw. Discussion in -sre. [18:45:14] dpifke: to confirm, the main memory issue is with flamegraph.pl doing its thing. Everything around it is/can be streamed, right? [18:46:08] Yes. It needs to build a bunch of data structures in memory. [18:46:23] k [18:46:39] But there's still something odd going on with parallel execution I'm trying to track down now. [20:24:44] Anyone want to +2 https://phabricator.wikimedia.org/T259167? I'd like to roll it out with the fix for cron spam. [20:25:05] Wrong link: https://gerrit.wikimedia.org/r/621348 [20:27:35] dpifke: Hm.. so 1GB of input results in 16GB perl memory flamegraph? [20:28:02] 1GB compressed. [20:28:17] The biggest files seem to be about 2.7GB right now, and those are using ~7GB RAM. [20:28:35] right [20:28:37] interesting [20:28:40] With 1GB at the limit, we end up in the death spiral I described in the bug. [20:29:02] Might be worth raising to Brendan Greg, I wonder how common our input size is and what his thoughts on the matter are [20:29:11] He might get a kick out of optimising it by 10 or 100x [20:29:18] (on the upstream gh repo) [20:29:27] I can do that later as well, if you prefer. [20:29:36] There's two behaviors I wasn't aware of: the subprocesses being moved under PID 1, and set -e not detecting one of the pipe processes dying. [20:29:39] * Krinkle +2's [20:30:54] There may be limits to how much a Perl script can be optimized - not sure how the Perl runtime does memory management. [20:31:36] If it were a Python script, the standard answer would be to rewrite it in another language. :) [21:54:28] AaronSchulz: what're your priorities for this week? Note that I have a few CRs assigned to you but current prod/sustainbility take precendence I think, e.g. T255700 and T252564 currently in that category on phab. [21:54:31] T255700: "Bad content model: expected wikitext but got javascript" while saving/stashing an edit - https://phabricator.wikimedia.org/T255700 [21:54:31] T252564: Let WANObjectCache store "sister keys" on the same backend as the main value key - https://phabricator.wikimedia.org/T252564 [21:57:06] * AaronSchulz is amending https://gerrit.wikimedia.org/r/c/mediawiki/core/+/611164/3/includes/libs/objectcache/MemcachedBagOStuff.php#29 [21:57:37] the preprocessor one two, some CR, and loadmonitor [21:58:32] lots of stuff is still not setup on my laptop now that's ubuntu/kde