[16:25:19] blerg. why is a forceHTTPS cookie being set in my mw-vagrant vm? [17:01:04] anomie: my task list says "Poke Brad to remove -2 on https://gerrit.wikimedia.org/r/#/c/160223/" [17:01:30] bd808: Thanks, I'll do that in a little bit [17:01:50] * bd808 is released from his duty [17:23:05] csteipp: http://grafana.wikimedia.org/#/dashboard/db/activity any idea why new accounts would rise like that? [17:23:14] seems like a metric error somehow [18:07:15] hey is anyone aware of what the purpose of extension:EventLogging is? [18:07:30] and what purpose requesting :8080/event.gif holds [18:10:53] https://www.mediawiki.org/wiki/Extension:EventLogging [18:13:58] https://pypi.python.org/pypi/monotonic/ "3130 downloads in the last day; 10212 downloads in the last week" [18:14:00] * ori blinks [18:18:47] thanks greg, but they don't really explain how it is used by other extensions [18:19:19] PJosepherum: the event.gif endpoint is a lightweight way to log data to the server from client-side code [18:19:59] javascript code constructs some object with data, like { userId: 123, foo: 'bar' }, that gets serialized to a query string and requested from event.gif? [18:20:01] cheers, so no file need actually exist? basically i'm just wondering how i can test that it is working, as it's a dependency for some other extension i'm using [18:20:14] the event.log is being written to, so i guess so [18:20:15] yes, there's no file expected at that url [18:20:19] yeah [18:21:02] cool, thanks for the explanation [18:56:46] ori: nice download stats. all praise the marketing power of the leap second :) [19:19:14] bd808: https://gerrit.wikimedia.org/r/#/c/160223/ is rebased and unblocked. Remember, it should be merged on Tuesday after 1.26wmf11 is cut. [19:21:02] anomie: I put my own -2 on it with a description [19:21:16] super exciting [19:21:17] ok [21:38:07] csteipp: can I ask you to prioritize https://phabricator.wikimedia.org/T103185 ? It's a security review for kzykhys/pygments, which is really a thing wrapper for Pygments utilizing symfony/Process (https://github.com/symfony/Process). The latter is a part of Symfony, which has a good security process AFAIK (http://symfony.com/doc/current/contributing/code/security.html). [21:39:50] ori: I'm swamped at the moment, but I'll try to get to it next week. What's the driver behind this? [21:41:23] csteipp: go to https://en.wikipedia.org/w/load.php?debug=false&lang=en&modules=startup&only=scripts&skin=vector&* , ctrl+f and search for 'geshi', see number of results, stare in shock and horror as you realize that this is loaded on every. single. pageview. [21:42:31] Ah, that problem.. [21:42:41] So why is that using symphony/process? [21:47:00] csteipp: I had a patch to use wfShellExec(); it just seemed uglier. [21:47:25] wfShellExec() sucks because it only gives you combined stdout and stderr or stdout only, you can't have separate streams [21:48:32] i wouldn't have introduced symphony/process for that convenience alone, but the fact that there existed a PHP library that wrapped Pygments with a nice PHP API (and used symfony/process behind the scenes) tipped the scale for me [22:43:09] ori: not sure why Geshi is loaded by default. I thought its RL material would only be loaded for pages having :/ [22:44:08] have good perf tracking folks and enjoy the week-end [22:56:12] MaxSem: What does your $wgWikiHieroService look like to use the labs instance? [22:56:43] csteipp, see the commit message [23:22:50] ori: do you know if our puppet rules ensure running for apache? Wondering if that is going to block/complicate the scap restart stuff [23:24:42] bd808: yes, they do. but submit your patch as though they won't, and i'll adapt the apache puppetization accordingly. [23:33:27] ori: next random question, how many mw servers should we restart at once? The list we have to work from right now is just everything there is with no differentiation between cluster role [23:42:22] bd808: anywhere from 10-20% seems sensible as a starting point [23:44:40] bd808: once there's a patch we can easily coordinate a brief experiment with ops where we depool progressively more app servers to see when the cluster starts to sweat. giuseppe did that shortly after the initial rollout of HHVM for estimating our carrying capacity then. [23:45:18] we don't actually have to push it to an extreme and find the point the cluster breaks; we just need to make sure that 20% is more than fine even during peak load and leave it at that. [23:46:26] hmmm.. actually do we know appservers from jobrunners from vidscalers in scap? I'm not sure that we do any more [23:47:17] the scalers and job runners aren't in pybal [23:50:54] ori: the list we have right now in scap is ~480 servers in general that we push MW to. 10% of that is too much I think [23:51:07] we don't have pybal yet [23:51:24] or should this wait for pybal access? [23:51:57] we have way more app servers than we need at the moment [23:52:10] like i said, we can determine a number empirically [23:52:23] re: pybal, no -- you can determine locally if the host is registered with pybal [23:52:58] you can run 'ifconfig lo:LVS' and see if it has an address assigned [23:53:30] but why do you need to distinguish app servers from job runners from scalers for the gradual depool? [23:54:58] some of those roles don't have many server right? so if the batch size is say 20 and all 20 at some point are scalers then that seems bad [23:55:44] but I guess at the moment that isn't a problem as the scalers won't have hhvm running and can be skipped [23:55:52] the role assignment happens in contiguous blocks, too [23:56:05] so if you shuffle the list of hosts, that's pretty good [23:59:07] could there be issues if it chooses all hosts in a group to restart at the same time as all from the same DC? [23:59:47] well codfw is just laying around so that's not a problem [23:59:58] at the moment, sure