[00:02:19] ori: good! kinda, we're focusing on cleaning up some other stuff in the jit for now [00:02:24] to try and give it larger regions of code to compile [00:02:41] is hhvm behaving itself, other than the statcache issue? [00:04:40] swtaarrs: the biggest pain-point is that the cold TC eventually gets exhausted and HHVM crashes and restarts [00:05:14] it happens more frequently now than it used to, and we're not sure why. does APC storage count against that limit? [00:05:18] ah yeah :( [00:05:24] ori: no, APC is all from malloc [00:05:30] how often do you naturally restart it? [00:05:42] or is that always what causes restarts? [00:06:00] paulbiss has a diff to garbage collect old, dead translated code that could help with that [00:06:22] we should try to get that in [00:06:25] the latter. _joe_'s theory is both funny and plausible: that 3.6 is much more stable than 3.1, and that the lack of semi-frequent segfaults means we run into the TC exhaustion issue more frequently [00:07:24] haha [00:07:33] yeah that could do that [00:07:43] we plan to switch to repo auth sometime soon™ [00:07:56] nice [00:08:04] it's been pretty awesome otherwise [00:08:10] ori: do you know how long it generally lasts before the TC fills up? [00:08:24] the longest we normally go without restarting is over the weekend [00:09:18] http://ganglia.wikimedia.org/latest/?r=week&cs=&ce=&c=API+application+servers+eqiad&h=&tab=m&vn=&hide-hf=false&m=HHVM.tccoldsize&sh=1&z=small&hc=4&host_regex=&max_graphs=0&s=by+name [00:09:21] 3-4 days, it looks like [00:09:22] on normal weekdays we have 3 pushes [00:09:26] ok [00:09:33] yeah, definitely longer than we run it [00:09:44] there could also be something in your php that's making it worse [00:10:12] we have a lot of code that appears to be specifically written to make things worse in general, so that's a possibility [00:10:24] haha [00:11:11] ori: if you can spare the memory, you could always increase the cold size in the config file to last longer [00:11:21] that's just a bandaid though [00:12:03] we push about as often as you do, but we don't restart hhvm, because our deploy process does not currently depool hosts while they are getting updated [00:12:18] yeah [00:12:18] this is a problem for a number of reasons, most of which predate HHVM [00:12:28] so we really just have to improve our deployment process [00:12:38] there's work happening on that front [00:13:25] nice [00:34:47] AaronSchulz: what's your gut reaction to the backend goals on https://phabricator.wikimedia.org/T101345 ? [00:35:48] we have june, july, august, september [00:36:28] i think APC caching of lua modules will bring us halfway there [00:50:46] bd808: Wanna land the header() stack? [03:35:42] bd808: If I wanted to change a log message in RL from INFO to DEBUG (to keep it out of wmf logs by default because it's too verbose), how would I do that? [03:35:56] https://github.com/wikimedia/mediawiki/blob/master/includes/resourceloader/ResourceLoader.php#L636 [03:36:02] That line about private modules being denied. [03:36:19] That's like 99% of the logs and it's not useful. It's just things accessing a page that we don't allow. [03:36:31] convert RL to PSR3 logging where you can specify the level [03:36:53] * bd808 looks for an example [03:38:25] Krinkle: something like Kunal did in https://gerrit.wikimedia.org/r/#/c/213011/ [03:39:04] And then we can set that logging channel to INFO in wmf-config to drop the spammy messages [03:40:29] Ah, I see [03:40:35] that's what wfDebugLog does essentiallu [03:40:40] exept instead of calling info(), call debug() [03:40:41] right? [03:40:44] right [03:40:49] cool [03:41:06] the config line to change would be https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/InitialiseSettings.php#L4303 [03:41:42] as an example: 'runJobs' => array( 'level' => 'debug', 'logstash' => 'info' ), [03:41:59] I see api.php also uses a logger now in its entry file [03:42:14] I'm wondering what this does to our parsability of the entry point files [03:42:19] is 'use' valid php4 syntax? [03:42:47] the plan is to slowly kill off wfDebug and wfDebugLog [03:42:59] hmmm.. it is not [03:43:06] Yeah. I'm not worried about it working in php4, the file just has to parse [03:43:12] are we still worried about that? [03:43:28] I don't know. I guess not [03:43:46] keeping index.php parseable would be nice for error displau [03:43:56] but not sure the other entry points are useful at this point [03:44:07] error: you are using ancient tech and we don't care :) [03:44:28] I mean php 5.3 is EOL for christ sakes [03:45:00] EOL for 4.4 was 6 years, 9 months ago [03:45:16] Yeah [03:45:41] I would really really really like to bump min version to 5.5 [03:46:28] of course that is majorly blocked by the 5.3 hosts in prod [03:55:10] bd808: Yeah, but let's start with dropping php4 further :P [03:55:31] That seems like it would be uncontroversial [03:56:18] want to make a wikitech-l post or should I? [03:59:17] bd808: Hmm yeah, basically we should drop the need for files to parse in PHP4. [03:59:33] bd808: Though it'll go up quite a bit at once if we want to allow 'use' in entry points [03:59:38] because that's new in PHP 5.3 [03:59:52] so our minimum php version warning won't just be obsolete for PHP4, it'll be obsolete entirely [04:00:23] hmmm [04:01:00] Hm.. how can I get "INFO" and "DEBUG" to be printed in my local mw logs? [04:01:10] Lookslike it's not in there by default? [04:01:33] It's not unless you use Monolog and a format that adds it [04:01:48] https://github.com/Krinkle/dotfiles/blob/master/hosts/KrinkleMac/mw-CommonSettings.php#L119-L128 [04:01:55] I tried to make the LegacyLogger output look just like the old logs [04:02:24] I'll make a past of my config somewhere [04:02:30] *paste [04:04:23] Ideally I'd be able to set wgMWLoggerDefaultSpi to a different class and get the Monolog formatting, but still have it use wgDebugLogGroups and wgDebugLogFile. [04:04:53] that is possible. It takes a bit of work [04:05:09] there is code to do it in the history of wmf-config [04:05:42] Seems like this would be the common use case for most devs and mediawiki-vagrant [04:05:53] perhaps even flip to be the default at some point. [04:06:09] Yeah I've thought about making a helper for it [04:06:12] but keeping the old format without the context and channel parts for back-compat under an opt-in class name. [04:06:34] Or as parameter to this class [04:07:20] Ah, interesting. Each legacy function has its own format function [04:07:30] yeah it's a mess [04:07:47] I really tired to make things look just as horrible as they always did [04:08:02] Why is wfDebugLog the default though? Could we not use wfDebugLog format only for wfDebugLog, and then for things that are logger aware directly (and not use wfDebugLog) get the new format by defaulta. [04:08:23] This old version of logging.php show how to convert to monolog fairly transparently -- https://raw.githubusercontent.com/wikimedia/operations-mediawiki-config/20fa1f08ef3f2ab83367519c374b1a6b5d7fbd81/wmf-config/logging.php [04:08:58] wfDebugLog is the default because it had named channels [04:09:08] Right [04:09:13] it's not its own channel [04:09:15] makes sense [04:12:22] So I change class to mw\logger\MonologSpi, and then args with ... something? [04:12:43] I could (and should) make a much simpler old settings -> monolog transform [04:13:10] $wmgMonologConfig is most of the magic [04:13:51] oh that version didn't log to files? [04:14:00] * bd808 stares at his old code [04:15:28] Looking at LegacySpi, I figured the following would do it, but I guess that was too easy [04:15:34] $wgMWLoggerDefaultSpi = array( [04:15:34] 'class' => '\\MediaWiki\\Logger\\MonologSpi', [04:15:34] 'args' => array( [04:15:34] https://phabricator.wikimedia.org/P725 [04:15:36] 'loggers' => array( [04:15:38] '@default' => 'MediaWiki\\Logger\\LegacyLogger' [04:15:40] ), [04:15:42] ), [04:15:44] ); [04:16:02] what are those processors? [04:16:12] that's the config I use in my vm [04:16:12] it only goes to one file [04:16:14] it has a nice format though [04:16:27] the processors seems like boilerplate. What does that mean? [04:16:32] they add metadata to each event [04:16:43] that's not default monolog? [04:17:06] nope. it's a framework you need to configure [04:17:36] * Krinkle will stick to mw default for now [04:17:37] the psr one is basically required in our setup [04:17:52] the wiki one I made to add dbname and some other bits [04:18:05] yeah [04:18:23] I don't want to choose these kind of things as a mediawiki developer though. [04:19:55] I think that explains a lot about how MediaWiki works :/ [04:19:58] And we probably don't want encourage people to all come up with their own log format. [04:20:17] epecially when sharing logs in irc or other support desks [04:20:52] it's good to expose the option though [04:21:54] I was talking at the hackathon about introducing a monolog setup [04:22:12] we could make a nice set of defaults and still allow customization [04:22:44] I'd like to get more of core converted to richer logging to make it make more sense [04:23:04] baby steps [04:23:26] I've "only" been working on this project for 18 months :) [04:24:25] Yeah, it'd be nice to be able to enable proper monolog formatting locally (whatever proper means, I guess we'll make our own standard, maybe the one we use at wmf/fluorine at the moment; as long as it includes all the important parts like channel,level,context etc.). Shoudl be configurable with nothing more than $wgMWLoggerDefaultSpi = array( 'class' => '\\MediaWiki\\Logger\\....something..' ); and t [04:24:25] hen the optoin to pass extra args to enable/disable/raise-min-level of individual channels. [04:25:38] anything beyond that is probably edge-case only of interest to large wiki farms and third parties that want custom formatting and/or custom writers to non-file (e.g. redis, udp whatev) [04:26:41] I was going to point you at the config that the mw-vagrant elk role sets up but I see that it is broken :/ [04:26:44] It's nice that LegacyHandler is re-usable this way. It deals only with writing to disk/udp etc. [04:28:36] alrighty, good night :) [04:28:36] o/ [04:28:41] o/ [05:20:58] gah. I can't figure out why the continue on my query isn't working. [05:21:21] I put in the returned continue value, but I get the same result over and over again. [13:46:57] ragesoss: Did you figure out your query continuing problem? I assume that's api.php? [15:35:24] anomie: I didn't figure it out, but I worked around it by querying fewer files at a time. [15:35:29] anomie: yes, api.php. [15:36:34] anomie: here's an example of the problem: https://commons.wikimedia.org/w/api.php?action=query&continue=&format=json&iilimit=50&iiprop=url&iiurlheight=480&pageids=3982584|3982588|3982595|3982603|3982606|3982617|3982634|3982639|3987381|3988044|3988237|3988332|3988424|3988480|3991285|3992308|3992901|3992908|3992939|3993018|3994073|3994076|3994083|3994195|3994505|3994506|3994508|3994510|3994584|3994630|3994692|3997992|4016017|4030078|4030104| [15:36:34] 4030351|4031166|4033567|4034071|4034081|4034085|4039921|4040180|4040191|4040194|4040220|4040222|4048502|4051075|4068154&prop=imageinfo [15:36:55] ragesoss: Thanks, I'll check it out once I'm done with SWAT [16:23:48] 10MediaWiki-Core-Team, 6Security, 6Security-Team, 5Patch-For-Review, and 2 others: XSS in language converter when used with Html class's tricky escaping - https://phabricator.wikimedia.org/T73394#1337901 (10dpatrick) [16:25:36] It would be really nice if wikibugs indicated that bug was already closed... [16:28:23] heh [16:28:30] file a bug? [16:37:37] legoktm: is there any easy way for me to test https://gerrit.wikimedia.org/r/#/c/215280 or can I just trust that you tested the heck out of it? [16:38:17] 10MediaWiki-Core-Team, 6Security, 5Patch-For-Review, 10Security-Core, 7Vuln-XSS: XSS in language converter when used with Html class's tricky escaping - https://phabricator.wikimedia.org/T73394#1337996 (10dpatrick) [16:38:34] bd808: if you have the queue set up, rename user:A --> user:B, then have B file a rename request to user:C, visit the queue entry for B-->C and you should see a log snippet [16:45:04] thanks :D [16:45:09] yw [16:45:24] do you have other SUL stuff roting away anywhere? [16:49:33] don't think so [16:51:07] good [16:56:55] Nikerabbit: Do you know much about LanguageConverter? [16:57:34] (not related to T73394 that I see wikibugs talking about) [17:07:14] anomie: if you mean the zh.wiki issue of today, that was being discussed in #wikimedia-operations [17:13:58] legoktm: ohai, I was told to ask you about UserMerge and ask A) are you loan-wolf'ing this? ('tis fine, just asking) and B) what should I follow/put in #roadmap? [17:14:12] Nemo_bis: I figured out the root cause of that. Part of it is obviously something is crashing that shouldn't be, but the other part is what LanguageConverter should be doing to not even get to that point. [17:16:39] anomie: is there a task for that? James will probably be following up with Tim about it later today [17:19:14] anomie: for this sort of thing you should just cc Liangent [17:20:26] and https://phabricator.wikimedia.org/T98386 will help [17:21:44] anomie: it's not really my cup of tea, I'm afraid [17:21:50] Nikerabbit: Thanks anyway [17:48:19] anomie: [17:48:20] /* Don't add zero-length strings, that will cause infinite loops in [17:48:20] search routines */ [17:48:21] if (!elem.size()) { [17:48:23] continue; [17:48:25] } [17:48:27] * ori squints [18:14:36] hello, can anyone please review https://gerrit.wikimedia.org/r/#/c/189161/ ? thanks [18:15:00] today is not a good language converter day :/ [18:22:42] Nikerabbit: can you rereview ^ [18:26:33] anomie: I think my example only crashed because calling fss_free() causes a double-free, but we don't use fss_free() in prod [18:26:57] so that seems like a separate bug [18:27:06] so i'm back to not having a repro case. i can't make your example crash. [18:27:23] * anomie looks [18:28:26] ori: hhvm --php -r 'fss_prep_replace( array( "" => "foo" ) );' doesn't do it for you? [18:28:48] hah, yes it does. ok, thanks. [18:30:17] aaaand i have a fix [18:30:46] \o/ [18:36:44] anomie: https://gerrit.wikimedia.org/r/#/c/215956/ [18:40:47] anomie: you can test on osmium [18:40:55] the build there has my patch applied [19:00:10] bd808: there are some weird filenames in /a/mw-log like "->tree_recurse('1177',.log" on fluorine, is that known? [19:04:51] Yeah that happens from time to time [19:10:18] AaronSchulz: do you have any idea why MassMessage jobs would take a long time to run? https://phabricator.wikimedia.org/T101427 [19:33:39] legoktm: that is a udp2log bug -- https://phabricator.wikimedia.org/T64082 [19:47:59] Hi guys [19:48:02] can anyone help me with Non-Gregorian calendars? [19:55:57] that probably depends on what you want to do [20:59:55] csteipp: could you grant gilles the right to view security bugs? [21:00:08] csteipp: I sent you an email about AuthManager that I'm hoping you'll have a chance to answer at some point this afternoon so I can look at it tomorrow morning. [21:00:33] anomie: Cool, I'll look at it. [21:01:05] ori: Is he planning on working on some? [21:01:28] Or did did he just want to get an idea of the issues we're having? [21:01:43] the latter [21:03:52] Cool. Done. [21:04:46] thanks! [22:58:47] hey [22:58:51] i need help with this: [22:58:51] 2015-06-04 22:56:23 mw1026 zhwiki fatal INFO: [46f0c536] /wiki/Jay_Chou ErrorException from line 72 of /srv/mediawiki/php-1.26wmf8/includes/libs/ReplacementArray.php: PHP Error: Object of class Article could not be converted to string [23:00:35] Article should have a __toString() [23:00:37] For one [23:01:22] line 72 is $this->fss = false; so I'm confused. [23:09:09] bd808: so the discussion has managed to get fragmented yet again to https://github.com/composer/composer/issues/4109 :|