[00:01:27] 3Wikimedia-Logstash, MediaWiki-Core-Team: Get HHVM logs into logstash - https://phabricator.wikimedia.org/T76119#790361 (10bd808) [00:06:39] 3MediaWiki-Core-Team: Low hanging fruit performance tweaks - https://phabricator.wikimedia.org/T76677 (10aaron) 3NEW p:3Triage a:3aaron [00:06:54] 3MediaWiki-Core-Team: Low hanging fruit performance tweaks - https://phabricator.wikimedia.org/T76677#808866 (10aaron) https://gerrit.wikimedia.org/r/177433 [00:08:03] legoktm: Indeed. That API [00:08:45] 3Wikimedia-Logstash, MediaWiki-Core-Team: Get HHVM logs into logstash - https://phabricator.wikimedia.org/T76119#808878 (10bd808) [00:09:55] legoktm: Hm.. I didn't know 'change' took multiple key/value pairs already, I thought you were speaking hyptothetically [00:10:02] legoktm: But indeed, the pipe issue [00:10:10] legoktm: Maybe those can be url escaped? [00:11:52] 3MediaWiki-Core-Team, Beta-Cluster: no log in deployment-bastion:/data/project/logs from "503 server unavailable" on beta labs - https://phabricator.wikimedia.org/T74275#760174 (10bd808) [00:12:59] * legoktm has no idea [00:20:09] 86.13% 325.384 1 - WikiPage::pageData [00:20:10] 86.12% 325.362 1 - DatabaseBase::selectRow [00:20:25] bd808: either the prof is wonky or something is seriously bad [00:21:53] I wouldn't rule either out without more investigation [00:22:07] Do the rest of the numbers make sense? [00:25:24] bd808: the slow stuff tends to have a few calls, often times 1, and the timings make sense (e.g. slightly slower for the next one, and since you know there is 1 call of each and know the code, you know it's stack depth) [00:25:36] the total also matches up to reality afaik [00:26:25] I'm not 99.99% sure of the data, but it matches up from what I can tell [00:27:04] So there's likely a really bad select then I guess [00:27:53] 3Release-Engineering, MediaWiki-Core-Team: Make ::mediawiki::syslog and ::mediawiki::php logging destinations configurable via hiera - https://phabricator.wikimedia.org/T1295#808937 (10bd808) [00:28:10] 3MediaWiki-Core-Team, Beta-Cluster: no log in deployment-bastion:/data/project/logs from "503 server unavailable" on beta labs - https://phabricator.wikimedia.org/T74275#808939 (10bd808) [00:28:11] 3Release-Engineering, MediaWiki-Core-Team: Make ::mediawiki::syslog and ::mediawiki::php logging destinations configurable via hiera - https://phabricator.wikimedia.org/T1295#808938 (10bd808) 5Open>3Resolved [00:33:47] well the thing is, the select is a old, simple, common, and harmless looking query [00:34:29] TimStarling: right now it looks like the slowest thing is the filemtime() call on the extension.json file to see if it has been updated, but it does each one individually. Do you have any ideas/suggestions on other ways to handle cache invalidation? [00:37:06] legoktm: could you do something like hetdeploy does and only stat one file (localsettings?) instead of lots of files? Then an update requires touching one thing but lots less io calls. [00:38:21] yeah, maybe stat calls could be switched off in configuration [00:38:33] if ( @filemtime( $filename ) >= filemtime( "$wmfConfigDir/InitialiseSettings.php" ) ) { [00:38:33] I don't think you would want to switch them off for default MW installs [00:39:17] HHVM has a stat cache which may make things faster if it is configured correctly [00:39:48] another possibility is to only do stat checks say once per second [00:40:02] i.e. a stat cache in APC with a very short expiry [00:40:38] but I'm not sure if it is worth the effort if HHVM's stat cache is fast enough [00:40:47] I'm using filemtime right now, let me try switching to stat and see if that makes a difference on hhvm [00:41:29] oh, no [00:41:35] filemtime() uses the stat cache [00:41:37] oh [00:41:45] but the stat cache is disabled by default, and it's disabled in production at the moment [00:42:03] should be Server.StatCache in the HDF configuration [00:42:57] maybe hhvm.server.stat_cache in the .ini configuration [00:43:16] yeah that's probably the new flag [00:43:29] interesting that it's disabled by default [00:43:38] in zend it is the opposite [00:43:54] legoktm: You can set that in mw-vagrant by adding it to the hierdata/common.yaml file [00:43:58] lots of FSFileBackend code had to integrate/fight with it [00:44:08] oh, I was about to edit /etc/hhvm/fcgi.ini [00:44:29] * legoktm will try hiera [00:44:30] You can do that too, but the next time you run puppet it will get wiped out [00:44:55] bd808: the profiling almost suggests hooks are to blame...but kind of dead ends [00:46:04] AaronS: I wonder if xhprof_sample_enable() traces would help figure out the code paths? [00:46:49] That mode just grabs the call stack every N seconds [00:47:27] so you get output of 1 line per sample with the active stack and a timestamp [00:47:42] hmm, it's about the same [00:48:29] legoktm: oh... mw-vagrant is going to be slow to do stats [00:48:49] it's using either nfs or a bastardized smb sharing [00:49:05] * bd808 just saw a lightbulb there [00:49:26] yeah, true [00:49:54] maybe the best idea is to just ignore stat overhead for now, and concentrate on other parts of the profile [00:50:02] ok [00:53:08] or just comment out the filemtime() calls and benchmark it like that [00:53:53] that's what I've done for now :P [00:56:00] <^d> bd808: Moved xhprof config from todo -> done on etherpad. [00:56:15] ^d: Sweet [00:57:37] <^d> I'm going to clean up Cirrus' profiling. [00:59:20] I saw something cirrus related in the hhvm logs ... [00:59:46] Warning: Search backend error during degraded_full_text search for '　　' after 35. Parse error on '　　': Encountered "" at line 1, column 2. [Called from CirrusSearch\ElasticsearchIntermediary::failure in /srv/mediawiki/php-1.25wmf10/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php at line 98] in /srv/mediawiki/php-1.25wmf10/includes/debug/MWDebug.php on line 300 [01:03:57] <^d> That's known. [01:04:08] <^d> It shouldn't be possible to search for an empty string like that. [01:04:19] <^d> I think Nik was trying to run it down last week. [01:07:19] <^d> bd808: Hehe, removing profiling made 2 of my classes completely MedaiWiki-free :p [01:07:31] woot [01:07:46] <^d> Oh wfDebugLog() [01:07:48] <^d> How did you sneak in there? [01:08:30] We (I?) need to think about how to start using the logger interface directly. I keep getting sidetracked on that [01:26:50] 3Scrum-of-Scrums, MediaWiki-extensions-WikibaseClient, MediaWiki-Core-Team, Wikidata: avoid fetching SiteList object from memcached - https://phabricator.wikimedia.org/T58602#809008 (10hoo) [01:42:11] AaronS: heh. I did something bad somewhere -- "message repeated 1202 times: [ Warning: Division by zero in /srv/mediawiki/php-1.25wmf11/includes/libs/Xhprof.php on line 277]" [01:54:23] 100 * $total / $main[$name] [02:05:05] MaxSem: https://gerrit.wikimedia.org/r/#/c/177464/ [02:05:13] 3MediaWiki-Core-Team: Document OrientDB research - https://phabricator.wikimedia.org/T76683#809038 (10aaron) p:5Triage>3High [02:05:15] 3MediaWiki-Core-Team: Document OrientDB research - https://phabricator.wikimedia.org/T76683#809021 (10aaron) p:5High>3Normal [03:30:07] <^d> AaronS: Do you think we'd ever write a stub profiler that's not ProfilerStub? [04:08:00] ^d: no [04:10:46] <^d> I didn't think so. I considered getting rid of isStub() as an implementation and just checking for instanceof ProfilerStub [04:11:21] <^d> I went for https://gerrit.wikimedia.org/r/#/c/177488/ instead. [04:13:19] ori: so if preview stashes edits, it only works for non-section edits [04:13:37] TimStarling: turns out xhprof gives garbage data [04:13:52] yeah? [04:13:53] same problem our profiler had before the cputime method was fixed to use the THREAD flag [04:14:01] it uses SELF atm [04:14:17] ahhh [04:14:43] looks like a search&replace fix [04:15:12] ori: so if preview stashes edits, it only works for non-section edits -- that's fine for now [04:16:19] have you done a facebook CLA? [04:16:53] AaronS: I'm not sure that we were right [04:17:37] the call to getrusage is preceded by this block: [04:17:40] <^d> AaronS: Also, ProfilerStub didn't need updating, it already returned true. [04:17:56] static inline uint64_t cpuTime(int64_t MHz) { [04:17:56] #ifdef CLOCK_THREAD_CPUTIME_ID [04:17:56] int64_t rval = Vdso::ClockGetTimeNS(CLOCK_THREAD_CPUTIME_ID); [04:17:58] if (rval >= 0) { [04:17:59] ahh, right, it was abstract [04:18:00] return rval; [04:18:02] } [04:18:04] #endif [04:18:13] CLOCK_THREAD_CPUTIME_ID should be defined [04:18:28] so when is Vdso::ClockGetTimeNS(CLOCK_THREAD_CPUTIME_ID) negative? [04:18:37] probably when it fails [04:19:09] argh [04:19:23] * TimStarling facepalm [04:19:54] ConfirmEdit needs to check if links are added to decide if it needs to show a captcha [04:20:54] it does that by calling $wgParser->parse() on the whole wikitext [04:21:32] instead of sharing a parse operation with SpamBlacklist, AbuseFilter etc. [04:21:51] <^d> Getting rid of that would probably cut save times a lot. [04:22:07] ori: is CLOCK_THREAD_CPUTIME_ID defined? [04:22:58] no kidding [04:23:16] it should be. which hhvm server are you curling? [04:23:21] i can strace getrusage calls [04:24:13] * AaronS isn't curling atm [04:28:48] ok, so this visualeditor API call I am looking at [04:29:05] it is not parsing the wikitext twice, as I had suspected [04:29:13] it is definitely parsing the entire page 4 f***ing times [04:29:55] each time taking about 16 seconds [04:31:17] and Roan said yesterday when I asked him about this action words to the effect of "Gabriel looked at this a while back and I think it was mostly network latency" [04:31:50] maybe if you're in mars orbit or something [04:34:37] remember: it's not slower -- it's doing more! [04:36:48] in a way, that is true [04:36:54] ;) [04:37:47] AaronS: getrusage _is_ getting called [04:39:36] but not, it appears, from runtime/ext/ext_hotprofiler.cpp:257 [04:40:17] it's luasandbox stuff, which uses RUSAGE_THREAD, as it should [04:42:28] ori: that was with forceprofiler=true? [04:42:42] I guess we are at square one then [04:43:00] yeah, it's using the vdso clockGetTimeNS thing [04:44:20] the fallback should probably still be getrusage w/RUSAGE_THREAD rather than RUSAGE_SELF so that may still be a bug but not one that is affecting us [04:57:11] someone broke ProfilerStandard [04:59:06] TimStarling: https://i.imgur.com/t0XHtgJ.gif :-) [05:05:58] Fiona: exactly [05:06:20] I am trying not to look at these strange RL messages that appear in the debug log next to the profiler output [05:07:42] strange RL messages? [05:08:33] TimStarling: can ParserOption::legacyOptions be removed now? [05:11:35] the fact that it is called from ParserOptions::legacyOptions() is a bit weird [05:11:57] maybe someone didn't understand what an etag is and just decided to leave it alone [05:12:06] "Used to get the old parser cache entries when available" [05:12:27] yes, it can be removed [05:12:38] just saying, you will have to work out what to do about getETag() [05:13:01] make it work, I guess, since it is probably broken by its use of legacyOptions() [05:18:17] ori: awight: I never understood why we should have a Special:RecordImpression anyway. If you trust the request logs, isn't it enough to have the request logs for the resource that you're interested in? And if you don't trust the request logs, wouldn't you need a Special:RecordRecordImpressionImpression, ad infinitum? [05:18:17] awight: ori: re Special:RecordImpression, the issue is that we need a record of *not* displaying a banner, as well. [05:19:06] but now I see in my debug console: [05:19:07] Failed to load resource: http://meta.wikimedia.org/wiki/Special:RecordImpression?country=US&uselang=…ket=2&bucketStart=1417514400&bucketEnd=1425290400&reason=close&result=hide net::ERR_BLOCKED_BY_CLIENT [05:19:09] and sure enough: [05:19:14] https://easylist-downloads.adblockplus.org/easyprivacy.txt [05:19:26] ||wikimedia.org/wiki/Special:RecordImpression? [07:08:40] <_joe_> ori: we should probably tell the adblockplus people [07:09:14] <_joe_> (I guess you're off by now) [07:09:47] _joe_: tell them what? [07:10:03] (morning) [07:10:23] <_joe_> that including wikimedia.org/wiki/Special:RecordImpression? is not that clever [07:13:21] <_joe_> ori: can I ask you something about VE? I know it uses API for rendering the page while you edit it; I never took the time to look if it uses API to save the page as well [07:13:43] _joe_: it does, yes [07:14:25] <_joe_> I got a very unpolite private email from a community member after my post to wikitech-l, with a bunch of questions and a couple of accusations, that I will disregard - but I wanted to answer correctly to the questions :) [07:18:50] <_joe_> ori: we should update https://www.mediawiki.org/wiki/HHVM, it's pretty outdated [07:19:28] <_joe_> I'll do that today [07:56:45] _joe_: i mean, the whole HHVM project only cut page load/save times by a factor of 2 or so [07:56:52] so we're bad people and we deserve to die [07:58:45] but as MZMcBride once said (): "Wikimedians complain loudly and congratulate softly." [07:59:59] <_joe_> ori: no, I was accused of mudding the waters with "contradicting information" [08:00:15] <_joe_> but I really can't care less, I answered politely [08:01:01] btw, unrelated: should shave off 4k from every cached page [08:02:27] <_joe_> whoa [08:02:48] also, https://twitter.com/fbOpenSource/status/540195516252831745 [08:02:50] legoktm: wow that was fast, thanks! [08:03:39] _joe_: there was something else i wanted to ask you.. [08:04:25] the fact that mw1017 has its own backend director is extremely useful for debugging, because it means you can live-hack / debug on that machine and see the impact immediately on testwiki [08:04:33] but that's just testwiki, which is really limiting [08:05:13] it would be great to re-use the separate pools VCL config to have one app server in its own pool, accessible via a cookie but serving all domains rather than just testwiki [08:05:25] <_joe_> yeah it would be of value [08:05:44] <_joe_> but I'd like to do that in the least-invasive-way possible [08:05:57] what do you have in mind? [08:06:02] <_joe_> also, we need not to cache any page served there [08:06:13] right [08:06:34] Nikerabbit: thanks for pointing that out, I had totally forgotten about LU :P [08:06:42] <_joe_> ori: nothing specific right now, or to be precise, a couple of ideas but I have to think about that [08:06:50] it could even be mw1017, it's very underutilized [08:07:04] having cookie-based routing rather than just hostname based routing to that backend would be ideal [08:07:47] <_joe_> ori: I'd like to introduce the concept of the "canary pool" where we can do canary releasing [08:07:58] yeah, that would be great [08:08:04] <_joe_> and of course we'd need a way to route requests there explicitly [08:08:07] +1 [08:08:41] <_joe_> yeah that's something I have to discuss with other ops, I'd like to have a framework to do that not only for mediawiki appservers [08:08:50] by the way, i know i congratulated you yesterday, but i'm still in shock that you guys went to 100% so fast [08:09:01] <_joe_> yeah well [08:09:10] <_joe_> Yuvi and alex really did speed that up [08:09:26] <_joe_> and I have to say, we automated the reimaging pretty well [08:09:45] <_joe_> it was basically matter of launching a script on palladium [08:09:45] yeah godog was complimenting your reimaging script [08:09:54] oh, i did notice one thing [08:10:22] <_joe_> ? [08:10:28] i think some of the newer installs have a newer kernel installed than the one they booted with [08:10:34] which means perf doesn't work [08:10:48] so they need a reboot [08:10:56] not urgent, just nice to have [08:13:07] <_joe_> oh yeah [08:13:07] <_joe_> I also have to dist-upgrade some of them [08:13:07] <_joe_> and a couple where we missed hyperthreading [08:13:07] <_joe_> but I want to stay away from servers today as much as possible :P [08:13:07] mw1224 is the one i ran into, i was wondering why perf was saying '' for everything [08:13:07] yeah i totally understand :) [08:13:07] <_joe_> :/ [08:13:07] <_joe_> btw, not now, but I'd like your review on https://gerrit.wikimedia.org/r/#/c/176334/ [08:13:16] <_joe_> it's ruby I know [08:13:18] oh sure, looks interesting [08:13:26] well, it's puppet / hiera, it has to be [08:13:50] <_joe_> I had to read more of the puppet source that is allowed by international war laws, probably [08:15:53] heheh [08:16:09] it is awful [08:16:14] puppet, not your patch [08:16:19] <_joe_> I figured you had to do the same for require_package [08:17:06] oh, i had to resort to a depth-first search for the solution [08:17:12] <_joe_> ori: every time I read "klass" I feel sad [08:17:26] at each step, see which methods i can call, and look for one that plausibly gets me closer to where i want to be [08:17:39] <_joe_> eheh [08:17:43] i kept doing that until i got stuck and then i backtracked [08:17:54] <_joe_> so funny to do [08:18:13] and then at the end i tried to make it shorter by identifying shortcuts, replacing A -> B -> C with A -> C where possible [08:18:21] so the result looks very much like i knew what i was doing [08:18:26] but that's very much not the case [08:18:37] <_joe_> but don't fear, they've just rewrote the whole thing in clojure, and jruby [08:18:46] you're not serious, are you? [08:18:50] <_joe_> I am [08:18:54] ... [08:19:06] <_joe_> http://puppetlabs.com/blog/puppet-server-bringing-soa-to-a-puppet-master-near-you [08:19:33] <_joe_> when rubysts discover the JVM and the term SOA, is when you need to run [08:19:48] clojure is not bad (it's a little sluggish and heavy) but it has even fewer breaks than ruby [08:19:56] they're going to make a horrible mess of it [08:20:00] <_joe_> "hanks to the excellent JRuby project, we’re able to run portions of the existing Puppet Ruby code from inside of our Clojure application" [08:20:06] oh no [08:20:08] please no [08:20:08] <_joe_> SERVED [08:20:19] <_joe_> worst of both worlds [08:20:24] <_joe_> the JVM and Ruby [08:21:10] well, plus [08:21:51] the JVM was designed for Java; it's not a generic language runtime like CLR [08:22:01] <_joe_> also, did I mention puppetdb is a pile of crap? I did try to use it at WORK~1 [08:22:07] so Ruby is bolted on awkwardly on top, and Clojure is bolted on awkwardly on top [08:22:13] <_joe_> well, it was a pile at least [08:22:13] and the JVM is bolted on awkwardly on top of Linux [08:22:45] i've never used it (puppetdb) [08:22:46] <_joe_> I always say that being a JVM sysadmin is a different job that being a linux sysadmin [08:22:50] bd808 hated it too iirc [08:22:58] <_joe_> ehej [08:23:04] <_joe_> maybe version 2 is better [08:23:39] so with this new platform [08:24:21] you have one language (puppet) getting converted into another language (ruby) getting evaluated by another language (clojure) which gets translated to another language (java) which gets translated into cross-platform bytecode (jvm) which gets translated into native byte code? [08:24:44] <_joe_> no [08:24:58] <_joe_> clojure gets translated directly to JVM bytecode AFAIR [08:25:07] oh right, fair point [08:25:41] <_joe_> so you have one language (puppet) that gets partly interpreted by ruby, partly by clojure, both of which are translated to JVM bytecode, and must interact with each other [08:25:45] but the clojure interpreter itself is written in java [08:25:46] <_joe_> if I got this right [08:25:54] <_joe_> ori: yes, true [08:25:58] <_joe_> and jruby as well [08:26:06] toss in .erb with more in-line ruby for fun [08:26:12] <_joe_> ahah [08:26:15] <_joe_> they are nuts [08:27:19] <_joe_> when I see such choices, I always feel like "whoa, it's so obviously bad, can a whole org become so clueless?" [08:27:35] <_joe_> and I fear we could go down the same rabbit hole of insanity [08:36:45] 09.13 < _joe_> I had to read more of the puppet source that is allowed by international war laws, probably [08:36:53] good thing USA didn't sign Hague conventions then ;) [09:21:33] _joe_: still around? [09:22:07] <_joe_> ori: of course [09:22:14] <_joe_> just deep into puppet testing :/ [09:22:52] when the servers were upgraded, the utilization of most system resources dropped sharply, with the exception of system cpu % [09:23:14] graphs such as this one are troubling: https://ganglia.wikimedia.org/latest/graph.php?r=week&z=large&c=Application+servers+eqiad&h=mw1081.eqiad.wmnet&jr=&js=&v=5.0&m=cpu_system&vl=%25&ti=CPU+System [09:24:11] i just looked at other metrics in graphite trying to find one that has undergone a similar regression [09:24:14] https://graphite.wikimedia.org/render/?width=586&height=308&_salt=1417684856.733&from=-1weeks&target=removeAbovePercentile(servers.mw1081.iostat.sda1.reads_byte.value%2C90) [09:24:22] some indication that we're performing a lot more disk reads [09:24:45] <_joe_> ori: that's expected as well, until we move to repoauth mode [09:24:58] <_joe_> disk is still not nearly a bottleneck anyway [09:25:24] <_joe_> (the system cpu time is kinda expected, from what I got from fb engineers) [09:25:24] yeah, if this is what it boils down to it's ok [09:25:52] <_joe_> ori: I remember one debugging session with bblack, we saw that hhvm stats() every file for every request it gets [09:27:20] <_joe_> I'm not sure about what apc did [09:28:16] <_joe_> (or, I'm sure it does the same, I just don't remember if there is some sort of internal cache [09:28:35] <_joe_> the good thing is, stat() is cached at the kernel level if you're not on NFS [09:58:16] _joe_: yes, it's newlstat, it looks like [11:40:03] 3MediaWiki-Core-Team: Track down and resolve cause for high system CPU% on HHVM API servers - https://phabricator.wikimedia.org/T75706#817290 (10ori) [14:52:11] 3MediaWiki-Core-Team: Track down and resolve cause for high system CPU% on HHVM API servers - https://phabricator.wikimedia.org/T75706#818854 (10MZMcBride) [15:06:20] 3CirrusSearch, MediaWiki-Core-Team: PHP Warnings when "startingOver" an index - https://phabricator.wikimedia.org/T75457#818879 (10Manybubbles) [15:07:07] <^d> manybubbles: Gahh, I thought we had gotten rid of it. [15:07:12] <^d> Oh yeah, that was --closeOk. [15:12:28] 3CirrusSearch, MediaWiki-Core-Team: PHP Warnings when "startingOver" an index - https://phabricator.wikimedia.org/T75457#818892 (10MZMcBride) [15:14:39] ^d, manybubbles: we are currently using elastica 1.3.0.0... I tried to update to 1.3.4.0 on twn but they added some ugly deprecation notices for some methods Translate uses, with no obvious way to detect which version is in use. Of course the replacements dont work on 1.3.0.0. So I'm just wondering how should I handle the situation. [15:15:12] ^d: yeah! [15:17:02] Nikerabbit: hmmmmmm - thats pretty mean. I wonder if there is an intermediate version we can get to. or maybe we can file an issue upstream..... [15:17:43] <^d> He's always been super eager to help :) [15:18:41] 3CirrusSearch, MediaWiki-Core-Team: PHP Warnings when "startingOver" an index - https://phabricator.wikimedia.org/T75457#818895 (10SmartK) [15:19:53] <^d> Nikerabbit: What got deprecated? [15:20:40] https://github.com/ruflin/Elastica/releases setFilter was deprecated in favour of setPostFilter, and setPostFilter changes the type of the first parameter at the same time [15:21:13] <^d> I see it [15:22:10] so I guess I could still pass array to it, without warnings for now [15:23:08] <^d> Passing an array to setPostFilter() also triggers the warning afaict. [15:23:32] could be, though I don't see it in https://github.com/ruflin/Elastica/pull/670/files [15:23:55] <^d> https://github.com/ruflin/Elastica/blob/master/lib/Elastica/Query.php#L422 [15:24:12] <^d> Deprecated: Elastica\Query::setPostFilter() passing filter as array is deprecated. Pass instance of AbstractFilter instead. #669 [15:24:24] right, it must have been added in some other commit [15:24:49] hmm - we probably shouldn't be using the post filtering though [15:24:49] so there is no obvious $elastica_version or method_exists I could use [15:25:16] manybubbles: well, I'm happy to replace it with something else as well :) [15:25:43] Nikerabbit: I believe a filtered query is generally better [15:26:00] post filtering is actually for a different (more obscure) use case [15:26:05] which is why they renamed it [15:26:52] manybubbles: okay... this is in the Special:SearchTranslations code I think [15:27:18] <^d> Nikerabbit: We could provide a method in the MW wrapper for elastica to get a version. [15:27:22] <^d> Info's in the composer.json [15:27:29] it uses boolfilter for filtering by language or group [15:27:44] ^d: currently I don't have direct dependency on Elastica extension though [15:28:00] <^d> Hmm ok [15:28:26] ^d: I would be more okay with it if it didn't yet another step now to install elastica library via composer ;) [15:29:28] <^d> I'm not a fan of it either, but we appear to have lost that battle :p [15:30:11] manybubbles: I'll see if I can easily get rid of setFilter [15:31:48] <^d> If we're going to use composer we should just pull Elastica into its various use places directly. [15:31:53] <^d> And get rid of the Elastica wrapper ext. [15:33:30] manybubbles: hmm I think I'm using filters because I also have facets... is that the obscure use case or is there alternative to that? [15:33:39] yes [15:33:43] facets is the use case [15:33:59] post filtering doesn't apply to faceting, iir [15:34:00] iirc [15:34:22] manybubbles: so then I can't get rid of them... except that facets are deprecated! [15:35:03] Nikerabbit: in favor of aggregations which is like a more general form of facets - but they both require post filtering if you want the filter not to apply to the aggregated data. [15:35:12] so - upshot is you have a good reason to use post filtering [15:35:58] hehe [15:39:03] * Nemo_bis not sure if this makes the situation better or worse [15:39:14] one alternative is of course for me to port over to 1.3.4.0 and update WMF and TWN at the same time [15:39:34] but not my favourite approach [15:45:10] 3CirrusSearch, MediaWiki-Core-Team: PHP Warnings when "startingOver" an index - https://phabricator.wikimedia.org/T75457#818968 (10SmartK) Just a short info with MediaWiki 1.24.0. When I use "php updateSearchIndexConfig.php --reindexAndRemoveOk --indexIdentifier now" as Chad suggested the same PHP Warning pops u... [15:46:26] <^d> Nikerabbit: I just tried 1.3.4.0 in my local dev install and was able to reindex fine which is a good exercise for the code usually. [15:56:03] ^d: yeah I don't doubt that [15:56:29] ^d: I'll file a bug about Translate so that I don't forget to resolve this issue somehow [15:56:34] <^d> Mmk. [15:57:43] manybubbles: https://wikitech.wikimedia.org/wiki/Search#Rolling_restarts Are those settings needed to be set every time? Or was it just for one upgrade? [15:58:35] Reedy: you mean cluster.routing.allocation.enable": "primaries" ? [15:58:55] that needs to be set to something while a node is down and then reset back to "all" when the node is back up [15:59:00] oh, duh [15:59:07] the script does that. actually, ^d has a nicer version [15:59:07] I should've really read it fully :) [15:59:20] ^d: gief :D [15:59:37] ^d: fix teh docs for your tool plz? [16:01:18] <^d> es-tool [16:01:29] <^d> on every elasticsearch node [16:01:31] <^d> has --help [16:02:07] <^d> you want `sudo es-tool restart-fast` [16:02:14] <^d> The output's a little buggy on it, but that's life. [16:02:35] Can we update manybubbles' rolling restart script? [16:02:54] <^d> What we *should* do is just replace the curl bits. [16:03:27] WFM [16:05:42] <^d> https://wikitech.wikimedia.org/wiki/Search#Restarting_a_node [16:05:44] <^d> Updated. [16:11:03] 3MediaWiki-Core-Team: Clean up ApiResult - https://phabricator.wikimedia.org/T76728 (10Anomie) 3NEW p:3Normal a:3Anomie [16:13:04] just updated the rolling restart script similarly [16:13:04] https://wikitech.wikimedia.org/w/index.php?title=Search&diff=136911&oldid=136910 [16:15:05] damn it, no es-tool on logstash boxes [16:15:34] ah, trusty only [16:15:55] bd808: ^^ another reason to get the boxes replaced/upgraded/reinstalled :P [16:16:34] +1 [16:16:53] replaced would be my preference I think [16:16:57] Did we decide if we need new hardware? [16:16:58] yeah... [16:17:49] Gage was going to bring it up at the ops meeting and mark said something on irc like "great idea. not this quarter" [16:17:49] I should probably open a procurement ticket then [16:17:58] ah, right [16:18:49] I wonder if we should just get them upgraded, rather than reinstalled [16:19:29] though, with the data on seperate drives now, a reinstall wouldn't be so much of an issue [16:19:43] <^d> wtf bouncer? [16:22:24] "cluster_name" : "production-logstash-eqiad", [16:22:24] "status" : "red", [16:22:35] that's not good (logstash1003) [16:22:35] bd808: ^^ [16:22:39] <^d> What's _cat/health say? [16:22:40] Reedy: everything is mirrored on all three nodes too, so as long as we keep one alive we are fine [16:22:51] <^d> curl localhost:9200/_cat/health?v is your friend [16:22:59] * bd808 shells in [16:23:32] <^d> Reedy: cat apis are your friend for status checks. http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cat.html [16:23:53] reedy@logstash1003:~$ curl -s localhost:9200/_cat/health?v [16:23:53] epoch timestamp cluster status node.total node.data shards pri relo init unassign [16:23:53] 1417710227 16:23:47 production-logstash-eqiad red 3 3 119 40 0 0 4 [16:24:09] <^d> It's the 4 unassigned, must be primaries. [16:25:24] <^d> Hmm [16:25:29] Corrupted index [corrupted_f1vhJijhRvmkw4dyyz36zw] in a log file [16:26:33] The index for logstash-2014.11.30 is awol [16:27:19] I'll kick logstash1001 and see if things get better or not [16:29:12] failed to start shard [16:29:12] org.apache.lucene.index.CorruptIndexException: [logstash-2014.11.26][0] Corrupted index [corrupted_f1vhJijhRvmkw4dyyz36zw] caused by: CorruptIndexException[codec footer mismatch: actual footer=962802488 vs expected footer=-1071082520 (resource: NIOFSIndexInput(path="/var/lib/elasticsearch/production-logstash-eqiad/nodes/0/indices/logstash-2014.11.26/0/index/_8nq.fdt"))] [16:29:28] So that shard is toast on 1001 [16:36:55] AaronSchulz: It might be worth backporting that xhprof_frame_end() fix. Lots of noise in logstash from it. [16:40:43] Reedy: https://gerrit.wikimedia.org/r/#/c/177560/ will stop a lot of noise in logstash [16:41:12] All the "Warning: xhprof_frame_end() expects exactly 0 parameters, 1 given in /srv/mediawiki/php-1.25wmf10/includes/libs/ScopedCallback.php on line 70" [16:41:46] Is it still swat window? [16:42:55] apparently yes [16:43:08] for another 18m [16:43:30] * Reedy merges [16:43:38] excellent [16:44:08] At lot of these noisy indexes and such stuff might be reasonable GCI tasks [16:44:40] yeah they might. most should be dead easy to fix [16:48:28] 3MediaWiki-Core-Team: HHVM log noise probably will be good GCI tasks - https://phabricator.wikimedia.org/T76733 (10Reedy) 3NEW p:3Triage [16:54:01] 3MediaWiki-API, MediaWiki-Core-Team: Clean up ApiResult - https://phabricator.wikimedia.org/T76728#819078 (10Reedy) [16:55:54] <^d> bd808: Thought on xhprof for vagrant. [16:56:04] <^d> What if we git::installed the sources. That's all we need for the php bits. [16:56:18] <^d> We can still get the pecl part from php5-xhprof. [16:56:28] <^d> That'd let us remove the unreliable execs [16:58:08] +1. There are a couple of "unofficial" composer packages for the ui too but just cloning is probably best [16:58:46] Evan had an epic rant about packagist on a bug and decided not to play with them. :/ [17:01:29] https://github.com/phacility/xhprof/pull/40 [17:04:27] anyone else having trouble with hphp complaining on fork on command line stuff? One of the cirrus maintenance tasks forks and I get broken pipe errors. everything runs fine, but I get broken pipe errors..... [17:06:35] 3MediaWiki-extensions-Scribunto, MediaWiki-Core-Team: An exception is thrown if an argument value passed to frame:expandTemplate has an invalid type - https://phabricator.wikimedia.org/T76609#819097 (10Anomie) [17:07:00] 3MediaWiki-extensions-Scribunto, MediaWiki-Core-Team: An exception is thrown if an argument value passed to frame:expandTemplate has an invalid type - https://phabricator.wikimedia.org/T76609#819101 (10Anomie) [17:07:40] 3MediaWiki-extensions-Scribunto, MediaWiki-Core-Team: An exception is thrown if an argument value passed to frame:expandTemplate has an invalid type - https://phabricator.wikimedia.org/T76609#807008 (10Anomie) The argument validation needs to be there, but the fact that the code on frwikibooks is passing invalid... [17:08:14] Curious read, bd808 [17:09:11] He has many good points, but also misses the central point of "this is *the* php package manager" [17:14:37] <^d> I don't mind composer. [17:14:45] <^d> I dislike packagist for similar reasons he outlined. [17:16:12] It's not any different in any other script registry that I know of. That's not an excuse but it's a common problem. [17:17:02] I've got packages registered with pip, npm, bower and none of them required "proof" of anything. Just a url to pull from. [17:17:06] well, on pypi you can gpg sign releases [17:17:19] you can gpg sign tags in git [17:17:40] I'm pretty sure you have to upload a tarball to pypi, if you just point it at git repo it won't work by default [17:17:57] Yeah, you're right about that [17:18:10] pecl/pear works with an upload model too [17:18:19] <^d> maven also requires signing. [17:18:22] and they are a pain to publish too [17:18:31] Also, you can now sign git pushes [17:18:53] <^d> central, at least. [17:18:54] https://developer.atlassian.com/blog/2014/12/git-2-2-0-released/ [17:19:30] legoktm: Awesome. We'll have that on the cluster in 2016! :/ [17:19:37] <^d> bd808: his point about naming is pretty valid though I think. [17:19:52] <^d> I mean, someone can do wikimedia/officialmediawikirelease on packagist :p [17:20:06] mediawiki/core on packagist points to a not-core repo [17:20:27] https://packagist.org/packages/mediawiki/core [17:20:34] Yeah because the name is just a name [17:20:47] There are not "organizations" in packagist [17:21:03] just 2 part naming to help avoid conflicts [17:21:07] <^d> Then they should be flat. [17:21:26] <^d> Because they are effectively flat then if the depth makes no difference beyond presentation. [17:21:34] They are really. It's just your eye that makes it seem like a hierarchy [17:22:26] or... I'm lying. Dman [17:22:39] Packages from mediawiki [17:22:42] that's not right [17:22:49] <^d> They're not really flat, but anyone can put anything in any bucket. [17:23:50] This package is very useless -- https://packagist.org/packages/mediawiki/core [17:24:12] "Meta package for MediaWiki to facilitate specifying required MediaWiki versions" but it has no versions? [17:24:19] <^d> duh [17:24:19] <^d> Anyway, I guess you trust the maintainer, not the packagist listing. [17:24:43] Ultimately you are trusting the backing git repo [17:24:58] and you should totally be pinning versions IMO [17:24:59] <^d> But like evan said, what if I change what repo packagist points to. [17:25:36] <^d> So unless you pin sha1s...it's not really doable. [17:26:56] <^d> Anybody can make a 1.5 branch ;-) [17:27:37] <^d> bd808: Anyway, it's all very grey. He probably shouldn't be throwing his hands up and saying "GAH NO COMPOSER" but he's got a lot of really valid/interesting points. [17:27:50] agreed [17:32:00] 3MediaWiki-Core-Team: Stash page previews into the edit stash - https://phabricator.wikimedia.org/T76737#819120 (10aaron) [17:58:30] 3Librarization, MediaWiki-Core-Team: Make MWException handle non-MW exceptions better - https://phabricator.wikimedia.org/T76652#819154 (10aaron) [17:59:49] mh google bot seems to have hit some pages at the wrong time and the snippet shows fundraising text, known already? http://esaurito.net/~godog/sshot/screenshot_6JSD2d.png [17:59:56] ori: wgBackendResponseTime agrees with the profiling on http://en.wikipedia.org/wiki/Death_of_Eric_Garner?forceprofile=true ; also it's all real time anyway (even if there were a cpu time bug, which there probably isn't) [18:00:13] * AaronS is still wondering what is going on...seems to be hooks there [18:01:22] 3MediaWiki-Core-Team: Stash page previews into the edit stash - https://phabricator.wikimedia.org/T76737#819158 (10aaron) https://gerrit.wikimedia.org/r/177514 [18:05:35] 3MediaWiki-Core-Team: Stash page previews into the edit stash - https://phabricator.wikimedia.org/T76737 (10RobLa-WMF) [18:10:54] godog: i saw that too. can you e-mail ops / fr-tech? [18:11:20] iirc there are ways to annotate content to tell google something should not be included in the search summary [18:11:27] 3MediaWiki-Core-Team: HHVM log noise probably will be good GCI tasks - https://phabricator.wikimedia.org/T76733#819176 (10Aklapper) [18:11:37] this has the potential of making our traffic tank [18:12:33] godog: oh good god [18:13:01] We should make all google snippets display the fundraising banner [18:13:01] greg-g: ditto, can you help escalate? i gotta get off the computer for a bit [18:13:12] ori: yes [18:13:15] ori greg-g sure, mailing now [18:13:24] thanks [18:15:05] legoktm: I just had a big realization about dropping php5.3 support in core. Besides the main cluster (almost done w00t!) we have wikitech and fundraising and possibly other MW boxes that will drag us down. :( [18:15:21] ://// [18:15:38] fundraising is on old stuff anyway isn't it? [18:15:39] is fundraising not switching over to HHVM? [18:15:54] they're on 1.23 I believe [18:16:01] Fundraising is maybe not a problem. As I understand it they lag everything [18:16:09] but wikitech will be a big deal [18:16:18] since it's on the train now [18:16:22] can we upgrade wikitech to trusty? :) [18:16:43] with a full labs outage of unknown duration [18:16:56] virt1000 does all the things [18:17:37] Andrew was grumpy about mw-v not supporting precise as that's how he's been testing things in the past [18:18:09] well it has to happen at some point anyways right? [18:19:36] The next time they reimage. We still had 10.10 boxes in tampa when they shut it down [18:19:48] 10.04 [18:20:01] <^d> Obviously google isn't using our search snippets. [18:20:10] <^d> /Cirrus/ doesn't include fundraising banners. [18:21:38] Would it be better if they did? Should we help them switch something? [18:43:47] ^d: could you review https://gerrit.wikimedia.org/r/#/c/177586/? should be trivial [18:44:10] <^d> bd808: No, they probably shouldn't. I'm being snarky :) [18:44:23] ^d: https://gerrit.wikimedia.org/r/#/c/177481/2 [18:44:51] ^d: I missed the snark bit on the msg packet. :) [18:45:55] I saw so much snark on the ./ thread on the annual fundraising that I my have become temporarily desensitized [18:46:04] thanks [18:46:38] <^d> yw [18:46:51] <^d> AaronS: Gimme a min. [18:50:17] AaronS: https://gerrit.wikimedia.org/r/#/c/177501/ [18:51:48] 8 - DatabaseBase::select@1 [18:52:01] bd808: how is that possible? there is no way any caller can do that afaik [18:57:35] <^d> AaronS: Done. [18:58:00] AaronS: ? which random thing I said today are you responding to? [18:58:44] 3MediaWiki-Core-Team: HHVM log noise probably will be good GCI tasks - https://phabricator.wikimedia.org/T76733#819279 (10fbstj) [18:58:51] AaronS: nothing, just me looking at forceprofile [18:59:02] bd808: you are personally accountable for xhprof's output now [18:59:09] WHY MUST YOU LIE??!?! [18:59:10] there is not recursion in select(), that must be wrong [19:00:42] ah I see it now. Well I'll get out the magic pen that scratches "i must not tell lies" into the back of my hand :) [19:00:44] ^d: https://gerrit.wikimedia.org/r/#/c/176981/ [19:06:11] bd808: I'm thinking about rewriting scopedProfileIn/scopedProfileOut to just be one method that takes a closure (bound first if convenient) [19:06:31] that way the code profiled runs inside the method calling the frame methods [19:07:18] * AaronS grumbles about php5.4 and http://php.net/manual/en/closure.bind.php [19:07:52] one can always use public *Internal() named methods until then [19:08:02] to deal with private/protected stuff [19:08:51] well, I guess it could be any callback, not just a closure [19:12:43] AaronS: Isn't doing that basically just turning the sub-profile section into a method? [19:13:10] well the hook and DB classes need custom names [19:13:28] I won't those to work [19:13:32] *want those [19:14:31] I don't care about the __METHOD__ . '-blah' stuff as much [19:14:57] * AaronS doesn't like those anyway [19:15:59] Have you tired blacklisting the wrapper method? [19:16:03] *tried [19:16:53] does that work? [19:17:07] It should [19:17:18] I haven't tried it with out classes [19:17:37] s/out/our/ [19:18:08] If you blacklist the method then it won't create and subsequently pop the stack and that should fix the problem [19:18:30] bd808: can you make a patch? I can test it [19:19:57] AaronS: It's just config. Add 'exclude' => [list of methods] to the config [19:21:59] So `$wgProfiler['exclude'] = array('ProfilerXhprof::scopedProfileIn');` [19:23:30] yeah, adding it to the constructor [19:26:42] bd808: wouldn't there still be the XhprofFrame::__construct problem? Can that be blacklisted too? [19:27:49] It can be yes, but we aren't using it. That method is the analog to our ProfilerXhprof::scopedProfileIn() call [19:28:05] Are you seeing that method in the traces? [19:28:37] Basically anything you see in the trace that is noise you can block with that exclude array [19:29:28] The c code scans that list on each function entry hook and decides if it will start a tracking frame or not [19:29:58] So setting it adds a small amount of overhead to all calls, but pretty small [19:30:15] 1 hash compute, a hash table lookup and possibly a linked list scan [19:30:18] ah, right, we aren't using XhprofFrame [19:30:49] bd808: btw, https://gerrit.wikimedia.org/r/#/c/177449/ [19:33:28] bd808: and if a function exits and is excluded it doesn't do frame cleanup either? [19:33:38] right [19:37:22] * AaronS stares at https://github.com/facebook/hhvm/blob/000d9ee25869e96f74912771e29e1b10e8660ef8/hphp/runtime/ext/ext_hotprofiler.cpp [19:41:31] heh, MemoProfiler is fun [19:43:01] Are they ignoring the config array? -- https://github.com/facebook/hhvm/blob/000d9ee25869e96f74912771e29e1b10e8660ef8/hphp/runtime/ext/ext_hotprofiler.cpp#L1458 [19:46:57] AaronS: I'm not seeing the analog of in the hhvm code but maybe I'm not looking in the right places [19:47:16] yeah, that's why I kept staring at it [19:47:19] I don't see it other [19:47:23] The `hp_ignore_entry(hash_code, symbol)` call there is the skipping magic [19:47:49] and it toggles the profile_curr flag which is checked on exit [19:50:46] bd808: so either we port that or do the 1-method/callback approach [19:57:44] AaronS: They don't check the blacklist I don't think -- https://github.com/facebook/hhvm/blob/92837355616f3a3a4cf15d8aaaf76fe44c6209f1/hphp/runtime/vm/event-hook.cpp#L290-L301 [19:57:50] That seems to be the measurement hook [20:00:02] We should make a repro that shows that they don't match the pecl method and file a bug I think. That's the place to get started. Then I guess it's about finding someone (you, me, hhvm core dev) to implement the matching behavior. [20:28:17] 3MediaWiki-extensions-Scribunto, MediaWiki-Core-Team: An exception is thrown if an argument value passed to frame:expandTemplate has an invalid type - https://phabricator.wikimedia.org/T76609#819415 (10JackPotte) OK, so I'll remove https://fr.wikibooks.org/wiki/Module:Navigation from the dozen of books where it... [20:37:50] 3Librarization, MediaWiki-Core-Team: Test monolog logging provider in beta - https://phabricator.wikimedia.org/T845#819419 (10bd808) [20:39:22] 3Librarization, MediaWiki-Core-Team: Test monolog logging provider in beta - https://phabricator.wikimedia.org/T845#819427 (10bd808) 5Open>3Resolved Monolog output to udp2log and redis seem to be working great. [20:45:41] 3Librarization, MediaWiki-Core-Team: Deploy Monolog logging configuration for WMF production - https://phabricator.wikimedia.org/T76759#819481 (10bd808) [20:49:36] 3Wikimedia-Logstash, MediaWiki-Core-Team: Ignore apache2 log2udp messages and transition rules to the syslog stream of the same events - https://phabricator.wikimedia.org/T76760 (10bd808) 3NEW p:3Normal [20:56:01] aude: will there ever be a setclaims API (not just setclaim)? Would be nice to not have so many bots making 3-7 edits in a row at the same timestamp to add multiple items. I guess there is the json blob api, but the format is not stable and that requires fetching everything first. [21:18:28] 3MediaWiki-extensions-CentralAuth, SUL-Finalization, MediaWiki-Core-Team: Expose users_to_rename table publicly - https://phabricator.wikimedia.org/T76774#819609 (10Legoktm) [21:32:44] AaronS: I see this in beta right now: [21:32:45] 21:21:08 [0ace469b] [no req] ConfigException from line 53 of /mnt/srv/mediawiki-staging/php-master/includes/config/GlobalVarConfig.php: GlobalVarConfig::get: undefined option: 'UDPProfilerPort' [21:32:49] from https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/label=deployment-bastion-eqiad,wikidb=simplewiki/5930/console [21:33:38] AaronS: I think I got stuck on much somehow [21:34:14] ^d: ^ [21:34:14] manybubbles: whoa. [21:34:23] ^d: see that error above [21:34:35] ^d: maybe related to https://gerrit.wikimedia.org/r/#/c/177561/ ? [21:36:54] in StatCounter.php: [21:36:55] $this->config->get( 'UDPProfilerHost' ), [21:36:55] $this->config->get( 'UDPProfilerPort' ) [21:37:26] mute, ramn it [21:37:28] damn it [21:37:31] wtf - I can't type any more [21:40:38] greg-g: silly, reverting [21:42:12] <^d> fucking config objects. [21:42:50] <^d> Wait, what? [21:42:50] <^d> Where? [21:43:15] beta cluster [21:43:15] <^d> StatCounter what? [21:43:37] <^d> how come my grep didn't find youuuu [21:43:38] <^d> I know the beta cluster bit. [21:43:45] * ^d sings songs about 99 problems/ [21:43:45] heh [21:43:47] :) [22:22:30] is there no #mediawiki project? like the old general/unknown? I can't find it [22:22:53] mediawiki-general-unknown I think [22:23:15] https://phabricator.wikimedia.org/tag/wikimedia-general-or-unknown/ [22:23:22] i think the phabricator migration unearthed what was previously implicit [22:23:23] great discoverability! [22:23:24] that's a different tag [22:23:30] that core has no ownership [22:23:42] https://phabricator.wikimedia.org/tag/mediawiki-general-or-unknown/ [22:24:08] except *us* [22:24:22] which is both awesome and horrible [22:24:28] mostly horrible [22:25:19] bd808 added hashtags: mediawiki, mediawiki-core [22:29:11] bd808: ori: that depends on the component, really [22:29:57] timo and i are the de-facto maintainers of js/css parts of core, for example (or so i think) [22:30:22] i18n owns the i18n parts [22:30:40] MatmaRex: great! you'll love https://gerrit.wikimedia.org/r/#/c/177501/ then :P [22:30:42] wikidata owns contenthandler [22:30:52] y'all own the databases and stuff [22:30:55] MatmaRex: *nod* Its a bit of a running joke in this room [22:31:07] ori: i liked it, but i currently have like 300 unread emails [22:31:22] you should mentor some GCI tasks [22:31:39] MatmaRex: http://i.imgur.com/FWu18Wt.png your move. [22:31:41] so that i don't have to do all of them [22:31:54] ori: ah, but you're never going to read these anyway [22:31:58] i'm going to read mine [22:32:00] you don't know that [22:32:09] or so i keep telling myself [22:37:37] <_joe_> I <3 PHP http://blog.ircmaxell.com/2014/12/what-about-garbage.html [22:39:14] <^d> ori, MatmaRex: inbox zero, every week. #iwin [22:40:27] <_joe_> tsk [22:40:33] * _joe_ adds chad to root@ [22:40:33] ^d: i have twenty or so emails that are my todo list, but apart from that, i usually also keep my inbox neat. i'm just drowning in better things to do this week [22:40:59] ^d: inbox zero everyday, I lose [22:41:21] * bd808 has email ocd [22:41:31] <^d> It's actually usually most days here too. [22:41:35] <^d> Sometimes 1 or 2 slip to the next. [22:41:43] <^d> But I never start the weekend with an inbox. [22:42:19] <_joe_> the secret to inbox zero is aggressive filters, so tht you can ignore most mails [22:42:32] <^d> Yes. [22:42:32] <^d> That. [22:42:35] _joe_: CheckSymLink=false fares even better: http://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&h=mw1081.eqiad.wmnet&m=cpu_report&s=by+name&mc=2&g=cpu_report&c=Application+servers+eqiad [22:42:47] <_joe_> so it's mostly "inbox whatever" [22:42:56] <_joe_> ori: I am looking at it [22:43:04] <_joe_> keep an eye on fatals [22:59:15] <^d> csteipp: Are you saying we shouldn't sync my confirmedit change in this afternoon's swat? It's already got like a half dozen +1s on it :p [23:01:11] csteipp: just fyi, re that add cc's to security bug, er, bug: that sync works for adding cc's, but not necessarily when you remove a cc (ie: you'd have to go in manually remove them from the ACL) [23:02:59] ^d: I'm not trying to block it. If enough people think it's the right thing, go for it. I'm just hopeful someone is watching the all the right places, because I don't have time to do it myself :) [23:03:44] https://test.wikipedia.org/wiki/Special:Log/block [23:04:00] greg-g: So if I add a new CC to a bug and it has the Security dropdown set, it currently adds them to the view and edit policies? [23:04:09] https://test.wikipedia.org/wiki/Special:Log/delete [23:04:11] no spam afais [23:04:20] <^d> legoktm: That doesn't count people who registered on testwiki and then got themselves blocked on enwiki. [23:04:27] <^d> Like csteipp said, SUL. [23:04:57] right [23:05:03] I'm looking at https://test.wikipedia.org/w/index.php?title=Special%3ALog&type=newusers&user=&page=&year=&month=-1&tagfilter= right now [23:05:21] Exactly. Hopefully someone is watching them globally, so we can say, the % of users who registered via testwiki and were later blocked on any wiki has gone up/down [23:06:55] <^d> I wonder if we should loop global sysops + stewards to this little test. [23:07:15] They should definitely be in the loop about it. [23:07:39] Did someone break skins for git installs again? [23:07:41] when did we turn off captcha on testwiki? [23:07:49] <^d> Good question. What's blame say [23:08:03] * legoktm blames [23:08:52] Github shows a picture of a rainbow unicorn if it takes too long to load. [23:09:41] https://gerrit.wikimedia.org/r/#/c/171998/ nov7 [23:09:51] <^d> 9b9992999 (MZMcBride 2014-11-0 [23:09:59] <^d> 07, even [23:10:27] <^d> Also, wtf is wmgEmergencyCaptcha for? [23:10:45] ptwiki I think [23:10:56] Nov 7, yeah. [23:10:57] 10 new users, not including User:TestForCaptchaStuff [23:11:04] :) [23:11:21] and not including the ones made by an already existing user testing stuff [23:11:32] <^d> Ah yes [23:12:37] <^d> Heh, the captcha config still points at /mnt/upload7/ [23:12:39] <^d> How cute. [23:14:23] https://test.wikipedia.org/wiki/Special:DeletedContributions/Dicky.co.id doesn't look like a spammer [23:15:20] There were two accounts that created on test.wp and then made edits to pt.wikt which was a bit weird, but didn't look malicious [23:16:21] Oh legoktm, do remember off hand if a user is autoconfirmed on one wiki, do that get passed on to wikis where they are autocreated via sul? [23:16:55] nope [23:17:10] there's a bug open for a "global autoconfirmed" kind of thing [23:17:31] but for now you have to get the autoconfirmed individually on each wiki [23:18:10] <^d> I wonder if I can get my autoconfirmed taken away on all wikis. [23:18:14] <^d> I want less privs! [23:19:38] you can use the abusefilter to revoke it I think. [23:26:11] <^d> Maybe we could push it back until like Monday. [23:26:18] <^d> So everyone knows? [23:26:32] <^d> (Although, we may jinx ourselves. By announcing it, we invite spammers to fuck with us :p) [23:32:21] Just send a heads up to stewards-l? [23:32:41] For just mediawiki.org I don't think it needs a big announcement. [23:33:35] TimStarling: what are you plans re: tidy? It seems to me that it might be quicker to strip the zend stuff from php_tidy_quick_repair and expose it via HNI directly, skipping EZC [23:34:22] I'm working on VisualEditor at the moment [23:35:03] I might have another look at tidy next week some time [23:35:11] I don't know what is quicker, because I don't know what the problem is [23:35:31] we don't have any reproduction procedure do we? [23:37:09] no. it looked like it was barfing on TIDY_APPLY_CONFIG_ZVAL [23:39:01] I spent a lot of time writing EZC, so it would be nice if there was some sort of application for it [23:39:48] obviously it would have been a lot quicker to rewrite luasandbox in HNI than to rewrite EZC, the same presumably could be said of any individual extension that uses it [23:40:53] OK. It'd be a learning exercise for me to try and debug further. Would you mind? [23:42:58] sure, go ahead [23:46:13] cool. the high(er) system cpu% issue on the hhvm turned out to be hhvm calling realpath() on symlinks on every request, to avoid duplicating work for source files that are referenced via both real and symbolic paths (or multiple symbolic paths) [23:46:36] RuntimeOption::CheckSymLink fixes it [23:46:49] good stuff [23:48:27] <^d> legoktm, csteipp: Quick informal poll. Yay or nay on sending that out in swat this afternoon? [23:48:40] it can also be mitigated by turning on ServerStatCache, which causes HHVM to cache stat calls, using inotify to invalidate [23:49:43] I +1'd it, so yay from me as long as someone emails the stewards. [23:49:53] ^d: I'm ok with it [23:50:08] <^d> Ok, I'll e-mail stewards. [23:50:28] And yeah, stewards-l should definitely get a notice, and someone should be on call to revert it over the next few days if anything goes crazy [23:50:31] * csteipp not it! [23:51:20] <^d> "Everyone who is not subscribed, and not on the send-only list, cannot send a message to the list members. Email instead the list owners or use stewards@wikimedia.org." [23:51:27] <^d> I guess I have to send to stewards@wm [23:51:47] oh, I'm whitelisted [23:51:50] I can send it [23:52:10] unless you already sent it to their OTRS queue [23:53:19] <^d> I'll send you my draft, one minute por favor. [23:58:03] <^d> sent. [23:58:37] <^d> Ah, add mediawiki.org to the "list of affected wikis" [23:58:41] <^d> Missed in revising. [23:59:11] and it's test.wikidata.org :) [23:59:48] <^d> meh, details.