[00:00:06] zero.wp isn't SUL enabled right? [00:00:16] <^d> I don't think so. [00:00:30] <^d> zero's fishbowlly I think. [00:00:31] it redirects to https://en.zero.wikipedia.org/wiki/Special:ZeroRatedMobileAccess for me [00:00:35] No, it's not supposed to be [00:00:56] ok, /me removes from email [00:01:50] sent [00:02:24] <^d> ty [00:07:31] 3MediaWiki-extensions-Scribunto, MediaWiki-Core-Team: An exception is thrown if an argument value passed to frame:expandTemplate has an invalid type - https://phabricator.wikimedia.org/T76609#820128 (10Jackmcbarn) Don't remove the module. Just fix the bug in it. [00:14:20] 3MediaWiki-extensions-Scribunto, MediaWiki-Core-Team: An exception is thrown if an argument value passed to frame:expandTemplate has an invalid type - https://phabricator.wikimedia.org/T76609#820141 (10bd808) [01:01:30] <^demon|away> legoktm, csteipp: Well that was a fun experiment for < 1hr. [01:01:44] Seriously. [01:01:56] Although I think it did answer the question [01:02:24] So Success! [01:02:27] <^demon|away> "Yes, captchas do stop a non-zero amount of spam" [01:03:49] <^demon|away> legoktm: We got a response from a steward. Could you respond so it'll go to the list? [01:17:21] <^demon|away> legoktm: Already done, nvm. [01:21:52] sorry, I was afk for a bit [01:24:21] <^demon|away> no worries [01:36:03] 3Collaboration-Team, MediaWiki-Core-Team, Scrum-of-Scrums: Enable $wgContentHandlerUseDB everywhere (bug 49193) - https://phabricator.wikimedia.org/T1217#820331 (10Spage) [01:36:04] 3Wikimedia-General-or-Unknown, Collaboration-Team, MediaWiki-Core-Team, Scrum-of-Scrums: Add ContentHandler columns to Wikimedia wikis, and set $wgContentHandlerUseDB = true - https://phabricator.wikimedia.org/T51193#820332 (10Spage) [01:43:28] <^demon|away> I got an e-mail inviting me to a webinar. [01:43:36] <^demon|away> I always want to respond and tell them webinar isn't a word. [01:45:47] wow! webinars exist, but not the word for them? :P [02:08:20] <^demon|away> MaxSem: It's for one of the silly elastic plugins you can pay money for :p [02:08:31] <^demon|away> Specifically the new "give elasticsearch security" one, lol. [02:09:12] the type of security you get by not exposing it to every haxx0r on the interwebz? [02:09:13] Hah, they have that out now? They were trying to recruit security people to work on it just a few months back... [02:11:58] Security as a Plugin sounds like Wordpress' Security as a Plugin [02:12:02] bah [02:12:12] 2nd security should be performance [02:13:19] <^demon|away> csteipp: http://www.elasticsearch.com/products/shield/ [02:15:44] <^demon|away> "Shield integrates with LDAP-based authentication systems as well as Active Directory, so your users don’t need to remember yet another password." [02:16:00] <^demon|away> We should totally use Active Directory for soa-auth. [02:18:15] why not Live ID? [02:18:53] <^demon|away> is that still a thing? [02:19:25] oh sorr,y it's called M$ account now [06:33:16] It's amazing that PECL is as large as it is. The Zend API is so offensive. [07:09:48] <_joe_> ori: PECL is large, not great :) [07:11:24] <_joe_> btw, I looked at mw1081, its system cpu is at 1.8% system vs 2.1% system of mw1080 [07:11:58] <_joe_> so definitely a gain, not sure if it's enough to justify running it in prod now [07:13:58] _joe_: you can't compare two hosts like that. system CPU% is not evenly distributed [07:14:07] you can easily tell that from http://ganglia.wikimedia.org/latest/?r=week&cs=&ce=&c=Application+servers+eqiad&h=&tab=m&vn=&hide-hf=false&m=cpu_system&sh=1&z=small&hc=4&host_regex=&max_graphs=0&s=by+name [07:14:31] <_joe_> what we should do (I think) is trying it on an API appserver [07:14:33] presumably certain types of requests cause it more than others, and varnish routing makes that affect some machines more than others [07:14:48] <_joe_> it's LVS [07:15:03] <_joe_> which uses wrr [07:15:07] <_joe_> so I don't see how [07:15:17] tell that to ganglia [07:15:36] <_joe_> look at the last 4 hours [07:15:39] <_joe_> http://ganglia.wikimedia.org/latest/?r=4hr&cs=&ce=&c=Application+servers+eqiad&h=&tab=m&vn=&hide-hf=false&m=cpu_system&sh=1&z=small&hc=4&host_regex=&max_graphs=0&s=by+name [07:16:10] <_joe_> apart from mw1041 and mw1072 (that are lacking hyperthreading) [07:16:23] <_joe_> it's pretty evenly around 2% [07:16:32] There was an error collecting ganglia data (127.0.0.1:8654): [07:16:41] http://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&title=&vl=&x=&n=&hreg[]=mw108%5B1%2C0%5D&mreg[]=cpu_system>ype=line&glegend=show&aggregate=1&embed=1&_=1417763692258 [07:16:45] <_joe_> (and you can also see mw1081 stands out) [07:17:17] so that drop is you toggling hyperthreading? [07:17:25] <_joe_> yep [07:17:33] <_joe_> being a percentage of cpu time [07:17:39] no, there are two very distinct drops [07:17:41] <_joe_> and being HT incredibly efficient [07:17:46] <_joe_> yes there are two [07:17:57] <_joe_> the first is stat_cache [07:18:00] <_joe_> the second is HT [07:18:46] <_joe_> awww most larger appservers have HT disabled [07:18:53] <_joe_> the ones I didn't do directly [07:19:03] <_joe_> sigh [07:20:17] <_joe_> seems that I gave yuvi a bad advice on how to check if HT is enabled [07:21:29] let's fix that quickly if possible [07:21:36] it's extremely confusing to have these variations [07:22:52] maybe combined with a dist-upgrade [07:24:40] it's also unfortunate that we were combining HT changes and configuration changes [07:25:33] combine that with seasonality and you end up with a graph that doesn't tell a decisive story about anything [07:27:03] it hadn't stopped dropping yet when you enabled hyperthreading [07:29:02] <_joe_> ehe [07:29:30] <_joe_> ok, well the average utilization of the cluster is still 11% [07:29:36] <_joe_> so it's not killing us [07:29:43] <_joe_> but yes, I do want consistency [07:29:54] <_joe_> (where we had a mess of mixed configs before) [07:31:37] sigh [07:32:08] we have three versions of hhvm installed [07:32:16] Installed: 3.3.0+dfsg1-1+wm3.1 [07:32:16] Installed: 3.3.0+dfsg1-1+wm4 [07:32:16] Installed: 3.3.0+dfsg1-1+wm5 [07:32:16] <_joe_> I thought 2 [07:32:26] <_joe_> where is 3.3.0+dfsg1-1+wm3.1 ? [07:32:34] mw1163, for example [07:32:42] <_joe_> I was aware of the 4-5 thing, which isn't a big deal tbh [07:32:46] <_joe_> given what changed [07:32:47] and mw1053 [07:33:05] <_joe_> ok, they were out of the cluster until yesterday so that makes sense [07:33:21] <_joe_> btw, 3.1 - 5 the only changes are: [07:33:39] <_joe_> 1) you patch to add the endpoint to the admin interface [07:33:51] <_joe_> 2) Tim's patch for tidy [07:34:09] and three kernels [07:34:14] 3.13.0-24-generic [07:34:14] 3.13.0-39-generic [07:34:14] 3.13.0-40-generic [07:34:17] <_joe_> that is expected [07:34:21] <_joe_> as well [07:34:50] <_joe_> we don't reboot/reinstall a kernel if there is a security issue [07:35:02] <_joe_> *there isn't [07:35:20] we should in this case, exactly so we can do comparisons between hosts [07:35:20] <_joe_> better [07:35:27] <_joe_> a security issue that affects us [07:35:46] <_joe_> do you have specific reasons to expect regressions between these versions? [07:36:03] <_joe_> it's a ubuntu minor version change, so there are no perf fixes [07:37:18] <_joe_> anyway, I have to reboot most servers to enable HT, so I'll dist-upgrade them anyway [07:37:33] the ability to reason conclusively about that with so many variables is beyond the reach of humans, so that's why it's better to control for as much as possible, imo [07:37:38] <_joe_> (that in fact does affect comparisons) [07:37:51] <_joe_> what about hardware differences? [07:38:02] <_joe_> they count much more than a minor kernel version [07:38:04] i thought about that, there's a missed opportunity here for a better server naming scheme [07:38:15] but that's probably out of the question now [07:38:27] <_joe_> which is _not_ expected to affect performance [07:38:29] <_joe_> is any way [07:38:58] not unexpected either [07:39:07] <_joe_> so yeah, maybe the rack the servers are in counts as well, given they may have different switches and all, but it's rapidly becoming the Poincare' theorem [07:39:17] <_joe_> no they would be unexpected [07:39:46] <_joe_> but a minor ubuntu kernel version is _not_ a performance factor [07:40:01] <_joe_> anyway, I have to reboot most servers to enable HT, so I'll dist-upgrade them anyway [07:40:12] <_joe_> and also that [07:40:13] <_joe_> :) [07:40:18] i'm happy to help by the way [07:40:35] <_joe_> do you have mgmt access? [07:40:59] don't i have access to the credentials? [07:41:01] <_joe_> mmmh I'm thinking this is a good occasion to expand the experiments paravoid did with tools from the os to change bios settings [07:41:05] i've never used it [07:41:22] <_joe_> ori: I think they are on iron, not sure [07:41:43] please let's have fewer variables before we add any [07:42:24] i should sleep, i'm a bit cranky (about something unrelated) and it's showing. sorry for grumbling. [07:42:51] <_joe_> yeah I agree, however the first 100 servers should be consistent [07:42:54] <_joe_> apart from 5 [07:43:01] <_joe_> so I will prioritize those [07:43:17] if you'd like me to help i can dedicate time to it tomorrow, just forward me whatever instructions you sent yuvi [07:43:18] <_joe_> and we have a solid baseline of identical (even in hw) machines [07:43:51] <_joe_> no I think having those 100 other the weekend is ok, the other 64 are almost all without HT [07:44:04] <_joe_> and I want to do that better than having to get in the bios [07:44:05] <_joe_> :) [07:44:42] cool. i'm looking at the tidy thing. i may not get anywhere, in which case tim will probably look at it sometime next week. [07:44:55] but i'm in the early optimistic stages of debugging [07:45:09] <_joe_> eheh [07:45:17] <_joe_> well, I think we have time [07:45:28] <_joe_> I also want to try an imagescaler soon-ish [07:45:41] sure [07:46:02] i'm off, good night, and thanks again [07:47:35] <_joe_> night [07:47:41] <_joe_> thank you for all the great work [07:47:54] <_joe_> and, take some time off this weekend, you greatly deserve it :) [07:51:42] 3MediaWiki-extensions-Scribunto, MediaWiki-Core-Team: An exception is thrown if an argument value passed to frame:expandTemplate has an invalid type - https://phabricator.wikimedia.org/T76609#820739 (10JackPotte) Too late for that. If nobody wants to assure a backward compatibility or an exception handling, you... [12:39:40] 3MediaWiki-extensions-Scribunto, MediaWiki-Core-Team: An exception is thrown if an argument value passed to frame:expandTemplate has an invalid type - https://phabricator.wikimedia.org/T76609#821135 (10Jackmcbarn) No, it's not too late for that. Whatever changes were made can be reverted. We are fixing the probl... [14:26:09] bd808|BUFFER: https://github.com/elasticsearch/elasticsearch/issues/8707#issuecomment-64954834 [14:57:04] 3MediaWiki-extensions-Scribunto, MediaWiki-Core-Team: An exception is thrown if an argument value passed to frame:expandTemplate has an invalid type - https://phabricator.wikimedia.org/T76609#821386 (10Anomie) >>! In T76609#820739, @JackPotte wrote: > If nobody wants to assure a backward compatibility or an exce... [15:08:26] 3MediaWiki-extensions-Scribunto, MediaWiki-Core-Team: An exception is thrown if an argument value passed to frame:expandTemplate has an invalid type - https://phabricator.wikimedia.org/T76609#821408 (10JackPotte) I was just noticing that before the last migration the books pages displayed their contents, and not... [16:01:03] ^d: if I want to add a remote branch to gerrit do I create it locally and just push it? [16:01:56] <^d> New branches, yeah since you're just creating a ref. [16:02:04] <^d> Subsequent pushes to that branch would go through review. [16:02:14] <^d> s/would/should/ [16:05:06] ^d: cool. thanks! [16:05:12] <^d> yw [16:06:25] manybubbles: ah. I'll try that. I tried forcing allocation yesterday but I didn't delete the on disk data first. [16:10:53] manybubbles: The corrupt/unallocated shard is gone. (and I didn't do anything to fix it) *shrug* [16:11:07] bd808: uh, yay? [16:11:19] yeah. I guess [16:11:29] mystery fix for mystery bug [16:12:09] sad [16:12:48] ^d: I'm going to cut a 1.4 branch release for the experimental highlighter plugin because users seem to want it. [16:12:54] I guess it's a good thing I don't fancy myself a system admin anymore or it would drive me nuts [16:12:57] so we'll have a 1.3 branch and a master branch [16:13:11] bd808: I think system administrators just have to live with some level of uncertainty [16:14:03] True. I wasted many days/nights trying to make sense of crash logs and such in the olden days [16:14:16] <^d> manybubbles: okie dokie. [16:14:25] But sometimes it worked and I'd find a jvm bug or a kernel problem [16:14:57] which were both kind of wasted efforts when Sun was my upstream and they really didn't care :/ [16:16:40] <_joe_> ori: all the appservers in the main pool have hyperthreading activated [16:54:34] hi bd808, I just did two things: [16:54:41] 1. approved Krinkle's request for access to the mediawiki composer account [16:54:53] 2. delegated future approvals to you [16:55:04] * bd808 goes mad with power [16:55:19] <^d> Can we remove gerritadmin@ from composer stuff. [16:55:25] Does that mean I have access to said account? [16:55:28] <^d> It's the /wrong user/ and Antoine and I get spammed. [16:56:06] bd808: I just assumed you did. [16:56:18] you can always approve yourself :-) [16:56:30] <^d> escalation! [16:56:36] I think me, krinkle, hashar, and ^d have access? [16:56:51] This is https://packagist.org/users/mediawiki/ ? [16:56:58] yes [16:57:06] ^d: what email should we set it to instead? [16:57:18] <^d> I don't know. [16:57:25] <^d> not me :) [16:57:55] Sounds like it needs a new list/alias [16:58:53] <^d> list could make sense. good place to announce new packages. [16:59:03] The packagist account is just a shared password thing right? They don't have proper organization accounts as I recall [16:59:11] <^d> ya [16:59:20] * bd808 should file a bug about that [17:00:47] Does password reset/other administrivia go to the registered account? [17:01:03] I would imagine it does, yes [17:01:17] * bd808 sees robla's +1 on https://github.com/composer/packagist/issues/163 [17:02:43] yeah, ^d was telling me about packagist issue 163 yesterday [17:03:32] it looked like Evan hadn't posted anything in the most relevant place to get upstream's attention, so I put it there. [17:04:20] 3MediaWiki-Core-Team, MediaWiki-extensions-GWToolset, Multimedia: Upload stopps regularly - https://phabricator.wikimedia.org/T76450#821582 (10hoo) [17:07:22] 3MediaWiki-Core-Team, MediaWiki-extensions-GWToolset, Multimedia: Upload stopps regularly - https://phabricator.wikimedia.org/T76450#821593 (10hoo) Now I found something, not sure how relevant it is: ``` error=GWToolset\Jobs\UploadMediafileJob::run: <Other contributors: A media file with the identical title... [17:07:40] manybubbles: geo search working in orient now, yay \o/ [17:07:54] <^d> sweet [17:07:55] we have lots of coords that need to be wrapped around so lucene doesn't whine [17:08:13] e.g. using lat 93N or lon 190W [17:08:59] robla, Krinkle|detached, legoktm: https://github.com/composer/packagist/issues/461 [Support for "organization" (shared access) accounts] [17:09:13] manybubbles: the slow queries not using collection indexes is caused by a bug where "contains (...)" is slow but "contains(...)" is fast [17:09:21] * AaronS should file a bug [17:09:51] <^d> Oh cool, I didn't know orientdb used lucene too. Yay shared upstreams. [17:10:09] ^d: it's a "drop a jar file in a dir" plugin, very easy [17:10:56] bd808: have we (or should we) snarf up the org accounts we're most likely to want? e.g. "wikimedia" [17:11:03] <^d> AaronS: yay java :) [17:11:24] <^d> Thankfully we've already got build mechanisms in place for prod for java, so that shouldn't be a big deal these days. [17:11:39] all the xml/framework/overengineering crap notwithstanding, there are some things I love about Java itself [17:11:55] I think we should be name squatting on wikimedia, yes. [17:12:31] <^d> wikimedia/what? [17:13:03] <^d> Oh, I guess the /user/ wikimedia [17:13:09] ^d: yeah [17:13:10] <^d> silly buckets. [17:13:35] <^d> AaronS: Well written java is nice to work with :) [17:15:09] robla: I not am squatting on https://packagist.org/users/wikimedia/ [17:15:15] s/not/now/ [17:15:27] excellent [17:15:30] so where do we track passwords for things like this? [17:15:59] Ops has them in a file somewhere. Krinkle requested them via RT [17:16:07] the password for "mediawiki" is with ops [17:16:38] *nod* Ok. So I need to probably file an RT to give the password over I guess [17:16:45] yup [17:16:57] and to get the password for the other account [17:17:09] also: yup [17:17:49] probably not putting the password in RT, but rather, arranging to meet an opsen in an undisclosed location [17:18:47] yeah. [17:19:25] perhaps through intermediaries who are given cyanide pills they are instructed to swollow at the completion of their mission. [17:19:44] or in a gpg message to Faidon :) [17:19:52] BORING! [17:20:25] * bd808 needs to put a key on his work laptop [17:28:03] ^d: btw, I figured the reason why the wikimedia/mediawiki account is managed by gerritadmin. It's because when you make a change and push a tag, gerrit syncs it to github, and the authenticating user with github is github.com/wmfgerrit, which becomes the creator/manager/admin for that package on packagist. [17:28:38] but we can add additional accounts though [17:28:40] it's viral [17:29:01] <^d> the e-mail shouldn't matter. [17:29:09] <^d> iirc It was because when hashar and I made the account and didn't have a better role account to use. [17:29:21] Yeah, the /github/ account, though. [17:30:22] we should rename it to wikimedia, and maybe virally add maintainers as appropiate. [17:30:37] I'm not sure an umbrella/sudo-like account as the default workflow to do anything at packagist is better. [17:31:06] I'd rather keep 'wikimedia' as a root thingy to login manually and as day-to-day just virally, from that account, grant access to other maintainers as needed. [17:31:21] also makes it easier to figure out who's maintaining what [17:31:23] maybe? [17:34:18] Krinkle: https://packagist.org/users/wikimedia/ [17:34:24] I got it today [17:34:43] I just put in an RT ticket to give the password to ops [17:36:50] bd808: OK [17:37:11] bd808: We should figure out how the 'mediawiki' account got attached to github/wmfgerrit so that we can get this to be the default one [17:37:25] for anything pushed through via gerrit->github->hook [17:37:45] it doesn't have to be attached [17:37:51] you just have to add the service hook on github [17:38:02] so whenever something is pushed github will notify packagist [17:39:00] 3MediaWiki-extensions-CentralAuth, SUL-Finalization, MediaWiki-Core-Team: Expose users_to_rename table publicly - https://phabricator.wikimedia.org/T76774#821626 (10Keegan) This is probably best achieved through a Special: page. I fully support having this ready by January. [17:40:22] There is a "connect your github account" button in the ui at packagist. [17:41:02] So somebody with the passwords for https://packagist.org/users/wikimedia/ and https://github.com/wmfgerrit could hook them up [17:42:56] what's the rt? [17:43:12] also, holy crap is this project going well [17:43:16] kudos Krinkle / bd808 [17:43:17] https://rt.wikimedia.org/Ticket/Display.html?id=8988 [17:43:27] and everyone else involved, obviously [17:43:45] i believed in it before it was cool! [17:44:25] legoktm and ^d have done much more actual work than I have so *high five* [17:45:18] <^d> go team! [17:45:26] 'mercia! [17:45:34] <^d> fuck yeah [17:45:55] 3wikidata-query-service, MediaWiki-Core-Team: Experiment with Wikidata Query Engine - https://phabricator.wikimedia.org/T1028#821650 (10Manybubbles) [17:46:15] bd808: i love this ticket [17:46:20] 3MediaWiki-Core-Team: [CirrusSearch] Split prefix searches from full text searches in pool counter - https://phabricator.wikimedia.org/T76490#821651 (10Manybubbles) [17:46:26] "please take this away from me, but then please give it back" [17:46:39] it makes sense, just funny [17:47:09] shared accounts are a pain in the ass [17:47:28] <^d> ya [17:47:49] * bd808 is ashamed that he has built software that required them before [17:48:04] 3CirrusSearch, MediaWiki-Core-Team: Cut Elasticsearch 1.4 compatible release for Experimental Highlighter - https://phabricator.wikimedia.org/T76872 (10Manybubbles) 3NEW p:3Low [17:48:16] i won't tell chris [17:48:29] heh. [17:48:52] 3MediaWiki-Core-Team: Cut Elasticsearch 1.4 Compatible Release of Wikimedia-Extra plugin - https://phabricator.wikimedia.org/T76873#821662 (10Manybubbles) [17:49:06] 3CirrusSearch, MediaWiki-Core-Team: Cut Elasticsearch 1.4 compatible release for Experimental Highlighter - https://phabricator.wikimedia.org/T76872#821654 (10Manybubbles) a:3Manybubbles [17:52:50] bd808: Ah, the github account used for packagist is determined by the hook, not by the user pushing the commit to github? [17:53:04] So it takes like a usename and auth token fixed for the repo, regardless who pushes. [17:53:15] I think that's right. Its tied to the repo [17:54:00] 3MediaWiki-Core-Team: Degrade large phrase queries - https://phabricator.wikimedia.org/T76874 (10Manybubbles) 3NEW p:3High a:3Manybubbles [17:55:05] 3CirrusSearch, MediaWiki-Core-Team: Release wikimedia-extra to pick up "Degrade large phrase queries" - https://phabricator.wikimedia.org/T76875 (10Manybubbles) 3NEW p:3Triage a:3Manybubbles [17:55:39] 3CirrusSearch, MediaWiki-Core-Team: Cut Elasticsearch 1.4 compatible release for Experimental Highlighter - https://phabricator.wikimedia.org/T76872#821698 (10Manybubbles) [17:59:46] 3Wikimedia-General-or-Unknown, Collaboration-Team, MediaWiki-Core-Team, Scrum-of-Scrums: Add ContentHandler columns to Wikimedia wikis, and set $wgContentHandlerUseDB = true - https://phabricator.wikimedia.org/T51193#821703 (10csteipp) [18:05:14] 3Librarization, MediaWiki-Core-Team: xhprof for MW - https://phabricator.wikimedia.org/T759#12620 (10Chad) [18:11:57] https://www.mediawiki.org/wiki/Manual:Profiling/Xhprof [18:12:05] I left some todos on that [18:12:16] https://www.mediawiki.org/wiki/Manual:Profiling needs an update too [18:13:00] bd808: ahahahahahah: http://docs.saltstack.com/en/latest/topics/ssh/roster.html [18:13:09] bd808, ^d: should "cdb/cdb" be named "wikimedia/cdb" instead? [18:13:33] legoktm: It could be. That would "preserve our brand" :) [18:13:37] <^d> I've got an edit window open for Manual:Profiling right now. [18:14:15] ori: *facepalm* [18:15:07] Salt is neat but I still think it sucks for push-based deployment tooling [18:15:37] And I honestly don't care if it's "better" than puppet as long as we are a puppet shop [18:15:59] bd808: +1 [18:16:14] puppet is slow and crazy but its our slow and crazy [18:16:16] all devops tools suck. and will always suck. You just have to pick some and learn to work with them [18:16:29] MaxSem: wanna review a pool counter change? [18:16:31] you know you wanna [18:17:16] bd808: completely different from http://docs.saltstack.com/en/latest/topics/targeting/nodegroups.html#targeting-nodegroups [18:17:25] salt just has five different implementations for everything [18:17:44] I'm completely opposed to the idea of a universal deployment tool that solves all problems because it will never exist. [18:17:56] There are too many variables to control for [18:18:05] Frameworks for such things, sure [18:18:18] like maven2 or something [18:18:33] manybubbles, sure [18:18:59] what change? [18:19:36] ori: not only does it have 5 implementations, but all of them have bugs and will be fixed in the next point release [18:20:40] legoktm: cool. I think we have a phab task for updating the docs somewhere. [18:20:49] that's my new answer to everything [18:20:57] there's a phab task for it [18:22:30] :P [18:23:58] https://phabricator.wikimedia.org/T76877 (we didntt have one quite yet) [18:27:43] <^d> Manual:Profiling updated. [18:27:53] <^d> Links to /Xhprof subpage too [18:31:12] MaxSem: well I'd like to teach it to take multiple locks per connection. so you could take a lock as a per user rate limit and then another one for access to the overall pool of resources. in particular this is to prevent someone from finding another query that hammers cirrus and redoing it 200 times a minute. It won't prevent ddos with it but it'd help. [18:31:39] woot :D [18:31:46] whyyyy is it mostly in rubyyyyy?! [18:32:41] 3Librarization, MediaWiki-Core-Team: xhprof for MW - https://phabricator.wikimedia.org/T759#821755 (10Chad) [18:34:25] https://github.com/wikimedia/mediawiki/commit/7da1cbc78c071c80cbd53f8c96baeee49ac20f59#commitcomment-8856057 o.O [18:34:45] MaxSem: integration tests. it didn't have any and i figured I'd use what we use for integration tests elsewhere [18:34:52] it didn't have any tests actually [18:34:58] and I didn't want to break it [18:37:13] <^d> manybubbles: re: speaking...they've got a form for that too :) [18:37:16] <^d> And they're still calling. [18:37:28] ah [18:37:36] well if you wanna do that they'd certainly take you [19:11:01] <^d> bd808: Are you thinking of oscon this year? [19:11:15] <^d> I haven't done it in years. Could be fun to send a contingent again. [19:11:52] I'd go for sure. I've never presented there but maybe we can dream up something for me to talk about. [19:12:34] it would be fun to be there as a real live FOSS boy instead of a corporate shill looking for help taking people's money [19:13:15] Plus my baby brother and a good drinking buddy live in PDX so I'm always up for a trip there :) [19:14:44] 3Wikimedia-General-or-Unknown, Collaboration-Team, MediaWiki-Core-Team, Scrum-of-Scrums: Add ContentHandler columns to Wikimedia wikis, and set $wgContentHandlerUseDB = true - https://phabricator.wikimedia.org/T51193#821844 (10EBernhardson) Basically it seems T73163 is what is stalled here, to solve that I thin... [19:16:12] ori: can you re-CR https://gerrit.wikimedia.org/r/#/c/177779/ ? [19:16:29] I didn't see your CR+2 and answered Umri's question anyway, ended up nullyfing your cr [19:17:55] Krinkle: Thanks for the feedback on that fatal log patch. I totally did not test with PHP5. :( I'll fix it up [19:18:30] Also how to I debug the qunit failure that is happening with it? -- https://integration.wikimedia.org/ci/job/mediawiki-core-qunit/33287/console [19:18:38] *how do I [19:19:45] bd808: At first I'd try reproducing locally ([[mw:Manual:JavaScript_unit_testing]]). If it passes locally, I'd fetch the Build artefacts from the build page and see if there's any mw-exception.log etc. [19:20:09] Looks like it might've affected load.php cache interface [19:20:31] not an actual unit test directly. More the backend falling away and random tests failing from what I can see [19:20:40] k. sounds reasonable. I've never run the qunit tests locally before so that will be good to learn a bit about [19:22:59] bd808: look at Chrome Dev Tools console, it'll have something like "Variable x undefined" (like the php warning log) [19:29:18] legoktm: It's a stalker [19:29:23] legoktm: I'm pretty sure I know who it is. [19:29:33] 3CirrusSearch, MediaWiki-Core-Team: Prefix search containing only a namespace finds weird results - https://phabricator.wikimedia.org/T76350#821968 (10Manybubbles) 5Open>3Resolved Closed by commit rECIR9686b53e4cdd. [19:33:09] <^d> manybubbles: Did you get any e-mails from phabricator about Cirrus the repo? [19:34:16] 3MediaWiki-Core-Team: Sudden drop in number of articles on nowiki on Nov29 (by 34k articles) - https://phabricator.wikimedia.org/T76356#822023 (10Ironholds) Went through the most recent XML dump (october) and found <=13,470,713 revisions.* *Grabbed the archive and rev_deleted revisions from the db and added tho... [19:34:49] 3MediaWiki-Core-Team: Sudden drop in number of articles on nowiki on Nov29 (by 34k articles) - https://phabricator.wikimedia.org/T76356#822025 (10Ironholds) 5Invalid>3Open a:5Ironholds>3None [19:35:14] <^d> Crap there they are. [19:35:25] ^d: ? [19:35:28] oh my [19:38:13] <^d> manybubbles: Are you still getting anymore or did they stop? [19:38:44] ^d: have 43 in inbox. look like they've stopped [19:39:54] Hm.. Is there a reason to use wfTimestamp() or wfTimestamp( TS_UNIX ) instead of time() ? [19:40:20] Main issue being that wfTimestamp returns a string and I need ints. Right now in resourceloader we often either forget to cast and get weird issues or we intval() around it, which is horrible. [19:40:28] returns the same thing from what I can see [19:41:18] use time() if unix time is what you want, no advantage to wf* stuff [19:41:43] <^d> Yeah, we should probably even do that more than we do. [19:42:46] OK. Cool [19:43:17] omg [19:43:24] There is a wfTimestampNow [19:43:27] and it takes no arguments [19:43:39] return wfTimestamp( TS_MW, time() ); [19:43:39] !!!!! [19:43:45] Oh wait [19:43:48] that's TS_MW [19:43:51] k.. [20:11:16] * AaronS chuckles at https://www.wikidata.org/wiki/Q7744702 [20:17:37] stuff's being imported into diffusion? whee [20:18:14] MaxSem: yeah [20:20:46] https://phabricator.wikimedia.org/daemon/ [20:20:51] no wonder search was lagging :P [20:22:45] <^d> Can non-admins see that? [20:22:47] <^d> Oh cool [20:22:58] <^d> I'm jealous. We should have that much insight to our jobqueue :) [20:23:39] <^d> Yeah, I did all extensions as a first batch. [20:25:54] Couldn't we have a specialpage that shows the job queue status? There's a maintenance script right? [20:26:29] bd808: you mean in mediawiki? yeah, there maintenance script [20:26:45] Hm.. how can a size reduction from 26.8KB to 25.9KB be the same as Content-Length going from 5456 to 5415? [20:26:53] 236,617 jobs [20:27:09] I made a change to a js file and in the browser it its ~ 50 length shorter and yet almost a kilobyte? What am I missing. [20:27:43] They're " characters (e.g. "123" -> 123; before gzip) [20:27:44] ^d: might want to lower the priority on those git tasks [20:27:47] ah crap, that's it. [20:27:52] the size is before gzip, the content-length is after. [20:28:19] <^d> manybubbles: what's lagging? [20:28:40] ^d: apparently search is not being updated? legoktm mentioned it above [20:28:59] manybubbles: yeah on phab, because it's indexing all the git repos that just got added [20:29:08] <^d> It is. It's a job. Indexing is also throwing stuff in the search queue. [20:29:10] PhabricatorRepositoryGitCommitMessageParserWorker [20:29:39] bd808: it used to be visible on Special:statistics but it got removed because of how inaccurate it was and people would freak out going "omg the job queue is too high stop editing" but it's still exposed in the API [20:29:43] I see a priority column below though - maybe you can lower the priority on the commit jobs? [20:29:48] or maybe it doesn't matter [20:30:17] <^d> I don't have the option to set that. [20:30:34] Sounds like a Java class :P http://projects.haykranen.nl/java/ [20:30:41] bd808: https://en.wikipedia.org/w/api.php?action=query&meta=siteinfo&siprop=statistics "jobs": 12640508, [20:30:41] legoktm: huh. Is it inaccurate because the rate of change is high or because the algorithm is flawed? [20:31:09] PhabricatorRepositoryGitCommitMessageParserWorkerServerContextDescriptorStub [20:31:12] I don't remember, this was back when all the jobs were stored in an sql table and I think it just used estimateRowCount [20:31:22] it might be better with redis? [20:31:26] *nod* that would be sketchy [20:32:07] It should be point in time exact with redis I would think. *shrug* I haven't poked at the job queue much at all [20:32:12] <^d> mwscript showJobs.php --wiki foowiki --group [20:32:18] <^d> --group is useful [20:32:26] yeah. I've used that in beta [20:32:40] to find out what was stuck [20:33:28] bd808: with the redis job queue the numbers are cached for a few seconds. [20:33:42] if you continually poll it it won't change but every 10 seconds and it changes a ton [20:34:06] I was mostly just responding to ^d's claim of jealously at having job queue status visibility [20:34:22] THat seems like an important thing to know operationally [20:34:57] <^d> Yeah, but phab has a pretty dashboard :) [20:35:35] It shouldn't be that difficult to expose showJobs.php --group in a wiki page [20:36:26] on https://phabricator.wikimedia.org/daemon/task/2707616/ I have options to "cancel task" or "free lease"...is that expected? [20:36:34] <^d> ummm. [20:37:26] <^d> i hope that's a bait-and-click. [20:37:43] I see the links too. Sometimes phab shows you things and then says "you can't do this" when you click though [20:37:46] <^d> I can't configure permissions on daemon app, that's a default thing [20:38:03] is there one we can try that wouldn't be sad to lose? [20:38:08] <^d> Try that one. [20:38:13] <^d> That mta job has failed 203 times. [20:38:21] <^d> (I was thinking of dropping it anyway) [20:38:33] heh. canceled [20:38:45] it asked me if i really wanted to first [20:38:45] o.o [20:38:49] <^d> .... [20:38:53] <^d> that seems bad. [20:39:16] another place that phab shows it was/is a behind the firewall tool [20:40:35] Krinkle: is it expected that there are vastly different qunit test results on the same wiki based on running in firefox vs chrome? [20:41:01] * legoktm finds food [20:41:14] * ^d is pinging chase & co [20:41:19] <^d> see if we need to upstream that. [20:46:30] bd808: No [20:47:44] Krinkle: Looks like it might be timing related somehow. I get a failure with "Expected 50 assertions, but 32 were run" and then many other failures saying expected N got N+1 in firefox [20:49:01] The test "test.mediawiki.qunit.testrunner: Loader status" also hates my wiki because I have VE loaded [20:49:39] * bd808 does not like tests that rely on externally controlled state [20:52:46] bd808: They don't. I have VE installed and the tests pass fine [20:52:57] bd808: The error is most likely genuine. One of the modules is broken. [20:53:38] expected: [] got: [ "ext.visualEditor.mwlink", "ext.visualEditor.mwmeta", "ext.visualEditor.test" ] [20:54:15] bd808: OK. Check your php error log (or look at the load.php request) for the stractrace of the exception thrown by resourceloader. [20:54:32] most likely a missing file due to an outdated git submodule [20:54:53] bd808: Does VE work in general on this wiki outside tests? [20:55:41] latest master of mediawiki, mwext-visualeditor and submodules updated [20:55:45] Krinkle: Good question. It's quite likely that my extensions are a bit out of date [20:56:15] I update core regularly but often neglect extensions [20:57:00] The test did pass when I disabled VE. I'll update everything and try again. [21:20:47] 3MediaWiki-Vendor, MediaWiki-Core-Team: Security review of Plancake e-mail parser library - https://phabricator.wikimedia.org/T74956#822613 (10csteipp) I thought there was more to it than just https://github.com/floriansemm/official-library-php-email-parser/blob/master/PlancakeEmailParser.php, but it looks like... [21:39:45] <^d> bd808: Ooh. http://tech.rhealitycheck.com/visualizing-mongodb-profiling-data-using-logstash-and-kibana/ [21:39:58] <^d> (swap mongodb for "thing you actually want to profile") [21:40:26] <^d> I totally think you can massage profiling data into a logstash setup. [21:40:51] Getting the data into logstash is easy [21:40:59] then we need a visulaization for it [21:41:13] I was looking at the classes that xhprof ships [21:41:35] there is a plugin interface to change the sotrage backend [21:42:10] but the project is lame to extend. There are several other projects flaoting around on github to make easier classes to work with [21:42:33] * bd808 reads the link [21:44:17] ^d: There is role:elk now for vagrant to give you a logstash server + kibana + monolog to play with [21:44:38] It configures your wiki to log to logstash via the redis queue [21:44:55] 3MediaWiki-Vendor, MediaWiki-Core-Team: Security review of Plancake e-mail parser library - https://phabricator.wikimedia.org/T74956#822655 (10csteipp) 5Open>3Resolved >>! In T74956#822613, @csteipp wrote: > It does pass user controlled data to the in_charset of iconv. I don't think there should be an issue... [21:45:05] You could take that and play with profiling data and dashboards [21:48:26] ^d: I was kind of thinking about adding elasticsearch backend support to this -- http://xhprof.io/ [21:50:00] <^d> I saw that. [21:50:05] <^d> I was bummed it was mongodb [21:50:08] <^d> :p [21:50:33] That one is mysql I thought. [21:50:44] there's another one that's mongo [21:50:56] <^d> oh yeah [21:51:03] <^d> xhprofui [21:51:09] <^d> No, xhgui [21:51:18] yeah -- https://github.com/perftools/xhgui [21:52:26] This seems like a space that could be "un-fragmented" by one good project, but we had the too many sideprojects discussion already [21:53:05] It's probably really just a variant of the "show me metrics" problem [21:53:55] where the solutions are almost always knocked together using everything in the free tier of heroku and then abandoned [22:25:50] [14:15:55] can someone confirm some bash magic for me? [22:25:50] [14:16:04] I have a list of a bunch of emails, with dupes [22:25:50] [14:16:15] I want all the dupes removed, so I did sort filename | uniq > out [22:25:56] the results seem to good to be true [22:29:50] hmm [22:29:53] I have a bug somewhere :/ [22:30:53] Krinkle|detached: \o/ thanks for merging the user options patch. i'm stoked to see the impact. [22:32:13] legoktm: that should work. `sort -u` does both at once too [22:32:40] ok [22:32:48] I think I have a bug somewhere earlier on in my list generation [22:33:46] I have a file for each wiki, an email address on each line that is unattached and unconfirmed [22:34:28] we only want to email users who are on those lists twice [22:35:03] so I ran a python script to run `comm` on every pair of possible wikis, which is where I think it has gone wrong. http://paste.fedoraproject.org/157101/78188301/ [22:35:18] cat * | sort | uniq -c [22:35:28] * legoktm cries [22:36:00] trying... [22:36:04] shell can do a lot of stuff really easily if you only find out the right filters :) [22:36:29] and now i just remove the lines that start with '1'? [22:36:55] yup; grep -Ev '^1' [22:37:13] well that would get 10 ... too [22:37:21] '^1 ' probably [22:37:31] <_joe_> hey, have you read the thread on robots.txt on ops@? [22:37:36] or use awk [22:37:44] <_joe_> that's the reason why google is indexing the FR banners [22:37:52] <_joe_> brad figured what the issue is [22:38:05] doh [22:38:17] not why, but why they haven't re-indexed? [22:38:20] right? [22:38:36] why they wandered in to the wrong space I bet [22:38:45] <_joe_> exactly [22:38:53] huh [22:38:58] <_joe_> so Jeff Green is opening a ticket for core [22:39:11] should we just null edit Robots.txt on enwp for now? [22:39:22] yes! [22:39:24] <_joe_> this is a heads-up that this is probably very important [22:39:28] <_joe_> legoktm: do that [22:40:14] The robots code is in mediawiki-config right? [22:41:05] https://en.wikipedia.org/w/index.php?title=MediaWiki%3ARobots.txt&diff=636814769&oldid=583410902 [22:42:23] that timestamp code is dumb [22:43:03] how would I use awk? the grep '^1 ' isn't working [22:43:32] legoktm: LMGTFY -- http://stackoverflow.com/questions/15316569/linux-print-all-lines-in-a-file-not-starting-with [22:43:41] :D [22:46:37] ok, final command was [22:46:39] cat *.csv | sort | uniq -c | awk '!/^ *1 / { print; }' > nodupes.list [22:47:02] and we're at 125970 emails woo [22:47:02] oh because the numbers are right justified [22:47:07] 3MediaWiki-Core-Team: robots.txt Last Modified header is wrong - https://phabricator.wikimedia.org/T76915 (10Jgreen) 3NEW p:3Unbreak! [22:47:18] much better than 2 million! [22:47:23] much [22:47:43] <_joe_> also, robots.txt doesn't get purged from varnish [22:47:48] <_joe_> apparently [22:48:06] _joe_: ugh. [22:48:30] <_joe_> curl -I http://en.wikipedia.org/robots.txt?sddwwqwvfw gives the right header, but the clean url doesn't [22:48:30] and we'd want to purge on git change or wiki change I suppose [22:48:50] <_joe_> that too [22:49:03] ffs [22:49:06] <_joe_> this is pretty complicated to get right it seems :P [22:49:24] we send 'Cache-Control: s-maxage=3600, must-revalidate, max-age=0' [22:49:32] so once an hour? [22:49:41] does varnish honor that? [22:49:53] <_joe_> I think so, yes [22:50:05] so not so bad then [22:50:11] <_joe_> no [22:50:17] we just need to fix up the date header [22:50:22] <_joe_> 1h resolution is "good enough" [22:50:23] * bd808 is looking at it now [22:50:24] <_joe_> yes [22:50:30] <_joe_> oh thanks [22:51:14] <_joe_> ori: all the appservers have hyperthreading enabled, the ~ 60 of them I rebooted are on the latest package+kernel as well [22:51:24] _joe_: i love you, man. [22:51:26] <_joe_> so basically all the larger machines [22:51:30] that's great! thank you. [22:51:37] how many are left? [22:51:49] <_joe_> without ht? [22:51:50] <_joe_> no one [22:51:53] <_joe_> all have ht [22:51:57] and hhvm package? [22:52:08] <_joe_> uhm I didn't count [22:52:12] no worries [22:52:20] <_joe_> but the difference between -4 and -5 is really minimal [22:52:32] <_joe_> the HT factor was quite important OTOTH [22:52:45] <_joe_> so I focused on that [22:52:55] yeah [22:53:00] makes complete sense [22:53:12] the cpu graphs are pretty awesome [22:53:20] <_joe_> now I'm off, and on monday I'm not sure I'll attend mwcore, it's bank holiday in Italy [22:53:35] <_joe_> so I would be technically off [22:53:48] seems you were right about the impact of checksymlink=off too [22:53:55] it's there, but it's not huge [22:54:09] <_joe_> we should try at higher load [22:54:09] cool, enjoy your long weekend [22:54:16] <_joe_> like, in api [22:54:26] i should have tidy fixed up by monday [22:54:32] i ditched EZC, doing a simple HNI extension instead [22:54:32] <_joe_> the appserver pool is _really_ bloated [22:55:08] <_joe_> good work on that :) [22:55:39] <_joe_> it's probably also going to be faster [22:55:48] nominally, perhaps [22:59:20] 3MediaWiki-General-or-Unknown, MediaWiki-Core-Team: robots.txt Last Modified header is wrong - https://phabricator.wikimedia.org/T76915#822775 (10greg) [23:01:10] ^d: https://gerrit.wikimedia.org/r/#/c/177940/1/w/robots.php,cm I spy a $wgArticle.... [23:01:34] ew [23:01:35] $wgTitle = Title::newFromText( 'Mediawiki:robots.txt' ); [23:01:35] $wgArticle = new Article( $wgTitle, 0 ); [23:02:19] legoktm: Is there a better way to do that dance? [23:02:31] * bd808 has the file open laready [23:03:38] I'm pretty sure we don't need the Article object [23:04:00] and they're named bad things [23:29:55] legoktm: How do you get the article text without an article object? [23:30:09] Revision::newFromTitle( $title ); [23:30:30] $rev->getContent()->serialize( CONTENT_FORMAT_something ); [23:31:34] And that's better than Artcle::getContent()? I really don't know much about actual wiki code bits [23:32:06] * @deprecated since 1.21; use WikiPage::getContent() instead [23:32:12] (that's Article::getContent()) [23:32:15] heh [23:33:04] confusingly, Article::getContent() returns a string, while WikiPage::getContent() returns \Content [23:33:19] but since we just want the text, we can skip WikiPage entirely and use Revision [23:39:46] legoktm: content->getNativeData() ? [23:40:02] yeah [23:40:11] k [23:55:14] blerg. you can't get the unix timestamp from a revision, just the MW timestamp [23:57:14] @deprecated: use WTF::getWTFrecursive() [23:57:49] * bd808 is now learning about the insanity of MWTimestamp [23:58:20] That's a crap ton of regex checks for something that the caller could/should know [23:58:27] happy friday bd808 [23:59:29] MaxSem: This code is ... useful but really really wasteful