[00:02:01] bd808: https://philsturgeon.uk/blog/2012/03/packages-the-way-forward-for-php/ [00:03:56] ori: That's a great one and ... basically the gist of what I'm trying to say I think [00:10:10] bd808: http://nelm.io/blog/2011/12/composer-part-1-what-why/ / http://nelm.io/blog/2011/12/composer-part-2-impact/ [00:11:06] and http://bergie.iki.fi/blog/composer_solves_the_php_code-sharing_problem/ [00:11:34] you weren't specifically asking for php/composer-related posts, but the terrain is too vast otherwise [00:13:39] fair enough. I'll read those and see if I can find the sort of thing I'm looking for. [00:13:50] bd808: lastly: "Code Reuse in Open Source Software" is oft-cited and it looks like it has all the right talking points [00:14:22] TikiWiki is one of their examples [00:14:57] cool. the research papers I had dredged up myself were too old to be compelling [00:16:29] http://neverworkintheory.org/2012/05/03/a-review-of-code-simplicity.html see point 5 [00:18:35] TimStarling: Your comment on https://phabricator.wikimedia.org/T85428, do we already log that? Or were you thinking add more logging in? [00:18:57] we don't log it [00:19:35] I was thinking, add the logging, then delete the memcached key a few times to see if anything poisons the cache as it refills [00:19:51] greg-g: Hey no submarining. I know what the answer is; now I just need facts to support it. [/me changes channel to Fox News] [00:20:33] bd808: I just found this site recently and I'm trying to go through the backlog [00:28:31] bd808: http://dl.acm.org/citation.cfm?id=1824763 :"we argue that OSS community networks characterized by small-world properties would positively influence the productivity of the member developers by providing them with speedy and reliable access to more quantity and variety of information and knowledge resource ... we found a statistically significant relationship between small-world properties of a community and the technical and co [00:28:31] mmercial success of the software produced by its members" [00:35:22] * bd808 reads http://www.academia.edu/8194566/Choosing_an_Appropriate_Task_to_Start_With_In_Open_Source_Software_Communities_a_Hard_Task [00:52:29] TimStarling: https://gerrit.wikimedia.org/r/#/c/182973/ [00:52:59] If you +2, I'll deploy and test that tommorrow. I agree, it sounds like cache [01:49:32] ori: can you restart qa-morebots? Everybody else in the morebots project seems to be offline or afk [02:18:43] bd808: sure, just a second [03:04:34] 3MediaWiki-Core-Team, Performance-Metrics-Dashboard: Parsoid performance analysis - https://phabricator.wikimedia.org/T85870#956993 (10tstarling) Parsoid API requests account for about 44% of the request rate at the API cluster. Parsoid's action=expandtemplates accounts for 31% of total API wall clock time as ac... [03:50:52] 3MediaWiki-Core-Team, Performance-Metrics-Dashboard: Parsoid performance analysis - https://phabricator.wikimedia.org/T85870#957003 (10tstarling) The most convincing argument in favour of not batching seems to be for latency optimisation. But latency is irrelevant for ParsoidCacheUpdate jobs. On wtp1001, I captu... [04:47:56] 3MediaWiki-Core-Team: Parsoid performance analysis - https://phabricator.wikimedia.org/T85870#957007 (10tstarling) [04:48:12] 3MediaWiki-Core-Team: Parsoid performance analysis - https://phabricator.wikimedia.org/T85870#956387 (10tstarling) I don't think that dashboard thing is related. [09:39:59] how does one get into the #wmf-nda group in phabricator? About half the links I've tried to open today tell me that I don't have access :/ [12:19:53] Nikerabbit: ask in #wikimedia-devtools :_D [12:20:20] Nikerabbit: the page at https://phabricator.wikimedia.org/tag/wmf-nda/ states that ops add people to it [12:24:01] hashar: are ops in devtools? ;) [12:24:16] Nikerabbit: phabricator folks are for sure :] [12:24:32] Nikerabbit: anyway the page seems to mention that ops are in charge of adding folks to thenda group [12:25:30] off for lunch [12:27:36] ok [14:35:22] _joe_: hey! feeling better? [14:36:01] if so please have a look at https://docs.google.com/a/wikimedia.org/spreadsheets/d/1MXikljoSUVP77w7JKf9EXN40OB-ZkMqT8Y5b2NYVKbU/edit#gid=0 [14:36:44] * ^d yawns, mutters something about the sun not being up yet [14:36:45] its how we're going to make the ultimate decision later today. That meeting is pretty late for you so I imagine you could put what you need on the spreadsheet [14:37:13] ^d: sometimes its nice to be on this coast [14:39:06] <^d> Not today! All cold and snowy :p [14:42:34] 3MediaWiki-Core-Team, MediaWiki-extensions-TitleBlacklist: Title blacklist intermittently failing, allowing users to edit things they shouldn't be able to - https://phabricator.wikimedia.org/T85428#957362 (10Anomie) I poked at this again this morning, and happened to have an affected eval.php. `TitleBlacklist::s... [14:44:28] Nikerabbit: Any thoughts on https://phabricator.wikimedia.org/T85428#957362, particularly the part about MessageCache? It looks like you're the last person to touch that code path in r71412. [14:48:07] anomie: ugh [14:48:25] anomie: in general, things will fail horrible if message cache loading fails [14:49:10] anomie: just this kind of stuff, invalid data get cached in other places because we return incorrect data instead of stop processing [14:49:12] Nikerabbit: If things will fail horrible and we can't make it halfway sane, can we just throw an exception or fatal somehow? [14:49:23] anomie: there is a chance it is related to some semirecent changes to messagecache, but unless proven I blame the above [14:49:27] * anomie prefers clean explosions to cache poisoning [14:50:12] anomie: I think Tim (long time ago) favoured not bringing the site down just because of that [14:50:53] anomie: it would be interesting, though not necessarily sane, expose these failures in more obvious way to see the magnitude. Perhaps there is some thing that can be fixed to reduce the failures so that how we handle it doesn't matter so much. [14:51:47] Nikerabbit: I also pinged in the ops channel about the memcached errors, but MessageCache returning the default message rather than hitting the DB seems like a bug to me too. [14:53:02] anomie: hmm I don't remember exactly how the fallbacks work, there are some pool counters to avoid servers melting... and usually things blow up if one thread gets stuck while loading the messages [14:53:29] I'm just guessing based on my experience on past (very old) events now [14:54:12] Nikerabbit: Well, if nothing else, please fill in an analysis of the MessageCache behavior in that bug, and we'll see what Tim says about things. [14:55:04] anomie: with analysis do you mean what I just rambled or me actually looking at the code? [14:56:23] Nikerabbit: Whatever seems useful as to why MessageCache does what it does in the face of the memcached failure, and why you don't think falling through to the code that loads the Revision there would be better than the status quo. [14:57:44] anomie: I don't really have opinion on the last part: I don't know if hundreds or thousand of queries to fetch messages will make db servers unhappy [14:58:44] I just figure more information is better [15:02:30] anomie: I'm afraid I have to defer anything that needs time and effort until Thursday [15:04:16] <^d> manybubbles: We avoided exposing other sorts before because we thought it would just confuse people on Special:Search. What do you think about just exposing them via the search api? [15:05:06] ^d: sorts are pretty nasty from a memory usage standpoint (its weird, not sure why really) so we'd have to do some performance investigation first. [15:05:11] <^d> Power users/bot owners/etc are interested in using search to build lists of things, and being able to sort the results is easier than having to either post process them or (potentially worse) make a second query to sort them. [15:05:53] <^d> "I want to sort by most recent edit date" came up again [15:36:18] * anomie sees we suddenly have 7 "TitleFactory" classes. [15:36:53] * anomie corrects that: 5 factories and 2 interfaces [15:37:33] Do we have a TitleFactoryFactory yet? [15:37:50] I don't see one [15:50:30] hashar: About? [15:53:58] Reedy: helllo [15:54:20] ohai [15:54:20] https://gerrit.wikimedia.org/r/#/c/182857/ [15:54:26] I'm removing an entry point that a test uses... [15:54:47] Wondering if I should just remove it from the test (obviously needs doing, maybe pre merge) [15:54:56] And whether I should just leave an empty stub in place instead.. [15:56:21] Reedy: the Jenkins/CI extension loader expect the entry point extensions/LabeledSectionTransclusion/LabeledSectionTransclusion.php [15:56:21] Oh [15:56:21] Yeah [15:56:21] I just looked at that [15:56:21] maybe there is a symlink [15:56:26] noticed the request in that file [16:03:02] Reedy: great [16:04:03] Reedy: The Bug: and Change-Id: fields should not be separated by an empty line [16:05:53] and looks like LabeledSectionTransclusion needs some love :-( [16:07:49] staticness [16:08:49] hashar: yay, both pass now [16:12:23] \O/ [16:12:52] off be back for the releng checkin in ~ 50 minutes [16:17:27] <^d> manybubbles: I wonder if we could put a stub in place for MWSearch. Just have it wrap Cirrus and do Cirrus stuff, but call itself the old thing so srbackend=LuceneSearch still works. [16:17:47] I'm not sure its worth it though [16:17:59] <^d> prolly not [16:21:23] manybubbles: Thanks for all the blog feedback. I'll be taking another shot today and will try not to bury the lead behind walls of boring. :) [16:21:33] bd808: I'm not doing much better myself with the search blog post. which we just let sit for forever [16:23:00] It's hard to switch gears from code to prose (at least for me) [16:29:26] <^d> I forgot how to write when I finished school. [16:59:59] 3MediaWiki-Core-Team, Security-Reviews: Security review of community extensions: Extension:AtomExporter, Extension:DownloadCounter, Extension:PasswordProtected - https://phabricator.wikimedia.org/T787#957674 (10csteipp) [17:00:49] <_joe_> manybubbles: not really feeling better, no. Today is also a bank holiday in Italy [17:00:58] cool [17:01:33] 3MediaWiki-extensions-ContentTranslation, ContentTranslation-Deployments, MediaWiki-Core-Team: Content Translation Beta Feature security review - https://phabricator.wikimedia.org/T85686#957675 (10Jsahleen) Chris, We have a few questions about this security review. # Is a security review mandatory for deploy... [17:01:42] <_joe_> but I'll take a look tomorrow [17:21:34] 3MediaWiki-Core-Team, MediaWiki-General-or-Unknown: HttpError should not be logged as an unhandled exception. - https://phabricator.wikimedia.org/T85795#957697 (10JanZerebecki) I can't figure out why currently these show up in logstash as type:HttpError instead of type:exception. Any ideas? [17:23:23] 3MediaWiki-extensions-ContentTranslation, ContentTranslation-Deployments, MediaWiki-Core-Team: Content Translation Beta Feature security review - https://phabricator.wikimedia.org/T85686#957698 (10csteipp) >>! In T85686#957675, @Jsahleen wrote: > Chris, > > We have a few questions about this security review. >... [17:28:05] 3MediaWiki-Core-Team, MediaWiki-General-or-Unknown: HttpError should not be logged as an unhandled exception. - https://phabricator.wikimedia.org/T85795#957705 (10bd808) >>! In T85795#957697, @JanZerebecki wrote: > I can't figure out why currently these show up in logstash as type:HttpError instead of type:excep... [17:31:36] 3MediaWiki-extensions-ContentTranslation, ContentTranslation-Deployments, MediaWiki-Core-Team: Content Translation Beta Feature security review - https://phabricator.wikimedia.org/T85686#957716 (10Jsahleen) Chris, Here are some links you can look at: https://www.mediawiki.org/wiki/Content_translation/Technical... [17:36:03] 3MediaWiki-Core-Team, MediaWiki-General-or-Unknown: HttpError should not be logged as an unhandled exception. - https://phabricator.wikimedia.org/T85795#957727 (10JanZerebecki) Thx. No wonder searching for type in that file didn't find where it was set to HttpError. [19:05:04] csteipp: did you have a chance to submit the monolog thing? [19:13:26] 3MediaWiki-Core-Team: Parsoid performance analysis - https://phabricator.wikimedia.org/T85870#957897 (10Aklapper) [19:16:29] btw anomie: https://www.mediawiki.org/w/index.php?title=API:Data_formats&diff=1345565&oldid=1225034 :P [19:29:08] 3MediaWiki-Core-Team, Security-Reviews, Phabricator: Aphlict security review - https://phabricator.wikimedia.org/T1286#957936 (10Qgil) There is ongoing work to get rid of Flash and use WebSockets. Should we resolve this task as Stalled in the meantime? See https://secure.phabricator.com/T6559 [19:35:20] legoktm: Let me do that right now [20:21:52] 3MediaWiki-Core-Team, Continuous-Integration, Librarization: Set up composer validate job for operations/mediawiki-config - https://phabricator.wikimedia.org/T76621#958050 (10Legoktm) 5Open>3Resolved I filed T85947 to convert the entire repository to use composer based testing. [20:37:48] ^demon|lunch: can I has https://gerrit.wikimedia.org/r/wikidata/gremlin ? it is time to create a gerrit repository for us. [20:37:59] or a phab repository if we want to try that [20:38:15] <^demon|lunch> Not ready for native to phab repos. [20:38:19] <^demon|lunch> Sec, will create. [20:38:31] cool [20:39:26] <^demon|lunch> Created, ldap/wmf is initial owner so you can tweak as needed. [20:49:50] ^demon|lunch: that means anyone at wmf can edit it and stuff? [20:49:58] <^demon|lunch> yep [21:23:48] writing this blog post is making me rethink MWLogger [21:24:22] what about it? [21:24:37] I think maybe we should ditch that entry point and instead just use Psr\Log\LoggerInterface directly [21:24:56] wasn't that my argument? :) [21:24:58] it's jsut a shell wrapper today so it would be easy to do [21:25:09] was it? I know Tyler pointed it out [21:25:20] it was. anyhow, +1. [21:26:08] If I can finish the blog post then I can start the patch. :) [21:26:24] ^demon|lunch: I can't give jenkins the permission to submit. or give it to myself [21:28:34] TimStarling: do you have any tests for the different pcre cache kinds in your pull request? [21:28:45] I'm going to change how it handles accessors and want to make sure I don't break stuff [21:28:53] I could just run our normal tests with the default changed I guess [21:44:04] ^demon|lunch: https://gerrit.wikimedia.org/r/#/c/181248/ [22:02:47] ^demon|lunch: hey! why can't i add jenkinsbot to permissions in my new project? [22:02:55] I can't see him when I type his name [22:07:53] TimStarling: If you happen to be available in the next little while, I can give you a little more detail about what I found on that TitleBlacklist bug (T85428). I asked Nikerabbit about the MessageCache part of it earlier, but he wasn't confident that fixing the fallback logic there wouldn't melt the DB or something like that. [22:20:27] hello [22:22:38] MessageCache could use the database if it was protected with a pool counter [22:22:58] originally it used the database but it routinely took the whole site down [22:23:21] but with PoolCounter it should be possible [22:25:01] eww, time to put it into mongodb? it's webscaaaaleeee! :P [22:28:13] seriously though, is memcached overloaded? [22:29:51] unlikely [22:30:45] actually if you look at that last log line, you see it did actually wait politely for another thread to load from the DB [22:30:46] but the global cache was still empty, so it didn't know what to do [22:31:02] it kind of expects keys set to memcached to at least last a few hundred milliseconds [22:31:17] that is also true of caches that use PoolCounter [22:32:06] and it tried to acquire the status key so that it could load from the DB itself, but that failed too of course [22:35:50] 3MediaWiki-Core-Team, MediaWiki-extensions-TitleBlacklist: Title blacklist intermittently failing, allowing users to edit things they shouldn't be able to - https://phabricator.wikimedia.org/T85428#958352 (10tstarling) It can't load from the DB with uncontrolled concurrency, so it tries to acquire a global lock... [22:36:55] $wgMemCachedTimeout = 0.25 * 1e6; // 250kus (a quarter of a second). [22:36:59] wtf is a kus? [22:37:44] kilomicroseconds? [22:37:45] lol [22:37:59] MaxSem: got to be [22:38:19] I'm sure there is a clause in the SI standard suggesting the death penalty if anyone every tries to chain prefixes [22:38:23] looks like someone wants the number in microseconds [22:38:37] just if they try to chain them backwards like that [22:39:55] If I send an email to ops and wikidata-tech and other people when they reply-all will they get bounces because ops is a private list? [22:40:09] or is it just that ops isn't archived publicly and anyone can send to it [22:40:34] it also has an electrocution while being hanged over fire for using imperial units, but it's not stopping anyone here in the us [22:41:31] manybubbles: non-member posts to the ops list are held [22:41:51] except from *.wikimedia.org [22:43:31] * anomie googles "kilomicrosecond", and apparently a bunch of wifi stuff uses that unit to mean "1024 microseconds". But wouldn't that be kibimicroseconds? [22:45:59] TimStarling: `forceprofile` became an unintended casualty of xhprof getting disabled. I talked about it with Aaron and we think Xenon can replace the random profiling of a sample of requests, but we still need xhprof for single-request profiling via forceprofile. Faidon was a bit worried about forceprofile being a DDoS vector and asked that you sanity-check. [22:48:32] why was xhprof disabled again? [22:48:41] is there a task about that? [22:48:57] yeah, i'll dig it up. it crashed periodically. it was reliable enough for the occasional forceprofile. [22:50:02] if it's reproducible then we can fix it [22:50:13] if it's not reproducible, it's presumably not a DoS vulnerability, right? [22:51:02] Faidon's concern didn't have anything to do with the crashes, as I understand it -- just the fact that profiling was expensive and that you could generate load easily by flooding us with forceprofile reqs. [22:51:27] how expensive? [22:51:37] there isn't a task, by the way -- I asked for a postmortem in the thread 'Profiling-related HHVM crashes' on the ops list and never got one, but there's a trace. [22:52:36] a single profiled request is several times slower than it would be without profiling, IIRC, but it doesn't produce measurable load on the system as a whole. [22:53:30] could use a shared password [22:53:37] forceprofile=xyzzy [22:54:26] yeah. it'd be a bit annoying to have to dig up. i thought maybe a special request header. that way one could set it permanently using any one of the popular extensions for header manipulations for ff or chrome. [22:55:12] the header would enable debugging generally, but you'd still need forceprofile=1 to enable profiling [22:55:15] I mean a short an obvious password, e.g. literally xyzzy [22:56:19] meh. that sort of dinky security isn't any better than the kind we get from forceprofile being obscure and mostly undocumented, but sure. [22:56:52] But you can change it if it starts to be abused [22:56:54] well, the other option is to leave it how it is, and add a password the first time it is used for a DoS attack [22:57:16] i'll add a password, csteipp has a point [22:59:14] url passwords are sooo easy to accidentally disclose... [23:00:14] 3MediaWiki-Core-Team, Wikimedia-Logstash, operations: Improve logstash - https://phabricator.wikimedia.org/T84895#958409 (10chasemp) p:5Triage>3Normal [23:00:46] what i'd really like to do is adapt the HHVM magic cookie varnish stuff and make it so a special header routes all of your requests to mw1017 [23:00:46] you know there are a couple of secret URLs in /w that have been there since about 2005 [23:01:00] so we could use that host to test any wiki, not just testwiki [23:01:38] then it wouldn't even need to be secret, because DDoSing mw1017 wouldn't do much harm [23:01:51] could do [23:04:18] 3MediaWiki-Core-Team, operations: Deploy multi-lock PoolCounter change - https://phabricator.wikimedia.org/T85071#958424 (10chasemp) p:5Triage>3Normal [23:04:42] 3MediaWiki-Core-Team, operations: Deploy multi-lock PoolCounter change - https://phabricator.wikimedia.org/T85071#938038 (10chasemp) p:5Normal>3Low This needs a better description of why and and what. [23:13:18] what should the policy be for handing out the password? I know folks like jackmcbarn use it. [23:13:59] so you are going to have a password? [23:14:17] if you do the mw1017 thing you could just leave the URL how it is [23:14:51] timed retry errors can be reproduced easily enough [23:15:06] yeah, ok [23:15:08] that makes sense [23:16:01] does any server have ddebs set up? [23:16:56] the repo? no, i don't think so. [23:17:30] ori: the only thing i use profiling for is scribunto, and isn't it totally separate from everything else that forceprofile controls? [23:17:59] yeah [23:19:10] ah, we do have libmemcached-dbg though [23:31:07] 3operations, MediaWiki-Core-Team: Come up with key performance indicators (KPIs) - https://phabricator.wikimedia.org/T784#958504 (10chasemp) Seems related to {T85829} but without an assignee can this really be high priority? [23:33:43] 3operations, MediaWiki-Core-Team, MediaWiki-ResourceLoader: Bad cache stuck due to race condition with scap between different web servers - https://phabricator.wikimedia.org/T47877#958510 (10chasemp) This seems important but has no assignee, can someone take this? Otherwise I will probably set to normal priorit... [23:54:26] 3MediaWiki-extensions-TitleBlacklist, MediaWiki-Core-Team: Title blacklist intermittently failing, allowing users to edit things they shouldn't be able to - https://phabricator.wikimedia.org/T85428#958568 (10tstarling) Actually, timed retry does appear to be disabled -- the error message is incorrect. An unrelat...