[00:26:52] _joe_ / ori : http://paste.tstarling.com/p/jCJUdt.html [00:27:28] what is special about those servers that makes them get memcached connection failures? [00:27:36] i was about to ask the same. [00:28:03] they were provisioned later. IIRC, we had an issue where nutcracker needed to be restarted at least once manually before it worked properly. [00:28:24] i'll poke around on mw1226 (picked randomly) [00:29:10] well picked, it happens to be the server I am already logged in to due to anomie previously reporting a connection failure on it ;) [00:29:26] hah! i'll pick another one [00:29:52] wat? you don't want a four-hand dissection? [00:30:13] "connection failure" most likely means connection refused [00:31:25] memcached handles timeouts separately, and it also misinterprets EAGAIN in a way which would probably hide EAGAIN errors [00:33:00] a few other errors such as EFAULT and ENETUNREACH will also be reported as "connection failure" [00:33:27] I have checked the uptime on nutcracker, it is long, so it's not restarting [00:34:31] [mw1205:~] $ stat -c %y /etc/nutcracker/nutcracker.yml [00:34:31] 2014-12-09 17:52:19.610694770 +0000 [00:34:31] [mw1205:~] $ ps -o "etime=" `pidof -s nutcracker` [00:34:33] 28-06:42:03 [00:34:35] yeah [00:34:45] I guess https://phabricator.wikimedia.org/T75949 was forgotten [00:35:06] so the only theory I have at the moment is that nutcracker's listen backlog is exceeded [00:35:48] I'm pretty sure FD limit exhaustion in the client would generate a different error [00:39:22] the listen backlog is 512 [00:39:59] which seems unlikely given that hhvm.server.thread_count = 128 but there you have it [00:42:33] not sure i understand. lsof on the nutcracker process doesn't show any open connections to memcached servers, and strace doesn't show it attempting to connect either, and the uptime of the process is close to the uptime for the host, so i think nutcracker was started with a default config which had a blank roster of backend memcached servers [00:43:21] http://paste.tstarling.com/p/hnUSAb.html [00:43:30] lots of connections [00:44:20] oh, right, my bad. i only read the origin-> bit as my eyes scanned down. [00:44:20] lots on mw1205 too [00:45:02] AaronS: that is a fair guess [00:45:37] it's probably not a nutcracker problem [00:46:02] how could it not be? [00:47:36] btw, $wgObjectCaches['mysql-multiwrite'] uses a different Memcached object than $wgObjectCaches['memcached-pecl'], right? [00:48:41] So that's 256 possible connections for 128 threads? Maybe there is other stuff like that too [00:49:38] the fact that those errors got much worse under hhvm is curious though [00:50:39] it's only some hhvm api servers [00:51:21] I just grepped for MEMCACHED_CONNECTION_FAILURE, actually it will also give that error if there is a server abort as well [00:51:31] but it won't do a timed retry in that case afaict [00:54:15] in netstat -s there is "TCPBacklogDrop"... [00:55:30] it's too small to explain all the errors [01:02:09] you can get detailed stats by telnetting to localhost, port 22222 [01:02:18] nutcracker serves stats on that port [01:04:26] yeah, that can't log backlog drops though [01:04:56] although I guess that's not relevant anymore [01:12:12] TimStarling: the set API hosts that hit this error is identical to the same is the set of hosts with weight=20 in pybal; the others have weight=15. So that supports the theory that this is caused by resource exhaustion of some kind. [01:12:30] s/is identical to the same is/is identical to/ [01:43:38] 3ContentTranslation-Deployments, MediaWiki-Core-Team, MediaWiki-extensions-ContentTranslation: Content Translation Beta Feature security review - https://phabricator.wikimedia.org/T85686#958661 (10csteipp) Ok, first issue. In poking around, the cx server is allowing reflected xss on that domain. Since it's on a... [01:48:31] TimStarling: do you think you'll have time to review the extension registration stuff anytime soon? Ideally I'd like to have that merged before the summit but I'm not sure if that's reasonable. [02:04:32] legoktm: you mean merge it? [02:05:02] I have done two code reviews already, probably the next step is a brief final code review and a merge [02:05:53] Yeah, I meant merging sorry [02:11:13] should be possible [02:30:50] 3MediaWiki-Core-Team: Investigate memcached-serious error spam that mostly effects HHVM API servers - https://phabricator.wikimedia.org/T75949#958708 (10ori) Affected hosts have a weight of 20 in PyBal; unaffected hosts have a weight of 15. [02:31:44] gotta run. tim, if you discover anything, update aaron's task -- i'm curious! [04:24:46] TimStarling: any more discoveries to share? fcntl(ptr->fd, F_GETFL, 0) could return -1 and set errno to EBADF if ptr->fd is invalid [04:26:26] that's consistent with libmemcached exiting the while (flags == -1 && (errno == EINTR || errno == EAGAIN)) loop (because the error is EBADF) and returning the error message you saw [04:29:21] [mw1205:~] $ sudo netstat --tcp | wc -l [04:29:21] 69107 [04:29:45] 62920 in TIME_WAIT [04:31:05] https://github.com/joyent/node/issues/1189#issuecomment-1381532 suggests tuning sysctl params to more aggressively reap connections helps [04:31:32] that comment is, curiously enough, from ed summers, who hacks on mediawiki code sometimes, iirc [04:35:12] 62920 is suspiciously close to 2^16 [04:36:17] (edsu is awesome) [04:39:52] yeah, he seems pretty cool [04:40:06] ori: I may have already asked this, but is there any way to make the edit performance dashboard start the y axis at 0? [04:40:35] i think so. which graph are you interested in, specifically? [04:40:49] this one: http://grafana.wikimedia.org/#/dashboard/db/Edit%20performance [04:41:14] well, with a specific time range [04:41:21] the latest data is kinda weird so it's naturally starting at 0 [04:41:38] but when I pick a couple days during teh transition it gives the picture I put in my blog post [04:43:10] TimStarling: http://paste.tstarling.com/p/mghfBH.html -- notice how all the ones up to mw1189 are in the 40-50k range but the remainder are 60-70k [04:43:15] swtaarrs: on it, just a sec [04:44:51] random internet commenters are acusing me of using "cheap tricks" :P [04:45:29] swtaarrs: click on the graph title, then edit, then 'axes & grids', then set 'min / auto' to 0 [04:45:36] swtaarrs: hah! lame! where? [04:45:52] nice, thanks [04:45:53] https://news.ycombinator.com/item?id=8848192 [04:46:00] also someone commented on the post itself [04:47:05] ah, that's not an unfriendly comment, as i read it. [04:47:13] yeah [04:47:27] I was selectively quoting it >_> [04:48:03] grafana did that by default, fwiw [04:48:41] yeah [05:06:11] ori: I'm not sure if EBADF is possible, but presumably it will fail somewhere in that function in the event of ephemeral port exhaustion [05:08:52] connect() should fail with EAGAIN judging by man connect [05:09:31] socket() won't fail because it doesn't need to allocate a source port [05:12:52] it will call connect_poll() , assuming that it is doing an asynchronous connection [05:22:47] it appears that libmemcached and twemproxy both support UNIX sockets [05:23:15] so that would be an option for solving it if it is ephemeral port exhaustion [05:23:31] the fcntl implementation is libc's, which wraps the system call, and which can return EBADF: http://www.gnu.org/software/libc/manual/html_node/Descriptor-Flags.html#Descriptor-Flags [05:23:46] yeah, I think I even have a task for that (switching to unix domain sockets) [05:24:09] we decided against it at the time just to keep the number of major changes down, but _joe_ might be up for it now [05:24:51] EBADF indicates that the file descriptor is not opened [05:25:07] but socket() opens it and if it fails, fcntl() won't be called [05:25:46] that's why I didn't think EBADF was possible [05:26:26] ah, hm [05:29:19] it should be easy enough to test, if we have a server that has twemproxy and that we don't mind breaking [05:30:21] osmium [05:30:28] the gift that keeps on giving [05:31:51] oh, i was looking at a different version of libmemcached, which is why i was fixated on the fcntl call [05:32:03] I'll do it [05:32:10] break osmium, that is [05:33:17] cool [05:34:42] oh, now i remember, i even wrote the patch to add unix domain socket support to the HHVM memcached client in anticipation of making the switch: https://github.com/facebook/hhvm/pull/2450 [05:40:30] well, it doesn't actually appear to fail [05:40:46] with this code: for ( $i = 0; $i < 65536; $i++ ) {$c = new Memcached; $c->addServer('localhost', 11211); $res = $c->getResultMessage(); if ( $res !== 'SUCCESS' ) { print "$res\n"; } } [05:43:02] it is a loop of connect/close as intended, generates plenty of TIME_WAIT sockets as intended [05:43:06] just doesn't fail [06:06:42] 3ContentTranslation-Deployments, MediaWiki-Core-Team, MediaWiki-extensions-ContentTranslation: Content Translation Beta Feature security review - https://phabricator.wikimedia.org/T85686#958847 (10santhosh) @csteipp, does this document https://www.mediawiki.org/wiki/Content_translation/cxserver help you to under... [07:03:03] <_joe_> hey I just woke up [07:03:17] <_joe_> the servers in Tim's list are api appservers [07:03:44] <_joe_> not "latest installed" ones [07:05:06] <_joe_> so it seems traffic-related more than anything else [07:06:57] <_joe_> well, I'll take a look later on if I have time [07:26:08] _joe_: they servers that error out have weight: 20 in pybal; those that don't have weight: 15, so it's resource starvation, possibly ephemeral ports [07:26:49] if you check the number of TCP connections in TIME_WAIT state on the servers in tim's list, and compare it to other API servers, you'll notice that the ones in tim's list have about 10 to 20k more connections open [07:26:54] <_joe_> probably ephemeral ports, yes [07:27:12] <_joe_> I already saw that for nagios checks btw [07:27:39] enabling tcp_tw_reuse or tcp_tw_recycle might help [07:28:45] <_joe_> using the unix socket to connect to twemproxy should also help :) [07:28:57] <_joe_> I'll take a look at both, maybe not today [07:29:21] * ori nods [07:29:50] <_joe_> I'm still not 100% ok [07:30:56] well, i think tim wanted to test twemproxy on unix sockets, so just let him do that, it sounds like he already got started on osmium [07:31:39] i'm off to bed, not feeling 100% myself [07:31:42] good night! [13:03:52] 3MediaWiki-Core-Team: HHVM log noise probably will be good GCI tasks - https://phabricator.wikimedia.org/T76733#959240 (10Aklapper) [13:04:31] 3MediaWiki-Core-Team: HHVM log noise probably will be good GCI tasks - https://phabricator.wikimedia.org/T76733#819066 (10Aklapper) [14:15:16] 3MediaWiki-extensions-TitleBlacklist, MediaWiki-Core-Team: Title blacklist intermittently failing, allowing users to edit things they shouldn't be able to - https://phabricator.wikimedia.org/T85428#959346 (10matmarex) [15:37:22] 3MediaWiki-Core-Team, Beta-Cluster: no log in deployment-bastion:/data/project/logs from "503 server unavailable" on beta labs - https://phabricator.wikimedia.org/T74275#959614 (10Aklapper) [15:42:23] So when are we dropping PHP 5.3 support? ;) [15:43:55] Reedy: After we migrated the misc. hosts [15:43:58] hopefully [15:44:12] heh [15:44:53] we can't be running MW on many misc hosts... [15:45:05] though, haven't tin and alike got hhvm anyway? [15:45:14] tin, terbium, snapshots*, tmh(?), image scalers(? [15:45:18] nope [15:47:30] <^d> We should go backwards instead. [15:47:35] <^d> Add more versions of PHP we support :) [15:48:36] 5.3 brought namespaces... so there's not much room [15:48:59] <^d> Namespaces are easier to remove than to add :p [15:49:28] :P [15:49:42] wmf14s changelog is gonna be huge, isn't it... [15:53:12] 12,608 bytes for just core [15:58:11] 3Phabricator, MediaWiki-General-or-Unknown, Project-Creators, MediaWiki-Core-Team: Allow to search tasks about MediaWiki core and core only (create MediaWiki umbrella project?) - https://phabricator.wikimedia.org/T76942#959825 (10Aklapper) p:5Normal>3High [16:48:42] Reedy: yeah, gonna be a fun day :) [16:49:20] greg-g: Just fixed up the changelog stuff so it reports/links bugs/tasks again [16:49:24] at least the ops quarterly review will be done by then :) [16:49:35] ah, right, phab phailure? [16:49:40] not really [16:49:47] just Bug: T12435 not matching etc [16:50:07] https://www.mediawiki.org/w/index.php?title=MediaWiki_1.25%2Fwmf11%2FChangelog&diff=1346476&oldid=1321380 [16:52:09] oh, right, that part [16:52:36] heh [16:53:16] (the getting it out of the commit message part, not the mw template part) [16:54:22] I'm not getting into the Task: 1234 discussion in commit summaries ;) [16:57:23] I read an essay a couple weeks ago about deployment freezes being a false security that really introduces more chaos unless there is an associated code introduction freeze. [16:57:45] heh [16:57:47] seems fair [16:58:54] in the magical future things will all be automated anyway right? ;) [16:58:55] bd808: yeah, I'd believe that, because now all of the (probably low) stress that would have happened over the holidays will happen all at once today :) [16:59:30] * bd808 actually doesn't know how much code change he caused is stacked up and waiting today [17:00:02] There's a lot in just core [17:06:20] 3MediaWiki-extensions-ContentTranslation, ContentTranslation-Deployments, MediaWiki-Core-Team: Content Translation Beta Feature security review - https://phabricator.wikimedia.org/T85686#960001 (10Arrbee) An update: we moved the deployment to next week (12th January - Monday). I hope that will ease the rush a bi... [17:14:31] ^d: ok, one last rebase for https://gerrit.wikimedia.org/r/#/c/174267/ would be nice [17:17:18] <^d> Heh, I can start on that :) [17:25:00] 15 extensions deleted from local clone so far [17:42:49] ori: Should I mass abandon my mod_proxy_fcgi changest in gerrit? [17:54:21] 3MediaWiki-General-or-Unknown, MediaWiki-Core-Team: HttpError should not be logged as an unhandled exception. - https://phabricator.wikimedia.org/T85795#960109 (10JanZerebecki) The goal is that everything that is logged can be easily differentiated according to its severity. Like fatals/errors/warnings/notices v... [17:54:45] 3operations, MediaWiki-Core-Team, MediaWiki-ResourceLoader: Bad cache stuck due to race condition with scap between different web servers - https://phabricator.wikimedia.org/T47877#960111 (10chasemp) @greg since this involves scap is this a release engineering thing? Would you be willing to kind of shepherd thi... [17:57:52] 3operations, MediaWiki-Core-Team, MediaWiki-ResourceLoader: Bad cache stuck due to race condition with scap between different web servers - https://phabricator.wikimedia.org/T47877#960136 (10bd808) I think this is really a bug about the fact that we don't depool/repool servers during a the rolling update process... [18:02:01] 3MediaWiki-extensions-ContentTranslation, ContentTranslation-Deployments, MediaWiki-Core-Team: Content Translation Beta Feature security review - https://phabricator.wikimedia.org/T85686#960146 (10csteipp) >>! In T85686#958847, @santhosh wrote: > @csteipp, does this document https://www.mediawiki.org/wiki/Conten... [18:05:13] I think I'll start branching [18:06:17] * Reedy remembers to revert live hacks first [18:08:30] protected static final String EMPTY_STRING = ""; [18:08:31] why [18:09:03] 3MediaWiki-General-or-Unknown, MediaWiki-Core-Team: HttpError should not be logged as an unhandled exception. - https://phabricator.wikimedia.org/T85795#960163 (10Legoktm) >>! In T85795#960109, @JanZerebecki wrote: > The goal is that everything that is logged can be easily differentiated according to its severit... [18:09:13] Does Java not have String.Empty or something? [18:11:52] <^d> manybubbles: Because it's java and we don't write java unless you abstract things. [18:12:09] <^d> :) [18:12:10] its even fucking protected [18:12:14] mind numbing crazy [18:12:15] "" [18:12:22] its the same fucking thing but shorter [18:12:32] and easier to read because its not a SHOUTING CONSTANT! [18:12:43] <^d> BUT HOW DO YOU KNOW WHAT ITS FOR THEN?!? [18:12:45] but, words! [18:12:47] <^d> UNLESS IT SHOUTS AT YOU? [18:20:08] a strict style checker will while at you about using a "magic string" if you don't make it a const [18:20:28] java hates duck typing and all it stands for [18:20:41] cult of compiler being smarter than then programmer [18:22:53] 3Phabricator, Security-Reviews, MediaWiki-Core-Team: Aphlict security review - https://phabricator.wikimedia.org/T1286#960189 (10csteipp) 5Open>3stalled >>! In T1286#957936, @Qgil wrote: > There is ongoing work to get rid of Flash and use WebSockets. Should we resolve this task as Stalled in the meantime? >... [18:39:27] ouch [18:39:30] wmf14 changelog is 48,979 bytes [18:39:46] https://www.mediawiki.org/wiki/MediaWiki_1.25/wmf14/Changelog [18:39:54] might be a long afternoon [18:40:35] and noting, that doesn't include wikidata [18:40:58] That's a ton of code cleanup changes in MobileFrontend [18:41:02] * bd808 gets worried [18:42:26] heh [18:43:29] I have deploy freezes for this reason personally. Lots of small changes that could/should have been sprinkled over 3 releases all in a big pile [18:43:52] when something goes wrong there is a lot of diffs to look at [18:44:10] s/have/hate/ [18:44:52] MF seems to break with only a few changes [18:47:11] <^d> It's not a long enough changelog unless it makes your machine go into swap [18:47:43] Well, making your browser hang due to the large page is good going [18:48:23] /easily done [18:50:49] 3operations, MediaWiki-Core-Team, MediaWiki-ResourceLoader: Bad cache stuck due to race condition with scap between different web servers - https://phabricator.wikimedia.org/T47877#960322 (10chasemp) So we think this is inherent to the design of our deploy mechanisms? @Joe, is there a ticket somwhere for revamp... [19:00:12] <^d> AaronS: https://gerrit.wikimedia.org/r/#/c/174267/ rebased. [19:00:24] <^d> Still a lot of custom profiling, but removes all the "obvious" stuff. [19:00:50] I guess the -2 can be removed [19:00:57] <^d> Yeah [19:01:46] <^d> {{done}} [19:04:23] "Everywhere on WMF that we want to profile has xhprof, and if it doesn't it'll be getting it soon." -- and it's turned off still right? [19:04:56] <^d> I thought it got turned back on, except for the sampled. [19:05:11] I saw some forceprofile chatter here yesterday but didn't pay close attention [19:06:39] ^d: What do I file a bug against for gerrit being overly eager in linking Txxxx text? -- see last comment on https://gerrit.wikimedia.org/r/#/c/183255/ [19:07:25] I pasted in a full link to phab there inside <...> [19:07:30] <^d> There already is a task I thought [19:07:30] and it went nutso [19:07:50] <^d> Gerrit uses a shitty regex implementation [19:07:58] java! [19:07:59] <^d> It's completely unintuitive and leads to crap like this [19:08:15] <^d> http://www.brics.dk/automaton/ [19:08:43] ugh. [19:09:08] pre compiled regex in java not good enough for them huh? [19:10:21] <^d> git wasn't good enough for them ;-) [19:10:41] fuck yeah, jgit [19:12:15] the syntax doesn't obviously have \b boundary markers which I *think* might fix the greed problem [19:13:16] <^d> I guess I'm just spoiled by having fancy tools like pcre :p [19:14:32] 3MediaWiki-extensions-TitleBlacklist, MediaWiki-Core-Team: Title blacklist intermittently failing, allowing users to edit things they shouldn't be able to - https://phabricator.wikimedia.org/T85428#960488 (10csteipp) a:5csteipp>3tstarling Tim is doing most of the work on this now [19:26:59] 3MediaWiki-Core-Team, MediaWiki-General-or-Unknown: HttpError should not be logged as an unhandled exception. - https://phabricator.wikimedia.org/T85795#960523 (10bd808) >>! In T85795#960163, @Legoktm wrote: >>>! In T85795#960109, @JanZerebecki wrote: >> The goal is that everything that is logged can be easily d... [19:30:29] ^d, AaronS: The merge of that profiler cleanup patch makes my blog post more truthy. :) [19:31:11] <^d> mmm, truthiness. [19:31:14] bd808: well, we xenon is basically doing most profiling [19:31:32] xhprof just for cases were you need all the info for a given request rather than just in aggregate [19:32:07] ^d: https://gerrit.wikimedia.org/r/#/c/183304/ [19:32:25] hmm, prolly needs release notes [19:32:43] yeah. I'm interested to see how the xenon stuff works out. It's a bit different to show the most common paths vs the slowest code in my mind but maybe I'm looking at things wrongly. [19:33:41] <^demon|lunch> AaronS: Yes, it does :) [19:33:41] well the slower it is the more snapshots it will be in, increasing its % [19:34:42] *nod* [19:36:12] bd808|LUNCH: looks like this is going to be a quick overall scap [19:36:38] 28m 32s [19:36:39] not bad [19:40:17] 3MediaWiki-Core-Team: Remove ProfilerStandard - https://phabricator.wikimedia.org/T86055#960570 (10aaron) 3NEW a:3aaron [19:46:31] Fatal error: Class undefined: ComposerAutoloaderInitea7a8b034df32856d6c1587c046d5521 in /srv/mediawiki/multiversion/vendor/autoload.php on line 7 [19:46:34] legoktm: ^ [19:49:16] uh [19:49:19] wtf [19:49:29] Reedy: where? [19:49:38] there was a few of them in the logstash fatalmonitor [19:49:39] Not sure if it's transient [19:50:18] Doesn't appear on fluorine [19:50:47] ok...we might want to set a prefix instead of using a random hash so that the class name doesn't change randomly [20:41:12] TimStarling: ok I'm finally going to push your pcre cache thing today. I wasn't able to completely get rid of the regression but got it small enough that we're ok with it [20:41:30] awesome [20:41:33] when I first push it, the default will be "static" but I'll follow it up with a change back to scalable [20:41:39] once I get our internal configs updated [20:42:17] sounds good [20:42:27] thanks for that [20:42:37] yeah, that's awesome [20:42:38] np [20:42:51] and sorry this took so long [20:52:37] 3MediaWiki-Core-Team, MediaWiki-API: list=logevents tries to return both the creating user's and the created user's ids as "userid" - https://phabricator.wikimedia.org/T73020#960748 (10Anomie) [20:52:45] 3MediaWiki-Core-Team, MediaWiki-API: let list=logevents use the new LogFormatter to format its output - https://phabricator.wikimedia.org/T35235#960750 (10Anomie) [20:53:12] 3MediaWiki-Core-Team, MediaWiki-API: let list=logevents use the new LogFormatter to format its output - https://phabricator.wikimedia.org/T35235#960754 (10Anomie) a:3Anomie [20:53:53] 3MediaWiki-Core-Team, MediaWiki-API: let list=logevents use the new LogFormatter to format its output - https://phabricator.wikimedia.org/T35235#363741 (10Anomie) [20:54:03] 3MediaWiki-Core-Team, MediaWiki-API: list=logevents tries to return both the creating user's and the created user's ids as "userid" - https://phabricator.wikimedia.org/T73020#766496 (10Anomie) [20:59:48] * bd808|LUNCH uses a time machine to visit 2003 -- https://github.com/wikimedia/mediawiki/tree/d82c14fb4fbac288b42ca5918b0a72f33ecb1e69 [21:01:48] <^d> Ancient commit history is fun [21:12:05] 3MediaWiki-Core-Team, MediaWiki-API: Annotate redirects from prop=info if target is an interwiki - https://phabricator.wikimedia.org/T85417#960808 (10Umherirrender) 5Open>3Resolved [21:21:31] 3MediaWiki-Core-Team: Parsoid performance analysis - https://phabricator.wikimedia.org/T85870#960882 (10Memeht) Heya, interesting results! Quick clarification: where the Parsoid timing metrics obtained from general logs or did you implement some performance instrumentation? [21:30:26] <^d> Anyone have thoughts on https://phabricator.wikimedia.org/T85565? I'm thinking MW's ForkController doesn't work in HHVM (it uses pcntl & friends) [21:31:28] ^d: https://gerrit.wikimedia.org/r/#/c/183304/ :) [21:32:57] <^d> thx for release notes [21:37:56] 3Librarization, MediaWiki-Core-Team: Split out utfnormal functions into standalone library - https://phabricator.wikimedia.org/T86069#960954 (10Legoktm) 3NEW [21:41:52] <^d> anomie: Fixed the @since annotations on https://gerrit.wikimedia.org/r/#/c/109669/ [22:03:43] ^d: https://gerrit.wikimedia.org/r/#/c/183374/ [22:03:59] after that, I think the other wfProfile* calls can just be nuked [22:04:15] still some ProfileSection calls to cleanup after that though [22:06:12] <^d> The ones for the entry points you mean? [22:06:23] <^d> Like wfProfileIn( 'api.php' ), etc? [22:08:04] anything starting with wfProfile* ;) [22:10:35] * AaronS will deal with ProfileSection [22:18:54] ori: does xenon use child class method names or the parents (if inherited)? [22:19:15] I thought they all used the parent in that case [22:20:10] I think it just uses the stack frames doesn't it. The same as you would get from a exception dump. [22:21:34] ^d: ^ [22:25:16] "This structure gathers a php and async stack trace when log is called." -- https://github.com/facebook/hhvm/blob/master/hphp/runtime/ext/xenon/ext_xenon.cpp#L56 [22:29:05] <^d> boo [22:29:26] ^d: also made https://gerrit.wikimedia.org/r/#/c/183385/ [22:33:07] 3MediaWiki-Core-Team, MediaWiki-General-or-Unknown: Drop PHP 5.3 support - https://phabricator.wikimedia.org/T75901#784809 (10Jdforrester-WMF) We can do this now, I assume? We're now at 100% HHVM. Should we do it for 1.25? [22:34:17] 3MediaWiki-Core-Team, MediaWiki-General-or-Unknown: Drop PHP 5.3 support - https://phabricator.wikimedia.org/T75901#961094 (10Reedy) >>! In T75901#961091, @Jdforrester-WMF wrote: > We can do this now, I assume? We're now at 100% HHVM. Should we do it for 1.25? We've still got some misc servers and such we're us... [22:34:30] 3MediaWiki-Core-Team, MediaWiki-General-or-Unknown: Drop PHP 5.3 support - https://phabricator.wikimedia.org/T75901#961095 (10Chad) >>! In T75901#961091, @Jdforrester-WMF wrote: > We can do this now, I assume? We're now at 100% HHVM. Should we do it for 1.25? We are not at 100%. Just apache stuff. Misc boxes, i... [22:34:54] 3MediaWiki-Core-Team, MediaWiki-General-or-Unknown: Drop PHP 5.3 support - https://phabricator.wikimedia.org/T75901#961097 (10Legoktm) >>! In T75901#961091, @Jdforrester-WMF wrote: > We can do this now, I assume? We're now at 100% HHVM. Should we do it for 1.25? We're not at 100% HHVM yet. imagescalers, job run... [22:35:14] kik [22:35:29] *lol [22:35:43] lol [22:39:37] heh. it would be very awkward to drop 5.3 before tin is upgraded [22:40:35] well, if we stopped people using maintenance scripts... [22:40:51] like l10nupdate [22:40:54] so lame [22:40:57] :) [22:41:05] it's broken anyway ;) [22:41:20] frack. we should really fix that [22:41:48] its the scap use right? lock file and ssh-agent both now? [22:42:20] I can't believe that folks aren't raging about that not working [22:42:50] maybe I'm just not watching the right phab projects to hear about it [22:45:51] https://phabricator.wikimedia.org/T85790 [22:48:39] Should we just roll back the changes that made it use scap? [22:49:07] Wasn't it working fine for a while? [22:49:16] I guess we talked about making scap clean up the lock file. [22:49:35] I thought it wasn't working since the new shared ssh-agent stuff was added [22:49:41] *added to scap [22:49:55] we reverted something [22:50:09] that patch that we tried to make that optional in scap was reverted [22:50:34] I have it open in a tab somewhere to "remind me" to take another crack at it [22:51:21] But basically the fancy patch ori made to check the socket read didn't work even if you had read rights on the file [22:51:40] so it was all rolled back [22:52:08] I might be wrong but I think if it gets past the scap lock it will fail next on the ssh [22:59:32] 3operations, MediaWiki-Core-Team: Come up with key performance indicators (KPIs) - https://phabricator.wikimedia.org/T784#961126 (10chasemp) p:5High>3Normal [23:05:52] 3MediaWiki-extensions-CentralAuth, MediaWiki-Core-Team, SUL-Finalization: Re-enable $wgCentralAuthAutoMigrate = true - https://phabricator.wikimedia.org/T78727#961132 (10Legoktm) It appears that the OOMs can be reproduced by visiting Special:MergeAccount directly, as seen in https://en.wikipedia.org/wiki/User_ta... [23:07:24] 3MediaWiki-extensions-CentralAuth, MediaWiki-Core-Team, SUL-Finalization: Investigate and fix OOMs caused during account globalization - https://phabricator.wikimedia.org/T78727#961134 (10Legoktm) p:5Triage>3High [23:08:31] * legoktm grumbles [23:16:26] 3ContentTranslation-Deployments, MediaWiki-extensions-ContentTranslation, MediaWiki-Core-Team: Content Translation Beta Feature security review - https://phabricator.wikimedia.org/T85686#961154 (10csteipp) modules/tools/ext.cx.tools.reference.js In the ReferenceCard.prototype.start function, I'm fairly sure an... [23:21:29] 3ContentTranslation-Deployments, MediaWiki-extensions-ContentTranslation, MediaWiki-Core-Team: Content Translation Beta Feature security review - https://phabricator.wikimedia.org/T85686#961164 (10csteipp) In general, you shouldn't be unserializing and serializing html with code like, $( '
' ).html( mwData.... [23:26:34] 3ContentTranslation-Deployments, MediaWiki-extensions-ContentTranslation, MediaWiki-Core-Team: Content Translation Beta Feature security review - https://phabricator.wikimedia.org/T85686#961168 (10csteipp) >>! In T85686#961164, @csteipp wrote: > In general, you shouldn't be unserializing and serializing html wit... [23:32:22] 3ContentTranslation-Deployments, MediaWiki-extensions-ContentTranslation, MediaWiki-Core-Team: Content Translation Beta Feature security review - https://phabricator.wikimedia.org/T85686#961186 (10csteipp) Not exploitable, but ext.cx.tools.reference.js line 251, you should verify that $section.data( 'source' ) i... [23:42:28] 3MediaWiki-Core-Team, MediaWiki-General-or-Unknown: Drop PHP 5.3 support - https://phabricator.wikimedia.org/T75901#961205 (10Jdforrester-WMF) [23:44:19] 3MediaWiki-Core-Team, MediaWiki-General-or-Unknown: Drop PHP 5.3 support - https://phabricator.wikimedia.org/T75901#784809 (10Jdforrester-WMF) >>! In T75901#961097, @Legoktm wrote: >>>! In T75901#961091, @Jdforrester-WMF wrote: >> We can do this now, I assume? We're now at 100% HHVM. Should we do it for 1.25? >...