[00:03:28] TimStarling: I amended your patch very slightly and pushed it to both production branches, and also pushed a mediawiki-config patch to enable the T87645 log bucket. I just ran wfDebugLog( 'T87645', '...' ) on tin to validate that the log bucket is indeed active (it is). There are no other entries on fluorine, and thus presumably no further occurrences of the bug. [00:03:54] yep, good [00:03:57] I saw that you merged it [00:08:22] I also flagged it to springle so that he's aware that there is an issue affecting production with data integrity implications. He brought up https://phabricator.wikimedia.org/T87716 , saying it'd be good to know if there is indeed a link between these two issues. [00:13:09] legoktm: I like your explanation of why authstack work is reasonable. Thanks! [00:15:08] :) [03:17:13] 3MediaWiki-Core-Team: Make FileRepo file cache misses use slaves instead of the master - https://phabricator.wikimedia.org/T88506#1013549 (10aaron) 3NEW [03:17:41] https://gerrit.wikimedia.org/r/#/c/184838/ [03:17:43] AaronSchulz: ^ [03:19:09] 3MediaWiki-Core-Team: Devise stashing strategy for multi-DC mediawiki - https://phabricator.wikimedia.org/T88493#1013558 (10aaron) [03:19:10] 3MediaWiki-Core-Team: Make FileRepo file cache misses use slaves instead of the master - https://phabricator.wikimedia.org/T88506#1013557 (10aaron) [03:19:47] 3MediaWiki-Core-Team: Make FileRepo file cache misses use slaves instead of the master - https://phabricator.wikimedia.org/T88506#1013549 (10aaron) [06:34:13] 3MediaWiki-Core-Team, MediaWiki-Special-pages, MediaWiki-extensions-CentralAuth: Unable to assign interwiki rights from Special:UserRights - Error 503 - https://phabricator.wikimedia.org/T88505#1013731 (10Legoktm) p:5High>3Unbreak! [06:37:08] 3MediaWiki-Core-Team, MediaWiki-Special-pages, MediaWiki-extensions-CentralAuth: Create common interface for User, UserRightsProxy, and CentralAuthGroupMembershipProxy - https://phabricator.wikimedia.org/T88510#1013735 (10Legoktm) 3NEW a:3Legoktm [06:42:46] 3MediaWiki-Core-Team, MediaWiki-Special-pages, MediaWiki-extensions-CentralAuth: Unable to assign interwiki rights from Special:UserRights - Error 503 - https://phabricator.wikimedia.org/T88505#1013515 (10Legoktm) I filed T88510 and T88511 as follow up tasks to prevent this from happening again. [06:54:51] legoktm: That's one weird interface name [06:55:13] hoo: yeaaaaah. I couldn't come up with anything better. suggestions welcome :D [06:55:21] UserGroupMembership ? [06:56:05] oder [06:56:23] UserGroupRelations [06:56:25] no, that sucks [06:57:03] UserGroupManager [06:57:54] oder :D [06:58:21] Context switching... [07:07:51] legoktm: Can I haz +2 https://gerrit.wikimedia.org/r/187925 ? [07:07:59] Would be nice to have it in the branch [07:08:30] same goes for the core patch about double escaping, but I guess reviewing that takes more time [07:10:51] I had looked at the escaping one before but was assuming that chris would +2 it [08:28:22] <_joe_> legoktm: any success with the new HHVM package? [08:28:31] <_joe_> I'll deploy it widely today [08:29:31] AWESOME [08:29:36] please close the bug afterwards [08:29:45] <_joe_> hoo: of course [08:30:10] Will stomp people on it afterwards... quite some things being blocked on that [08:30:35] legoktm: Anything else blocking auto-merge-renable? [08:30:42] If not, I'd go for that today [08:35:43] <_joe_> hoo: what is blocked on that, apart from the SUL bug? [08:37:03] _joe_: Nothing I'm aware of [08:37:10] Only several parts of CentralAuth [08:38:37] <_joe_> ok [10:54:44] bleugh [10:54:56] I have a load of dead extensions checked out, deleting them obviously fixes that [10:55:00] but then they seem to come back again via some vodoo [12:44:37] 3MediaWiki-Core-Team, SUL-Finalization, MediaWiki-extensions-CentralAuth: Investigate and fix OOMs caused during account globalization - https://phabricator.wikimedia.org/T78727#1014206 (10Joe) [12:44:38] 3MediaWiki-Core-Team, SUL-Finalization, MediaWiki-extensions-CentralAuth: Cherry-pick and deploy fd41d20010 from facebook/hhvm - https://phabricator.wikimedia.org/T86813#1014204 (10Joe) 5Open>3Resolved [12:44:56] 3MediaWiki-Core-Team, SUL-Finalization, MediaWiki-extensions-CentralAuth: Cherry-pick and deploy fd41d20010 from facebook/hhvm - https://phabricator.wikimedia.org/T86813#977091 (10Joe) [12:46:06] <_joe_> legoktm: there you go, package deployed everywhere [12:50:08] Awesome... let's shoot for auto merge [13:59:41] 3MediaWiki-Core-Team, MediaWiki-extensions-CentralAuth, SUL-Finalization: Investigate and fix OOMs caused during account globalization - https://phabricator.wikimedia.org/T78727#1014370 (10hoo) 5Open>3Resolved a:3hoo Should be good now. [14:35:42] _joe_: "Using RESTBase as an integration layer for everything is SOA done wrong." Hear, hear! [14:36:03] <_joe_> anomie: :) [14:36:32] <_joe_> anomie: I just have PTSD from such an horror in my last dayjob, that's why I'm so vocal [15:21:58] 3MediaWiki-Core-Team, wikidata-query-service, Wikidata: Write example queries in different query languages - https://phabricator.wikimedia.org/T86786#1014784 (10Lydia_Pintscher) p:5Triage>3High [15:29:27] ori: Do we even know if that performance regression is in PHP or JS? I see both your graphs are measuring stuff client-side. [15:36:15] anomie: no (as in, we don't know) [15:41:43] ori: Sanity check: it's not because of I6e3167f, is it? [15:44:38] anomie: that's in wmf14 as well, though [15:45:37] Oh... git log confused me. [15:58:36] ori: anything I should/can do to help with this? [15:59:06] * Reedy blames VE [16:00:52] <^d> bd808, manybubbles: Let's join the hangout at least 10mins early so we can at least attempt to put the ducks in a row [16:01:02] +1 [16:01:16] works for me [16:01:28] many meetings today. many meetings every day [16:01:56] * bd808 may need to attach headphones permanently [16:02:30] aural implants [16:02:50] fiber behind my left ear [16:04:38] <^d> No, you clearly need a larry king microphone [16:04:40] <^d> Big table mic [16:04:44] <^d> Way cooler than headset [16:04:58] Telefunkin U47 [16:05:18] *Telefunken U47 [16:05:48] <^d> We have an article on [[Neumann U47]] [16:05:54] A nice warm tube mic to give me that Wolfman Jack sound [16:06:54] The telefunken is the modern version of that [16:08:24] I've got rather large collection of mics, one is an old electrovoice announcer's mike, would probably do the job nicely [16:08:41] mic .. I should name it mike though, or bob. [16:09:48] * bd808 imagines a mic lowering from the ceiling over his desk. "Let's get ready to produuuuct!" [16:17:16] <^d> Well crap, I just broke one of our nice coffee mugs :( [16:17:31] <^d> In the sink, no less, so there's now glass in the drain I have to fish out [16:18:47] <^d> I liked that mug [16:18:58] <^d> Double walled glass [16:41:34] ^d: nice mug. sorry [16:42:36] ^d: robla mentioned something about being able to make a workboard that had two tags - like a filtered version of the mw.core workboard. I think I'd like to convert the wikidata query service board into that. do you know where the docs are for that? I don't know where I'd start searching. [16:42:38] <^d> Life goes on. At least I'd finished my coffee before I broke it ;-) [16:43:29] <^d> I dunno how to make that the default [16:43:41] <^d> I know there's filters for a workboard. I suppose those are linkable? [16:43:56] <^d> Like search queries? [16:44:04] hmmm - I'll dig around [16:45:16] ori: Do we have any information on this performance regression besides "something in the entire stack"? [16:49:01] manybubbles: open a workboard, click "filter", click "advanced filter" [16:49:08] manybubbles: example -- https://phabricator.wikimedia.org/project/board/37/query/LnW5WRS_xspN/ [16:50:53] <^d> Ah yes, what I thought [16:51:34] So who is helping Ori look at the performance regression? [16:52:00] I think we need somebody to step up and coordinate efforts on this [16:52:14] this is why i want ability to do a diff on xdebug or xenon between two profiles... [16:52:27] any further info on the performance issue? [16:52:42] greg-g, Reedy: who should be in charge? [16:53:55] aude: All I've seen in various backscrolls is that group1 to wmf15 started a 1s front end latency bump and rolling back to wmf14 fixed it [16:54:15] so "something" in wmf15 is really bad for frontend perfornance [16:54:44] hm [16:56:15] http://performance.wikimedia.org/xenon/svgs/daily/2015-02-04.svgz [16:56:22] what do the colors mean? red? [16:56:41] went from orange to red http://performance.wikimedia.org/xenon/svgs/daily/2015-02-02.svgz [16:56:44] colors man nothing [16:56:47] *mean [16:56:56] only width and height matter [16:56:57] ok [16:57:09] bd808: +1, we need some organization if we're going to halt all deployments until it gets fixed. [16:57:18] width is how often it was seen [16:57:31] height is the depth of the stack trace obviously [16:57:34] and maybe one of these http://performance.wikimedia.org/xenon/logs/hourly/ [16:57:38] bd808: not me, I'm in 1:1s all day today :/ [16:57:39] if we rolled back [16:57:46] could be looking at [16:57:46] anomie: is that you volunteering? [16:58:13] Here is a link to Ori's smoking gun graph -- https://docs.google.com/a/wikimedia.org/spreadsheets/d/1IVR_gx1uj0gUKWorD34x3wfyII3-XcKVSQchCWvPf3A/pubchart?oid=688143338&format=interactive [16:58:20] bd808: You probably don't want me in charge of organization ;) [16:59:12] anomie: you sell yourself short, but point taken [16:59:13] ok [16:59:25] ^d: can you help help wrangle troops to figure out the perf issue? [16:59:30] * bd808 has a conf call; back in 30 mins [16:59:42] * greg-g has two back to back [17:00:34] * ^d is trying to understand what he's reading on the flame graph [17:00:37] I'm looking at the performance issue, but not having much luck in narrowing things down. I haven't been able to reproduce it yet in a basic vagrant with a handful of extensions. [17:03:19] https://www.mediawiki.org/wiki/MediaWiki_1.25/wmf15 anything suspicious [17:04:17] What exactly are our "smoking gun" graphs measuring? https://i.imgur.com/2sPxYNg.png looks like from the time the browser starts loading the page until jQuery's onload, while https://i.imgur.com/NFbyOBu.png is more ambiguous. [17:05:33] If "time to mediaWikiLoadComplete" means "what the NavigationTiming extension reports as mediaWikiLoadComplete", that looks like it'd be the time between when resources/src/startup.js runs and just after jQuery's load event. [17:11:06] <^d> Do we have some xhprof data to go with this? Right now I'm shooting blind. [17:14:36] do we collect data like this on beta, by any chance? [17:15:19] that would be awesome to figure out timing of whatever change that caused this [17:16:06] ori: ^ [17:16:17] * aude thinks unlikely but would be godo for the future [17:17:43] 3MediaWiki-Core-Team, wikidata-query-service: Mourn Titan - https://phabricator.wikimedia.org/T88550#1015196 (10Manybubbles) [17:18:18] 3MediaWiki-Core-Team, wikidata-query-service: Figure out if Neo4j is a possible alternative to Titan - https://phabricator.wikimedia.org/T88571#1015198 (10Manybubbles) [17:18:31] 3MediaWiki-Core-Team, wikidata-query-service: Figure out if Neo4j is a possible alternative to Titan - https://phabricator.wikimedia.org/T88571#1014978 (10Manybubbles) [17:18:47] 3MediaWiki-Core-Team, wikidata-query-service: Find another TinkerPop 3 implementation to add to the Wikidata Query candidate list - https://phabricator.wikimedia.org/T88551#1015201 (10Manybubbles) [17:18:49] <_joe_> manybubbles: uh? [17:18:53] <_joe_> neo4j? [17:18:56] <_joe_> why? [17:19:06] <_joe_> and then, java again? [17:19:08] _joe_: see the email about titan? [17:19:20] _joe_: we've opened up investigations on arangodb as well [17:19:21] <_joe_> nope, not reading the emails at the moment [17:19:28] <_joe_> ok [17:19:28] bad news :( [17:19:29] _joe_: ah, well Titan is probably dead [17:19:43] <_joe_> we still have haedus and capella for testing, luckily [17:20:46] 3MediaWiki-Core-Team, wikidata-query-service: Investigate ArangoDB for Wikidata Query - https://phabricator.wikimedia.org/T88549#1015208 (10Manybubbles) [17:20:57] 3MediaWiki-Core-Team, wikidata-query-service: Consume etherpad from wikidata query developer session - https://phabricator.wikimedia.org/T88341#1015210 (10Manybubbles) [17:23:22] <_joe_> manybubbles: wow, nice [17:23:32] 3MediaWiki-Core-Team: Investigate hand building a gremlin implementation against Elasticsearch for wikidata query - https://phabricator.wikimedia.org/T88577#1015218 (10Manybubbles) 3NEW a:3Manybubbles [17:23:48] 3MediaWiki-Core-Team, wikidata-query-service: Investigate hand building a gremlin implementation against Elasticsearch for wikidata query - https://phabricator.wikimedia.org/T88577#1015227 (10Manybubbles) [17:23:50] its fun, yeah [17:23:51] <^d> Ooooh, more Elasticsearch :D [17:24:01] * ^d impulsively votes for that [17:24:03] ^d: read the comment [17:24:08] "will it stretch!?" [17:24:11] its one of those "if we can't find anything better" [17:24:38] <^d> Reedy: You can put anything in Elasticsearch! [17:25:09] <_joe_> manybubbles: re-considering RDF datastores may be an option? [17:25:21] <_joe_> I kind of remember some of them seemed nice to me [17:25:24] <^d> New startup idea: Pet boarding, but store people's pets in Elasticsearch [17:25:33] * ^d quits, goes to make millions [17:25:55] <_joe_> ^d: s/elasticsearch/restbase/g [17:26:07] ^d: step one: invent star trek transporter technology [17:26:13] <^d> _joe_: bahahahahahahaha [17:26:16] 3MediaWiki-Core-Team, wikidata-query-service, Wikidata: Investigate Titan supernode issues - https://phabricator.wikimedia.org/T85173#1015237 (10JanZerebecki) 5Open>3declined [17:26:23] ^d: step two: fix pattern buffer issue they always had [17:26:32] Do the lords prayer with restbase instead [17:26:34] <^d> fucking pattern buffers man [17:26:54] 3MediaWiki-Core-Team, wikidata-query-service, Wikidata: Investigate Titan supernode issues - https://phabricator.wikimedia.org/T85173#1015241 (10Manybubbles) Yup. This is totally no needed any more. We might still need some of the Gremlin tasks but sure not this. [17:27:13] 3MediaWiki-Core-Team, wikidata-query-service: Mourn Titan - https://phabricator.wikimedia.org/T88550#1015244 (10MaxSem) Weep weep. [17:27:30] <^d> _joe_: Well Tim always said you can do anything with MediaWiki ("feed your cat or launch nuclear missiles"), restbase is the new MW so...... :p [17:28:07] <_joe_> ^d: gabriel genuinely says otherwise. The problem here is we're terrible at cross-team communication [17:28:35] is someone investigating the performance regression? wmf15 has a lot of changes :/ [17:28:46] anomie, ^d, aude: You may be able to get xhprof data using the X-Wikimedia-Debug header + forceprofile=true [17:28:59] group0 would still be on wmf15 [17:29:28] legoktm: welcome to the party! We need a leader. [17:29:30] legoktm: We're all shooting in the dark, mostly. We don't even know if the slowdown was in PHP or client-side JS. [17:30:05] I would try, but I'm just in-between meetings, booked until 11am pacific [17:30:33] bd808: ah, yes although wonder if it's js or backend or both [17:30:43] ^d: do you have time to just be aware of who is working on it and making sure it stays high prio for them (whoever)? [17:30:50] where it==perf issue [17:30:56] * aude doesn't know where to start looking [17:31:12] <_joe_> greg-g: tsk tsk, you should ask an ETA [17:31:24] <_joe_> good managers always ask for an ETA [17:31:30] and hover [17:31:34] <^d> anomie: Work on getting us some real xhprof data between wmf14 and 15. I want to at least figure out where this is coming from. [17:31:39] and then respond with "you can do it in half that time, go!" [17:32:17] We have scrum of scrums in an hour. I can scream for help there. [17:32:21] <_joe_> greg-g: an accenture drone once asked me "Can you give me an ETA on getting an ETA on the issue"? [17:32:24] bd808: please do [17:32:31] _joe_: ........ [17:32:45] <_joe_> greg-g: you can imagine my answer, I guess [17:32:58] I mean, that's a logical sentence with words put together grammatically correct and it has an understandable meaning, but... [17:33:04] "you'll know when I know" is my typical answer [17:33:10] bd808++ [17:33:30] hah [17:33:33] bd809-- [17:33:34] <_joe_> bd808: mine was like "go f*** yourself with a very hot tube", but in italian [17:33:41] bd808-- [17:33:50] "somewhere between 5 minutes and the heat-death of the universe" [17:33:56] bd807++ [17:33:59] <_joe_> anomie: eheh right :) [17:34:13] Nick bd809 is now registered to your account. [17:34:26] tacotuesday wins! [17:34:34] haha [17:34:51] <_joe_> php -r '$a="bd808" + "5 apples"; echo "$a\n";' [17:34:56] <_joe_> I win :) [17:36:04] why did that coerce to 5? [17:36:58] isset( $_GET['forceprofile'] ) && isset( $_SERVER['HTTP_X_WIKIMEDIA_DEBUG'] ) [17:37:01] That reminds me of T51580, where a side conversation revealed that specifying a block duration of "a potato" gives a block that expired an hour ago. [17:37:15] bd808 == 0; 5 apples == 5; php == carzy pants [17:37:32] I can't just forceprofile=1? [17:37:34] bd808: coercing strings to numbers just gets the first number in the text. I think [17:38:12] wingswednesday: nope. there's are firefox and chrome helpers to set the header [17:38:26] commit messages like "Fix FIXMEs" are great [17:38:31] That's more annoying :( [17:38:56] https://addons.mozilla.org/en-US/firefox/addon/wikimedia-debug-header/ [17:39:12] https://github.com/wikimedia/ChromeWikimediaDebug [17:39:33] chrome -- https://chrome.google.com/webstore/detail/wikimediadebug/binmakecefompkjggiklgjenddjoifbb [17:40:51] Installed extension, still not getting text profiling output [17:41:13] did you click the button? [17:41:16] :) [17:41:17] Yes [17:41:34] stuff like OutputPage::getBottomScripts seems to take longer [17:41:35] "Our plan is to have a spike to experiment to determine whether there are any early roadblocks in the proposed solution." Uhh... bd808? Does that basically mean there's a plan to plan to plan to see if there are problems in the plan? [17:42:25] https://gist.github.com/filbertkm/384e01263adfa6786f84 [17:42:26] vs [17:42:38] wingswednesday: with the firefox plugin enabled and -- https://www.mediawiki.org/wiki/Manual:Developing_libraries?forceprofile=true -- I get xhprof data in the view-source [17:42:52] https://gist.github.com/filbertkm/69779cf6343e8a065efc [17:42:55] * wingswednesday doesn't use firefox :p [17:43:06] but there are big difference between test2 and enwiki, surely [17:44:29] getBottmScripts taking longer might mean someone just added a bunch of RL modules [17:44:55] which would effect frontend timing too [17:44:57] hm [17:45:05] https://gerrit.wikimedia.org/r/#/c/182875/5/includes/OutputPage.php is the only thing [17:45:09] i see touching output page [17:45:14] in core [17:45:18] works on enwiki but not mw.org? [17:45:19] might be in extension [17:45:32] anomie: think of that "spike" as what you would do before you started writing an RFC to know what need to say in that RFC. Not sure where that is in the redirection level, but that's the positive interpretation [17:46:00] wingswednesday: oh yeah. ori didn't whitelist that domain in the initial chrome plugin! [17:46:42] wingswednesday: https://github.com/wikimedia/ChromeWikimediaDebug/commit/499b794604e913a6e63daca8a226a710763cd233 [17:47:15] Ah, ok [17:47:18] I bet he didn't update the chrome store version [17:47:57] release software is hard, yo [17:48:00] releasing* [17:49:19] aude, legoktm: Among other things, test2wiki has $wgEnableJavaScriptTest true which probably explains the large increase in time spent in RL [17:49:26] ah [17:49:36] * aude tries test* [17:51:41] etherpad at https://etherpad.wikimedia.org/p/1.25wmf15-perf-regression for notes [17:51:48] bd808: I don't like this btw [17:51:49] 7.66% 50.967 3175 - ProfilerXhprof::scopedProfileIn [17:51:54] I think we can do better there. [17:51:54] 3MediaWiki-Core-Team, wikidata-query-service: Consume etherpad from wikidata query developer session - https://phabricator.wikimedia.org/T88341#1015340 (10JanZerebecki) [17:52:13] https://gist.github.com/filbertkm/5a3ada51718ab61148bc [17:52:19] more similar to enwiki [17:53:27] 3MediaWiki-Core-Team, Wikidata, wikidata-query-service: Write example queries in different query languages - https://phabricator.wikimedia.org/T86786#1015352 (10JanZerebecki) T88341 will have suggestions for queries [17:54:22] aude: I only see about a 2ms difference between enwiki and testwiki on getBottomScripts and friends [17:54:27] yeah [17:55:12] aude: That and guessing https://i.imgur.com/NFbyOBu.png is between setup.js and jQuery load makes me suspect client-side JS even more. [17:55:22] I see way more run_inits in test than enwiki [17:55:29] 4 vs like 30-something. [17:55:41] i don't see anything obvious in forceprofile [17:57:08] Do we have any graphs of average request service time as seen server-side? [17:57:55] http://gdash.wikimedia.org/dashboards/totalphp/, but gdash sucks [17:58:25] ... Yay graphs with no data [17:58:56] I can't find the data in graphite either. [17:59:02] mediawiki -all seems empty [17:59:03] request time would be super easy since we set wgBackendResponseTime on every page [17:59:05] great timing to have the graphite migration :/ [17:59:39] of just some sample pages or whatnot [17:59:44] I haven't been able to get data out of graphite in weeks. [17:59:46] godog: what's the extent of lost data during the graphite migration? [17:59:48] I don't blame migration [17:59:51] ah [18:00:00] godog: apologies if it's none more than normal :) [18:00:10] I blame graphite being a pain in the ass :p [18:01:59] manybubbles: did you figure out the workboard thing? ^^^ [18:02:11] robla: yeah [18:02:13] got it [18:02:27] slowly moving wikidata query stuff to it so it stays up to date with core [18:02:28] \o/ [18:03:50] one thing I'd love to investigate in my copious spare time is whether it'd be possible to keep two workboards in sync with the right Herald rules [18:04:39] Timeline of revert pieced together in -- https://etherpad.wikimedia.org/p/1.25wmf15-perf-regression [18:05:04] https://gerrit.wikimedia.org/r/#/c/180697/ was cherry-picked to wmf12, and then merged for wmf15 [18:06:24] greg-g: I am aiming at none (backfill from old machine is due tomorrow) but unlikely it'll be zero, guesstimating 3h/4h for some metrics [18:08:38] https://gerrit.wikimedia.org/r/#/c/182890/ [18:10:10] legoktm: that would be ironic [18:12:36] are the graphs we're looking at created by EL data? [18:12:47] I believe so, yes [18:13:01] They're frontend measurements from NavigationTiming as far as I can tell, so probably. [18:14:11] i really don't know what i'm looking at but in chrome profiling, [18:14:19] is see more onreadystatechange stuff towards the end [18:14:25] that i don't see for enwiki [18:14:32] but again, don't know what i'm doing really... [18:16:44] 3MediaWiki-Core-Team, wikidata-query-service: Figure out if Neo4j is a possible alternative to Titan - https://phabricator.wikimedia.org/T88571#1015409 (10JanZerebecki) The AGPL3 is a fine license. Compliance with it is only a minimum amount of more work than GPL3. But we can have that discussion in T76158. To... [18:17:41] So aside from a bunch of mobile and oojs stuff I skipped, that EL one is the only one that sticks out as me as something that is served to all users [18:18:20] one seems to be a gadget (formWizard...) [18:20:37] is ori up yet? [18:21:57] looking at some page on mw.org seems better comparision in terms of js profiling [18:22:25] chrismcmahon pointed out something possibly unrelated but for completeness -- https://phabricator.wikimedia.org/T87446 -- jquery undefined object warning that has been fowling browser tests [18:22:48] seems more likely that would cause loss of data rather than slowdown however [18:23:27] How do we bisect in prod :/ [18:23:47] beta!!! (doesn't help at this point though) [18:23:59] would love data to be collected regularly on beta [18:24:08] bd808: Live-hack on mw1017? [18:24:30] if we can recreate there, then yeah it would be possible [18:24:47] so first we need to establish that we can see the problem [18:24:52] * anomie is tempted to live-hack mw1017 to downgrade group0 to wmf14 and see if anyone can see differences [18:26:46] ok [18:27:28] Reedy & legoktm, can you +1 on https://gerrit.wikimedia.org/r/#/c/188086/ please? [18:27:53] done [18:28:10] wow, that was fast:) [18:31:18] csteipp: I'm in the SoS hangout but planning on mostly lurking today [18:40:04] commons spends quite a bit of js time on the anonymous i18n gadget... [18:40:23] that's wmf14 though [18:40:44] the profile is quite different than that for enwiki [18:41:39] I wonder if removing these several hundred profiling calls will offset the perf loss we took :p [18:42:53] wingswednesday: heh. [18:57:05] 3MediaWiki-Core-Team, CirrusSearch, MediaWiki-Maintenance-scripts: Hook into namespaceDupes so we can update the search index when namespaces change - https://phabricator.wikimedia.org/T88587#1015524 (10Chad) 3NEW [18:57:14] manybubbles: happy times ^ [18:57:31] wingswednesday: oh nice [18:57:56] we can probably cheat on the namespcedupes we've already run [18:58:19] Yeah, we just ran it on enwiki (elsewhere too?) so entries are busted at the moment. [18:58:31] (I've got reports of a User Talk page showing up in a mainspace prefix search) [19:00:43] manybubbles, bd808: I'm on the hangout, afk for a sec while I make a coffee [19:05:40] what happened to titan? is there an email or bug about it? [19:06:20] legoktm: there was an email to ops [19:06:46] oh, I missed that. /me reads [19:06:49] basically, the company behind it was bought and they are no longer to be developing titan :( [19:06:55] so we need to pick something else [19:07:08] * aude cries [19:09:15] 'We're using Titan in mission-critical applications. We really need commercial support from you guys' so lets kill the product! [19:11:15] on commons, looks like warn* and deprecated* js functions are slowish [19:11:30] and deprecated stuff is used all over the place in the anon i18n gadget [19:11:34] Use of "wgUserLanguage" is deprecated. Use mw.config instead. [19:19:43] 3MediaWiki-Core-Team, wikidata-query-service: Mourn Titan - https://phabricator.wikimedia.org/T88550#1015638 (10Legoktm) RIP. http://www.zdnet.com/article/datastax-snaps-up-aurelius-and-its-titan-team-to-build-new-graph-database/ [19:32:37] bd808: hey [19:33:05] ori: howdy. in an interview for another 30 mins [19:33:32] * ori nods [19:36:18] 3MediaWiki-Core-Team, wikidata-query-service: Mourn Titan - https://phabricator.wikimedia.org/T88550#1015688 (10Smalyshev) https://www.youtube.com/watch?v=vodd6C5ryUU [20:00:54] csteipp: i dumped on tim since the diff addresses his comments from 2013, but you may want to review it as well [20:09:51] SMalyshev, https://www.youtube.com/watch?v=wbWOVfY-rxU :P [20:15:41] * bd808 is drained [20:16:15] likewise, I'm going afk for awhile. [20:17:18] bd808: I think I've hit my human:human interaction quota for the week [20:17:22] so, where are we at on the wmf15->14 / RL stuff? [20:17:31] did touching stuff fix anything? [20:18:02] I haven't heard but I haven't been looking for a bit [20:18:31] bblack: Reports are that touching everything UploadWizard-related fixed that problem, if that's the "RL stuff" [20:18:31] it was VE problems right? RL cache not rolling back gracefully? [20:18:41] ah UW [20:19:07] No idea where we're at on the performance thing, although I saw ori speak earlier. [20:20:05] I'm not actually investigating it, I'm not sure how people got that idea [20:20:11] I'm taken up with VE stuff [20:20:23] i would say try wmf15 with commons [20:20:29] to get some profiling data [20:20:40] commons is very different than enwiki / mediawiki.org etc [20:20:42] that sounds fine, as long as there is some commitment to watching the graphs and making sure there aren't any major regressions [20:20:46] ori: You're the one who said "halt all deployments until this is fixed!", shouldn't you be working on it? [20:20:49] * aude has to go though [20:20:57] so someone else has to do [20:21:03] anomie: no, why would you say that? [20:21:21] but doing a comparision, maybe someone will figure out what the issue is or maybe it is fixed [20:22:02] ori: I think you can recruit help, but since we just stopped the train on this it should be all hands on deck until it is resolved [20:22:37] Random ideas have been posted to https://etherpad.wikimedia.org/p/1.25wmf15-perf-regression [20:22:49] so, the reason that e-mail went out at 2:49 AM SF time last night is because I was up until then triaging this [20:22:58] but we don't have a clear idea of how to reproduce and isolate the issue [20:23:04] jokes about my insomnia aside, I would have gone to bed earlier if it were not for that, so I am not being lazy here [20:23:26] I don't think that finger was pointed; sorry if it sounded that way [20:23:27] because commons is very different, it's tricky [20:23:39] No one said you were being lazy for going to bed. But you're not in bed anymore and said you're not working on it. [20:23:44] * aude notes the anonymous i18n gadget needs love there [20:23:46] but you did say VE was more important and that seems wrong [20:23:50] for one thing [20:24:40] well, it seemed like there is an emerging consensus (cf also greg's email from a few minutes ago) that we should try again [20:24:53] why don't we do that, and then triage further efforts accordingly [20:25:10] * anomie has no objection to that plan [20:25:18] wmf [20:25:21] I'm only ok if that's the right thing, I haven't seen anyone really propose it in a "I'm willing to do it/own it" kind of way [20:25:22] wfm even [20:25:22] who's going to do the actual deploy? i can re-push the patch i reverted last night [20:25:43] greg-g: i share your disappointment :P [20:25:49] ori: /me nods [20:26:01] OK, I'll revert the revert [20:26:06] thanks [20:26:36] * bd808|LUNCH will be back in 15 or so [20:26:45] good timing :) [20:27:04] 3MediaWiki-Core-Team, Librarization, Deployment-Systems: Have a check for reported security issues in dependencies - https://phabricator.wikimedia.org/T74193#1015834 (10Krenair) [20:27:13] bd808|LUNCH: I know you feel like strangling me :P [20:41:38] what graphs should I be watching/ [20:41:38] ? [20:42:05] 3MediaWiki-Core-Team, wikidata-query-service: Mourn Titan - https://phabricator.wikimedia.org/T88550#1015880 (10Manybubbles) >>! In T88550#1015688, @Smalyshev wrote: > https://www.youtube.com/watch?v=vodd6C5ryUU 10 hours of sad violin brightened my day! [20:47:48] * bd808 hugs ori [20:47:59] what graph do I need to stare intently at? [20:48:07] 3MediaWiki-Core-Team, operations: move misc mw maintenance scripts into mw puppet module - https://phabricator.wikimedia.org/T88597#1015891 (10Dzahn) 3NEW [20:49:02] these are deceiving, it seems: http://gdash.wikimedia.org/dashboards/frontend/ [20:49:10] the revert/deploy happened at :33 [20:49:32] (20:33, that is) [20:50:12] it'd be nice if we had those graphs broken down by wmfXX [20:51:01] yeah that would be swell but tricky to manage in graphite [20:51:38] 3MediaWiki-Core-Team, operations: move misc mw maintenance scripts into mw puppet module - https://phabricator.wikimedia.org/T88597#1015900 (10Dzahn) [20:51:50] the nav timing graphs were borked for almost two weeks this month, to my embarrassment [20:52:09] shit happens [20:52:20] i have a pattern of neglecting things and then making up for it with a fever of focused activity. it works for some things, but not metrics, since data lost is data lost. [20:53:17] https://graphite.wikimedia.org/render/?title=mediaWikiLoadStart%20to%20document.onload%20on%20desktop%20sites,%20last%206%20hours&vtitle=milliseconds&from=-6hours&width=1024&height=500&until=now&areaMode=none&hideLegend=false&lineWidth=1&lineMode=connected&target=alias%28color%28frontend.navtiming.mediaWikiLoadComplete.desktop.overall.mean,%22blue%22%29,%22mean%22%29&target=alias%28color%28frontend.navtiming.mediaWikiLoadComplete.desktop.overall.9 [20:53:17] 9percentile,%22red%22%29,%2299th%20percentile%22%29 [20:53:24] long url is long [20:53:42] jump at 19:40Z [20:54:01] so 1.5h ago about [20:54:38] I hate graphite urls [20:55:04] I love them because I know how to edit them but the paste badly [21:03:00] 3MediaWiki-Core-Team: Document current MediaWiki PHP authn stack - https://phabricator.wikimedia.org/T88195#1015909 (10Legoktm) My initial and completely un-organized notes are at https://www.mediawiki.org/wiki/User:Legoktm/authn. It only covers the GUI login process (haven't gotten to API), and only covers exte... [21:04:54] grrr, graphite urls are a crime against humanity [21:13:09] Heh, "overly long graphite url" would make a good cards against humanity card [21:13:15] only when playing with other engineers though [21:14:04] there is the devops against humanity ;) [21:14:25] Meh :p [21:14:37] https://github.com/bridgetkromhout/devops-against-humanity [21:14:56] I'm just waiting for July when exploding kittens comes out :) [21:17:01] oh god: https://groups.google.com/forum/#!topic/aureliusgraphs/VFsO-mYOo-0 [21:17:12] I'm full of regret even for offering [21:22:47] manybubbles: this is how careers are born :) [21:25:06] bd808: Are we supposed to chat before we fill out that form? [21:25:14] Or are we supposed to do it on our own thoughts? [21:25:24] let's just do personal thoughts [21:25:28] mmk [21:25:32] less group think influence [21:25:57] makes sense [21:26:35] * bd808 gags each time he sees how much random js is needed to power greenhouse [21:27:18] Gah, why on earth is google analytics needed for a private thing like this? [21:27:41] scripts sourced from 8 random domains :/ [21:28:00] Ghostery found 5 trackers: app.greenhouse.io [21:28:15] google analyics is the only decent web analytics app left unfortunately [21:28:48] Ok, use it on the public pages. [21:28:57] I don't like when auth'd areas "need" it [21:29:14] why is "Object oriented design" a skill for the project manager interview? [21:30:15] $5 says those are canned for all applicants [21:30:23] Or it was copy+pasted from another req [21:33:19] speaking of that, have you seen the latest services job posting. Very different tone than most I've collaborated on [21:34:21] https://boards.greenhouse.io/wikimedia/jobs/27696 [21:34:31] ori: do we have zookeeper running anywhere? [21:34:42] For kafka, right? [21:34:47] kafka uses zookeeper I think [21:34:52] yeah [21:34:53] yup [21:35:05] puppet says yes [21:35:48] Speaking of kafka, this released in the last couple of days -- https://github.com/yahoo/kafka-manager [21:36:59] bd808: what do you mean by very different tone? [21:37:16] are you upset that it seems to be pitching a very restbase-specific role? [21:37:31] no, it jsut seemed more "rockstar" than normal [21:37:41] I may have been projecting [21:37:57] right. i think it stands out, too, but i think it's a good thing. gabriel does an excellent job with them, and has the hires to show for it. [21:38:18] i actually think we should consciously emulate it for some other reqs (with his permission) [21:38:41] *nod* experiments with these things are good [21:39:23] For a couple of jobs at Kount we ran a/b tests of sorts by posting pretty different reqs to different services [21:39:44] the results were mixed though. not enough data to decide [21:39:53] Are we forming a band? [21:39:56] * wingswednesday saw rockstar [21:41:18] ori: the mediaWikiLoadComplete graph I'm starting at is up but the rise came 60 minutes before the wmf15 revert-revert. [21:41:32] so I'm not seeing a correlation really [21:42:10] i'm in the waiting room of my doc office, which is why i'm chatty but unproductive. i'll take a look, but may not be able to dive deeply for another hour. [21:42:21] coolio [21:42:53] should we turn Reedy loose on wmf16 or suggest that tomorrow would be better? [21:43:18] I'd hate to see another 2-week release stack up honestly [21:43:43] you call it [21:44:06] * bd808 tries and fails to find someone to pass the buck too [21:44:27] well, we can still blame Reedy [21:44:48] Reedy: what's your take? Would running the train tomorrow be an option? [21:45:12] if not then I'm a go for today; if so, I'd give it time to stew [21:45:36] but I don't want to hold until next week for wmf16; too much change [21:45:53] that's kind of the boat we're already in with wmf15 :/ [21:46:02] *nod* [21:46:19] is wmf16 cut yet? [21:46:54] nope [21:50:03] I'm not going to drive the train myself today, so we wait for Reedy to show up and weigh in I think. [21:50:35] What needs doing for wmf15 still? [21:52:14] wmf15 is back on group1, so the next step would be pedias and the wmf16 branch to be cut and sent to group0 [21:52:33] "normal wednesday train" [21:52:52] just 4-6 hours behind schedule [21:53:35] Like I said holding the train to tomorrow would be groovy with me, but holding until next wednesday seems like a bad idea [21:53:42] ohai [21:54:03] we haz a Reedy! [21:54:14] now where is my bukkit? [21:54:34] We could move wikipedias tonight... [21:54:51] I'd rather not have to be up another couple of hours to do wmf16 tonight though [21:55:01] But waiting till next week seems madness [21:55:02] reasonable [21:55:30] it would be just as easy to do it all tomorrow too right? [21:55:47] yeah [21:55:57] what's your gut say? [21:56:07] 3MediaWiki-Core-Team, wikidata-query-service: Figure out if Neo4j is a possible alternative to Titan - https://phabricator.wikimedia.org/T88571#1016037 (10Smalyshev) @JanZerebecki is master/slave not enough for us? The only write scenario is updates, and these can be done on master, all queries are read-only and... [21:56:20] do we know the perf issue is fixed? [21:56:36] no, but we don't know if it is really there either [21:56:55] the graph jumped again but it did it 60 minutes before the revert [21:56:55] heh [21:58:52] doing wmf16 tomorrow sounds reasonable [21:59:07] bd808: Did someone mix thiotimoline into graphite? ;) [23:20:27] greg-g: i think it's fine to go with wmf16 today [23:26:06] Reedy: ? [23:26:16] I'm just about to go to bed [23:26:33] I didn't want to be up late with it... Yet I'm stil up 2 hours later :( [23:27:35] are you still up to doing it, if we start now? [23:27:36] i can help [23:28:30] for wmf16? reedy wanted to do it tomorrow, and I'm inclined to let him make that call :) [23:29:11] +1 for tomorrow. Sam should sleep [23:29:37] how is " are you still up to doing it, if we start now?" different from "I'm inclined to let him make that call" [23:30:00] it isn't [23:30:50] I'm just giving Reedy support that he can say no without letting anyone down [23:30:54] sorry, I just didn't want to push reedy to do it tonight [23:31:31] if Reedy doesn't want to be cool, that's fine [23:31:40] I'm kidding. Go to bed. Sorry for pushing. [23:31:46] :P [23:32:24] (you weren't pushing, I just wanted to make sure *I* wasn't pushing by giving the permission to do it now) [23:33:28] 3MediaWiki-Core-Team, MediaWiki-Configuration, MediaWiki-Vagrant: Update MediaWiki-Vagrant's role system to use extension registration - https://phabricator.wikimedia.org/T86990#1016364 (10Legoktm) 5Open>3Resolved a:3Legoktm [23:34:35] * bd808 gets in a fist fight over who is being more gracious [23:35:12] * greg-g compliments bd808 on his fisticuffs style [23:39:34] Yeah, I'd really rather not tonight :) [23:39:54] I was nodding off to sleep a few hours ago, so went up to bed, and then was more awake (grr) [23:40:04] I'll no doubt be awake again in about 6 hours :P [23:41:27] Reedy: get off the laptop! no blue light! :) [23:41:43] sleep! [23:41:54] QUICKLY! [23:44:37] honestly, we replaced a few bulbs with amber colored ones (to limit the blue light) and it works pretty well for Rowan [23:45:27] csteipp: have you picked a day/time for sketching out the authstack work? [23:47:33] bd808: Not yet... that was pretty much next on my list. Let me do that. [23:59:45] greg-g, can we deploy it without Sam? :P