[00:06:10] * AaronS watches orient choke on https://www.wikidata.org/wiki/Q15646406 [00:06:37] what's special aobut that item? [00:07:21] Sub\urban [00:07:29] I bet it's that backslash [00:07:45] https://de.wikipedia.org/wiki/Suburban_(Zeitschrift) [00:07:50] which is legit afaik [00:35:29] hmm, it's not the two items with special chars...at least I can't repro with simple examples [00:35:34] AaronS: ResourceLoaderWikiModule could benefit from using the BloomFilter to check whether the wiki page exists; what's the best way to tap into that? [00:35:36] including the \ [00:37:24] ori: are there many requests for garbage titles? [00:39:14] AaronS: yes -- for example, many wikis have Monobook.css, but few have Monobook.js [00:39:33] wouldn't simple key/value caching handle that? [00:39:51] I guess so, but this seemed cleaner [00:40:16] what could maybe use it is pending changes, since only a tiny portion of pages are protected [00:40:23] * AaronS is too lazy to do that any time soon [00:45:40] Krinkle: This hhvm issue could be related to the unexpected suppressed error logging -- https://github.com/facebook/hhvm/issues/4407 [00:55:13] manybubbles: three bug reports so far :) [01:21:19] bd808: https://github.com/orientechnologies/orientdb/issues/2424 scares me [01:21:31] I bet when it finally gets fixed, then the work arounds will break [01:21:44] escaping/encoding is like that :) [01:23:08] looks nasty all right. [01:44:34] AaronS: are you OK w/https://gerrit.wikimedia.org/r/178350 ? [01:44:57] I asked about it earlier, never got a reply though [01:45:10] sorry, I must have missed it. what did you ask? [01:45:12] were you saying zend was somehow causing cache missed? [01:45:40] *misses [01:45:49] no, i'm saying the fact that api requests are served by zend makes the preemptive save unlikely to be any faster than just waiting for the 'submit' and submitting to hhvm [01:46:12] that makes more sense [01:46:32] reimaging the api cluster was blocked on the tidy thing, but now that that's resolved i think it'll happen this week [01:46:46] but in the meantime we're just taxing users on wmf11 with additional network traffic [03:34:23] Interesting. phabricator.wikimedia.org runs PHP/5.5 [03:34:28] Or is that hhvm in disguise? [03:34:36] (according to http response headers) [03:38:20] if correct, that means we're running both php 5.3, hhvm and php 5.5 in prod. [03:38:27] PHP/5.5.9-1ubuntu4.5 [03:55:39] 3MediaWiki-Core-Team, CirrusSearch: Investigate if anyone is still using lsearchd - https://phabricator.wikimedia.org/T77921#832798 (10jeremyb) I guess this use case is rather obvious but we need lsearchd in order to compare results between old and new. e.g. T75977 [04:29:34] Dec 9 04:28:21 mw1017 hhvm: #012Fatal error: $this is null in /srv/mediawiki/php-1.25wmf10/extensions/Math/MathRenderer.php on line 205 [04:29:36] blech [04:41:17] TimStarling: if you have a moment, https://gerrit.wikimedia.org/r/#/c/178430/ is a one-line patch that should fix a fatal in production. [04:46:19] thanks [05:29:58] charming [05:30:04] the unit tests depend on some external web service [06:01:55] Not the only case in extensions iirc [06:33:12] 3MediaWiki-Core-Team, CirrusSearch: Add per user concurrent search request limiting - https://phabricator.wikimedia.org/T76497#832870 (10aaron) a:3Manybubbles [06:34:15] <_joe_> "unit" tests? [09:11:52] 3MediaWiki-Core-Team, MediaWiki-General-or-Unknown: At least one user is permanently unable to use Special:RecentChanges on translatewiki.net - https://phabricator.wikimedia.org/T74228#833045 (10Nikerabbit) [09:13:01] 3MediaWiki-Core-Team, MediaWiki-General-or-Unknown: At least one user is permanently unable to use Special:RecentChanges on translatewiki.net - https://phabricator.wikimedia.org/T74228#833048 (10Nikerabbit) p:5High>3Normal Priority cannot be high if nobody is working on it. [09:41:10] 3MediaWiki-Core-Team, MediaWiki-General-or-Unknown: At least one user is permanently unable to use Special:RecentChanges on translatewiki.net - https://phabricator.wikimedia.org/T74228#833064 (10Nemo_bis) [10:06:27] 3MediaWiki-Core-Team, MediaWiki-General-or-Unknown: At least one user is permanently unable to use Special:RecentChanges on translatewiki.net - https://phabricator.wikimedia.org/T74228#833114 (10Nikerabbit) [10:46:41] 3MediaWiki-Core-Team, MediaWiki-API: Generator list duplicated - https://phabricator.wikimedia.org/T77974 (10Reedy) 3NEW p:3Triage a:3Anomie [13:15:10] 3MediaWiki-Core-Team, Parsoid-Team, MediaWiki-Parser: Mechanism to find usages of raw-html messages - https://phabricator.wikimedia.org/T21291#833374 (10Nemo_bis) [13:57:44] 3Code-Review, MediaWiki-Core-Team: Wikimedia code repository browser in Phabricator - https://phabricator.wikimedia.org/T752#833429 (10Nemo_bis) I like that top bar search by hash (e.g. 00cf19dc65) works, seemingly everywhere. [14:29:30] 3MediaWiki-Core-Team, MediaWiki-API: Generator list duplicated - https://phabricator.wikimedia.org/T77974#833513 (10Anomie) [14:46:10] manybubbles, ^d: T76967 seems like someone complaining about something search-related, but beyond that I don't know what he's doing there to even know which "search" might be involved. [15:08:13] anomie: looking [15:19:17] replied [15:27:22] manybubbles: Thanks [15:28:57] 3MediaWiki-Core-Team, MediaWiki-API: Generator list duplicated - https://phabricator.wikimedia.org/T77974#833666 (10Anomie) [15:29:38] 3MediaWiki-Core-Team, MediaWiki-API: Generator list duplicated - https://phabricator.wikimedia.org/T77974#833667 (10Anomie) 5Open>3Resolved [15:36:46] <^d> *yawn* morning folks [15:53:00] morning! [16:18:21] 3MediaWiki-Core-Team, Multimedia: SHA-1 file name support - https://phabricator.wikimedia.org/T1210#833717 (10Gilles) Just to get a sense of where we're at @aaron, now that I finally have time to look at this. Do you think we have bad/insufficient/ok/good test coverage for this change? In particular I wonder if... [16:22:22] Error: Could not retrieve catalog from remote server: Error 400 on SERVER: custom functions must be called with a single array that contains the arguments. For example, function_example([1]) instead of function_example(1) at /etc/puppet/modules/mediawiki/manifests/hhvm.pp:6 on node i-0000044e.eqiad.wmflabs [16:22:32] Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type sudo::user at /etc/puppet/modules/base/manifests/monitoring/host.pp:87 on node i-0000044e.eqiad.wmflabs [16:22:39] * bd808 grumbles [16:55:27] 3MediaWiki-Core-Team, MediaWiki-extensions-SecurePoll: When translations are changed on control wiki the change should be updated on jump wikis - https://phabricator.wikimedia.org/T74576#833779 (10Nemo_bis) > the translation interface on the 'control' wiki What's this? [17:24:36] 3MediaWiki-Core-Team, MediaWiki-extensions-SecurePoll: Avoid holding many open DB connections when saving SecurePoll edits - https://phabricator.wikimedia.org/T78013 (10Anomie) 3NEW p:3Triage a:3Anomie [17:26:25] 3MediaWiki-Core-Team, MediaWiki-API: list=logevents in API shows type/action of suppressed and revdeleted log entries - https://phabricator.wikimedia.org/T74222#833936 (10Anomie) 5Resolved>3Open [17:32:23] Did we merge something that is dumping json to the php error log channel? hhvm in beta is writing json stack traces from MWMultiVersion [17:34:17] <^d> It should go to a new log I'd hope. [17:34:23] <^d> But yeah, Timo had a patch up for that. [17:35:21] https://phabricator.wikimedia.org/P134 [17:35:30] it's showing up in syslog from hhvm [17:35:36] at least in beta [17:36:08] which I guess would be from error_log() [17:36:30] <^d> I don't see it merged in mw/core master. [17:36:36] <^d> Must be something else... [17:37:02] <^d> https://gerrit.wikimedia.org/r/#/c/177095/ [17:37:05] <^d> isn't merged. [17:37:15] * bd808 will git grep for things [17:37:25] It could be hhvm changes too I guess [17:38:07] The log stream has a "Fatal error: Invalid host name (www.es.wikipedia.beta.wmflabs.org). in /srv/mediawiki/multiversion/MWMultiVersion.php on line 366" followed by that json exception dump [17:38:19] as 2 separate syslog events [17:46:02] The origin is a trigger_error() call in MWMultiVersion [17:46:10] * bd808 looks deeper [17:50:50] * bd808 found it [17:51:34] ^d: mystery solved -- https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/HHVMRequestInit.php [17:52:13] morning [17:52:26] \o [17:52:37] <^d> yo [18:01:07] 3MediaWiki-Core-Team, Release-Engineering: Make ::mediawiki::syslog and ::mediawiki::php logging destinations configurable via hiera - https://phabricator.wikimedia.org/T1295#833982 (10greg) [18:02:40] ^d: bd808: exception-json has been in mediawiki for a while though [18:02:44] maybe it's that one? [18:02:51] <^d> No, it was something different. [18:03:01] Krinkle: It's a special hhvm multiverison hack [18:03:53] ^d: looks like php backtrace() [18:03:58] jsonified [18:03:59] yeah [18:15:29] ori: does eventlogging go anywhere on Beta Cluster? there are requests to event.gif on beta's bits at least. [18:15:43] ori: ebernhardson is asking, I'm just relaying really [18:25:51] * ^d stabs apt [18:26:38] sounds like you want something more snappy? ;) [18:27:23] * YuviPanda installs OS X on legoktm’s servers [18:27:43] <^d> W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/universe/i18n/Translation-en Hash Sum mismatch [18:27:44] <^d> :( [18:34:46] legoktm: are you in hurry with the api prop patch? I'll prolly get to it on thursday [18:35:48] Nikerabbit: I don't think so, that should be fine [18:42:45] greg-g: yes, they go to deployment-eventlogging02 [18:42:52] nuria from analytics should be able to help [18:43:51] ori: cool, thanks sir [18:46:51] ori: I guess the edit stash stuff can be enable [18:47:19] AaronS: yep [18:55:31] legoktm: What would it take to make the wikibugs bot ignore things that only involve a particular project? Easy(ish)? [18:56:44] legoktm: My use case is that I have a personal project for testing out self-calendaring and I may get kind of spammy with the things I put in it. -- https://phabricator.wikimedia.org/project/view/929/ [18:58:48] bd808: should be easy. Should it not go to -feed? [18:59:13] I don't mind if it goes to feed, but probably not anywhere else [18:59:33] * bd808 doesn't want to flood -dev for example [19:00:20] ok, give me a few minutes [19:00:27] I'd be fine with it not going to -feed either but don't really care. I'm not going to use it to track my lunch order or anything. :) [19:11:19] <^d> I finished the xhprof refactoring for vagrant. [19:11:42] sweet. I'll check it out after lunch [19:11:46] it would have been a few minutes except the tests are failing....argh [19:12:12] legoktm: No worries. I actually expected a "sure, submit a patch" response [19:13:20] I'm not sure how the bot is running right now >.> [19:13:33] ori: you should turn off half your servers and save some power :) [19:14:24] hhvm is pretty great [19:15:51] The coolest things about hhvm are the little things that a lot of folks will never notice. Like being able to log fatal errors. That is awesome. [19:16:23] swtaarrs: it's a shame tim's pcre patch stalled tho :/ [19:16:59] yeah I keep starting to look at it and getting distracted :( [19:17:13] I might grab someone else and lock us in a room until we get through it [19:18:53] there's a pun about lock-free data structures here somewhere, but i'm too tired this morning to find it [19:23:26] bd808|LUNCH: scap is brokened [19:23:34] for every host [19:23:34] bd808|LUNCH: your project should be ignored now [19:23:35] 19:23:09 sudo -u mwdeploy -n -- /usr/bin/rsync -l tin.eqiad.wmnet::common/wikiversions*.{json,cdb} /srv/mediawiki on mw1127 returned [255]: Host key verification failed [19:24:15] ori: ^^ [19:24:30] _joe_: ^ [19:24:48] lol [19:25:24] <_joe_> Reedy: ^^ [19:25:30] swtaarrs: ^^ [19:25:44] <_joe_> Reedy: this is strange actually [19:25:52] <_joe_> but I'm a bit too tired to look now [19:26:08] <_joe_> it's probably some puppet-based fuckup [19:26:21] <_joe_> we can just depool the server for now and look tomorrow? [19:26:29] the = 266 [19:26:40] of them, that is [19:27:01] <_joe_> greg-g: uh? [19:27:12] 14:25 < Reedy> !log that is a lie, 266 hosts failed [19:27:13] <_joe_> greg-g: all the appservers have the same error? [19:27:20] <_joe_> oh ok [19:27:22] yup [19:27:32] _joe_: All gave Host key verification failed [19:27:35] <_joe_> then someone broke something [19:27:40] right. [19:27:44] did someone push a scap update? [19:27:46] * Reedy checks [19:27:47] <_joe_> escalate this with opsens in a better timezone [19:28:04] i tried to set hostkeyverification off on tin just now, didn't help [19:28:07] because it's on the scap proxies [19:28:07] who do you recommend? [19:28:15] Who's there? ;) [19:28:52] <_joe_> it wasn't broke when anomie swatted this morning [19:29:09] <_joe_> (pacific time) [19:29:13] brandon, chase, rob, chris, jeff ? [19:35:23] ^d: https://gerrit.wikimedia.org/r/#/c/178458/ [20:08:04] HHVM: [20:08:11] 1) https://phabricator.wikimedia.org/T75181#809065 Is this task resolved? [20:08:21] 2) does James_F|Away 's comment there need a followup task created? [20:08:44] 3) Can the BetaFeature be removed/graduated now that everyone is using it? [20:09:33] quiddity: The beta feature should just be removed. There is no going back now. :) [20:09:51] <^d> Well we could go back to php5 :) [20:12:54] 3) filed as https://phabricator.wikimedia.org/T78044 [20:19:50] 3MediaWiki-Core-Team: $wgThumbnailMinimumBucketDistance and $wgThumbnailBuckets not documented in RELEASE NOTES - https://phabricator.wikimedia.org/T78053#834460 (10Kghbln) [20:26:29] <_joe_> ^d: if we go back, you're welcome to reimage all appservers; all 260 of them :) [20:27:12] <^d> gogogo [20:28:50] ori: hmm? [20:29:39] <^d> manybubbles: gah, shit :( [20:29:42] <^d> demon@fluorine:/a/mw-log$ grep srbackend=LuceneSearch api.log | wc -l [20:29:42] <^d> 196779 [20:30:09] hmmm - that is a bunch [20:34:10] Can't we just use php on 14.04? [20:34:40] <_joe_> Reedy: well, once you reconfigure everything, maybe [20:34:47] heh :) [20:34:51] <_joe_> I'd still require you to reimage the servers [20:35:01] <_joe_> just to see you endure my same pain :P [20:35:18] _joe_: I need root access [20:35:22] There's an RT ticket for it [20:35:23] ;D [20:48:43] ^d: looks like its mostly one ip address and something else on nlwiki [20:49:03] <^d> Historically too? [20:53:10] 3MediaWiki-Core-Team: Low hanging fruit performance tweaks - https://phabricator.wikimedia.org/T76677#834616 (10Aklapper) [20:54:49] 3MediaWiki-Core-Team, Wikimedia-General-or-Unknown: History of Utilisateur:Binabik@frwiki missing after global rename - https://phabricator.wikimedia.org/T76979#834618 (10Legoktm) I've restored the page histories for Utilisateur:Binabik and Utilisateur:Binabik/Modèle (the current revision is still a self-redirec... [20:57:55] ^d: I wonder how php5-json got installed on my vm? I totally agree however and wonder wtf ubunutu/debian was thinking by making it a separate package. [20:59:45] I think that was because of the licensing issue [20:59:47] bd808: The php5-json in Debian isn't the json that's built into PHP. [21:00:29] Ah. It's the libjson fork without do-no-evil? [21:05:45] Yes [21:06:03] (or something without do-no-evil, anyway) [21:06:12] *nod* [21:06:20] <^d> So, bad commit. [21:06:25] <^d> Or, not needed really [21:06:37] ffs. I've uploaded an unaccessible file on commons [21:06:47] It was already installed for me on 2 vms [21:06:48] <^d> gj [21:06:52] but it hurts nothing [21:07:36] I can't move it, and I can't delete it [21:07:36] ffs [21:12:51] I'd really like to cleanup all the installed stuff in /vagrant by moving to a /vagrant/src directory. I suppose that would be exactly the kind of breaking change that folks are talking about on the mailing list :) [21:13:49] https://phabricator.wikimedia.org/T78060 [21:13:52] * Reedy sighs [21:14:24] <^d> I might just expose phab on a port. [21:14:30] <^d> It'd be easier than that wmftest stuff. [21:14:47] bah. the named vhosts are easy [21:14:49] or should be [21:15:05] what's tricky about phab? [21:15:41] <^d> Do I have to do anything in my /etc/hosts for wmftest support? [21:15:47] <^d> I just use localhost:8080 for my wiki :) [21:15:56] nope. it's live dns we serve [21:16:09] *.local.wmftest.net == 127.0.0.1 [21:16:37] <^d> webpage can't be displayed :\ [21:18:16] Is that docroot right? [21:18:35] DocumentRoot <%= @deploy_dir %>/public [21:18:42] <^d> Ah, whoops. [21:18:47] <^d> Fixing that + rewrites. [21:19:16] also... that's puppet/modules/iegreview/templates/apache.conf.erb [21:19:44] in the gerrit commit [21:20:15] <^d> Yeah whoops. [21:23:09] <^d> lol. [21:23:23] <^d> helps to clone the right repo. [21:24:04] <^d> I cloned arcanist into ./phabricator :p [21:24:36] heh. move fast and break things! [21:29:44] <^d> Whee, http://phabricator.local.wmftest.net:8080/ works [21:30:12] cool [21:30:54] * bd808 hopes that CableOne figures out what part between his house and their head end is broken soon [21:31:21] having my connection drop for 2 minutes every 4-6 hours is getting really really old [21:32:40] The tech who came out yesterday showed me their monitoring graphs that clearly pointed to the problem being in their network equipment. Then he ignored that for 2 hours while he tried to prove the problem was in my house wiring. :/ [21:32:47] ori: fyi, I talked with the team this morning about hhvm on jenkins slaves, hashar is brain dumping on the phab task for it now, and I gave him (and by extension you) permission to rope in everyone he needs (including zeljko or dan or mukunda as approppriate). joe and antoine/zeljko have good timezone overlap so hopefully those three can get it done, but if there's anything you can help with (direction/idea-wise), please do, giving antoine a backlo [21:32:56] #tldr [21:33:29] greg-g: we need to reimage salves with 14.04 and then it's bascially done right? [21:33:35] *slaves [21:33:47] bd808: apparently not? https://phabricator.wikimedia.org/T75356#834842 [21:34:01] * bd808 looks [21:34:36] also, I would share a hair pulling story about me and ATT not getting along for about 2 months, which ended with a free couple months from them, but I've blocked most of it out of memory [21:35:16] hashar is making it too hard [21:35:29] hhvm slaves, php5.3 slaves [21:35:47] no fancy switching between version [21:36:08] or go full out and use phpenv like Travis does [21:37:09] I tested the xhprof PECL changes I made recently using a VM built from the Travis chef config [21:37:17] It worked like a charm [21:37:37] sense slapping appreciated :) [21:45:42] Are timeline images not generated on the image scalers? [21:46:01] AaronS: ^^ [21:47:25] I guess not [21:47:28] As it'll just shell out to timelineFile [21:50:16] anomie: I need to fix an api thing for mobile and want some advice. [21:50:37] when using generator=search&gsrinfo=suggestion they don't get anything back [21:51:29] manybubbles: Yeah, that's expected. It could be done with the same sort of generator-data thing, but that's a bit ick. [21:51:31] where should that come back? [21:51:45] anomie: they want it :( [21:51:56] without it they lose the suggestion which is good for misspellings [21:52:48] Do they also want a pony? [21:53:09] one with a horn on the forehead [21:56:06] Generators are for generating, and not for getting all sorts of extra info... [21:56:20] * bd808 sends mobile-apps http://www.djangopony.com/ [21:58:29] anomie: https://gerrit.wikimedia.org/r/178660 [21:59:05] thanks [22:06:59] anomie: about? [22:07:09] I think your opensearch merge into core has a few issues [22:07:10] SELECT page_namespace,page_title,rd_namespace,rd_title FROM `page`,`redirect` WHERE (rd_from = page_id) AND (rd_interwiki IS NULL OR rd_interwiki = '') AND () [22:07:30] Tue Dec 9 22:06:43 UTC 2014 mw1132 enwiktionary ApiOpenSearch::search 10.64.32.30 1064 You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ')' at line 1 (10.64.32.30) SELECT page_namespace,page_title,rd_namespace,rd_title FROM `page`,`redirect` WHERE (rd_from = page_id) AND (rd_interwiki IS NULL OR rd_interwiki = '') AND () [22:08:22] Reedy: ... Ah, I see what's going on there. [22:08:25] 3MediaWiki-Core-Team, MediaWiki-API: Query error in ApiOpenSearch::search - https://phabricator.wikimedia.org/T78074 (10Reedy) 3NEW p:3Triage a:3Anomie [22:10:12] I guess there's no pages in the linkbatch? [22:12:19] Exactly [22:12:27] https://gerrit.wikimedia.org/r/178667 [22:13:51] anomie: Guess we should swat it... [22:13:53] sigh :) [22:15:06] 3MediaWiki-Core-Team: Make Parser::parse tolerate recursion - https://phabricator.wikimedia.org/T76651#835080 (10Welterkj) @matmarex, thanks for the info about recursiveTagParse. The situation I'm grappling with is shown in the backtrace in T75073#763545. WikitextContent::getParserOutput called parser::parse i... [22:16:53] 3MediaWiki-Core-Team: Make Parser::parse handle recursion better (throw exception or tolerate it) - https://phabricator.wikimedia.org/T76651#835093 (10Welterkj) [23:18:26] 3MediaWiki-Core-Team: Make Parser::parse handle recursion better (throw exception or tolerate it) - https://phabricator.wikimedia.org/T76651#835325 (10matmarex) I vaguely recall that cloning $wgParser at the wrong time used to cause PHP to explode spectacularly, Brian might remember the details. You can always... [23:24:43] legoktm: Any reason you didn't make checkuser its own vagrant role when you setup AccountInfo? [23:25:45] probably didn't think of it [23:26:21] should be trivial to split out [23:33:42] 3Parsoid, MediaWiki-Interface, MediaWiki-Core-Team, Mobile-Web: Switch MobileFrontend over to using Parsoid for its read HTML - https://phabricator.wikimedia.org/T76970#835354 (10Jdlrobson) [23:47:38] legoktm: Do you think adding composer install support to extensions would be a good GCI task? Or a crazy waste of time? [23:47:46] Or both: :) [23:48:10] rand() extension we care about presumably? ;) [23:48:26] yeah. start with the wmf installed ones [23:59:40] bd808: waste of time because the RfC is still up in the air [23:59:54] legoktm: hmmm. ok