[15:43:32] test test [15:44:08] http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-traffic/20160120.txt [15:44:16] ^ recursive reference to itself now :P [15:44:30] hello logs [15:44:56] !log does this work too? [15:45:03] I guess that's not that bot [15:45:23] yeah that's morebots [15:45:39] right [15:47:47] <_joe_> nope that's someotherbot [15:48:04] <_joe_> but I guess we want to keep !log entries in -operations [15:48:41] <_joe_> so, pybal on etcd, I think we'd have time before the ops meeting [15:49:03] <_joe_> let me check that there are no discrepancies between the on-disk version and what is on etcd atm [15:49:24] well we have !log coming and going from multiple channels already really [15:49:35] getting logmsgbot in here would echo !log from elsewhere to here [15:49:51] and getting morebots in here with the right config would let !log from here echo elsewhere [15:50:02] but whatever, it can wait [15:55:18] ok after reading 10 times the explanation about the vcl transformation/inclusion I believe that I got a grasp of what's happening. I am wondering what is the "time to change a vcl file without setting everything on fire" for a new engineer [15:55:30] :D [15:59:30] <_joe_> elukey: it took me a month or so, and a sizeable amount of gut-inducing psychotropes [16:00:13] :) [16:00:29] it's nasty, and most things to do with the cp* machines are dangerous live changes [16:00:54] one of the many ongoing things is to slowly refactor all of that into a better place [16:01:11] modules/varnish isn't the right place for site-specific VCL and config anyways [16:01:21] and templates/vanrish/ sucks too, it's not in any module [16:02:05] and the whole way we use multiple definitions of the standard vcl_foo subroutines is hard to follow too [16:02:06] _joe_: all right maybe I'll be able to do it in 6 months, we'll see :D [16:02:29] probably the next stopping points in slowly making it better are something like: [16:03:26] 1) move modules/varnish/templates/vcl/wikimedia.vcl.erb somewhere in modules/role/templates/cache/ referenced from modules/role/manifests/cache/base.pp, or something like that? [16:03:38] 2) move the templates/varnish/ templates into the role module similarly? [16:04:17] 3) make it so there's only one definition of each varnish-defined entrypoint subroutine, and thus include order matters less and is less confusing [16:04:50] as in either have the central mediawiki.vcl template have the only vcl_recv and such, and have it call standard names like text_frontend_recv, or something along those sorts of lines [16:05:32] or put vcl_recv and such in the included text-frontend.vcl.inc.erb and have that central template have none of the standard ones, and instead names its routines something like common_vcl_recv, which the other ones "call" [16:05:50] or something like any of that, but better [16:06:01] get it to where it's obvious what the flow of control is [16:06:53] right now it goes every which way. the central and per-cluster/layer VCL templates both define instances of various top-level vcl_foo subroutines, and define custom subroutines, and sometimes one calls a custom subroutine from the other. it's madness. [16:08:37] the templates moving around the puppet directories seems simple on the surface, but there's a lot of complexity to deal with in the related classes too [16:09:22] there's something of a related plan which was kinda going the wrong way in https://phabricator.wikimedia.org/T96847 [16:20:34] <_joe_> ouch, i have another meeting before the ops meeting [16:20:35] <_joe_> gee [16:20:42] <_joe_> no change today I guess [16:27:03] bblack: I'm adding some traffic-related points to the ops meeting etherpad [16:27:22] ema: awesome [16:29:40] https://gerrit.wikimedia.org/r/#/c/265281/ will get us some more bot traffic in here [16:30:09] I left off operations since we're all in -ops anyways, but it will highlight the traffic-related phab updates in here for relevancy [16:30:49] oh that's nice [16:35:13] hmm I need to get this in chanserv control too, and of course now we're all -o from various quits/splits :P [16:45:08] hey [16:47:52] thank you :) [16:51:40] so I don't lose it for now while I dork around with chanserv after the meeting :) [16:54:30] Traffic, operations: test ticket - https://phabricator.wikimedia.org/T124182#1948370 (Krenair) NEW [16:54:41] Traffic, operations: test ticket - https://phabricator.wikimedia.org/T124182#1948378 (Krenair) Open>Invalid a:Krenair [16:54:51] ugh, +c [16:54:54] Krenair: thanks :) [16:54:58] I can fix that [16:56:04] It might be that I didn't need to restart the bot. It did some auto-update of the repo and recognised the channels conf change, but then only appeared in the channel when I made the ticket to test it [16:56:13] oh well [16:58:55] anyways, ops meeting starting, I'll figure out the rest of chanserv later... [17:05:15] 10Traffic, 10MediaWiki-General-or-Unknown, 10MobileFrontend-Feature-requests, 6operations: Fix mobile purging - https://phabricator.wikimedia.org/T124165#1948456 (10BBlack) ping to check bot [17:55:34] 10Traffic, 10Fundraising-Backlog, 10Unplanned-Sprint-Work, 6operations, 5Patch-For-Review: Firefox SPDY-coalesces requests to geoiplookup over text-lb, causing GeoIP IPv6 failures - https://phabricator.wikimedia.org/T121922#1948639 (10BBlack) 5Open>3Resolved a:3BBlack Assuming this is no longer an... [18:03:09] hi [18:04:11] hi :) [18:05:07] this channel was originally just made for one conversation a few weeks back. Then I realized that everything I was saying to @ema about traffic-y stuff would be better here than in /msg, and so now it's a "real" channel [18:05:24] we have bots and logs and stuff :) [18:06:20] hehe nice! yeah I think I'll mostly lurk, still interested in what's going on [18:06:52] ops tends to be too spammy for some of the involved conversations on varnishy things [18:07:00] and the other option isn't logged and isn't ideal either [18:07:04] so here we are :) [18:23:41] hey godog :) [18:24:00] ciao ema / ema_ ! [18:24:38] so I'm about to go afk, last update for today is that varnish vmods do not seem to be compatible between versions [18:25:13] eg: I've tried building libvmod-vslp against varnish 4.0 and it does not work with 4.1 [18:25:57] it's nice that you don't need the whole varnish source tree for building them, libvarnishapi1 seems to be enough [18:26:35] 10Traffic, 6operations: compressed http responses without content-length not cached by varnish - https://phabricator.wikimedia.org/T124195#1948818 (10Dzahn) [18:26:53] also: http://www.gossamer-threads.com/lists/varnish/misc/37939 [18:27:02] 10Traffic, 6operations: compressed http responses without content-length not cached by varnish - https://phabricator.wikimedia.org/T124195#1948821 (10BBlack) Yeah copying a bit from IRC discussion at the time: basically anytime apache's configured to gzip stuff, it's going to have a deflate buffer size limit c... [18:27:12] 10Traffic, 6operations: compressed http responses without content-length not cached by varnish - https://phabricator.wikimedia.org/T124195#1948829 (10Dzahn) you could call it just an Apache config issue instead of varnish, but tagged "Traffic" anyways, feel free to remove it again if you think it shouldn't be [18:28:00] ema: any chance it will work for 4.1.0 vs 4.1.1 though, hypothetically? [18:28:01] it is relatively straightforward to package stuff like libvmod-vslp and play with it, but then for each varnish upgrade we need to rebuild the vmods as well [18:28:16] or even 4.1.0-wmf1 vs 4.1.0-wmf2 or whatever the version scheme looks like [18:28:38] 4.1.0-wmf1 vs 4.1.0-wmf2 should not be a problem, but I haven't tried yet [18:29:08] 10netops, 6operations, 10ops-codfw, 5codfw-rollout, 3codfw-rollout-Apr-Jun-2015: Codfw mediawiki appservers from any rows but row A can't communicate with the dhcp server - https://phabricator.wikimedia.org/T92815#1948845 (10Aklapper) [18:29:17] 10Wikimedia-Apache-configuration, 10Wikimedia-Site-Requests, 6operations, 5codfw-rollout, 3codfw-rollout-Apr-Jun-2015: Configure mediawiki to operate in the Dallas DC - https://phabricator.wikimedia.org/T91754#1948850 (10Aklapper) [18:30:06] ema: I think before it was generating a hash at build time or something is what I mean [18:30:17] so every unique "make" -> must rebuild vmods against it [18:30:38] 4.0 -> 4.1 probably has legitimate API changes anyways [18:31:04] exactly, 4.0 -> 4.1 does have API changes [18:31:27] I'll let you know tomorrow whether 4.1.0-wmf1 -> 4.1.0-wmf2 works :) [18:31:40] as long as we can get compat on the point-releases and wmf releases, I think we're better off not bundling [18:31:43] ok, cya tomorrow [19:07:54] 10Wikimedia-Apache-configuration: https://test.wikipedia.org/wiki/Bug%3F?action=history doesn't show the history page, unlike https://test.wikipedia.org/w/index.php?title=Bug%3F&action=history - https://phabricator.wikimedia.org/T123276#1949050 (10Kipod) >>! In T123276#1948781, @Krinkle wrote: > Works for me. >... [19:09:55] so, been tracking down the purge thing on the MW side.... [19:10:28] https://github.com/wikimedia/mediawiki/blob/master/includes/Title.php#L3540 - getCdnUrls there sets the list of URLs to purge for an article, and it does two by default - the standard canonical one and action=history [19:10:44] at the bottom it does: Hooks::run( 'TitleSquidURLs', array( $this, &$urls ) ); [19:12:00] and then over in mediawiki-config, we have this which MobileFrontend consumes: https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/InitialiseSettings-labs.php#L272 [19:12:13] which is wmgMobileUrlTemplate template with stuff like 'default' => '%h0.m.%h1.%h2.%h3.%h4', [19:12:37] so I suspect the "right" answer is that MobileFrontEnd extension needs to gain a TitleSquidURLs hook [19:13:15] and in that hook, it should look at the existing array of $urls set from Title.php, and transform them with wgMobileUrlTemplate into a copy with mobile domainnames and then append those to the list [19:14:19] (and that all needs to be configurable somehow I guess, since the default behavior is probably-working for others now) [19:43:37] 10Wikimedia-Apache-configuration: https://test.wikipedia.org/wiki/Bug%3F?action=history doesn't show the history page, unlike https://test.wikipedia.org/w/index.php?title=Bug%3F&action=history - https://phabricator.wikimedia.org/T123276#1949229 (10BBlack) Well the description is a little convoluted, but what I'm... [19:57:59] https://gerrit.wikimedia.org/r/#/c/265316/ [19:58:54] hi greg :) [20:25:36] ahah, look at this cohort of migration watches you have gathered! [20:26:09] ottomata: no migration for now [20:26:15] I should update the ticket! [20:26:36] we had some forgotten dependencies, which are now blocking the task: https://phabricator.wikimedia.org/T109286 [20:27:44] 10Traffic, 6Zero, 6operations, 5Patch-For-Review: Merge mobile cache into text cache - https://phabricator.wikimedia.org/T109286#1949446 (10BBlack) The traffic move from mobile->text is now on hold (we did convert codfw, then we rolled back) due to purge-related issues that need to be addressed first, in b... [20:28:25] Oh [20:28:26] ok [20:28:27] thanks [20:28:29] hm [20:28:50] I'm not sure who to talk to about getting the purges right on the MW side really, but I think I have a half-assed patch to MobileFrontEnd that might work now heh [20:32:42] hehe [20:32:43] cool [20:33:10] are you still planning on proceeding soonish? (like within a week), or is this blocked on some bigger team coordination and deploy? [20:35:12] ottomata: as soon as I can, but at this point we're blocked on some kind of MW patch getting written->reviewed->deployed (to all prod wikis) [20:35:37] I don't have any idea what our timeline is yet [20:35:47] https://gerrit.wikimedia.org/r/#/c/265316/ is my step 1 so far [20:36:07] but I wouldn't be confident pushing for a merge without someone with MFE/core experience looking at the problem first [21:40:08] 7domains, 6operations: traffic stats for typo domains - https://phabricator.wikimedia.org/T124237#1949803 (10Dzahn) 3NEW a:3Dzahn [22:02:27] 7domains, 6operations: traffic stats for typo domains - https://phabricator.wikimedia.org/T124237#1949879 (10Dzahn) | domain | period | date | hits | | wikiepdia.com | 1d | 2016-01-19 | 71 | | wikiepdia.org | 1d | 2016-01-19 | 104| | wikimediacommons.co.uk | 1d | 2016-01-19 | | | wikimediacommons.eu | 1d | 201... [23:35:04] 10Traffic, 6Phabricator, 6Release-Engineering-Team, 6operations, 5Patch-For-Review: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#1950387 (10mmodell) git-ssh.wikimedia.org has an ipv6 address in DNS, however, it's not yet active due to lack of time to work on this. We need t... [23:43:48] 10Traffic, 6Phabricator, 6Release-Engineering-Team, 6operations, 5Patch-For-Review: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#1950423 (10Dzahn) >>! In T100519#1950387, @mmodell wrote: > git-ssh.wikimedia.org has an ipv6 address in DNS, however, it's not yet active due to...