[00:08:44] 10Traffic, 06Operations: Content purges are unreliable - https://phabricator.wikimedia.org/T133821#2245593 (10BBlack) [00:09:54] 10Traffic, 06Operations, 13Patch-For-Review: Decrease max object TTL in varnishes - https://phabricator.wikimedia.org/T124954#1970940 (10BBlack) [00:09:56] 10Traffic, 10MediaWiki-Cache, 10MediaWiki-JobQueue, 10MediaWiki-JobRunner, and 3 others: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan - https://phabricator.wikimedia.org/T124418#1955365 (10BBlack) [00:10:00] 10Traffic, 07Varnish, 06Operations: Install XKey vmod - https://phabricator.wikimedia.org/T122881#1916107 (10BBlack) [00:10:03] 10Traffic, 06Operations: Content purges are unreliable - https://phabricator.wikimedia.org/T133821#2245614 (10BBlack) [00:10:24] 10Traffic, 06Operations: Content purges are unreliable - https://phabricator.wikimedia.org/T133821#2245593 (10BBlack) 05Open>03stalled p:05Triage>03High [00:10:37] 10Traffic, 06Operations: Content purges are unreliable - https://phabricator.wikimedia.org/T133821#2245593 (10BBlack) [00:10:40] 10Traffic, 06Commons, 06Operations, 10media-storage, and 2 others: upload-lb.ulsfo.wikimedia.org still allow access to some deleted files - https://phabricator.wikimedia.org/T133819#2245416 (10BBlack) [00:10:46] 10Traffic, 06Operations: Content purges are unreliable - https://phabricator.wikimedia.org/T133821#2245593 (10BBlack) [00:10:49] 10Traffic, 06Commons, 06Operations, 10media-storage, and 2 others: Deleted files sometimes remain visible to non-privileged users if permanently linked - https://phabricator.wikimedia.org/T109331#2245623 (10BBlack) [00:11:01] 10Traffic, 06Operations: Content purges are unreliable - https://phabricator.wikimedia.org/T133821#2245593 (10BBlack) [00:11:05] 10Traffic, 06Commons, 10MediaWiki-File-management, 06Multimedia, and 2 others: Image cache issue when 'over-writing' an image on commons - https://phabricator.wikimedia.org/T119038#2245625 (10BBlack) [00:20:45] 10Traffic, 06Operations: Content purges are unreliable - https://phabricator.wikimedia.org/T133821#2245639 (10BBlack) [01:03:18] 10Traffic, 06Operations: Content purges are unreliable - https://phabricator.wikimedia.org/T133821#2245711 (10BBlack) I perhaps should've noted this in the description, but we also attempted one partial general improvement and reverted it. The improvement was to move from the singular multicast address we hav... [01:29:45] 10netops, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: Allocate labs subnet in dallas - https://phabricator.wikimedia.org/T115491#2245799 (10chasemp) [01:29:47] 10netops, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: Allocate labs subnet in dallas - https://phabricator.wikimedia.org/T115491#1725570 (10chasemp) [01:29:51] 10netops, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: Allocate subnet for labs test cluster instances - https://phabricator.wikimedia.org/T115492#2245801 (10chasemp) [01:29:55] 10netops, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: Allocate subnet for labs test cluster instances - https://phabricator.wikimedia.org/T115492#1725583 (10chasemp) [01:29:59] 10Traffic, 06Operations, 10Phabricator, 06Release-Engineering-Team, 13Patch-For-Review: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#2245804 (10chasemp) [01:30:05] 10Traffic, 06Operations, 10Phabricator, 06Release-Engineering-Team, 13Patch-For-Review: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#1528619 (10chasemp) [02:53:51] 07HTTPS, 10Traffic, 06Operations, 13Patch-For-Review: enable https for (ubuntu|apt|mirrors).wikimedia.org - https://phabricator.wikimedia.org/T132450#2246162 (10Chmarkine) >>! In T132450#2244769, @faidon wrote: > I'm not convinced https for that is a good idea. apt doesn't support it by default — apt-trans... [03:12:43] 10Traffic, 06Operations, 10Phabricator, 06Release-Engineering-Team, 13Patch-For-Review: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#2246257 (10chasemp) [03:12:49] 10Traffic, 06Operations, 10Phabricator, 06Release-Engineering-Team, 13Patch-For-Review: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#1529683 (10chasemp) [03:12:55] 10Traffic, 06Operations, 10Phabricator, 06Release-Engineering-Team, 13Patch-For-Review: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#1529683 (10chasemp) [03:13:01] 10Traffic, 06Operations, 10Phabricator, 06Release-Engineering-Team, 13Patch-For-Review: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#1529708 (10chasemp) [03:13:26] 10Traffic, 06Operations, 10Phabricator, 06Release-Engineering-Team, 13Patch-For-Review: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#1620392 (10chasemp) [03:13:31] 10Traffic, 06Operations, 10Phabricator, 06Release-Engineering-Team, 13Patch-For-Review: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#1620392 (10chasemp) [03:13:48] 10Traffic, 06Operations, 10Phabricator, 06Release-Engineering-Team, 13Patch-For-Review: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#1645325 (10chasemp) [04:44:04] 10Traffic, 06Commons, 06Operations, 10media-storage, and 2 others: Deleted files sometimes remain visible to non-privileged users if permanently linked - https://phabricator.wikimedia.org/T109331#1546384 (10Jay8g) I too can still see the file, and I get the same IP from ping. [05:42:23] bblack: just read the messages about the error page, thanks! It makes sense now.. probably on Monday when I'll add the maintenance flag I'll get my refresh! [08:09:31] 07HTTPS, 10Traffic, 06Operations, 13Patch-For-Review: enable https for (ubuntu|apt|mirrors).wikimedia.org - https://phabricator.wikimedia.org/T132450#2247110 (10faidon) See: https://packages.debian.org/sid/apt-transport-https https://launchpad.net/ubuntu/trusty/+package/apt-transport-https Also check out... [08:49:41] 10netops, 06Operations: analytics hosts frequently tripping 'port utilization threshold' librenms alerts - https://phabricator.wikimedia.org/T133852#2247332 (10fgiunchedi) [08:51:34] 10netops, 06Operations: analytics hosts frequently tripping 'port utilization threshold' librenms alerts - https://phabricator.wikimedia.org/T133852#2247344 (10fgiunchedi) p:05Triage>03Normal [09:46:12] 10Traffic, 06Discovery, 06Operations, 10Wikidata, 10Wikidata-Query-Service: Wikidata Query Service REST endpoint returns truncated results - https://phabricator.wikimedia.org/T133490#2247526 (10fgiunchedi) p:05Triage>03Normal [09:55:59] 10Traffic, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium): Move gallium to an internal host? - https://phabricator.wikimedia.org/T133150#2247551 (10fgiunchedi) 05Open>03stalled p:05Triage>03Normal [09:56:17] 10Traffic, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium): Move gallium to an internal host? - https://phabricator.wikimedia.org/T133150#2223557 (10fgiunchedi) [10:25:32] 10Traffic, 06Operations, 13Patch-For-Review: confctl: improve/upgrade --tags/--find - https://phabricator.wikimedia.org/T128199#2247688 (10Joe) 05Open>03Resolved [10:50:10] 07Varnish, 10Wikimedia-Apache-configuration, 06Operations: Data passed to HHVM ($_SERVER variables) is a mixed bag of already-decoded and non-decoded nonsense - https://phabricator.wikimedia.org/T132629#2247749 (10fgiunchedi) p:05Triage>03Normal [11:07:50] 07Varnish, 10Wikidata: Varnish seems to sometimes mangle uncompressed API results - https://phabricator.wikimedia.org/T133866#2247787 (10hoo) [11:11:06] 10Wikimedia-Apache-configuration, 10MediaWiki-extensions-UrlShortener: Configure apache rules for UrlShortener - https://phabricator.wikimedia.org/T133485#2247809 (10Dereckson) a:03Dereckson [11:25:45] bblack: new varnish works fine on cp1008, uploaded to carbon [11:26:38] 07HTTPS, 10Traffic, 06Operations, 10Wiki-Loves-Monuments-General: configure https for www.wikilovesmonuments.org - https://phabricator.wikimedia.org/T118388#2247901 (10SindyM3) @JanZerebecki Via Let's Encrypt, it is not possible to have all the necessary certificates. I have contacted the Wikimedia DE. W... [11:56:49] 10Wikimedia-Apache-configuration: create pk.wikimedia.org and redirect to http://loveforkarachi.org/wp/ - https://phabricator.wikimedia.org/T56780#619446 (10Dereckson) I've sent Saqib a mail this 2016-04-28. [11:58:46] 10Wikimedia-Apache-configuration: create pk.wikimedia.org and redirect to http://loveforkarachi.org/wp/ - https://phabricator.wikimedia.org/T56780#2247971 (10Dereckson) [12:14:22] moritzm: thanks [12:16:14] 07HTTPS, 10Traffic, 06Operations, 13Patch-For-Review: enable https for (ubuntu|apt|mirrors).wikimedia.org - https://phabricator.wikimedia.org/T132450#2248063 (10Chmarkine) I just tried on a new install of Ubuntu 12.04.5 Desktop, and apt-transport-https is installed out of box. ``` apt-transport-https: I... [12:20:40] 10Wikimedia-Apache-configuration, 10MediaWiki-extensions-ZeroBanner, 06Zero, 07Mobile: m.wikipedia.org incorrectly redirects to en.m.wikipedia.org - https://phabricator.wikimedia.org/T69015#2248087 (10Dereckson) a:05Yurik>03Dereckson Let's keep the zero. domain like this, as it's advertised to be used... [12:43:17] 10Wikimedia-Apache-configuration, 10MediaWiki-extensions-ZeroBanner, 06Zero, 07Mobile, 13Patch-For-Review: m.wikipedia.org incorrectly redirects to en.m.wikipedia.org - https://phabricator.wikimedia.org/T69015#730151 (10BBlack) We should probably look at aligning the varnish mobile-related redirect/rewri... [12:43:40] 10Traffic, 10Wikimedia-Apache-configuration, 10MediaWiki-extensions-ZeroBanner, 06Operations, and 3 others: m.wikipedia.org incorrectly redirects to en.m.wikipedia.org - https://phabricator.wikimedia.org/T69015#2248243 (10BBlack) [12:52:58] 10Traffic, 06Operations: Varnish configuration for mobile domains should be coherent with Apache configuration - https://phabricator.wikimedia.org/T133895#2248306 (10Dereckson) [13:00:41] 10Traffic, 06Operations: Varnish configuration for mobile domains should be coherent with Apache configuration - https://phabricator.wikimedia.org/T133895#2248387 (10Dereckson) >>! In T69015#2248241, @BBlack wrote: > Specifically the 3x set req.http.MobileHost regex lines (lines 34-36 currently), they seem to,... [13:20:23] 10Traffic, 06Operations: Varnish configuration for mobile domains should be coherent with Apache configuration - https://phabricator.wikimedia.org/T133895#2248432 (10Dereckson) Actually, the text-frontend.inc.vcl.erb code is mostly correct: `regsub(req.http.MobileHost, "^(www\.)?(mediawiki|wikimediafoundation... [13:36:49] 10Traffic, 06Operations, 06Performance-Team, 13Patch-For-Review: Support HTTP/2 - https://phabricator.wikimedia.org/T96848#2248503 (10BBlack) Continuing on the saga above: building the ko for various kernels either doesn't work at all or requires different versions of the systemtap building tools than what... [13:50:53] 10Traffic, 06Operations, 06Performance-Team, 13Patch-For-Review: Support HTTP/2 - https://phabricator.wikimedia.org/T96848#2248553 (10BBlack) And... the output looks like that object was from a previous version of the source, the one in P2719 which lacks detection of an ALPN negotiation which specifies SPD... [13:59:51] 10Traffic, 06Operations, 06Performance-Team, 13Patch-For-Review: Support HTTP/2 - https://phabricator.wikimedia.org/T96848#2248582 (10BBlack) Searched all the cache nodes for other builds that might have been left behind. Found a working compile of the latest source for 3.19.0-2 in @ema's homedir on cp104... [14:07:16] 10Traffic, 06Operations: Move login.wikimedia.org to its own IP address - https://phabricator.wikimedia.org/T82877#2248598 (10fgiunchedi) [14:10:10] 10Traffic, 06Operations: Move login.wikimedia.org to its own IP address - https://phabricator.wikimedia.org/T82877#2248605 (10BBlack) 05Open>03declined Everything related to what this ticket is about has changed since it was written, and there was never any movement on it anyways. If someone still thinks... [15:06:55] 10Traffic, 06Operations, 10ops-eqiad: lvs1012 eth1 NIC Link flapping - https://phabricator.wikimedia.org/T133915#2248747 (10faidon) [15:47:35] 07HTTPS, 10Traffic, 06Operations, 10Wiki-Loves-Monuments-General: configure https for www.wikilovesmonuments.org - https://phabricator.wikimedia.org/T118388#2248879 (10JanZerebecki) A CSR will only provide one certificate. > Via Let's Encrypt, it is not possible to have all the necessary certificates. Ca... [16:05:47] 10Traffic, 06Operations, 06Performance-Team, 13Patch-For-Review: Support HTTP/2 - https://phabricator.wikimedia.org/T96848#2248930 (10BBlack) The initial 1-hour run is done, and there didn't seem to be any adverse effects. For the record, this is how the raw results format looks per-host: ``` total: 36039... [16:13:41] 10Traffic, 06Operations, 06Performance-Team, 13Patch-For-Review: Support HTTP/2 - https://phabricator.wikimedia.org/T96848#2248953 (10BBlack) In the overall 1H test data, the net result is that when we make the switch: * ~33% of our client connections will upgrade from SPDY to H/2 (which is a very minor i... [16:20:53] 10Traffic, 06Operations, 06Performance-Team, 13Patch-For-Review: Support HTTP/2 - https://phabricator.wikimedia.org/T96848#2248957 (10BBlack) (FTR: 24H sample started at 16:17 UTC, but stashbot didn't log it here) [16:35:59] 10Traffic, 06Operations, 06Performance-Team, 13Patch-For-Review: Support HTTP/2 - https://phabricator.wikimedia.org/T96848#2248985 (10faidon) >>! In T96848#2248953, @BBlack wrote: > I'd say this looks like a fine tradeoff to me. 2% net drop back to H/1, vs moving forward on standards and slightly-improvin... [17:14:46] 10Traffic, 10MediaWiki-Parser, 06Operations, 06Parsing-Team, and 6 others: Banners fail to show up occassionally on Russian Wikivoyage - https://phabricator.wikimedia.org/T121135#1870420 (10dr0ptp4kt) @wrh2, okay if we close this task now? [17:39:50] 10Traffic, 10MediaWiki-Parser, 06Operations, 06Parsing-Team, and 6 others: Banners fail to show up occassionally on Russian Wikivoyage - https://phabricator.wikimedia.org/T121135#2249139 (10Wrh2) I haven't seen the issue re-occur since the change went live, so I think it should be OK to close this issue no... [17:44:25] 10Traffic, 06Operations, 06Performance-Team, 13Patch-For-Review: Support HTTP/2 - https://phabricator.wikimedia.org/T96848#2249142 (10ori) >>! In T96848#2248985, @faidon wrote: >>>! In T96848#2248953, @BBlack wrote: >> I'd say this looks like a fine tradeoff to me. 2% net drop back to H/1, vs moving forwa... [17:53:14] 10Traffic, 10MediaWiki-Parser, 06Operations, 06Parsing-Team, and 6 others: Banners fail to show up occassionally on Russian Wikivoyage - https://phabricator.wikimedia.org/T121135#2249193 (10Jdlrobson) 05stalled>03Resolved Thanks @Wrh2 for confirming. Please do reopen this bug if it reoccurs. If you can... [17:59:46] 10Traffic, 06Operations, 10ops-eqiad: lvs1012 eth1 NIC Link flapping - https://phabricator.wikimedia.org/T133915#2249217 (10Cmjohnson) Swapped the spf+ asw=a5 xe-0/0/13....appears to have fixed the issue [18:04:06] 10Traffic, 06Operations, 10ops-eqiad: lvs1012 eth1 NIC Link flapping - https://phabricator.wikimedia.org/T133915#2249236 (10MoritzMuehlenhoff) 05Open>03Resolved This was caused by T133915, apt and puppet run working fine now. Closing. [18:04:10] 10Traffic, 10domains, 06Operations, 06WMF-Legal: wikipedia.lol - https://phabricator.wikimedia.org/T88861#2249238 (10ZhouZ) [18:06:47] 10Traffic, 06Operations, 10ops-eqiad: lvs1012 eth1 NIC Link flapping - https://phabricator.wikimedia.org/T133915#2248747 (10MoritzMuehlenhoff) Wrong tab.. I meant to close T133832 [19:21:53] 10Traffic, 10MediaWiki-Parser, 06Operations, 06Parsing-Team, and 5 others: Banners fail to show up occassionally on Russian Wikivoyage - https://phabricator.wikimedia.org/T121135#2249524 (10bmansurov) [19:25:08] 07Varnish, 10Beta-Cluster-Infrastructure: Beta cluster varnish sets overly broad domain on GeoIP cookie - https://phabricator.wikimedia.org/T133936#2249530 (10bd808) [19:29:33] 07Varnish, 10Beta-Cluster-Infrastructure: Beta cluster varnish sets overly broad domain on GeoIP cookie - https://phabricator.wikimedia.org/T133936#2249530 (10hashar) Looks like templates/varnish/geoip.inc.vcl.erb  function `geo_get_top_cookie_domain()` which does not take in account the additional level of su... [19:59:13] 10Wikimedia-Apache-configuration: svn.wikimedia.org/doc/ should redirect to doc.wikimedia.org - https://phabricator.wikimedia.org/T109950#2249658 (10Dereckson) a:03Dereckson [21:33:58] 07Varnish, 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Beta cluster varnish sets overly broad domain on GeoIP cookie - https://phabricator.wikimedia.org/T133936#2249982 (10bd808) 05Open>03Resolved a:03BBlack Verified new header: `Set-Cookie: GeoIP=COUNTRY:REGION:CITY:LAT:LON:v4; Path=/; secure;... [21:48:28] 10Traffic, 10netops, 06Operations, 13Patch-For-Review: Anycast (Auth)DNS - https://phabricator.wikimedia.org/T98006#2250022 (10BBlack) Actually, amending the network thoughts above: should use 198.35.27.0/24 + 185.15.5**8**.0/24. Using 58 instead of 56 leaves us a contiguous /23 for more future flexibilit... [23:53:28] 10Wikimedia-Apache-configuration: create pk.wikimedia.org and redirect to http://loveforkarachi.org/wp/ - https://phabricator.wikimedia.org/T56780#2250315 (10Dzahn) the suggested target of the redirect doesn't seem to exist anymore: Iceweasel can't find the server at www.loveforkarachi.org.