[01:07:27] 10Traffic, 10Operations, 10Accessibility, 10Browser-Support-Internet-Explorer: Wikipedia no longer accessible to those using some braille devices - https://phabricator.wikimedia.org/T185582#3922665 (10matmarex) MediaWiki still has basic support for IE 6, but I think Wikipedia no longer allows IE 6 to conne... [02:32:01] 10Traffic, 10Mail, 10Operations: convert mail servers from GS to LE certificates - https://phabricator.wikimedia.org/T159346#3922809 (10herron) [02:36:10] 10Traffic, 10Operations: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#3922815 (10herron) [02:36:13] 10Traffic, 10Mail, 10Operations: convert mail servers from GS to LE certificates - https://phabricator.wikimedia.org/T159346#3922813 (10herron) 05Open>03Resolved Resolving as MX were migrated to LE certificates during renewal task T159346 [02:50:49] 10Traffic, 10Gerrit, 10Operations, 10Patch-For-Review: Switch on http/2 in apache for gerrit - https://phabricator.wikimedia.org/T180978#3922822 (10demon) 05Open>03declined Declining outright per what I said in T185645#3922220: > Also, I'd rather move it behind LVS: T165631 [12:19:12] 10Traffic, 10netops, 10Operations: Wikipedia access in the United Kingdom is experiencing connection issues. - https://phabricator.wikimedia.org/T185687#3923101 (10TheDragonFire) [12:23:41] 10Traffic, 10netops, 10Operations: Wikipedia access in the United Kingdom is experiencing connection issues. - https://phabricator.wikimedia.org/T185687#3923102 (10TheDragonFire) benoliver999 reports that `text-lb.esams.wikimedia.org` times out, but `ulsfo`, `eqiad` and `codfw` are good. [13:19:06] 10Traffic, 10netops, 10Operations: Wikipedia access in the United Kingdom is experiencing connection issues. - https://phabricator.wikimedia.org/T185687#3923212 (10TheDragonFire) 05Open>03Resolved a:03TheDragonFire Cautiously closing as connectivity is back up. Somewhat speculative, but circumstantial... [15:31:00] 10Traffic, 10netops, 10Operations: Wikipedia access in the United Kingdom is experiencing connection issues. - https://phabricator.wikimedia.org/T185687#3923353 (10Aklapper) 05Resolved>03declined a:05TheDragonFire>03None [18:23:19] 10Traffic, 10Operations, 10Accessibility, 10Browser-Support-Internet-Explorer: Wikipedia no longer accessible to those using some braille devices - https://phabricator.wikimedia.org/T185582#3923484 (10Cameron11598) They actually were receiving a different error and were just able to locate it. > "Browse... [18:31:01] 10Traffic, 10Operations, 10Accessibility, 10Browser-Support-Internet-Explorer: Wikipedia no longer accessible to those using some braille devices - https://phabricator.wikimedia.org/T185582#3923504 (10Reedy) They might be able to improve the situation by changing the SSL certificates. Beyond that, there's... [18:31:48] 10Traffic, 10Operations, 10Accessibility, 10Browser-Support-Internet-Explorer: Wikipedia no longer accessible to those using some braille devices - https://phabricator.wikimedia.org/T185582#3920199 (10Joe) I can confirm that our current SSL settings do not work with IE8 on windows XP. This is due to inhere... [18:40:08] 10Traffic, 10Operations, 10Accessibility, 10Browser-Support-Internet-Explorer: Wikipedia no longer accessible to those using some braille devices - https://phabricator.wikimedia.org/T185582#3923523 (10Joe) @Cameron11598 I couldn't find on the product page for that device any information on the OS/browser i... [18:43:13] 10Traffic, 10Operations, 10Accessibility, 10Browser-Support-Internet-Explorer: Wikipedia no longer accessible to those using some braille devices - https://phabricator.wikimedia.org/T185582#3923538 (10Joe) >>! In T185582#3922270, @Reedy wrote: > Error seems to be related to SSL settings. > > https://suppo... [18:45:05] 10Traffic, 10Operations, 10Accessibility, 10Browser-Support-Internet-Explorer: Wikipedia no longer accessible to those using some braille devices - https://phabricator.wikimedia.org/T185582#3923539 (10Reedy) Right. But there should be TLS settings to enable (if they're not enabled already), ala https://hel... [18:56:44] 10Traffic, 10Operations, 10Accessibility, 10Browser-Support-Internet-Explorer: Wikipedia no longer accessible to those using some braille devices - https://phabricator.wikimedia.org/T185582#3923572 (10Volker_E) @Joe There's a product specification page (I linked another article I've found but wanted to lin... [20:32:55] 10Traffic, 10Operations, 10Accessibility, 10Browser-Support-Internet-Explorer: Wikipedia no longer accessible to those using some braille devices - https://phabricator.wikimedia.org/T185582#3920199 (10TheDJ) I've thrown a tweet at HIMS Inc. Maybe they can help. https://twitter.com/dj_hartman/status/9566256... [20:38:25] 10Traffic, 10Operations, 10Phabricator: Phabricator is slow on uk ISP's - https://phabricator.wikimedia.org/T185718#3923666 (10Zoranzoki21) Vip mobile Serbia too Telenor Serbia work ok P. S. 😄 Deploy this https://gerrit.wikimedia.org/r/#/c/404095/ [20:38:39] 10Traffic, 10Operations, 10Phabricator: Phabricator is slow on uk ISP's - https://phabricator.wikimedia.org/T185718#3923685 (10Paladox) [20:39:14] 10Traffic, 10Operations, 10Phabricator: Phabricator is slow on mobile ISP's - https://phabricator.wikimedia.org/T185718#3923686 (10Zoranzoki21) [20:39:36] 10Traffic, 10Operations, 10Phabricator: Phabricator is slow on mobile ISP's and home ISP's - https://phabricator.wikimedia.org/T185718#3923687 (10Paladox) [20:41:02] 10Traffic, 10Operations, 10Phabricator: Phabricator is slow on mobile ISP's and home ISP's - https://phabricator.wikimedia.org/T185718#3923666 (10Paladox) Maybe this should be UBN status. The slowness is really hard to do anything on here. [20:44:16] 10Traffic, 10Operations, 10Phabricator: Phabricator is slow on mobile ISP's and home ISP's - https://phabricator.wikimedia.org/T185718#3923689 (10Zoranzoki21) I think to we first should contact these providers. I will contact vip mobile to ask. [20:45:32] 10Traffic, 10Operations, 10Phabricator: Phabricator is slow on mobile ISP's and home ISP's - https://phabricator.wikimedia.org/T185718#3923690 (10Paladox) As it happends to bt and three in the uk. I doin't think there's a point unless they managed to have the same problem at the same time which is unlikely. [20:47:23] 10Traffic, 10Operations, 10Phabricator: Phabricator is slow on mobile ISP's and home ISP's - https://phabricator.wikimedia.org/T185718#3923691 (10Zoranzoki21) >>! In T185718#3923690, @Paladox wrote: > As it happends to bt and three in the uk. I doin't think there's a point unless they managed to have the sam... [20:50:26] 10Traffic, 10Operations, 10Phabricator: Phabricator is slow on mobile ISP's and home ISP's - https://phabricator.wikimedia.org/T185718#3923693 (10Zoranzoki21) Tech support from vip mobile told me to you check logs. [20:55:34] 10Traffic, 10Operations, 10Phabricator: Phabricator is slow on mobile ISP's and home ISP's - https://phabricator.wikimedia.org/T185718#3923697 (10Paladox) [20:57:19] 10Traffic, 10Operations, 10Phabricator: Phabricator is slow on mobile ISP's and home ISP's - https://phabricator.wikimedia.org/T185718#3923698 (10Zoranzoki21) >>! In T185718#3923693, @Zoranzoki21 wrote: > Tech support from vip mobile told me to you check logs. I asked my friend which work in telenor to chec... [20:57:55] 10Traffic, 10Operations, 10Phabricator: Phabricator is slow on mobile ISP's and home ISP's - https://phabricator.wikimedia.org/T185718#3923699 (10Zoranzoki21) [20:58:46] 10Traffic, 10Operations, 10Phabricator: Phabricator is slow on mobile ISP's and home ISP's - https://phabricator.wikimedia.org/T185718#3923666 (10Legoktm) Please follow the instructions at and provide the requested information so ops can di... [20:59:00] 10Traffic, 10Operations, 10Phabricator: Phabricator is slow on mobile ISP's and home ISP's - https://phabricator.wikimedia.org/T185718#3923704 (10Zoranzoki21) I tested too on mts network. I can not open phabricator how much is slow. I updated task description to problems too happening in Serbia. [21:00:36] 10Traffic, 10Operations, 10Phabricator: Phabricator is slow on mobile ISP's and home ISP's - https://phabricator.wikimedia.org/T185718#3923706 (10Zoranzoki21) >>! In T185718#3923702, @Legoktm wrote: > Please follow the instructions at and p... [21:04:31] 10Traffic, 10Operations, 10Phabricator: Phabricator is slow on mobile ISP's and home ISP's - https://phabricator.wikimedia.org/T185718#3923707 (10Zoranzoki21) Curl for phabricator tested from my test server: ``` mediawiki@server:~$ curl -v https://phabricator.wikimedia.org... [21:04:44] 10Traffic, 10Operations, 10Phabricator: Phabricator is slow on mobile ISP's and home ISP's - https://phabricator.wikimedia.org/T185718#3923708 (10Paladox) Pinging ae2.cr2-esams.wikimedia.org returns timeout errors (including it's ip). ``` curl -v phabricator.wikimedia.org * Rebuilt URL to: phabricator.wikim... [21:11:23] 10Traffic, 10Operations, 10Phabricator: Phabricator is slow on mobile ISP's and home ISP's - https://phabricator.wikimedia.org/T185718#3923709 (10Zoranzoki21) >>! In T185718#3923708, @Paladox wrote: > Pinging ae2.cr2-esams.wikimedia.org returns timeout errors (including it's ip). > > ``` > curl -v phabricat... [21:15:38] 10Traffic, 10Analytics-Data-Quality, 10Operations: Vet reliability of the response_size field for data analysis purposes - https://phabricator.wikimedia.org/T185350#3923717 (10Tbayer) >>! In T185350#3919078, @faidon wrote: > Interesting! So with a ratio in:out of approximately 25:1 (based on January's figure... [21:24:08] 10Traffic, 10Operations, 10Phabricator: Phabricator is slow on mobile ISP's and home ISP's - https://phabricator.wikimedia.org/T185718#3923730 (10Paladox) This is ping data ``` ping -c 4 phabricator.wikimedia.org PING phabricator.wikimedia.org (91.198.174.217): 56 data bytes 64 bytes from 91.198.174.217: ic... [21:35:40] 10Traffic, 10Operations, 10Phabricator: Phabricator is slow on mobile ISP's and home ISP's - https://phabricator.wikimedia.org/T185718#3923740 (10Paladox) Here's my recording of it http://recordit.co/SCP6EtgIKK [21:38:31] 10Traffic, 10Operations, 10Accessibility, 10Browser-Support-Internet-Explorer: Wikipedia no longer accessible to those using some braille devices - https://phabricator.wikimedia.org/T185582#3923741 (10Volker_E) Thanks @TheDJ – I've also sent them an email about the cause with request for participation on t... [21:39:39] 10Traffic, 10Operations, 10Accessibility, 10Browser-Support-Internet-Explorer: Wikipedia no longer accessible to those using some braille devices - https://phabricator.wikimedia.org/T185582#3923751 (10Reedy) If it's running CE 6.0... I bet it's basically going to be a case of "the OS doesn't support it, bu... [21:42:23] 10Traffic, 10Operations, 10Phabricator: Phabricator is slow on mobile ISP's and home ISP's - https://phabricator.wikimedia.org/T185718#3923757 (10Paladox) Here's how long it takes for it to load phab https://phabricator.wikimedia.org/F12791592 [21:45:15] 10Traffic, 10Operations, 10Phabricator: Phabricator is slow on mobile ISP's and home ISP's - https://phabricator.wikimedia.org/T185718#3923759 (10Zoranzoki21) >>! In T185718#3923757, @Paladox wrote: > Here's how long it takes for it to load phab https://phabricator.wikimedia.org/F12791592 Heh, I can not see... [22:07:12] 10Traffic, 10Operations, 10Phabricator: Phabricator is slow on mobile ISP's and home ISP's - https://phabricator.wikimedia.org/T185718#3923781 (10Paladox) tcptraceroute shows: ``` sudo tcptraceroute phabricator.wikimedia.org Password: Selected device en0, address 192.168.1.231, port 58435 for outgoing pack... [22:09:21] 10Traffic, 10Operations, 10Phabricator: Phabricator is slow on mobile ISP's and home ISP's - https://phabricator.wikimedia.org/T185718#3923666 (10Dzahn) I restarted Apache on the Phabricator server and apparently that fixed it. [22:10:11] 10Traffic, 10Operations, 10Phabricator: Phabricator is slow on mobile ISP's and home ISP's - https://phabricator.wikimedia.org/T185718#3923786 (10Zoranzoki21) >>! In T185718#3923784, @Dzahn wrote: > I restarted Apache on the Phabricator server and apparently that fixed it. Now is faster loading phabricator... [22:10:28] 10Traffic, 10Operations, 10Phabricator: Phabricator is slow on mobile ISP's and home ISP's - https://phabricator.wikimedia.org/T185718#3923787 (10Dzahn) 05Open>03Resolved a:03Dzahn 17:07 < mutante> !log restarting apache on phabricator server 17:07 < paladox> yay it fast again [22:24:42] 10Traffic, 10Operations, 10Accessibility, 10Browser-Support-Internet-Explorer: Wikipedia no longer accessible to those using some braille devices - https://phabricator.wikimedia.org/T185582#3923793 (10TheDJ) Mainstream support for it ended in 2013.. https://support.microsoft.com/en-us/lifecycle/search?alph... [23:03:50] 10Traffic, 10Operations, 10Performance-Team: load.php response taking 160s (of which only 0.031s in Apache) - https://phabricator.wikimedia.org/T181315#3923835 (10Imarlier) Couple of things that I noticed about this, which seem to generally support Gilles' conclusion: //[[ https://logstash.wikimedia.org/app...