[00:02:51] 10Traffic, 06Operations, 05WMF-deploy-2016-08-09_(1.28.0-wmf.14): Decom bits.wikimedia.org hostname - https://phabricator.wikimedia.org/T107430#2525342 (10Krinkle) [01:04:28] 07HTTPS, 10Traffic, 06Operations, 10Wikimedia-Blog: Switch blog to HTTPS-only - https://phabricator.wikimedia.org/T105905#1454386 (10EdErhart-WMF) @BBlack can you recheck this? Automattic just shipped a few changes for us today. [02:11:18] 07HTTPS, 10Traffic, 06Operations, 10Wikimedia-Blog: Switch blog to HTTPS-only - https://phabricator.wikimedia.org/T105905#2525620 (10BBlack) >>! In T105905#2525418, @EdErhart-WMF wrote: > @BBlack can you recheck this? Automattic shipped a few changes for us today. The blog is still sending the response h... [11:44:16] 07HTTPS, 10Traffic, 06Operations, 07Tracking: [DO NOT USE] SSL related (tracking) [superseded by #HTTPS] - https://phabricator.wikimedia.org/T29946#2526687 (10Danny_B) 05Resolved>03Invalid p:05Normal>03Lowest a:05chasemp>03None [11:44:18] 07HTTPS, 10Traffic, 06Operations, 07Tracking: [DO NOT USE] SSL related (tracking) [superseded by #HTTPS] - https://phabricator.wikimedia.org/T29946#2526694 (10Danny_B) [17:31:58] 10Traffic, 10Varnish, 06Operations: Varnish 4 stalls with two consecutive Range requests using HTTP persistent connections - https://phabricator.wikimedia.org/T142233#2528064 (10ema) [17:45:48] ema: eep, that sounds like a real v4 bug [17:56:08] bblack: it does [17:57:16] while trying to find out what was going on I found this funny thing: [17:57:22] sudo tcpdump -n -v -i lo 'tcp[tcpflags] == tcp-syn' [17:57:31] curl http://localhost [17:57:34] no SYN [17:57:42] curl http://127.0.0.1 [17:57:43] SYN [17:58:33] and that went straight into the pile of mysteries of the week [17:59:32] also note that T142233 does not only happen right after a cache miss [17:59:32] T142233: Varnish 4 stalls with two consecutive Range requests using HTTP persistent connections - https://phabricator.wikimedia.org/T142233 [17:59:56] ema: because localhost is resolving as v6, and youre tcpdump's bpf filter isn't matching tcp6 :) [17:59:57] it's reproducible by first running the script with REPRODUCE = False and shortly afterwards with REPRODUCE = True [18:00:13] bblack: oooooohhhh [18:02:39] dinner time, bbl [19:25:43] 10Traffic, 10Varnish, 06Operations: Analyze Range requests on cache_upload frontend - https://phabricator.wikimedia.org/T142076#2528601 (10ema) This is a 24 hours sample collected on hosts in ulsfo, eqiad and esams. **cp4005 - Total 123158948** |norange |122827836|99.73% of total |range | 331004|0.2... [19:45:32] bblack: so the norange percentage in the 30 min sample was in line with the 24h sample [19:46:01] quite some more highrange in esams [19:46:46] and also the 31% sillyrange in eqiad is interesting [20:42:01] 10Traffic, 10Fundraising-Backlog, 06Operations, 13Patch-For-Review: Switch Varnish's GeoIP code to libmaxminddb/GeoIP2 - https://phabricator.wikimedia.org/T99226#2528897 (10DStrine) [23:40:03] 07HTTPS, 10Traffic: Make wikimedia italia https - https://phabricator.wikimedia.org/T142268#2529622 (10Paladox) [23:40:14] 07HTTPS, 10Traffic, 06Operations: Make wikimedia italia https - https://phabricator.wikimedia.org/T142268#2529635 (10Paladox) [23:44:19] 07HTTPS, 10Traffic, 06Operations: Make wikimedia italia https - https://phabricator.wikimedia.org/T142268#2529622 (10Peachey88) It's not hosted on the cluster [23:44:57] 07HTTPS, 10Traffic, 06Operations: Make wikimedia italia https - https://phabricator.wikimedia.org/T142268#2529641 (10Paladox) Hi nope, but Wikimedia lanaguges do not make it easy to report this stuff. [23:46:33] 07HTTPS, 10Traffic, 06Operations: Make wikimedia italia https - https://phabricator.wikimedia.org/T142268#2529622 (10Dzahn) I think maybe we could have one single ticket like "wikimedia chapters that dont have working https". (and then send some mails about it)