[00:01:10] 10Traffic, 10Varnish, 10Cloud-Services, 10Operations, and 3 others: Fix RESTBase support for wikitech.wikimedia.org - https://phabricator.wikimedia.org/T102178#1358396 (10Krinkle) @GWicke At which point was wikimedia.org (or www.wikimedia.org?) a wiki? Assuming this would've been the foundation wiki (curre... [01:28:12] 10Traffic, 10Varnish, 10Cloud-Services, 10Operations, and 3 others: Fix RESTBase support for wikitech.wikimedia.org - https://phabricator.wikimedia.org/T102178#3403100 (10GWicke) @krinkle, your comments sounds like it might have been intended for {T133178}. [01:30:03] 10Traffic, 10Operations, 10RESTBase, 10RESTBase-API, 10Services (next): RESTBase support for www.wikimedia.org missing - https://phabricator.wikimedia.org/T133178#2224310 (10GWicke) @krinkle, I don't know the exact time period www.wikimedia.org was active as a wiki, and yes it was the foundation wiki. II... [02:04:20] 10Traffic, 10Operations, 10Community-Liaisons (Jul-Sep 2017): Communicate dropping IE8-on-XP support (a security change) to affected editors and other community members - https://phabricator.wikimedia.org/T163251#3403138 (10Aklapper) [04:42:37] 10Traffic, 10Operations, 10Pybal, 10User-Joe: Pybal not happy with DNS delays - https://phabricator.wikimedia.org/T154759#3403250 (10ema) Twisted uses `ThreadedResolver` (internet/base.py) as the default resolver (I got sidetracked for a while and thought that ResolverBase in names/common.py was the one, b... [06:11:22] 10Traffic, 10DNS, 10Operations, 10Pybal, 10Patch-For-Review: pybal DNS lookup issues causing outage risks - https://phabricator.wikimedia.org/T103921#1402514 (10ema) >>! In T103921#1417034, @mark wrote: > PyBal resolves IPs once at startup for managing the IPVS state. Did we neglect to extend that to Idl... [06:54:26] 10Traffic, 10Operations, 10Prometheus-metrics-monitoring: Enable diamond PowerDNSRecursor collector on dnsrecursors - https://phabricator.wikimedia.org/T169600#3403331 (10ema) [07:18:56] 10Traffic, 10Operations, 10Pybal, 10User-Joe: Pybal not happy with DNS delays - https://phabricator.wikimedia.org/T154759#3403354 (10MoritzMuehlenhoff) If we use rotate in resolv.conf, I guess we'd need to couple it with "options timeout:1", otherwise it wouldn't be very effeective with the default timeout... [07:34:02] random thought: it is really great now to have webrequests in logstash, but at the same time the format of the events is really frustrating because it doesn't carry good info from Varnish, like what happened to the backend and why the 50X occurred [07:35:04] we could do something different, namely add another varnishkafka instance only for ReqStatus ~ 50* that pushes to a special kafka topic events carrying all the VSL tags that we'd like to see [07:35:21] so we'll have a more complete stream of data from Varnish [07:36:14] it shouldn't require a lot of work, just a new vk instance and a new kafkatee config on oxygen (or logstash pulling directly from kafka) [07:36:19] Cc: godog --^ [07:53:45] elukey: makes a lot of sense! also for 4xx would be equally simple I think? [07:54:17] oh yes, just a matter of telling to vk what to push [08:24:06] 10Traffic, 10MobileFrontend, 10Operations, 10Patch-For-Review, 10Reading-Web-Backlog (Tracking): Remove disableImages handling from VCL - https://phabricator.wikimedia.org/T168013#3403448 (10phuedx) 05Open>03Resolved a:03phuedx This looks like it's done. Feel free to reopen if it ain't. Obvs. Than... [08:37:23] paravoid: https://vincent.bernat.im/en/blog/2017-performance-progression-ipv4-route-lookup-linux [09:27:31] 10Traffic, 10Operations, 10Pybal, 10User-Joe: Pybal not happy with DNS delays - https://phabricator.wikimedia.org/T154759#3403680 (10ema) >>! In T154759#3403354, @MoritzMuehlenhoff wrote: > If we use rotate in resolv.conf, I guess we'd need to couple it with "options timeout:1", otherwise it wouldn't be ve... [11:09:20] XioNoX: yeah I've seen it :) [11:49:34] 10Traffic, 10Operations, 10Pybal, 10User-Joe: Pybal not happy with DNS delays - https://phabricator.wikimedia.org/T154759#3404022 (10ema) @mark's comment on a related issue (T103921#1417034) made me go dig a bit deeper into our IdleConnection monitor implementation. The version currently in prod [[https://... [12:47:48] 10netops, 10Operations: deploy diffscan2 - https://phabricator.wikimedia.org/T169624#3404104 (10ayounsi) [12:55:44] 10netops, 10Operations: deploy diffscan2 - https://phabricator.wikimedia.org/T169624#3404140 (10MoritzMuehlenhoff) Since the tool (after an initially vetted run) only shows difference to the previous run I think it should simply be sent to the standard root mails. (We probably need a little tweak so that it do... [13:00:51] 10HTTPS, 10Traffic, 10MW-1.30-release-notes, 10Operations, and 2 others: Enable HTTPS for swift clients - https://phabricator.wikimedia.org/T160616#3404157 (10fgiunchedi) @aaron yeah it will need some love, in the meantime I've patched in https support so swiftrepl will DTRT. Looks like this completes the... [13:57:41] paravoid, ema, what IP ranges are zero-rated? [14:03:32] 10netops, 10Operations: zombie rstp configuration - https://phabricator.wikimedia.org/T169637#3404423 (10ayounsi) [14:34:00] 10netops, 10Operations, 10fundraising-tech-ops, 10ops-codfw: codfw: rack frack refresh equipment - https://phabricator.wikimedia.org/T169643#3404554 (10ayounsi) [14:37:55] 10netops, 10Operations, 10ops-eqiad: eqiad: rack frack refresh equipment - https://phabricator.wikimedia.org/T169644#3404579 (10ayounsi) [15:00:09] 10netops, 10Operations: zombie rstp configuration - https://phabricator.wikimedia.org/T169637#3404706 (10faidon) I know nothing about that, likely a stock config that wasn't cleaned up and left cruft behind. [15:47:56] 10Traffic, 10ArchCom-RfC, 10Operations, 10Services (designing): Make API usage limits easier to understand, implement, and more adaptive to varying request costs / concurrency limiting - https://phabricator.wikimedia.org/T167906#3349120 (10Tgr) ORES is considering useragent-based connection limiting (we ar... [16:51:07] XioNoX: is there a list? would be glad to get it. btw https://phabricator.wikimedia.org/T131211#2160420 [21:17:25] 10Traffic, 10Commons, 10Operations, 10Thumbor: PNG thumbnail gives a 429 error - https://phabricator.wikimedia.org/T169678#3405600 (10Bawolff) https://upload.wikimedia.org/wikipedia/commons/thumb/c/c7/444_album_cover.png/600px-444_album_cover.png gives: ``` HTTP/2.0 429 No Reason Phrase Date: Tue, 04 Jul... [21:20:37] 10Traffic, 10Commons, 10Operations, 10Thumbor: PNG thumbnail gives a 429 error - https://phabricator.wikimedia.org/T169678#3405604 (10Bawolff) [Even if this particular image is subsequently fixed, it should be investigated why there is no friendly html error message returned]