[02:32:45] <wikibugs>	 10Domains, 10Traffic, 06Operations, 10Wikimedia-Site-requests, 13Patch-For-Review: Private wiki for Project Grants Committee - https://phabricator.wikimedia.org/T143138#2558051 (10MZMcBride) What about <https://grants.wikimedia.org/>? There's already a private wiki here. Will that work?  If <https://iegc...
[11:09:14] <ema>	 varnishkafka.py --generate-pyconf failed on cp4018, transient failure
[11:09:55] <ema>	 running puppet again fixed it, but I think we've seen it failing multiple times in the past
[11:10:30] <ema>	 Error: /usr/bin/python /usr/lib/ganglia/python_modules/varnishkafka.py --generate-pyconf /etc/ganglia/conf.d/varnishkafka-statsv.pyconf --key-prefix=statsv --tmax=60 /var/cache/varnishkafka/statsv.stats.json returned 1 instead of one of [0]
[11:15:28] <wikibugs>	 10Traffic, 10MediaWiki-General-or-Unknown, 06Operations, 06Release-Engineering-Team, and 5 others: Make sure we're not relying on HTTP_PROXY headers - https://phabricator.wikimedia.org/T140658#2560365 (10Aklapper)
[13:05:24] <bblack>	 ema: yeah all I know on that is it's always been an off and on problem.  I've never really understood the whole flow of how the vk pyconf things works in function or puppet-dependency terms, but it always seems like there's some nasty loop built in.
[13:09:47] <ema>	 bblack: right, I've tried running it by hand a couple of times and it works well (and fast)
[13:10:34] <ema>	 bblack: still need to keep puppet disabled on cp1065 or can I re-enable it?
[13:10:50] <bblack>	 re-enable :)
[13:11:08] <bblack>	 my mind must be falling apart, I'm doing that more and more lately (forgetting to re-enable after experiments)
[13:11:42] <bblack>	 I was using cp1065 to figure out how to kill the Chrome/41/Win request-spam
[13:12:08] <bblack>	 turns out 401 is the only way I found heh
[13:21:17] <ema>	 oh interesting
[13:22:09] <ema>	 so 400 isn't enough
[13:22:36] <ema>	 I see, it avoids the redirect but they keep trying
[13:30:37] <bblack>	 yeah the root of their behavior is a constant spam of requests to '/'.  If that returns 301 they commonly follow it.
[13:30:56] <bblack>	 but returning various [45]xx on the initial query to slash, while it stops the redirect, doesn't stop the spam of requests to /
[13:31:01] <bblack>	 but 401 does :)
[13:31:20] <bblack>	 it's fascinating that Retry-After on a 503 didn't work either.  So much for standards :P
[13:31:36] <ema>	 those browsers are savages
[13:31:48] <bblack>	 yup
[13:57:06] <wikibugs>	 10Traffic, 10ArticlePlaceholder, 06Operations, 10Wikidata: Performance and caching considerations for article placeholders accesses - https://phabricator.wikimedia.org/T142944#2560681 (10hoo) >>! In T142944#2557015, @BBlack wrote: > 30 minutes isn't really reasonable, and neither is spamming more purge tra...
[14:10:49] <wikibugs>	 10Traffic, 10ArticlePlaceholder, 06Operations, 10Wikidata: Performance and caching considerations for article placeholders accesses - https://phabricator.wikimedia.org/T142944#2560699 (10BBlack) I think I'm lacking a lot of context here about these special pages and placeholders.  But my bottom line though...
[15:04:44] <wikibugs>	 10Traffic, 10ArticlePlaceholder, 06Operations, 10Wikidata: Performance and caching considerations for article placeholders accesses - https://phabricator.wikimedia.org/T142944#2560827 (10hoo) >>! In T142944#2560699, @BBlack wrote: > I think I'm lacking a lot of context here about these special pages and pl...
[15:57:00] <ema>	 bblack: so it looks like we still need to port varnishprocessor and varnishmedia for cache_upload
[15:57:30] <ema>	 I started doing that and I discovered this: https://github.com/wikimedia/operations-puppet/blob/production/modules/varnish/files/varnishmedia#L51
[15:58:18] <ema>	 the conditionals if 'RxHeader' and if 'TxHeader' return always False, I think
[15:59:04] <ema>	 we don't extract those tags when calling varnishlog.varnishlog towards the end of the file
[15:59:43] <ema>	 and indeed there are no metrics under media.thumbnail.varnish.responses
[16:00:20] <ema>	 as well as under media.thumbnail.varnish.reqs.if_none_match
[16:00:27] <wikibugs>	 10Domains, 10Traffic, 06Operations, 10Wikimedia-Site-requests, 13Patch-For-Review: Private wiki for Project Grants Committee - https://phabricator.wikimedia.org/T143138#2561115 (10Mjohnson_WMF) MZMcBride, I've asked Katy Love about https://grants.wikimedia.org/.  I'm not familiar with that wiki.  I'll le...
[16:00:54] <bblack>	 ema: so what you're saying is even under v3, varnishmedia isn't actually doing anything today?
[16:01:08] <ema>	 bblack: no, it is doing something
[16:01:17] <ema>	 other metrics are being reported correctly
[16:01:52] <ema>	 media.thumbnail.varnish.reqs.all for example
[16:02:38] <bblack>	 ok
[16:03:24] <bblack>	 I'd say if those parts are apparently-unused, start with a commit to the v3 part that removes the dead code with a note that it can be fixed or added back later if someone cares in the commitmsg
[16:03:41] <bblack>	 and then proceed with the v3/v4 split, and then hopefully we don't re-visit this until post-v4 if at all
[16:04:51] <ema>	 sounds reasonable. In particular, the if 'RxHeader' part makes little sense, it would basically increment reqs.if_none_match for each and every received header unless I'm missing something
[16:05:25] <ema>	 so yeah just "fixing" it by adding RxHeader to the list of emitted tags would not be that useful
[16:10:26] <godog>	 indeed, and also make the metrics list grow very big as it stands
[19:29:25] <wikibugs>	 10Traffic, 10Varnish, 06Operations: Convert text cluster to Varnish 4 - https://phabricator.wikimedia.org/T131503#2561905 (10BBlack)
[19:29:27] <wikibugs>	 10Traffic, 10Fundraising-Backlog, 06Operations, 13Patch-For-Review: Switch Varnish's GeoIP code to libmaxminddb/GeoIP2 - https://phabricator.wikimedia.org/T99226#2561904 (10BBlack)
[19:36:52] <wikibugs>	 10netops, 06Operations: configure port for frdb1001 - https://phabricator.wikimedia.org/T143248#2561911 (10Jgreen)
[22:06:13] <wikibugs>	 10Traffic, 10MediaWiki-extensions-UniversalLanguageSelector, 06Operations: ULS GeoIP should use the Cookie - https://phabricator.wikimedia.org/T143270#2562516 (10BBlack)
[22:06:41] <wikibugs>	 10Traffic, 10Fundraising-Backlog, 06Operations, 13Patch-For-Review: Switch Varnish's GeoIP code to libmaxminddb/GeoIP2 - https://phabricator.wikimedia.org/T99226#2562532 (10BBlack)
[22:06:44] <wikibugs>	 10Traffic, 10MediaWiki-extensions-UniversalLanguageSelector, 06Operations: ULS GeoIP should use the Cookie - https://phabricator.wikimedia.org/T143270#2562516 (10BBlack)
[22:09:27] <wikibugs>	 10Traffic, 10MediaWiki-extensions-CentralNotice, 06Operations: CN: Stop using the geoiplookup HTTPS service (always use the Cookie) - https://phabricator.wikimedia.org/T143271#2562534 (10BBlack)
[22:09:58] <wikibugs>	 10Traffic, 10Fundraising-Backlog, 06Operations, 13Patch-For-Review: Switch Varnish's GeoIP code to libmaxminddb/GeoIP2 - https://phabricator.wikimedia.org/T99226#2562548 (10BBlack)
[22:10:01] <wikibugs>	 10Traffic, 10MediaWiki-extensions-CentralNotice, 06Operations: CN: Stop using the geoiplookup HTTPS service (always use the Cookie) - https://phabricator.wikimedia.org/T143271#2562534 (10BBlack)
[22:14:33] <wikibugs>	 10Traffic, 10Fundraising-Backlog, 06Operations, 13Patch-For-Review: Switch Varnish's GeoIP code to libmaxminddb/GeoIP2 - https://phabricator.wikimedia.org/T99226#2562582 (10BBlack) I'm planning to merge up the new version of the Varnish GeoIP code from 371d7cc737d0 in the next couple of days.  If anyone ha...