[01:21:27] 10Traffic, 10Analytics, 06Operations, 06Performance-Team: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2328835 (10Jdlrobson) [13:12:31] 10Traffic, 06Operations, 13Patch-For-Review, 07Performance: Lots of Title::purgeExpiredRestriction from API DELETE FROM `page_restrictions` WHERE (pr_expiry < '20160517063108') without batching/throttling potentially causing lag on s5-api - https://phabricator.wikimedia.org/T135470#2329838 (10jcrespo) I wo... [13:26:41] 10netops, 10Monitoring, 06Operations, 03Scap3 (Scap3-Adoption-Phase1): Deploy libreNMS with scap3 - https://phabricator.wikimedia.org/T129136#2329902 (10faidon) a:03akosiaris [14:02:58] hiyaaaa [14:03:00] qq [14:03:03] are yall still messing with misc? [14:03:13] we still get a pretty frequent notices of loss from that cluster [14:04:44] with loss == varnishkafka loss right ottomata ? [14:06:49] yes [14:19:56] ottomata: misc has been re-upgraded to v4, but except for that we're not messing with it [14:20:05] hm [14:20:11] yall aren't restarting it now and then? over the last week? [14:20:27] let me check when exactly we finished the re-upgrade [14:20:30] or like a few hours ago? [14:20:38] no, certainly not a few hours ago [14:20:50] we've seen loss in each of the last 3 hours [14:22:32] elukey: issues with vk perhaps? [14:24:52] not sure, I still didn't fully get the loss calculation [14:25:39] IIRC should be vk stats vs webrequest logs, but still a bit fuzzy [14:26:36] so requests that vk registers but that never reach kafka/hadoop for some reason [14:26:43] again, IIRC [14:27:30] but ottomata did some investigation on them and afaik the loss was ok, not sure about now. Andrew, thoughts? [14:27:54] well, we should revisit, now that varnish isnt' being restarted [14:32:27] ema: seems to be misc-only [14:37:15] ottomata: do we have a phab task somewhere? [14:37:27] otherwise I am going to create one [14:37:52] there are a bunch of vk errors on cp2025 for example, but IIRC unrelated [14:37:54] May 24 13:09:34 cp2025 varnishkafka[111604]: KAFKAERR: Kafka error (-195): kafka1013.eqiad.wmnet:9092/13: Failed to connect to broker at [kafka1013.eqiad.wmnet]:9092: Connection refused [14:38:29] ema that sounds like when elukey recentlyr restarted that broker [14:38:31] i think unrelated [14:38:32] ja [14:38:43] elukey: i don't think we have a task [14:43:32] all right opening one [14:58:48] 10Traffic, 06Analytics-Kanban, 06Operations: Verify why varnishkafka stats and webrequest logs count differs - https://phabricator.wikimedia.org/T136314#2330347 (10elukey) p:05Triage>03High [14:59:05] ema --^ [14:59:18] will try to work on it asap [15:55:55] 07HTTPS, 10Traffic, 06Operations, 05MW-1.27-release-notes, 13Patch-For-Review: Insecure POST traffic - https://phabricator.wikimedia.org/T105794#1451756 (10Danmichaelo) >>! In T105794#2314347, @bd808 wrote: > @Steinsplitter reported to me on irc that >> for protocol relative urls in mwclient, scheme='htt... [16:07:32] 10Wikimedia-Apache-configuration, 13Patch-For-Review: create pk.wikimedia.org and redirect to wikimediapakistan.org - https://phabricator.wikimedia.org/T56780#2330697 (10Dereckson) Server works again. No HTTPS. [16:16:43] 07HTTPS, 10Traffic, 06Operations, 05MW-1.27-release-notes, 13Patch-For-Review: Insecure POST traffic - https://phabricator.wikimedia.org/T105794#2330749 (10Steinsplitter) >>! In T105794#2330598, @Danmichaelo wrote: >>>! In T105794#2314347, @bd808 wrote: >> @Steinsplitter reported to me on irc that >>> fo... [16:36:51] 10Traffic, 06Analytics-Kanban, 06Operations: Verify why varnishkafka stats and webrequest logs count differs - https://phabricator.wikimedia.org/T136314#2330849 (10ema) When are we seeing those inconsistencies? Any specific timeframes? [16:45:12] 10Traffic, 06Analytics-Kanban, 06Operations: Verify why varnishkafka stats and webrequest logs count differs - https://phabricator.wikimedia.org/T136314#2330888 (10elukey) [16:45:45] 10Traffic, 06Analytics-Kanban, 06Operations: Verify why varnishkafka stats and webrequest logs count differs - https://phabricator.wikimedia.org/T136314#2330335 (10elukey) >>! In T136314#2330849, @ema wrote: > When are we seeing those inconsistencies? Any specific timeframes? Need to query Hive and Hadoop... [17:26:08] 10Traffic, 06Analytics-Kanban, 06Operations: Verify why varnishkafka stats and webrequest logs count differs - https://phabricator.wikimedia.org/T136314#2330335 (10Nuria) 1. On labs or perhaps prod: Generate lots of request and a sighup and see if all requests ids are present, try to find repro for dropping... [17:28:32] 10Traffic, 06Analytics-Kanban, 06Operations: Verify why varnishkafka stats and webrequest logs count differs - https://phabricator.wikimedia.org/T136314#2331150 (10Ottomata) https://gist.github.com/ottomata/7048012 [18:28:16] 10Traffic, 06Operations, 13Patch-For-Review, 07Performance: Lots of Title::purgeExpiredRestriction from API DELETE FROM `page_restrictions` WHERE (pr_expiry < '20160517063108') without batching/throttling potentially causing lag on s5-api - https://phabricator.wikimedia.org/T135470#2331377 (10aaron) 05Ope... [18:35:59] 10Wikimedia-Apache-configuration, 13Patch-For-Review: create pk.wikimedia.org and redirect to wikimediapakistan.org - https://phabricator.wikimedia.org/T56780#2331451 (10Dzahn) give it about half an hour from now and this should be resolved [18:38:48] 10Wikimedia-Apache-configuration, 13Patch-For-Review: create pk.wikimedia.org and redirect to wikimediapakistan.org - https://phabricator.wikimedia.org/T56780#2331466 (10Dzahn) tested that on canary server mw1017 and with apache-fast-test before merge. https://pk.wikimedia.org * 301 Moved Permanently http://... [20:34:13] 10Traffic, 06Operations, 13Patch-For-Review: Raise cache frontend memory sizes significantly - https://phabricator.wikimedia.org/T135384#2331912 (10BBlack) So far cp3048 seems slightly better off with the new params at 156G virt + 79G resident, but it will take days to level into a new normal (not recorded a... [22:15:42] 10Traffic, 06Discovery, 10Maps, 06Operations, and 2 others: allow maps cluster Varnish cache purging - https://phabricator.wikimedia.org/T112836#2332246 (10Yurik) [22:15:46] 10Traffic, 10Varnish, 06Discovery, 10Maps, and 3 others: Tilerator should purge Varnish cache - https://phabricator.wikimedia.org/T109776#2332245 (10Yurik) [22:18:09] 10Traffic, 10Varnish, 06Discovery, 10Maps, and 3 others: Tilerator should purge Varnish cache - https://phabricator.wikimedia.org/T109776#2332266 (10Yurik) [22:18:13] 10Traffic, 06Discovery, 10Maps, 06Operations, and 2 others: allow maps cluster Varnish cache purging - https://phabricator.wikimedia.org/T112836#2332265 (10Yurik) 05Open>03Resolved [22:45:26] 10Traffic, 10Varnish, 06Discovery, 10Maps, and 2 others: Tilerator should purge Varnish cache - https://phabricator.wikimedia.org/T109776#2332371 (10Yurik)