[00:02:48] bblack: unrelated, but the other day I was looking at vhtcpd purge commands issued and there's some baseline plus 100+k/s every now and then, does it seem realistic? https://grafana.wikimedia.org/dashboard/db/prometheus-varnish-dc-stats?from=1481150223938&to=1481152814479 [03:22:43] godog: yes, realistic, but subject to some interpretation. Every PURGE multicasts to every cache node. So when you sum the PURGEs from all cache nodes, you're seeing PURGEs issued by MW/jobrunner/etc multiplied by cache node count [03:23:23] godog: it is valid in some sense that our aggregate caches are executing that many purge requests all total, but it's also kind of unfair to total it that way when throwing numbers back at MW :) [03:24:53] godog: if we take cache_text in isolation, it has 30 cache nodes. So if you see 100K/sec there, that's really about 3.3K/sec issued by MW towards cache_text. [11:26:44] I've upgraded cp1008 and cp3006 (maps esams) to varnish 4.1.4-1wm1, no obvious issues [11:27:18] also all the varnishtests are green, which is nice :) [11:50:29] after lunch I'll start upgrading maps/misc [17:20:01] 10Traffic, 10Mobile-Content-Service, 06Operations, 10RESTBase, and 4 others: Refreshing mobile-sections does not purge mobile-sections-lead - https://phabricator.wikimedia.org/T152690#2857373 (10mobrovac) [17:21:12] elukey: I sent out the mw-config removal of the bits docroot a bit ago, all seems quiet. [18:00:05] 10Traffic, 10Mobile-Content-Service, 06Operations, 10RESTBase, and 4 others: Refreshing mobile-sections does not purge mobile-sections-lead - https://phabricator.wikimedia.org/T152690#2857492 (10mobrovac) Ok, this should be resolved now. There was an issue with the EventBus HTTP Proxy service rejecting cer... [19:23:15] 10netops, 06Labs, 10Labs-Infrastructure, 06Operations, and 3 others: Provide read-only access to OpenStack APIs from WMF IP space - https://phabricator.wikimedia.org/T150092#2857881 (10Andrew) [19:42:38] 10Traffic, 10Mobile-Content-Service, 06Operations, 10RESTBase, and 4 others: Refreshing mobile-sections does not purge mobile-sections-lead - https://phabricator.wikimedia.org/T152690#2857914 (10Dbrant) 05Open>03Resolved Looks resolved now; thanks! [21:11:14] 10Traffic, 10Mobile-Content-Service, 06Operations, 10RESTBase, and 4 others: Refreshing mobile-sections does not purge mobile-sections-lead - https://phabricator.wikimedia.org/T152690#2858188 (10Mholloway) 05Resolved>03Open Back again: https://en.wikipedia.org/api/rest_v1/page/mobile-sections/Monogram... [21:14:22] 10Traffic, 10Mobile-Content-Service, 06Operations, 10RESTBase, and 4 others: Refreshing mobile-sections does not purge mobile-sections-lead - https://phabricator.wikimedia.org/T152690#2858199 (10Pchelolo) @Mholloway I see 'toy manufacturer' description in both links and on wikidata item [21:16:31] 10Traffic, 10Mobile-Content-Service, 06Operations, 10RESTBase, and 4 others: Refreshing mobile-sections does not purge mobile-sections-lead - https://phabricator.wikimedia.org/T152690#2858212 (10Mholloway) Must have just synced. The mobile-sections-lead version was lagging with "company" (the old descript... [21:20:39] 10Traffic, 10Mobile-Content-Service, 06Operations, 10RESTBase, and 4 others: Refreshing mobile-sections does not purge mobile-sections-lead - https://phabricator.wikimedia.org/T152690#2858235 (10Pchelolo) @Mdholloway Hm. So we have 2 queues in Change-Prop - one for main events and one for events derived fr... [21:23:30] 10Traffic, 10Mobile-Content-Service, 06Operations, 10RESTBase, and 4 others: Refreshing mobile-sections does not purge mobile-sections-lead - https://phabricator.wikimedia.org/T152690#2858257 (10Mholloway) @Pchelolo Sounds great, thank you! [21:26:21] 10Traffic, 10Mobile-Content-Service, 06Operations, 10RESTBase, and 4 others: Refreshing mobile-sections does not purge mobile-sections-lead - https://phabricator.wikimedia.org/T152690#2858261 (10Pchelolo) Created https://github.com/wikimedia/change-propagation/pull/145 [22:20:58] 10Traffic, 10Mobile-Content-Service, 06Operations, 10RESTBase, and 4 others: Refreshing mobile-sections does not purge mobile-sections-lead - https://phabricator.wikimedia.org/T152690#2858541 (10mobrovac) OK, this should be now fixed once and for all :) Please @Mholloway and @Dbrant recheck and resolve the... [23:27:20] ema, have you read my patch in gerrit?