[06:24:01] <elukey>	 morning people, as fyi I just restarted the varnish backend on cp3040/3033 due to 503s 
[07:52:34] <ema>	 elukey: thank you!
[13:05:09] <paravoid>	 did you folks see the cp* cronspam today?
[13:05:15] <paravoid>	 /etc/cron.daily/logrotate:
[13:05:15] <paravoid>	 Job for varnishkafka-webrequest.service failed. See 'systemctl status varnishkafka-webrequest.service' and 'journalctl -xn' for details.
[13:05:18] <paravoid>	 error: error running non-shared postrotate script for /var/cache/varnishkafka/webrequest.stats.json of '/var/cache/varnishkafka/webrequest.stats.json '
[13:05:21] <paravoid>	 run-parts: /etc/cron.daily/logrotate exited with return code 1
[13:05:23] <paravoid>	 for a bunch of hosts
[13:08:59] <wikibugs_>	 10netops, 10Cloud-Services, 10Operations, 10ops-eqiad: labsdb1001's switch port negociating at 100M - https://phabricator.wikimedia.org/T177130#3650565 (10faidon)
[13:52:39] <elukey>	 sigh
[13:52:54] <elukey>	 I didn't get those emails
[13:52:56] <elukey>	 spam?
[13:53:12] <elukey>	 of course
[13:55:52] <elukey>	 ahhh these are spare::systems
[13:56:01] <elukey>	 but the cronjob is still there
[13:56:15] <elukey>	 I was really scared for a brief moment :D
[13:59:15] <elukey>	 cumin 'cp4* and R:class = role::spare::system' 'rm -f /etc/logrotate.d/varnishkafka*'
[13:59:34] <elukey>	 paravoid: --^
[14:00:01] <paravoid>	 ok
[14:25:23] <bblack>	 sorry, I caused that with the decom, I didn't re-install after re-roling them
[14:25:35] <bblack>	 tried to get away with the simpler "stop all the cp*-specific daemons" approach
[14:50:02] <wikibugs_>	 10netops, 10Operations, 10fundraising-tech-ops: remove fundraising firewall rules related to ganglia - https://phabricator.wikimedia.org/T176319#3650965 (10Jgreen) 05Open>03Resolved this is done
[15:48:23] <paravoid>	 bblack: is T175636 waiting on you or godog?
[15:48:23] <stashbot>	 T175636: prometheus -> grafana stats for per-numa-node meminfo - https://phabricator.wikimedia.org/T175636
[15:51:53] <bblack>	 paravoid: whoever first has time? we can probably downgrade priority to Low too, since "numa_networking: isolate" apparently doesn't work out so hot even on our new assymetric cache nodes :)
[15:52:15] <wikibugs_>	 10Traffic, 10Operations, 10monitoring, 10Patch-For-Review: prometheus -> grafana stats for per-numa-node meminfo - https://phabricator.wikimedia.org/T175636#3651253 (10BBlack) p:05Normal>03Low
[15:52:42] <bblack>	 it'd still be a nice-to-have in the non-isolated case to stare at, and we may yet end up using isolate on the LVSes if it tests well there
[15:53:50] <godog>	 from the sort-of-related-but-not-really department https://medium.com/netflix-techblog/serving-100-gbps-from-an-open-connect-appliance-cdb51dda3b99
[15:56:45] <bblack>	 yeah tons of useful details in there :)
[15:59:21] <godog>	 indeed, I wish we had a nicer way to share interesting links/reads we find over the internet
[16:23:22] <wikibugs_>	 10Traffic, 10Operations, 10Performance-Team (Radar): Upgrade to Varnish 5 - https://phabricator.wikimedia.org/T168529#3651406 (10BBlack) This task hasn't been updated for various IRC/Hangouts discussions since.  We did decide to move forward with V5 upgrades.  Arzhel has built preliminary packages, and we ha...
[16:23:52] <wikibugs_>	 10Traffic, 10Operations, 10Performance-Team (Radar): Upgrade cache_misc to Varnish 5 - https://phabricator.wikimedia.org/T177233#3651409 (10BBlack)
[16:59:42] <XioNoX>	 godog: maybe add a section at the end of the weekly meeting notes?
[17:00:19] <XioNoX>	 Or a wikitech page that people can subscribe to, so it's public
[17:01:50] <godog>	 XioNoX: good ideas! yeah wikitech might work better as it is public
[17:02:44] <godog>	 I'll get sth started and mention it at the next h/o meeting
[17:47:00] <wikibugs_>	 10Traffic, 10Operations, 10Reading-Admin: TEST: redirect small portion of unauthenticated desktop users to mobile web - https://phabricator.wikimedia.org/T117826#3651798 (10CKoerner_WMF)
[17:59:56] <wikibugs_>	 10Traffic, 10Operations, 10Performance-Team (Radar): Upgrade to Varnish 5 - https://phabricator.wikimedia.org/T168529#3651828 (10Gilles) Is the plan to use it with hitch in front of it rather than nginx? Or just an upgrade for now and we'll see about that part later?
[18:56:10] <wikibugs_>	 10Traffic, 10Operations, 10Performance-Team (Radar): Upgrade to Varnish 5 - https://phabricator.wikimedia.org/T168529#3651975 (10BBlack) Definitely not looking at Hitch presently.  Just swapping out Varnish4 for Varnish5 in the existing software stack for both the frontend and backend cache processes.  Forwa...