[00:42:11] 10Traffic, 10Operations, 10Wikimedia-Stream: stream.wikimedia.org - redirect http(s) to docs - https://phabricator.wikimedia.org/T70528#3450929 (10Krinkle) 05Open>03Resolved and all redirect to now. [08:05:27] 10Traffic, 10Operations, 10ops-eqiad: Degraded RAID on cp1008 - https://phabricator.wikimedia.org/T171028#3451439 (10ema) [09:10:14] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: codfw row C switch upgrade - https://phabricator.wikimedia.org/T170380#3451571 (10ayounsi) 05Open>03Resolved Switch upgrade took ~9min and went as expected. Icinga paged about some svc.codfw services unreachable, followup task to be opened. [09:18:47] 10Traffic, 10netops, 10Operations: Investigate lvs IP pages during codfw row C switch upgrade - https://phabricator.wikimedia.org/T171032#3451590 (10fgiunchedi) [10:40:04] 10netops, 10Operations, 10Patch-For-Review: Enabling IGMP snooping on QFX switches breaks IPv6 (HTCP purges flood across codfw) - https://phabricator.wikimedia.org/T133387#3451811 (10fgiunchedi) a:03ayounsi [10:47:00] 10Traffic, 10Operations, 10Patch-For-Review: python-varnishapi daemons seeing "Log overrun" constantly - https://phabricator.wikimedia.org/T151643#2823504 (10fgiunchedi) looks like this was fixed, @ema @elukey ? [10:59:10] 10Traffic, 10Operations: Fix lvs1001-6 storage - https://phabricator.wikimedia.org/T136737#3451852 (10fgiunchedi) [12:08:36] 10Traffic, 10Operations, 10Patch-For-Review: python-varnishapi daemons seeing "Log overrun" constantly - https://phabricator.wikimedia.org/T151643#3452088 (10elukey) Logged on cp4006 and I can see `Jul 19 12:03:09 cp4006 varnishstatsd[2413]: Log overrun` in the syslog, so nope :( [12:21:24] 10Traffic, 10Operations, 10Patch-For-Review: python-varnishapi daemons seeing "Log overrun" constantly - https://phabricator.wikimedia.org/T151643#3452120 (10ema) Yeah, all other daemons have been fixed but `varnishstatsd` seems to still be affected by this issue. [12:38:04] 10Traffic, 10Operations, 10Wikimedia-Planet, 10Patch-For-Review: mixed-content issues on planet.wikimedia.org - https://phabricator.wikimedia.org/T141480#2500163 (10fgiunchedi) I tried loading en.planet.wikimedia.org and got no mixed content alerts today, there were some osm linked that failed to load due... [12:55:18] 10Traffic, 10MediaWiki-API, 10Operations, 10Patch-For-Review: Varnish does not cache Action API responses when logged in - https://phabricator.wikimedia.org/T155314#2940086 (10fgiunchedi) This seems to WAI, only responses for anons are cached in varnish, anything left to do? [12:55:21] 10Traffic, 10Operations, 10Patch-For-Review: python-varnishapi daemons seeing "Log overrun" constantly - https://phabricator.wikimedia.org/T151643#3452265 (10ema) Incidentally, while looking at entirely different stuff on esams recdns hosts, I've noticed that the vast majority of our DNS traffic there is due... [13:22:28] 10Traffic, 10Operations: Strongswan Icinga check: do not report issues about depooled hosts - https://phabricator.wikimedia.org/T148976#3452356 (10ema) Icinga [[ https://www.icinga.com/docs/icinga1/latest/en/extcommands2.html|external commands]] include `SCHEDULE_SVC_DOWNTIME`, which seems handy. We could perh... [13:34:54] 10Traffic, 10Operations, 10Community-Liaisons (Jul-Sep 2017): Communicate dropping IE8-on-XP support (a security change) to affected editors and other community members - https://phabricator.wikimedia.org/T163251#3452402 (10Johan) @BBlack Sorry for late reply, been OoO. Difficult to say, given that this isn... [13:35:12] 10Traffic, 10Operations, 10Community-Liaisons (Jul-Sep 2017), 10User-Johan: Communicate dropping IE8-on-XP support (a security change) to affected editors and other community members - https://phabricator.wikimedia.org/T163251#3452403 (10Johan) a:03Johan [14:05:40] 10Traffic, 10MediaWiki-API, 10Operations, 10Patch-For-Review: Varnish does not cache Action API responses when logged in - https://phabricator.wikimedia.org/T155314#3452500 (10Tgr) >>! In T155314#3452263, @fgiunchedi wrote: > This seems to WAI, only responses for anons are cached in varnish, anything left... [14:07:43] 10Traffic, 10MediaWiki-API, 10Operations, 10Patch-For-Review: Varnish does not cache Action API responses when logged in - https://phabricator.wikimedia.org/T155314#3452507 (10Tgr) [14:11:25] 10Traffic, 10MediaWiki-API, 10Operations, 10Patch-For-Review: Varnish does not cache Action API responses when logged in - https://phabricator.wikimedia.org/T155314#3452522 (10Tgr) This is not a huge deal right now as clients have to explicitly ask for API responses to be cached in Varnish (although some t... [14:37:23] 10Traffic, 10Operations, 10Patch-For-Review: prometheus-vhtcpd-stats cronspamming if vhtcpd is not running yet - https://phabricator.wikimedia.org/T157353#3452607 (10fgiunchedi) 05Open>03Resolved a:03fgiunchedi We're checking for file presence too now [14:48:47] 10netops, 10Operations, 10monitoring: internal network packet loss alerting - https://phabricator.wikimedia.org/T83196#3452641 (10fgiunchedi) +#netops as this sounds similar to {T169860} [14:53:44] 10netops, 10Operations, 10monitoring: internal network packet loss alerting - https://phabricator.wikimedia.org/T83196#3452661 (10ayounsi) 05Open>03Resolved a:03ayounsi It's fine to close. We already have better latency/loss monitoring than I guess in 2013/2015. T169860 might improve that even more. [14:58:37] 10Domains, 10Traffic, 10Operations, 10Patch-For-Review: wikiknihy.cz - transfer to Wikimedia Czech Republic? - https://phabricator.wikimedia.org/T127573#2047253 (10fgiunchedi) I've ran again a hive query to get some idea about this domain, 670 hits in june ``` 0: jdbc:hive2://analytics1003.eqiad.wmnet:100... [15:38:02] 10Traffic, 10Operations, 10Wikimedia-Planet, 10Patch-For-Review: mixed-content issues on planet.wikimedia.org - https://phabricator.wikimedia.org/T141480#3452829 (10Dzahn) Yes, not all planet feeds use https yet. [16:13:38] 10Traffic, 10Operations, 10Zero: Security: Is it safe to enable Zero spoofing - https://phabricator.wikimedia.org/T120631#3452969 (10fgiunchedi) [19:14:11] 10Traffic, 10Operations, 10Patch-For-Review: python-varnishapi daemons seeing "Log overrun" constantly - https://phabricator.wikimedia.org/T151643#3453850 (10ema) So it looks like the varnishstatsd overruns occur mostly in ulsfo: ``` $ sudo grep -F 'Log overru' /srv/syslog/syslog.log | grep cp4 -c 20737 18:... [20:26:18] 10Traffic, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Operations, and 14 others: Purge Varnish cache when a banner is saved - https://phabricator.wikimedia.org/T154954#3454232 (10DStrine)