[05:12:08] 10netops, 10Operations: ripe-atlas-eqiad IPv6 unreachable - https://phabricator.wikimedia.org/T258018 (10ayounsi) According to https://radar.qrator.net portmap is open to the world but I was not able to reproduce. [09:26:26] 10netops, 10Operations, 10observability, 10Patch-For-Review, 10User-fgiunchedi: Migrate role::netmon to Buster - https://phabricator.wikimedia.org/T247967 (10fgiunchedi) [09:54:20] 10netops, 10Operations, 10observability, 10Patch-For-Review, 10User-fgiunchedi: Migrate role::netmon to Buster - https://phabricator.wikimedia.org/T247967 (10fgiunchedi) [10:02:48] 10netops, 10Operations, 10observability, 10Patch-For-Review, 10User-fgiunchedi: Migrate role::netmon to Buster - https://phabricator.wikimedia.org/T247967 (10fgiunchedi) [12:11:40] euwiki, frwiktionary, ptwikiversity banned on ATS first and Varnish later, all smooth [12:12:23] except that I passed the HOSTS_QUERY to cumin before -b and it told me I'm not cool [12:12:41] cumin A:cp-text -b1 [...] [12:13:06] but that's on me for getting too creative [12:16:35] 10Traffic, 10Desktop Improvements, 10Operations, 10Performance-Team (Radar): CDN cache revalidation on several wikis for desktop improvements deployment - https://phabricator.wikimedia.org/T256750 (10ema) euwiki, frewiktionary, and ptwikiversity done today. All good. [14:47:32] 10Traffic, 10Operations, 10serviceops, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10ema) [14:49:36] 10Traffic, 10Operations, 10serviceops, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10ema) 05Open→03Stalled [14:49:40] 10Traffic, 10Operations, 10Patch-For-Review: ATS production-ready as a backend cache layer - https://phabricator.wikimedia.org/T207048 (10ema) [14:57:24] 10Traffic, 10Operations: Backend naming in VCL needs to use fqdn+port - https://phabricator.wikimedia.org/T138546 (10ema) 05Open→03Declined Varnish backends are gone since T227432, this is now unnecessary. [15:02:11] 10Traffic, 10Operations, 10SRE-swift-storage, 10Patch-For-Review: Some PNG thumbnails and JPEG originals delivered as [text/html] content-type and hence not rendered in browser - https://phabricator.wikimedia.org/T162035 (10ema) 05Open→03Resolved >>! In T162035#5415777, @gerritbot wrote: > Change 53033... [15:03:01] 10Traffic, 10Operations, 10Patch-For-Review: Explicitly limit varnishd transient storage - https://phabricator.wikimedia.org/T164768 (10ema) 05Open→03Resolved a:03ema Both cache and upload now have limited transient: ` hieradata/role/common/cache/text.yaml:profile::cache::varnish::frontend::transient_... [16:37:50] 10Traffic, 10Operations, 10observability, 10serviceops: Alert "LVS api codfw port 80/tcp - MediaWiki API cluster- api.svc.eqiad.wmnet IPv4" is flapping - https://phabricator.wikimedia.org/T258614 (10jcrespo) [16:41:48] 10Traffic, 10Operations, 10observability, 10serviceops: Alert "LVS api codfw port 80/tcp - MediaWiki API cluster- api.svc.eqiad.wmnet IPv4" is flapping - https://phabricator.wikimedia.org/T258614 (10Dzahn) https://icinga.wikimedia.org/cgi-bin/icinga/history.cgi?host=api.svc.codfw.wmnet&service=LVS+api+codf... [16:47:01] 10Traffic, 10Operations, 10observability, 10serviceops: Alert "LVS api codfw port 80/tcp - MediaWiki API cluster- api.svc.eqiad.wmnet IPv4" is flapping - https://phabricator.wikimedia.org/T258614 (10jcrespo) [16:48:10] 10Traffic, 10Operations, 10observability, 10serviceops: Alert "LVS api codfw port 80/tcp - MediaWiki API cluster- api.svc.eqiad.wmnet IPv4" is flapping - https://phabricator.wikimedia.org/T258614 (10CDanis) First occurrence was June 17th, 15:10 UTC: `Jun 17 15:10:38 icinga1001 icinga: SERVICE ALERT: api.s... [17:30:04] 10Traffic, 10Operations, 10observability, 10serviceops: Alert "LVS api codfw port 80/tcp - MediaWiki API cluster- api.svc.eqiad.wmnet IPv4" is flapping - https://phabricator.wikimedia.org/T258614 (10Dzahn) a:03Dzahn mw2335 - mw2339 are configured as API appservers in confctl but they are regular appserv... [18:12:58] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: Remove multicast - https://phabricator.wikimedia.org/T257573 (10faidon) 05Resolved→03Open We still seem to have remnants of PIM-RP: ` faidon@re0.cr2-codfw> show configuration | display set | match 208.80.153.194 set interfaces lo0 unit... [19:51:51] 10Traffic, 10Operations, 10observability, 10serviceops: Alert "LVS api codfw port 80/tcp - MediaWiki API cluster- api.svc.eqiad.wmnet IPv4" is flapping - https://phabricator.wikimedia.org/T258614 (10Dzahn) So this happened whenever the check ended up talking to one of the servers in that 2335 - 2339 range.... [19:52:38] 10Traffic, 10Operations, 10observability, 10serviceops: Alert "LVS api codfw port 80/tcp - MediaWiki API cluster- api.svc.eqiad.wmnet IPv4" is flapping - https://phabricator.wikimedia.org/T258614 (10Dzahn) 05Open→03Resolved [19:53:09] 10Traffic, 10Operations, 10observability, 10serviceops: Alert "LVS api codfw port 80/tcp - MediaWiki API cluster- api.svc.eqiad.wmnet IPv4" is flapping - https://phabricator.wikimedia.org/T258614 (10Dzahn) [20:07:53] 10Traffic, 10Operations, 10observability, 10serviceops: monitoring for mismatched LVS realserver addresses/configurations - https://phabricator.wikimedia.org/T258648 (10CDanis)