[11:11:33] 06Traffic, 06Data-Engineering, 06Infrastructure-Foundations, 13Patch-For-Review: WMF-Last-Access-Global cookie set on wrong domain when accessing static assets - https://phabricator.wikimedia.org/T367346#10834307 (10Vgutierrez) @mforns I've submitted https://gerrit.wikimedia.org/r/c/operations/puppet/+/114... [14:33:32] 10netops, 06Infrastructure-Foundations, 07sre-alert-triage: Alert in need of triage: BGP status (instance cr2-drmrs) - https://phabricator.wikimedia.org/T393991#10835035 (10cmooney) p:05Triage→03Low [14:38:18] 06Traffic, 06Data-Engineering, 06Infrastructure-Foundations, 13Patch-For-Review: WMF-Last-Access-Global cookie set on wrong domain when accessing static assets - https://phabricator.wikimedia.org/T367346#10835061 (10mforns) @Vgutierrez thanks a lot for that! This should fix the issue and honor the filter t... [14:55:13] 06Traffic, 06Data-Engineering, 06Infrastructure-Foundations, 13Patch-For-Review: WMF-Last-Access-Global cookie set on wrong domain when accessing static assets - https://phabricator.wikimedia.org/T367346#10835125 (10mforns) Thinking about the potential effects of this change... I think it might add new cou... [14:59:58] 06Traffic, 06Experimentation Lab, 13Patch-For-Review: SDS 2.4.4 Edge Uniques Production Cookie Deployment - https://phabricator.wikimedia.org/T391411#10835133 (10Vgutierrez) [15:16:55] 06Traffic, 06Infrastructure-Foundations, 10Data-Engineering (Q4 2025 April 1st - June 30th), 13Patch-For-Review: WMF-Last-Access-Global cookie set on wrong domain when accessing static assets - https://phabricator.wikimedia.org/T367346#10835223 (10Ahoelzl) a:03mforns [15:25:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudvirt1068:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [15:29:13] err [15:29:20] why are we getting alerts for cloudvirt instances here? [15:29:41] sukhe: ^^ probably related to your work on puppet error alerting? [15:35:48] FIRING: [3x] PuppetZeroResources: Puppet has failed generate resources on cloudvirt1068:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [15:36:33] 06Traffic, 06Infrastructure-Foundations, 10Data-Engineering (Q4 2025 April 1st - June 30th): WMF-Last-Access-Global cookie set on wrong domain when accessing static assets - https://phabricator.wikimedia.org/T367346#10835418 (10Vgutierrez) 05Open→03Resolved test request on cp7001 after applying the p... [15:40:48] FIRING: [3x] PuppetZeroResources: Puppet has failed generate resources on cloudvirt1068:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [15:43:01] 06Traffic: Deb package for github.com/fabled/lua-maxminddb - https://phabricator.wikimedia.org/T394504#10835476 (10Fabfur) 05Open→03In progress [15:45:48] FIRING: [7x] PuppetZeroResources: Puppet has failed generate resources on cloudvirt1068:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [15:55:48] FIRING: [8x] PuppetZeroResources: Puppet has failed generate resources on cloudvirt1068:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [15:57:33] lol [16:00:48] FIRING: [7x] PuppetZeroResources: Puppet has failed generate resources on cloudvirt1069:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [16:19:25] vgutierrez: no, I have not done any work on that and we do not own cloudvirt* [16:19:47] yeah.. I'm totally aware that we don't own those instances [16:19:48] thanks [16:19:59] oho [16:20:01] The mystery thicken [16:20:20] in the email from alertmanager we get.. 3 alerts for alertname=PuppetZeroResources cluster=misc team=traffic [16:20:33] that alert or those instances are wrongly flagged as team=traffic [16:28:42] 06Traffic, 06Experimentation Lab, 13Patch-For-Review: SDS 2.4.4 Edge Uniques Production Cookie Deployment - https://phabricator.wikimedia.org/T391411#10835709 (10Vgutierrez) [17:05:48] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on cloudvirt1074:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [17:09:48] I suspect the wmcs alerts are happening because it's using role_owner in the alert [17:09:54] e.g. "puppet_agent_resources_total * on (instance) group_left (team) role_owner == 0" [17:10:12] https://w.wiki/EE72 [17:10:27] notice it's doing both wmcs and traffic as the team [17:40:04] this has been already clarified in -sre [19:14:04] 06Traffic, 06Data-Engineering-Radar, 10Observability-Logging, 13Patch-For-Review: Shutdown varnishkafka instances - https://phabricator.wikimedia.org/T393772#10836716 (10Fabfur) [19:26:02] FIRING: SLOMetricAbsent: varnish-combined esams - https://slo.wikimedia.org/?search=varnish-combined - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [19:26:17] FIRING: [2x] SLOMetricAbsent: haproxy-combined - https://slo.wikimedia.org/?search=haproxy-combined - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [19:26:26] FIRING: SLOMetricAbsent: varnish-combined ulsfo - https://slo.wikimedia.org/?search=varnish-combined - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [19:31:02] RESOLVED: [2x] SLOMetricAbsent: haproxy-combined - https://slo.wikimedia.org/?search=haproxy-combined - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [19:31:10] RESOLVED: [3x] SLOMetricAbsent: varnish-combined codfw - https://slo.wikimedia.org/?search=varnish-combined - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [19:31:26] RESOLVED: [2x] SLOMetricAbsent: varnish-combined esams - https://slo.wikimedia.org/?search=varnish-combined - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [23:19:40] FIRING: [3x] VarnishHighThreadCount: Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [23:24:40] FIRING: [8x] VarnishHighThreadCount: Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [23:34:40] FIRING: [8x] VarnishHighThreadCount: Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [23:39:40] FIRING: [8x] VarnishHighThreadCount: Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [23:44:40] RESOLVED: [8x] VarnishHighThreadCount: Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount