[09:44:59] 10netops, 06Infrastructure-Foundations, 06SRE: Investigate gnmic metric gaps and counters going to zero - https://phabricator.wikimedia.org/T382396#10415035 (10fgiunchedi) Thank you all for looking into this -- let's indeed see how `3m` (or larger) goes and if that is satisfactory! >>! In T382396#10413490,... [15:03:36] 10netops, 06Infrastructure-Foundations, 10Observability-Metrics, 06SRE: replace check_ripe_atlas Python script with a check_prometheus backed by atlasexporter data - https://phabricator.wikimedia.org/T251155#10415860 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Done in {T370506} [16:05:37] Hello! I'd like to ask for some review and babysitting for deploying during the week of january 6. https://phabricator.wikimedia.org/T353817#10411130 [16:05:37] this is eventlogging decom varnish vcl work we've tried to deploy before, and are ready to try again. [16:06:29] ottomata: Thanks! I'll take a look and I'm happy to be present during deployment [16:16:00] 10netops, 06Infrastructure-Foundations, 06SRE: Investigate gnmic metric gaps and counters going to zero - https://phabricator.wikimedia.org/T382396#10416237 (10cmooney) >>! In T382396#10415035, @fgiunchedi wrote: > Yes and that's almost always the case, my understanding though is that the samples may not alw... [16:21:59] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: WMF RIPE Atlas probe in Eqiad offline - https://phabricator.wikimedia.org/T382518 (10cmooney) 03NEW p:05Triage→03Low [16:25:57] 10netops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: WMF RIPE Atlas probe in Eqsin offline - https://phabricator.wikimedia.org/T382519 (10cmooney) 03NEW p:05Triage→03Low [16:28:59] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: WMF RIPE Atlas probe in Eqiad offline - https://phabricator.wikimedia.org/T382518#10416342 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=7fe2fd80-b4a4-43f7-ba5a-5238c44bbd7a) set by cmooney@cumin1002 for 30 days,... [16:35:49] 10netops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: WMF RIPE Atlas probe in Eqsin offline - https://phabricator.wikimedia.org/T382519#10416397 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=68d77968-a0dd-4bd1-94ad-66be8ab508c5) set by cmooney@cumin1002 for 30 days, 0:00:00 on 2... [17:22:40] FIRING: [8x] VarnishHighThreadCount: Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [17:22:50] 06Traffic, 06Data Products, 06Data-Engineering, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: New software: haproxykafka - https://phabricator.wikimedia.org/T370668#10416668 (10Krinkle) [17:37:40] FIRING: [15x] VarnishHighThreadCount: Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [17:41:09] 06Traffic, 06Experimentation Lab: Cookie % has been rejected because it is foreign and does not have the "Partitioned" attribute - https://phabricator.wikimedia.org/T375256#10416763 (10Milimetric) [17:42:40] FIRING: [16x] VarnishHighThreadCount: Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [17:57:40] FIRING: [8x] VarnishHighThreadCount: Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [18:02:40] RESOLVED: [8x] VarnishHighThreadCount: Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [22:18:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [22:23:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX