[09:12:49] 10netops, 06Infrastructure-Foundations, 06SRE: Investigate gnmic metric gaps and counters going to zero - https://phabricator.wikimedia.org/T382396 (10fgiunchedi) 03NEW [09:13:36] 10netops, 06Infrastructure-Foundations, 06SRE: Productionize gnmic network telemetry pipeline - https://phabricator.wikimedia.org/T369384#10411780 (10fgiunchedi) No worries at all @cmooney, I've opened {T382396} to investigate/followup on the two issues you mentioned [10:19:27] 10netops, 06Infrastructure-Foundations, 06SRE: Investigate gnmic metric gaps and counters going to zero - https://phabricator.wikimedia.org/T382396#10411938 (10cmooney) > What is the dashboard and the underlying expression in the graph above? That one came from here I think: https://grafana.wikimedia.org/g... [10:20:01] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 4 others: Reimage wikikube-worker1290 in eqiad as a replacement for wikikube-ctrl1001 - https://phabricator.wikimedia.org/T379790#10411941 (10JMeybohm) [10:20:15] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 4 others: Reimage wikikube-worker1290 in eqiad as a replacement for wikikube-ctrl1001 - https://phabricator.wikimedia.org/T379790#10411942 (10JMeybohm) [11:39:52] 10netops, 06Infrastructure-Foundations, 06SRE: Investigate gnmic metric gaps and counters going to zero - https://phabricator.wikimedia.org/T382396#10412135 (10cmooney) @fgiunchedi yeah I'm pretty sure it's only gaps in the data we are seeing, for instance here: https://grafana.wikimedia.org/goto/_GSV1TIHR... [13:37:31] 10netops, 06Infrastructure-Foundations, 06SRE: Investigate gnmic metric gaps and counters going to zero - https://phabricator.wikimedia.org/T382396#10412405 (10fgiunchedi) Indeed the underlying data/samples are there as expected: I tested this theory by removing all functions and look at the raw data, which... [14:41:24] 10SRE-tools, 06Infrastructure-Foundations, 13Patch-For-Review: Add an ownership field to cookbooks. - https://phabricator.wikimedia.org/T379258#10412580 (10BTullis) >>! In T379258#10409987, @Volans wrote: > In an early draft I had thought of adding working groups to the list of possible groups but talking wi... [16:35:51] Congratulations on getting the wmf group management into bitu slyngs and moritzm. I know that has been a long hike from the first time y'all hoed to have that feature. [16:49:50] 10netops, 06Infrastructure-Foundations, 06SRE: Management routers: use BGP instead of OSPF - https://phabricator.wikimedia.org/T294845#10413071 (10cmooney) 05Resolved→03Open >>! In T294845#8758882, @ayounsi wrote: > This is completed in drmrs, the same will be applied to the other sites when we bring L3... [18:14:27] 10Mail, 06Infrastructure-Foundations, 06Trust-and-Safety: Emails from wikimediats.zendesk.com fails DMARC policy - https://phabricator.wikimedia.org/T378285#10413423 (10jhathaway) @JAbrams and I met yesterday to test using Zendesk's authenticated SMTP connector. We were unable to test the full flow, but we w... [18:23:03] 10netops, 06Infrastructure-Foundations, 06SRE: Investigate gnmic metric gaps and counters going to zero - https://phabricator.wikimedia.org/T382396#10413490 (10cmooney) >>! In T382396#10412404, @fgiunchedi wrote: > Indeed the underlying data/samples are there as expected: I tested this theory by removing all... [18:28:38] 10netops, 06Infrastructure-Foundations, 06SRE: Investigate gnmic metric gaps and counters going to zero - https://phabricator.wikimedia.org/T382396#10413543 (10CDanis) >>! In T382396#10413490, @cmooney wrote: > But we can deal with that if that is the cause. The goal of the "irate" is that we want as much g... [18:34:36] 10netops, 06Infrastructure-Foundations, 06SRE: Investigate gnmic metric gaps and counters going to zero - https://phabricator.wikimedia.org/T382396#10413565 (10cmooney) >>! In T382396#10413543, @CDanis wrote: > It's fine to make the time window longer with `irate()` -- it will always pick the two most-recent... [20:28:55] FIRING: MaxConntrack: Max conntrack at 82.42% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [20:33:55] RESOLVED: MaxConntrack: Max conntrack at 82.42% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack