[08:44:29] <wikibugs>	 10Wikimedia-Apache-configuration, 06Infrastructure-Foundations: Do we still need the RW_PROTO env variable in wiki Apache configs? - https://phabricator.wikimedia.org/T382358#10411725 (10Joe) It's definitely not strictly needed now that we're fully on k8s (minus two mw-debug servers), where access to port 80 i...
[08:50:32] <wikibugs>	 10Wikimedia-Apache-configuration, 06Infrastructure-Foundations: Do we still need the RW_PROTO env variable in wiki Apache configs? - https://phabricator.wikimedia.org/T382358#10411732 (10Joe) I just realized I wasn't extremely clear above - that variable isn't strictly needed anymore but removing it would mean...
[09:12:49] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Investigate gnmic metric gaps and counters going to zero - https://phabricator.wikimedia.org/T382396 (10fgiunchedi) 03NEW
[09:13:36] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Productionize gnmic network telemetry pipeline - https://phabricator.wikimedia.org/T369384#10411780 (10fgiunchedi) No worries at all @cmooney, I've opened {T382396} to investigate/followup on the two issues you mentioned
[10:19:27] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Investigate gnmic metric gaps and counters going to zero - https://phabricator.wikimedia.org/T382396#10411938 (10cmooney) > What is the dashboard and the underlying expression in the graph above?  That one came from here I think:  https://grafana.wikimedia.org/g...
[10:20:01] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 4 others: Reimage wikikube-worker1290 in eqiad as a replacement for wikikube-ctrl1001 - https://phabricator.wikimedia.org/T379790#10411941 (10JMeybohm)
[10:20:15] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 4 others: Reimage wikikube-worker1290 in eqiad as a replacement for wikikube-ctrl1001 - https://phabricator.wikimedia.org/T379790#10411942 (10JMeybohm)
[11:39:52] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Investigate gnmic metric gaps and counters going to zero - https://phabricator.wikimedia.org/T382396#10412135 (10cmooney) @fgiunchedi yeah I'm pretty sure it's only gaps in the data we are seeing, for instance here:  https://grafana.wikimedia.org/goto/_GSV1TIHR...
[13:37:31] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Investigate gnmic metric gaps and counters going to zero - https://phabricator.wikimedia.org/T382396#10412405 (10fgiunchedi) Indeed the underlying data/samples are there as expected: I tested this theory by removing all functions and look at the raw data, which...
[15:23:27] <wikibugs>	 10Wikimedia-Apache-configuration, 06Infrastructure-Foundations: Do we still need the RW_PROTO env variable in wiki Apache configs? - https://phabricator.wikimedia.org/T382358#10412785 (10matmarex) 05Open→03Invalid Thanks, I'm happy with that answer, and having this task is good enough as documentation...
[16:49:50] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Management routers: use BGP instead of OSPF - https://phabricator.wikimedia.org/T294845#10413071 (10cmooney) 05Resolved→03Open >>! In T294845#8758882, @ayounsi wrote: > This is completed in drmrs, the same will be applied to the other sites when we bring L3...
[18:19:09] <jinxer-wm>	 FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[18:23:03] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Investigate gnmic metric gaps and counters going to zero - https://phabricator.wikimedia.org/T382396#10413490 (10cmooney) >>! In T382396#10412404, @fgiunchedi wrote: > Indeed the underlying data/samples are there as expected: I tested this theory by removing all...
[18:24:09] <jinxer-wm>	 RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[18:28:38] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Investigate gnmic metric gaps and counters going to zero - https://phabricator.wikimedia.org/T382396#10413543 (10CDanis) >>! In T382396#10413490, @cmooney wrote: > But we can deal with that if that is the cause.  The goal of the "irate" is that we want as much g...
[18:34:36] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Investigate gnmic metric gaps and counters going to zero - https://phabricator.wikimedia.org/T382396#10413565 (10cmooney) >>! In T382396#10413543, @CDanis wrote: > It's fine to make the time window longer with `irate()` -- it will always pick the two most-recent...