[00:29:22] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown: Different age of history logged in and out - https://phabricator.wikimedia.org/T246185 (10Luke081515) [00:30:39] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown: Different age of history logged in and out when from the EU (but not SF office) - https://phabricator.wikimedia.org/T246185 (10Jdforrester-WMF) [00:46:57] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown: Different age of history logged in and out when from the EU (but not SF office) - https://phabricator.wikimedia.org/T246185 (10Aklapper) Hmm, I'm afraid I cannot confirm at 00:44 UTC from Central Europe. Both logged in and private mode window show the... [00:59:29] 10netops, 10Operations: can aggregated netflow data include the router it was sampled from? - https://phabricator.wikimedia.org/T246186 (10CDanis) [01:42:49] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install LVS200[7-10] - https://phabricator.wikimedia.org/T196560 (10wiki_willy) [06:24:39] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown: Different age of history logged in and out when from the EU (but not SF office) - https://phabricator.wikimedia.org/T246185 (10Conny) Got yesterday info for this via Servicehotline [1] for page https://de.wikipedia.org/wiki/Benutzer:Gerhard_Steigerwald... [07:38:39] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install LVS200[7-10] - https://phabricator.wikimedia.org/T196560 (10Vgutierrez) @Papaul I think I can handle that as well. I'll let you know, the first one will be lvs2006 [07:54:01] 10Traffic, 10Operations, 10conftool, 10Patch-For-Review, and 2 others: Figure out a security model for etcd - https://phabricator.wikimedia.org/T97972 (10akosiaris) Per @volans recommendation, adding a use case here: * Be able to read conftool data via automated means (i.e. scripts) in order to cross chec... [08:01:16] 10netops, 10Operations: PyBal BGP group prefix-limit 50 teardown - https://phabricator.wikimedia.org/T246110 (10akosiaris) For what is worth, the Kubernetes groups (v4, v6) have copy pasted the PyBal group, so it probably makes sense to follow the same approach for those as well. There is one extra interesting... [10:00:45] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown: Different age of history logged in and out when from the EU (but not SF office) - https://phabricator.wikimedia.org/T246185 (10Aklapper) I'm wondering if this could have anything do with the yesterday's earlier problem at https://wikitech.wikimedia.org... [10:51:53] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown: Different age of history logged in and out when from the EU (but not SF office) - https://phabricator.wikimedia.org/T246185 (10Luke081515) As an logged out user I'm now able to see the newest version. However, I've temporary revoked the review status... [12:27:24] 10netops, 10Operations: PyBal BGP group prefix-limit 50 teardown - https://phabricator.wikimedia.org/T246110 (10ayounsi) > [1] I say at least, because in the a bad scenario we experienced, due to operator error (yours truly), summarization failed and we ended up advertising something like 40 /32 prefixes. If t... [13:05:04] 10netops, 10Operations: can aggregated netflow data include the router it was sampled from? - https://phabricator.wikimedia.org/T246186 (10ayounsi) It's possible by adding the keys `in_iface`, `out_iface` and `export_proto_sysid` to `modules/pmacct/templates/nfacctd.conf.erb` See: https://github.com/pmacct/pma... [13:10:37] 10netops, 10Operations: can aggregated netflow data include the router it was sampled from? - https://phabricator.wikimedia.org/T246186 (10ayounsi) About this specific peak, we could also configure pmacct to send BGP events to Kafka using BMP, see https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L2230 [13:53:48] 10netops, 10Operations: RRDP status alert - https://phabricator.wikimedia.org/T245121 (10ayounsi) As it keeps flapping I (temporarily) disabled the alert in eqiad, and we can rely in codfw if there is an actual issue. [15:40:07] 10netops, 10Operations: PyBal BGP group prefix-limit 50 teardown - https://phabricator.wikimedia.org/T246110 (10akosiaris) >>! In T246110#5919494, @ayounsi wrote: >> [1] I say at least, because in the a bad scenario we experienced, due to operator error (yours truly), summarization failed and we ended up adver... [17:59:16] someone knows how the esams depool has been down? With the rebalancing of traffic to codfw? I don't think the repool is a regular one. Or is it? [18:05:56] XioNoX: not regular in the sense that we didn't change admin_state but "config", was done in I32e93cbe80 [19:44:43] 10netops, 10Operations: BFD session alerts due to inconsistent status on cr3-knams - https://phabricator.wikimedia.org/T240659 (10ayounsi) 05Open→03Resolved No more issues. [20:01:20] 10netops, 10Operations: PyBal BGP group prefix-limit 50 teardown - https://phabricator.wikimedia.org/T246110 (10ayounsi) Ok, looks like a good one would be: ` set protocols bgp group PyBal family inet unicast prefix-limit maximum 1000 teardown 20 set protocols bgp group PyBal family inet6 unicast prefix-limit...