[10:26:23] 10netops, 10Operations: BGP peering sessions with corp partially down in ulsfo - https://phabricator.wikimedia.org/T239893 (10elukey) [10:27:47] hello traffic folks, it seems that we lost the active connection graph in the Load Balancer dashboard in icinga (cc godog ) [10:28:33] sorry I don't have time to look into it right now, but wanted to mention it [10:31:22] thanks volans I'll take a look [10:33:45] ah yeah I'll adjust the dashboard, new metric name [10:33:46] thx [10:35:59] 10Traffic, 10Operations, 10observability: 'LVS connections' graph on Load Balancers dashboard takes a rate of a gauge - https://phabricator.wikimedia.org/T236700 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Fixed now and 'load balancers' dashboard adjusted [10:37:41] 10Traffic, 10Operations: Unclear LVS bandwidth graph in "load balancers" dashboard - https://phabricator.wikimedia.org/T174432 (10fgiunchedi) >>! In T174432#3565169, @ema wrote: >>>! In T174432#3562830, @BBlack wrote: >> Are the non-icmp graphs somehow LVS-specific? > > Yes, the metrics are: node_ipvs_backend... [10:47:04] 10netops, 10Operations: Facebook BGP peering links down in ulsfo - https://phabricator.wikimedia.org/T239896 (10elukey) [17:55:32] 10netops, 10Operations: Facebook BGP peering links down in ulsfo - https://phabricator.wikimedia.org/T239896 (10colewhite) p:05Triage→03Normal [17:58:14] 10netops, 10Operations: BGP peering sessions with corp partially down in ulsfo - https://phabricator.wikimedia.org/T239893 (10colewhite) p:05Triage→03Normal [19:29:15] ema: if not already planned, as part of the whole ATS work there should also be a task to update the current switchdc cookbooks to the new reality, it should simplify it (and speed it up) quite a bit [19:30:06] yeah we get to add the new part where we have to make commits to the puppet and then dns repos, with a 5 minute delay between each, and then wait 30 minutes for propogation, once for every applayer service, right? [19:31:23] lol [19:33:26] wait what? [19:36:23] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: Convert DNS servers to Buster - https://phabricator.wikimedia.org/T239667 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by bblack on cumin1001.eqiad.wmnet for hosts: ` ['authdns2001.wikimedia.org'] ` The log can be found in `/var/log/wmf-... [19:38:08] cdanis: [19:39:07] is that what the old one had to do? [19:41:46] no, the old one had to do some puppet changes and a synchronized enable+run puppet in a specific order on the various layers of varnish [19:42:27] and to be sure it was ok, we were showing the diff to the operator and ask for confirmation [19:42:30] see https://gerrit.wikimedia.org/r/plugins/gitiles/operations/cookbooks/+/master/cookbooks/sre/switchdc/mediawiki/04-switch-traffic.py [19:43:24] to speed up the thing we were disabling puppet before the RO period in a previous step so that we could rebase+merge on gerrit waiting jekins before the RO period started [20:03:55] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: Convert DNS servers to Buster - https://phabricator.wikimedia.org/T239667 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['authdns2001.wikimedia.org'] ` and were **ALL** successful. [20:44:51] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: Convert DNS servers to Buster - https://phabricator.wikimedia.org/T239667 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by bblack on cumin1001.eqiad.wmnet for hosts: ` ['authdns1001.wikimedia.org'] ` The log can be found in `/var/log/wmf-... [21:05:59] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: Convert DNS servers to Buster - https://phabricator.wikimedia.org/T239667 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['authdns1001.wikimedia.org'] ` and were **ALL** successful. [21:13:20] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: Convert DNS servers to Buster - https://phabricator.wikimedia.org/T239667 (10BBlack) 05Open→03Resolved ` (12) authdns[1001,2001].wikimedia.org,dns[1001-1002,2001-2002,3001-3002,4001-4002,5001-5002].wikimedia.org... [21:13:22] 10Traffic, 10netops, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar): Anycast AuthDNS - https://phabricator.wikimedia.org/T98006 (10BBlack)