[01:46:39] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12131165 (10Papaul) [07:19:49] 10netops, 06Infrastructure-Foundations, 06SRE: JunOS: Investigate BGP PIC Edge / Protection - https://phabricator.wikimedia.org/T432381#12131427 (10ayounsi) Another optimization to consider: https://www.juniper.net/documentation/us/en/software/junos/bgp/topics/topic-map/bgp-route-prioritization.html [07:20:25] FIRING: SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:50:25] FIRING: [2x] SystemdUnitFailed: ferm.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:06:28] FIRING: KeyholderUnarmed: 2 unarmed Keyholder key(s) on cumin1003:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [08:20:25] RESOLVED: SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:36:28] RESOLVED: KeyholderUnarmed: 1 unarmed Keyholder key(s) on cumin1003:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [08:56:18] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Create alerting for saturation on sub-rated interfaces - https://phabricator.wikimedia.org/T374614#12131571 (10ayounsi) As said on IRC, dunno if I'm opening some kind of pandora box by using starlark/python, but that seems like a good appro... [11:37:25] FIRING: SystemdUnitFailed: stunnel4.service on netmon1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:47:25] RESOLVED: SystemdUnitFailed: stunnel4.service on netmon1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:08:27] 10netbox, 10netops, 06Infrastructure-Foundations: Automatically run Capirca Netbox script regularly - https://phabricator.wikimedia.org/T361549#12132133 (10ayounsi) a:03ayounsi On Netbox-next: I created an "event-rule" - https://netbox-next.wikimedia.org/extras/event-rules/1/ {F94211857} Configured to ru... [12:35:17] 10netbox, 10netops, 06Infrastructure-Foundations: Automatically run Capirca Netbox script regularly - https://phabricator.wikimedia.org/T361549#12132210 (10cmooney) That's a great approach very nice! Haven't looked at the event-rule config but overall the idea is good with me +1 [13:09:14] hello, we have a new person to approve: https://ldap.toolforge.org/user/jniren-ctr