[06:05:58] 06Traffic, 10conftool, 07Epic: Deleting ipblocks when there are uncommitted changes causes failures - https://phabricator.wikimedia.org/T378435 (10Joe) 03NEW [07:41:10] 06Traffic, 10conftool, 07Epic: Deleting ipblocks when there are uncommitted changes causes failures - https://phabricator.wikimedia.org/T378435#10271064 (10Joe) p:05Triage→03High [09:51:03] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Support PyBal routes announced with lower priority than "backup" - https://phabricator.wikimedia.org/T354839#10271470 (10Vgutierrez) Gven the limitations to run pybal and liberica on the same hosts, we want to run liberica on separate hosts with a h... [09:57:20] XioNoX: https://phabricator.wikimedia.org/T354839#10271470 --> should I open a task for this? [10:03:08] vgutierrez: sure, always a good idea to open a task. That will be a temporary community iirc? FYI all our communities are defined in https://wikitech.wikimedia.org/wiki/IP_and_AS_allocations#BGP_communities and :2 is already used [10:03:30] XioNoX: yes.. it should be something temporary [10:03:48] cool, we can maybe re-use 14907:10 in that case [10:04:06] or 14907:6 [10:04:36] cool.. I'll create the task and let you pick the community that you consider suitable for this [10:06:55] vgutierrez: when do you need it? [10:07:54] XioNoX: I'm ready to reimage lvs1013 as soon as I get approval on the role from fab.fur and s.ukhe, so running initial tests this week would be great [10:08:32] vgutierrez: sweet! [10:10:12] 10netops, 06Infrastructure-Foundations: Testing liberica with ncredir@eqiad - https://phabricator.wikimedia.org/T378453 (10Vgutierrez) 03NEW [10:10:19] that's T378453 [10:10:20] T378453: Testing liberica with ncredir@eqiad - https://phabricator.wikimedia.org/T378453 [13:56:15] 06Traffic, 10ops-magru, 06SRE: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged) - https://phabricator.wikimedia.org/T376737#10272741 (10ssingh) Hi @RobH: thanks for writing this up. The instructions, hostnames (and serial numbers) all look good. The date/time also work for Traf... [16:36:29] 06Traffic, 10CX-cxserver, 10RESTBase Sunsetting: Block RESTBase cxserver v1 endpoints in favor of the new endpoints - https://phabricator.wikimedia.org/T375616#10273694 (10MSantos) [17:35:47] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10273979 (10Dwisehaupt) @cmooney @Jclark-ctr Got confirmation that the date shift is good. We are all set t... [18:10:25] FIRING: SystemdUnitFailed: prometheus-nft-throttling-denylist.service on durum2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:10:30] hmm [18:14:33] ^ related to the nftables change, daniel is looking into it. doesn't affect the service, just the stats [18:15:25] FIRING: [3x] SystemdUnitFailed: prometheus-nft-throttling-denylist.service on durum2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:15:38] 06Traffic, 10ops-magru, 06SRE: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged) - https://phabricator.wikimedia.org/T376737#10274253 (10RobH) @papaul: As the point of contact between DC Ops and #netops, did you want to handle the router rules/ACLs to allow us to reinstall all o... [18:20:25] FIRING: [5x] SystemdUnitFailed: prometheus-nft-throttling-denylist.service on durum1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:25:25] FIRING: [9x] SystemdUnitFailed: prometheus-nft-throttling-denylist.service on durum1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:30:25] FIRING: [9x] SystemdUnitFailed: prometheus-nft-throttling-denylist.service on durum1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:35:25] FIRING: [8x] SystemdUnitFailed: prometheus-nft-throttling-denylist.service on durum1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:35:30] ^ should be resolving [18:38:39] I ran puppet and systemctl start for that on 14 durum hosts [18:40:25] RESOLVED: [8x] SystemdUnitFailed: prometheus-nft-throttling-denylist.service on durum1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:40:39] mutante: nice! [19:45:43] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10274577 (10Jclark-ctr) @cmooney fyi i have 10x of the 100g green handled optics [20:20:13] 06Traffic, 10ops-magru, 06SRE: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged) - https://phabricator.wikimedia.org/T376737#10274656 (10Papaul) @RobH yes i can take care of that [20:59:45] 06Traffic, 10ops-magru, 06SRE: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged) - https://phabricator.wikimedia.org/T376737#10274774 (10RobH)