[02:48:56] FIRING: [4x] SystemdUnitFailed: squid-logrotate.service on install2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:11:05] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [04:11:05] RESOLVED: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [06:48:06] 10netops, 10Ganeti, 06Infrastructure-Foundations, 13Patch-For-Review: esams: move sandbox vlan to routed Ganeti - https://phabricator.wikimedia.org/T403580#11146558 (10MoritzMuehlenhoff) >>! In T403580#11142998, @ayounsi wrote: > @MoritzMuehlenhoff I tried to create the VM using `sudo cookbook sre.ganeti.m... [06:48:56] FIRING: [4x] SystemdUnitFailed: squid-logrotate.service on install2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:53:41] 10netops, 10Ganeti, 06Infrastructure-Foundations, 13Patch-For-Review: esams: move sandbox vlan to routed Ganeti - https://phabricator.wikimedia.org/T403580#11146886 (10ayounsi) [09:02:47] 10netops, 10Ganeti, 06Infrastructure-Foundations, 13Patch-For-Review: esams: move sandbox vlan to routed Ganeti - https://phabricator.wikimedia.org/T403580#11146920 (10ayounsi) 05Open→03Resolved The RIPE re-generated an image using the /32 and /128 netmask. The install went perfectly fine. [09:13:56] FIRING: [5x] SystemdUnitFailed: squid-logrotate.service on install2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:16:05] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [11:16:05] RESOLVED: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [12:37:13] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Switch frack eqiad frdata-codfw NAT to frdata2002.frack.codfw.wmnet - https://phabricator.wikimedia.org/T403718 (10Jgreen) 03NEW [12:38:07] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Switch frack eqiad frdata-codfw NAT to frdata2002.frack.codfw.wmnet - https://phabricator.wikimedia.org/T403718#11147607 (10Jgreen) [12:42:42] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Reuse old payments-codfw LVS-DR IP for frmx2002 NAT - https://phabricator.wikimedia.org/T403719 (10Jgreen) 03NEW [12:52:53] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Reuse old payments-codfw LVS-DR IP for frmx2002 NAT - https://phabricator.wikimedia.org/T403719#11147667 (10Jgreen) p:05Triage→03Medium [13:13:56] FIRING: [5x] SystemdUnitFailed: squid-logrotate.service on install2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:28:44] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Move pfw1b-codfw to rack F5 - https://phabricator.wikimedia.org/T401297#11147869 (10Papaul) I talked to @Jgreen on IRC about the schedule, there is a maintenance window during from September 22nd to the 26th so this will be a best time for the m... [13:32:37] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Reuse old payments-codfw LVS-DR IP for frmx2002 NAT - https://phabricator.wikimedia.org/T403719#11147894 (10ayounsi) 05Open→03Resolved nat added [13:34:10] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Switch frack eqiad frdata-codfw NAT to frdata2002.frack.codfw.wmnet - https://phabricator.wikimedia.org/T403718#11147901 (10Jgreen) [13:34:41] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Switch frack eqiad frdata-codfw NAT to frdata2002.frack.codfw.wmnet - https://phabricator.wikimedia.org/T403718#11147903 (10ayounsi) 05Open→03Resolved a:03ayounsi All good there too [14:46:14] 10netops, 06Infrastructure-Foundations, 06SRE: Management routers: use BGP instead of OSPF - https://phabricator.wikimedia.org/T294845#11148288 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=829d4d0b-c9d0-4961-b07b-d12e8f1ac430) set by pt1979@cumin2002 for 2:00:00 on 1 host(s) and their... [15:14:01] FIRING: [5x] SystemdUnitFailed: squid-logrotate.service on install2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:13:56] FIRING: [4x] SystemdUnitFailed: squid-logrotate.service on install2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:53:50] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Management routers: use BGP instead of OSPF - https://phabricator.wikimedia.org/T294845#11150505 (10Papaul) mr1-ulsfo is now running BGP . All OSPF entries on mr1-ulsfo, cr3-ulsfo and cr4-ulsfo for the management network removed. [23:13:56] FIRING: [4x] SystemdUnitFailed: squid-logrotate.service on install2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed