[00:08:25] FIRING: SystemdUnitFailed: dump_proxy_ranges.service on puppetserver1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:49:52] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN: Setup VRRP on both routers for the new subnets - https://phabricator.wikimedia.org/T427393#11961688 (10Papaul) [00:54:05] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN: Setup VRRP on both routers for the new subnets - https://phabricator.wikimedia.org/T427393#11961689 (10Papaul) VRRP is up on cr2-eqsin ` cr2-eqsin> show interfaces terse | match "ae1.512|ae1.522" et-0/0/1.512 up... [04:08:40] FIRING: SystemdUnitFailed: dump_proxy_ranges.service on puppetserver1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:03:25] RESOLVED: SystemdUnitFailed: dump_proxy_ranges.service on puppetserver1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:44:03] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Investigate internal rejected prefixes - https://phabricator.wikimedia.org/T423384#11962295 (10ayounsi) `lang=python { address="10.128.0.20", afi_safi="IPV4_UNICAST", instance="asw1-22-ulsfo:9804", job="gnmi", network_instance_name="default", peer_a... [09:01:52] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Investigate internal rejected prefixes - https://phabricator.wikimedia.org/T423384#11962424 (10ayounsi) Another one was: ` pfw1-codfw> show route receive-protocol bgp 208.80.153.202 hidden extensive inet.0: 26 destinations, 29 routes (26 active... [09:22:34] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Investigate internal rejected prefixes - https://phabricator.wikimedia.org/T423384#11962509 (10ayounsi) Another one, pfw1 re-advertises its uplinks subnets to cr1/2-codfw: ` cr2-codfw# run show route receive-protocol bgp 208.80.153.203 hidden extens... [12:07:25] FIRING: SystemdUnitFailed: dump_proxy_ranges.service on puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:58:32] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, and 2 others: EQSIN: Setup VRRP on both routers for the new subnets - https://phabricator.wikimedia.org/T427393#11963146 (10Papaul) @BCornwall hello can you please provide me with one CP node in rack 604 that i can use later on today to test th... [13:00:31] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#11963150 (10Papaul) [13:13:34] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, and 2 others: EQSIN: Setup VRRP on both routers for the new subnets - https://phabricator.wikimedia.org/T427393#11963174 (10Papaul) [13:15:10] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, and 2 others: EQSIN: Setup VRRP on both routers for the new subnets - https://phabricator.wikimedia.org/T427393#11963178 (10Papaul) [13:20:11] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, and 2 others: EQSIN: Setup VRRP on both routers for the new subnets - https://phabricator.wikimedia.org/T427393#11963191 (10Papaul) [14:20:32] 10SRE-tools, 10Ceph, 06cloud-services-team, 10Cloud-VPS, and 2 others: Enhancements to wmcs.ceph.roll_reboot_osds - https://phabricator.wikimedia.org/T427295#11963500 (10BLiviero-WMF) @Andrew do we understand _why_ this is happening? what research has been done to develop the understanding? [15:21:05] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, and 2 others: EQSIN: Setup VRRP on both routers for the new subnets - https://phabricator.wikimedia.org/T427393#11963925 (10BCornwall) Hi, @papaul: cp5032 is depooled/downtimed and ready for reimaging. [15:24:31] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, and 2 others: EQSIN: Setup VRRP on both routers for the new subnets - https://phabricator.wikimedia.org/T427393#11963940 (10Papaul) @BCornwall thank you will do that after lunch doing some onsite work [16:07:40] FIRING: SystemdUnitFailed: dump_proxy_ranges.service on puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:02:25] RESOLVED: SystemdUnitFailed: dump_proxy_ranges.service on puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:52:30] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN: Setup VRRP on both routers for the new subnets - https://phabricator.wikimedia.org/T427393#11964838 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cp5032.eqsin.wmnet with OS... [20:14:02] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN: Setup VRRP on both routers for the new subnets - https://phabricator.wikimedia.org/T427393#11965068 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cp5032.eqsin.wmnet with OS trix...