[01:49:43] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:reth2 (fasw1-f5 2x25G) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [05:49:43] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:reth2 (fasw1-f5 2x25G) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [07:40:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 125.20, 91.66, 45.47 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [07:43:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 103.38, 93.83, 53.68 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [07:45:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 86.81, 94.34, 59.46 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [07:48:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 87.64, 102.24, 69.13 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [07:50:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 46.04, 68.64, 58.89 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [07:53:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 44.97, 68.95, 64.12 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [08:10:17] RECOVERY - check_load on frban2002 is OK: LOAD OK - total load average: 3.77, 5.86, 20.66 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [08:13:13] RECOVERY - check_load on frban1002 is OK: LOAD OK - total load average: 5.91, 6.49, 21.98 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [09:49:44] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:reth2 (fasw1-f5 2x25G) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [13:49:44] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:reth2 (fasw1-f5 2x25G) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [14:10:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 123.94, 120.35, 98.59 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:13:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 148.53, 129.78, 104.17 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [14:15:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 80.55, 105.70, 99.82 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:18:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 118.30, 128.45, 111.78 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [14:20:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 157.41, 128.39, 110.59 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:23:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 144.97, 130.92, 116.76 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [14:25:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 88.55, 118.26, 112.27 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:28:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 132.22, 138.20, 124.18 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [14:30:19] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 82.81, 109.50, 111.51 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:33:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 145.19, 142.91, 129.82 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [14:35:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 105.34, 107.65, 110.39 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:38:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 104.98, 140.20, 133.74 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [14:40:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 120.57, 116.69, 113.43 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:43:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 111.49, 123.28, 128.45 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [14:45:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 94.87, 110.75, 112.43 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:48:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 111.96, 117.53, 124.28 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [14:50:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 101.54, 123.58, 119.33 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:53:11] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 128.36, 127.40, 126.35 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [14:55:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 155.77, 136.87, 125.66 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:58:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 94.83, 103.95, 116.36 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [15:00:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 98.25, 121.16, 122.57 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [15:03:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 82.02, 88.31, 106.24 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [15:05:19] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 156.65, 137.96, 128.57 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [15:08:11] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 40.49, 64.08, 91.31 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [15:10:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 133.31, 133.32, 129.17 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [15:13:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 27.04, 42.34, 74.47 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [15:15:19] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 113.21, 134.30, 131.78 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [15:18:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 9.35, 25.72, 58.79 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [15:20:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 145.40, 131.29, 130.33 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [15:25:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 60.91, 102.79, 119.61 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [15:30:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 48.30, 73.73, 103.08 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [15:35:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 29.85, 48.13, 84.20 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [15:38:11] RECOVERY - check_load on frban1002 is OK: LOAD OK - total load average: 4.22, 5.67, 20.29 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [15:40:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 15.83, 31.56, 67.40 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [15:45:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 24.39, 18.47, 51.33 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [16:00:17] RECOVERY - check_load on frban2002 is OK: LOAD OK - total load average: 45.06, 13.68, 24.98 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [17:15:19] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 129.47, 83.38, 41.21 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [17:18:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 125.66, 109.45, 63.54 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [17:20:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 116.75, 109.21, 64.60 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [17:23:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 93.27, 103.16, 73.75 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [17:25:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 74.87, 96.08, 71.97 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [17:28:11] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 42.56, 69.00, 67.61 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [17:30:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 51.78, 69.58, 67.52 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [17:33:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 5.86, 32.63, 52.58 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [17:35:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 5.67, 28.99, 50.46 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [17:48:13] RECOVERY - check_load on frban1002 is OK: LOAD OK - total load average: 5.51, 6.08, 22.67 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [17:49:44] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:reth2 (fasw1-f5 2x25G) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [17:50:17] RECOVERY - check_load on frban2002 is OK: LOAD OK - total load average: 4.50, 8.30, 23.67 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load