[00:00:55] (03PS5) 10Wfan: Re-enable active paypal-ec recurring donor to update their EmailPreference Primary Email [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1237626 (https://phabricator.wikimedia.org/T416056) [00:21:50] (03CR) 10Wfan: [C:03+2] Delete unused DonorPortal messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1245478 (https://phabricator.wikimedia.org/T111677) (owner: 10Ejegg) [01:15:31] 03Fundraising Sprint: Dami's Crib 😀, 06Fundraising-Backlog, 07fr-current-sprint: Use new secondary email types to match contacts in WMFContact::Save - https://phabricator.wikimedia.org/T418660 (10Lars) 03NEW [01:49:43] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:reth2 (fasw1-f5 2x25G) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [05:49:43] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:reth2 (fasw1-f5 2x25G) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [07:19:45] (03CR) 10Raimond Spekking: "This comment is valid for translations into all languages: It would be easier for translatewiki staff like me (volunteer) if translations " [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1245466 (https://phabricator.wikimedia.org/T415682) (owner: 10Ejegg) [09:49:43] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:reth2 (fasw1-f5 2x25G) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [13:23:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 113.68, 87.18, 46.16 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [13:25:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 128.90, 102.61, 56.35 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [13:28:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 99.56, 104.16, 65.02 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [13:30:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 166.22, 127.71, 78.66 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [13:33:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 90.81, 107.41, 78.29 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [13:35:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 155.52, 131.42, 93.10 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [13:38:09] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 129.08, 100.87, 82.29 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [13:40:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 134.21, 127.31, 102.36 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [13:43:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 165.28, 138.28, 103.36 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [13:45:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 96.77, 112.05, 103.66 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [13:48:11] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 103.55, 118.98, 104.77 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [13:49:43] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:reth2 (fasw1-f5 2x25G) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [13:50:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 93.08, 99.91, 101.01 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [13:53:11] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 149.65, 124.39, 109.88 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [13:55:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 105.80, 111.58, 106.45 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [13:58:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 105.05, 112.50, 108.76 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [14:00:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 114.54, 115.43, 109.96 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:03:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 127.32, 118.56, 112.31 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [14:05:19] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 176.12, 150.67, 126.63 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:08:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 133.06, 124.07, 115.63 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [14:10:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 150.64, 146.73, 131.07 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:13:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 100.69, 102.60, 108.18 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [14:15:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 116.56, 126.66, 126.44 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:18:11] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 146.24, 126.02, 116.97 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [14:20:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 148.65, 133.03, 128.48 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:23:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 153.59, 139.76, 124.52 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [14:25:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 128.11, 133.36, 130.11 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:28:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 113.77, 131.32, 125.81 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [14:30:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 105.29, 113.72, 121.61 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:33:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 117.17, 126.61, 126.69 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [14:35:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 130.47, 114.74, 119.47 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:38:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 113.47, 119.13, 123.25 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [14:40:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 146.19, 134.98, 126.44 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:43:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 132.42, 123.63, 124.05 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [14:45:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 158.36, 151.64, 136.13 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:48:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 95.24, 106.96, 116.70 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [14:50:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 81.30, 114.53, 125.27 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:53:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 108.08, 127.01, 124.60 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [14:55:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 63.84, 87.76, 111.21 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:58:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 177.91, 158.15, 138.36 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [15:00:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 29.76, 56.49, 91.58 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [15:03:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 146.23, 159.77, 145.50 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [15:05:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 25.41, 39.83, 74.78 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [15:08:11] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 71.20, 110.73, 128.93 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [15:10:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 2.42, 16.44, 55.01 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [15:13:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 39.28, 66.01, 104.55 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [15:18:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 22.12, 41.34, 83.43 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [15:23:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 3.00, 17.21, 61.34 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [15:25:17] RECOVERY - check_load on frban2002 is OK: LOAD OK - total load average: 3.54, 4.43, 23.52 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [15:43:13] RECOVERY - check_load on frban1002 is OK: LOAD OK - total load average: 3.18, 3.78, 19.42 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [17:49:43] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:reth2 (fasw1-f5 2x25G) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [18:56:21] (03CR) 10CI reject: [V:04-1] build: Updating dependencies [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1246163 (owner: 10Libraryupgrader) [21:24:28] (03CR) 10CI reject: [V:04-1] build: Updating composer dependencies [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1246240 (owner: 10Libraryupgrader) [21:49:43] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:reth2 (fasw1-f5 2x25G) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [23:14:33] (03CR) 10CI reject: [V:04-1] build: Updating composer dependencies [extensions/FundraiserLandingPage] - 10https://gerrit.wikimedia.org/r/1246302 (owner: 10Libraryupgrader)