[01:45:41] PROBLEM - Host frpm1001 is DOWN: PING CRITICAL - Packet loss = 100% [01:50:11] PROBLEM - check_puppetrun on frauth1001 is CRITICAL: CRITICAL: Puppet has 31 failures. Last run 5 minutes ago with 31 failures. Failed resources (up to 3 shown) [01:55:11] PROBLEM - check_puppetrun on frauth1001 is CRITICAL: CRITICAL: Puppet has 31 failures. Last run 10 minutes ago with 31 failures. Failed resources (up to 3 shown) [01:55:12] PROBLEM - check_puppetrun on frlog1001 is CRITICAL: CRITICAL: Puppet has 26 failures. Last run 9 minutes ago with 26 failures. Failed resources (up to 3 shown) [02:00:11] PROBLEM - check_puppetrun on frauth1001 is CRITICAL: CRITICAL: Puppet has 31 failures. Last run 1 minute ago with 31 failures. Failed resources (up to 3 shown) [02:00:11] PROBLEM - check_puppetrun on bismuth is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 5 minutes ago with 27 failures. Failed resources (up to 3 shown) [02:00:12] PROBLEM - check_puppetrun on frlog1001 is CRITICAL: CRITICAL: Puppet has 26 failures. Last run 1 minute ago with 26 failures. Failed resources (up to 3 shown) [02:05:11] PROBLEM - check_puppetrun on bismuth is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 10 minutes ago with 27 failures. Failed resources (up to 3 shown) [02:05:11] PROBLEM - check_puppetrun on frauth1001 is CRITICAL: CRITICAL: Puppet has 31 failures. Last run 2 minutes ago with 31 failures. Failed resources (up to 3 shown) [02:05:12] PROBLEM - check_puppetrun on frlog1001 is CRITICAL: CRITICAL: Puppet has 26 failures. Last run 6 minutes ago with 26 failures. Failed resources (up to 3 shown) [02:10:11] PROBLEM - check_puppetrun on bismuth is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 30 seconds ago with 27 failures. Failed resources (up to 3 shown) [02:10:11] PROBLEM - check_puppetrun on frauth1001 is CRITICAL: CRITICAL: Puppet has 31 failures. Last run 7 minutes ago with 31 failures. Failed resources (up to 3 shown) [02:10:12] PROBLEM - check_puppetrun on frlog1001 is CRITICAL: CRITICAL: Puppet has 26 failures. Last run 11 minutes ago with 26 failures. Failed resources (up to 3 shown) [02:10:52] fundraising-tech-ops, Operations, ops-eqiad: frpm1001 is dead, looks like hardware failure - https://phabricator.wikimedia.org/T177710#3667572 (Jgreen) [02:11:00] fundraising-tech-ops, Operations, ops-eqiad: frpm1001 is dead, looks like hardware failure - https://phabricator.wikimedia.org/T177710#3667585 (Jgreen) p:Triage>Unbreak! [02:15:11] PROBLEM - check_puppetrun on frauth1001 is CRITICAL: CRITICAL: Puppet has 31 failures. Last run 1 minute ago with 31 failures. Failed resources (up to 3 shown) [02:15:11] PROBLEM - check_puppetrun on bismuth is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 5 minutes ago with 27 failures. Failed resources (up to 3 shown) [02:15:12] PROBLEM - check_puppetrun on frlog1001 is CRITICAL: CRITICAL: Puppet has 26 failures. Last run 16 minutes ago with 26 failures. Failed resources (up to 3 shown) [02:20:11] PROBLEM - check_puppetrun on frauth1001 is CRITICAL: CRITICAL: Puppet has 31 failures. Last run 6 minutes ago with 31 failures. Failed resources (up to 3 shown) [02:20:11] PROBLEM - check_puppetrun on payments1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:20:12] PROBLEM - check_puppetrun on bismuth is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 10 minutes ago with 27 failures. Failed resources (up to 3 shown) [02:20:12] PROBLEM - check_puppetrun on frdev1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:20:13] PROBLEM - check_puppetrun on frlog1001 is CRITICAL: CRITICAL: Puppet has 26 failures. Last run 4 minutes ago with 26 failures. Failed resources (up to 3 shown) [02:20:14] PROBLEM - check_puppetrun on frdb1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:20:31] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:25:11] PROBLEM - check_puppetrun on bismuth is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 28 seconds ago with 27 failures. Failed resources (up to 3 shown) [02:25:11] PROBLEM - check_puppetrun on frauth1001 is CRITICAL: CRITICAL: Puppet has 31 failures. Last run 11 minutes ago with 31 failures. Failed resources (up to 3 shown) [02:25:12] PROBLEM - check_puppetrun on payments1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:25:12] PROBLEM - check_puppetrun on payments1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:25:13] PROBLEM - check_puppetrun on payments1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:25:13] PROBLEM - check_puppetrun on civi1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:25:14] PROBLEM - check_puppetrun on frlog1001 is CRITICAL: CRITICAL: Puppet has 26 failures. Last run 9 minutes ago with 26 failures. Failed resources (up to 3 shown) [02:25:14] PROBLEM - check_puppetrun on frdev1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:25:15] PROBLEM - check_puppetrun on frdb1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:25:33] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:30:11] PROBLEM - check_puppetrun on frqueue1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:30:11] PROBLEM - check_puppetrun on pay-lvs1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:30:12] PROBLEM - check_puppetrun on pay-lvs1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:30:12] PROBLEM - check_puppetrun on civi1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:30:13] PROBLEM - check_puppetrun on bismuth is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 5 minutes ago with 27 failures. Failed resources (up to 3 shown) [02:30:13] PROBLEM - check_puppetrun on americium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:30:14] PROBLEM - check_puppetrun on frauth1001 is CRITICAL: CRITICAL: Puppet has 31 failures. Last run 2 minutes ago with 31 failures. Failed resources (up to 3 shown) [02:30:14] PROBLEM - check_puppetrun on frdb1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:30:15] PROBLEM - check_puppetrun on frdev1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:30:19] PROBLEM - check_puppetrun on payments1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:30:19] PROBLEM - check_puppetrun on payments1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:30:19] PROBLEM - check_puppetrun on payments1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:30:19] PROBLEM - check_puppetrun on frlog1001 is CRITICAL: CRITICAL: Puppet has 26 failures. Last run 1 minute ago with 26 failures. Failed resources (up to 3 shown) [02:30:19] PROBLEM - check_puppetrun on frdb1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:30:31] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:35:11] PROBLEM - check_puppetrun on frqueue1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:35:11] PROBLEM - check_puppetrun on americium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:35:12] PROBLEM - check_puppetrun on pay-lvs1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:35:12] PROBLEM - check_puppetrun on pay-lvs1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:35:13] PROBLEM - check_puppetrun on bismuth is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 10 minutes ago with 27 failures. Failed resources (up to 3 shown) [02:35:13] PROBLEM - check_puppetrun on civi1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:35:14] PROBLEM - check_puppetrun on frauth1001 is CRITICAL: CRITICAL: Puppet has 31 failures. Last run 2 minutes ago with 31 failures. Failed resources (up to 3 shown) [02:35:14] PROBLEM - check_puppetrun on frdb1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:35:15] PROBLEM - check_puppetrun on payments1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:35:15] PROBLEM - check_puppetrun on payments1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:35:16] PROBLEM - check_puppetrun on payments1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:35:16] PROBLEM - check_puppetrun on frlog1001 is CRITICAL: CRITICAL: Puppet has 26 failures. Last run 6 minutes ago with 26 failures. Failed resources (up to 3 shown) [02:35:17] PROBLEM - check_puppetrun on frdev1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:35:18] PROBLEM - check_puppetrun on frdb1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:35:31] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:40:11] PROBLEM - check_puppetrun on pay-lvs1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:40:12] PROBLEM - check_puppetrun on pay-lvs1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:40:12] PROBLEM - check_puppetrun on civi1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:40:13] PROBLEM - check_puppetrun on payments1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:40:13] PROBLEM - check_puppetrun on frqueue1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:40:13] PROBLEM - check_puppetrun on payments1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:40:14] PROBLEM - check_puppetrun on payments1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:40:14] PROBLEM - check_puppetrun on frauth1001 is CRITICAL: CRITICAL: Puppet has 31 failures. Last run 7 minutes ago with 31 failures. Failed resources (up to 3 shown) [02:40:15] PROBLEM - check_puppetrun on bismuth is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 50 seconds ago with 27 failures. Failed resources (up to 3 shown) [02:40:19] RECOVERY - check_puppetrun on americium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:40:19] PROBLEM - check_puppetrun on frlog1001 is CRITICAL: CRITICAL: Puppet has 26 failures. Last run 11 minutes ago with 26 failures. Failed resources (up to 3 shown) [02:40:19] PROBLEM - check_puppetrun on frdev1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:40:19] RECOVERY - check_puppetrun on frdb1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:40:19] PROBLEM - check_puppetrun on frdb1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:40:31] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:45:12] PROBLEM - check_puppetrun on frauth1001 is CRITICAL: CRITICAL: Puppet has 31 failures. Last run 1 minute ago with 31 failures. Failed resources (up to 3 shown) [02:45:13] PROBLEM - check_puppetrun on frqueue1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:45:13] PROBLEM - check_puppetrun on pay-lvs1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:45:13] PROBLEM - check_puppetrun on pay-lvs1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:45:13] PROBLEM - check_puppetrun on bismuth is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 5 minutes ago with 27 failures. Failed resources (up to 3 shown) [02:45:13] PROBLEM - check_puppetrun on civi1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:45:14] PROBLEM - check_puppetrun on frlog1001 is CRITICAL: CRITICAL: Puppet has 26 failures. Last run 16 minutes ago with 26 failures. Failed resources (up to 3 shown) [02:45:14] PROBLEM - check_puppetrun on tellurium is CRITICAL: CRITICAL: Puppet has 23 failures. Last run 5 minutes ago with 23 failures. Failed resources (up to 3 shown) [02:45:14] PROBLEM - check_puppetrun on payments1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:45:15] PROBLEM - check_puppetrun on payments1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:45:15] RECOVERY - check_puppetrun on payments1001 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [02:45:16] RECOVERY - check_puppetrun on frdev1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:45:17] PROBLEM - check_puppetrun on frdb1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:45:21] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:50:11] PROBLEM - check_puppetrun on pay-lvs1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:50:11] PROBLEM - check_puppetrun on frqueue1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:50:12] PROBLEM - check_puppetrun on frauth1001 is CRITICAL: CRITICAL: Puppet has 31 failures. Last run 6 minutes ago with 31 failures. Failed resources (up to 3 shown) [02:50:12] PROBLEM - check_puppetrun on civi1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:50:13] RECOVERY - check_puppetrun on pay-lvs1001 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [02:50:13] PROBLEM - check_puppetrun on bismuth is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 10 minutes ago with 27 failures. Failed resources (up to 3 shown) [02:50:14] PROBLEM - check_puppetrun on tellurium is CRITICAL: CRITICAL: Puppet has 23 failures. Last run 10 minutes ago with 23 failures. Failed resources (up to 3 shown) [02:50:14] RECOVERY - check_puppetrun on payments1002 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [02:50:15] RECOVERY - check_puppetrun on payments1003 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [02:50:15] PROBLEM - check_puppetrun on frlog1001 is CRITICAL: CRITICAL: Puppet has 26 failures. Last run 4 minutes ago with 26 failures. Failed resources (up to 3 shown) [02:50:16] PROBLEM - check_puppetrun on frdb1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:50:31] RECOVERY - check_puppetrun on payments1004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:55:11] RECOVERY - check_puppetrun on frqueue1002 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [02:55:11] PROBLEM - check_puppetrun on frauth1001 is CRITICAL: CRITICAL: Puppet has 31 failures. Last run 11 minutes ago with 31 failures. Failed resources (up to 3 shown) [02:55:12] PROBLEM - check_puppetrun on bismuth is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 1 minute ago with 27 failures. Failed resources (up to 3 shown) [02:55:12] RECOVERY - check_puppetrun on pay-lvs1002 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [02:55:13] RECOVERY - check_puppetrun on tellurium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:55:13] PROBLEM - check_puppetrun on civi1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:55:14] PROBLEM - check_puppetrun on frlog1001 is CRITICAL: CRITICAL: Puppet has 26 failures. Last run 9 minutes ago with 26 failures. Failed resources (up to 3 shown) [02:55:15] RECOVERY - check_puppetrun on frdb1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:00:11] RECOVERY - check_puppetrun on bismuth is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [03:00:11] RECOVERY - check_puppetrun on frauth1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:00:12] RECOVERY - check_puppetrun on civi1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:00:12] RECOVERY - check_puppetrun on frlog1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:41:51] RECOVERY - Host frpm1001 is UP: PING OK - Packet loss = 0%, RTA = 0.43 ms [14:44:29] fundraising-tech-ops, Operations, ops-eqiad: frpm1001 is dead, looks like hardware failure - https://phabricator.wikimedia.org/T177710#3667572 (Cmjohnson) The system board for frpm1001 will need to be rplaced, the server will not power on, tried reseating power supplies, DIMM, CPU1 with no luck. Th... [16:01:14] PROBLEM - check_minfraud_secondary on payments1001 is CRITICAL: CRITICAL - Socket timeout after 61 seconds [16:01:15] PROBLEM - check_minfraud_secondary on payments1002 is CRITICAL: CRITICAL - Socket timeout after 61 seconds [16:01:15] PROBLEM - check_minfraud_secondary on payments1003 is CRITICAL: CRITICAL - Socket timeout after 61 seconds [16:01:21] PROBLEM - check_minfraud_secondary on payments2002 is CRITICAL: CRITICAL - Socket timeout after 61 seconds [16:01:22] PROBLEM - check_minfraud_secondary on payments2003 is CRITICAL: CRITICAL - Socket timeout after 61 seconds [16:01:24] PROBLEM - check_minfraud_secondary on payments2001 is CRITICAL: CRITICAL - Socket timeout after 61 seconds [16:01:31] PROBLEM - check_minfraud_secondary on payments1004 is CRITICAL: CRITICAL - Socket timeout after 61 seconds [16:05:12] RECOVERY - check_minfraud_secondary on payments1001 is OK: HTTP OK: Status line output matched 200,301,302 - 385 bytes in 0.016 second response time [16:05:13] RECOVERY - check_minfraud_secondary on payments1002 is OK: HTTP OK: Status line output matched 200,301,302 - 385 bytes in 0.016 second response time [16:05:14] RECOVERY - check_minfraud_secondary on payments1003 is OK: HTTP OK: Status line output matched 200,301,302 - 385 bytes in 0.019 second response time [16:05:16] RECOVERY - check_minfraud_secondary on payments2002 is OK: HTTP OK: Status line output matched 200,301,302 - 385 bytes in 0.141 second response time [16:05:17] RECOVERY - check_minfraud_secondary on payments2003 is OK: HTTP OK: Status line output matched 200,301,302 - 385 bytes in 0.142 second response time [16:05:18] RECOVERY - check_minfraud_secondary on payments2001 is OK: HTTP OK: Status line output matched 200,301,302 - 385 bytes in 0.145 second response time [16:05:21] RECOVERY - check_minfraud_secondary on payments1004 is OK: HTTP OK: Status line output matched 200,301,302 - 385 bytes in 0.015 second response time [22:24:17] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Add editorconfig files to our codebase to deal with whitespace pain? - https://phabricator.wikimedia.org/T177725#3668035 (Eileenmcnaughton) [22:36:11] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Add editorconfig files to our codebase to deal with whitespace pain? - https://phabricator.wikimedia.org/T177725#3668062 (Eileenmcnaughton) [22:52:37] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Add editorconfig files to our codebase to deal with whitespace pain? - https://phabricator.wikimedia.org/T177725#3668080 (Eileenmcnaughton) This is the psr2 version https://github.com/pfefferle/.editorconfigs/blob/master/php-psr-2/.editorconfig