[00:07:32] PROBLEM - Citoid on deployment-sca01 is CRITICAL: connect to address 172.16.5.13 and port 1970: Connection refused [00:12:33] RECOVERY - Citoid on deployment-sca01 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.028 second response time [00:38:07] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.063 second response time [00:44:05] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [00:54:05] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.035 second response time [04:51:04] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [06:42:10] PROBLEM - Citoid on deployment-sca02 is CRITICAL: connect to address 172.16.5.112 and port 1970: Connection refused [06:44:53] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [06:47:10] RECOVERY - Citoid on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.026 second response time [06:51:05] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.034 second response time [06:57:07] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [09:47:04] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.121 second response time [09:53:04] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [13:18:31] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:00:20] 10Release-Engineering-Team (Kanban), 10MediaWiki-Release-Tools, 10VisualEditor, 10User-Ryasmeen, 10User-zeljkofilipin: fatal: Project not found: p/VisualEditor/VisualEditor - https://phabricator.wikimedia.org/T218694 (10freephile) The r/p path is still present in .gitmodules for REL1_32 (and others). Th... [15:03:55] 10Gerrit: reviewer-bot sometimes not working - https://phabricator.wikimedia.org/T222983 (10valhallasw) I cannot find any reference to this changeset in the bot logs. This suggests the changeset never showed up on the mediawiki-commits mailing list (which is odd), the email was filtered out as spam (again, uncle... [15:11:10] 10Gerrit: reviewer-bot sometimes not working - https://phabricator.wikimedia.org/T222983 (10valhallasw) There's another odd thing here. If we look at changeset 508721, GRB adds the reviewer at 21:30 UTC. The bot start logs show the following: ` Tue May 7 21:20:12 UTC 2019: Running as task 2915083 @ tools-sgeex... [15:13:50] 10Release-Engineering-Team (Kanban), 10MediaWiki-Release-Tools, 10VisualEditor, 10User-Ryasmeen, 10User-zeljkofilipin: fatal: Project not found: p/VisualEditor/VisualEditor - https://phabricator.wikimedia.org/T218694 (10Paladox) We redirect /r/p/*/info/* to /r/*/info/*. Though I guess would be good if th... [15:23:04] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.023 second response time [15:29:05] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [18:35:08] PROBLEM - Citoid on deployment-sca02 is CRITICAL: connect to address 172.16.5.112 and port 1970: Connection refused [18:45:10] RECOVERY - Citoid on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.028 second response time [19:00:03] Project mediawiki-core-code-coverage-docker build #4245: 04FAILURE in 4 hr 0 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-docker/4245/ [23:19:05] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.632 second response time [23:25:02] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [23:30:04] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.024 second response time [23:36:03] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused