[03:51:06] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [03:56:05] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.030 second response time [05:20:52] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<10.00%) [06:45:54] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:19:19] PROBLEM - Citoid on deployment-sca02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:24:09] RECOVERY - Citoid on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.025 second response time [12:00:04] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [12:11:50] RECOVERY - Content Translation Server on deployment-sca01 is OK: HTTP OK: HTTP/1.1 200 OK - 904 bytes in 0.021 second response time [12:40:03] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.044 second response time [13:36:04] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [13:41:04] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.039 second response time [13:47:04] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [13:52:07] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.261 second response time [14:44:26] PROBLEM - Host deployment-ms-be03 is DOWN: CRITICAL - Host Unreachable (172.16.5.51) [14:45:19] PROBLEM - Host deployment-ms-be04 is DOWN: CRITICAL - Host Unreachable (172.16.4.129) [14:58:03] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [15:03:04] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.030 second response time [15:34:06] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [15:49:04] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.024 second response time [16:40:27] PROBLEM - Host deployment-ms-fe02 is DOWN: CRITICAL - Host Unreachable (172.16.5.66) [16:40:46] PROBLEM - Host deployment-poolcounter04 is DOWN: CRITICAL - Host Unreachable (172.16.5.58) [16:41:09] 10Beta-Cluster-Infrastructure: Migrate away from Debian Jessie to Debian Stretch - https://phabricator.wikimedia.org/T218729 (10Krenair) [16:51:05] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [17:11:05] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.023 second response time [17:58:28] 10Beta-Cluster-Infrastructure: Migrate away from Debian Jessie to Debian Stretch - https://phabricator.wikimedia.org/T218729 (10Krenair) [18:13:47] 10Beta-Cluster-Infrastructure: Migrate away from Debian Jessie to Debian Stretch - https://phabricator.wikimedia.org/T218729 (10Krenair) [19:13:03] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [19:18:03] (03PS1) 10Umherirrender: [TemplateData] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/506876 [19:21:14] (03PS1) 10Umherirrender: [TemplateWizard] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/506878 [19:23:54] (03PS1) 10Umherirrender: [TextExtracts] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/506880 [19:27:03] (03PS1) 10Umherirrender: [ThrottleOverride] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/506882 [19:28:05] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.040 second response time [19:30:52] (03PS1) 10Umherirrender: [TrustedXFF] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/506884 [20:05:05] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [21:39:35] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.34.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T220730 (10hashar) I might pair this one with @LarsWirzenius if there is any interest. And probably check to shift it to an earlier time which better accommodate European... [21:49:30] RECOVERY - Citoid on deployment-sca01 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.031 second response time [22:25:00] PROBLEM - Content Translation Server on deployment-sca01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:34:50] RECOVERY - Content Translation Server on deployment-sca01 is OK: HTTP OK: HTTP/1.1 200 OK - 904 bytes in 0.026 second response time [22:54:10] (03PS1) 10Hashar: Add MinvervaNeue and Vector to gate [integration/config] - 10https://gerrit.wikimedia.org/r/506889 (https://phabricator.wikimedia.org/T202030) [22:54:25] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Readers-Web-Backlog, 10Patch-For-Review: CI: Minerva PHPUnit tests should be included in shared extension gate job - https://phabricator.wikimedia.org/T202030 (10hashar) a:03hashar [23:06:04] 10Continuous-Integration-Infrastructure: wmf-quibble-vendor-mysql-hhvm-docker job sometime take 40+ minutes to run - https://phabricator.wikimedia.org/T222023 (10hashar) [23:36:05] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused