[03:01:55] 10Release-Engineering-Team, 10MobileFrontend, 10Browser-Tests, 10MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), and 2 others: AssertionError: false === true at thereShouldBeALinkToCreateMyUserPage on wmf-quibble PHP jobs - https://phabricator.wikimedia.org/T221860 (10Jdlrobson) **This will need to be fixed in w... [05:35:43] 10Release-Engineering-Team, 10MobileFrontend, 10Browser-Tests, 10MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), and 2 others: AssertionError: false === true at thereShouldBeALinkToCreateMyUserPage on wmf-quibble PHP jobs - https://phabricator.wikimedia.org/T221860 (10Jdlrobson) Several runs of the above (with c... [05:39:53] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<20.00%) [06:03:35] 10Release-Engineering-Team, 10MobileFrontend, 10Browser-Tests, 10MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), and 3 others: AssertionError: false === true at thereShouldBeALinkToCreateMyUserPage on wmf-quibble PHP jobs - https://phabricator.wikimedia.org/T221860 (10Jdlrobson) [06:39:51] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [06:42:12] 10Continuous-Integration-Config, 10MediaWiki-extensions-Translate: Translate coverage publish step fails with: Incorrect whitelist config, no code coverage will be generated. - https://phabricator.wikimedia.org/T221913 (10abi_) Possible related patch - https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Tran... [10:13:58] PROBLEM - Host deployment-cache-upload04 is DOWN: CRITICAL - Host Unreachable (172.16.5.14) [10:29:14] 10Gerrit, 10Repository-Admins, 10Shape Expressions, 10Wikidata, 10Wikidata-Campsite: rename repository for WikibaseSchema - https://phabricator.wikimedia.org/T221946 (10Lucas_Werkmeister_WMDE) [11:53:46] (03CR) 10WMDE-leszek: "yes please." [integration/config] - 10https://gerrit.wikimedia.org/r/460527 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [11:56:21] (03CR) 10WMDE-leszek: Create selenium-daily-beta-WikibaseLexeme Jenkins job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/460527 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [13:08:34] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: On deployment-prep scap cache_git_info takes 12 minutes (that is too slow) - https://phabricator.wikimedia.org/T204762 (10hashar) >>! In T204762#5132043, @Krenair wrote: >>>! In T204762#4595548, @thcipriani wrote: >> Takes a while on... [13:18:30] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:33:37] !log shut off deployment-conf03 after discussion with otto.mata and elu.key - it seems ancient, broken, unused. T218729 [13:33:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:33:40] T218729: Migrate away from Debian Jessie to Debian Stretch - https://phabricator.wikimedia.org/T218729 [13:34:30] 10Beta-Cluster-Infrastructure: Migrate away from Debian Jessie to Debian Stretch - https://phabricator.wikimedia.org/T218729 (10Krenair) [13:35:09] PROBLEM - Citoid on deployment-sca02 is CRITICAL: connect to address 172.16.5.112 and port 1970: Connection refused [13:39:30] PROBLEM - Citoid on deployment-sca01 is CRITICAL: connect to address 172.16.5.13 and port 1970: Connection refused [13:40:09] RECOVERY - Citoid on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.028 second response time [13:40:16] PROBLEM - Host deployment-conf03 is DOWN: CRITICAL - Host Unreachable (172.16.5.30) [13:49:38] PROBLEM - Content Translation Server on deployment-sca02 is CRITICAL: connect to address 172.16.5.112 and port 8080: Connection refused [13:52:09] PROBLEM - Citoid on deployment-sca02 is CRITICAL: connect to address 172.16.5.112 and port 1970: Connection refused [13:59:05] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [14:00:37] * paladox reports https://bugs.chromium.org/p/gerrit/issues/detail?id=10782 [14:04:04] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.028 second response time [14:09:40] RECOVERY - Content Translation Server on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 904 bytes in 0.027 second response time [14:10:05] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [14:10:50] PROBLEM - Content Translation Server on deployment-sca01 is CRITICAL: connect to address 172.16.5.13 and port 8080: Connection refused [14:14:31] RECOVERY - Citoid on deployment-sca01 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.024 second response time [14:15:05] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.025 second response time [14:15:38] PROBLEM - Content Translation Server on deployment-sca02 is CRITICAL: connect to address 172.16.5.112 and port 8080: Connection refused [14:20:40] PROBLEM - Citoid on deployment-sca01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:31:35] 10Release-Engineering-Team: Puppet catalog compiler - increasing max concurrent jobs - https://phabricator.wikimedia.org/T221969 (10herron) [14:35:32] RECOVERY - Citoid on deployment-sca01 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.045 second response time [14:40:38] RECOVERY - Content Translation Server on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 904 bytes in 0.025 second response time [14:41:05] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [14:46:39] PROBLEM - Content Translation Server on deployment-sca02 is CRITICAL: connect to address 172.16.5.112 and port 8080: Connection refused [16:47:51] PROBLEM - Content Translation Server on deployment-sca01 is CRITICAL: connect to address 172.16.5.13 and port 8080: Connection refused [16:48:38] PROBLEM - Citoid on deployment-sca01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:49:08] PROBLEM - Citoid on deployment-sca02 is CRITICAL: connect to address 172.16.5.112 and port 1970: Connection refused [16:52:04] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [16:52:37] PROBLEM - Content Translation Server on deployment-sca02 is CRITICAL: connect to address 172.16.5.112 and port 8080: Connection refused [16:54:10] RECOVERY - Citoid on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.027 second response time [16:57:05] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.032 second response time [16:57:38] RECOVERY - Content Translation Server on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 904 bytes in 0.027 second response time [16:57:49] RECOVERY - Content Translation Server on deployment-sca01 is OK: HTTP OK: HTTP/1.1 200 OK - 904 bytes in 0.024 second response time [17:03:05] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [17:03:32] RECOVERY - Citoid on deployment-sca01 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.026 second response time [17:03:47] PROBLEM - Content Translation Server on deployment-sca02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:04:02] PROBLEM - Content Translation Server on deployment-sca01 is CRITICAL: (Service Check Timed Out) [17:06:09] PROBLEM - Citoid on deployment-sca02 is CRITICAL: connect to address 172.16.5.112 and port 1970: Connection refused [17:08:39] RECOVERY - Content Translation Server on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 904 bytes in 0.058 second response time [17:09:40] PROBLEM - Citoid on deployment-sca01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:48] RECOVERY - Content Translation Server on deployment-sca01 is OK: HTTP OK: HTTP/1.1 200 OK - 904 bytes in 0.029 second response time [17:19:40] PROBLEM - Content Translation Server on deployment-sca02 is CRITICAL: connect to address 172.16.5.112 and port 8080: Connection refused [17:23:05] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.035 second response time [17:24:30] RECOVERY - Citoid on deployment-sca01 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.025 second response time [17:24:49] PROBLEM - Content Translation Server on deployment-sca01 is CRITICAL: connect to address 172.16.5.13 and port 8080: Connection refused [17:30:41] PROBLEM - Citoid on deployment-sca01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:22:00] 10Deployments, 10Release-Engineering-Team, 10HHVM, 10MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), 10Upstream: hhvm aborts on extensions/CirrusSearch/tests/phan/stubs/intl.php blocking deployment sync - https://phabricator.wikimedia.org/T221814 (10Smalyshev) [18:23:52] 10Gerrit, 10LDAP: Gerrit: Cannot assign user name "vladi2016" to account XXXX; name already in use. - https://phabricator.wikimedia.org/T220867 (10MisterSynergy) I do have the same problem, waiting for a fix now. [19:00:14] (03PS1) 10Umherirrender: [GettingStarted] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/506727 [19:07:26] (03PS1) 10Umherirrender: [Score] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/506729 [19:10:45] (03PS1) 10Umherirrender: [ShortUrl] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/506731 [19:14:04] (03PS1) 10Umherirrender: [SiteMatrix] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/506733 [19:18:10] (03PS1) 10Umherirrender: [SkinPerPage] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/506736 [19:21:24] (03PS1) 10Umherirrender: [SendGrid] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/506737 [19:23:28] (03CR) 10D3r1ck01: [C: 03+1] "Thank you @Umherirrender :), over to Hashar :)" [integration/config] - 10https://gerrit.wikimedia.org/r/506737 (owner: 10Umherirrender) [19:24:56] (03CR) 10Umherirrender: [C: 04-1] "Waiting for I586294ced8d4790866d89bf3e870f0b55321a11d" [integration/config] - 10https://gerrit.wikimedia.org/r/506729 (owner: 10Umherirrender) [19:32:16] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:37:04] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.025 second response time [19:53:44] (03PS2) 10Umherirrender: [GettingStarted] Add dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/506727 [20:33:05] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [20:38:04] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.026 second response time [21:06:10] RECOVERY - Citoid on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.046 second response time [21:20:58] 10Gerrit, 10Patch-For-Review: Support setting site notice in PolyGerrit - https://phabricator.wikimedia.org/T215323 (10Paladox) 05Open→03Declined Upstream support this now. [21:21:23] 10Gerrit: Support setting site notice in PolyGerrit - https://phabricator.wikimedia.org/T215323 (10Paladox) [21:24:04] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [21:29:05] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.030 second response time [22:00:05] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [22:05:04] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.031 second response time [22:51:05] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [22:56:04] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.029 second response time [22:58:15] 10Gerrit, 10Patch-For-Review, 10Security: Disable TLS 1.0 and 1.1 in apache for gerrit.wikimedia.org - https://phabricator.wikimedia.org/T221499 (10Paladox) p:05Triage→03Normal [23:02:13] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: CRITICAL - Socket timeout after 10 seconds