[00:01:03] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [00:52:00] PROBLEM - Host Graphite Labs is DOWN: check_ping: Invalid hostname/address - graphite-labs.wikimedia.org [00:52:57] PROBLEM - Host Graphite Labs is DOWN: check_ping: Invalid hostname/address - graphite-labs.wikimedia.org [00:54:23] Project beta-code-update-eqiad build #262933: 04FAILURE in 1 min 22 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/262933/ [00:55:52] Network issues? [00:59:28] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:04:03] Project mediawiki-core-doxygen-docker build #9713: 04FAILURE in 0.15 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/9713/ [01:04:22] Project beta-code-update-eqiad build #262934: 04STILL FAILING in 1 min 22 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/262934/ [01:04:23] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 46377 bytes in 4.901 second response time [01:14:23] Project beta-code-update-eqiad build #262935: 04STILL FAILING in 1 min 22 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/262935/ [01:15:01] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:19:59] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 46914 bytes in 4.502 second response time [01:20:24] Project beta-update-databases-eqiad build #36591: 04FAILURE in 23 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/36591/ [01:24:28] Project beta-code-update-eqiad build #262936: 04STILL FAILING in 1 min 27 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/262936/ [01:27:45] Hmm [01:28:51] paladox: I'd ignore it and go to bed if I were you :) [01:29:01] Ok :) [01:34:14] RECOVERY - Puppet staleness on deployment-ms-fe03 is OK: OK: Less than 1.00% above the threshold [3600.0] [01:34:20] RECOVERY - Puppet errors on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [2.0] [01:34:24] Project beta-code-update-eqiad build #262937: 04STILL FAILING in 1 min 23 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/262937/ [01:34:41] RECOVERY - Puppet staleness on deployment-sessionstore02 is OK: OK: Less than 1.00% above the threshold [3600.0] [01:34:51] RECOVERY - Free space - all mounts on deployment-db06 is OK: OK: All targets OK [01:35:05] RECOVERY - Puppet staleness on deployment-memc05 is OK: OK: Less than 1.00% above the threshold [3600.0] [01:35:17] RECOVERY - Puppet errors on deployment-ms-be05 is OK: OK: Less than 1.00% above the threshold [2.0] [01:35:20] RECOVERY - Puppet staleness on deployment-acme-chief03 is OK: OK: Less than 1.00% above the threshold [3600.0] [01:35:23] RECOVERY - Free space - all mounts on deployment-logstash2 is OK: OK: deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (No valid datapoints found) deployment-prep.deployment-logstash2.diskspace._var_lib_elasticsearch.byte_percentfree (No valid datapoints found) [01:35:24] RECOVERY - Puppet staleness on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [3600.0] [01:35:24] RECOVERY - Puppet staleness on integration-slave-docker-1048 is OK: OK: Less than 1.00% above the threshold [3600.0] [01:35:24] RECOVERY - Free space - all mounts on deployment-deploy02 is OK: OK: All targets OK [01:35:35] RECOVERY - Free space - all mounts on integration-slave-docker-1058 is OK: OK: All targets OK [01:35:39] RECOVERY - Puppet errors on deployment-docker-citoid01 is OK: OK: Less than 1.00% above the threshold [2.0] [01:35:43] RECOVERY - Free space - all mounts on deployment-acme-chief03 is OK: OK: All targets OK [01:35:44] RECOVERY - Free space - all mounts on deployment-aqs01 is OK: OK: All targets OK [01:35:44] RECOVERY - Puppet errors on integration-slave-jessie-1004 is OK: OK: Less than 1.00% above the threshold [2.0] [01:35:44] RECOVERY - Puppet staleness on deployment-eventlog05 is OK: OK: Less than 1.00% above the threshold [3600.0] [01:35:56] RECOVERY - Free space - all mounts on deployment-jobrunner03 is OK: OK: All targets OK [01:35:57] RECOVERY - Puppet errors on deployment-hadoop-test-3 is OK: OK: Less than 1.00% above the threshold [2.0] [01:35:57] RECOVERY - Puppet staleness on deployment-db05 is OK: OK: Less than 1.00% above the threshold [3600.0] [01:36:00] RECOVERY - Free space - all mounts on integration-cumin is OK: OK: All targets OK [01:36:13] RECOVERY - Host Graphite Labs is UP: PING OK - Packet loss = 0%, RTA = 3.99 ms [01:36:18] RECOVERY - Free space - all mounts on deployment-memc04 is OK: OK: All targets OK [01:36:36] RECOVERY - Puppet errors on deployment-db05 is OK: OK: Less than 1.00% above the threshold [2.0] [01:36:43] RECOVERY - Puppet errors on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [2.0] [01:36:45] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [2.0] [01:36:59] RECOVERY - Free space - all mounts on deployment-schema-2 is OK: OK: All targets OK [01:37:15] RECOVERY - Free space - all mounts on deployment-memc07 is OK: OK: All targets OK [01:37:18] RECOVERY - Puppet staleness on deployment-sca02 is OK: OK: Less than 1.00% above the threshold [3600.0] [01:37:19] RECOVERY - Puppet staleness on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [3600.0] [01:37:21] RECOVERY - Puppet staleness on deployment-snapshot01 is OK: OK: Less than 1.00% above the threshold [3600.0] [01:37:31] RECOVERY - Puppet errors on deployment-aqs02 is OK: OK: Less than 1.00% above the threshold [2.0] [01:37:31] RECOVERY - Puppet staleness on deployment-chromium02 is OK: OK: Less than 1.00% above the threshold [3600.0] [01:37:35] RECOVERY - Free space - all mounts on deployment-ircd is OK: OK: All targets OK [01:37:37] RECOVERY - Puppet staleness on integration-slave-docker-1050 is OK: OK: Less than 1.00% above the threshold [3600.0] [01:37:37] RECOVERY - Puppet staleness on deployment-hadoop-test-1 is OK: OK: Less than 1.00% above the threshold [3600.0] [01:37:43] RECOVERY - Puppet errors on integration-castor03 is OK: OK: Less than 1.00% above the threshold [2.0] [01:37:53] RECOVERY - Puppet staleness on integration-slave-docker-1041 is OK: OK: Less than 1.00% above the threshold [3600.0] [01:37:57] RECOVERY - Puppet staleness on deployment-mcs01 is OK: OK: Less than 1.00% above the threshold [3600.0] [01:38:02] RECOVERY - Host Graphite Labs is UP: PING OK - Packet loss = 0%, RTA = 1.95 ms [01:38:12] RECOVERY - Puppet staleness on deployment-docker-citoid01 is OK: OK: Less than 1.00% above the threshold [3600.0] [01:38:13] RECOVERY - Free space - all mounts on deployment-sessionstore01 is OK: OK: All targets OK [01:38:17] RECOVERY - Puppet staleness on deployment-mwmaint01 is OK: OK: Less than 1.00% above the threshold [3600.0] [01:38:17] RECOVERY - Puppet staleness on integration-castor03 is OK: OK: Less than 1.00% above the threshold [3600.0] [01:38:22] RECOVERY - Free space - all mounts on deployment-aqs02 is OK: OK: All targets OK [01:38:33] RECOVERY - Puppet staleness on integration-puppetmaster01 is OK: OK: Less than 1.00% above the threshold [3600.0] [01:44:23] Project beta-code-update-eqiad build #262938: 04STILL FAILING in 1 min 22 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/262938/ [01:54:25] Yippee, build fixed! [01:54:25] Project beta-code-update-eqiad build #262939: 09FIXED in 1 min 24 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/262939/ [02:11:02] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [02:12:16] Yippee, build fixed! [02:12:16] Project mediawiki-core-doxygen-docker build #9714: 09FIXED in 8 min 12 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/9714/ [02:21:28] Yippee, build fixed! [02:21:29] Project beta-update-databases-eqiad build #36592: 09FIXED in 1 min 28 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/36592/ [02:31:04] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [04:11:00] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:11:03] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:51:06] hello releng! [06:51:24] I need some help with moving this: https://gerrit.wikimedia.org/r/c/mediawiki/services/kartotherian/+/534419 forward [06:51:49] I believe that patch should trigger docker image build for kartotherian and publish it as well. [07:01:47] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:34:57] (03CR) 10Awight: "What about adding this to the php.ini for the docker image, so it applies to every invocation of PHP?" [integration/quibble] - 10https://gerrit.wikimedia.org/r/534468 (https://phabricator.wikimedia.org/T219694) (owner: 10Ladsgroup) [08:11:03] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [08:36:54] 10Project-Admins, 10translatewiki.net: Convert T41480 ("Issues affecting translatewiki.net") to a project tag - https://phabricator.wikimedia.org/T231991 (10Nikerabbit) I was thinking what similar tags already exist. #wikimedia-production-error is very similar. If we use that pattern, it would be #translatewik... [08:58:51] (03CR) 10Awight: "Hmm, I guess I'm suggesting copying a config file to both /etc/php/7.0/fpm/conf.d/ and to /etc/php/7.0/cli/conf.d/" [integration/quibble] - 10https://gerrit.wikimedia.org/r/534468 (https://phabricator.wikimedia.org/T219694) (owner: 10Ladsgroup) [09:13:19] 10Project-Admins: New Phabricator project for Wikicontrib - https://phabricator.wikimedia.org/T231268 (10Aklapper) 05Resolved→03Open a:05Aklapper→03None I [just saw](https://lists.wikimedia.org/pipermail/wikitech-l/2019-September/092517.html) that https://github.com/wikimedia/WikiContrib/issues exists, h... [09:33:09] (03CR) 10Kosta Harlan: "> Patch Set 3:" [integration/quibble] - 10https://gerrit.wikimedia.org/r/534468 (https://phabricator.wikimedia.org/T219694) (owner: 10Ladsgroup) [10:03:11] Glad gerrit survived the network issues last night :) [10:11:02] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [10:23:32] hey all:) [10:24:00] I did a scap sync file and it didn't get logged on IRC [10:24:04] thus on SAL [11:14:58] (03CR) 10Awight: "> Or ini_set( 'output_buffering', 'Off' ) in DevelopmentSettings.php?" [integration/quibble] - 10https://gerrit.wikimedia.org/r/534468 (https://phabricator.wikimedia.org/T219694) (owner: 10Ladsgroup) [11:32:05] effie: you can always manually !log in #wikimedia-operations [11:32:21] yes, that is what I did [12:02:36] (03PS7) 10Hashar: Diff declared deps in CI and in registration json [integration/config] - 10https://gerrit.wikimedia.org/r/504437 [12:02:38] (03PS1) 10Hashar: FundraisinerLandingPage requires EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/535161 [12:02:40] (03PS1) 10Hashar: Shibboleth requires PluggableAuth [integration/config] - 10https://gerrit.wikimedia.org/r/535162 [12:02:42] (03PS1) 10Hashar: Add BlueSpiceEditNotifyConnector dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/535163 [12:02:44] (03PS1) 10Hashar: BlueSpcePagesVisited requires BlueSpiceWhoIsOnline [integration/config] - 10https://gerrit.wikimedia.org/r/535164 [12:02:46] (03PS1) 10Hashar: BlueSpiceUE* require BlueSpiceUniversalExport [integration/config] - 10https://gerrit.wikimedia.org/r/535165 [12:02:48] (03PS1) 10Hashar: BlueSpiceVisualEditorConnector requires OOJSPlus [integration/config] - 10https://gerrit.wikimedia.org/r/535166 [12:05:18] (03CR) 10jerkins-bot: [V: 04-1] Diff declared deps in CI and in registration json [integration/config] - 10https://gerrit.wikimedia.org/r/504437 (owner: 10Hashar) [12:06:35] (03PS2) 10Hashar: FundraiserLandingPage requires EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/535161 [12:06:43] (03CR) 10Hashar: [C: 03+2] FundraiserLandingPage requires EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/535161 (owner: 10Hashar) [12:06:54] (03CR) 10Hashar: [C: 03+2] Shibboleth requires PluggableAuth [integration/config] - 10https://gerrit.wikimedia.org/r/535162 (owner: 10Hashar) [12:07:03] (03CR) 10Hashar: [C: 03+2] Add BlueSpiceEditNotifyConnector dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/535163 (owner: 10Hashar) [12:09:16] 10Continuous-Integration-Config, 10Code-Health-Metrics, 10Code-Health: Error in code coverage reports for extensions in codehealth-pipeline - https://phabricator.wikimedia.org/T232195 (10kostajh) Ah, it's only failing for extensions which have no unit tests, and which have 0% coverage in the junit.xml. These... [12:09:34] 10Diffusion, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201907), 10Operations, and 4 others: Cannot connect to vcs@git-ssh.wikimedia.org (since move from phab1001 to phab1003) - https://phabricator.wikimedia.org/T224677 (10MoritzMuehlenhoff) The Debian Stretch point... [12:11:03] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [12:11:07] (03Merged) 10jenkins-bot: FundraiserLandingPage requires EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/535161 (owner: 10Hashar) [12:15:25] !log Reloading Zuul for https://gerrit.wikimedia.org/r/535161 " FundraiserLandingPage requires EventLogging" [12:15:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:18:42] (03PS2) 10Hashar: Shibboleth requires PluggableAuth [integration/config] - 10https://gerrit.wikimedia.org/r/535162 [12:18:45] (03PS2) 10Hashar: Add BlueSpiceEditNotifyConnector dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/535163 [12:18:48] (03PS2) 10Hashar: BlueSpcePagesVisited requires BlueSpiceWhoIsOnline [integration/config] - 10https://gerrit.wikimedia.org/r/535164 [12:18:51] (03PS2) 10Hashar: BlueSpiceUE* require BlueSpiceUniversalExport [integration/config] - 10https://gerrit.wikimedia.org/r/535165 [12:18:53] (03PS2) 10Hashar: BlueSpiceVisualEditorConnector requires OOJSPlus [integration/config] - 10https://gerrit.wikimedia.org/r/535166 [12:18:55] (03PS8) 10Hashar: Diff declared deps in CI and in registration json [integration/config] - 10https://gerrit.wikimedia.org/r/504437 [12:19:01] (03CR) 10Hashar: "Fixed flake8 issues" [integration/config] - 10https://gerrit.wikimedia.org/r/504437 (owner: 10Hashar) [12:19:19] (03CR) 10Hashar: [C: 03+2] Shibboleth requires PluggableAuth [integration/config] - 10https://gerrit.wikimedia.org/r/535162 (owner: 10Hashar) [12:19:36] (03CR) 10Hashar: [C: 03+2] Add BlueSpiceEditNotifyConnector dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/535163 (owner: 10Hashar) [12:19:41] (03CR) 10Hashar: [C: 03+2] BlueSpcePagesVisited requires BlueSpiceWhoIsOnline [integration/config] - 10https://gerrit.wikimedia.org/r/535164 (owner: 10Hashar) [12:19:56] (03CR) 10Hashar: [C: 03+2] BlueSpiceUE* require BlueSpiceUniversalExport [integration/config] - 10https://gerrit.wikimedia.org/r/535165 (owner: 10Hashar) [12:19:59] (03CR) 10Hashar: [C: 03+2] BlueSpiceVisualEditorConnector requires OOJSPlus [integration/config] - 10https://gerrit.wikimedia.org/r/535166 (owner: 10Hashar) [12:23:08] (03Merged) 10jenkins-bot: Shibboleth requires PluggableAuth [integration/config] - 10https://gerrit.wikimedia.org/r/535162 (owner: 10Hashar) [12:23:11] (03Merged) 10jenkins-bot: Add BlueSpiceEditNotifyConnector dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/535163 (owner: 10Hashar) [12:26:14] (03Merged) 10jenkins-bot: BlueSpcePagesVisited requires BlueSpiceWhoIsOnline [integration/config] - 10https://gerrit.wikimedia.org/r/535164 (owner: 10Hashar) [12:26:19] (03Merged) 10jenkins-bot: BlueSpiceUE* require BlueSpiceUniversalExport [integration/config] - 10https://gerrit.wikimedia.org/r/535165 (owner: 10Hashar) [12:26:22] (03Merged) 10jenkins-bot: BlueSpiceVisualEditorConnector requires OOJSPlus [integration/config] - 10https://gerrit.wikimedia.org/r/535166 (owner: 10Hashar) [12:26:58] !log Reloading Zuul to add multiple extension dependencies [12:26:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:36:01] (03PS5) 1020after4: Phatality plugin for kibana [releng/phatality] - 10https://gerrit.wikimedia.org/r/531047 (https://phabricator.wikimedia.org/T230752) [12:43:54] thcipriani https://gerrit.wikimedia.org/r/monitoring [12:44:13] is that the issue again? [12:44:46] 466,115ms for mediawiki-config is kind of long [12:46:02] https://www.gerritcodereview.com/2019-09-09-gerrit-3.1-release-and-2.15-planned-eol.html 2.15 is offically EOL in nov [12:49:29] ah, seems to have resolved it's self. [12:56:25] Amir1: kostajh: Thanks again for the recent unit test push! I enjoy the improved runtime every day... [13:30:05] <3 [13:34:56] fyi gerrit-slave is now removed from acme :) [14:09:00] James_F: hello... [14:09:10] can you take a look at https://gerrit.wikimedia.org/r/c/mediawiki/services/kartotherian/+/534419 pls? [14:09:11] Hey. [14:09:16] Sure. [14:09:20] thanks! [14:11:04] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [14:13:08] 10Beta-Cluster-Infrastructure: Interface admin rights request for meta.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T232341 (10Pcoombe) [14:21:28] Project beta-update-databases-eqiad build #36604: 04FAILURE in 1 min 26 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/36604/ [14:23:19] awight: glad to hear it! [14:24:27] (03PS1) 10Hashar: Shibboleth disable selenium tests [integration/config] - 10https://gerrit.wikimedia.org/r/535198 [15:01:26] (03CR) 10Hashar: [C: 03+2] Shibboleth disable selenium tests [integration/config] - 10https://gerrit.wikimedia.org/r/535198 (owner: 10Hashar) [15:03:53] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201909): integration-config-zuul-layout-validate-docker takes too long - https://phabricator.wikimedia.org/T232287 (10hashar) [15:04:08] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201909): integration-config-zuul-layout-validate-docker takes too long in Jenkins due to huge output - https://phabricator.wikimedia.org/T232287 (10hashar) [15:04:28] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201909), 10Jenkins: integration-config-zuul-layout-validate-docker takes too long in Jenkins due to huge output - https://phabricator.wikimedia.org/T232287 (10hashar) + #jenkins since that might be the actual root cause? :-\ [15:04:50] (03Merged) 10jenkins-bot: Shibboleth disable selenium tests [integration/config] - 10https://gerrit.wikimedia.org/r/535198 (owner: 10Hashar) [15:05:08] !log Reloading Zuul for https://gerrit.wikimedia.org/r/535198 Shibboleth disable selenium tests [15:05:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:21:32] Yippee, build fixed! [15:21:32] Project beta-update-databases-eqiad build #36605: 09FIXED in 1 min 31 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/36605/ [15:34:34] Project beta-scap-eqiad build #266237: 04FAILURE in 1.2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/266237/ [15:44:29] Project beta-scap-eqiad build #266238: 04STILL FAILING in 3.5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/266238/ [15:45:04] "16:44:29 RuntimeError: Scap failed!: Call to mwscript eval.php stderr: Warning: Use of undefined constant MW_ENTRY_POINT - assumed 'MW_ENTRY_POINT' (this will throw an Error in a future version of PHP) in /srv/mediawiki-staging/php-master/maintenance/Maintenance.php on line 23" [15:51:52] https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/533981/2/maintenance/Maintenance.php [15:52:08] The name needs to be a string [15:52:12] (03CR) 10Krinkle: [C: 03+1] "I vaguely recall it not being possible to turn off output_buffering within a web request (too late?). But worth a shot." [integration/quibble] - 10https://gerrit.wikimedia.org/r/534468 (https://phabricator.wikimedia.org/T219694) (owner: 10Ladsgroup) [15:54:25] Project beta-scap-eqiad build #266239: 04STILL FAILING in 0.69 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/266239/ [15:54:40] FTR, https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/535219/ [15:55:14] 10Gerrit, 10Wikimedia-General-or-Unknown, 10Documentation, 10Epic, and 4 others: Update Gerrit /r/p/ links to /r/ - https://phabricator.wikimedia.org/T218844 (10thcipriani) 05Open→03Resolved a:03Paladox >>! In T218844#5171654, @gerritbot wrote: > Change 507787 **merged** by Dzahn: > [operations/puppe... [15:55:21] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Patch-For-Review: Upgrade to Gerrit 2.16.10 - https://phabricator.wikimedia.org/T200739 (10thcipriani) [15:56:31] 10Gerrit, 10Wikimedia-General-or-Unknown, 10Documentation, 10Epic, and 4 others: Update Gerrit /r/p/ links to /r/ - https://phabricator.wikimedia.org/T218844 (10Paladox) \o/ [15:57:19] 10Gerrit, 10Release-Engineering-Team-TODO (201909), 10Documentation: Update Gerrit documentation on mediawiki.org before upgrading to Gerrit 2.16.x / PolyGerrit UI - https://phabricator.wikimedia.org/T227562 (10thcipriani) [15:57:24] 10Gerrit, 10Release-Engineering-Team-TODO (201909), 10Documentation: Update Gerrit documentation on mediawiki.org before upgrading to Gerrit 2.16.x / PolyGerrit UI - https://phabricator.wikimedia.org/T227562 (10thcipriani) p:05Triage→03Normal [16:01:26] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Patch-For-Review: Upgrade to Gerrit 2.16.11.1 - https://phabricator.wikimedia.org/T200739 (10Paladox) [16:04:26] Project beta-scap-eqiad build #266240: 04STILL FAILING in 1.2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/266240/ [16:11:02] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [16:11:59] (03CR) 10Jforrester: [C: 03+1] "Do we want to try to wire this into the repo tests?" [integration/config] - 10https://gerrit.wikimedia.org/r/504437 (owner: 10Hashar) [16:14:29] Project beta-scap-eqiad build #266241: 04STILL FAILING in 0.86 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/266241/ [16:24:27] Project beta-scap-eqiad build #266242: 04STILL FAILING in 0.68 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/266242/ [16:25:07] 10Release-Engineering-Team-TODO (201909), 10Browser-Tests: selenium-daily-beta-MediaWiki possibly broken by password changes - https://phabricator.wikimedia.org/T232357 (10Jdforrester-WMF) [16:34:33] Project beta-scap-eqiad build #266243: 04STILL FAILING in 0.9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/266243/ [16:44:36] Project beta-scap-eqiad build #266244: 04STILL FAILING in 0.72 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/266244/ [16:54:30] Project beta-scap-eqiad build #266245: 04STILL FAILING in 3.7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/266245/ [16:57:27] (03CR) 10Jforrester: "> Patch Set 1: Code-Review+1" [integration/config] - 10https://gerrit.wikimedia.org/r/534522 (owner: 10Jforrester) [17:06:51] Yippee, build fixed! [17:06:51] Project beta-scap-eqiad build #266246: 09FIXED in 2 min 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/266246/ [17:07:36] yay [17:15:04] 10Diffusion, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201907), 10Operations, and 4 others: Cannot connect to vcs@git-ssh.wikimedia.org (since move from phab1001 to phab1003) - https://phabricator.wikimedia.org/T224677 (10mmodell) Thank you @MoritzMuehlenhoff for y... [17:32:05] (03PS1) 10Daimona Eaytoy: Also allow @phan-template as alias of @template [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/535241 [17:32:42] (03PS2) 10Daimona Eaytoy: Also allow @phan-template as alias of @template [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/535241 (https://phabricator.wikimedia.org/T232256) [17:40:27] (03CR) 10Thcipriani: [C: 03+2] make-deployment-calendar: Create wikitech:Deployments [tools/release] - 10https://gerrit.wikimedia.org/r/530006 (https://phabricator.wikimedia.org/T114488) (owner: 10Thcipriani) [17:42:44] (03Merged) 10jenkins-bot: make-deployment-calendar: Create wikitech:Deployments [tools/release] - 10https://gerrit.wikimedia.org/r/530006 (https://phabricator.wikimedia.org/T114488) (owner: 10Thcipriani) [17:44:33] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [17:49:48] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Zuul, 10Upstream: Zuul: Implement support for customizing status_url to include the change.id - https://phabricator.wikimedia.org/T65744 (10Krinkle) [17:50:34] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Zuul, 10Upstream: Zuul: Implement support for customizing status_url to include the change.id - https://phabricator.wikimedia.org/T65744 (10Krinkle) I've done some work on the Zuul status page again recently. If we can get this field wo... [17:55:23] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-jhuneidi is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 hphp_invoke - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 353 bytes in 0.041 second response time [18:11:02] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [18:30:30] 10Continuous-Integration-Config, 10Code-Health-Metrics, 10Code-Health: Error in code coverage reports for extensions in codehealth-pipeline - https://phabricator.wikimedia.org/T232195 (10thcipriani) >>! In T232195#5474581, @kostajh wrote: > Ah, it's only failing for extensions which have no unit tests, and w... [18:34:36] Project beta-code-update-eqiad build #263039: 04FAILURE in 1 min 36 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/263039/ [18:44:35] Yippee, build fixed! [18:44:36] Project beta-code-update-eqiad build #263040: 09FIXED in 1 min 34 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/263040/ [18:44:57] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [19:27:26] hiya [19:27:40] it looks like deployment-puppetmaster03 is having trouble updating via git sync upstream [19:27:47] error: could not apply 8c6d2feb... LOCAL HACK: Add puppetdb passwords [19:30:51] ottomata, hmm, I did just sort out the operations/puppet repo there [19:30:55] didn't look at labs/private yet [19:31:05] behind 44 commits :( [19:31:11] ahhh ok [19:31:11] cool [19:31:19] i thought it was from the ops/puppet one, i see my commit there now [19:31:52] (03PS9) 10Hashar: Diff declared deps in CI and in registration json [integration/config] - 10https://gerrit.wikimedia.org/r/504437 [19:32:07] ottomata, fixed [19:32:27] danke [19:37:41] (03CR) 10Hashar: "> Do we want to try to wire this into the repo tests?" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/504437 (owner: 10Hashar) [19:47:36] 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO, 10Code-Stewardship-Reviews: Code Stewardship Review: Collection Extension - https://phabricator.wikimedia.org/T224922 (10Steelpillow) I would endorse the comments by '''Tgr''' so far as they go, but it is important to remember that ma... [20:02:34] 10Phabricator (Upstream), 10Upstream: Allow Herald to remove flags (under "Maniphest Tasks > Personal") - https://phabricator.wikimedia.org/T231623 (10epriestley) Filed upstream as . Resolved upstream by . [20:09:37] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Patch-For-Review: Upgrade to Gerrit 2.16.11.1 - https://phabricator.wikimedia.org/T200739 (10hashar) Gerrit 2.15 will reach its end of life at the next Gerrit hackathon in November 2019. Announcement: https://w... [20:11:04] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [20:15:18] 10Continuous-Integration-Infrastructure, 10DataValues, 10Wikidata, 10Patch-For-Review, 10Wikidata-Campsite (Wikidata-Campsite-Iteration-∞): RuntimeException breaking CI builds - https://phabricator.wikimedia.org/T232063 (10hashar) Fixed by https://github.com/DataValues/DataValues/commit/35fbe96d0843f9abd... [20:19:07] 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.32-release, and 5 others: install.php --with-extensions silently ignores extensions whose dependencies are not satisfied - https://phabricator.wikimedia.org/T225512 (10hashar) 05Open... [20:37:45] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<30.00%) [20:45:10] 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.32-release, and 5 others: install.php --with-extensions silently ignores extensions whose dependencies are not satisfied - https://phabricator.wikimedia.org/T225512 (10hashar) And I wr... [20:49:03] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [21:22:27] (03PS2) 10Brennen Bearnes: mediawiki: replace with blubber-compatible Apache + FPM image [releng/dev-images] - 10https://gerrit.wikimedia.org/r/525842 (https://phabricator.wikimedia.org/T222494) [21:23:25] CI heavily backlogged since about 16:00 -- i.e. about 5.5 hours now :( [21:23:58] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [21:41:19] this is even worse than what has become the 'usual' lately :/ [21:42:16] some test prio jobs queued for over half an hour [21:42:26] sorry, not 'test-prio', 'test' [21:48:23] hrm, there are available executors afaict. [21:51:39] yeah, 40 and 50 are empty [21:56:01] nothing looks suspect in the zuul log, it's still processing changes. There is a bit of a backup on jenkins...which doesn't usually happen since we don't rely on the scheduler there very often. [21:59:11] 22:56:02 A dependency error was encountered while installing the extension "MinervaNeue": Could not find the registration file for the extension "MobileFrontend" [21:59:11] 22:56:02 [12387160f4db169a2121e2f1] [no req] Wikimedia\Services\ServiceDisabledException from line 423 of /workspace/src/includes/libs/services/ServiceContainer.php: Service disabled: DBLoadBalancerFactory [21:59:11] 22:56:02 Backtrace: [21:59:58] http://tyler.zone/waiting-on-executor.png looks suspicious wrt jenkins backup... [22:04:34] 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.32-release, and 5 others: install.php --with-extensions silently ignores extensions whose dependencies are not satisfied - https://phabricator.wikimedia.org/T225512 (10Reedy) I don't t... [22:05:17] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [22:10:38] cdanis: catching up now [22:29:16] James_F: What's wrong with having to two different distinct dependancy lists!? [22:29:38] Reedy: Indeed. :-( [23:01:22] 10Continuous-Integration-Config: A dependency error in CI tests - https://phabricator.wikimedia.org/T232413 (10Zoranzoki21) [23:14:23] 10Continuous-Integration-Config, 10Documentation: doc.wikimedia.org contains empties on end of page - https://phabricator.wikimedia.org/T231421 (10Zoranzoki21) 05Open→03Declined Understandably, I withdraw then this request.