[00:31:17] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [02:16:44] 10Continuous-Integration-Infrastructure: integration-slave-jessie-1002 out of disk - https://phabricator.wikimedia.org/T185821#3926027 (10Paladox) [02:16:51] 10Continuous-Integration-Infrastructure: integration-slave-jessie-1002 out of disk - https://phabricator.wikimedia.org/T185821#3926037 (10Paladox) p:05Triage>03Unbreak! [02:47:28] 10Gerrit, 10Release-Engineering-Team, 10Operations: Add prometheus exporter to Gerrit - https://phabricator.wikimedia.org/T184086#3926059 (10Paladox) @demon do you want to add this plugin in your 2.14 patch or will you do a follow up adding this? [02:49:44] 10Gerrit, 10Patch-For-Review: Replace using certificates with tokens when using its-phabricator - https://phabricator.wikimedia.org/T178385#3926060 (10Paladox) @chad this needs to be done at the same time of the upgrade (replace certificates in the private repo or where you had it :)) [02:54:30] 10Gerrit, 10Gerrit-Migration: Provide static dump of Gerrit - https://phabricator.wikimedia.org/T617#3926061 (10Paladox) p:05Normal>03Lowest [03:00:09] Project mediawiki-core-code-coverage-php7 build #50: 04FAILURE in 9.4 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/50/ [04:57:40] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<11.11%) [05:12:32] PROBLEM - Puppet errors on deployment-memc06 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [05:47:30] RECOVERY - Puppet errors on deployment-memc06 is OK: OK: Less than 1.00% above the threshold [0.0] [05:55:24] PROBLEM - Free space - all mounts on integration-slave-jessie-1001 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1001.diskspace._srv.byte_percentfree (<11.11%) [06:31:49] 10Continuous-Integration-Infrastructure: integration-slave-jessie-1002 out of disk - https://phabricator.wikimedia.org/T185821#3926027 (10greg) ``` gjg@integration-slave-jessie-1002:/srv/jenkins-workspace/workspace$ du -h -d 1 135M ./apps-android-wikipedia-lint 31M ./mwgate-jsduck 97M ./operations-mw-config-typo... [07:02:20] 10Continuous-Integration-Infrastructure: integration-slave-jessie-1002 out of disk - https://phabricator.wikimedia.org/T185821#3926122 (10TerraCodes) [07:02:38] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:12:44] RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [07:18:43] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [07:35:38] PROBLEM - Free space - all mounts on integration-slave-jessie-1002 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1002.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1002.diskspace._srv.byte_percentfree (<100.00%) [07:43:44] RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [07:49:45] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:14:43] RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [10:54:38] Jenkins as always is not normal [10:54:44] Example: https://integration.wikimedia.org/ci/job/operations-mw-config-typos/15597/console [11:20:25] Help please [11:27:31] 10Continuous-Integration-Config, 10Jenkins, 10Zuul: Jenkis fail with reason to have problems with using git - https://phabricator.wikimedia.org/T185831#3926264 (10Zoranzoki21) [11:27:53] 10Continuous-Integration-Config, 10Jenkins, 10Zuul: Jenkis fail with reason to have problems with using git - https://phabricator.wikimedia.org/T185831#3926266 (10TerraCodes) [11:27:55] 10Continuous-Integration-Infrastructure: integration-slave-jessie-1002 out of disk - https://phabricator.wikimedia.org/T185821#3926269 (10TerraCodes) [11:28:46] 10Continuous-Integration-Infrastructure: integration-slave-jessie-1002 out of disk - https://phabricator.wikimedia.org/T185821#3926270 (10Zoranzoki21) OMG [11:32:23] 10Continuous-Integration-Infrastructure: integration-slave-jessie-1002 out of disk - https://phabricator.wikimedia.org/T185821#3926273 (10TerraCodes) This is causing a lot of tests to fail. [11:34:19] 10Continuous-Integration-Infrastructure: integration-slave-jessie-1002 out of disk - https://phabricator.wikimedia.org/T185821#3926274 (10Zoranzoki21) This should be resolved until tommorow first SWAT to Jenkins no made more problems. [12:51:40] Reedy: contint seems to have disks full again? [12:51:42] https://integration.wikimedia.org/ci/job/operations-mw-config-php55lint/17923/console [13:32:54] 10Continuous-Integration-Infrastructure: integration-slave-jessie-1002 out of disk - https://phabricator.wikimedia.org/T185821#3926436 (10Zoranzoki21) >>! In T185821#3926385, @MarcoAurelio wrote: > I guess the failure at https://gerrit.wikimedia.org/r/#/c/406474/ is also related to this one? @MarcoAurelio I thi... [13:46:28] 10Continuous-Integration-Infrastructure, 10User-Jayprakash12345: Error cloning remote repo 'origin' in CI - https://phabricator.wikimedia.org/T185839#3926455 (10Jayprakash12345) [13:46:46] 10Continuous-Integration-Infrastructure: Error cloning remote repo 'origin' in CI - https://phabricator.wikimedia.org/T185839#3926466 (10Jayprakash12345) [13:49:40] 10Continuous-Integration-Infrastructure: integration-slave-jessie-1002 out of disk - https://phabricator.wikimedia.org/T185821#3926467 (10Zoranzoki21) 05Open>03Resolved Now work without problems. I think to this can be closed as resolved. [13:52:53] 10Continuous-Integration-Infrastructure: Error cloning remote repo 'origin' in CI - https://phabricator.wikimedia.org/T185839#3926473 (10Zoranzoki21) 05Open>03Resolved Minor bug. All is ok now. [13:56:02] 10Continuous-Integration-Infrastructure: integration-slave-jessie-1002 out of disk - https://phabricator.wikimedia.org/T185821#3926481 (10TerraCodes) 05Resolved>03Open Its not fixed. [13:59:44] 10Continuous-Integration-Infrastructure: integration-slave-jessie-1002 out of disk - https://phabricator.wikimedia.org/T185821#3926484 (10Zoranzoki21) 05Open>03Resolved >>! In T185821#3926481, @TerraCodes wrote: > Its not fixed. I runned recheck on few patches where is it problem and now he no show this err... [14:02:14] 10Continuous-Integration-Infrastructure: integration-slave-jessie-1002 out of disk - https://phabricator.wikimedia.org/T185821#3926489 (10TerraCodes) 05Resolved>03Open Nope, its not resolved. See https://integration.wikimedia.org/ci/job/operations-mw-config-typos/15607/console and https://integration.wikimed... [14:06:28] 10Continuous-Integration-Infrastructure: Error cloning remote repo 'origin' in CI - https://phabricator.wikimedia.org/T185839#3926492 (10Zoranzoki21) 05Resolved>03Open @TerraCodes Problems with your patches are related to this task. Opening because is not resolved, [14:06:54] 10Continuous-Integration-Infrastructure: integration-slave-jessie-1002 out of disk - https://phabricator.wikimedia.org/T185821#3926495 (10Zoranzoki21) 05Open>03Resolved >>! In T185821#3926489, @TerraCodes wrote: > Nope, its not resolved. See https://integration.wikimedia.org/ci/job/operations-mw-config-typos... [14:08:50] 10Continuous-Integration-Infrastructure: integration-slave-jessie-1002 out of disk - https://phabricator.wikimedia.org/T185821#3926500 (10TerraCodes) [14:08:52] 10Continuous-Integration-Infrastructure: Error cloning remote repo 'origin' in CI - https://phabricator.wikimedia.org/T185839#3926502 (10TerraCodes) [14:09:27] 10Continuous-Integration-Infrastructure: integration-slave-jessie-1002 out of disk - https://phabricator.wikimedia.org/T185821#3926027 (10TerraCodes) 05Resolved>03Open T185839 is due to it being out of space as said in the task description. [14:11:22] 10Continuous-Integration-Infrastructure: integration-slave-jessie-1002 out of disk - https://phabricator.wikimedia.org/T185821#3926505 (10Jayprakash12345) @Zoranzoki21, Don't close until problem should be permanently fixed. [14:12:37] 10Continuous-Integration-Infrastructure: integration-slave-jessie-1002 out of disk - https://phabricator.wikimedia.org/T185821#3926506 (10Zoranzoki21) >>! In T185821#3926505, @Jayprakash12345 wrote: > @Zoranzoki21, Don't close until problem should be permanently fixed. Ok [14:53:17] PROBLEM - Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) [15:01:11] (03CR) 10Umherirrender: Whitelist few users in CI whitelist (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/406242 (owner: 10Zoranzoki21) [15:48:53] Yippee, build fixed! [15:48:53] Project mediawiki-core-code-coverage-php7 build #51: 09FIXED in 48 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/51/ [16:08:23] (03Draft1) 10MarcoAurelio: Archive extension-CommunityApplications [integration/config] - 10https://gerrit.wikimedia.org/r/406501 [16:08:25] (03PS2) 10MarcoAurelio: Archive extension-CommunityApplications [integration/config] - 10https://gerrit.wikimedia.org/r/406501 [16:09:08] (03PS3) 10MarcoAurelio: Archive extension-CommunityApplications [integration/config] - 10https://gerrit.wikimedia.org/r/406501 (https://phabricator.wikimedia.org/T185844) [16:16:19] Project mwext-phpunit-coverage-publish build #326: 04FAILURE in 0.27 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/326/ [16:33:18] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:44:59] (03Draft1) 10MarcoAurelio: Archive extension-CommunityHiring [integration/config] - 10https://gerrit.wikimedia.org/r/406511 (https://phabricator.wikimedia.org/T185845) [16:45:04] (03PS2) 10MarcoAurelio: Archive extension-CommunityHiring [integration/config] - 10https://gerrit.wikimedia.org/r/406511 (https://phabricator.wikimedia.org/T185845) [16:46:20] (03CR) 10Zoranzoki21: [C: 031] Archive extension-CommunityHiring [integration/config] - 10https://gerrit.wikimedia.org/r/406511 (https://phabricator.wikimedia.org/T185845) (owner: 10MarcoAurelio) [17:28:50] PROBLEM - Puppet errors on deployment-snapshot01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:00:27] PROBLEM - Free space - all mounts on integration-slave-jessie-1001 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1001.diskspace._srv.byte_percentfree (<22.22%) [18:13:41] Yippee, build fixed! [18:13:41] Project mwext-phpunit-coverage-publish build #327: 09FIXED in 40 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/327/ [18:15:43] (03CR) 10Umherirrender: [C: 031] Archive extension-CommunityApplications [integration/config] - 10https://gerrit.wikimedia.org/r/406501 (https://phabricator.wikimedia.org/T185844) (owner: 10MarcoAurelio) [18:15:57] (03CR) 10Umherirrender: [C: 031] Archive extension-CommunityHiring [integration/config] - 10https://gerrit.wikimedia.org/r/406511 (https://phabricator.wikimedia.org/T185845) (owner: 10MarcoAurelio) [18:25:39] RECOVERY - Free space - all mounts on integration-slave-jessie-1002 is OK: OK: integration.integration-slave-jessie-1002.diskspace._mnt.byte_percentfree (No valid datapoints found) [18:26:37] 10Gerrit: Gerrit rebase button returns: "500 Internal server error" - https://phabricator.wikimedia.org/T185849#3926755 (10MarcoAurelio) [18:34:29] 10Gerrit: Gerrit rebase button returns: "500 Internal server error" - https://phabricator.wikimedia.org/T185849#3926755 (10Paladox) Do you have access to https://logstash.wikimedia.org/ ? [18:35:15] 10Gerrit: Gerrit rebase button returns: "500 Internal server error" - https://phabricator.wikimedia.org/T185849#3926769 (10MarcoAurelio) No, I do not @Paladox; sorry. [18:54:28] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:04:18] (03CR) 10Zoranzoki21: [C: 031] Newsletter: Add Echo as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/401407 (owner: 10Florianschmidtwelzow) [20:19:13] 10Continuous-Integration-Config, 10MediaWiki-extensions-General, 10Google-Code-in-2017, 10Jenkins, 10Patch-For-Review: Add MediaWiki extensions to zuul/layout.yaml in so Jenkins can run builds - https://phabricator.wikimedia.org/T183674#3926856 (10Zoranzoki21) [20:22:55] (03PS1) 10Zoranzoki21: Add MediaWiki extensions to zuul/layout.yaml in so Jenkins can run builds [integration/config] - 10https://gerrit.wikimedia.org/r/406524 (https://phabricator.wikimedia.org/T183674) [20:23:34] 10Continuous-Integration-Config, 10MediaWiki-extensions-General, 10Google-Code-in-2017, 10Jenkins, 10Patch-For-Review: Add MediaWiki extensions to zuul/layout.yaml in so Jenkins can run builds - https://phabricator.wikimedia.org/T183674#3860109 (10Zoranzoki21) >>! In T183674#3926868, @gerritbot wrote: >... [20:23:57] (03CR) 10jerkins-bot: [V: 04-1] Add MediaWiki extensions to zuul/layout.yaml in so Jenkins can run builds [integration/config] - 10https://gerrit.wikimedia.org/r/406524 (https://phabricator.wikimedia.org/T183674) (owner: 10Zoranzoki21) [20:29:32] (03PS2) 10Zoranzoki21: Add MediaWiki extensions to zuul/layout.yaml in so Jenkins can run builds [integration/config] - 10https://gerrit.wikimedia.org/r/406524 (https://phabricator.wikimedia.org/T183674) [20:33:30] (03CR) 10Zoranzoki21: [C: 031] Change linting types on ZppixBot repo [integration/config] - 10https://gerrit.wikimedia.org/r/405828 (owner: 10MacFan4000) [20:33:40] (03CR) 10Zoranzoki21: [C: 031] "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/405828 (owner: 10MacFan4000) [20:34:39] (03PS3) 10Zoranzoki21: Add few extensions to zuul/layout.yaml in so Jenkins can run builds [integration/config] - 10https://gerrit.wikimedia.org/r/406524 (https://phabricator.wikimedia.org/T183674) [20:34:45] (03CR) 10jerkins-bot: [V: 04-1] Change linting types on ZppixBot repo [integration/config] - 10https://gerrit.wikimedia.org/r/405828 (owner: 10MacFan4000) [20:45:18] 10Continuous-Integration-Infrastructure: integration-slave-jessie-1002 out of disk - https://phabricator.wikimedia.org/T185821#3926887 (10TerraCodes) p:05Unbreak!>03High Changing from UBN to high because of that this is no longer making jerkins -1 everything. [21:08:22] PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:11:46] (03Abandoned) 10MacFan4000: Change linting types on ZppixBot repo [integration/config] - 10https://gerrit.wikimedia.org/r/405828 (owner: 10MacFan4000) [21:35:38] 10Phabricator, 10Zero: Disallow uploading unwanted files to Phabricator (WP Zero abuse?) - https://phabricator.wikimedia.org/T185784#3926921 (10Aklapper) @Zoranzoki21: This task is about a //specific// problem. Please see previous work and related tasks first. Generic 'solutions' like disabling uploading is o... [21:36:43] 10Phabricator, 10Zero: Disallow uploading unwanted files to Phabricator (WP Zero abuse?) - https://phabricator.wikimedia.org/T185784#3926922 (10Zoranzoki21) >>! In T185784#3926921, @Aklapper wrote: > @Zoranzoki21: This task is about a //specific// problem. Please see previous work and related tasks first. > G... [21:37:38] (03Draft2) 10MacFan4000: Whitelist MacFan4000s email [integration/config] - 10https://gerrit.wikimedia.org/r/406528 [21:39:54] 10Phabricator, 10Zero: Disallow uploading unwanted files to Phabricator (WP Zero abuse?) - https://phabricator.wikimedia.org/T185784#3926924 (10Aklapper) @Zoranzoki21: Please do read existing and linked tasks first before commenting. What you commented has been discussed already. No need to discuss again. [21:40:51] 10Phabricator, 10Zero: Disallow uploading unwanted files to Phabricator (WP Zero abuse?) - https://phabricator.wikimedia.org/T185784#3926925 (10Zoranzoki21) >>! In T185784#3926924, @Aklapper wrote: > @Zoranzoki21: Please do read existing and linked tasks first before commenting. What you commented has been dis... [22:48:58] When someone gets a chance can i speak to releng member in private? If not, i can email releng's internal email if needed [22:55:05] 10Continuous-Integration-Config, 10Operations, 10Operations-Software-Development: Puppet tox: properly lint both Py2 and Py3 files - https://phabricator.wikimedia.org/T184435#3927000 (10Volans) [23:37:34] PROBLEM - Free space - all mounts on deployment-kafka-jumbo-2 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka-jumbo-2.diskspace.root.byte_percentfree (<100.00%) [23:53:27] hello [23:53:32] it seems to have all fixed itself [23:57:49] Zppix: please email me directly, greg@ wikimedia