[00:05:15] PROBLEM - Puppet errors on integration-slave-docker-1007 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [00:09:42] PROBLEM - Puppet errors on integration-slave-docker-1005 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [00:10:29] PROBLEM - Puppet errors on deployment-memc06 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [00:18:32] PROBLEM - Puppet errors on integration-slave-docker-1003 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [00:19:08] PROBLEM - Puppet errors on integration-slave-jessie-1004 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [00:20:21] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [00:20:59] PROBLEM - Puppet errors on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [00:21:29] Hmm [00:21:33] puppet errors [00:24:13] PROBLEM - Puppet errors on integration-slave-docker-1002 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [00:24:53] ^^ that's due https://gerrit.wikimedia.org/r/#/c/403730/ [00:25:44] PROBLEM - Puppet errors on integration-slave-jessie-1003 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [00:26:18] PROBLEM - Puppet errors on integration-slave-docker-1006 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [00:32:15] PROBLEM - Puppet errors on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [00:33:21] PROBLEM - Puppet errors on integration-slave-docker-1004 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [00:33:45] mutante ^^ [00:36:10] Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Resource Statement, Duplicate declaration: File[/usr/local/bin/apache-status] is already declared in file /etc/puppet/modules/httpd/manifests/init.pp:87; cannot redeclare at /etc/puppet/modules/apache/manifests/monitoring.pp:23 at /etc/puppet/ [00:36:10] modules/apache/manifests/monitoring.pp:23:5 at /etc/puppet/modules/contint/manifests/localvhost.pp:23 on node integration-slave-docker-1004.integration.eqiad.wmflabs [00:36:20] Reedy [00:36:20] yep [00:36:22] known [00:36:26] i've reverted the change [00:36:34] I was posting the actual error [00:36:49] oh i see [00:36:49] ok [00:45:33] RECOVERY - Puppet errors on deployment-memc06 is OK: OK: Less than 1.00% above the threshold [0.0] [01:04:31] PROBLEM - Free space - all mounts on deployment-kafka-jumbo-1 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka-jumbo-1.diskspace.root.byte_percentfree (<100.00%) [01:10:17] RECOVERY - Puppet errors on integration-slave-docker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [01:14:40] RECOVERY - Puppet errors on integration-slave-docker-1005 is OK: OK: Less than 1.00% above the threshold [0.0] [01:28:32] RECOVERY - Puppet errors on integration-slave-docker-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [01:29:08] RECOVERY - Puppet errors on integration-slave-jessie-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [01:29:12] RECOVERY - Puppet errors on integration-slave-docker-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [01:30:22] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [01:31:00] RECOVERY - Puppet errors on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [01:31:16] RECOVERY - Puppet errors on integration-slave-docker-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [01:35:45] RECOVERY - Puppet errors on integration-slave-jessie-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [01:37:15] RECOVERY - Puppet errors on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [01:38:23] RECOVERY - Puppet errors on integration-slave-docker-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [02:34:28] 10Phabricator, 10Release-Engineering-Team, 10Operations: Consider ssd's for phabricator - https://phabricator.wikimedia.org/T185796#3925461 (10Paladox) [05:11:40] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<55.56%) [07:01:39] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:35:25] PROBLEM - Free space - all mounts on integration-slave-jessie-1001 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1001.diskspace._srv.byte_percentfree (<11.11%) [11:52:25] 10Scap, 10User-Ladsgroup: Port scap tests to pytest - https://phabricator.wikimedia.org/T179885#3925574 (10Ladsgroup) 05Open>03Resolved It's done I think [12:32:30] PROBLEM - puppet last run on contint2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:02:30] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [14:01:17] 10Continuous-Integration-Config, 10Pywikibot-core: Jenkins output for pywikibot job is hard to read - https://phabricator.wikimedia.org/T117570#3925621 (10Xqt) p:05Triage>03Lowest [14:25:25] PROBLEM - Free space - all mounts on integration-slave-jessie-1001 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1001.diskspace._srv.byte_percentfree (<11.11%) [14:51:18] (03CR) 10Zoranzoki21: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/256413 (owner: 10Hashar) [14:52:28] (03CR) 10jerkins-bot: [V: 04-1] Integration tests with make-wmf-branch [integration/config] - 10https://gerrit.wikimedia.org/r/256413 (owner: 10Hashar) [15:27:44] PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:33:43] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:41:31] 10Phabricator, 10Zero: Disallow uploading unwanted files to Phabricator (WP Zero abuse?) - https://phabricator.wikimedia.org/T185784#3925259 (10Zoranzoki21) One option: Disable option for uploading. And all ok. [16:11:31] (03PS3) 10Zoranzoki21: Whitelist few users in CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/406242 [16:24:40] PROBLEM - Free space - all mounts on integration-slave-jessie-1003 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1003.diskspace._srv.byte_percentfree (<100.00%) [16:38:21] 10Phabricator: Phabricator account deletion request - https://phabricator.wikimedia.org/T185703#3925686 (10TheFanoulis) Mhm, I see. Also, how did a deletion task win so many subs 😂 [16:39:08] (03CR) 10Zoranzoki21: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/405828 (owner: 10MacFan4000) [16:39:13] (03CR) 10Zoranzoki21: "check experimental" [integration/config] - 10https://gerrit.wikimedia.org/r/405828 (owner: 10MacFan4000) [16:40:10] (03CR) 10jerkins-bot: [V: 04-1] Change linting types on ZppixBot repo [integration/config] - 10https://gerrit.wikimedia.org/r/405828 (owner: 10MacFan4000) [16:45:03] (03CR) 10MacFan4000: "Not sure how I can fix that" [integration/config] - 10https://gerrit.wikimedia.org/r/405828 (owner: 10MacFan4000) [16:47:41] (03PS4) 10Zoranzoki21: Whitelist few users in CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/406242 [18:05:25] PROBLEM - Free space - all mounts on integration-slave-jessie-1001 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1001.diskspace._srv.byte_percentfree (<22.22%) [18:39:23] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:17:02] HI, I have few tasks for you [19:17:21] First abandon all changes from this user: https://gerrit.wikimedia.org/r/#/q/owner:MigDinny [21:40:08] Project mwext-phpunit-coverage-publish build #316: 04FAILURE in 13 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/316/ [21:41:02] Yippee, build fixed! [21:41:02] Project mwext-phpunit-coverage-publish build #317: 09FIXED in 53 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/317/ [21:41:41] Project mwext-phpunit-coverage-publish build #319: 04FAILURE in 6.5 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/319/ [21:44:11] Project mwext-phpunit-coverage-publish build #320: 04STILL FAILING in 2 min 29 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/320/ [21:44:12] Project mwext-phpunit-coverage-publish build #321: 04STILL FAILING in 0.18 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/321/ [21:44:37] Yippee, build fixed! [21:44:38] Project mwext-phpunit-coverage-publish build #322: 09FIXED in 25 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/322/ [21:45:53] no space left ^^ [21:45:56] legoktm ^^ [21:46:17] we should create it with larger space :)