[00:06:43] PROBLEM - Puppet run on deployment-ms-fe01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [00:20:34] 06Release-Engineering-Team, 15User-greg: Create RelEng FY1617Q1 QR slides (due 10/14) - https://phabricator.wikimedia.org/T146515#2663812 (10greg) [00:21:08] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:21:19] 06Release-Engineering-Team, 15User-greg: Update RelEng skillmatrix (due 10/13) - https://phabricator.wikimedia.org/T146516#2663826 (10greg) [00:22:08] 06Release-Engineering-Team, 15User-greg: Compile time spent graphs for FY1617Q1 - https://phabricator.wikimedia.org/T146517#2663840 (10greg) [00:22:40] 06Release-Engineering-Team, 15User-greg: Gather KPI data for FY1617Q1 - https://phabricator.wikimedia.org/T146518#2663854 (10greg) [00:23:48] 06Release-Engineering-Team, 15User-greg: Gather workflow metrics for FY1617Q1 - https://phabricator.wikimedia.org/T146519#2663868 (10greg) [00:25:41] who was idoine in here earlier? [00:26:00] :) [00:26:06] a sneaky lurker [00:26:33] I give you one guess [00:27:13] LOL [00:27:24] legoktm hashar :) [00:27:36] man, ruin the fun, I wanted him to guess :) [00:28:17] heh [00:28:22] oh sorry [00:33:13] RECOVERY - Puppet run on deployment-ores-redis is OK: OK: Less than 1.00% above the threshold [0.0] [00:41:44] RECOVERY - Puppet run on deployment-ms-fe01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:50:33] RECOVERY - Puppet run on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [02:00:18] PROBLEM - Puppet run on deployment-restbase02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [02:15:18] RECOVERY - Puppet run on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [02:15:29] PROBLEM - Puppet run on deployment-cache-text04 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [02:55:29] RECOVERY - Puppet run on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [03:26:58] PROBLEM - Puppet run on deployment-eventlogging03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [05:12:41] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Build an apt repository on deployment-prep (for testing packages from jenkins) - https://phabricator.wikimedia.org/T146497#2664070 (10mmodell) Yuvipanda: cool, thanks! I've got it cherry-picked on beta which works fin... [05:21:07] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Build an apt repository on deployment-prep (for testing packages from jenkins) - https://phabricator.wikimedia.org/T146497#2664071 (10mmodell) [05:26:40] PROBLEM - Puppet staleness on deployment-aqs01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [05:28:52] 06Release-Engineering-Team, 06Developer-Relations, 10Wikimedia-Blog-Content: blog.wikimedia.org post on Phabricator improvements - https://phabricator.wikimedia.org/T141457#2664078 (10mmodell) Thanks @aklapper! I approve of your edits and I second the request to get this published before it's really old news... [06:35:30] PROBLEM - Puppet run on deployment-tmh01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [07:01:08] RECOVERY - Long lived cherry-picks on puppetmaster on deployment-puppetmaster is OK: OK: Less than 100.00% above the threshold [0.0] [07:10:29] RECOVERY - Puppet run on deployment-tmh01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:11:15] 10Continuous-Integration-Config, 10MediaWiki-Unit-tests: All code style CI tests should be run even if some fail - https://phabricator.wikimedia.org/T146445#2664095 (10hashar) May I suggest you run the tests locally? MediaWiki has grunt v1.0.1 and `./node_modules/.bin/grunt --help` has: --force A way to... [08:12:27] PROBLEM - Puppet run on deployment-cache-text04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:52:29] RECOVERY - Puppet run on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [08:52:50] 10Continuous-Integration-Config, 10MediaWiki-Unit-tests: All code style CI tests should be run even if some fail - https://phabricator.wikimedia.org/T146445#2664104 (10Tgr) >>! In T146445#2664095, @hashar wrote: > May I suggest you run the tests locally? I can do that, sure. For a beginner it would probably b... [09:38:22] PROBLEM - Puppet run on deployment-eventlogging04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:46:44] 06Release-Engineering-Team, 06Developer-Relations, 10Wikimedia-Blog-Content: blog.wikimedia.org post on Phabricator improvements - https://phabricator.wikimedia.org/T141457#2664186 (10Qgil) @MelodyKramer is your very helpful default contact with Comms. :) [10:13:19] RECOVERY - Puppet run on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [10:48:48] 10Continuous-Integration-Config, 10MediaWiki-Unit-tests: All code style CI tests should be run even if some fail - https://phabricator.wikimedia.org/T146445#2664208 (10Paladox) In package.json we have the script then test in there grunt test, do grunt test --force [11:24:13] PROBLEM - Puppet run on deployment-ores-redis is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [12:04:13] RECOVERY - Puppet run on deployment-ores-redis is OK: OK: Less than 1.00% above the threshold [0.0] [14:55:43] PROBLEM - Puppet run on deployment-mx is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:30:45] RECOVERY - Puppet run on deployment-mx is OK: OK: Less than 1.00% above the threshold [0.0] [15:32:55] PROBLEM - Puppet run on deployment-db03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:46:48] Yippee, build fixed! [15:46:48] Project selenium-MobileFrontend ยป firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #169: 09FIXED in 24 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/169/ [16:12:52] RECOVERY - Puppet run on deployment-db03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:25:35] twentyafterfour, so is this aptly repo only to be used for things via jenkins or can we add additional packages? [16:32:56] PROBLEM - Keyholder status on deployment-tin02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:11:50] 10Continuous-Integration-Config, 10MediaWiki-Unit-tests: All code style CI tests should be run even if some fail - https://phabricator.wikimedia.org/T146445#2664491 (10hashar) For mediawiki/core there is also a `grunt watch`. I can imagine it is meant to run the tests whenever file change but it does not quite... [17:18:15] Krenair: sure you can add more packages [17:18:25] but you'll need to sudo to the jenkins-deploy user to do it [17:20:20] ok [18:03:54] PROBLEM - Puppet run on deployment-db03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:25:27] 10Gerrit, 07Upstream: Droping down and selcting username from past inputs through keyboard is not functional - https://phabricator.wikimedia.org/T146541#2664583 (10AlexMonk-WMF) [18:53:00] PROBLEM - Puppet run on deployment-eventlogging03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:23:59] 10Beta-Cluster-Infrastructure: New wiki cluster wikipedia indonesian language - https://phabricator.wikimedia.org/T143557#2664696 (10Aklapper) 05stalled>03declined No reply by @Mbrt to the two requests here, hence unfortunately closing this report, as no further information has been provided. @Mbrt: Please... [19:32:59] RECOVERY - Puppet run on deployment-eventlogging03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:33:49] RECOVERY - Puppet run on integration-slave-trusty-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [19:47:36] PROBLEM - Puppet run on deployment-parsoid09 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [19:49:54] PROBLEM - Puppet run on deployment-db03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [19:50:34] PROBLEM - Puppet run on deployment-db04 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [19:51:01] 10Continuous-Integration-Config, 10MediaWiki-Unit-tests: All code style CI tests should be run even if some fail - https://phabricator.wikimedia.org/T146445#2664736 (10Tgr) `watch` probably uses enotif of unixlikes, but that probably does not work via NFS or whatever vagrant is configured to use for file shari... [19:59:55] RECOVERY - Puppet run on deployment-db03 is OK: OK: Less than 1.00% above the threshold [0.0] [20:00:40] !log beta: dropping deployment-tin (ubuntu) replaced by deployment-tin02 (jessie). Primary is still deployment-mira (https://gerrit.wikimedia.org/r/#/c/312654/ T144578 ) [20:00:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:02:33] PROBLEM - Puppet run on deployment-mira is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:02:48] !log deployment-mira: ssh-keyscan deployment-tin02.deployment-prep.eqiad.wmflabs >> /etc/ssh/ssh_known_hosts [20:02:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:05:36] RECOVERY - Puppet run on deployment-db04 is OK: OK: Less than 1.00% above the threshold [0.0] [20:07:42] PROBLEM - Puppet run on deployment-cache-upload04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [20:07:58] PROBLEM - Puppet run on deployment-tin02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:08:10] PROBLEM - SSH on deployment-tin is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:08:53] !log deployment-tin is shutdown. Replaced by Jessie deployment-tin02 [20:08:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:09:32] PROBLEM - Host deployment-tin is DOWN: CRITICAL - Host Unreachable (10.68.17.240) [20:22:35] RECOVERY - Puppet run on deployment-parsoid09 is OK: OK: Less than 1.00% above the threshold [0.0] [21:24:36] PROBLEM - Puppet run on deployment-kafka05 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [22:04:36] RECOVERY - Puppet run on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0] [22:05:53] wmf-insecte: ping [22:05:53] hashar you may not issue bot commands in this chat! [22:06:56] wmf-insecte: help [22:06:56] hashar you may not issue bot commands in this chat! [22:06:57] hashar irc is having a denial attack [22:07:09] paladox: yeah that is recurring :( [22:07:15] Yep [22:07:17] I am not sure why folks hit freenode really [22:07:23] Yep [22:08:22] anyway gotta sleep :D [22:08:56] Ok [22:08:57] bye [22:55:37] PROBLEM - Puppet run on deployment-kafka05 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [23:30:38] RECOVERY - Puppet run on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0] [23:31:12] PROBLEM - Free space - all mounts on deployment-cache-upload04 is CRITICAL: CRITICAL: deployment-prep.deployment-cache-upload04.diskspace._srv_vdb.byte_percentfree (<44.44%)