[00:37:18] PROBLEM - Puppet run on deployment-eventlogging04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [00:54:14] PROBLEM - Puppet run on deployment-ores-redis is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [01:12:19] RECOVERY - Puppet run on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [01:34:15] RECOVERY - Puppet run on deployment-ores-redis is OK: OK: Less than 1.00% above the threshold [0.0] [02:38:21] PROBLEM - Puppet run on deployment-eventlogging04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [03:13:18] RECOVERY - Puppet run on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [04:17:55] Yippee, build fixed! [04:17:55] Project selenium-MultimediaViewer » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #131: 09FIXED in 21 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/131/ [04:34:29] PROBLEM - Puppet staleness on deployment-salt02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [06:25:14] PROBLEM - Puppet run on deployment-ores-redis is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [07:00:13] RECOVERY - Puppet run on deployment-ores-redis is OK: OK: Less than 1.00% above the threshold [0.0] [08:50:04] Project performance-webpagetest-wpt-org build #1869: 04FAILURE in 18 min: https://integration.wikimedia.org/ci/job/performance-webpagetest-wpt-org/1869/ [10:52:33] PROBLEM - Puppet run on deployment-mediawiki02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [11:32:32] RECOVERY - Puppet run on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [12:58:01] Yippee, build fixed! [12:58:02] Project performance-webpagetest-wpt-org build #1870: 09FIXED in 25 min: https://integration.wikimedia.org/ci/job/performance-webpagetest-wpt-org/1870/ [14:49:31] RECOVERY - Puppet staleness on deployment-salt02 is OK: OK: Less than 1.00% above the threshold [3600.0] [15:21:09] legoktm https://github.com/phpenv/phpenv :) [15:21:36] all we need now is to build php in the different versions, ie 5.3, 5.5, 5.6 and 7, 7.1 [15:21:52] and install them then we can get rid of precise [15:23:14] legoktm oh i think there already built [15:23:17] travis ci uses it [15:44:45] Project selenium-MobileFrontend » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #146: 04FAILURE in 22 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/146/ [17:42:03] 10Gerrit, 07Upstream: Provide a Github-like web editor for Gerrit to make small edits to code - https://phabricator.wikimedia.org/T116246#2607686 (10Quiddity) I've added some instructions/reminders at https://www.mediawiki.org/wiki/Gerrit/Navigation#Editing_via_the_web-interface I'm not sure if that's the best... [19:00:38] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:09:49] PROBLEM - puppet last run on scandium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:13:43] can someone contact ops? [19:13:49] we got a lot of puppet issues [19:13:51] yeah [19:13:53] Luke081515: they are on it [19:14:00] thx :) [19:14:09] Luke081515: at least a couple euro ops are looking at it [19:14:23] ok :) [19:14:37] I just noticed, that -operations gets constantly flooded [19:15:54] hashar: maybe somebody can quiet icinga meanwhile? [19:16:11] so we can use operations at least for some communitcation :) [19:18:02] Luke081515: yeah maybe. Though that is currently done via the back channel [19:18:11] (private irc channel to deal with infra issues) [19:18:16] hm, ok [19:18:25] at least then the other 200 users don't get flooded [19:18:28] ;) [19:18:31] not sure how one can quite the IRC bot. I am afraid it will spam a bit [19:18:33] :D [19:18:49] a) op and set /mode +b icinga-wm!*@* [19:19:01] b) disable the tool running it / the host [19:23:33] PROBLEM - Puppet run on deployment-mediawiki02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [19:24:28] Luke081515, it's just typical - none of them are around now [19:24:34] :O [19:24:58] And wm-bot isn't op in there, so I can't do anything. [19:37:57] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:39:38] 10Continuous-Integration-Config, 13Patch-For-Review: Add "fail-archived-repositories" to commits to mediawiki/extensions/ApiSandbox in Gerrit - https://phabricator.wikimedia.org/T127012#2607751 (10MarcoAurelio) a:05MarcoAurelio>03None I don't think I should be the assignee. I can't decide on this one. [19:40:29] hashar hi, oh your on tonight? [19:40:31] :) [19:46:31] 10Continuous-Integration-Config, 10Tool-Labs-tools-stewardbots, 13Patch-For-Review: Implement jenkins tests on labs/tools/stewardbots - https://phabricator.wikimedia.org/T128503#2607759 (10MarcoAurelio) Is it possible to expand the tests to gate-and-submit (I think they work already, not sure) and postmerge?... [20:03:33] RECOVERY - Puppet run on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:11:45] Luke081515: it is all back on prod [20:12:31] tom29739: thanks for showing up :) ops have another IRC channel to chat when the -operations one is spammed so not a big deal [20:12:35] hashar, thx :) [20:12:36] paladox: only went to check the puppet spam [20:12:45] havent done anything myself [20:12:51] Oh ok [20:12:54] beside watching / reporting about a few metrics [20:13:03] Didnt notice any puppet spam until now [20:13:17] anyway solved and it is sleep time for me :D [20:13:19] It has been doing puppet fails there and here again today but not as much as now [20:13:20] * hashar waves [20:13:42] PROBLEM - Puppet run on deployment-mathoid is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:53:42] RECOVERY - Puppet run on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [20:58:08] 10Gerrit, 07Upstream: Gerrit's new side-by-side diff screen sometimes cuts off the last few characters of a line - https://phabricator.wikimedia.org/T144565#2607796 (10Paladox) This seems to be fixed in polygerrit. Gerrit's new ui. Seee https://go-review.googlesource.com/?polygerrit=1 and https://go-review.goo... [21:14:42] PROBLEM - Puppet run on deployment-mathoid is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:27:59] 10Beta-Cluster-Infrastructure, 07Puppet: Puppet failing on deployment-conf03 due to missing files - https://phabricator.wikimedia.org/T144703#2607811 (10AlexMonk-WMF) [21:29:41] RECOVERY - Puppet run on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0]