[01:53:45] !Ping [01:54:55] {Pöñg} [02:27:50] 6Labs, 10wikitech.wikimedia.org, 7Wikimedia-log-errors: Hook SMWParseData::onTitleMoveComplete has invalid call signature; Parameter 3 to SMWParseData::onTitleMoveComplete() expected to be a reference, value given - https://phabricator.wikimedia.org/T118649#1809975 (10aaron) Removing the & from SMWParseData:... [02:42:51] PROBLEM - Puppet failure on tools-exec-1210 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [02:46:10] 6Labs, 10Tool-Labs, 3labs-sprint-118, 3labs-sprint-119: Enforce that containers from a user run with the uid assigned to that user - https://phabricator.wikimedia.org/T116504#1810005 (10yuvipanda) p:5Low>3High [03:17:52] RECOVERY - Puppet failure on tools-exec-1210 is OK: OK: Less than 1.00% above the threshold [0.0] [04:26:41] 6Labs, 10Tool-Labs: tools-webgrid-lighttpd-1412 refuses ssh connections - https://phabricator.wikimedia.org/T118654#1810040 (10yuvipanda) 5Open>3Resolved I've recreated and repooled it. [04:27:41] 6Labs: Move all instances using role::lamp::labs to role::simplelamp - https://phabricator.wikimedia.org/T118784#1810043 (10yuvipanda) This still needs to make sure that the mounting of mysql in /mnt in the older role by default is dealt with somehow. [06:50:49] PROBLEM - Puppet failure on tools-exec-1216 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:10:31] 6Labs: Move all instances using role::lamp::labs to role::simplelamp - https://phabricator.wikimedia.org/T118784#1810114 (10yuvipanda) a:3yuvipanda [07:16:15] RECOVERY - Puppet failure on tools-precise-dev is OK: OK: Less than 1.00% above the threshold [0.0] [07:25:42] RECOVERY - Puppet failure on tools-exec-1216 is OK: OK: Less than 1.00% above the threshold [0.0] [07:43:09] 6Labs, 5Patch-For-Review: Move all instances using role::lamp::labs to role::simplelamp - https://phabricator.wikimedia.org/T118784#1810159 (10yuvipanda) I'm changing the instances one by one! so far changed: # accouts-db2 # wikidata-unicorn # utrs-primary # map (couldn't ssh, probably too broken to be useful... [08:11:00] 6Labs, 6operations: Write a diamond collector to collect active ssh sessions - https://phabricator.wikimedia.org/T118827#1810194 (10yuvipanda) 3NEW [08:11:08] 6Labs, 6operations: Write a diamond collector to collect active ssh sessions - https://phabricator.wikimedia.org/T118827#1810201 (10yuvipanda) a:3yuvipanda [08:11:18] 6Labs, 10Tool-Labs, 6operations: Write a diamond collector to collect active ssh sessions - https://phabricator.wikimedia.org/T118827#1810194 (10yuvipanda) [08:11:40] 6Labs, 6Phabricator: Archive all the labs-sprint-* tags - https://phabricator.wikimedia.org/T118828#1810204 (10yuvipanda) 3NEW a:3chasemp [08:52:53] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: Unused/broken Special:NovaVolume - https://phabricator.wikimedia.org/T118832#1810265 (10zhuyifei1999) 3NEW [09:01:33] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: Unused/broken Special:NovaVolume - https://phabricator.wikimedia.org/T118832#1810272 (10zhuyifei1999) [09:15:42] 6Labs, 3Labs-Sprint-108, 3Labs-Sprint-109, 3Labs-Sprint-111, 5Patch-For-Review: Simple method to have a per-project debian repository - https://phabricator.wikimedia.org/T104194#1810291 (10Aklapper) > yuvipanda moved this task to Done on the Labs-Sprint-111 workboard. @yuvipanda: Is there more work left... [09:16:37] 6Labs, 10Tool-Labs, 3Labs-Sprint-101, 3Labs-Sprint-102, and 2 others: Puppetize toolserver.org redirect configuration - https://phabricator.wikimedia.org/T85165#1810293 (10Ricordisamoa) [09:17:22] 6Labs, 10Tool-Labs, 7Database, 3labs-sprint-117: tools.citationhunt can't access databases - https://phabricator.wikimedia.org/T109972#1810300 (10Aklapper) > yuvipanda moved this task to Done on the labs-sprint-117 workboard. @yuvipanda: Is there more work left to do here? Or should this task be closed as... [09:20:59] 6Labs, 6Phabricator: Archive all the labs-sprint-* tags - https://phabricator.wikimedia.org/T118828#1810303 (10Aklapper) 5Open>3Resolved I've archived all projects. > They're no longer being used. @yuvipanda: What else is used instead? Where can I find out about plans and progress of the Labs team? https:... [09:21:09] 6Labs, 6Phabricator: Archive all the labs-sprint-* tags - https://phabricator.wikimedia.org/T118828#1810305 (10Aklapper) a:5chasemp>3Aklapper [09:21:58] 6Labs, 6Phabricator: Archive all the labs-sprint-* tags - https://phabricator.wikimedia.org/T118828#1810308 (10yuvipanda) @chasemp can probably explain better :) I think we're just assigning tasks to ourselves and using task assignment only. Thanks for archiving! [12:11:54] 6Labs, 10Tool-Labs: Define base Wikimedia Docker container - https://phabricator.wikimedia.org/T118446#1810607 (10MoritzMuehlenhoff) Excerpt from an IRC discussion earlier the day: [08:14] moritzm: yeah, but then there are a lot of official images too - so question is do we just allow them all (s... [14:39:47] 6Labs, 10Tool-Labs: Provide webservice bigbrotherrc for actively used tools - https://phabricator.wikimedia.org/T90569#1810804 (10coren) [14:39:49] 6Labs, 10Tool-Labs, 10Tool-Labs-tools-Erwin's-tools, 7Monitoring: monitor webservice / 504 errors for erwin - https://phabricator.wikimedia.org/T90800#1810802 (10coren) 5Open>3declined As far as I can tell, the tool has been working properly for several months. In addition, bigbrother is deprecated so... [14:40:21] 6Labs, 10Tool-Labs: Make Flow database available / accessible on Labs/Tools - https://phabricator.wikimedia.org/T69397#1810805 (10coren) a:5coren>3None [14:42:20] 6Labs, 10Tool-Labs, 10Labs-Team-Backlog: Set up A-based SPF for tools.wmflabs.org - https://phabricator.wikimedia.org/T104733#1810808 (10coren) a:5coren>3None This needs to be revisited once the LDAP backend has changed (since the opendj schema does not properly allow TXT records in domain entries) [14:53:14] 6Labs, 10Tool-Labs, 10Labs-Team-Backlog: Set up A-based SPF for tools.wmflabs.org - https://phabricator.wikimedia.org/T104733#1810829 (10valhallasw) What's the ETA for the LDAP backend change? I'm also confused why this is blocking changing the current record -- that record got in there somehow? Keep in mind... [14:58:47] 6Labs, 10Tool-Labs: Determine and deploy proper h_vmem resources for execution nodes - https://phabricator.wikimedia.org/T107665#1810843 (10coren) [14:58:48] 6Labs, 10Tool-Labs: Puppetize gridengine master configuration - https://phabricator.wikimedia.org/T95747#1810844 (10coren) [14:58:49] 6Labs, 10Tool-Labs, 5Patch-For-Review: Puppetize that h_vmem is a consumable resource - https://phabricator.wikimedia.org/T107821#1810841 (10coren) 5Open>3Resolved This was done in `gridengine::master` by virtue of `complex-99-default` setting h_vmem as consumable, but the attached patchset making griden... [15:05:48] 6Labs, 10Tool-Labs: Help a new volunteer dev get his timeline script on Tool Labs - https://phabricator.wikimedia.org/T110557#1810866 (10coren) a:5coren>3None Leaving up for grabs as it's not clear that tools is the proper venue at all. [15:07:29] 6Labs, 7Database: Database replicas: replicate user.user_touched - https://phabricator.wikimedia.org/T92841#1810873 (10coren) a:5Springle>3jcrespo (This was assigned directly to Sean and likely fell between the cracks because of it) [15:21:05] 6Labs, 7Database: Database replicas: replicate user.user_touched - https://phabricator.wikimedia.org/T92841#1810902 (10jcrespo) Do you want me to fill in that column to sanitarium from production (and take out any nullifying process, if any there)? [15:21:32] 6Labs, 6Phabricator: Archive all the labs-sprint-* tags - https://phabricator.wikimedia.org/T118828#1810904 (10chasemp) >>! In T118828#1810303, @Aklapper wrote: > I've archived all projects. > >> They're no longer being used. > @yuvipanda: What else is used instead? Where can I find out about plans and progre... [15:23:05] 6Labs, 7Database: Database replicas: replicate user.user_touched - https://phabricator.wikimedia.org/T92841#1810909 (10coren) @jcrespo: Yes, that's correct - the data in that column (and the views that use it) have been cleared by legal as part of T60196. [15:23:47] 6Labs, 6Phabricator: Archive all the labs-sprint-* tags - https://phabricator.wikimedia.org/T118828#1810913 (10Aklapper) Ah neat. (I hope that board is linked from somewhere™ (wikipage? Phab project desc?) so anyone outside of the Labs team could also find out what's cooking? Cause transparency, you know.) [15:29:14] 6Labs, 10Tool-Labs, 5Patch-For-Review: Redirect Dispenser's tools - https://phabricator.wikimedia.org/T116757#1810937 (10coren) 5Open>3Resolved [16:08:13] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:08:13] ^^ related, being investigated [16:08:55] Coren: can I do anything? [16:14:18] does self hosted puppetmaster not work in deployment-prep? [16:21:45] 10MediaWiki-extensions-OpenStackManager: Unused/broken Special:NovaVolume - https://phabricator.wikimedia.org/T118832#1811082 (10Krenair) [16:31:13] (03PS1) 10Alexandros Kosiaris: Add ssh/mysql/mysql.key secret [labs/private] - 10https://gerrit.wikimedia.org/r/253626 [16:37:58] 6Labs, 7Monitoring, 5Patch-For-Review, 7Shinken, 7Upstream: shinken.wmflabs.org redirects on https-login to http - https://phabricator.wikimedia.org/T85326#1811145 (10Krenair) I don't really know what's going on upstream but it sounds like a load of things have been changed, the patch might not still be... [16:38:36] 6Labs, 5Patch-For-Review, 7Puppet: dynamicproxy: Move list of blocked user agents to hiera - https://phabricator.wikimedia.org/T90844#1811147 (10Krenair) 5Open>3Resolved https://gerrit.wikimedia.org/r/#/c/249182/ [16:40:39] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Add ssh/mysql/mysql.key secret [labs/private] - 10https://gerrit.wikimedia.org/r/253626 (owner: 10Alexandros Kosiaris) [16:41:15] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10Labs-Infrastructure: Can't delete NovaProxy instance with malformed DNS hostname - https://phabricator.wikimedia.org/T69927#1811165 (10Krenair) [16:43:28] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10Labs-Infrastructure: Can't delete NovaProxy instance with malformed DNS hostname - https://phabricator.wikimedia.org/T69927#1811173 (10Krenair) >>! In T69927#1787337, @yuvipanda wrote: > okay, I've built and deployed this, and now we get: > > ```Successfully... [16:43:36] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10Labs-Infrastructure: Can't delete NovaProxy instance with malformed DNS hostname - https://phabricator.wikimedia.org/T69927#1811177 (10Krenair) a:5Krenair>3yuvipanda [19:19:10] andrewbogott: just a fyi, I cleaned up some more of the 'default' roles and puppet variables in wikitech that're available to all instances. [19:19:21] andrewbogott: primary difference is that a whole bunch of salt stuff that most people didn't need is gone [19:19:33] and can be re-added on a per-project basis if needed [19:19:39] same for the mnt lvm [19:19:43] the srv one is still there [19:19:45] That’s great! [19:20:11] andrewbogott: yeah I tried getting rid of the role::labs::instance and realm ones earlier but that broke new instances :( [19:20:22] andrewbogott: since if those are hidden then OSM doesn't actually apply them to new instances [19:20:34] so that's a bit sad since people can kill their instances by modifying those values I guess [19:21:24] yeah, it’s pretty broken. I had a phab task for that somewhere... [19:21:29] but I’d rather move things to hiera than fix it [19:21:42] andrewbogott: ooooh, yeah, maybe we can do that [19:22:12] andrewbogott: switch realm to be read from hiera instead of ldap and include role::labs::instance via hiera too [19:22:44] yep, would be better I think. Or at least another step away from wikitech [19:23:27] andrewbogott: yeah. is there a bug? point me to it and I'll take it up [19:23:46] andrewbogott: I'm in 'goddamit must cleanup all the things' mode. about 45 cleanup commits yesterday mostly removing stuff.. [19:24:00] hm… https://wikitech.wikimedia.org/wiki/Ldap_hosts might be out of date [19:24:03] but it has a roadmap [19:25:31] andrewbogott: yeah I guess needs update in light of the Designate usage now [19:25:58] among others. I’ll look [19:34:29] !log tools blanked 'realm' for tools-bastion-01 to figure out what happens [19:34:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [19:39:11] !log tools created tools-worker-03 to be k8s worker node [19:39:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [19:59:15] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Spacemartin was created, changed by Spacemartin link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Spacemartin edit summary: Created page with "{{Tools Access Request |Justification=I plan to implement a Pinyin translator for the Chinese Wikipedia, to help language learners. |Completed=false |User Name=Spacemartin }}" [20:17:53] 6Labs: Provide a simple way to backup arbitrary files from instances - https://phabricator.wikimedia.org/T104206#1812133 (10Halfak) Just +1ing this. I'm struggling to implement a robust backup strategy for #ores and #wikilabels right now. Because I don't want the instability of NFS, I'm currently SCPing copi... [20:39:29] 6Labs, 10Tool-Labs: webgrid nodes have very limited swap (500MB) - https://phabricator.wikimedia.org/T118419#1812212 (10valhallasw) To clarify, this was manually applied to `tools-webgrid-lighttpd-1415.tools.eqiad.wmflabs`, and still needs to be applied to the rest (https://gerrit.wikimedia.org/r/#/c/252506/).... [20:40:59] YuviPanda: ^ can we do something with hiera to easily do a staggered deploy? [20:41:44] meh, probably not easily, as the change sort of goes through everything. Bah. [20:43:09] I suppose puppet agent -disable and then enabling host by host should work [20:59:59] 6Labs, 10Tool-Labs: Document how to turn shadow into master - https://phabricator.wikimedia.org/T91133#1812235 (10coren) 5Open>3Resolved The (single step) procedure is documented; including a note about puppet that will shortly become true. [22:22:48] andrewbogott: YuviPanda been puzzling over it for awhile and I think the most straightforward mechanism is https://gerrit.wikimedia.org/r/#/c/253770/ for labtest. we basically hinge a ton of logic on a "subsite" aka realm (labs) within eqiad and to jump to not doing it in codfw while mixing a prod labstore and labcontrol box is pretty drastic, so this is my baby step [22:26:38] chasemp: so… what’s the problem with just using hiera settings specific to the affected hosts? [22:26:53] Won’t there be a ton more if/then switching if we make it realm-specific? [22:27:30] it solves two different problems, we are doing hiera() lookups someplaces as a kind of global and it's site based more or less [22:27:46] and we are doing realm checks in some places for prod vs labs and also labs vs openstack on labs [22:28:00] so there are places we are splitting teh fork in both methods [22:28:23] adding a realm for labtest doesn't negate some hiera things that are overridden at the host level for these hosts [22:28:44] ideally we don't use the realm here at all but this is going to be iterative I imagine [22:29:46] hm… ok. I would have predicted that all of the realm checks (prod vs. openstack-on-labs) can be replaced with hiera globals [22:29:52] but I must be forgetting some things [22:30:06] so we can but hiera has no realm equiv there is site and host and role [22:30:43] the real wierd thing is untangling the novaconfig dict that is a few levels of puppet inheretence deep that has to be parameterized to be hieraized [22:31:04] and basically I'm starting here to see how it goes [22:31:15] ah -- [22:31:40] it doesn’t have to be parameterized if it just yanks globals out of hiera [22:31:49] like a global dict? [22:32:08] or just a bunch of globals [22:32:33] maybe but we are overriding and realm modifying in a lot of places and there is no way to do that via hiera for this context [22:32:39] Like, right now some things like “which box is the labs controller?” are only answered in one place [22:32:55] if those are moved into parameters then we’ll have to answer the question in multiple places, per-role [22:33:18] when you say ‘no way to do it’ you mean ‘no way to do it without duplicating hiera config for each server’ right? [22:33:37] yeah if we duped ourselves to death it could be done that way [22:33:44] I guess I mean no way to do it that isn't worse than where we are now [22:34:17] ok [22:35:09] I don’t necessarily object to doing it based on realm, as long as you’re still planning to rip out all of the labs-on-labs code :) [22:35:29] yes and actually I spent some time looking into making eqiad/$realm.yaml work out [22:35:37] rather than if $realms [22:35:39] but small steps [22:37:22] hiera seems to indicate it will do combined hieararchy arrays/dicts [22:37:46] but it seems not ot actually work, not sure what's up there [23:29:59] 6Labs, 10Tool-Labs, 5Patch-For-Review: webgrid nodes have very limited swap (500MB) - https://phabricator.wikimedia.org/T118419#1812489 (10yuvipanda) Let's set a time and do a staggered deploy. [23:30:16] valhallasw`cloud: I think we can just merge it, test it on one instance and let it be [23:30:24] valhallasw`cloud: I think for precise it just won't take effect until restarted [23:30:30] valhallasw`cloud: and since these are webgrid nodes we can restart with abandon [23:43:46] PROBLEM - Puppet failure on tools-exec-1208 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [23:43:59] PROBLEM - Puppet failure on tools-exec-1409 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [23:44:06] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1210 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [23:44:15] PROBLEM - Puppet failure on tools-exec-1201 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [23:44:17] uh oh [23:44:45] PROBLEM - Puppet failure on tools-bastion-02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [23:45:21] PROBLEM - Puppet failure on tools-exec-1410 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:45:41] PROBLEM - Puppet failure on tools-exec-1214 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [23:46:08] (am investigating) [23:46:47] PROBLEM - Puppet failure on tools-webgrid-generic-1405 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [23:47:07] it's python-yaml conflict?! [23:47:09] wat [23:47:23] PROBLEM - Puppet failure on tools-submit is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:47:35] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1205 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [23:47:45] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [23:47:55] PROBLEM - Puppet failure on tools-exec-1217 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [23:48:07] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [23:48:19] PROBLEM - Puppet failure on tools-exec-1212 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [23:48:19] PROBLEM - Puppet failure on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [23:48:39] PROBLEM - Puppet failure on tools-exec-1216 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [23:48:49] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1408 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:49:05] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1209 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [23:49:26] PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [23:51:04] PROBLEM - Puppet failure on tools-exec-1209 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [23:52:20] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1202 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:52:20] PROBLEM - Puppet failure on tools-exec-1220 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:52:32] PROBLEM - Puppet failure on tools-exec-1218 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [23:53:28] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1401 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:54:02] PROBLEM - Puppet failure on tools-exec-1207 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [23:54:14] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1201 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [23:54:16] PROBLEM - Puppet failure on tools-exec-1215 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [23:55:22] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1411 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [23:55:27] PROBLEM - Puppet failure on tools-exec-1405 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [23:56:09] PROBLEM - Puppet failure on tools-exec-1404 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [23:57:19] PROBLEM - Puppet failure on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [23:57:19] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1404 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:57:19] PROBLEM - Puppet failure on tools-exec-1219 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [23:57:29] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1203 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [23:57:55] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1204 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [23:58:05] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1407 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [23:58:05] PROBLEM - Puppet failure on tools-exec-1203 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:58:05] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1406 is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0] [23:58:06] PROBLEM - Puppet failure on tools-exec-1408 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [23:58:59] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1207 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [23:59:11] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1208 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:59:13] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1402 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [23:59:25] PROBLEM - Puppet failure on tools-exec-1205 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]