[00:06:08] 3Wikimedia-Labs-Infrastructure: List of SVN users who were not migrated - https://phabricator.wikimedia.org/T60687#974887 (10Qgil) For definitions of Invalid and Declined, see https://www.mediawiki.org/wiki/Bug_management/Bug_report_life_cycle > A report is given the Invalid status (...) when it is a change tha... [00:52:10] 3Wikimedia-Labs-Infrastructure: List of SVN users who were not migrated - https://phabricator.wikimedia.org/T60687#974987 (10jayvdb) [00:58:09] 3Wikimedia-Labs-Infrastructure, Labs-Team: Debian Jessie image for Labs - https://phabricator.wikimedia.org/T75592#763305 (10Krinkle) [00:58:17] 3Wikimedia-Labs-Infrastructure, Labs-Team: Debian Jessie image for Labs - https://phabricator.wikimedia.org/T75592#763305 (10Krinkle) [01:52:16] PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<33.33%) [01:52:22] Coren: How do I get to where I need to be to start bouncer 6669 (wm-bot3) using https://wikitech.wikimedia.org/wiki/Nova_Resource:Bots/Documentation/wm-bot#How_to_fix_1_or_more_disconnected_instances ?? [01:52:49] I'm logged into tools-dev right now. [01:54:54] stwalkerster: can you help me figure how to get to where I need to go? [01:55:51] When I cd /mnt and try find . -maxdepth 1 -type d it shows the directory as empty. [02:13:53] PROBLEM - Puppet failure on tools-exec-15 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [02:41:17] 3Wikimedia-Labs-wikistats: completely remove or globally add the "views" column in stats tables - https://phabricator.wikimedia.org/T38293#975140 (10Dzahn) @Nemo_bis: the bug(s) are: https://phabricator.wikimedia.org/T381 https://phabricator.wikimedia.org/T74420 [02:42:39] What does "If you get permission denied You forgot to put your cloak to admins file" mean? https://wikitech.wikimedia.org/wiki/Nova_Resource:Bots/Documentation/wm-bot#If_you_get_permission_denied [02:43:53] RECOVERY - Puppet failure on tools-exec-15 is OK: OK: Less than 1.00% above the threshold [0.0] [02:50:50] PROBLEM - Puppet failure on tools-exec-wmt is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [02:57:39] Reedy: Can you help me restart wm-bot3's bouncer (6669)? [02:57:58] https://wikitech.wikimedia.org/wiki/Nova_Resource:Bots/Documentation/wm-bot#How_to_fix_1_or_more_disconnected_instances [02:58:34] I can do the kill and conn, but I get permission denied when I try to nohup. [03:12:15] RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK [03:12:55] Why is https://tools.wmflabs.org/copyvios/ giving me a 404 result? Something wrong with Labs or the tool? [03:14:49] Josve05a: Ask again now that Earwig has reconnected... [03:15:46] I did in -helpers [03:20:50] RECOVERY - Puppet failure on tools-exec-wmt is OK: OK: Less than 1.00% above the threshold [0.0] [06:36:53] PROBLEM - Puppet failure on tools-webgrid-tomcat is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:50:14] PROBLEM - Puppet failure on tools-exec-03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:02:45] 3Tool-Labs: Make http (404, 302, 301 etc) statistics for toolserver.org - https://phabricator.wikimedia.org/T85167#975282 (10Nemo_bis) [07:04:35] 3Tool-Labs: Toolserver redirect configuration broken after domain move - https://phabricator.wikimedia.org/T85166#975283 (10Nemo_bis) This is a rather severe bug. Does WMF have copies of the user/project .htaccess files? [07:06:53] RECOVERY - Puppet failure on tools-webgrid-tomcat is OK: OK: Less than 1.00% above the threshold [0.0] [07:20:18] RECOVERY - Puppet failure on tools-exec-03 is OK: OK: Less than 1.00% above the threshold [0.0] [09:44:27] (03PS1) 10Yuvipanda: Send #puppet to -operations [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184859 [09:45:13] (03CR) 10Giuseppe Lavagetto: [C: 031] Send #puppet to -operations [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184859 (owner: 10Yuvipanda) [09:45:24] (03CR) 10Yuvipanda: [C: 032] Send #puppet to -operations [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184859 (owner: 10Yuvipanda) [09:45:47] (03Merged) 10jenkins-bot: Send #puppet to -operations [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184859 (owner: 10Yuvipanda) [09:45:50] !log tools.wikibugs Updated channels.yaml to: dfdc89a5152f26f3072973a5747e7d4e27711ed1 Send #puppet to -operations [09:46:36] YuviPanda: behold the auto-merge magic! [09:46:46] valhallasw`cloud: yeah, but that didn’t seem to work.. [09:46:50] valhallasw`cloud: is it ignoring yellow tags? [09:46:51] ? [09:47:01] YuviPanda: yes, it is. Those are /tags/, after all :-p [09:47:09] I should still be able to filter by themn :P [09:47:16] yes/no [09:47:28] in this case, puppet issues should pop up on -operations [09:47:39] they are, after all, the only people who can fix it (most of the times) [09:47:58] and that’s true for a lot of operations things [09:48:02] https://phabricator.wikimedia.org/T86525 + https://phabricator.wikimedia.org/T1316 [09:49:11] valhallasw`cloud: we should also add tags to that, I think [09:49:25] what’s that called? [09:49:41] YuviPanda: I don't think tags should be reported on irc in general [09:49:46] no-one cares about 'easy' [09:49:53] valhallasw`cloud: but we do care about #puppet [09:49:55] it's rather the inverse: 'puppet' should probably not be a tag [09:50:02] hmm [09:50:05] what else should it be? [09:50:17] maybe a project, maybe a user group [09:50:21] like, https://phabricator.wikimedia.org/T86745 for example [09:50:27] the underlying problem is in analytics [09:50:33] but puppet tag is appropriate [09:50:41] idk, puppet isn’t really a project. [09:50:55] or a group of user [09:50:58] tag is the most accurate thing there [09:51:46] it's on a completely different level than 'easy' [09:51:49] so no, I disagree [09:52:32] valhallasw`cloud: how about, we allow *filtering* by them [09:52:33] otoh, blocked-on-* are also tags atm [09:52:35] valhallasw`cloud: but don’t report them? [09:52:43] valhallasw`cloud: yup, because again, tag is what makes sense, I think [09:53:02] valhallasw`cloud: at least as a compromise (allow filtering but not reporting) [09:53:27] valhallasw`cloud: I also see no problem with reporting Easy in the output :P [09:53:45] YuviPanda: I do. 3 line irc reports suck. [09:53:52] valhallasw`cloud: hehe :) [09:53:57] I kicked them out for a reason [09:53:59] valhallasw`cloud: yeah, I’m not going to debate on that. [09:54:10] but in any case, filtering should be available [09:54:19] right, but doesn’t seem to work [09:54:25] but with tags, T86525 becomes more important [09:54:45] YuviPanda: I know. T1316. [09:55:04] valhallasw`cloud: oh, mind if I fix? [09:55:30] YuviPanda: go ahead; just remember to restart both services afterwards [09:55:36] YuviPanda: and give T86525 some thought [10:02:09] (03PS1) 10Yuvipanda: Disentangle tag lists for filtering vs display [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184864 (https://phabricator.wikimedia.org/T1316) [10:02:12] valhallasw`cloud: [10:13:20] YuviPanda: https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcRzs7Eh_A6h4VHJ9w5ale2KQoLaa3E12k1rGlJoDuUYWRmQbkty [10:13:35] valhallasw`cloud: go on. [10:13:49] I haven’t tested it at all [10:13:58] and I didn’t even use set intersection [10:14:06] :D [10:14:45] (03CR) 10Merlijn van Deen: [C: 04-1] Disentangle tag lists for filtering vs display (032 comments) [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184864 (https://phabricator.wikimedia.org/T1316) (owner: 10Yuvipanda) [10:15:50] valhallasw`cloud: hmm [10:16:08] valhallasw`cloud: makes sense to move the disabled to messagebuilder too, yeah [10:16:14] *nod* [10:16:26] not sure if we want to /filter/ on them [10:16:31] valhallasw`cloud: and the rest was a uegh [10:16:35] :D [10:19:28] (03PS2) 10Yuvipanda: Disentangle tag lists for filtering vs display [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184864 (https://phabricator.wikimedia.org/T1316) [10:20:37] valhallasw`cloud: uh, projects can be either list or dict now [10:23:10] valhallasw`cloud: ^ [10:23:12] err [10:23:12] (03PS3) 10Yuvipanda: Disentangle tag lists for filtering vs display [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184864 (https://phabricator.wikimedia.org/T1316) [10:23:14] I mean [10:23:14] ^ [10:40:57] (03CR) 10Merlijn van Deen: [C: 04-1] "filtering on project should also be adapted to the new format" (031 comment) [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184864 (https://phabricator.wikimedia.org/T1316) (owner: 10Yuvipanda) [10:49:54] valhallasw`cloud: filtering on project should just work? [10:50:07] valhallasw`cloud: it’s just a for project in projects, and that should pass the unfiltered list through [10:50:10] and iterate over the keys [10:50:13] which is what we want [10:52:50] Ah ok [10:52:59] there's also an inline comment :-p [10:53:06] eh, no, that's not what we want [10:53:11] we probably dont [10:53:26] we probably don't want to match on archived projects, for example [10:55:47] valhallasw`cloud: but that should be fixed by removing archived projects in channels.py, I think [10:55:50] rather than in code [10:56:01] don’t think the code should enforce that [10:56:16] maybe [10:56:53] (03PS4) 10Yuvipanda: Disentangle tag lists for filtering vs display [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184864 (https://phabricator.wikimedia.org/T1316) [11:00:34] (03CR) 10Merlijn van Deen: [C: 031] Disentangle tag lists for filtering vs display [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184864 (https://phabricator.wikimedia.org/T1316) (owner: 10Yuvipanda) [11:00:44] YuviPanda: lgtm, please deploy <3 [11:00:50] tschhhhh [11:00:51] :D [11:00:52] ok [11:01:11] (03CR) 10Yuvipanda: [C: 032] Disentangle tag lists for filtering vs display [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184864 (https://phabricator.wikimedia.org/T1316) (owner: 10Yuvipanda) [11:01:20] valhallasw`cloud: I guess I’ll just have to restart both bots [11:01:24] YuviPanda: yeah [11:01:29] (03Merged) 10jenkins-bot: Disentangle tag lists for filtering vs display [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184864 (https://phabricator.wikimedia.org/T1316) (owner: 10Yuvipanda) [11:01:31] fab something something [11:02:19] just fab deploy [11:02:39] !log tools.wikibugs yuvipanda: Deployed 2b66af26ca2a7343d0743423a4c9fcc6b8296e5e Disentangle tag lists for filtering vs display wb2-phab, wb2-irc [11:02:43] Logged the message, Master [11:02:50] YuviPanda: :-p [11:03:02] valhallasw`cloud: now to see if it comes back up :P [11:03:12] might take a bit, because grid. [11:04:07] wb2-phab is still queued [11:04:15] that’s strange [11:05:02] ok, executing now [11:05:50] !log tools.wikibugs yuvipanda: Deployed 2b66af26ca2a7343d0743423a4c9fcc6b8296e5e Disentangle tag lists for filtering vs display wb2-phab, wb2-irc [11:05:54] just restarting [11:07:41] Hello. I've been checking the databases in Tool Labs and I can't find 'text table' and where the article talk pages are. Anyone can help me? Thanks. [11:09:16] moccard: heya! text isn’t stored in the databases in the WMF infrastructure (would be too huge) and hence not available on toollabs. You would have to use the API or parse the dumps to get text [11:10:05] well, I would be interested in the text length (bytes) and the number of outlinks for each page. any idea where to get that besides the dumps? [11:10:27] the dumps are huge to process... [11:10:55] moccard: both these should be available, I think [11:11:06] looking through https://www.mediawiki.org/wiki/Manual:Database_layout [11:11:13] https://www.mediawiki.org/wiki/Manual:Page_table has page_len column [11:11:25] https://www.mediawiki.org/wiki/Manual:Pagelinks_table has internal links [11:11:38] then there’s https://www.mediawiki.org/wiki/Manual:Externallinks_table for external links [11:13:21] yes, I guess that with pagelinks I can find inlinks to an article and outlinks from it. :) thanks. [11:13:27] yw! [11:13:46] what about article the talk page? [11:13:49] moccard: also there’s quarry.wmflabs.org which lets you run some queries (less than 10min runtime) from a web interface. Useful for poking around, etc :) [11:14:02] moccard: text you mean? Text isn’t available, but all other info should still be present [11:14:12] page table isn’t just pages in main namespace, but in all namespaces [11:16:43] what other info from an article talk page? [11:16:50] ? [11:16:56] they aren’t treated any differently than normal pages [11:18:41] I see! [11:21:02] If I want to process dumps in Tool Labs, should I have the file in my user space or is there any other path where they are available? [11:21:28] moccard: they are in /public/dumps [11:23:01] but they are compressed. [11:23:48] moccard: yeah, you should be able to stream them and uncompress them at the same time somewhat easily in most programming languages [11:24:02] there are also libraries that make it easy for you [11:24:03] like https://github.com/halfak/mediawiki-utilities [11:27:01] thanks! your a wikijedi! [11:27:37] moccard: :D yw [11:28:09] moccard: halfak might be able to help with questions about that library :) there’s also a #wikimedia-research channel that you might find interesting [11:35:28] yup! thanks. is that library installed in tool labs btw? [11:36:19] moccard: not sure, but yo7u can use virtualenv (and is reccomended to use virtualenv) [11:40:55] thanks YuviPanda [11:51:06] valhallasw`cloud: hmm, it’s trying to join #/dev/null and failing [11:51:07] and dying [11:51:13] YuviPanda: lolwut [11:51:45] valhallasw`cloud: oh, no [11:51:56] it’s actually failing with [11:51:56] ssl.SSLError: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:598) [11:52:07] O_o [11:52:15] but [11:52:17] the last thing it does [11:52:18] is [11:52:18] DEBUG irc3.wikibugs > JOIN #/dev/null [11:52:21] CRITICAL irc3.wikibugs connection lost (47763145995320): None [11:52:24] so perhaps that’s the problem [11:53:07] YuviPanda: probably. https://github.com/wikimedia/labs-tools-wikibugs2/blob/2b66af26ca2a7343d0743423a4c9fcc6b8296e5e/channelfilter.py#L43 should probably remove /dev/null [11:53:08] and irc3 should suck less ;D [11:53:27] heh [12:03:09] (03PS1) 10Yuvipanda: All-channels shouldn't include /dev/null [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184878 [12:03:11] valhallasw`cloud: ^ [12:03:34] actually [12:03:39] YuviPanda: you're not even using set differences! [12:04:06] (03CR) 10Merlijn van Deen: [C: 032] All-channels shouldn't include /dev/null [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184878 (owner: 10Yuvipanda) [12:04:14] it's fine ;-) [12:04:27] (03Merged) 10jenkins-bot: All-channels shouldn't include /dev/null [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184878 (owner: 10Yuvipanda) [12:04:50] (03PS1) 10Yuvipanda: Stop trying to join channels that start with # [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184879 [12:04:51] valhallasw`cloud: ^ better [12:04:56] valhallasw`cloud: set is used to dedupe :P [12:05:31] (03CR) 10Merlijn van Deen: [C: 04-1] Stop trying to join channels that start with # (031 comment) [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184879 (owner: 10Yuvipanda) [12:05:47] YuviPanda: how about only join channels that start with # :P [12:05:55] valhallasw`cloud: I… clearly should sleep more [12:05:56] today [12:06:01] :D [12:07:36] (03PS2) 10Yuvipanda: Stop trying to join channels that don't start with # [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184879 [12:08:09] valhallasw`cloud: better [12:08:29] (03PS3) 10Merlijn van Deen: Only join actual channels [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184879 (owner: 10Yuvipanda) [12:08:40] (03CR) 10Merlijn van Deen: [C: 032] Only join actual channels [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184879 (owner: 10Yuvipanda) [12:08:43] :> [12:08:52] valhallasw`cloud: doing a deploy [12:08:55] (03Merged) 10jenkins-bot: Only join actual channels [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184879 (owner: 10Yuvipanda) [12:09:02] kk [12:09:03] thx [12:09:19] !log tools.wikibugs yuvipanda: Deployed 8736032750b4fead35646ea9120621bf9d0ccb7e Only join actual channels wb2-phab, wb2-irc [12:09:22] Logged the message, Master [12:10:57] !log tools.wikibugs yuvipanda: Deployed 8736032750b4fead35646ea9120621bf9d0ccb7e Only join actual channels wb2-phab, wb2-irc [12:11:00] Logged the message, Master [12:11:41] that looks promising :( [12:13:43] valhallasw`cloud: wut [12:13:44] DEBUG irc3.wikibugs > JOIN #wikimedia-collaboration [12:13:48] CRITICAL irc3.wikibugs connection lost (46980363360632): None [12:13:59] maybe the ssl thing still is the real issue [12:14:05] that’s possible [12:14:07] highly [12:14:09] what’s the ssl thing? [12:14:28] 12:51 ssl.SSLError: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:598) < that [12:14:37] just shut off the ssl connection [12:14:42] hmm [12:15:22] valhallasw`cloud: ^ [12:15:23] (03PS1) 10Yuvipanda: Turn ssl off [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184881 [12:15:23] err [12:15:25] damn [12:15:26] valhallasw`cloud: ^ [12:15:34] (03PS2) 10Merlijn van Deen: Turn ssl off [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184881 (owner: 10Yuvipanda) [12:15:43] YuviPanda: fix port plz [12:16:11] 6667 or something [12:16:20] yeah [12:16:21] (03PS3) 10Yuvipanda: Turn ssl off [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184881 [12:16:29] I’m being very braindead today [12:16:30] hmm [12:16:32] (03PS4) 10Merlijn van Deen: Turn ssl off [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184881 (owner: 10Yuvipanda) [12:16:36] (03CR) 10Merlijn van Deen: [C: 032] Turn ssl off [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184881 (owner: 10Yuvipanda) [12:16:48] (03Merged) 10jenkins-bot: Turn ssl off [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184881 (owner: 10Yuvipanda) [12:17:18] valhallasw`cloud: doing deploy again [12:17:34] !log tools.wikibugs yuvipanda: Deployed 9521ec19491d35ebc40fdccb34e75e0bd7f9399f Turn ssl off wb2-phab, wb2-irc [12:17:37] Logged the message, Master [12:19:44] valhallasw`cloud: no luck [12:19:56] valhallasw`cloud: I don’t think my recent patch is what is causing the issue. I think just it was that required a restart [12:20:08] yeah think so too [12:20:09] DEBUG irc3.wikibugs > JOIN #wikidata [12:20:11] CRITICAL irc3.wikibugs connection lost (47573307717280): None [12:21:33] YuviPanda: telnetting to irc.freenode also is very flaky [12:21:40] nothing I send is responded to [12:22:28] hmm [12:22:32] maybe they blacklisted our IP? [12:24:52] hrm. [12:25:54] meh, I can connect [12:26:15] just forgot that you need to NICK abc before anything happens [12:26:46] hmm [12:28:13] YuviPanda: just wondering, what are all these tables which start by ep_ such as ep_courses? I see there are many other tables which are not in the scheme in mediawiki.org. Can I find doc somewhere? [12:28:44] moccard: they’re from https://www.mediawiki.org/wiki/Extension:Education_Program [12:28:49] usually you just search mw.org [12:30:44] I see there are many other besides education program so I guess they are from other extensions [12:31:05] moccard: yup. [12:32:08] :wikibugs!~wikibugs2@wikimedia/bot/pywikibugs QUIT :Excess Flood [12:32:08] ERROR :Closing Link: wikimedia/bot/pywikibugs (Excess Flood) [12:32:11] TOO MANY CHANNELS [12:32:20] * valhallasw`cloud laughs manically [12:32:21] valhallasw`cloud: oh [12:32:24] lol [12:32:38] should we throttle? [12:33:30] maybe. not immediately sure how, though [12:33:30] needs yield from magic [12:34:32] Bye! [12:34:34] thanks for all [12:35:17] YuviPanda: I'm going to make a quick hack now [12:36:51] valhallasw`cloud: ok [12:37:44] (03PS1) 10Merlijn van Deen: SSL back, quick channel join hack [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184883 [12:37:47] YuviPanda: ^ [12:38:16] I iz confused, where did your ssl change go [12:38:40] (03PS2) 10Merlijn van Deen: SSL back, quick channel join hack [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184883 [12:40:08] (03CR) 10Merlijn van Deen: [C: 032] SSL back, quick channel join hack [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184883 (owner: 10Merlijn van Deen) [12:40:21] (03Merged) 10jenkins-bot: SSL back, quick channel join hack [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184883 (owner: 10Merlijn van Deen) [12:40:33] !log tools.wikibugs valhallasw: Deployed c7a327dee5e8987aba84cc332cec2194c8675eb3 SSL back, quick channel join hack wb2-irc [12:41:18] start, fscking job [12:42:29] ok, wikibugs is back in -dev and will join here on the next event [12:42:31] I think [12:43:09] meh, no [12:44:23] 3Wikibugs: wikibugs has too many channels, causing join flood - https://phabricator.wikimedia.org/T86758#975732 (10valhallasw) [12:44:25] YuviPanda: back up.\ [12:48:40] valhallasw`cloud: wheeeee [12:49:08] YuviPanda: see T86758 [12:49:26] but really irc3 should just do throttling [13:01:28] 3Wikibugs: Always show matched project - https://phabricator.wikimedia.org/T86759#975751 (10valhallasw) 3NEW [13:10:16] valhallasw`cloud: legoktm y’know what would be nice? [13:10:29] RTD.org style tool for wikimedia [13:10:30] well [13:10:33] at least for toollabs / labs docs [13:15:53] YuviPanda: it's called wikitech [13:16:05] valhallasw`cloud: well, idk. one huge page sucks [13:16:15] but multiple smaller pages suck for ctrl-f [13:16:52] YuviPanda: wikitech has search, amirite [13:17:05] sucky, but search nonetheless [13:17:14] just need to limit it to Tools/Help pages [13:18:25] dunno [13:18:29] it also looks like shit :P [13:18:33] * YuviPanda isn’t a vector fan [14:26:53] 3Wikibugs: wikibugs has too many channels, causing join flood - https://phabricator.wikimedia.org/T86758#975846 (10valhallasw) https://github.com/gawel/irc3/issues/31 [14:56:18] 3Tool-Labs, Pywikibot-compat-to-core, pywikibot-core: Install all pywikibot python dependencies on tool labs - https://phabricator.wikimedia.org/T86015#975897 (10yuvipanda) 5Open>3stalled So, socketio client has been installed as well. Open Questions: # is python-search still needed if it is just an alias... [14:56:19] 3Tool-Labs: Packages to be added to toollabs puppet - https://phabricator.wikimedia.org/T55704#975899 (10yuvipanda) [14:58:22] YuviPanda: you should make a daemonize API-compatible clone that submits jobs to the grid instead ;-D [15:58:50] YuviPanda: can you have a look on https://phabricator.wikimedia.org/T86771 [15:58:58] it needs to be put in proper projects [15:59:01] idk which are those [15:59:39] petan: I’m not sure either, but I’ve added a few people as CC [16:01:52] petan: also, please post the version of the script you're using. 'basically' the same can still be fundamentally different ;-) [16:02:36] I will but since the error is clearly related to socketIO and I didn't change anything on that part, it's pretty much not important [16:35:56] 3Tool-Labs: Toolserver redirect configuration broken after domain move - https://phabricator.wikimedia.org/T85166#976215 (10jeremyb-phone) [16:37:53] 3Tool-Labs: Toolserver tilserver (toolserver.org/tiles/) redirect - https://phabricator.wikimedia.org/T86739#976228 (10jeremyb-phone) [17:08:24] PROBLEM - Puppet failure on tools-exec-06 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [17:09:56] PROBLEM - Puppet failure on tools-webgrid-05 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:11:52] PROBLEM - Puppet failure on tools-exec-wmt is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:11:53] PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [17:11:54] PROBLEM - Puppet failure on tools-static is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:12:22] PROBLEM - Puppet failure on tools-webgrid-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:12:52] PROBLEM - Puppet failure on tools-webgrid-tomcat is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:13:02] PROBLEM - Puppet failure on tools-mail is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:14:09] PROBLEM - Puppet failure on tools-exec-08 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:14:52] PROBLEM - Puppet failure on tools-exec-15 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:14:58] PROBLEM - Puppet failure on tools-exec-13 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:15:12] PROBLEM - Puppet failure on tools-exec-07 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:16:15] PROBLEM - Puppet failure on tools-login is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [17:16:29] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:16:47] PROBLEM - Puppet failure on tools-uwsgi-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [17:16:56] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [17:17:08] PROBLEM - Puppet failure on tools-dev is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:17:24] PROBLEM - Puppet failure on tools-webgrid-03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:17:34] PROBLEM - Puppet failure on tools-exec-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:17:47] PROBLEM - Puppet failure on tools-exec-catscan is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:18:13] PROBLEM - Puppet failure on tools-webgrid-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:18:31] PROBLEM - Puppet failure on tools-exec-12 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:19:36] Oh, what /now/ [17:19:58] PROBLEM - Puppet failure on tools-shadow is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:20:11] Coren: that’s just a wikitech outage. deploy in progress [17:28:06] I'm not getting it - puppet actually runs fine with no errors. [17:29:21] Coren: yup. wikitech is back up [17:29:37] Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Error from DataBinding 'hiera' while looking up 'apt::unattendedupgrades::ensure': Reading data from Tools failed: NoMethodError: undefined method `[]' for nil:NilClass on node i-000006a5.eqiad.wmflabs [17:29:50] it should probably have a better error message [17:31:15] RECOVERY - Puppet failure on tools-login is OK: OK: Less than 1.00% above the threshold [0.0] [17:37:14] PROBLEM - Puppet failure on tools-login is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:38:24] RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0] [17:39:57] RECOVERY - Puppet failure on tools-webgrid-05 is OK: OK: Less than 1.00% above the threshold [0.0] [17:41:51] RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0] [17:41:53] RECOVERY - Puppet failure on tools-exec-wmt is OK: OK: Less than 1.00% above the threshold [0.0] [17:42:09] Coren: ^ recovering now :) [17:42:19] RECOVERY - Puppet failure on tools-webgrid-02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:42:53] RECOVERY - Puppet failure on tools-webgrid-tomcat is OK: OK: Less than 1.00% above the threshold [0.0] [17:43:01] RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [17:43:31] RECOVERY - Puppet failure on tools-exec-12 is OK: OK: Less than 1.00% above the threshold [0.0] [17:44:59] RECOVERY - Puppet failure on tools-exec-13 is OK: OK: Less than 1.00% above the threshold [0.0] [17:45:13] RECOVERY - Puppet failure on tools-exec-07 is OK: OK: Less than 1.00% above the threshold [0.0] [17:46:29] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [17:47:09] RECOVERY - Puppet failure on tools-dev is OK: OK: Less than 1.00% above the threshold [0.0] [17:47:35] RECOVERY - Puppet failure on tools-exec-02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:47:45] RECOVERY - Puppet failure on tools-exec-catscan is OK: OK: Less than 1.00% above the threshold [0.0] [17:49:55] RECOVERY - Puppet failure on tools-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [17:59:51] RECOVERY - Puppet failure on tools-exec-15 is OK: OK: Less than 1.00% above the threshold [0.0] [18:01:58] RECOVERY - Puppet failure on tools-static is OK: OK: Less than 1.00% above the threshold [0.0] [18:04:06] RECOVERY - Puppet failure on tools-exec-08 is OK: OK: Less than 1.00% above the threshold [0.0] [18:06:48] RECOVERY - Puppet failure on tools-uwsgi-01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:07:16] RECOVERY - Puppet failure on tools-login is OK: OK: Less than 1.00% above the threshold [0.0] [18:07:24] RECOVERY - Puppet failure on tools-webgrid-03 is OK: OK: Less than 1.00% above the threshold [0.0] [18:08:16] RECOVERY - Puppet failure on tools-webgrid-01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:24:39] PROBLEM - Puppet failure on tools-submit is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [20:27:50] PROBLEM - Puppet failure on tools-redis is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [20:28:50] * Coren greatly desires that puppet had physical form so that he could strangle it. [20:52:46] RECOVERY - Puppet failure on tools-redis is OK: OK: Less than 1.00% above the threshold [0.0] [20:54:42] RECOVERY - Puppet failure on tools-submit is OK: OK: Less than 1.00% above the threshold [0.0] [20:58:14] PROBLEM - Puppet failure on tools-login is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [21:04:15] PROBLEM - Puppet failure on tools-exec-10 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [21:04:59] PROBLEM - Puppet failure on tools-webproxy is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [21:06:58] PROBLEM - Puppet failure on tools-exec-11 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [21:07:16] PROBLEM - Puppet failure on tools-exec-09 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [21:10:55] PROBLEM - Puppet failure on tools-webgrid-05 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [21:10:57] PROBLEM - Puppet failure on tools-exec-15 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [21:12:59] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [21:13:17] PROBLEM - Puppet failure on tools-webgrid-02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [21:13:55] PROBLEM - Puppet failure on tools-webgrid-tomcat is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:14:33] PROBLEM - Puppet failure on tools-exec-12 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [21:15:07] PROBLEM - Puppet failure on tools-exec-08 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [21:18:05] PROBLEM - Puppet failure on tools-dev is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [21:18:47] PROBLEM - Puppet failure on tools-exec-catscan is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:19:51] PROBLEM - Puppet failure on tools-exec-04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [21:22:46] PROBLEM - Puppet failure on tools-webgrid-04 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [21:23:18] RECOVERY - Puppet failure on tools-login is OK: OK: Less than 1.00% above the threshold [0.0] [21:23:46] PROBLEM - Puppet failure on tools-redis is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [21:29:22] PROBLEM - Puppet failure on tools-exec-06 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [21:32:28] Hello! [21:33:05] I am wondering how long I should expect my tool labs account to be created/approved [21:34:05] PROBLEM - Puppet failure on tools-mail is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:34:11] RECOVERY - Puppet failure on tools-exec-10 is OK: OK: Less than 1.00% above the threshold [0.0] [21:35:57] PROBLEM - Puppet failure on tools-exec-13 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [21:36:57] RECOVERY - Puppet failure on tools-exec-11 is OK: OK: Less than 1.00% above the threshold [0.0] [21:37:31] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [21:39:15] PROBLEM - Puppet failure on tools-login is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [21:40:39] PROBLEM - Puppet failure on tools-submit is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [21:40:49] PROBLEM - Puppet failure on tools-exec-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:40:57] PROBLEM - Puppet failure on tools-shadow is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [21:42:47] PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:42:57] RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [21:43:05] RECOVERY - Puppet failure on tools-dev is OK: OK: Less than 1.00% above the threshold [0.0] [21:43:19] RECOVERY - Puppet failure on tools-webgrid-02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:43:55] RECOVERY - Puppet failure on tools-webgrid-tomcat is OK: OK: Less than 1.00% above the threshold [0.0] [21:45:05] RECOVERY - Puppet failure on tools-exec-08 is OK: OK: Less than 1.00% above the threshold [0.0] [21:48:47] RECOVERY - Puppet failure on tools-exec-catscan is OK: OK: Less than 1.00% above the threshold [0.0] [21:53:59] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [22:00:52] RECOVERY - Puppet failure on tools-exec-15 is OK: OK: Less than 1.00% above the threshold [0.0] [22:00:56] RECOVERY - Puppet failure on tools-webgrid-05 is OK: OK: Less than 1.00% above the threshold [0.0] [22:02:32] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [22:04:02] RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [22:04:34] RECOVERY - Puppet failure on tools-exec-12 is OK: OK: Less than 1.00% above the threshold [0.0] [22:05:57] RECOVERY - Puppet failure on tools-exec-13 is OK: OK: Less than 1.00% above the threshold [0.0] [22:07:41] RECOVERY - Puppet failure on tools-webgrid-04 is OK: OK: Less than 1.00% above the threshold [0.0] [22:09:15] RECOVERY - Puppet failure on tools-login is OK: OK: Less than 1.00% above the threshold [0.0] [22:09:53] RECOVERY - Puppet failure on tools-exec-04 is OK: OK: Less than 1.00% above the threshold [0.0] [22:09:57] RECOVERY - Puppet failure on tools-webproxy is OK: OK: Less than 1.00% above the threshold [0.0] [22:10:42] RECOVERY - Puppet failure on tools-submit is OK: OK: Less than 1.00% above the threshold [0.0] [22:10:48] RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:10:56] RECOVERY - Puppet failure on tools-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [22:11:52] PROBLEM - Puppet failure on tools-exec-15 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [22:12:44] RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0] [22:12:52] PROBLEM - Puppet failure on tools-exec-wmt is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [22:13:48] RECOVERY - Puppet failure on tools-redis is OK: OK: Less than 1.00% above the threshold [0.0] [22:14:20] PROBLEM - Puppet failure on tools-webgrid-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [22:14:54] PROBLEM - Puppet failure on tools-webgrid-tomcat is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:15:16] PROBLEM - Puppet failure on tools-mail is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [22:17:00] PROBLEM - Puppet failure on tools-exec-13 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [22:17:20] RECOVERY - Puppet failure on tools-exec-09 is OK: OK: Less than 1.00% above the threshold [0.0] [22:18:20] PROBLEM - Puppet failure on tools-webgrid-03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:18:33] PROBLEM - Puppet failure on tools-exec-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:18:45] PROBLEM - Puppet failure on tools-webgrid-04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [22:19:27] RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0] [22:19:45] PROBLEM - Puppet failure on tools-exec-catscan is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [22:20:53] PROBLEM - Puppet failure on tools-exec-04 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:24:01] RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [22:25:13] PROBLEM - Puppet failure on tools-exec-10 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:28:16] PROBLEM - Puppet failure on tools-exec-09 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:28:20] 3Wikibugs: IRC bot to report Phabricator activity exactly like Wikibugs does with Bugzilla - https://phabricator.wikimedia.org/T131#977501 (10matmarex) [22:28:21] 3Wikibugs: wikibugs should Ignore CC changes - https://phabricator.wikimedia.org/T647#977499 (10matmarex) 5Resolved>3Open Happening again. Related to Phab update today? [22:28:44] PROBLEM - Puppet failure on tools-exec-05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:31:16] PROBLEM - Puppet failure on tools-exec-03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:32:52] PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [22:35:37] 3Wikibugs: wikibugs test bug - https://phabricator.wikimedia.org/T1152#977519 (10Legoktm) test [22:36:01] 3Wikibugs: wikibugs test bug - https://phabricator.wikimedia.org/T1152#977521 (10Legoktm) [22:36:34] 3Wikibugs: wikibugs test bug - https://phabricator.wikimedia.org/T1152#19999 (10Legoktm) [22:38:35] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:39:21] RECOVERY - Puppet failure on tools-webgrid-02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:40:03] RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [22:41:53] RECOVERY - Puppet failure on tools-exec-15 is OK: OK: Less than 1.00% above the threshold [0.0] [22:42:02] RECOVERY - Puppet failure on tools-exec-13 is OK: OK: Less than 1.00% above the threshold [0.0] [22:42:52] RECOVERY - Puppet failure on tools-exec-wmt is OK: OK: Less than 1.00% above the threshold [0.0] [22:44:54] RECOVERY - Puppet failure on tools-webgrid-tomcat is OK: OK: Less than 1.00% above the threshold [0.0] [22:48:23] RECOVERY - Puppet failure on tools-webgrid-03 is OK: OK: Less than 1.00% above the threshold [0.0] [22:48:33] RECOVERY - Puppet failure on tools-exec-02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:48:41] RECOVERY - Puppet failure on tools-webgrid-04 is OK: OK: Less than 1.00% above the threshold [0.0] [22:48:53] (03PS1) 10Legoktm: Fix wikibugs after Phabricator update [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/185079 (https://phabricator.wikimedia.org/T647) [22:49:45] RECOVERY - Puppet failure on tools-exec-catscan is OK: OK: Less than 1.00% above the threshold [0.0] [22:50:26] (03CR) 10Legoktm: [C: 032] Fix wikibugs after Phabricator update [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/185079 (https://phabricator.wikimedia.org/T647) (owner: 10Legoktm) [22:50:45] (03Merged) 10jenkins-bot: Fix wikibugs after Phabricator update [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/185079 (https://phabricator.wikimedia.org/T647) (owner: 10Legoktm) [22:50:49] RECOVERY - Puppet failure on tools-exec-04 is OK: OK: Less than 1.00% above the threshold [0.0] [22:52:17] !log tools.wikibugs legoktm: Deployed 492438a4da3bd10a6e53bd248c997a02edb9d781 Fix wikibugs after Phabricator update wb2-phab [22:52:22] Logged the message, Master [22:53:46] RECOVERY - Puppet failure on tools-exec-05 is OK: OK: Less than 1.00% above the threshold [0.0] [22:54:42] 3Wikibugs: IRC bot to report Phabricator activity exactly like Wikibugs does with Bugzilla - https://phabricator.wikimedia.org/T131#977605 (10Legoktm) [22:54:44] 3Wikibugs: wikibugs should Ignore CC changes - https://phabricator.wikimedia.org/T647#977603 (10Legoktm) 5Open>3Resolved Sigh. Should be fixed again again now. [22:56:12] RECOVERY - Puppet failure on tools-exec-03 is OK: OK: Less than 1.00% above the threshold [0.0] [22:57:52] RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0] [22:58:18] RECOVERY - Puppet failure on tools-exec-09 is OK: OK: Less than 1.00% above the threshold [0.0] [22:59:22] PROBLEM - Puppet failure on tools-webgrid-03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:59:44] PROBLEM - Puppet failure on tools-webgrid-04 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [23:00:11] RECOVERY - Puppet failure on tools-exec-10 is OK: OK: Less than 1.00% above the threshold [0.0] [23:00:57] PROBLEM - Puppet failure on tools-webproxy is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [23:11:02] PROBLEM - Puppet failure on tools-mail is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [23:19:12] PROBLEM - Puppet failure on tools-webgrid-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [23:19:36] PROBLEM - Puppet failure on tools-exec-02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [23:21:39] PROBLEM - Puppet failure on tools-submit is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [23:23:29] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [23:26:11] PROBLEM - Puppet failure on tools-exec-10 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [23:29:24] RECOVERY - Puppet failure on tools-webgrid-03 is OK: OK: Less than 1.00% above the threshold [0.0] [23:29:43] RECOVERY - Puppet failure on tools-webgrid-04 is OK: OK: Less than 1.00% above the threshold [0.0] [23:31:00] RECOVERY - Puppet failure on tools-webproxy is OK: OK: Less than 1.00% above the threshold [0.0] [23:41:00] RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [23:44:14] RECOVERY - Puppet failure on tools-webgrid-01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:49:39] RECOVERY - Puppet failure on tools-exec-02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:51:15] RECOVERY - Puppet failure on tools-exec-10 is OK: OK: Less than 1.00% above the threshold [0.0] [23:51:39] RECOVERY - Puppet failure on tools-submit is OK: OK: Less than 1.00% above the threshold [0.0]