[00:40:53] Cyberpower678: how did your solution turn out? [00:41:08] I haven't implemented it yet. [00:41:35] Let me know. :) [00:41:37] I need a new OAuth Key. The old one belonged to Hedonil. [00:42:33] The solution is to have the OAuth redirect back to the xtools-core with a return to address stored. It will then redirect back to the appropriate location. [00:43:16] I'm 99.9% sure it will work. I'm working on it's implementation. [01:38:50] 3Wikimedia-Labs-Infrastructure: "Stale file handle" for /public/dumps/ - https://phabricator.wikimedia.org/T87013#989433 (10Krinkle) I'm getting the same error on the puppet run for all instances in the "cvn" projects. It started January 16th. ``` Jan 22 01:13:27 cvn-app4 puppet-agent[5403]: (/Stage[main]/Role... [02:04:04] YuviPanda, welcome to the xtools family. You have been added to the maintainers list on the bottom of the tool. [02:09:53] 3translatewiki.net, Labs-Team: Vulnerability scanning from Wikimedia Labs IP address - https://phabricator.wikimedia.org/T87352#989484 (10Nikerabbit) 3NEW [02:15:58] YuviPanda, when a job is in the "t" state, what does that meam? [02:28:47] T13|something, something bugged up in labs again. [02:29:00] We've got 9 webservices running. [02:31:14] Okay. [02:31:28] I'm only seeing the one.. [02:31:49] How was it all split up? [02:32:54] T13|something, that's because I just killed them and cleaned it up. [02:33:08] IDK [02:33:13] I just know they were there. [02:34:29] * Cyberpower678 wonders how he can have so many issues moving ONE script. Hedonil didn't make this very move friendly. [02:36:00] Well... [02:36:34] Perhaps if we can't figure out what Hedonil was trying to do... [02:36:53] Maybe we should work on a rewrite from scratch? [02:36:55] Hedonil's inactive, possibly retired. [02:37:10] What Hedonil did was a rewrite from scratch. [02:37:33] Yes, and apparently it was incomplete and is buggy as hell.. [02:37:53] T13|something, At least it works better than before his rewrite. [02:41:28] That may be, but it wasn't complete and he's not around to help finish it. [02:42:24] have we tried contacting him via email? [02:42:32] So, we can either try to patch it up and put band-aids on it, or we can do our own rewrite. [02:42:53] For some reason it keeps trying to write to xtools and not xtools-ec [02:43:07] I don't know where that is happening. [02:43:07] Cyberpower678 would know more about trying to email him than I. [02:43:22] MusikAnimal, I already tried 7 times. [02:43:25] No response [02:43:49] perfect [02:53:06] just a thought, if it's any easier, you could move the edit counter's backend to a new tool and set up an API endpoint that the frontend of xtools could query [02:53:44] since that's the part that's consuming all the resouces [02:53:48] *resources [02:54:12] I think the issue with the session data being written from xtools-ec to xtools, and failing causing the script to terminate. [02:54:29] I need to figure out where it's happening from. [02:54:34] And modify it. [02:55:40] FOUND IT [02:57:26] woot [03:07:14] Fixed one, a million more issues crop up. :/ [03:07:39] * Cyberpower678 thinks [03:10:28] MusikAnimal, ^ [03:10:47] haha of course [03:10:57] baby steps [03:11:31] * Cyberpower678 sifts through the 17 Stack traces that got thrown at him. [03:12:36] First issue. replica.my.cnf is using wrong credentials for the DBs. [03:15:09] Next issue, redirect location to xtools-ec for credentials to maintain security. [03:22:03] MusikAnimal, now to fix a bad include path. [03:22:48] glad to hear of progress! wish I could help [03:23:04] I'm going to make a point to dab a little with PHP [03:23:47] YuviPanda, question. How should I link to your page. YuviPanda or Yuvipanda? [03:34:47] T13|something, almost ready [03:35:33] Kk [03:35:45] T13|sleeps, don't sleep yet. [03:36:27] ;) [03:36:50] 45 minutes left on my movie [03:54:17] PROBLEM - Puppet failure on tools-webgrid-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [03:54:28] T13|sleeps, done [03:54:39] It's now running on it's own service [04:03:54] YuviPanda, T13|sleeps: Success. The edit counter has been moved to xtools-ec and a redirect placed in the old spot. [04:04:35] Cyberpower678: yay, nice. [04:04:41] Try it out. [04:04:46] Cyberpower678: mind if I move it to run on trusty? has a newer php version, should be faster. [04:04:54] Cyberpower678: can we set up a cron job to restart it if it's mem usage goes over 1k GB? [04:05:14] T13|sleeps, let's see if stays running first. [04:05:22] I will in a bit. 10 min or so. [04:05:33] YuviPanda, go ahead, but it's gaurded by bigbrother [04:06:10] YuviPanda, T13|sleeps I thought it would interest everyone that I left everything but index on xTools [04:07:36] !log tools.xtools-ec moved to trusty, started via webservice2 --release trusty start [04:07:41] Cyberpower678: ^ [04:07:42] Logged the message, Master [04:07:44] is running ont rusty now [04:07:46] *trusty [04:09:10] Since there is no point in cloning the template code, should make maintaining easier. [04:09:30] Stupid internet [04:09:34] YuviPanda, :-) [04:11:25] God damnit [04:11:31] FATAL ERROR [04:12:59] Cyberpower678: hmm? [04:13:22] Cyberpower678: if you have to restart the tool for some reason, use webservice2 --release trusty restart [04:13:26] instead of webservice restart [04:13:50] No. Somethings wrong with...something. [04:13:53] ISK [04:13:56] *IDK [04:14:17] ok [04:24:18] RECOVERY - Puppet failure on tools-webgrid-02 is OK: OK: Less than 1.00% above the threshold [0.0] [04:26:19] Is someone tinkering with the tool right now? [04:26:24] YuviPanda, T13|sleeps ^ [04:26:43] not me [04:26:47] why? [04:27:11] I keep getting random errors, and now it's not even functional. Can you deactivate trusty? [04:27:31] ok [04:28:36] T13|sleeps, I have to stop working on the tool, can you take over and fix the errors? [04:28:47] Or YuviPanda? [04:28:48] !log tools.xtools-ec moved back to precise at Cyberpower678's request [04:28:52] Logged the message, Master [04:28:56] PHP isn't really my cup of tea, Cyberpower678 :) [04:29:35] T13|sleeps, YuviPanda: Moving away from Trusty fixes every single bug. [04:29:56] well, that means that the code doesn't work on PHP newer than 5.3 [04:30:34] Meaning we have to look for functions that got removed/deprecated between the release of Precise and Trusty [04:31:02] PHP 5.3 and PHP 5.5, that is [04:31:31] I'm not going to hunt for that ATM [04:31:42] Cyberpower678: uhm, you should probably remove my name from the maintainers list. I haven't done any work on this. [04:32:14] I've added you because you are now a maintainer. You wanted to join xTools I thought? [04:32:25] I wanted to help out with any tool-labs related unstabilities, sure. [04:32:33] but not the code itself. [04:32:35] Oh. [04:32:37] I'm not doing anything yet. Getting ready for bed, first classes of the semester tomorrow. [04:32:48] sorry if that was unclear [04:32:57] Shall I remove your access to xTools then? [04:33:15] Cyberpower678: I have root on toollabs :) But yeah, feel free to remove me from there. [04:33:22] YuviPanda: can you help with setting up xtools.wmflabs.org? [04:33:43] T13|sleeps: well, we would probably happily give you your own instance, but remember you have to administer that instance yourself [04:33:48] Instead of it being tools.wmflabs.org/xtools? [04:33:58] if services inside that instance go down, you are responsible for that. [04:34:05] so basically, you get a VM [04:34:11] with root access [04:34:17] That is one of our goals for xtools [04:34:21] and then with great power comes great responsibility :) [04:35:01] We're not quite ready for it yet, but it is a goal. :) [04:35:23] YuviPanda, I thought we decided against that in favor of splitting the tools. Don't tell me I ripped my hair out trying to fix a moved script for nothing. [04:35:36] Which I know both Cyberpower678 and musikanimal support. [04:35:41] Cyberpower678: no, I think deciding against that is a good idea. I was just responding to T13|sleeps asking for xtools.wmflabs.org [04:36:03] you can't get *.wmflabs.org without going the 'separate projects' route [04:36:13] I would still like to see xtools.wmflabs.org [04:37:08] We can do that and have the tools split there. [04:37:44] We'll create a mini toollabs setup. :p [04:38:14] YuviPanda: now that xtools-ec is moved back, what's the current restart command? [04:38:14] * YuviPanda will still reccomend against that [04:38:20] T13|sleeps: just 'webservice restart' [04:38:44] If we need to restart [04:38:49] Kk just checking. [04:38:51] Something tells me no. [04:39:12] I tried loading my stats and it's been spinning five minutes now.. [04:39:35] Just refreshed the page and trying again [04:40:35] Loads fine for me [04:41:42] Timed out. Internal error [04:42:15] Okay. Loaded that time [04:42:16] Purge your cache [04:42:25] Bad redirect I think. [04:43:11] Redirect is fine [04:43:18] I tested it myself [04:43:51] http://tools.wmflabs.org/xtools/ec/?user=Technical+13&project=en.wikipedia.org [04:44:06] Or not. [04:44:08] :/ [04:44:21] times out with that url, then when I click the URL on the internal error page it quickly loads. [04:44:55] You can look at the redirect yourself, I swear there shouldn't be anything wrong with it. [04:45:30] I updated the GitHub repo [04:47:13] Do you see anything wrong? [04:47:55] T13|sleeps, ^ [05:19:36] Fixed. [05:20:03] It didn't like the protocol relative way it was being done.. [05:20:33] I had to define ( isset( $_SERVER['HTTPS'] ) ? 'https' : 'http' ) [05:21:34] http://php.net/manual/en/reserved.variables.server.php [05:21:48] http://php.net/manual/en/function.header.php [05:22:05] Did I mention I love php.net/manual/en? [05:22:23] CP678 ^^ (you're not here....) [05:24:41] CP678|iPhone: you stable now? [05:26:01] CP678|iPhone: I might have fixed it. [05:26:07] It didn't like the protocol relative way it was being done.. [05:26:10] I had to define ( isset( $_SERVER['HTTPS'] ) ? 'https' : 'http' ) [05:26:13] http://php.net/manual/en/reserved.variables.server.php [05:26:17] http://php.net/manual/en/function.header.php [05:26:20] Did I mention I love php.net/manual/en? [05:26:25] good night. [05:52:41] PROBLEM - Free space - all mounts on tools-login is CRITICAL: CRITICAL: tools.tools-login.diskspace.root.byte_percentfree.value (<44.44%) [06:02:39] RECOVERY - Free space - all mounts on tools-login is OK: OK: All targets OK [06:07:25] PROBLEM - Puppet failure on tools-exec-06 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:34:53] PROBLEM - Puppet failure on tools-webgrid-tomcat is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [06:37:23] RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0] [06:39:21] PROBLEM - Puppet failure on tools-webgrid-03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:04:52] RECOVERY - Puppet failure on tools-webgrid-tomcat is OK: OK: Less than 1.00% above the threshold [0.0] [07:09:21] RECOVERY - Puppet failure on tools-webgrid-03 is OK: OK: Less than 1.00% above the threshold [0.0] [07:48:24] PROBLEM - Puppet failure on tools-exec-06 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:18:23] RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0] [09:00:20] PROBLEM - Puppet failure on tools-webgrid-03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:25:22] RECOVERY - Puppet failure on tools-webgrid-03 is OK: OK: Less than 1.00% above the threshold [0.0] [10:34:02] PROBLEM - Puppet failure on tools-mail is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [11:04:02] RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [11:29:23] PROBLEM - Puppet failure on tools-exec-06 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:38:58] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:46:31] 3Wikimedia-Labs-General: Installation of rlwrap to trusty.tools.wmflabs.org - https://phabricator.wikimedia.org/T87368#989779 (10doctaxon) 3NEW [11:47:55] PROBLEM - Puppet failure on tools-exec-11 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [11:52:23] why do I always get "error: [Errno 101] Network is unreachable"? is there any firewall to avoid flooding? [11:54:23] RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0] [12:04:00] RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [12:17:55] RECOVERY - Puppet failure on tools-exec-11 is OK: OK: Less than 1.00% above the threshold [0.0] [12:19:07] PROBLEM - Puppet failure on tools-dev is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [12:24:44] PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [12:44:04] RECOVERY - Puppet failure on tools-dev is OK: OK: Less than 1.00% above the threshold [0.0] [12:49:45] RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0] [13:13:51] PROBLEM - Puppet failure on tools-exec-wmt is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [13:14:59] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [13:34:34] PROBLEM - Puppet failure on tools-webgrid-generic-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [13:38:53] RECOVERY - Puppet failure on tools-exec-wmt is OK: OK: Less than 1.00% above the threshold [0.0] [13:39:25] 3Phabricator, Wikimedia-Labs-Other: phab-01.wmflabs.org test instance's statuses are out of date - https://phabricator.wikimedia.org/T76943#989926 (10Qgil) Would it be good enough to just edit https://phab-01.wmflabs.org/config/edit/maniphest.statuses/ ? Is there documentation anywhere about how to update the L... [13:43:25] 3Wikimedia-Labs-General, Project-Creators, Wikimedia-Labs-Other: Labs-General vs Labs-Other - https://phabricator.wikimedia.org/T87372#989928 (10Qgil) 3NEW [13:45:00] RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [14:04:39] RECOVERY - Puppet failure on tools-webgrid-generic-01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:10:23] PROBLEM - Puppet failure on tools-exec-06 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:35:23] RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0] [14:53:56] PROBLEM - Puppet failure on tools-static is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:55:21] PROBLEM - Puppet failure on tools-webgrid-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:19:00] RECOVERY - Puppet failure on tools-static is OK: OK: Less than 1.00% above the threshold [0.0] [15:20:21] RECOVERY - Puppet failure on tools-webgrid-02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:00:14] PROBLEM - Puppet failure on tools-login is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:06:13] PROBLEM - Puppet failure on tools-exec-03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:08:15] PROBLEM - Puppet failure on tools-exec-09 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:09:01] PROBLEM - Puppet failure on tools-exec-11 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:24:47] PROBLEM - Puppet failure on tools-exec-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:36:14] RECOVERY - Puppet failure on tools-exec-03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:38:56] RECOVERY - Puppet failure on tools-exec-11 is OK: OK: Less than 1.00% above the threshold [0.0] [16:45:15] RECOVERY - Puppet failure on tools-login is OK: OK: Less than 1.00% above the threshold [0.0] [16:49:14] anyone online who happens to know how to run the job runner in vagrant? [16:49:45] RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:49:51] Hi, I want to edit the crontab for a tool to have it occasionally restart. Is it safe to use "jlocal" to restart the service locally (as opposed to submitting the job to the grid)? Restarting takes about 3-5 seconds [16:49:54] Bigbrother monitoring doesn't help as the service stalls and does not stop [16:50:46] this is on tools-login [16:58:15] RECOVERY - Puppet failure on tools-exec-09 is OK: OK: Less than 1.00% above the threshold [0.0] [17:35:12] PROBLEM - Puppet failure on tools-webgrid-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [17:35:58] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:55:57] PROBLEM - Puppet failure on tools-exec-13 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [18:01:00] RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [18:03:40] Flaking puppet grrr [18:05:13] RECOVERY - Puppet failure on tools-webgrid-01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:25:56] RECOVERY - Puppet failure on tools-exec-13 is OK: OK: Less than 1.00% above the threshold [0.0] [18:51:01] PROBLEM - Puppet failure on tools-mail is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [18:52:55] 3Wikimedia-Labs-wikitech-interface: cannot create foo.foo.wmflabs.org domain - https://phabricator.wikimedia.org/T56664#990105 (10jeremyb-phone) [19:01:45] 3Wikimedia-Labs-General, Project-Creators, Wikimedia-Labs-Other: Labs-General vs Labs-Other - https://phabricator.wikimedia.org/T87372#990111 (10scfc) #Wikimedia-Labs-General is for issues surrounding setup of Wikimedia Labs projects, while #Wikimedia-Labs-Other is for issues in projects hosted in Wikimedia Labs... [19:21:01] RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [19:23:13] is this accurate and replication stopped for some db ? http://tools.wmflabs.org/betacommand-dev/cgi-bin/replag [19:23:55] PROBLEM - Puppet failure on tools-shadow is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [19:31:23] PROBLEM - Puppet failure on tools-exec-06 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [19:38:31] PROBLEM - Puppet failure on tools-exec-12 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [19:48:57] RECOVERY - Puppet failure on tools-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [19:56:28] RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0] [19:57:06] PROBLEM - Puppet failure on tools-exec-08 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:01:20] PROBLEM - Puppet failure on tools-webgrid-03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:03:35] RECOVERY - Puppet failure on tools-exec-12 is OK: OK: Less than 1.00% above the threshold [0.0] [20:26:21] RECOVERY - Puppet failure on tools-webgrid-03 is OK: OK: Less than 1.00% above the threshold [0.0] [20:27:05] RECOVERY - Puppet failure on tools-exec-08 is OK: OK: Less than 1.00% above the threshold [0.0] [20:31:23] If I created a Web proxy on tools to an instance is there a propagation delay? [20:34:03] And question 2: does the proxy go 80->specifiedport or specifiedport -> specifiedport? [20:36:57] PROBLEM - Puppet failure on tools-exec-13 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:37:01] PROBLEM - Puppet failure on tools-mail is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:39:51] PROBLEM - Puppet failure on tools-exec-04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:40:35] notconfusing: iirc zero delay (the registration on the proxy is blocking) and 80->specifiedport [20:42:43] 3Tool-Labs: Replace all references to "tools" by references to $instanceproject in operations/puppet and labs/toollabs - https://phabricator.wikimedia.org/T87387#990246 (10scfc) 3NEW a:3scfc [20:50:52] PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:51:58] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:57:18] PROBLEM - Puppet failure on tools-webgrid-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [21:01:57] RECOVERY - Puppet failure on tools-exec-13 is OK: OK: Less than 1.00% above the threshold [0.0] [21:02:02] RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [21:09:52] RECOVERY - Puppet failure on tools-exec-04 is OK: OK: Less than 1.00% above the threshold [0.0] [21:14:14] PROBLEM - Puppet failure on tools-exec-07 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:15:54] PROBLEM - Puppet failure on tools-webgrid-tomcat is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [21:20:55] RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0] [21:21:59] RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [21:22:19] RECOVERY - Puppet failure on tools-webgrid-02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:31:55] PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [21:38:06] PROBLEM - Puppet failure on tools-exec-08 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [21:40:57] RECOVERY - Puppet failure on tools-webgrid-tomcat is OK: OK: Less than 1.00% above the threshold [0.0] [21:44:17] RECOVERY - Puppet failure on tools-exec-07 is OK: OK: Less than 1.00% above the threshold [0.0] [21:49:55] PROBLEM - Puppet failure on tools-exec-11 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [22:01:53] RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0] [22:03:10] RECOVERY - Puppet failure on tools-exec-08 is OK: OK: Less than 1.00% above the threshold [0.0] [22:10:38] PROBLEM - Puppet failure on tools-submit is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:16:01] 3translatewiki.net, Wikimedia-Labs-General: Vulnerability scanning from Wikimedia Labs IP address - https://phabricator.wikimedia.org/T87352#990338 (10Nemo_bis) [22:19:32] (03CR) 10Legoktm: [C: 032] -WikidataRepo is now -WikidataRepository [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184599 (owner: 10Merlijn van Deen) [22:19:51] (03Merged) 10jenkins-bot: -WikidataRepo is now -WikidataRepository [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184599 (owner: 10Merlijn van Deen) [22:19:57] RECOVERY - Puppet failure on tools-exec-11 is OK: OK: Less than 1.00% above the threshold [0.0] [22:21:38] !log tools.wikibugs Updated channels.yaml to: 6e130fecc19a39a5caed12ca0dda25ad28df62f0 -WikidataRepo is now -WikidataRepository [22:21:44] Logged the message, Master [22:29:16] PROBLEM - Puppet failure on tools-exec-09 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [22:40:40] RECOVERY - Puppet failure on tools-submit is OK: OK: Less than 1.00% above the threshold [0.0] [22:54:19] RECOVERY - Puppet failure on tools-exec-09 is OK: OK: Less than 1.00% above the threshold [0.0] [23:45:01] PROBLEM - Puppet failure on tools-shadow is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]