[00:01:31] is there a tool anyone has which allows us to test Mediawiki POST queries to the API? [00:09:17] Magog_the_Ogre: Special:ApiSandbox [00:10:33] Magog_the_Ogre: also I open Firebug, make a similar request, Copy as cURL from the Net tab, then fiddle with it in a shell. [00:10:50] I have what I think is a bug in the API [00:10:56] the problem is, it only appears intermittently [00:11:24] so I'm forced to copy all 500 (!) titles when reporting the bug. Obviously GET will not suffice. :) [00:12:53] doesn't seem that page will suffice spagewmf, as it doesn't allow me to query multiple properties at the same time :( [00:41:17] * JD|cloud wonders how https://wikitech.wikimedia.org/wiki/Main_Page lists 2 TB of RAM in use... is that just the amount we have physically? [01:18:34] 3Wikimedia-Labs-wikistats: completely remove or globally add the "views" column in stats tables - https://phabricator.wikimedia.org/T38293#972628 (10Dzahn) 5Open>3Resolved all remnants removed, with the only exception that largest_csv.php still has a value for some wikis [03:44:27] 3Wikibugs, Phabricator: Set up dumping Phabricator's project taxonomy to a wiki - https://phabricator.wikimedia.org/T85096#972737 (10scfc) @valhallasw: Thanks! [05:40:24] hmm, I'm having trouble connecting to tools-login [05:40:27] If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [05:40:27] Permission denied (publickey,hostbased). [06:04:31] L235: I can help if you're still there [06:04:58] It works for me, so that's something :) Can you tell me when it last worked for you? [06:36:03] PROBLEM - Puppet failure on tools-mail is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [07:06:01] RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [07:14:59] Coren: I'm not aware of storage requirements, I thought we no longer had a quota [07:16:18] Let's try $ sudo du -shc /home [07:48:56] 3LabsDB-Auditor: Make labsdb1003 replicate same way as labsdb1002 and labsdb1001 - https://phabricator.wikimedia.org/T78835#972850 (10yuvipanda) 5Open>3Resolved a:3yuvipanda @springle says he's finished this before christmas! :) [08:07:41] 7.1G [08:07:47] I hope that's not the issue? [08:21:01] !log extdist deleted extdist2 instance [08:21:04] hah [08:21:04] Logged the message, Master [08:21:09] legoktm: :) [08:21:16] legoktm: one less X in shinken :) [08:21:55] woo [08:27:27] legoktm: how do I set up tox for labsdb-auditor? [08:27:39] that, or more accurately, how can I bribe you into doing that? :D [08:29:00] YuviPanda: like creating the tox.ini file or setting up jenkins jobs? [08:29:07] legoktm: latter [08:29:43] if you file a bug for it and assign me I can get to it tomorrow when I'm actually awake [08:29:47] legoktm: wheee sure [08:30:29] 3LabsDB-Auditor, Continuous-Integration: Setup jenkins jobs for labsdb-auditor - https://phabricator.wikimedia.org/T86622#972953 (10yuvipanda) 3NEW a:3Legoktm [08:30:33] legoktm: ^ [08:34:33] YuviPanda: Host DOWN alert for extdist2! <-- should I do anything about that? [08:34:43] legoktm: nope, it goes away on next puppet run [08:34:48] ok [08:34:55] legoktm: and gone now [09:18:37] 3LabsDB-Auditor, Continuous-Integration: Setup jenkins jobs for labsdb-auditor - https://phabricator.wikimedia.org/T86622#973002 (10hashar) The tox based convention for python and related CI configuration are described at https://www.mediawiki.org/wiki/Continuous_integration/Tutorials/Test_your_python should be... [09:18:45] 3LabsDB-Auditor, Continuous-Integration: Setup jenkins jobs for labsdb-auditor - https://phabricator.wikimedia.org/T86622#973004 (10hashar) p:5Triage>3Normal [09:28:32] 3Tool-Labs-tools-Other: OAuth sometimes fails on AutoList2 - https://phabricator.wikimedia.org/T78247#973032 (10yuvipanda) [09:31:15] 3Tool-Labs-tools-Other: OAuth sometimes fails on AutoList2 - https://phabricator.wikimedia.org/T78247#973034 (10yuvipanda) You should probably file AutoList issues against https://bitbucket.org/magnusmanske/autolist/issues?status=new&status=open [09:48:22] YuviPanda: good afternoon. Do you know whether tools.wmflabs.org is a reverse proxy ? And whether it add X-Forwarded-For header ? :] [09:48:46] hashar: heya! it is a reverse proxy, but it doesn’t add XFF because the privacy policy doesn’t allow leaking IP [09:49:39] WTF! :-D [09:49:59] though one can gather it by adding a Javascript and a web service to gather it [09:50:30] hashar: heh, yeah [09:51:01] YuviPanda: Merlijn wants to expose a web service that would be triggered by Jenkins [09:51:11] hashar: yeah, been following that :) [09:51:12] and we wanted to have an IP whitelist to prevent abuse [09:51:17] hashar: can’t do IP whitelist, sadly. [09:51:21] hashar: general labs proxy sets XFF [09:51:28] yeah found that [10:29:17] (03PS1) 10Merlijn van Deen: Updated config.json example [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184597 [10:29:36] (03CR) 10jenkins-bot: [V: 04-1] Updated config.json example [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184597 (owner: 10Merlijn van Deen) [10:40:28] (03PS2) 10Merlijn van Deen: Updated config.json example [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184597 [10:41:08] (03CR) 10Merlijn van Deen: [C: 032] "(basically a doc update & merging to test the auto-pull)" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184597 (owner: 10Merlijn van Deen) [10:41:45] (03Merged) 10jenkins-bot: Updated config.json example [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184597 (owner: 10Merlijn van Deen) [10:43:11] legoktm: ^ 't is working! [10:46:00] (03PS1) 10Merlijn van Deen: -WikidataRepo is now -WikidataRepository [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184599 [11:37:49] YuviPanda: is labmonxxx a bare metal server in the labs infra just like the other virtXX servers? [11:38:13] valhallasw`cloud: we could even customize the message reported on success of abs-tools-wikibugs2-autopull [11:38:53] - name: beta-mediawiki-config-update-eqiad [11:38:53] branch: ^master$ [11:38:53] success-message: 'Change has been deployed on the EQIAD beta cluster' [11:40:42] hashar: it is [12:04:03] hashar: ah, good to know. I'm pretty happy with it as-is, to be honest :-) [12:05:00] YuviPanda: https://gerrit.wikimedia.org/r/#/c/184497/ please +2 if you're ok with MIT, otherwise please suggest another license :-) [12:12:37] YuviPanda: great :-) [12:17:31] (03CR) 10Yuvipanda: [C: 032] Add LICENSE and CREDITS [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184497 (owner: 10Merlijn van Deen) [12:17:53] (03Merged) 10jenkins-bot: Add LICENSE and CREDITS [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184497 (owner: 10Merlijn van Deen) [13:25:49] hi andrewbogott_afk, Coren, YuviPanda: can I haz 1 ip for huggle project? [13:25:57] hey petan [13:26:08] petan: what are you going to use it for? if it’s for a web interface there’s the proxy.. [13:26:20] I need to launch some service for huggle that listen on port 8822 [13:26:32] idk how to accomplish that using a proxy [13:26:35] it's not http protocol [13:26:58] petan: yeah, if it isn’t http protocol won’t work on proxy. [13:27:13] petan: I’ll allocate one in about 5mins? [13:27:18] ok great [13:29:58] !log huggle increased floating IP quota to 1 [13:30:00] petan: ^ [13:30:07] yay [13:30:08] thanks [14:48:16] PROBLEM - Puppet failure on tools-exec-03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:49:25] PROBLEM - Puppet failure on tools-exec-09 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:50:47] * Coren checks puppet [14:50:52] PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:50:56] PROBLEM - Puppet failure on tools-exec-11 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:51:23] Ah. My fault this time. [14:51:28] PROBLEM - Puppet failure on tools-exec-06 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:51:59] But also: harmless. *grumblel* [14:52:22] Coren: what happened? which patch? [14:52:50] Not a patch; wip on NFS made /public/backup handles stale and makes puppet whine. [14:53:34] ah [14:53:35] right [14:53:51] PROBLEM - Puppet failure on tools-exec-wmt is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:53:55] I wish it was possible to tell puppet "don't worry if that doesn't work" in a resource. [14:53:55] PROBLEM - Puppet failure on tools-webgrid-05 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:54:21] 3Wikimedia-Labs-Infrastructure: List of SVN users who were not migrated - https://phabricator.wikimedia.org/T60687#973667 (10Aklapper) >>! In T60687#636177, @scfc wrote: > I assume that *all* SVN users have been migrated to LDAP/Gerrit/wikitech/Labs (causing such errors as bug #61967), so I don't see a way to di... [14:54:49] PROBLEM - Puppet failure on tools-uwsgi-01 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [14:54:55] PROBLEM - Puppet failure on tools-webgrid-tomcat is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:55:18] PROBLEM - Puppet failure on tools-webgrid-02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:55:34] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:55:56] PROBLEM - Puppet failure on tools-static is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:56:15] That should actually go away next puppet run. [14:56:43] But I'm probably better of removing the mount entirely for the time being. [14:57:02] Coren: also !log so I can learn what you’re doing :) [14:57:02] PROBLEM - Puppet failure on tools-mail is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:57:15] PROBLEM - Puppet failure on tools-webgrid-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:57:15] PROBLEM - Puppet failure on tools-exec-07 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:57:35] PROBLEM - Puppet failure on tools-exec-12 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:57:58] PROBLEM - Puppet failure on tools-exec-13 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:57:58] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:58:14] PROBLEM - Puppet failure on tools-login is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:01:53] PROBLEM - Puppet failure on tools-exec-15 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:03:16] RECOVERY - Puppet failure on tools-login is OK: OK: Less than 1.00% above the threshold [0.0] [15:07:51] PROBLEM - Puppet failure on tools-exec-04 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:07:56] PROBLEM - Puppet failure on tools-shadow is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:09:13] PROBLEM - Puppet failure on tools-login is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:10:24] PROBLEM - Puppet failure on tools-webgrid-03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:10:40] PROBLEM - Puppet failure on tools-submit is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:10:48] PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:11:29] yikes [15:11:30] wikitech is down.. [15:11:33] (Cannot contact the database server: Too many connections (208.80.154.18)) [15:11:34] PROBLEM - Puppet failure on tools-exec-02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:11:42] PROBLEM - Puppet failure on tools-webgrid-04 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [15:11:44] PROBLEM - Puppet failure on tools-exec-05 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:11:48] PROBLEM - Puppet failure on tools-redis is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:11:48] PROBLEM - Puppet failure on tools-exec-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:12:10] PROBLEM - Puppet failure on tools-exec-10 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:12:36] Glaisher: yikes. [15:12:52] yikes yikes [15:12:58] PROBLEM - Puppet failure on tools-webproxy is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:13:23] Glaisher: back up [15:13:31] nice :) [15:14:16] RECOVERY - Puppet failure on tools-exec-09 is OK: OK: Less than 1.00% above the threshold [0.0] [15:17:06] PROBLEM - Puppet failure on tools-exec-08 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:18:06] PROBLEM - Puppet failure on tools-dev is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:18:14] RECOVERY - Puppet failure on tools-exec-03 is OK: OK: Less than 1.00% above the threshold [0.0] [15:18:48] PROBLEM - Puppet failure on tools-exec-catscan is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:21:24] RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0] [15:23:52] RECOVERY - Puppet failure on tools-exec-wmt is OK: OK: Less than 1.00% above the threshold [0.0] [15:25:15] PROBLEM - Puppet failure on tools-exec-09 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:27:25] PROBLEM - Puppet failure on tools-exec-06 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:27:41] Stupid [bleep]ing piece of [bleep] [bleep] puppet [bleep]. [15:29:13] PROBLEM - Puppet failure on tools-exec-03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:31:23] PROBLEM - Puppet failure on tools-exec-wmt is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:35:55] RECOVERY - Puppet failure on tools-exec-11 is OK: OK: Less than 1.00% above the threshold [0.0] [15:40:48] YuviPanda: Still getting the nil:NilClass thing intermittently [15:40:52] RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0] [15:44:46] RECOVERY - Puppet failure on tools-uwsgi-01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:44:58] RECOVERY - Puppet failure on tools-webgrid-tomcat is OK: OK: Less than 1.00% above the threshold [0.0] [15:45:58] Coren: ugh, might still be mysql [15:46:01] Coren: I’ve to go now though :( [15:47:06] RECOVERY - Puppet failure on tools-exec-08 is OK: OK: Less than 1.00% above the threshold [0.0] [15:47:14] RECOVERY - Puppet failure on tools-exec-07 is OK: OK: Less than 1.00% above the threshold [0.0] [15:49:51] Coren, I popped in due to the virt1000 warnings, but I guess it's our monthly puppet DOS attack [15:50:01] ….and… now it's back [15:50:16] * andrewbogott didn't touch anything [15:50:21] Puppet: inject some random in your day! [15:50:31] andrewbogott: oh, did shinken email you? [15:50:37] indeed [15:50:41] YuviPanda: yep -- it works! [15:50:43] andrewbogott: it was preceeded by a wikitech outage [15:50:51] from mysql [15:50:55] It's one and the same -- just an http flood [15:51:17] sigh [15:51:35] anyway, I’m moving cities again tonight, brb going to catch a bus [15:51:39] I don't know if it's just that the stars align and every labs instance runs puppet at the same second? [15:51:46] Anyway, I'm going back to bed [15:52:05] andrewbogott: yup, good night [15:52:52] RECOVERY - Puppet failure on tools-exec-04 is OK: OK: Less than 1.00% above the threshold [0.0] [15:52:58] RECOVERY - Puppet failure on tools-webproxy is OK: OK: Less than 1.00% above the threshold [0.0] [15:52:58] RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [15:52:58] RECOVERY - Puppet failure on tools-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [15:53:06] RECOVERY - Puppet failure on tools-dev is OK: OK: Less than 1.00% above the threshold [0.0] [15:53:46] RECOVERY - Puppet failure on tools-exec-catscan is OK: OK: Less than 1.00% above the threshold [0.0] [15:53:56] RECOVERY - Puppet failure on tools-webgrid-05 is OK: OK: Less than 1.00% above the threshold [0.0] [15:54:16] RECOVERY - Puppet failure on tools-login is OK: OK: Less than 1.00% above the threshold [0.0] [15:55:18] RECOVERY - Puppet failure on tools-webgrid-02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:55:20] RECOVERY - Puppet failure on tools-webgrid-03 is OK: OK: Less than 1.00% above the threshold [0.0] [15:55:39] RECOVERY - Puppet failure on tools-submit is OK: OK: Less than 1.00% above the threshold [0.0] [15:55:45] RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0] [15:55:47] PROBLEM - Puppet failure on tools-uwsgi-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:56:43] RECOVERY - Puppet failure on tools-exec-05 is OK: OK: Less than 1.00% above the threshold [0.0] [15:56:47] RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:57:13] RECOVERY - Puppet failure on tools-exec-10 is OK: OK: Less than 1.00% above the threshold [0.0] [15:57:25] RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0] [15:57:32] YuviPanda: you around? [15:57:45] milimetric: kindof. not for long [15:57:46] ‘sup [15:57:54] just wanted to monitor an instance on labs [15:58:04] milimetric: ah, which one? [15:58:05] PROBLEM - Puppet failure on tools-exec-08 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:58:08] dan-pentaho [15:58:08] milimetric: and what kind of monitoring? [15:58:12] i just need to know if it's up or not [15:58:17] PROBLEM - Puppet failure on tools-exec-07 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:58:29] milimetric: http://shinken.wmflabs.org/all?global_search=dan-pentaho# [15:58:33] milimetric: already being monitored :) [15:58:35] you will get alerts [15:58:49] PROBLEM - Puppet failure on tools-exec-04 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:58:57] PROBLEM - Puppet failure on tools-shadow is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:59:07] PROBLEM - Puppet failure on tools-dev is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:59:09] doh, YuviPanda what's the user/pass there? [15:59:19] guest/guest [15:59:22] :) [15:59:25] thanks muchly [15:59:27] nite! [15:59:48] PROBLEM - Puppet failure on tools-exec-catscan is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:00:16] PROBLEM - Puppet failure on tools-login is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:01:22] PROBLEM - Puppet failure on tools-webgrid-03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:01:40] PROBLEM - Puppet failure on tools-submit is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:01:46] PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:02:46] PROBLEM - Puppet failure on tools-exec-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:03:12] PROBLEM - Puppet failure on tools-exec-10 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:03:58] PROBLEM - Puppet failure on tools-webproxy is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:07:46] PROBLEM - Puppet failure on tools-exec-05 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:13:58] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:19:14] RECOVERY - Puppet failure on tools-exec-03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:20:56] RECOVERY - Puppet failure on tools-static is OK: OK: Less than 1.00% above the threshold [0.0] [16:21:50] RECOVERY - Puppet failure on tools-exec-15 is OK: OK: Less than 1.00% above the threshold [0.0] [16:22:00] RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [16:23:15] RECOVERY - Puppet failure on tools-exec-07 is OK: OK: Less than 1.00% above the threshold [0.0] [16:24:07] RECOVERY - Puppet failure on tools-dev is OK: OK: Less than 1.00% above the threshold [0.0] [16:25:17] RECOVERY - Puppet failure on tools-login is OK: OK: Less than 1.00% above the threshold [0.0] [16:25:51] RECOVERY - Puppet failure on tools-uwsgi-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:26:21] RECOVERY - Puppet failure on tools-webgrid-03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:26:33] RECOVERY - Puppet failure on tools-exec-02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:27:13] RECOVERY - Puppet failure on tools-webgrid-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:27:33] RECOVERY - Puppet failure on tools-exec-12 is OK: OK: Less than 1.00% above the threshold [0.0] [16:27:45] RECOVERY - Puppet failure on tools-exec-05 is OK: OK: Less than 1.00% above the threshold [0.0] [16:28:05] RECOVERY - Puppet failure on tools-exec-08 is OK: OK: Less than 1.00% above the threshold [0.0] [16:28:11] RECOVERY - Puppet failure on tools-exec-10 is OK: OK: Less than 1.00% above the threshold [0.0] [16:28:57] RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [16:29:00] RECOVERY - Puppet failure on tools-webproxy is OK: OK: Less than 1.00% above the threshold [0.0] [16:29:48] RECOVERY - Puppet failure on tools-exec-catscan is OK: OK: Less than 1.00% above the threshold [0.0] [16:30:16] RECOVERY - Puppet failure on tools-exec-09 is OK: OK: Less than 1.00% above the threshold [0.0] [16:30:32] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [16:31:41] RECOVERY - Puppet failure on tools-submit is OK: OK: Less than 1.00% above the threshold [0.0] [16:31:43] RECOVERY - Puppet failure on tools-webgrid-04 is OK: OK: Less than 1.00% above the threshold [0.0] [16:32:50] RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:36:45] RECOVERY - Puppet failure on tools-redis is OK: OK: Less than 1.00% above the threshold [0.0] [16:39:54] RECOVERY - Puppet failure on tools-exec-wmt is OK: OK: Less than 1.00% above the threshold [0.0] [16:40:46] PROBLEM - Puppet failure on tools-exec-catscan is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:42:40] PROBLEM - Puppet failure on tools-submit is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [16:42:59] RECOVERY - Puppet failure on tools-exec-13 is OK: OK: Less than 1.00% above the threshold [0.0] [16:43:45] PROBLEM - Puppet failure on tools-exec-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:48:52] RECOVERY - Puppet failure on tools-exec-04 is OK: OK: Less than 1.00% above the threshold [0.0] [16:52:51] PROBLEM - Puppet failure on tools-exec-15 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:53:58] PROBLEM - Puppet failure on tools-exec-13 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:54:14] PROBLEM - Puppet failure on tools-exec-07 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:54:58] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:58:40] valhallasw`cloud: yay :D [16:58:43] Coren: are there any know outages? [16:58:53] legoktm: ? [16:59:08] Betacommand: Puppet is currently being flaky, but that shouldn't affect anything except for the noise. [17:08:58] RECOVERY - Puppet failure on tools-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [17:12:42] RECOVERY - Puppet failure on tools-submit is OK: OK: Less than 1.00% above the threshold [0.0] [17:13:48] RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:16:44] RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0] [17:17:40] 3Tool-Labs-tools-Other: OAuth sometimes fails on AutoList2 - https://phabricator.wikimedia.org/T78247#973988 (10scfc) 5Open>3Invalid a:3scfc (What @yuvipanda said.) [17:22:51] RECOVERY - Puppet failure on tools-exec-15 is OK: OK: Less than 1.00% above the threshold [0.0] [17:23:59] RECOVERY - Puppet failure on tools-exec-13 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:59] RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:13] RECOVERY - Puppet failure on tools-exec-07 is OK: OK: Less than 1.00% above the threshold [0.0] [17:30:45] RECOVERY - Puppet failure on tools-exec-catscan is OK: OK: Less than 1.00% above the threshold [0.0] [17:57:36] 3Gerrit-Patch-Uploader, Wikimedia-Git-or-Gerrit: Gerrit-patch-uploader fails under git 1.9 - https://phabricator.wikimedia.org/T86304#974034 (10Krinkle) I encountered a similar problem on Mac. I didn't check the git versions at the time, but I suspect it might be the same. The git version that ships with OS X "... [18:32:42] is toolsbeta-package-builder.eqiad.wmflabs actively being used? [18:51:20] paravoid: Not by me [21:21:16] Coren, can random labs projects have their bugs tracked in phabricator? I can't find a project... [21:21:57] Well, you'd want to have a project created for 'em, but sure - we were entirely willing to open bz products for them too. [21:31:17] PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<22.22%) [21:33:12] 3Wikimedia-Labs-Infrastructure: List of SVN users who were not migrated - https://phabricator.wikimedia.org/T60687#974598 (10scfc) 5Open>3Invalid a:3scfc (My understanding: "Declined" = "I could fix this, but I won't"; "Invalid": "It's objectively impossible to resolve this".) [21:41:16] RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK [21:55:16] 3Tool-Labs: Tool Labs: jsub starts multiple instances of tasks declared as "once" - https://phabricator.wikimedia.org/T62862#656832 (10Krinkle) Still happening. Just had to kill a dozen instances of ecmabot. ``` tools.ecmabot@tools-login:~$ qstat job-ID prior name user state submit/start at... [22:15:41] 3LabsDB-Auditor, Continuous-Integration: Setup jenkins jobs for labsdb-auditor - https://phabricator.wikimedia.org/T86622#974703 (10Legoktm) 5Open>3Resolved [22:28:29] Job 5319489 won't die [22:28:36] Keeps restarting [22:28:47] See ecmabot-wm in wikimedia-dev etc. [22:29:08] I've renamed the executable so the grid can't run it but it's still restarting constantly [22:29:18] It seems qdel is overidden by -continuous [22:29:28] it kills it, and then immediately comes back [22:29:31] but still with the same job id. [22:37:49] Hello. I am a newbe in Tool Labs. I am starting to code my scripts to process data from the database. However, I am trying to understand the tables and it is rather difficult. I don't know all the fields and don't find their equivalents in the real Wikipedia. Any advice? Thanks. [22:41:52] the schema is documented at https://www.mediawiki.org/wiki/Manual:Database_layout [22:41:56] should be relatively up to date [22:47:43] thanks legoktm, i didn't know about this doc. [22:51:17] PROBLEM - Puppet failure on tools-webgrid-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [23:00:53] PROBLEM - Puppet failure on tools-webgrid-tomcat is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:02:30] lol, ok, no qstat doesn't work anymore [23:02:31] error: unable to send message to qmaster using port 6444 on host "tools-master.eqiad.wmflabs": can't resolve host name [23:04:51] PROBLEM - Puppet failure on tools-exec-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [23:05:07] PROBLEM - Puppet failure on tools-dev is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [23:20:56] RECOVERY - Puppet failure on tools-webgrid-tomcat is OK: OK: Less than 1.00% above the threshold [0.0] [23:21:20] RECOVERY - Puppet failure on tools-webgrid-02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:25:10] RECOVERY - Puppet failure on tools-dev is OK: OK: Less than 1.00% above the threshold [0.0] [23:29:49] RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:45:26] 3Labs-Team: fix http://openmeetings.wmflabs.org/ - https://phabricator.wikimedia.org/T86698#974844 (10chasemp)