[00:06:02] !log tools Remove user maximilianklein from tools.cite-o-meter (on request) [00:06:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [00:29:35] 06Labs, 06Operations, 07Tracking: Cleanup tools nfs share on labstore1004/5 - https://phabricator.wikimedia.org/T156982#2991854 (10madhuvishy) [00:33:01] 06Labs, 06Operations, 07Tracking: Cleanup tools nfs share on labstore1004/5 - https://phabricator.wikimedia.org/T156982#2991870 (10madhuvishy) Finding all log/err/out files >1G by doing `find /srv/tools/shared/tools/ -type f -size +1G > gt1g` `cat gt1g | grep -e \.log -e \.error -e \.err -e debug\.txt -e \.o... [00:39:40] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#2991885 (10DeltaQuad) >>! In T143349#2965177, @chasemp wrote: >>>! In T143349#2962963, @chasemp wrote: >> Greetings @Crazycomputers @DeltaQuad @Tparis from https://wikitech.wikimedia.org... [00:49:54] 06Labs, 10Tool-Labs: Investigate alternatives to dedicated exec node for gifti's tools - https://phabricator.wikimedia.org/T99130#2991896 (10scfc) [00:49:58] 06Labs, 10Tool-Labs, 07Epic: Find a solution for tools-exec-gift on Trusty - https://phabricator.wikimedia.org/T156981#2991895 (10scfc) [00:52:33] !log deployment-prep added mhurd as member [00:52:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [01:04:04] 06Labs, 10Tool-Labs, 10Mail: Move tools-mail to trusty - https://phabricator.wikimedia.org/T96299#2991931 (10Andrew) [01:04:06] 06Labs, 10Tool-Labs, 10Mail: Attempt to clone tools-mail and dist-upgrade it - https://phabricator.wikimedia.org/T156051#2991928 (10Andrew) 05Open>03Resolved a:03Andrew This worked fine, much to my surprise. [01:37:42] PROBLEM - Puppet run on tools-exec-1411 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [01:54:08] !log social-tools deleted social-tools1 instance that was precise and not used (T143349) [01:54:10] ashley: ^^ [01:54:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Social-tools/SAL [01:54:12] T143349: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349 [01:54:34] legoktm: cool, thanks :) [01:55:00] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#2992032 (10Legoktm) [02:17:42] RECOVERY - Puppet run on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [02:24:00] 06Labs, 10Tool-Labs-tools-Other, 06Community-Tech-Tool-Labs, 06Developer-Relations, and 3 others: Create an authoritative and well promoted catalog of Wikimedia tools - https://phabricator.wikimedia.org/T115650#2992063 (10srishakatux) [02:25:46] 10Tool-Labs-tools-Pageviews: Add % mobile to Topviews - https://phabricator.wikimedia.org/T156081#2992066 (10MusikAnimal) 05Open>03Resolved a:03MusikAnimal [02:50:49] madhuvishy: yeah [02:52:04] 06Labs, 10Horizon, 06Operations, 13Patch-For-Review, 07Puppet: Puppet tab in Horizon unusably slow - https://phabricator.wikimedia.org/T149589#2992084 (10Andrew) Updates: * The cache is working properly * Rendering in version Mitaka is much faster, so this will be less of an issue as soon as we're abl... [03:17:51] 06Labs, 10Horizon, 06Operations, 13Patch-For-Review, 07Puppet: Puppet tab in Horizon unusably slow - https://phabricator.wikimedia.org/T149589#2992120 (10scfc) I'm not particular fond of that idea (but can offer no alternative) because then "click here to test a role" becomes a) "submit a patch to add fi... [04:18:01] 06Tool-Labs-standards-committee: Figure out how communications and meetings will work for the Tool Labs standards committee - https://phabricator.wikimedia.org/T156075#2992174 (10Huji) For private chats, I created #wikimedia-labs-standards-committee and gave all of you access. It is invite only (you might want t... [04:18:59] sorry did anyone talk to me? I accidentally closed the window [05:14:33] 10Tool-Labs-tools-Other, 06Wikisource, 06Wikisource-Community-User-Group: ia-upload convert pdf to djvu and uploaded as 0 × 0 pixels file - https://phabricator.wikimedia.org/T142939#2992215 (10Samwilson) 05Open>03Invalid This is being tracked at https://github.com/wikisource/ia-upload/issues/11 [05:44:58] !log tools.precise-tools Deployed 83085e4 [05:45:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.precise-tools/SAL [06:11:12] !log tools.precise-tools Deployed 38ea551; webservices that switch to trusty now drop off the list! [06:11:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.precise-tools/SAL [06:15:28] 06Labs, 10Tool-Labs, 06Tool-Labs-standards-committee: Create a wall for tools migration to trusty - https://phabricator.wikimedia.org/T156386#2992292 (10bd808) >>! In T156386#2991462, @zhuyifei1999 wrote: > Ah yeah, I migrated all mine. But that's not a week ago yet. :P > > I'm considering whether grid stat... [06:23:18] !log tools.precise-tools Deployed 067c93a Fix cache expiration [06:23:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.precise-tools/SAL [06:26:55] !log tools.precise-tools Added cron to update list data hourly [06:26:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.precise-tools/SAL [06:46:08] RECOVERY - Free space - all mounts on tools-exec-1221 is OK: OK: tools.tools-exec-1221.diskspace._public_dumps.byte_percentfree (No valid datapoints found) [07:34:44] 06Labs, 10Tool-Labs, 10Mail: Move tools-mail to trusty - https://phabricator.wikimedia.org/T96299#2992325 (10yuvipanda) 05Open>03Resolved a:03yuvipanda w00t, we were able to do the in-place upgrade for it:D [07:34:46] 06Labs, 10Tool-Labs, 07Epic: Phase out precise instances from Tool Labs - https://phabricator.wikimedia.org/T94790#2992328 (10yuvipanda) [07:39:06] Hey, I can't connect to the new instance I just created: wikidata-lexeme.wikidata-dev.eqiad.wmflabs it asks for password and I have done soft reboot and hard reboot, and waited for hours but still no success [07:39:12] is it a known issue [08:20:49] @seen EranRoz [08:20:49] zhuyifei1999_: Last time I saw eranroz they were quitting the network with reason: Remote host closed the connection N/A at 2/1/2017 11:06:11 PM (9h14m38s ago) [08:24:52] How can I be sure my tool is not on precise? (I don't trust https://tools.wmflabs.org/precise-tools/ cuz my tool is not listed on there and I still got the email) [08:25:17] 06Tool-Labs-standards-committee: Figure out how communications and meetings will work for the Tool Labs standards committee - https://phabricator.wikimedia.org/T156075#2992447 (10zhuyifei1999) >>! In T156075#2992174, @Huji wrote: > For private chats, I created #wikimedia-labs-standards-committee and gave all of... [08:30:04] revi: yeah the email and the dashboard use somewhat different algorithms [08:30:49] I run the commands without -l release=blah, do I really need to put the trusty forcibly? [08:31:06] default should point to trusty [08:31:35] (I use jstart only, no webservice) [08:31:53] hmm [08:32:07] wait, let me put on a full list [08:32:24] I just gave the force trusty but anyway sure [08:32:29] (the one the email uses) [08:32:35] kk [08:32:41] the toolname is tc-rc [08:44:10] !log tools.precise-tools Deploying daed824 [08:44:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.precise-tools/SAL [08:46:55] revi: https://tools.wmflabs.org/precise-tools/?all <= start-tc 1 2017-01-30 12:01 [08:47:02] hmmmm [08:47:11] the time of the precise job exit [08:47:12] then jstart is still defaulting to precise [08:47:22] hmm is it? [08:47:23] oh that's the last exit? [08:47:33] I thought that date to be last 'start' [08:47:43] yeah last exit [08:47:46] then k [08:48:11] I use jstart for that embedded data bot and it's on trusty [08:48:38] I was just curious why my bot landed on precise [08:49:10] when I was using default jstart -mem 512m (file) [08:51:02] hmm weird [08:51:43] btw, if you want to check if it's on precise or trusty, just check qstat [08:51:56] tools-exec-NNNN [08:52:57] if NNNN is 12NN it's precise (codename for ubuntu 12.04), if 14NN it's trusty (codename for for ubuntu 14.04) [08:53:17] !log tools.precise-tools Deploying d1c65b5 [08:53:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.precise-tools/SAL [10:16:13] 06Tool-Labs-standards-committee: Figure out how communications and meetings will work for the Tool Labs standards committee - https://phabricator.wikimedia.org/T156075#2992596 (10Ladsgroup) I'm `Amir1` as mentioned in my user profile here (@Ladsgroup) and [[ https://meta.wikipedia.org/wiki/User:Ladsgroup| everyw... [10:17:26] Amir1: yeah, but I don't see your cloak, only a hostmask [10:17:57] you can /cs invite yourself though [10:18:00] I should've been identified [10:18:47] /whois Amir1 => uid102662@gateway/web/irccloud.com/x-jolztxiiatfzvmry [10:19:33] Yeah, but no one else can pick this nick [10:20:39] anyways, huji set a chanserv access so you can join after /cs invite #wikimedia-labs-standards-committee [10:21:01] well, anyone can, for 30 seconds... [10:24:35] zhuyifei1999_: I'm in [11:03:31] 06Tool-Labs-standards-committee: Figure out how communications and meetings will work for the Tool Labs standards committee - https://phabricator.wikimedia.org/T156075#2992760 (10eranroz) I'm eranroz as mentioned in phabricator profile and in [[https://meta.wikimedia.org/wiki/User:%D7%A2%D7%A8%D7%9F|metawiki]] [11:05:05] @seen eranroz [11:05:05] zhuyifei1999_: Last time I saw eranroz they were quitting the network with reason: Remote host closed the connection N/A at 2/1/2017 11:06:11 PM (11h58m54s ago) [11:16:01] 06Tool-Labs-standards-committee: Figure out how communications and meetings will work for the Tool Labs standards committee - https://phabricator.wikimedia.org/T156075#2992763 (10zhuyifei1999) For clarification, I meant https://meta.wikimedia.org/wiki/IRC/Cloak. The standard practice is [[https://en.wikipedia.or... [13:13:57] (03PS1) 10Volans: Add missing dummy secrets from production [labs/private] - 10https://gerrit.wikimedia.org/r/335643 [13:17:43] (03CR) 10Volans: "I've found them doing a full puppet compiler:" [labs/private] - 10https://gerrit.wikimedia.org/r/335643 (owner: 10Volans) [13:19:31] !log tools.precise-tools Deploying 8ff1214 [13:19:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.precise-tools/SAL [13:24:04] Amir1: saw a message from you in backscroll, it's not a known issue for an instance to be stuck in that state. usually that means puppet is not running there correctly to seed the instance. What instance and project is it? [13:25:09] !log tools.precise-tools Deploying 35bfbdc [13:25:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.precise-tools/SAL [13:31:55] !log tools.precise-tools Deploying 8980803 [13:31:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.precise-tools/SAL [13:34:51] 10Tool-Labs-tools-Other: ptwikis/Filters: Incorrect naming of projects - https://phabricator.wikimedia.org/T157025#2993066 (10MarcoAurelio) [14:00:44] (03CR) 10Filippo Giunchedi: [C: 031] Add missing dummy secrets from production [labs/private] - 10https://gerrit.wikimedia.org/r/335643 (owner: 10Volans) [14:02:11] 06Labs, 10Tool-Labs, 06Tool-Labs-standards-committee: Create a wall for tools migration to trusty - https://phabricator.wikimedia.org/T156386#2993148 (10Huji) Should we add a link to this wall on https://wikitech.wikimedia.org/wiki/Tools_Precise_deprecation ? And should we expand that page to clearly state w... [14:25:50] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Labs instance ci-jessie-wikimedia-498353 can not be deleted - https://phabricator.wikimedia.org/T156636#2993248 (10hashar) ci-jessie-wikimedia-498353 eventually got deleted so the original issue is solved. Maybe when nova-api got restar... [14:36:13] 06Labs, 10Tool-Labs, 06Tool-Labs-standards-committee: Create a wall for tools migration to trusty - https://phabricator.wikimedia.org/T156386#2993259 (10zhuyifei1999) >>! In T156386#2993148, @Huji wrote: > Should we add a link to this wall on https://wikitech.wikimedia.org/wiki/Tools_Precise_deprecation ? +... [14:36:16] 06Labs, 10Horizon, 06Operations, 13Patch-For-Review, 07Puppet: Puppet tab in Horizon unusably slow - https://phabricator.wikimedia.org/T149589#2993261 (10Andrew) @scfc part of why I'm conflicted about this issue is that wmf Ops keep asserting that no one would ever use this gui to discover roles that the... [14:55:15] 06Labs, 10Tool-Labs, 06Tool-Labs-standards-committee: Create a wall for tools migration to trusty - https://phabricator.wikimedia.org/T156386#2993341 (10Huji) >>! In T156386#2993259, @zhuyifei1999 wrote: > I don't think I get what you meant exactly by "what it takes", but more docs are usually good :) (See a... [16:06:44] 06Labs, 10Tool-Labs, 10Continuous-Integration-Config, 13Patch-For-Review: operations/software/tools-webservice (and operations/software/tools-manifest?) do not run Debian tests - https://phabricator.wikimedia.org/T156651#2982620 (10hashar) I have added a non voting job to build the package. Unfortunately i... [16:20:53] 06Labs, 10Tool-Labs, 10Continuous-Integration-Config, 13Patch-For-Review: operations/software/tools-webservice (and operations/software/tools-manifest?) do not run Debian tests - https://phabricator.wikimedia.org/T156651#2993714 (10hashar) I fixed the instance that lacked a proper base image for Trusty. D... [16:20:57] 06Labs, 10Tool-Labs, 10Continuous-Integration-Config, 13Patch-For-Review: operations/software/tools-webservice (and operations/software/tools-manifest?) do not run Debian tests - https://phabricator.wikimedia.org/T156651#2993715 (10hashar) I fixed the instance that lacked a proper base image for Trusty. D... [16:54:26] Hello all. [16:54:38] o/ JustBerry [16:54:45] Nothing shows up at (https://toolsadmin.wikimedia.org/tools/id/editcompare) or (https://toolsadmin.wikimedia.org/tools/id/purger) [16:55:14] If there's nothing in those tools, could I delete them? I can't really see any files associated with them in my ssh [16:55:17] Those are the config pages in the admin console. What are you actually looking for? [16:55:28] https://tools.wmflabs.org/editcompare/ says no webservice is running [16:55:44] same for https://tools.wmflabs.org/purger/ [16:55:51] bd808: In my ssh, though, I can't see any of my files. [16:55:56] I'm trying to "sort out" my labs account right now [16:56:11] Get an assessment of what I can trash, create another tool if need be, and get rolling so to speak [16:56:50] After ssh into the tools bastion, you need to run `become editcompare` or `become purger` to be switched to the shared tool account [16:57:25] each tool account is a special unix user that anyone who has been added as a maintainer can access [16:57:36] bd808: do I have to logout after becoming to become another tool? [16:58:19] yes, you need to logout of the tool's account to get back to yours and then you can become the next tool [16:59:00] so become tool-name-here; do some stuff; logout; become some-other-tool [16:59:12] * bd808 has a meeting now [16:59:53] bd808: sounds good, thanks [16:59:55] MaxSem: o/ [17:00:03] JustBerry: for the question you asked in PM about deleting a tool... we have this tracking bug -- T133777 [17:00:04] T133777: Tools that should get deleted (tracking) - https://phabricator.wikimedia.org/T133777 [17:00:09] ? [17:00:26] just being friendly as you joined I think MaxSem [17:00:33] hey:) [17:12:41] 06Labs, 10Tool-Labs, 07Tracking: Delete tool named `learning` - https://phabricator.wikimedia.org/T157057#2993909 (10JustBerry) [17:12:52] bd808: ^^ done [17:13:22] 06Labs, 10Tool-Labs, 07Tracking: Delete tool `learning` - https://phabricator.wikimedia.org/T157057#2993924 (10JustBerry) [17:46:59] 06Labs, 10Tool-Labs: Delete tool `learning` - https://phabricator.wikimedia.org/T157057#2994098 (10bd808) [17:51:19] 06Labs, 10Tool-Labs, 07Tracking: Tools that should get deleted (tracking) - https://phabricator.wikimedia.org/T133777#2242852 (10JustBerry) @TerraCodes Should other subtasks (like you did for T133778) also have tags changed from Tool-Labs to Tool-Labs-tools-Other? [18:00:21] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Investigation: Plan for rewriting XTools - https://phabricator.wikimedia.org/T154551#2994143 (10MusikAnimal) Here's what I've got, mostly copied from @Matthewrbowker's comment: **What are the current components of XTools?** (striked have already been rewritte... [19:11:10] jdlrobson: what's up? 2fa problems? [19:11:33] bd808: Having problems with 2FA on labs (cc tgr) :( I changed my phone recently and was using the Google Authenticator app.. which now doesn't have the account on it... so im locked out of my account I think? [19:11:38] I need to schedule a SWAT deploy [19:11:49] (new phone doesnt have the wikitech account in authenticator) [19:11:54] and there are no alternative ways to login [19:12:18] jdlrobson: do you still have the old phone? [19:12:50] jdlrobson: ah. let me find the page with the emergency reset procedure [19:13:26] jdlrobson: https://wikitech.wikimedia.org/wiki/Password_reset#Reset_two_factor_authentication [19:13:53] Using the "write a request on a labs server" works pretty well [19:13:53] bd808: tgr has suggested i create a ticket [19:14:04] I can handle it for you here and now :) [19:14:08] gonna have issues with my staff wikipedia account too.. :/ new phones dont port authenticator accounts [19:14:12] just help be be sure you are you [19:14:27] *help me [19:14:35] we are two meters away so I can vouch for him :) [19:14:53] is that enough? [19:14:58] tgr: :) you can do that labswiki db edit then too [19:15:01] you could both be imposters [19:15:03] i can vouch that tgr can vouch for me [19:15:05] in all seriousness [19:15:11] will do [19:15:16] usually for staff we hop on a hangout [19:15:16] tgr: ssh to silver and then do it [19:15:39] I just figured we need a written record in Phab but won't insist [19:16:06] we'll !log it for entry in the SAL as a record [19:16:15] "Jon (WMF)" and wikitech jdlrobson [19:18:50] chasemp: how about a gif? [19:19:06] highly prized :) [19:19:50] https://usercontent.irccloud-cdn.com/file/xzkA7Yyf/P_20170202_111555.gif [19:20:08] ^ chasemp PROOF [19:21:23] Hi, im wondering could i have some help with floating IP please? The phabricator project had a floating ip but when i try to add it, i get this error Error: Unable to associate floating IP. [19:22:48] jdlrobson: I"m guessing bd808 is helping you out of band this channel but let me just say, I see no newspaper with todays date on that gif ;) [19:23:21] chasemp: does wikitech have its own SAL project or should I just use the main one? [19:23:57] in -ops is ideal for something like this, esp with wikitech creds tied to gerrit and etc [19:24:54] thanks tgr [19:25:02] you just didnt ask... https://usercontent.irccloud-cdn.com/file/EqMn7Qzm/P_20170202_112338.gif [19:25:21] it shows the project has 1 out of 1 floating ip's, just i carn't seem to add it to one of the instances. [19:25:25] jdlrobson: my favorite moment of the day thanks [19:25:30] Anyone know how I can only display unique usernames from this example? https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&format=json&uselang=user&prop=revisions&titles=User%3AJustBerry&rvprop=user&rvlimit=max [19:25:34] i think i had this problem before with another project. [19:25:44] paladox: we are all heading into a meeting atm so it may take a bit [19:25:51] Didn't see an API call for it, but just thought I might ask. [19:25:53] ok [19:26:05] tgr: use the main sal. We have a tools sal but not a general labs one [19:27:46] Danny_B: "(~danny_b@internal-server-nat.wmflabs.org) joined the channel" -- Do you run a bouncer in Labs? [19:29:44] should be done [19:30:03] thanks trg [19:30:07] err tgr [19:32:36] chasemp: gifs keep the world running [19:38:49] thanks tgr bd808 chasemp - can get in now :) [19:40:48] JustBerry: prop=contributors? [19:41:42] tgr: wew, thanks [20:33:20] PROBLEM - Puppet run on tools-docker-builder-03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [20:39:04] !log tools import docker-engine 1.11.2 (currently running version) and 1.12.6 (latest version) into aptly [20:39:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:40:03] 06Labs, 06Operations, 07kubernetes: docker-engine pulled into our repositories only keeps the latest version - https://phabricator.wikimedia.org/T153416#2994611 (10yuvipanda) I just manually imported docker-engine version tools is using into aptly, and it all seems to be ok. [20:43:21] RECOVERY - Puppet run on tools-docker-builder-03 is OK: OK: Less than 1.00% above the threshold [0.0] [20:59:02] 06Labs, 10Tool-Labs, 10Continuous-Integration-Config, 13Patch-For-Review: operations/software/tools-webservice (and operations/software/tools-manifest?) do not run Debian tests - https://phabricator.wikimedia.org/T156651#2994645 (10scfc) >>! In T156651#2993715, @hashar wrote: > […] > Looks like it is all h... [21:51:54] PROBLEM - Puppet run on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [22:06:45] 06Labs, 10Tool-Labs: zoomviewer is using 837G out of 8T for Tools - https://phabricator.wikimedia.org/T136190#2326298 (10madhuvishy) Hi @dschwen, zoomviewer is currently using 408G/8T of tools NFS storage. Could you consider cleaning up old data, and also revisiting adding the cleanup command to the tools cron... [22:06:53] 06Labs, 10Tool-Labs, 10Continuous-Integration-Config, 13Patch-For-Review: operations/software/tools-webservice (and operations/software/tools-manifest?) do not run Debian tests - https://phabricator.wikimedia.org/T156651#2994837 (10hashar) tools-webservice now has a the debian-glue job voting. For tools-m... [22:07:17] 06Labs, 10Tool-Labs, 10Continuous-Integration-Config, 13Patch-For-Review: operations/software/tools-webservice (and operations/software/tools-manifest?) do not run Debian tests - https://phabricator.wikimedia.org/T156651#2982620 (10hashar) p:05Triage>03Normal [22:10:20] 06Labs, 10Tool-Labs: templatetiger is using 613G in Tools out of 8T - https://phabricator.wikimedia.org/T136192#2994846 (10madhuvishy) Hi @Kolossos, templatetiger is currently using 868G/8T of tools NFS storage. This is pretty significant usage, could you consider cleaning the directory up? [22:16:55] 06Labs, 10Tool-Labs: Delete tool massaction - https://phabricator.wikimedia.org/T157079#2994857 (10madhuvishy) [22:26:54] RECOVERY - Puppet run on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [22:33:39] 06Labs, 10Tool-Labs: zoomviewer is using 837G out of 8T for Tools - https://phabricator.wikimedia.org/T136190#2994927 (10dschwen) Done. I put the find command into a small script and added it to the crontab (via jsub). [22:35:38] 06Labs, 10Tool-Labs: Delete tool massaction - https://phabricator.wikimedia.org/T157079#2994943 (10Addshore) All data can be deleted :) Thanks! [22:35:49] 06Labs, 10Tool-Labs, 10MassAction, 15User-Addshore: Delete tool massaction - https://phabricator.wikimedia.org/T157079#2994944 (10Addshore) [22:51:41] 06Labs, 10Tool-Labs, 10MassAction, 15User-Addshore: Delete tool massaction - https://phabricator.wikimedia.org/T157079#2994990 (10bd808) [22:51:43] 06Labs, 10Tool-Labs, 07Tracking: Tools that should get deleted (tracking) - https://phabricator.wikimedia.org/T133777#2994989 (10bd808) [22:53:00] 06Labs, 10Tool-Labs, 10MassAction, 15User-Addshore: Delete tool 'massaction' - https://phabricator.wikimedia.org/T157079#2994995 (10Addshore)