[02:16:40] we aren't experiencing any NFS issues right now, are we? [02:23:02] my bot stopped running today at 17:40 GMT [02:23:16] 6Labs, 10Tool-Labs: Permission issues and/or failure to load Ruby environment on trusty - https://phabricator.wikimedia.org/T106170#1563055 (10MusikAnimal) This started happening again today. Same errors as reported above, and again I'm able to execute the script directly without any issues. The last time it r... [02:23:27] ^ there's the phab report [02:24:53] I guess this is what I get for using Ruby :( [02:27:10] aha! just heard that DefconBot was down, it last ran at 17:30 https://en.wikipedia.org/w/index.php?title=Special%3AContributions&target=DefconBot [02:27:20] around the same time my bot went down [02:33:02] 6Labs, 10Tool-Labs, 5Patch-For-Review: Make sure gridengine-exec starts on boot - https://phabricator.wikimedia.org/T109728#1563069 (10scfc) a:3scfc [02:43:27] 6Labs, 10Tool-Labs: Labs_lvm::Volume[separate-tmp] is noisy on execution hosts - https://phabricator.wikimedia.org/T109933#1563078 (10scfc) 3NEW a:3scfc [04:06:08] 6Labs, 10Tool-Labs: Permission issues and/or failure to load Ruby environment on trusty - https://phabricator.wikimedia.org/T106170#1563104 (10scfc) When I look at `~tools.musikbot/exec.err`, the last errors are: ``` Exception `Errno::EACCES' at /data/project/musikbot/.rbenv/versions/2.2.1/lib/ruby/2.2.0/logg... [04:12:30] 6Labs, 10Tool-Labs: Permission issues and/or failure to load Ruby environment on trusty - https://phabricator.wikimedia.org/T106170#1563111 (10scfc) And a closer look at `~tools.musikbot/exec.err` shows multiple instances of: ``` Exception `LoadError' at /data/project/musikbot/.rbenv/versions/2.2.1/lib/ruby/2... [13:35:10] !log heritage After backporting all local changes to Gerrit, updating local checkout to latest Git version. [13:35:11] heritage is not a valid project. [13:35:39] !log tools.heritage After backporting all local changes to Gerrit, updating local checkout to latest Git version. [13:35:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master [13:44:59] Hi! I wanted to know if anyone here has used nodejs on tool labs. I'm having some trouble with it. [16:40:20] 6Labs, 3Labs-Sprint-107, 3Labs-Sprint-108, 3Labs-Sprint-109: Evaluate kubernetes for use on Tool Labs - https://phabricator.wikimedia.org/T107993#1563642 (10scfc) Question of understanding: I have only watched a Kubernetes talk once and IIRC the setup was a central file that defined how many containers of... [16:57:13] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Samtar was created, changed by Samtar link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Samtar edit summary: Created page with "{{Tools Access Request |Justification=Working on The Wikipedia Library's library card system |Completed=false |User Name=Samtar }}" [17:09:55] 6Labs, 6Release-Engineering: Cleanup quaity-assurance labs project - https://phabricator.wikimedia.org/T108087#1563652 (10zeljkofilipin) I am not using them, as far as I am concerned, delete them at will. [17:14:30] 6Labs, 6Release-Engineering: Cleanup quaity-assurance labs project - https://phabricator.wikimedia.org/T108087#1563655 (10hashar) All three instances have been created back in March 2015. The project administrators are @ori and @cmcmahon . No idea what that project might have been, probably safe to delete. [17:27:00] 6Labs, 10Wikimedia-Site-Requests, 10wikitech.wikimedia.org: Investigate the creation of a wiki for the dev community that is organizing around Labs - https://phabricator.wikimedia.org/T70818#1563669 (10Krenair) [17:31:32] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Samtar was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=174577 edit summary: [19:02:19] Hello all, I need to know what the best place would be to request the creation of a Labs tool. [19:02:35] In this case, the conversion of a set of command-line scripts for web use. [19:03:11] you want to request a developer builds something for you? [19:03:17] or just the space to run it yourself in labs? [19:04:01] I got a space on Labs and played around with it a bit. [19:04:33] What I realized was that I don't have the time or motivation to deal with servers and rewriting the scripts...I don't think it's difficult, per se, but it requires someone with more experience than me. [19:05:04] So basically I'm asking for the best place to contact experienced web-ops who can do the legwork. [19:07:01] The tools are three: importer scripts, written in Python and meant to be run via the command line, which assist with frequent tasks at the Wikipedia Signpost. [19:07:07] So far I've been running them locally. [19:07:32] 6Labs, 3Labs-Sprint-107, 3Labs-Sprint-108, 3Labs-Sprint-109: Evaluate kubernetes for use on Tool Labs - https://phabricator.wikimedia.org/T107993#1563795 (10yuvipanda) >>! In T107993#1563642, @scfc wrote: > Question of understanding: I have only watched a Kubernetes talk once and IIRC the setup was a centr... [19:07:39] But I won't be doing it forever and the people running the Signpost are not technically experienced enough to run anything via the command line. Clearly putting it up online is the only tenable solution. [19:08:38] Anyway judging my the silence I'm guessing there aren't any direct methods? [19:09:27] I know there are pages for bot requests, a page for lab tools requests doesn't seem out of the ballpark. [19:11:00] ResMar: yes, there are no direct methods. [19:11:05] ResMar: you can try emailing the labs-l list to check [19:12:23] Yeah, alright. That sucks. [19:13:06] Argh. There's no way these guys could power through installing pywikibot. [19:13:08] I barely managed it. [19:13:29] Alright, nothing to be done, thanks. [19:13:46] huh [19:41:56] 6Labs: labstore1002 out of space in vg to create new snapshots - https://phabricator.wikimedia.org/T109954#1563836 (10yuvipanda) 3NEW [20:12:17] 6Labs: labstore1002 out of space in vg to create new snapshots - https://phabricator.wikimedia.org/T109954#1563850 (10yuvipanda) I deleted the old snapshots, should be space now. I'll wait for the next runs to pick up to see what happens. [20:13:18] 6Labs, 7Blocked-on-Operations: labstore1002 out of space in vg to create new snapshots - https://phabricator.wikimedia.org/T109954#1563836 (10yuvipanda) [20:13:37] 6Labs, 6operations: labstore1002 out of space in vg to create new snapshots - https://phabricator.wikimedia.org/T109954#1563855 (10yuvipanda) [20:28:36] 6Labs, 6operations: labstore1002 out of space in vg to create new snapshots - https://phabricator.wikimedia.org/T109954#1563887 (10yuvipanda) So this also means backups have been broken for about a week. [20:29:05] 6Labs, 6operations: labstore1002 out of space in vg to create new snapshots - https://phabricator.wikimedia.org/T109954#1563888 (10yuvipanda) p:5Triage>3High [21:25:16] 6Labs, 10Tool-Labs: Decommission tools-exec-catscan - https://phabricator.wikimedia.org/T109871#1564317 (10scfc) 5Open>3Resolved [21:25:17] 6Labs, 10Tool-Labs: document the need and usage patterns for special exec hosts - https://phabricator.wikimedia.org/T99067#1564319 (10scfc) [22:26:01] 6Labs, 5Patch-For-Review, 7Puppet: Could not find data item labs_recursor - https://phabricator.wikimedia.org/T107205#1564451 (10scfc) 5Open>3Resolved a:3scfc The linked patch should have fixed this bug for new self-hosted puppetmasters; on your existing instance (worst case) this should require `sudo... [22:57:35] 6Labs, 3Labs-Sprint-104, 3Labs-Sprint-105, 7Puppet: Allow per-host hiera overrides via wikitech - https://phabricator.wikimedia.org/T104202#1564500 (10scfc) a:3scfc [23:16:25] 6Labs, 10Wikimedia-Site-Requests, 10wikitech.wikimedia.org: Investigate the creation of a wiki for the dev community that is organizing around Labs - https://phabricator.wikimedia.org/T70818#1564551 (10MZMcBride) >>! In T70818#727452, @MZMcBride wrote: > I strongly agree with comment 4. A year later, I stil...