[01:02:48] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/JustBerry was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=1121504 edit summary: 
[01:43:48] <wikibugs>	 06Labs, 06Operations, 13Patch-For-Review: audit labs versus production ssh keys - https://phabricator.wikimedia.org/T108078#2864021 (10RobH) a:05RobH>03None
[04:04:35] <wikibugs_>	 06Labs, 07Puppet: Make changing puppetmasters for Labs instances more easy - https://phabricator.wikimedia.org/T152941#2864105 (10scfc)
[09:14:53] <wikibugs_>	 06Labs, 10Tool-Labs, 06Operations, 10Phabricator, and 2 others: Install Arcanist in toollabs::dev_environ - https://phabricator.wikimedia.org/T139738#2864252 (10hashar) Puppet patch https://gerrit.wikimedia.org/r/#/c/297975/ will add arcanist on tools development environment for Trust and Jessie.  Precise...
[10:34:08] <wikibugs_>	 06Labs, 10Labs-Infrastructure: Please, update Mono to the actual version - https://phabricator.wikimedia.org/T152949#2864391 (10MaxBioHazard)
[11:04:52] <wikibugs_>	 06Labs, 10Labs-Infrastructure: Please, update Mono to the actual version - https://phabricator.wikimedia.org/T152949#2864391 (10hashar) From http://packages.ubuntu.org/mono-complete http://packages.debian.org/mono-complete  | Distribution | `mono-complete` version | Labs? |--|--|-- | Ubuntu Precise | 2.10.8 |...
[11:05:17] <wikibugs_>	 06Labs, 10Labs-Infrastructure, 10Tool-Labs: Please, update Mono to the actual version - https://phabricator.wikimedia.org/T152949#2864443 (10hashar)
[12:51:32] <wikibugs_>	 06Labs, 10Labs-Infrastructure, 10Tool-Labs: Please, update Mono to the actual version - https://phabricator.wikimedia.org/T152949#2864571 (10Reedy)
[12:52:30] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10Tool-Labs: Please, update Mono to the actual version - https://phabricator.wikimedia.org/T152949#2864391 (10Reedy) They do provide built debs for various Debian based versions -- http://www.mono-project.com/docs/getting-started/install/linux/#debian-ubuntu-and-derivatives (no...
[12:54:25] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10Tool-Labs: Update Mono to newer stable release - https://phabricator.wikimedia.org/T152949#2864589 (10Reedy)
[13:21:12] <chasemp>	 zhuyifei1999_: if you have a use case for 100's of gigs please don't drop it on NFS atm, especially Tools relatively limited NFS share, a ticket would be a good start to understanding your storage needs
[13:22:21] <zhuyifei1999_>	 chasemp: it's not tool labs, but http://wikitech.wikimedia.org/wiki/Nova_Resource:Video
[13:24:06] <chasemp>	 zhuyifei1999_: this does the heavy lifting for video2commons then?
[13:24:17] <zhuyifei1999_>	 yeah
[13:24:47] <zhuyifei1999_>	 this thing does the heavy stuffs for that tool
[13:24:50] <chasemp>	 zhuyifei1999_: how do you currently store things now, local disk only?
[13:25:13] <zhuyifei1999_>	 yes, as far as what the tool uses
[13:26:23] <zhuyifei1999_>	 there are stuffs in /data/project that idek what they are
[13:26:43] <chasemp>	 zhuyifei1999_: so stopgap is you could use the scratch share which has the space generally and has been separated from general Tools NFS for just these reasons
[13:27:01] <chasemp>	 but that has a few downsides, namely it's not reserved, and it can still be overwhelmed
[13:27:12] <chasemp>	 but soon as in this week that woudl be OK I imagine to try
[13:27:36] <chasemp>	 but at 3T that hasn't gone above 1T in months
[13:27:44] <chasemp>	 I bet you would be ok until we reasoned out something better
[13:28:06] <zhuyifei1999_>	 would this non-ldapness screw up permissions?
[13:28:26] <chasemp>	 I'm not sure I understand the question, the same ldap-ness should apply
[13:28:39] <chasemp>	 can you elaborate?
[13:29:09] <zhuyifei1999_>	 I mean the service user that stores the files on the NFS is not attached to ldap
[13:29:49] <chasemp>	 it's a local user to video project VMs that generates files that need to be consumed by the toollabs video2commons bot?
[13:29:52] <chasemp>	 is that the gist
[13:30:10] <zhuyifei1999_>	 and would that break file permissions?
[13:30:33] <zhuyifei1999_>	 no, it stores file temporary for server-side upload requests
[13:31:06] <chasemp>	 two things, it shouldn't behave any differently than it does now, and it will create oddness if you don't manage the gid/uid of the generated files
[13:31:08] <zhuyifei1999_>	 the toollabs tool is only a web ui
[13:32:20] <chasemp>	 break file permissions is a broad domain, there are ways to make it work and intra-project you should be fine
[13:34:57] * zhuyifei1999_ just checked, with "$ sudo -- sudo -u v2ccelery touch test" on /data/scratch on encoding01 the generated file has the same uid + gid as tools.admin-beta
[13:36:04] <zhuyifei1999_>	 what's the safe zone for uids? or should I just ldap-ize it with a service user?
[13:36:06] <chasemp>	 one thign I would advise if you are going to go down this road is stop thinking in terms of human readable users and groups, and only in numeric uid/gid
[13:36:29] <zhuyifei1999_>	 hmm
[13:36:38] <chasemp>	 make a service user will make it global to Labs atm so that's the most common sense thing to do I imagine
[13:40:15] <chasemp>	 zhuyifei1999_: I have a question for you, if you don't mind :) say there is a large cache of images/videos and we wanted to upload only things not currently on commons (I'm guessing there is some existing process for this?) what's the best thing? in relation to https://phabricator.wikimedia.org/T152632
[13:40:34] <chasemp>	 it would be great if you could comment there a bit to share wisdom
[13:40:43] * zhuyifei1999_ looks
[13:45:19] <zhuyifei1999_>	 chasemp: I don't get this. If most of the images are from Flickr, we can just go ask a tool like flickr2commons to mine for us right?
[13:46:11] <chasemp>	 very possibly yes
[13:46:12] <zhuyifei1999_>	 (except for videos, which can be handled with video2commons, but it does not handle floods well currently)
[13:47:29] <chasemp>	 this a thing where the researchers who have carries this cache of media forward know almost nothing about tools/labs and I'm trying to bridge the gap here or ask the right ppl but I'm not sure how flickr2commons works now for this
[13:47:39] <chasemp>	 and if the solutions translates 
[13:50:50] * zhuyifei1999_ thinks the interface of https://tools.wmflabs.org/flickr2commons/#interface_language=en is quite intuitive, right?
[13:54:12] <chasemp>	 it is generally, this source is not flickr itself though, only historically 
[13:54:32] <zhuyifei1999_>	 oh
[13:55:07] <chasemp>	 and it's also 100-150T which is /the entire size of commons now/
[13:57:33] <zhuyifei1999_>	 chasemp: I guess https://commons.wikimedia.org/wiki/User:Fae might be able to help a lot. he does a lot of batch uploads
[13:57:45] <zhuyifei1999_>	 *https://commons.wikimedia.org/wiki/User:F%C3%A6
[13:59:27] <zhuyifei1999_>	 and about the size, if it's too crazy to do it client side, idk if a server side batch upload will work, but that will need a lot of preparation
[14:00:07] <zhuyifei1999_>	 https://commons.wikimedia.org/wiki/Help:Server-side_upload
[14:00:40] <zhuyifei1999_>	 this can be similar to https://commons.wikimedia.org/wiki/Help:Server-side_upload#What_to_do_if_files_represent_hundred_of_GB_to_several_TB.3F
[14:01:07] <zhuyifei1999_>	 but that's only several TB, not 100+ TB
[14:02:46] <wikibugs>	 06Labs, 06Operations: Explore hosting the multimedia commons use case - https://phabricator.wikimedia.org/T152632#2864700 (10zhuyifei1999) [[https://commons.wikimedia.org/wiki/Help:Server-side_upload#What_to_do_if_files_represent_hundred_of_GB_to_several_TB.3F|commons:Help:Server-side upload#What to do if file...
[14:24:23] <chasemp>	 yep, thanks zhuyifei1999_
[14:24:32] <zhuyifei1999_>	 np
[15:54:12] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Migrate existing labs users from the old servers, if possible using roles and start maintaining users on the new database servers, too - https://phabricator.wikimedia.org/T149933#2864970 (10jcrespo) Fixed. It turned out that you need GRANT OPTION an...
[16:53:18] <Reedy>	 valhallasw`cloud: Is https://www.mediawiki.org/wiki/Git/Reviewers broken atm?
[16:53:21] <Reedy>	 (well, the bot)
[16:53:27] <valhallasw`cloud>	 Reedy: uh.
[16:53:40] <valhallasw`cloud>	 https://tools.wmflabs.org/gerrit-reviewer-bot/ suggests not
[16:53:47] <Reedy>	 niedzielski is complaining reviewers aren't being added...
[16:54:06] <valhallasw`cloud>	 mmm
[16:54:26] <valhallasw`cloud>	 maybe gerrit changed their api. Let me test
[16:56:38] <valhallasw`cloud>	 ... id_rsa is missing?!
[16:59:33] <valhallasw`cloud>	 hm no, just in a different location
[17:00:47] <valhallasw`cloud>	 "fatal: "04f4bbedf0ff2448de46562917adb39df1a7fbf0" no such change"
[17:00:48] <valhallasw`cloud>	 interesting
[17:02:06] <valhallasw`cloud>	 so for some reason gerrit doesn't accept git hashes anymore
[17:02:23] <Reedy>	 valhallasw`cloud: yeah, it's changed a lot of shit
[17:02:27] <Reedy>	 it removed some of the prefixes too
[17:03:37] <Reedy>	 https://github.com/wikimedia/mediawiki-extensions/commit/558f66816eb9553c94be3454e738ce5ea040f0f2
[17:03:42] <Reedy>	 /r/p to /r
[17:04:13] <paladox>	 Reedy they made submodules securer.
[17:04:19] <Reedy>	 "securer"
[17:04:27] <Reedy>	 security through obscurity
[17:04:31] <paladox>	 Yeh
[17:04:35] <paladox>	 It was a horrible
[17:04:36] <paladox>	 change
[17:04:53] <paladox>	 it makes upgrading harder.
[17:05:48] <valhallasw`cloud>	 oddly enough gerrit_reviewer_bot has not borked on that change
[17:05:53] <Reedy>	 lol
[17:07:44] <valhallasw`cloud>	 oh, because it's just /r/p, not /r/<rest api>/
[17:09:20] <Reedy>	 valhallasw`cloud: have you got a table to flip?
[17:09:35] * valhallasw`cloud fears the worst
[17:09:45] <valhallasw`cloud>	 watskeburt?
[17:19:33] <paladox>	 It seems that it dosent support commits now
[17:19:34] <paladox>	 https://gerrit-review.googlesource.com/Documentation/cmd-set-reviewers.html
[17:19:36] <paladox>	 only change id
[17:20:26] <paladox>	 https://github.com/gerrit-review/gerrit/commit/0be8b71c5cc6c11e060d5ed1c251f6ade445b4e0
[17:20:39] <paladox>	 valhallasw`cloud was broken by ^^
[17:21:03] <valhallasw`cloud>	 sending change-id instead of commit hash is easy enough
[17:21:04] <valhallasw`cloud>	 not a problem
[17:21:29] <paladox>	 ok
[18:48:27] <grrrit-wm>	 (03PS1) 10Yurik: Publish to #wikimedia-interactive [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/326496 
[18:48:41] <chasemp>	 yurik: what is -interactive?
[18:49:04] <yurik>	 chasemp, ?  the name of the team
[18:49:31] <chasemp>	 ok thanks TIL
[18:50:28] <yurik>	 :)
[19:29:50] <jynus>	 apparently there will be some ongoing productio maintenance that could cause extra load on s7 shard databases
[19:30:08] <jynus>	 no impact on labs resources, except that it may lag s7 for some hours
[19:31:15] <jynus>	 More info on: https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20161212T1830
[19:34:43] <chasemp>	 ok thanks jynus
[19:36:12] <grrrit-wm>	 (03CR) 10Alex Monk: Publish to #wikimedia-interactive (031 comment) [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/326496 (owner: 10Yurik) 
[20:10:10] <Cyberpower678>	 yuvipanda how many jobs can a tool run concurrently before jsub begins to queue the rest?
[20:19:21] <valhallasw`cloud>	 Cyberpower678: there are no per-user limits at the moment iirc
[20:19:50] <Cyberpower678>	 really.  Last time jsub only ran 16 at a time and then left the rest queued until one job finished.
[20:20:49] <wikibugs_>	 06Labs: Request increased quota for etytree labs project - https://phabricator.wikimedia.org/T152417#2866173 (10Andrew) I've added a new instance type ('flavor') to your project called 'bigram' with 36Gb of RAM.  I've also increased the project quota to permit creation of one bigram instance in addition to the i...
[20:21:49] <Cyberpower678>	 valhallasw`cloud: ^^
[20:21:53] <valhallasw`cloud>	 the only limit is the voluntary user_slot
[20:22:06] <valhallasw`cloud>	 and if you request enough resources, sure, they will queue
[20:22:12] <valhallasw`cloud>	 and everyone else will also have to wait
[20:22:24] <bd808>	 Cyberpower678: but... please don't test out a denial of service attack against the grid
[20:22:35] <valhallasw`cloud>	 so don't do that
[20:22:37] <Cyberpower678>	 bd808: not my intention.
[20:24:00] <Cyberpower678>	 The reason why the cyberbot exec node existed was because I had too many jobs running concurrently through the numerous bot tasks that were running.  I realized some of them were stuck in the queue state and would never start since all of the running jobs were continuous.
[20:24:39] <Cyberpower678>	 That was at least according to Coren
[20:26:11] <valhallasw`cloud>	 no, the reason for having a single machine is sharing VMEM. You need less memory in total if all jobs using the same libraries are running one one machine than if they are running on different machines.
[20:27:00] <valhallasw`cloud>	 if jobs are stuck in queued state, that means that everyone is stuck in a queued state (well, depending on which resources you're asking for)
[20:29:02] <Cyberpower678>	 valhallasw`cloud I can garauntee you that wasn't the reason.  I had too many jobs running so Coren created a node for me to use where I can queue an unlimited number of jobs until the memory ran out.
[20:30:13] <bd808>	 so, you were breaking everything and Coren made a sandbox where you could only hurt yourself (except for stress on NFS)?
[20:31:31] <valhallasw`cloud>	 https://phabricator.wikimedia.org/T99067#1284789
[20:31:33] <valhallasw`cloud>	 "or because their use pattern renders worst-case allocation very suboptimal."
[20:31:57] <valhallasw`cloud>	 I guess you fall in the latter category then -- maybe your tools allocate a large amount of memory without actually using it actively?
[20:33:02] <Cyberpower678>	 One of my bot scripts did, but I decommissioned it.
[20:33:13] <Cyberpower678>	 Too memory intensive
[20:33:20] <Cyberpower678>	 Too demanding]
[20:39:27] <fifitu>	 asta la casa
[20:39:49] <fifitu>	 !whoise
[20:40:17] <Cyberpower678>	 fifitu English please
[20:40:31] <fifitu>	 chat 0.0.0
[20:54:16] <Cyberpower678>	 fifitu is there anything you need help with?
[21:00:28] <grrrit-wm>	 (03PS2) 10Yurik: Publish to #wikimedia-interactive [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/326496 
[21:06:05] <grrrit-wm>	 (03CR) 10Alex Monk: [C: 032] Publish to #wikimedia-interactive [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/326496 (owner: 10Yurik) 
[21:06:46] <grrrit-wm>	 (03Merged) 10jenkins-bot: Publish to #wikimedia-interactive [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/326496 (owner: 10Yurik) 
[21:11:12] <wikibugs>	 !log tools.wikibugs Updated channels.yaml to: 8ed4192969cedb7c733e0ab57d7362b4ca807951 Publish to #wikimedia-interactive
[21:11:58] <wikibugs_>	 !log tools.wikibugs Updated channels.yaml to: 8ed4192969cedb7c733e0ab57d7362b4ca807951 Publish to #wikimedia-interactive
[21:13:15] <Krenair>	 stashbot, help
[21:13:15] <stashbot>	 See https://wikitech.wikimedia.org/wiki/Tool:Stashbot for help.
[21:13:21] <Krenair>	 bd808, is it broken?
[21:14:00] <valhallasw`cloud>	 Krenair: you're looking for labslogbot I think
[21:14:04] <valhallasw`cloud>	 who isn't here
[21:14:13] <valhallasw`cloud>	 stashbot also logs things, but not to the SAL
[21:14:14] <stashbot>	 See https://wikitech.wikimedia.org/wiki/Tool:Stashbot for help.
[21:14:20] <Krenair>	 honestly at this point I've lost track of what bot does what
[21:14:29] <chasemp>	 same
[21:14:53] <Krenair>	 wasn't that a morebots instance that got replaced by new stashbot functionality?
[21:14:57] <valhallasw`cloud>	 https://tools.wmflabs.org/sal/tools.wikibugs < hrm.
[21:15:05] * valhallasw`cloud shrugs
[21:31:07] <wikibugs_>	 06Labs: Request increased quota for etytree labs project - https://phabricator.wikimedia.org/T152417#2866389 (10Epantaleo) That's great, thank you! I just deleted the old instance.   I hope the project will run indefinitely. As mentioned before, if I manage to reduce the amount of memory needed I'll write a note.