[00:14:34] <mwalker>	 Coren, are you someone to ping if I want additional resources added to a quota?
[00:14:39] <mwalker>	 *in labs general
[00:14:54] <Coren>	 I'm one of  'em.
[00:14:58] <Coren>	 Whatcha need?
[00:15:07] <mwalker>	 I need an additional instance in the deployment-prep project
[00:15:12] <mwalker>	 we're maxed out
[00:15:42] <mwalker>	 turns out betalabs is quite big
[00:34:42] <Coren>	 mwalker: It is.  Just an instance?  You're okay with ram and CPUs?
[00:35:09] <mwalker>	 yepyep; looks like we have decent headroom for the moment
[00:36:30] <mwalker>	 17 free cores; and 90GB free ram
[00:36:46] <Coren>	 Gave you two.  Use them wisely, grasshopper.
[00:36:51] <mwalker>	 heh
[00:36:54] <mwalker>	 thanks :)
[01:01:35] <Betacommand>	 COREN
[01:01:52] <Coren>	 I doubt the all caps are necessary.  What's up?
[01:02:11] <Betacommand>	 Coren: jlocal not found
[01:03:05] <Coren>	 I didn't get the time to do a build of the package today.  Lemme dump a symlink in /usr/bin for the time being.
[01:03:46] <Coren>	 {{done}}.  It's crappy, but it'll get overwritten by the package anyways.
[01:04:02] <Betacommand>	 thanks
[02:41:46] <DispenserAFK>	 Why is F4 in vim somehow mapped to delete the first line of the file?  Its not my .vimrc that's causing it
[02:48:54] <Dispenser>	 Stupid shit wrong in labs -_-
[03:25:36] <Dispenser>	 `vim -u NONE` skips all vimrc files, is the binary borked? F2-F5 are broken for me :-(
[03:29:47] <Dispenser>	 Ok, so PuTTY was in Linux mode no ESC[n~ for the function keys
[10:26:20] <Steinsplitter>	 Coren: ping
[11:30:52] <wikibugs>	 3Wikimedia Labs / 3tools: Removal of shell access [User:The Herald] - 10https://bugzilla.wikimedia.org/64792 (10Steinsplitter) 3NEW p:3Unprio s:3enhanc a:3Marc A. Pelletier The user is a hatcollector, pleas remove his shell access. Pleas delete also the nonsense vm in bastion (local-theherald).  http...
[11:46:47] <phe>	 guillom tu n'avais pas eu ce pb il y a quelques temps ? https://fr.wikipedia.org/wiki/Discussion_utilisateur:Phe#Badmood_sous_IP
[11:47:55] <phe>	 errm, c'était Gribeco, pas toi
[12:38:21] <wikibugs>	 3Wikimedia Labs: Removal of shell access [User:The Herald] - 10https://bugzilla.wikimedia.org/64792#c1 (10Tim Landscheidt) p:5Unprio>3Lowest a:5Marc A. Pelletier>3None I don't know if we care that much about hat collectors until they actually cause disturbance :-).  He also didn't create a VM in Bastio...
[12:39:20] <wikibugs>	 3Wikimedia Labs / 3wikitech-interface: Project Bastion has a service group "local-rezabot" - 10https://bugzilla.wikimedia.org/62537#c1 (10Tim Landscheidt) And now there are three (cf. also bug #64792):  | dn: cn=bastion.local-cobain,ou=servicegroups,dc=wikimedia,dc=org |         objectClass: groupofnames |...
[13:34:05] <Qcoder00>	 Hi
[13:34:20] <Qcoder00>	 catsan2 is currently offline... Reason?
[13:52:14] <Betacommand>	 Qcoder00: link?
[13:56:40] <Qcoder00>	 http://tools.wmflabs.org/catscan2/catscan2.php
[13:56:46] <Qcoder00>	 Gives an error
[14:03:52] <Betacommand>	 Qcoder00: have you tried contacting him?
[14:04:11] <Betacommand>	 I know my web service has died a few time recently for some unknown reason
[14:04:51] <wikibugs>	 3Wikimedia Labs: Removal of shell access [User:The Herald] - 10https://bugzilla.wikimedia.org/64792#c2 (10Marc A. Pelletier) a:3Andrew Bogott It's harmless, but I agree that the ability to create service groups for non-admins of a project probably should be selectable per-project.  Whatcha think, Andrew?
[14:05:06] <Qcoder00>	 I thought I'd ask here in case it was a technical issue with the labs server
[15:39:34] <wikibugs>	 3Wikimedia Labs: Removal of shell access [User:The Herald] - 10https://bugzilla.wikimedia.org/64792#c3 (10Steinsplitter) yeah! let we do unexperienced user on the cluster whatever they want! :)
[15:40:52] <valhallasw>	 Steinsplitter: that's what we are doing already anyway.
[15:41:19] <Steinsplitter>	 bad.
[15:41:59] <Steinsplitter>	 is Tim Landscheidt on irc?
[15:42:27] <Steinsplitter>	 i go to revert his nonsense change on this bug.
[15:42:41] <valhallasw>	 scfc_de is Tim Landscheidt
[15:43:05] <valhallasw>	 Steinsplitter: why is that bad? Just having a Labs account does not let you do a lot, really.
[15:43:20] <wikibugs>	 3Wikimedia Labs: Removal of shell access [User:The Herald] - 10https://bugzilla.wikimedia.org/64792 (10Steinsplitter) 5NEW>3RES/WON
[15:51:06] <wikibugs>	 3Wikimedia Labs / 3wikitech-interface: Project Bastion has a service group "local-rezabot" - 10https://bugzilla.wikimedia.org/62537#c2 (10Steinsplitter) Unused groups should be deleted imho. And it shouldn't be a big thing to delete this groups.
[15:53:10] <valhallasw>	 Steinsplitter, why do you care?
[15:54:02] <Steinsplitter>	 I have an allergy against such people.
[17:23:15] <mwalker>	 Reedy, are you still around?
[17:24:10] <mwalker>	 I'm attempting to set up a mwlib/pdf rendering server in beta labs -- but I don't seem to have sudo on the new instance
[17:34:52] <wikibugs>	 3Wikimedia Labs / 3tools: IRC: Too many user connections (global) - 10https://bugzilla.wikimedia.org/64802 (10Dirk Beetstra) 3NEW p:3Unprio s:3major a:3Marc A. Pelletier My bots run into problems connecting to IRC:  LINKREPORTER 000009 9: returned error: Closing Link: 208.80.155.255 (Too many user co...
[17:39:44] <bd808>	 mwalker: It looks like you aren't in the list of "under_NDA" users for the deployment-prep project
[17:40:04] <mwalker>	 ah; that's fun
[17:40:26] <bd808>	 I'm trying to figure out how that list if made.
[17:42:51] <bd808>	 mwalker: I think I fixed it for you.
[17:43:05] <mwalker>	 ah; yes you did :)
[17:43:18] <bd808>	 !log deployment-prep Added mwalker to under_NDA sudoers group
[17:43:19] <mwalker>	 is there anyone on monday I need to talk to in order to do paperwork?
[17:43:21] <labs-morebots>	 Logged the message, Master
[17:44:44] <bd808>	 mwalker: You've got a user page on officewiki. That's enough paperwork for me :)
[17:44:55] <mwalker>	 heh; fairnuf :p
[17:46:24] <bd808>	 You should follow the steps at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/How_code_is_updated#Converting_a_host_to_use_local_puppetmaster_and_salt_master to join that box with the beta puppet master and salt cluster
[17:48:16] <mwalker>	 yepyep; doing so now :)
[17:54:45] <mwalker>	 !log deployment-prep signed salt key for i-00000339.eqiad.wmflabs (deployment-pdf01)
[17:54:47] <labs-morebots>	 Logged the message, Master
[17:58:21] <mwalker>	 !log deployment-prep Converted i-00000339.eqiad.wmflabs (deployment-pdf01) to use local puppet & salt masters
[17:58:23] <labs-morebots>	 Logged the message, Master
[18:10:12] <mwalker>	 !deployment-prep Updated kernel on deployment-pdf01 (manually set console=ttyS0 to match older installed kernels)
[18:10:12] <wm-bot>	 deployment-prep is a project to test mediawiki at beta.wmflabs.org before putting it to prod
[18:10:28] <mwalker>	 !log deployment-prep Updated kernel on deployment-pdf01 (manually set console=ttyS0 to match older installed kernels)
[18:10:31] <labs-morebots>	 Logged the message, Master
[18:25:05] <wikibugs>	 3Wikimedia Labs / 3tools: IRC: Too many user connections (global) - 10https://bugzilla.wikimedia.org/64802#c1 (10Steinsplitter) s:5major>3normal Only freenode can change this limit. Have you tried it with ipv6?
[18:31:35] <wikibugs>	 3Wikimedia Labs / 3tools: IRC: Too many user connections (global) - 10https://bugzilla.wikimedia.org/64802#c2 (10Merlijn van Deen) Info on requestion exemption:  20:29 < Chiyo> which, if that is the case, you can ask for an iline 20:29 < Chiyo> please e-mail ilines@freenode.net with the IP address range in q...
[18:37:35] <wikibugs>	 3Wikimedia Labs / 3tools: IRC: Too many user connections (global) - 10https://bugzilla.wikimedia.org/64802#c3 (10Dirk Beetstra) (In reply to Steinsplitter from comment #1) > Only freenode can change this limit. Have you tried it with ipv6?  I am not sure how IPV6 would solve this (except that Freenode probab...
[18:38:56] <Beetstra>	 valhallasw: are you Dutch?
[18:39:03] <valhallasw>	 Beetstra: yes.
[18:39:35] <Beetstra>	 wouldn't have guessed that with this username
[18:40:12] <Beetstra>	 Anyway: Bedankt dat je contact hebt opgenomen met Freenode :-)
[18:40:41] <valhallasw>	 I haven't been in contact other than asking for details
[18:41:20] <Beetstra>	 Well, that is a good start
[18:41:59] <Beetstra>	 My bots have been .. strange for a couple of days because of this
[18:42:17] <Beetstra>	 And now I am running a test-instance which simply fails to connect most of the time
[18:42:35] <wikibugs>	 3Wikimedia Labs / 3tools: IRC: Too many user connections (global) - 10https://bugzilla.wikimedia.org/64802#c4 (10Merlijn van Deen) Also, from which host is this?   tools-login is  tools-dev is 208.80.155.132 tools-exec-0y is 208.80.155.14(y-1)   which is probably why I haven't had any issues with IRC bots --...
[18:42:42] <valhallasw>	 Ohh.
[18:42:46] <Beetstra>	 Now I must confess I already use 7 lines
[18:42:52] <valhallasw>	 You're running them in your own project?
[18:43:29] <Beetstra>	 No, they run on the instances .. at the moment it looks like tools-exec-10 is 'full'
[18:44:28] <valhallasw>	 I see, -10 has an anomalous IP
[18:44:30] <Beetstra>	 this bot uses standard 3 (using the 4th for testing), the other three bots are running on 5, 9 and also 10
[18:44:47] <Beetstra>	 Maybe it is 10 only then
[18:44:58] <valhallasw>	 why do you need three + 1 connections?
[18:45:04] <Beetstra>	 flooding
[18:45:15] <valhallasw>	 no, it seems not all exec hosts have an external ip
[18:45:32] <valhallasw>	 I think you can get a flooding exempt, actually. Not 100% sure.
[18:46:16] <Beetstra>	 That would maybe be another solution, good thing of this is that these three lines also keep the load lower, so the bot is more responsive
[18:46:30] <valhallasw>	 right
[18:46:59] <Beetstra>	 (if I pump those ~80 reports per minute through one instance, it has no time left to do something else)
[18:48:03] <Beetstra>	 and that 80 is a 1-day average .. I don't know what peak-speeds are
[18:50:20] <wikibugs>	 3Wikimedia Labs / 3tools: IRC: Too many user connections (global) - 10https://bugzilla.wikimedia.org/64802#c5 (10Dirk Beetstra) Note: the instance giving problems at the moment is tools-exec-10, I don't know however if the observed cases with my other bots were also while that particular bot was running on t...
[18:52:33] <valhallasw>	 basically, -9 and -10 use the same external IP
[18:52:35] <wikibugs>	 3Wikimedia Labs / 3tools: IRC: Too many user connections (global) - 10https://bugzilla.wikimedia.org/64802#c6 (10Merlijn van Deen) Okay, I should have tested further than -01..-04:  tools-exec-01 is 208.80.155.140 tools-exec-02 is 208.80.155.141 tools-exec-03 is 208.80.155.142 tools-exec-04 is 208.80.155.143...
[18:52:42] <valhallasw>	 so that's probably why you are hitting the limit
[18:53:58] <Beetstra>	 Ah
[18:54:32] <Beetstra>	 I am using 5 lines, trying to add number 6 .. probably the standard limit is something like 10, maybe taken by other bots
[18:55:16] <valhallasw>	 Might actually be less. Let me try :-p
[18:55:49] <valhallasw>	 oh, that's not practical.
[18:56:53] <Beetstra>	 I think 09 is running 9 bots .. now I don't know how many are having IRC .. 
[18:58:04] <Beetstra>	 -10 runs something like 11 bots
[19:00:34] <valhallasw>	 I'm not sure what the limit is, but it's > 10
[19:00:38] <valhallasw>	 and then I got bored :-p
[19:01:45] <Beetstra>	 hehe
[19:02:00] <Beetstra>	 ah .. that is maybe why .. my bot and cluebotng are running on the same instance .. 
[19:02:06] <Beetstra>	 My bot has problems keeping up
[19:07:10] <Beetstra>	 Anyway, time to go to bed .. it's 10
[19:07:14] <Beetstra>	 nightnight!
[19:31:05] <wikibugs>	 3Wikimedia Labs / 3tools: IRC: Too many user connections (global) - 10https://bugzilla.wikimedia.org/64802#c7 (10Tim Landscheidt) (In reply to Merlijn van Deen from comment #6) > [...]  > So probably exec-09 and exec-10 are using a centralized labs NAT IP? [...]  Just to confirm: (Fixed) public IPs have been...
[20:25:47] <valhallasw>	 Coren: could you clean this from my Gerrit? https://gerrit.wikimedia.org/r/#/c/102721/