[00:58:35] <anomie>	 Coren: ping
[00:59:47] <Coren>	 anomie: Pong.
[01:01:07] <anomie>	 Coren: I just told AnomieBOT to re-exec itself, and gridengine decided it had to die and sent it SIGUSR2. I don't see how it could have used the whole 256m memory requested when it doesn't any other time; do you have a log file you can check to see why it got killed?
[01:01:36] <Coren>	 anomie: It's rarely very informative, but I can look to see.
[01:01:57] <Coren>	 anomie: You don't happen to have its job number handy, do you?
[01:02:27] <anomie>	 Possibly 1284
[01:02:40] <Coren>	 Found it.  1320
[01:02:57] <anomie>	 Oh, 1284 was the job before last. Yeah, 1320
[01:04:08] <Coren>	 o_O according to the accounting table, your job exited normally with a status of 0.
[01:04:41] <Coren>	 maxvmem      1.115G
[01:04:42] <anomie>	 When it receives SIGUSR2 it decides it's going to be killed and tries to complete an orderly shutdown.
[01:05:34] <Coren>	 Ah, so it did exit orderly.  Maybe it ran away widely when you told it to re-exec?
[01:05:50] <Coren>	 Can I see the code that invokes?
[01:05:56] <anomie>	 I wonder how it managed to get such a high vmem. Especially since I was restarting it since the most memory-intensive process had died due to that MySQL lost connection thing
[01:06:16] <Coren>	 Maybe its restart method has a flaw?
[01:06:26] <Coren>	 BTW, there is a happy fun shortcut if you want a restart:
[01:06:29] <Coren>	 exit(1)
[01:06:32] <Coren>	 :-)
[01:07:22] <Coren>	 As for the mysql lost connection thing, I have a sneaking suspicion I'm going to look at right now.  I'd rather that didn't happen ever.
[01:09:15] <Coren>	 Ah.  Nope.  Mysqld reports no error, nor did it restart.
[01:09:23] <anomie>	 That bit of the code is in /data/project/anomiebot/bot/bot.pl. The child processes are in the loop at lines 96-122, and the master is in 151-188. According to the log, the child processes exited cleanly, then the bot re-execed bot.pl, then it successfully started the child processes (the same way it does every other time). But then it got some strange error about "libgcc_s.so.1 must be installed for pthread_cancel to work" (any pthread usage must
[01:09:23] <anomie>	  be in Perl, AnomieBOT isn't threaded), then it got SIGUSR2.
[01:10:12] <anomie>	 The mysql error would have been around 12 May 03:47:01 UTC, I didn't get a chance to check it until now
[01:10:42] <Coren>	 The libgcc_s.so.1 error is a symptom of hitting the hard memory limit.
[01:12:34] <Coren>	 Oooo.  You don't reap the chidlren!
[01:13:11] <anomie>	 Lines 126-135?
[01:13:48] <Coren>	 Ah, with a handler.
[01:13:50] <Coren>	 Hm.
[01:18:32] <anomie>	 If you want to restart it to test anything, feel free. The script /data/project/anomiebot/restartbot.sh will trigger a restart.
[07:30:44] <petan>	 @requests-on
[07:30:44] <wm-bot>	 Requests were enabled
[07:30:57] <petan>	 @requests
[07:30:57] <wm-bot>	 Warning: There is 1 user waiting for shell: Miguel2706 (waiting 559 minutes) 
[07:35:52] <wm-bot>	 Warning: There is 1 user waiting for shell: Miguel2706 (waiting 564 minutes) 
[07:49:17] <wm-bot>	 Warning: There is 1 user waiting for shell: Miguel2706 (waiting 577 minutes) 
[08:02:42] <wm-bot>	 Warning: There is 1 user waiting for shell: Miguel2706 (waiting 591 minutes) 
[08:03:03] <petan>	 !sa
[08:03:03] <wm-bot>	 There are multiple keys, refine your input: sal, say, 
[08:03:06] <petan>	 !sal
[08:03:06] <wm-bot>	 https://labsconsole.wikimedia.org/wiki/Server_Admin_Log see it and you will know all you need
[08:03:57] <petan>	 !sal del
[08:03:58] <wm-bot>	 Successfully removed sal
[08:04:00] <petan>	 !sal is https://wikitech.wikimedia.org/wiki/Labs_Server_Admin_Log
[08:04:00] <wm-bot>	 Key was added
[08:04:25] <petan>	 Coren is motd on -login broken purposefuly?
[08:16:07] <wm-bot>	 Warning: There is 1 user waiting for shell: Miguel2706 (waiting 604 minutes) 
[08:19:44] <petan>	 !sh
[08:19:45] <wm-bot>	 http://bit.ly/10eZZoa
[08:25:20] <petan>	 !rq is https://wikitech.wikimedia.org/wiki/Shell_Request/$url_encoded*?action=edit https://wikitech.wikimedia.org/wiki/User_talk:$url_encoded1?action=edit&section=new&preload=Template:ShellGranted https://wikitech.wikimedia.org/wiki/Special:UserRights/$url_encoded1 
[08:25:20] <wm-bot>	 Key was added
[08:25:24] <petan>	 !rq test
[08:25:24] <wm-bot>	 https://wikitech.wikimedia.org/wiki/Shell_Request/$url_encoded*?action=edit https://wikitech.wikimedia.org/wiki/User_talk:$url_encoded1?action=edit&section=new&preload=Template:ShellGranted https://wikitech.wikimedia.org/wiki/Special:UserRights/$url_encoded1 
[08:25:37] <wm-bot>	 Type @commands for list of commands. This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.10.8.0 source code licensed under GPL and located at https://github.com/benapetr/wikimedia-bot
[08:26:00] <petan>	 !rq del
[08:26:00] <wm-bot>	 Successfully removed rq
[08:26:20] <petan>	 !rq is https://wikitech.wikimedia.org/wiki/Shell_Request/$url_encoded_1?action=edit https://wikitech.wikimedia.org/wiki/User_talk:$url_encoded_1?action=edit&section=new&preload=Template:ShellGranted https://wikitech.wikimedia.org/wiki/Special:UserRights/$url_encoded_1
[08:26:20] <wm-bot>	 Key was added
[08:26:25] <petan>	 !rq bfsdg
[08:26:25] <wm-bot>	 https://wikitech.wikimedia.org/wiki/Shell_Request/bfsdg?action=edit https://wikitech.wikimedia.org/wiki/User_talk:bfsdg?action=edit&section=new&preload=Template:ShellGranted https://wikitech.wikimedia.org/wiki/Special:UserRights/bfsdg
[09:13:24] <wm-bot>	 Warning: There is 1 user waiting for shell: Artem Korzhimanov (waiting 0 minutes) 
[09:26:49] <wm-bot>	 Warning: There is 1 user waiting for shell: Artem Korzhimanov (waiting 13 minutes) 
[09:36:42] <petan>	 !rq Artem_Korzhimanov
[09:36:42] <wm-bot>	 https://wikitech.wikimedia.org/wiki/Shell_Request/Artem_Korzhimanov?action=edit https://wikitech.wikimedia.org/wiki/User_talk:Artem_Korzhimanov?action=edit&section=new&preload=Template:ShellGranted https://wikitech.wikimedia.org/wiki/Special:UserRights/Artem_Korzhimanov
[10:12:54] <addshore>	 petan: can you add rschen7754 to tools? :)
[10:38:20] <addshore>	 !screenfix is script /dev/null
[10:38:21] <wm-bot>	 Key was added
[10:43:47] <hashar>	 !log deployment-prep  deployment-cache-bits03 + role::protoproxy::ssl::beta  (should give us https on bits.beta.wmflabs.org
[10:43:50] <labs-morebots>	 Logged the message, Master
[10:45:46] <grv99>	 ^ :D
[10:45:53] <grv99>	 this bot is awesome :P
[10:46:20] <addshore>	 does anyone have a script that remvoed red links? :D
[10:51:42] <addshore>	 ignore the above :D
[11:14:15] <hashar>	 !log deployment-prep deployment-varnish-t3 : updating local puppet repo
[11:14:17] <labs-morebots>	 Logged the message, Master
[11:15:38] <hashar>	 !log deployment-prep deployment-varnish-t3 applying role::protoproxy::ssl::beta
[11:15:41] <labs-morebots>	 Logged the message, Master
[11:20:25] <hashar>	 !log deployment-prep deployment-squid applying role::protoproxy::ssl::beta
[11:20:28] <labs-morebots>	 Logged the message, Master
[11:44:29] <hashar>	 !log deployment-prep We somehow have HTTPS on beta now! https://en.wikipedia.beta.wmflabs.org/wiki/Main_Page  still have to fix up the cert names though.
[11:44:31] <labs-morebots>	 Logged the message, Master
[11:50:26] <hashar>	 MaxSem: hi. Beta more or less has HTTPS now https://en.m.wikipedia.beta.wmflabs.org/wiki/Main_Page  :-D
[11:58:57] <hashar>	 !log deployment-prep l10n cache is broken since Apr 30th 15:21 utC
[11:58:59] <labs-morebots>	 Logged the message, Master
[12:25:48] <MaxSem>	 props hashar
[12:25:55] <petan>	 Coren is motd on -login broken purposefuly?
[12:26:10] <Coren>	 No.  Odd.
[12:26:47] <Coren>	 The /etc/update-motd.d has all the right stuff in it.
[12:26:49] <Coren>	 ... and all the other motds are okay.
[12:27:41] <Coren>	 Huh.  Why did update-motd get uninstalled?
[12:28:03] <petan>	 no idea
[12:28:07] <petan>	 I wasn't on that server since sunday
[12:28:18] <petan>	 puppet I guess
[12:28:23] <hashar>	 !log deployment-prep fixed l10n cache ownership: chown -R l10nupdate:l10nupdate /data/project/apache/common-local/php-master/cache/l10n/
[12:28:26] <labs-morebots>	 Logged the message, Master
[12:28:42] <hashar>	 !log deployment-prep refreshing l10n cache
[12:28:44] <labs-morebots>	 Logged the message, Master
[12:31:43] <hashar>	 ah Coren good morning :-]
[12:31:54] <Coren>	 Heyo hashar.
[12:32:18] <hashar>	 so a few weeks ago we had some instances from the beta project to use the NFS server instead of glusterFS
[12:32:24] <hashar>	 with labsnfs::client or something
[12:32:35] <hashar>	 Ì was wondering how we could make that the default for newly created instance on that project
[12:32:36] <hashar>	 :-D
[12:33:14] <petan>	 Coren I created new feature
[12:33:32] <Coren>	 hashar: I don't think we can, although it should work if you set it in the 'configuration' section when you create the instance.
[12:33:36] <petan>	 Coren if you create /etc/motd.sysadmin it will be displayed only to people who are listed in /data/project/.system/sysadmins
[12:33:41] <Coren>	 So that it starts with the right classes.
[12:33:51] <petan>	 so in case we have some special issue with certain server, like must not run puppet etc
[12:33:54] <petan>	 like on -db
[12:33:54] <hashar>	 is that configuration done in puppet base ? 
[12:33:56] <Coren>	 petan: Simpler method:  check for membership in local-admin
[12:34:10] <petan>	 you can place it in that file so that everybody who is sysadmin knows that
[12:34:24] <petan>	 aha ok
[12:36:12] <petan>	 ok updated
[12:37:45] <petan>	 Coren when you ssh to -login you should see some to-do list
[12:38:11] <petan>	 Coren is there any other box where puppet must not be run?
[12:38:15] <Coren>	 petan: Except it doesn't.  Did you do that just on -login atm?
[12:38:41] <petan>	 Coren it /should/ work everywhere if /etc/motd.sysadmin exist on that box
[12:38:58] <petan>	 it works to me... dunno if it works to you
[12:39:11] <petan>	 it doesn't work if you suppress tips :P
[12:39:18] <petan>	 because it is part of same motd script
[12:39:30] <Coren>	 Aha.  I suppressed tips long ago.  :-)
[12:39:33] <petan>	 hehe
[12:39:40] <petan>	 I could probably make it replace tips
[12:39:49] <petan>	 instead of displaying both
[12:40:10] <Coren>	 Hm.  Something really funky is going on with motd on -login.  What the?
[12:40:37] <petan>	 stat: cannot stat `/var/lib/puppet/state/classes.txt': Permission denied
[12:40:38] <petan>	 run-parts: /etc/update-motd.d//97-last-puppet-run exited with return code 1
[12:40:39] <petan>	 Coren ^
[12:40:47] <petan>	 maybe that?
[12:40:59] <petan>	 well, with sudo it works
[12:41:33] <Coren>	 That's supposed to be invoked from pam_motd
[12:44:23] <Coren>	 Aha!  The symlink to /var/run/motd got trampled.
[12:44:33] <Coren>	 fix't
[12:44:38] <Coren>	 I wonder how /that/ happened.
[12:45:59] <hashar>	 !log deployment-prep upgrading bunch of boxes.
[12:46:02] <labs-morebots>	 Logged the message, Master
[12:47:02] <hashar>	 !log rebooting -squid for kernel upgrade
[12:47:02] <labs-morebots>	 rebooting is not a valid project.
[12:47:12] <hashar>	 !log deployment-prep rebooting -squid for kernel upgrade
[12:47:13] <labs-morebots>	 Logged the message, Master
[12:49:44] <hashar>	 !log deployment-prep rebooting -sql for kernel / mysql upgrade
[12:49:51] <labs-morebots>	 Logged the message, Master
[12:49:53] <hashar>	 !log deployment-prep rebooting -cache-upload03 for kernel upgrade
[12:49:55] <labs-morebots>	 Logged the message, Master
[12:53:01] <hashar>	 !log deployment-prep deleting deployment-lucene, we are using search01 and searchidx01
[12:53:04] <labs-morebots>	 Logged the message, Master
[12:56:22] <wm-bot>	 Warning: There is 1 user waiting for shell: Eclectiqus (waiting 0 minutes) 
[12:57:13] <petan>	 !rq Eclectiqus
[12:57:13] <wm-bot>	 https://wikitech.wikimedia.org/wiki/Shell_Request/Eclectiqus?action=edit https://wikitech.wikimedia.org/wiki/User_talk:Eclectiqus?action=edit&section=new&preload=Template:ShellGranted https://wikitech.wikimedia.org/wiki/Special:UserRights/Eclectiqus
[13:00:30] <petan>	 !labs-l is https://lists.wikimedia.org/mailman/listinfo/labs-l
[13:00:30] <wm-bot>	 Key was added
[13:01:39] <petan>	 !msys-git is http://msysgit.github.com/
[13:01:39] <wm-bot>	 Key was added
[13:01:50] <petan>	 !nagios.wmflabs.org is 208.80.153.210
[13:01:50] <wm-bot>	 This key already exist - remove it, if you want to change it
[13:03:18] <petan>	 !pypi is https://pypi.python.org/pypi/stdeb#the-commands
[13:03:19] <wm-bot>	 Key was added
[13:03:32] <petan>	 !rb is broken? report a bug: https://bugzilla.wikimedia.org/enter_bug.cgi?product=Wikimedia%20Labs
[13:03:32] <wm-bot>	 Key was added
[13:03:35] <petan>	 !report alias rb
[13:03:36] <wm-bot>	 Created new alias for this key
[13:04:20] <petan>	 !say is $*
[13:04:21] <wm-bot>	 This key already exist - remove it, if you want to change it
[13:04:26] <petan>	 !say I love petan
[13:04:26] <wm-bot>	 I love petan
[13:04:31] <petan>	 lol
[13:05:02] <petan>	 !seen is you probably wanted to use @seen
[13:05:03] <wm-bot>	 Key was added
[13:05:55] <petan>	 !a unalias
[13:05:55] <wm-bot>	 Alias removed!
[13:16:41] <Coren>	 !log tools reboot -dev, need to test kernel upgrade
[13:16:43] <labs-morebots>	 Logged the message, Master
[15:33:15] <petan>	 addshore does irc cloud shows a diff when topic is changed?
[15:33:23] <petan>	 if not I made 1 more feature that beats it
[15:48:05] <petan>	 Coren|AFK you fixed motd?
[15:51:39] <petan>	 Coren|AFK if you add /data/project/.system/bin to your path, I made command motded that edits the local sysadmin motd, so that in case you have some local-box notes you can put it there
[15:51:53] <petan>	 Coren|AFK such as "don't run puppet here" so that we don't get in similar issues as with db :P
[15:52:19] <petan>	 Coren|AFK: also I need to modify 1 line in /etc/profile how could I do that in puppet
[15:58:45] <scfc_de>	 petan: Can't you add a file in /etc/profile.d?
[15:58:58] <petan>	 that could work
[16:05:21] <scfc_de>	 addshore: Can you see where you started deploy.php? :-)  http://ganglia.wmflabs.org/latest/graph.php?r=week&z=xlarge&c=tools&h=tools-login&v=32&m=sge_running&jr=&js=
[16:37:48] <addshore>	 scfc_de: I know :P
[16:37:59] <addshore>	 Im tempted to increase the number of jobs I run
[16:45:50] <scfc_de>	 addshore: Go for it.  The exec nodes are idling most of the time with loads less than 0.2.
[16:46:04] <scfc_de>	 ("qstat -f")
[16:46:08] <addshore>	 Going to wait for it to get most of the way through this run
[16:46:18] <addshore>	 then might bump it up to 20 procs?
[16:46:31] <addshore>	 *jobs ;p
[16:46:41] <scfc_de>	 BTW, with Wikidata, will there be any need for interwiki bots in the future?
[16:47:49] <scfc_de>	 On Toolserver, there is just one non-interactive interwiki bot allowed, and I wonder if it makes sense to have a similar rule on Tools, or this topic will dry out in a few weeks.
[16:51:23] <addshore>	 interwiki bots generally are now just wikidata bots
[16:52:17] <scfc_de>	 k
[16:53:35] <apmon>	 MaxSem: Hi
[16:54:00] <addshore>	 !screenfix
[16:54:00] <wm-bot>	 script /dev/null
[16:54:36] <apmon>	 Did you get rados/ceph running on labs?
[16:54:39] <MaxSem>	 hey apmon
[16:54:44] <MaxSem>	 nope
[16:55:01] <MaxSem>	 instance creation was broken for some time
[16:55:30] <apmon>	 oh :-S
[16:58:57] <addshore>	 scfc_de: how do you think it would handle another 69 jobs run right now, this would clear my backlog
[16:59:04] <apmon>	 paravoid: With debian 7 now released and the freeze presumably over, would it be a good time at having another stab at getting mod_tile / renderd into the debian repositories?
[17:02:42] <addshore>	 im going to take that as a, go for it addshore!
[17:02:50] <scfc_de>	 addshore: I think it should be the grid's job to handle anything :-).
[17:03:04] <paravoid>	 apmon: I guess :)
[17:05:28] <addshore>	 well, there you go scfc_de , lets see how it handles it ;p
[17:06:33] <apmon>	 I'll try and update the packaging scripts for debian 7, and fix the remaining deblint warnings.
[17:06:51] <apmon>	 After that, could you have another look to see if they do meet the standards?
[17:06:59] <MaxSem>	 I've no idea about packaging
[17:07:19] <MaxSem>	 paravoid is the packaging guy:)
[17:08:01] <apmon>	 yup, sorry, that was in response to paravoid
[17:08:29] <scfc_de>	 addshore: It seems to handle it well :-).  But you could down the rate at which new processes are started from every minute to let's say, every 15 minutes or so.
[17:08:41] <Coren>	 The addbot hordes have come!
[17:08:48] <paravoid>	 I will, I need to find some time to devote to OSM
[17:09:00] <paravoid>	 sorry to have disappeared on you, too many things at once :/
[17:09:12] <addshore>	 yep :) well hopfully after my DB gets even smaller I will have less jobs to submit anyway :D
[17:09:29] <addshore>	 Coren: enjoy the ride, the only thing that is stoll on bots is my db now
[17:09:53] * addshore  wishes the ganglia memory thing was fixed :<

[17:10:29] <Coren>	 addshore: I think it's a side effect of the switch of swap and won't get fixed until a reboot.
[17:11:00] <apmon>	 cheers. I haven't really found too much time for things either :-/
[17:18:40] <scfc_de>	 Coren: Do you know why job 1862 is in Eqw?
[17:18:51] <addshore>	 because it broke? ;p
[17:19:07] <addshore>	 05/13/2013 21:48:08 [50404:9470]: error: can't open output file "/data/project/cyberbot//HOME/Cyberb
[17:19:45] <scfc_de>	 Ah, okay, that explains that.  So you can break a job by "-o".
[17:20:33] <addshore>	 scfc_de: for jsut so you know :) qstat -j <ID> gives you lots of lovely infomation ;p
[17:23:00] <scfc_de>	 addshore: *facepalm* I have used that command before, but sometimes my brain just seems too lazy :-).
[17:23:14] <addshore>	 hehe :)
[17:24:22] <scfc_de>	 Well, the cybers should then probably qdel it (but I don't know what's the status of the work on his crontab).
[17:53:46] <Ryan_Lane>	 andrewbogott: hey, the mediawiki puppet updates are looking good. thanks for working on that!
[17:54:45] <andrewbogott>	 Ryan_Lane:  Thanks!  Except the openid stuff is still broken which is making wikinaut sad :(
[17:54:50] <Ryan_Lane>	 heh
[17:55:14] <Ryan_Lane>	 can you move the $wgServer setting above the orig/LocalSettings.php include?
[17:55:19] <Ryan_Lane>	 then it would work
[17:55:24] <Ryan_Lane>	 because you could override it
[17:57:09] <andrewbogott>	 The mediawiki auto setup puts a fixed $wgServer setting in orig/LocalSettings.php.  So fresh installs would have to be hand-tuned in that case.
[17:57:20] <andrewbogott>	 Hopefully we'll just get the openid bug fixed
[17:57:24] * Ryan_Lane  nods

[18:08:25] <andrewbogott>	 Ryan_Lane, are you still planning on doing the essex->folsom upgrade tomorrow?  And, is there anything I can do in support of that?
[18:09:12] <Ryan_Lane>	 andrewbogott: yep. doing that tomorrow.
[18:46:41] <wm-bot>	 Warning: There is 1 user waiting for shell: Basti2342 (waiting 0 minutes) 
[18:55:22] <Iluvatar_>	 Hello! Sorry for my bad English (translate.google) Help me, please :) Problem: Python reports "WARNING: Skipped '/data/project/iluvatarbot/pywikipedia/user-config.py': owned by someone else." What am I doing wrong? (the problem described here: https://wikitech.wikimedia.org/wiki/User_talk:JackPotte )
[19:00:10] <wm-bot>	 Warning: There is 1 user waiting for shell: Basti2342 (waiting 13 minutes) 
[19:07:23] <Coren>	 Iluvatar_: You need to make sure that the owner of the file is the same as that which is running the script.
[19:08:07] <Coren>	 Iluvatar_: From the tool account, you can use 'take <file>' to take its ownership.
[19:13:38] <valhallasw>	 Ryan_Lane / Coren: http://www.mediawiki.org/wiki/Amsterdam_Hackathon_2013/Topics#Using_Wikimedia_Labs.2C_migrating_from_Toolserver "Wikimedia Labs - In this part of the session, you will be guided through setting up your first labs project." < do you mean a labs project or a service group?
[19:13:40] <wm-bot>	 Warning: There is 1 user waiting for shell: Basti2342 (waiting 27 minutes) 
[19:13:49] <valhallasw>	 in the first case, I think it shouldn't be under 'migrating from the toolserver'
[19:16:33] <valhallasw>	 Iluvatar_: chown youruser:youruser user-config.py
[19:17:08] <valhallasw>	 Iluvatar_: are you scheduling it in your own cron or in local-iluvatarbot's cron?
[19:17:08] <Iluvatar_>	 Coren: local-iluvatarbot@tools-login:~$ take $HOME/pywikipedia/user-config.py
[19:17:57] <valhallasw>	 Iluvatar_: that should be OK if you run the bot under the iluvatarbot account
[19:18:52] <Coren>	 valhallasw: At AMS, I think Ryan mean Tool Labs project.  I'll double check to make sure:  he might well also want to talk about setting up projects.
[19:19:23] <valhallasw>	 Coren: OK. I'm trying to get all tools-related communication as consistent as possible
[19:19:23] <Coren>	 valhallasw: Certainly I am here to talk about Tool Labs specifically.
[19:20:10] <Coren>	 Yeah, the terminology is...  not what I would have picked.  :-)
[19:20:29] <valhallasw>	 using 'Tool Labs' consistently would help
[19:20:59] <scfc_de>	 "Tools", please (with capital T).
[19:21:29] <Coren>	 I think I've been pretty consistent; but the terminology isn't all that conductive to clarity.  Tool Labs is a project of Labs, wher toolserver projects can be moved as tools (not projects).  :-)
[19:22:08] <valhallasw>	 :-)
[19:22:47] <Coren>	 And then there are service groups, which are local to the projects, which are use in the Tool Labs to implement tools.  :-)
[19:23:14] <Coren>	 There is a very important lesson in all this:  don't let sysadmins name stuff.  :-)
[19:23:46] <valhallasw>	 :-)
[19:27:10] <wm-bot>	 Warning: There is 1 user waiting for shell: Basti2342 (waiting 40 minutes) 
[19:39:49] <andrewbogott>	 Ryan_Lane, could you have a look at https://gerrit.wikimedia.org/r/#/c/51798/ ?  
[19:40:12] <andrewbogott>	 oops, wait, jenkins hates me...
[19:40:45] <wm-bot>	 Warning: There is 1 user waiting for shell: Basti2342 (waiting 54 minutes) 
[19:46:41] <valhallasw>	 Coren: can we get something like wm-bot for gaining access to Tools Labs?
[19:47:05] <valhallasw>	 (I mean: a page for requests + 'N users waiting for access' nags)
[19:47:43] <Coren>	 valhallasw: That'd make sense.  Lemme mull on the "right way" to do it, and I'll see what I can do that'd work best with SMW.
[19:47:50] <valhallasw>	 cool.
[19:48:36] <scfc_de>	 Are there people waiting for access?!
[19:49:28] <valhallasw>	 scfc_de: no, I'm just thinking a way to have a more formal way of requesting access, other than 'nag someone on IRC'
[19:50:40] <andrewbogott>	 OK… now Jenkins is happy.  So, again, Ryan_Lane, https://gerrit.wikimedia.org/r/#/c/51798/
[19:50:48] <Ryan_Lane>	 looking
[19:51:33] <Ryan_Lane>	 andrewbogott: why /a?
[19:54:14] <wm-bot>	 Warning: There is 1 user waiting for shell: Basti2342 (waiting 67 minutes) 
[19:54:35] <andrewbogott>	 Ryan_Lane, is /a present by default in our images?
[19:54:49] <andrewbogott>	 I think I had a run where the /a/backup bit failed for lack of /a
[19:59:34] <Ryan_Lane>	 andrewbogott: /a is not present
[19:59:43] <Ryan_Lane>	 and we should stop using /a places :)
[19:59:49] <Ryan_Lane>	 that's some weird domas-ism
[19:59:56] <Ryan_Lane>	 back in a little bit
[20:00:26] <andrewbogott>	 The /a/backup stuff is in that file from before, I don't know who uses it.
[20:07:49] <wm-bot>	 Warning: There is 1 user waiting for shell: Basti2342 (waiting 81 minutes) 
[20:12:18] <Damianz>	 Ryan_Lane: !
[20:23:35] <greenrosetta>	 where my database at?
[20:40:00] <petan>	 greenrosetta probably on tools-db
[20:40:28] <greenrosetta>	 no.. the one with en-wiki
[20:41:01] <Ryan_Lane>	 Damianz: !
[20:41:12] <Damianz>	 Ryan_Lane: HAI
[20:41:14] <Coren>	 greenrosetta: Real Soon Now™
[20:41:18] * Damianz  gives Ryan_Lane a virtual cookie

[20:41:31] <Damianz>	 Been like forever since you talked in here when I was around, it's like being old
[20:41:38] <Ryan_Lane>	 :D
[20:42:03] <Ryan_Lane>	 yeah, I've been working my actual hours, rather than like 60 hours a week
[20:42:16] <Damianz>	 that's no fun, Coren will take over the world
[20:42:34] <Ryan_Lane>	 if tool labs is considered the world, sure :)
[20:42:46] <Damianz>	 a tiny little bit of the world, he can't have my bath
[20:42:56] <Coren>	 Ryan_Lane: Wait, our actual hours aren't 60-70h/week?  :-)
[20:43:01] <Ryan_Lane>	 Coren: :D
[20:43:04] <Ryan_Lane>	 feels like it
[20:43:12] <valhallasw>	 Coren: http://tools.wmflabs.org/nlwikibots/public_html/ 
[20:43:17] <Ryan_Lane>	 when I work my actual hours people think I've disappeared
[20:43:29] <Damianz>	 Or died
[20:43:35] <Ryan_Lane>	 indeed :)
[20:43:38] <Coren>	 valhallasw: Oooo!  Pretties!
[20:43:50] <valhallasw>	 Coren: it's in /data/project/nlwikibots/public_html/public_html/content
[20:44:07] <Damianz>	 On another note, I need some cool opensource projects to work on... brain is dead
[20:44:08] <Ryan_Lane>	 we need to get some html/css gurus in here :)
[20:44:30] <Damianz>	 Ryan_Lane: Don't you have like a crazy long haired designer dude with an awesome tatoo
[20:44:40] <Ryan_Lane>	 yeah, but he does design
[20:44:41] <Damianz>	 s/tatoo/tattoo/
[20:44:45] <Ryan_Lane>	 we'd still need a frontend person
[20:44:47] <Damianz>	 that's like css :P
[20:45:22] <Ryan_Lane>	 maybe we should standardize on bootstrap :)
[20:45:28] <Damianz>	 <3 bootstrap
[20:45:37] * valhallasw  suggests vector ;-)

[20:45:57] <Damianz>	 Or 960 grid
[20:45:57] <Ryan_Lane>	 I'm not a huge fan of vector
[20:45:58] <Damianz>	 But that's more of a layout thing
[20:46:06] <Ryan_Lane>	 vector is just a design
[20:46:17] <valhallasw>	 Ryan_Lane: I'm mainly a huge fan of consistency
[20:46:21] <Ryan_Lane>	 bootstrap is layout, standard colors, etc
[20:46:37] <Ryan_Lane>	 valhallasw: see: http://twitter.github.io/bootstrap/
[20:46:38] <Coren>	 valhallasw: No it isn't anymore.  Now it's at /.  Yoink!
[20:46:50] <Damianz>	 We could just steal the webtools design or w/e it's called - must be oss
[20:46:51] <Damianz>	 :D
[20:46:59] <valhallasw>	 Coren: cool! Now only the style.css remains ;-)
[20:47:00] <Ryan_Lane>	 webtools design?
[20:47:11] <Coren>	 valhallasw: Shift-reload.  Your CSS is cached.  :-)
[20:47:15] <valhallasw>	 oh, right
[20:47:20] * valhallasw  feels silly

[20:47:27] <Damianz>	 s/webtools/webplatform/
[20:47:46] <Damianz>	 also webplatform is slow
[20:47:49] * Damianz  closes tab

[20:47:52] <Ryan_Lane>	 ahh
[20:48:01] <Ryan_Lane>	 yeah, their skin is in gerrit
[20:48:09] <Ryan_Lane>	 webplatform is indeed slow
[20:48:14] <Ryan_Lane>	 I've told them why
[20:48:15] <Damianz>	 It's kinda bootstrapish though, the top bar anyway
[20:48:21] <Damianz>	 Ryan_Lane: Mediawiki
[20:48:22] * Damianz  trolll

[20:48:23] <Coren>	 valhallasw: I also changed "add / remove users" to "* maintainers" to be more clear
[20:48:24] <Ryan_Lane>	 heh
[20:48:31] <Ryan_Lane>	 Damianz: because most of their pages disable cache
[20:48:43] <Damianz>	 OH CAUSE THAT'S SENSIBLE
[20:48:53] <Ryan_Lane>	 they aren't doing it on purpose
[20:49:05] <Ryan_Lane>	 they are using mediawiki features that disable it without knowing they are doing so
[20:49:07] <Damianz>	 There again... our largest site has only like the front page cached in Akami (and believe me, that ain't cheep)
[20:49:08] <Ryan_Lane>	 yay mediawiki
[20:49:26] <Damianz>	 Could you not override that using rules and then write an ext to fire off a clear on edit or such....
[20:49:39] <Damianz>	 Though that could get tricky with reviewed pages, probably hookified
[20:49:42] <Ryan_Lane>	 webplatform is reasonably fast if you are logged out and hit cached pages
[20:49:46] <valhallasw>	 Coren: sure
[20:49:49] <Damianz>	 Though mediawiki sucks, like resource loader means CLEAR ALL THE CACHE
[20:50:02] <Ryan_Lane>	 RL is actually pretty great
[20:50:07] <Ryan_Lane>	 it's one of mediawiki better features
[20:50:26] <Ryan_Lane>	 bits reloads from a full purge in seconds
[20:50:33] <Damianz>	 Not when you change like 1 css file and have to reload all of them :( or enable a plugin and boom all pages
[20:51:09] <Damianz>	 Though with bits... I guess it's not bad since varnish is awesome
[20:51:26] <Ryan_Lane>	 yep. when using a frontend cache it's nice
[20:51:47] <Ryan_Lane>	 mediawiki is designed in such a way that a frontend cache is mostly necessary
[20:54:41] <ori-l>	 or rather: mediawiki was not designed in such a way that you can avoid using a frontend cache
[20:55:26] <Damianz>	 I'd totally go for mediawiki just generating static html and shipping that to servers =D
[20:55:43] <Ryan_Lane>	 ori-l: well, you can, but it's really slow :)
[20:56:22] <Ryan_Lane>	 maybe at some point I'll need to add a frontend cache to openstack's wiki, but it's fast enough right now
[21:11:35] <petan>	 Coren why did you restart tools-dev?
[21:22:11] <labs-logs-bottie>	 !log tools petrb: created a separate volume for /tmp on login so that temp files do not fragment root fs and it does not get filled up by them, it also makes it easier to track filesystem usage
[21:22:12] <labs-morebots>	 Logged the message, Master
[21:23:04] <petan>	 !sal
[21:23:04] <wm-bot>	 https://wikitech.wikimedia.org/wiki/Labs_Server_Admin_Log
[21:23:22] <petan>	 Coren nvm
[21:24:04] <petan>	 heh, that is not really a big kernel update
[21:27:30] <petan>	 addshore is your bot still using bsql01?
[21:40:52] <Ryan_Lane>	 petan: are you using LVM for vdb on that host?
[22:07:43] <Ryan_Lane>	 andrewbogott: https://gerrit.wikimedia.org/r/#/c/51798/9/manifests/openstack.pp,unified
[22:07:47] <Ryan_Lane>	 like 468
[22:08:05] <Ryan_Lane>	 isn't webserver::php5 going to include pgp5?
[22:08:08] <Ryan_Lane>	 err
[22:08:10] <Ryan_Lane>	 php5
[22:08:58] <andrewbogott>	 I think it doesn't.  Checking...
[22:09:29] <Ryan_Lane>	 heh. I hate the way we handle apache :)
[22:10:16] <andrewbogott>	 Looks like it includes mod-php5 but not php itself.
[22:10:37] * Ryan_Lane  nods

[22:10:40] <Ryan_Lane>	 ok
[22:11:28] <Ryan_Lane>	 it looks like some other php classes were removed, though?
[22:12:04] <Ryan_Lane>	 php5-mysql, php-apc, and imagemagick? are they included elsewhere?
[22:12:24] <andrewbogott>	 They are, but, let me see where
[22:20:43] <andrewbogott>	 Ryan_Lane:  I no longer see whatever conflict prompted me to remove those.  So I'll restore that line.
[22:20:50] <Ryan_Lane>	 ok :)
[22:24:21] <Coren>	 Speaking of, Ryan_Lane, did you keep an eye on the growing puppet config of tools?
[22:24:43] <Ryan_Lane>	 well, I saw a bunch of stuff +2'd, but didn't get a chance to review it
[22:25:21] <Coren>	 90% is boring stuff, just adding packages users requested.
[22:25:54] * Ryan_Lane  nods

[22:26:12] <Ryan_Lane>	 the one that had caught my attention was the email stuff :D
[22:28:00] <Coren>	 Well, I wasn't expecting to do the MTA so quickly, but I wanted to catch the escaping daemon mail.  :-)
[22:28:10] <Ryan_Lane>	 heh
[22:28:31] <Coren>	 paravoid was being annoyed by it (understandably)
[22:29:21] <Coren>	 Did you see the evil workaround I did to the fact that we can't use exported resources?  It's surprisingly well-working.
[22:36:36] <Ryan_Lane>	 Coren: I didn't
[22:37:00] <Ryan_Lane>	 you know, they wouldn't be annoyed by email from instances if they set up a labs relay :)
[22:37:00] <andrewbogott>	 Is someone editing labs pages?  A bunch of <h3> tags just appeared on the configure instance pages.
[22:37:02] <Damianz>	 Btw - we should totally move to modules and puppet 3 with puppetdb, I actually sorta like exported resources again and environments are sexy
[22:37:14] <Ryan_Lane>	 andrewbogott: umm. I don't think so
[22:37:17] <Ryan_Lane>	 andrewbogott: example?
[22:37:21] <Ryan_Lane>	 I just upgraded mediawiki
[22:37:26] <andrewbogott>	 pick an instance, click 'configure'
[22:37:39] <andrewbogott>	 Ah, that's probably it, some slight change in parsing
[22:37:55] <Ryan_Lane>	 wow. wtf
[22:38:11] <Ryan_Lane>	 well, this looks like an issue with htmlform
[22:38:23] <andrewbogott>	 yeah
[22:38:47] <Coren>	 Damianz: Agreed.  Right now, I work around exported resources by stuffing what would have gone in them in plaintext files on the project shared filesystem, and collect them from the other instances.  It's a hack.
[22:39:10] <Damianz>	 that sorta works..
[22:39:14] <Ryan_Lane>	 andrewbogott: 'label' => Html::element( 'h3', array(), "$puppetgroupname:" ),
[22:39:43] <Damianz>	 I get around not using exported resources by handling the logic externally and shoving it down via an ENC in some cases, which is uber hacky but works
[22:40:29] <Ryan_Lane>	 andrewbogott: looks like htmlform escapes label now?
[22:41:05] <andrewbogott>	 Is our code setting it h3 or is htmlform?
[22:41:12] <Ryan_Lane>	 our code
[22:41:15] <Ryan_Lane>	 using htmlform
[22:41:28] <andrewbogott>	 well, escaping seems sort of sensible...
[22:41:36] <Ryan_Lane>	 not really
[22:42:02] <Ryan_Lane>	 ah
[22:42:03] <Ryan_Lane>	 label-raw
[22:44:46] <scfc_de>	 Coren: But the mailrelay relays to /dev/null ATM?
[22:44:50] <Ryan_Lane>	 andrewbogott: fixed
[22:45:18] <andrewbogott>	 Are those still h3s?  They look much bigger than before
[22:45:28] <Ryan_Lane>	 same size
[22:45:30] <andrewbogott>	 Although maybe before the <h3> was ignored entirely
[22:45:31] <Ryan_Lane>	 they are h3s
[22:45:42] <andrewbogott>	 Probably I'm just imagining it :)
[22:45:45] <Ryan_Lane>	 nah. it looked like this before :)
[22:45:56] <Coren>	 scfc_de: No, it delivers locally, but only internally generated mail.
[22:45:57] <Damianz>	 It's not the size that matters, it's what you do with it
[22:45:59] <Damianz>	 ...apparently...
[22:46:28] <Coren>	 scfc_de: No MX points at it.
[22:47:23] <scfc_de>	 Coren|Dinner: "scfc@tools-login:~$ echo Test | mail -s Test scfc" -> "No mail for scfc" :-(.
[22:53:13] <paravoid>	 Coren|Dinner: having labs outgoing mails going to the project owners is orthogonal to having a domain/incoming mail
[22:58:25] <andrewbogott>	 Ryan_Lane, I removed that line from https://gerrit.wikimedia.org/r/#/c/51798/ and it's still working fine for my purposes.
[22:59:40] <Ryan_Lane>	 andrewbogott: mysql-client also seems to have been removed?
[22:59:56] <Ryan_Lane>	 line 153?
[23:00:36] <andrewbogott>	 That comes in via the dep on generic::mysql::packages::client
[23:00:57] <Coren|Dinner>	 scfc_de: How odd.
[23:00:57] <Ryan_Lane>	 ah
[23:00:58] <Ryan_Lane>	 right
[23:00:58] <Ryan_Lane>	 ok
[23:01:26] <Ryan_Lane>	 +2'd
[23:01:41] <andrewbogott>	 thanks
[23:01:57] <Coren>	 paravoid: Except that the "from" address needs to make sense.
[23:02:13] <Coren>	 scfc_de: Hm.  I think something might not have noticed the change in domain name.
[23:02:25] <paravoid>	 from can be no-reply@wmflabs.org, who cares?
[23:02:36] <paravoid>	 (or whatever)
[23:04:14] <Ryan_Lane>	 paravoid: can you do the ferm stuff as a module? :)
[23:04:19] <Ryan_Lane>	 I'm doing a review of it now
[23:04:26] <paravoid>	 hm?
[23:04:28] <paravoid>	 it is a module
[23:04:35] <Ryan_Lane>	 https://gerrit.wikimedia.org/r/#/c/61744/1/manifests/base.pp,unified
[23:04:37] <Coren>	 Aha!  The service isn't refreshed on config file change.
[23:04:53] <scfc_de>	 Coren: The service on -login or on the relay?
[23:04:56] <Ryan_Lane>	 ah. I see. that's just the "role" part of it
[23:04:58] <paravoid>	 yes
[23:05:58] <Ryan_Lane>	 looks like we're always doing deny by default when using ferm?
[23:06:49] <Ryan_Lane>	 that's likely sane. I think 99% of the time we want that
[23:06:54] <Coren>	 scfc_de: Relay.  Did it by hand, will puppetize in a bit.  Ryan: What do I do for domain in the meantime?  Keep toolabs.org until we have the wrestling match with Legal (because this is known to work) or put tools.wmflabs.org for consistency?  Either way, I'm not going to allow incoming yet.
[23:07:28] <Ryan_Lane>	 Coren: for outgoing? I guess tools.wmflabs.org
[23:07:39] <Coren>	 kk
[23:07:57] <paravoid>	 Ryan_Lane: yeah I'm ambivalent about that
[23:08:13] <paravoid>	 but I think let's leave it like that until we need otherwise
[23:08:18] * Ryan_Lane  nods

[23:08:20] <scfc_de>	 Coren: "/var/mail/scfc": 1 message 1 new: Yippie!
[23:08:23] <paravoid>	 I'm also wondering what to do about OUTPUT's policy
[23:08:45] <Ryan_Lane>	 well, in general we don't need to do anything with it
[23:08:52] <Coren>	 scfc_de: Don't use it yet, I gots stuff to change.  :-)
[23:09:32] <Ryan_Lane>	 it may be good to have some policies we can drop in for special cases, though
[23:09:59] <paravoid>	 well
[23:10:57] <scfc_de>	 Coren: I hope cron uses it when it can't execute a job :-).
[23:11:13] <Coren>	 scfc_de: It does, which is the whole point of the exercise.
[23:15:53] <Coren>	 Ryan_Lane: Want to do the honors of +2 the revert to tools.wmflabs.org?  :-)
[23:17:54] <Coren>	 You snooze you loose.  :-)
[23:17:55] <Ryan_Lane>	 Coren: which change is this? :)
[23:18:01] <Coren>	 https://gerrit.wikimedia.org/r/#/c/63797/
[23:18:02] <Ryan_Lane>	 you mean toollabs.org?
[23:18:22] <Ryan_Lane>	 heh
[23:18:25] <Coren>	 revert /to/ tools.wmflabs.org
[23:18:27] <Ryan_Lane>	 well, I guess this reverts a little more than just that :)
[23:18:40] <Ryan_Lane>	 +2'd
[23:18:47] <Coren>	 Well, also the service refresh and all.
[23:20:36] * Coren  headdesks

[23:22:24] <Coren>	 Except that a service refresh is "notify =>", not "refresh =>"
[23:23:31] <scfc_de>	 :-)
[23:26:35] <Coren>	 Ryan_Lane: So, now, I have a working outgoing relay than can (eventually) also accept incoming.  It'll be trivial to point it at a global relay when we want to do that.
[23:27:57] <Ryan_Lane>	 sounds good
[23:30:36] <Ryan_Lane>	 heh
[23:30:38] <Ryan_Lane>	 good test email