[00:08:59] <gwicke>	 k, thanks!
[00:19:46] <legoktm>	 found the bug, working on a patch
[00:27:02] <grrrit-wm>	 (03PS1) 10Legoktm: redis2irc: join channels after reloading config [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/180365 
[00:27:27] <grrrit-wm>	 (03PS2) 10Legoktm: redis2irc: join channels after reloading config [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/180365 
[00:27:35] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] redis2irc: join channels after reloading config [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/180365 (owner: 10Legoktm)
[00:27:53] <grrrit-wm>	 (03Merged) 10jenkins-bot: redis2irc: join channels after reloading config [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/180365 (owner: 10Legoktm)
[00:28:43] <wm-bot>	 !log tools.wikibugs legoktm: Deployed 8502072659ddb8c55ae45026d54867771e3122e7 redis2irc: join channels after reloading config wb2-irc
[00:28:45] <labs-morebots>	 Logged the message, Master
[00:29:03] <YuviPanda>	 legoktm: why wm-bot?! ;(
[00:29:17] <legoktm>	 YuviPanda: oh, I don't think we merged valhallasw`cloud's patch yet.
[00:29:24] <YuviPanda>	 oh, right
[00:29:36] <YuviPanda>	 it’s 6AM, so I shouldn’t be merging patches
[00:39:39] <wm-bot>	 !log tools.wikibugs legoktm: Deployed 8502072659ddb8c55ae45026d54867771e3122e7 redis2irc: join channels after reloading config wb2-irc
[00:39:41] <labs-morebots>	 Logged the message, Master
[00:40:42] <grrrit-wm>	 (03PS1) 10Legoktm: fab: Turn `pull` into a task (for channels.yaml-only updates) [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/180370 
[00:41:18] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] fab: Turn `pull` into a task (for channels.yaml-only updates) [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/180370 (owner: 10Legoktm)
[00:41:32] <grrrit-wm>	 (03Merged) 10jenkins-bot: fab: Turn `pull` into a task (for channels.yaml-only updates) [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/180370 (owner: 10Legoktm)
[02:50:01] <wikibugs>	 3Tool-Labs: program created by proprietary compiler allowed on labs? - https://phabricator.wikimedia.org/T74253#852391 (10LuisV_WMF) I think it's against the spirit of the TOU, if not necessarily the letter - the idea was that if the original maintainer vanishes, others can reasonably pick up and continue their...
[04:20:40] <andrewbogott>	 Coren, if still awake:  https://dpaste.de/3en7
[04:21:09] <andrewbogott>	 That's roughly what was happening with my attempts yesterday as well.  Everything looks good until after the reboot, and then…maintenance mode
[06:42:37] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]  
[06:54:02] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-15 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]  
[07:07:37] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-04 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:24:01] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-15 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:48:38] <matanya>	 andrewbogott: why precise on hp virts ?
[07:49:06] <andrewbogott>	 matanya: because the other virt nodes are running precise and I don't have the patience to put a million ifs in the manifests right now.
[07:49:22] <matanya>	 makes sense :)
[07:50:04] <matanya>	 andrewbogott: who is it gooing with bootstrap-vs apart from partman voodoo ?
[07:51:14] <andrewbogott>	 matanya: it pretty much works!  Coren worked on lvm stuff a bit -- now instances are doing this:  https://tools.wmflabs.org/paste/view/0dd299e2
[07:53:45] <matanya>	 hmm, interesting
[09:09:29] * valhallasw`cloud hits legoktm
[09:09:43] <valhallasw`cloud>	 although maybe that pull option is better than mine :P
[09:10:02] <valhallasw`cloud>	 https://gerrit.wikimedia.org/r/#/c/180318/ :-p
[09:10:36] <grrrit-wm>	 (03Abandoned) 10Merlijn van Deen: change default behavior to just-pull [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/180318 (owner: 10Merlijn van Deen)
[09:13:42] <legoktm>	 valhallasw`cloud: oops
[09:14:25] <valhallasw`cloud>	 legoktm: no problem :-p
[09:18:53] <valhallasw`cloud>	 legoktm: the pull option is better anyway, as it shouldn't log to irc
[09:19:01] <valhallasw`cloud>	 because wikibugs already does this itself
[09:50:35] <grrrit-wm>	 (03PS5) 10Merlijn van Deen: Report IRC using Python and Yuvi's ircnotifier [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/180322 
[09:50:50] <valhallasw`cloud>	 legoktm: ^ :>
[12:51:08] <YuviPanda>	 valhallasw`cloud: hmm, was thinking - should make hhvm available on toollabs :)
[12:55:52] <wikibugs>	 3Tool-Labs: Make HHVM based webservices available on toollabs - https://phabricator.wikimedia.org/T78783#853119 (10yuvipanda) 3NEW
[13:09:21] <shinken-wm>	 RECOVERY - Puppet staleness on tools-webproxy is OK: OK: Less than 1.00% above the threshold [3600.0]  
[13:23:18] <valhallasw`cloud>	 YuviPanda: who cares about php =p
[13:25:12] <YuviPanda>	 valhallasw`cloud: a sadly large number of people
[13:25:20] <YuviPanda>	 valhallasw`cloud: also, this lets us write code in Hack :P
[13:27:29] <YuviPanda>	 valhallasw`cloud: should be trivial to make some of the long running bots use it
[13:46:47] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[13:59:23] <wikibugs>	 3Tool-Labs: Make HHVM based webservices available on toollabs - https://phabricator.wikimedia.org/T78783#853260 (10yuvipanda) hhvm is now installed and could potentially be used for bots / other things that run from the CLI. I've emailed labs-l asking for volunteers.
[14:00:02] <Coren>	 andrewbogott_afk: I'm up now.  Looking.
[14:14:43] <Coren>	 andrewbogott_afk: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=758808 seems very similar.  And I see dozens of Fedora issues related to filesystem timing out under systemd that appear similar.
[14:16:48] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-05 is OK: OK: Less than 1.00% above the threshold [0.0]  
[14:20:47] <Coren>	 andrewbogott_afk: Further research show that at least six distros have that issue in/around systemd
[14:21:25] <Coren>	 andrewbogott_afk: It appears to be a combination of poor ordering of dependencies and short timeouts.
[15:12:21] <paravoid>	 is sudo-ldap still being used in labs?
[15:15:59] <Coren>	 paravoid: It is.
[15:16:17] <Coren>	 paravoid: There's no plan to phase it out.  Is there an issue?
[15:18:44] <paravoid>	 no, I'm doing some cleanups
[15:20:17] <paravoid>	 and I thought I remember Ryan saying that he wanted to drop it
[15:44:27] <Coren>	 It's not obvious how we could, short of distributing sudoers by puppet that are constructed from the wikitech db...  which seems even more brittle to me.
[15:57:04] <paravoid>	 Coren: grep -A 2 Coren modules/quarry/manifests/init.pp
[15:57:09] <paravoid>	 Coren: have you figured it out?
[15:58:33] <Coren>	 Hm.  I don't recall the issue that led to this.  Lemme see what I can dig up.
[15:58:44] <paravoid>	 thanks
[15:59:02] <paravoid>	 it says "Temp" and it's there since July, so... :)
[16:08:05] <Coren>	 I'm not sure I was really involved in that discussion, but it seems to be related to/fixed by https://gerrit.wikimedia.org/r/#/c/153723/
[16:08:45] <Coren>	 YuviPanda: Do you recall what led you to ^^?
[16:40:00] <YuviPanda>	 Coren: oh yeah
[16:40:16] <YuviPanda>	  Coren you could only sudo as users from LDAP 
[16:40:21] <YuviPanda>	 Not others 
[16:40:24] <YuviPanda>	 Including root
[16:40:31] <YuviPanda>	 Which Andrew fixed with that 
[16:41:21] <Coren>	 So that temp fix should be obsolete now?
[16:44:25] <YuviPanda>	 Yeah
[16:44:29] <YuviPanda>	 I'll remove soon
[16:50:41] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<11.11%)  
[16:55:40] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK  
[17:46:48] <wikibugs>	 3Labs-Team: facter: VM detection incorrect in labs - https://phabricator.wikimedia.org/T78813#853700 (10Gage) 3NEW
[17:53:07] <grrrit-wm>	 (03CR) 10Legoktm: [C: 031] Report IRC using Python and Yuvi's ircnotifier (031 comment) [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/180322 (owner: 10Merlijn van Deen)
[17:54:31] <valhallasw`cloud>	 legoktm: correct, but I think the venv is actually loaded on 'become'
[17:57:47] <legoktm>	 valhallasw`cloud: ah ok. 
[21:31:57] <KTC>	 bd808, free?
[21:32:16] <bd808>	 KTC: what's up?
[21:32:25] <KTC>	 questions about vagrant
[21:32:50] <bd808>	 on a laptop or in labs?
[21:33:00] <KTC>	 my computer
[21:33:05] <KTC>	 i.e. how to use it properly
[21:33:18] <KTC>	 re. wikimania scholarships
[21:33:43] <bd808>	 ok. what parts are not clear?
[21:34:38] <KTC>	 If I write some code that make db changes say, test it, and decide I want to revert everything and start again. What's the quickest way to do that?
[21:35:26] <bd808>	 I guess it depends on what you did to the db. Either undo the schema changes manually or drop and recreate the db
[21:35:53] <bd808>	 if you completely drop the database and then run `vagrant provision` it should rebuild it from the master schema
[21:36:23] <KTC>	 great, thanks :)
[22:08:53] <YuviPanda>	 valhallasw`cloud: what was your fastcgi template thing people were using?
[22:12:24] <YuviPanda>	 legoktm: ^
[22:29:16] <legoktm>	 YuviPanda: the one in gerrit-patch-uploader
[22:30:57] <YuviPanda>	 legoktm: cool, thanks :)
[22:35:33] <wikibugs>	 3Labs-Team: apt-get update fails some times causing puppet failure alarms - https://phabricator.wikimedia.org/T76771#854364 (10yuvipanda) 5Open>3Resolved a:3yuvipanda This is completely fixed now - apt-get update is run by the cron that runs puppet, and so is all good :)
[22:37:45] <YuviPanda>	 !log touched /data/project/repo/Packages so tools-webproxy stops complaining about that not xisting and never running apt-get
[22:37:45] <labs-morebots>	 touched is not a valid project.
[22:38:08] <YuviPanda>	 !log tools touched /data/project/repo/Packages so tools-webproxy stops complaining about that not xisting and never running apt-get
[22:38:11] <labs-morebots>	 Logged the message, Master
[23:16:04] <wikibugs>	 3Quarry: Switch Quarry to use Material Design Bootstrap theme - https://phabricator.wikimedia.org/T76140#854487 (10yuvipanda) 5Open>3declined a:3yuvipanda But has a somewhat fucked up license, so... no.  https://github.com/FezVrasta/bootstrap-material-design/blob/master/LICENSE.md