[01:16:56] <YuviPanda>	 !log tools.lollrit-wm attempting to move to kubernetes
[01:16:56] <labs-morebots>	 tools.lollrit-wm is not a valid project.
[01:17:04] <YuviPanda>	 !log tools-lollrit-wm attempting to move to kubernetes
[01:17:04] <labs-morebots>	 tools-lollrit-wm is not a valid project.
[01:17:12] <YuviPanda>	 !log tools attempting to move to kubernetes
[01:17:15] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[01:17:15] <YuviPanda>	 err
[01:17:22] <YuviPanda>	 !log tools attempting to move grrrit-wm to kubernetes
[01:17:25] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[01:30:24] <wikibugs>	 6Labs, 6operations: Investigate whether to use Debian's jessie-backports - https://phabricator.wikimedia.org/T107507#1643731 (10scfc) (I submitted a change for #Tool-Labs to enable backports for specific packages only at https://gerrit.wikimedia.org/r/#/c/238662/.)
[03:00:17] <wikibugs>	 6Labs, 10Tool-Labs: Write admission controller disabling mounting of unauthorized volumes - https://phabricator.wikimedia.org/T112718#1643779 (10yuvipanda) 3NEW
[03:02:53] <wikibugs>	 6Labs, 10Tool-Labs, 3Labs-Sprint-114: Make sure that docker0 bridge comes up after flannel network is established - https://phabricator.wikimedia.org/T111893#1643785 (10yuvipanda) This caused issues in tools-worker-02 now, preventing internet access. I had to do ` iptables -t nat -A POSTROUTING ! -d <pod-ip-...
[03:50:44] <wikibugs>	 6Labs, 10Labs-Infrastructure: glancesync cron is failing - https://phabricator.wikimedia.org/T112719#1643794 (10Andrew) 3NEW a:3Andrew
[04:32:13] <wikibugs>	 6Labs, 10Tool-Labs, 10Labs-Infrastructure: Can't delete rule in default security group for the Tools project - https://phabricator.wikimedia.org/T112492#1643831 (10yuvipanda) Affects cloudadmins too
[04:32:22] <wikibugs>	 6Labs, 10Tool-Labs, 10Labs-Infrastructure: Can't delete rule in default security group for the Tools project - https://phabricator.wikimedia.org/T112492#1643832 (10yuvipanda) /cc @Andrew
[04:34:24] <wikibugs>	 6Labs, 10Tool-Labs: Setup DNS for kubernetes services - https://phabricator.wikimedia.org/T111914#1643839 (10yuvipanda) There's skydns + kube2sky running on a 2pod replica in the cluster now.
[04:40:11] <wikibugs>	 6Labs, 10Tool-Labs, 10Labs-Infrastructure: Can't delete rule in default security group for the Tools project - https://phabricator.wikimedia.org/T112492#1643862 (10yuvipanda) p:5Lowest>3High
[04:40:18] <YuviPanda>	 Krenair: can you open one for yours as well?
[04:40:38] <YuviPanda>	 Krenair: also since that's opened up in default, that should've technically fucked up all the other instances on the project too
[04:41:03] <Krenair>	 another ticket?
[04:41:08] <YuviPanda>	 Krenair: yes, different issue no?
[04:41:19] <YuviPanda>	 Krenair: opening up a security group is not supposed to block egress or something crazy like that
[04:42:21] <Krenair>	 Well I tried another host in the project which was fine
[04:53:37] <wikibugs>	 6Labs, 10Tool-Labs, 10Labs-Infrastructure: Can't delete rule in default security group for the Tools project - https://phabricator.wikimedia.org/T112492#1643889 (10Krenair) Ran into this issue myself, did some digging:  ```alex@alex-laptop:~/Development/MediaWiki/extensions/OpenStackManager (master)$ ssh sil...
[05:03:40] <wikibugs>	 6Labs, 10Labs-Infrastructure, 3ToolLabs-Goals-Q4: Move LabsDB aliases to DNS - https://phabricator.wikimedia.org/T63897#1643908 (10Krenair) a:5coren>3Krenair
[07:59:28] <GitHub139>	 [13flask-mwoauth] 15valhallasw closed pull request #14: Update link to OAuth consumer registration form (06master...06patch-1) 02http://git.io/vZbmC
[07:59:29] <GitHub33>	 [13flask-mwoauth] 15valhallasw pushed 2 new commits to 06master: 02http://git.io/vZNau
[07:59:29] <GitHub33>	 13flask-mwoauth/06master 14094c62c 15Robin Krahl: Update link to OAuth consumer registration form...
[07:59:29] <GitHub33>	 13flask-mwoauth/06master 149a028c5 15Merlijn van Deen: Merge pull request #14 from robinkrahl/patch-1...
[08:00:30] <krrrit-wm>	 (03CR) 10Merlijn van Deen: [C: 032] "Also a sane option on SGE, actually." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/238619 (owner: 10Yuvipanda)
[08:00:33] <krrrit-wm>	 (03Merged) 10jenkins-bot: Log to stdout only [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/238619 (owner: 10Yuvipanda)
[08:01:56] <krrrit-wm>	 (03CR) 10Merlijn van Deen: [C: 04-1] "Assuming comments are allowed, could you add a link to https://github.com/nodejs/docker-node/blob/master/4.0/onbuild/Dockerfile ? Assuming" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/238622 (owner: 10Yuvipanda)
[09:10:34] <wikibugs>	 6Labs, 10Tool-Labs, 7Icinga: Add a monitoring check for *.wmflabs.org certificate - https://phabricator.wikimedia.org/T112645#1644236 (10hashar) p:5Unbreak!>3Normal
[09:10:44] <wikibugs>	 6Labs, 10Tool-Labs, 7Icinga, 7Monitoring: Add a monitoring check for *.wmflabs.org certificate - https://phabricator.wikimedia.org/T112645#1641229 (10hashar)
[09:11:10] <wikibugs>	 6Labs, 10Tool-Labs: *.wmflabs.org https certificate expired (tools.wmflabs.org) - https://phabricator.wikimedia.org/T112608#1644240 (10hashar) Left to be done is to add a monitoring probe for the certificate expiry {T112645}
[14:56:32] <wikibugs>	 6Labs, 3Labs-Sprint-114, 3Labs-sprint-112, 3ToolLabs-Goals-Q4, 3labs-sprint-113: Fix documentation & puppetization for labs NFS - https://phabricator.wikimedia.org/T88723#1644949 (10coren)
[14:56:34] <wikibugs>	 6Labs, 6operations, 10ops-eqiad, 3Labs-Sprint-114, 3ToolLabs-Goals-Q4: Make certain ports and cables between the labstores and shelves are numbered/named and labeled, and make sure that the diagram(s) reflect that. - https://phabricator.wikimedia.org/T112549#1644947 (10coren) 5Open>3Resolved Diagram...
[14:57:06] <wikibugs>	 6Labs, 3Labs-Sprint-114, 3Labs-sprint-112, 3ToolLabs-Goals-Q4, 3labs-sprint-113: Fix documentation & puppetization for labs NFS - https://phabricator.wikimedia.org/T88723#1644952 (10coren) 5Open>3Resolved This should be reasonably complete with the updated diagram.
[14:59:32] <krrrit-wm>	 (03PS1) 10Greg Grossmeier: Remove ECT and old/dead projects from -devtools [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/238763 
[14:59:55] <wikibugs>	 6Labs, 3Labs-Sprint-114, 3labs-sprint-113: Evaluate gridengine's use of NFS and (possibly) move it to a different volume - https://phabricator.wikimedia.org/T111797#1644959 (10coren) Given that gridengine's traffic accounts for roughly only 20% of the tools volume's load, and that itself hovers only at 15% o...
[14:59:55] <krrrit-wm>	 (03PS2) 10Greg Grossmeier: Remove ECT and old/dead projects from -devtools [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/238763 
[15:01:54] <krrrit-wm>	 (03CR) 10Merlijn van Deen: [C: 032] Remove ECT and old/dead projects from -devtools [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/238763 (owner: 10Greg Grossmeier)
[15:02:08] <krrrit-wm>	 (03Merged) 10jenkins-bot: Remove ECT and old/dead projects from -devtools [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/238763 (owner: 10Greg Grossmeier)
[15:02:21] <valhallasw`cloud>	 YuviPanda: grrit+kubernetes=krrrit?
[15:04:40] <wikibugs>	 !log tools.wikibugs Updated channels.yaml to: 08ac39ff3184c179434bb9f36187adcea5ea8f24 Remove ECT and old/dead projects from -devtools
[15:04:43] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL, Master
[15:59:35] <hashar>	 YuviPanda: Coren : andrewbogott I have enabled a job to run on nodepool.  It is barely triggered but that means nodepool is booting/deleting instances on labs infra for real now
[16:00:13] <Coren>	 hashar: Cool beans!
[16:03:55] <hashar>	 currently limited to 5 instances
[16:10:36] <andrewbogott>	 hashar: https://gerrit.wikimedia.org/r/#/c/238515/ seems to be working as advertised… I’ll keep an eye out but I think we’re in good shape image-leak-wise
[16:15:40] <gifti>	 YuviPanda: didn't you want to make an announcement for kubernetes?
[16:24:26] <andrewbogott>	 Coren, I need some test creds for postgres on labsdb1004; is that something you know how to do?
[16:25:53] <Coren>	 andrewbogott: Actually, no.  I mean, I could create a user in the db, certainly, but I don't know how it'd mesh properly with what has been done previously.  Alex or Yuvi?
[16:42:47] <valhallasw`cloud>	 gifti: I'm not sure if there are any docs yet
[16:43:39] <YuviPanda>	 valhallasw`cloud: yes
[16:43:44] <YuviPanda>	 And there are no docs 
[16:43:52] <YuviPanda>	 gifti: yes I should...
[16:44:12] <valhallasw`cloud>	 I suggest some basic docs before announcing :-p
[16:44:33] <valhallasw`cloud>	 otherwise people want to try it but won't understand what is/isn't possible
[16:45:00] <valhallasw`cloud>	 also tools-k8s-bastion-01 doesn't have HBA working
[16:46:19] <YuviPanda>	 valhallasw`cloud: they can't try it 
[16:46:27] <YuviPanda>	 Nobody has any credentials there 
[16:46:36] <YuviPanda>	 Which is also why I haven't announced 
[16:46:47] <YuviPanda>	 Still tightening up the multi tenancy 
[16:53:25] <YuviPanda>	 valhallasw`cloud: btw am also disallowing non root access for the workers 
[16:53:45] <YuviPanda>	 Kube allows you to 'attach' to s remote pod and run bash...
[16:53:54] <YuviPanda>	 So people can debug things that way if need be 
[16:54:15] <YuviPanda>	 Is nicer since it gives you the exact same environment you specified 
[16:58:12] <Coren>	 Yes, those boxen need to be infrastructrure - there's no more reason for endusers to be there than there are for the virt hosts.
[17:02:17] <wikibugs>	 6Labs, 10Tool-Labs: Permission issues and/or failure to load Ruby environment on trusty - https://phabricator.wikimedia.org/T106170#1645595 (10MusikAnimal) Happening again. This time I'm getting `Permission denied @ rb_sysopen - perm_clerk.log (Errno::EACCES)` which I've seen before. I've tried `chmod 0777 per...
[17:02:44] <MusikAnimal>	 ^ the mystery remains
[17:05:37] <wikibugs>	 6Labs, 10Tool-Labs: Permission issues and/or failure to load Ruby environment on trusty - https://phabricator.wikimedia.org/T106170#1645611 (10valhallasw) Could this be related to T112388?
[17:06:08] <valhallasw`cloud>	 MusikAnimal: is it happening on a specific exec host?
[17:06:31] <MusikAnimal>	 trusty
[17:07:00] <wikibugs>	 6Labs, 10Tool-Labs: Permission issues and/or failure to load Ruby environment on trusty - https://phabricator.wikimedia.org/T106170#1645615 (10coren) @MusikAnimal: does your script chdir() at any point?
[17:07:04] <valhallasw`cloud>	 a specific host. So e.g. tools-exec-1401
[17:07:15] <MusikAnimal>	 oh sorry, not sure actually
[17:07:47] <Coren>	 MusikAnimal: Ah, didn't notice you here.  So yes, does your code change directories at all while it runs?
[17:08:04] <wikibugs>	 6Labs, 10Tool-Labs: Permission issues and/or failure to load Ruby environment on trusty - https://phabricator.wikimedia.org/T106170#1645618 (10MusikAnimal) @coren the cronjob calls a bash script that does change a directory: ``` export PATH="$HOME/.rbenv/bin:$PATH" export PATH="/data/project/musikbot/.rbenv/ve...
[17:08:08] <MusikAnimal>	 ^ yep
[17:09:00] <MusikAnimal>	 the issue is not predictable, I'm confident it will eventually start working again. It doesn't always complain about the same file either
[17:09:22] <MusikAnimal>	 sometimes it will error out for an hour, sometimes for a whole day, etc
[17:09:27] <valhallasw`cloud>	 so which runs errored out?
[17:09:31] <valhallasw`cloud>	 at which times?
[17:09:41] <valhallasw`cloud>	 or which jobids?
[17:10:10] <MusikAnimal>	 not sure about jobids, as it creates a new one every time, but it stopped working around 14:00 UTC
[17:10:17] <MusikAnimal>	 *16:00 UTC
[17:10:33] <valhallasw`cloud>	 and all later ones failed?
[17:11:07] <MusikAnimal>	 yep, this time complaining that it didn't have permission to open/write to the log file
[17:12:15] <MusikAnimal>	 mind you I run the script directly and it works, it's only through the cronjob that it fails
[17:12:37] <Coren>	 MusikAnimal: ... when do you invoke rotate_logs.sh?
[17:13:23] <MusikAnimal>	 0 */2 * * *, so what is that, twice daily I think?
[17:13:44] <Coren>	 Every other hour.
[17:13:48] <MusikAnimal>	 err no, once daily, 10 past the hour
[17:13:53] <MusikAnimal>	 oh
[17:14:09] <MusikAnimal>	 I dunno haha, it's just to ensure the logs don't get crazy big
[17:14:56] <valhallasw`cloud>	 it's nfs.
[17:15:03] <MusikAnimal>	 I figured
[17:15:10] <valhallasw`cloud>	 lemme paste what I just got
[17:15:21] <mutante>	 MusikAnimal: crontab of which user? your own?
[17:15:32] <MusikAnimal>	 yeah, musikbot
[17:15:43] <mutante>	 and when you run it manually, are you also musikbot?
[17:15:48] <MusikAnimal>	 yep
[17:16:35] <Coren>	 valhallasw`cloud: Where is that paste?
[17:16:46] <valhallasw`cloud>	 Coren: https://phabricator.wikimedia.org/T106170#1645712
[17:17:00] <MusikAnimal>	 there we go
[17:17:13] <MusikAnimal>	 I figured given the inconsistency it was NFS related
[17:17:26] <wikibugs>	 6Labs, 10Tool-Labs: Permission issues and/or failure to load Ruby environment on trusty - https://phabricator.wikimedia.org/T106170#1645792 (10valhallasw) And: ``` valhallasw@tools-exec-1401:~$ cat /data/project/musikbot/MusikBot/perm_clerk.log | wc -l cat: /data/project/musikbot/MusikBot/perm_clerk.log: Permi...
[17:17:34] <MusikAnimal>	 weird thing is I haven't heard about others' having this problem?
[17:17:48] <valhallasw`cloud>	 so I think it might indeed be the same issue as the one we see on the static web hosts
[17:18:15] <Coren>	 valhallasw`cloud: I'm not getting that result when I try this - I'm getting the right count everywhere.
[17:19:03] <Coren>	 (As tools.musikbot, which I'm guessing is what you were trying)
[17:19:07] <valhallasw`cloud>	 That's... weird. Could it be 'fixed' because of my remount?
[17:19:19] <Coren>	 valhallasw`cloud: What did you remount where?
[17:19:22] <MusikAnimal>	 about to find out
[17:19:28] <valhallasw`cloud>	 I tried as my own user, but all the hosts seem to work now for me as well
[17:19:37] <valhallasw`cloud>	 Coren: mount -o remount /data/project   on tools-exec-1401
[17:20:13] <Coren>	 valhallasw`cloud: That should actually be a noop if you're not changing any mount option.  But it shouldn't have affected the other nodes anyways.
[17:20:23] * Coren puzzles a bit.
[17:20:46] <MusikAnimal>	 bam, it worked
[17:20:53] <MusikAnimal>	 I think remounting does the trick
[17:21:04] <krrrit-wm>	 (03CR) 10Legoktm: "Even though projects are dead, you probably still want them in your channel as people will occasionally bump old bugs ;)" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/238763 (owner: 10Greg Grossmeier)
[17:22:23] <Coren>	 Even if it did, I can't think of a way it could possibly affect more than one node.
[17:22:42] <valhallasw`cloud>	 Coren: tools-exec-1405 still reports permission denied
[17:22:55] <Coren>	 valhallasw`cloud: Allright; don't touch it.  :-)
[17:23:00] <valhallasw`cloud>	 :-)
[17:23:37] <Coren>	 ... okay, I found what fixed it previously.
[17:23:48] <Coren>	 Is it working for you too now, valhallasw`cloud?
[17:24:02] <valhallasw`cloud>	 yes
[17:24:08] <valhallasw`cloud>	 -1410 is still broken
[17:24:22] <valhallasw`cloud>	 what did you do? ;-)
[17:24:26] <Coren>	 valhallasw`cloud: Try this.  Do a ls -l of the file then try again.
[17:24:52] <valhallasw`cloud>	 that fixes it. What?
[17:25:06] <Coren>	 That's not proper behaviour; it means the kernel is - somehow - not using the right stat() result on some files at least in some contexts.
[17:25:22] <Coren>	 I can't seem to cause the issue on precise.
[17:25:24] <krrrit-wm>	 (03PS1) 10Legoktm: Wait at least 1 second before pushing into redis [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/238792 (https://phabricator.wikimedia.org/T112032) 
[17:25:27] <valhallasw`cloud>	 I don't remember ls fixing the tools-web-static issue, though, so that might still be slightly different
[17:25:38] <valhallasw`cloud>	 but let's wait until that issue resurfaces
[17:26:04] <legoktm>	 valhallasw`cloud: wanna review ^ ?
[17:26:09] <valhallasw`cloud>	 legoktm: it's evil
[17:26:10] <valhallasw`cloud>	 I like it
[17:26:17] <legoktm>	 :D
[17:26:47] <krrrit-wm>	 (03CR) 10Merlijn van Deen: [C: 032] "Evil evil evil evil" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/238792 (https://phabricator.wikimedia.org/T112032) (owner: 10Legoktm)
[17:27:02] <krrrit-wm>	 (03Merged) 10jenkins-bot: Wait at least 1 second before pushing into redis [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/238792 (https://phabricator.wikimedia.org/T112032) (owner: 10Legoktm)
[17:27:42] <ircnotifier>	 !log tools.wikibugs legoktm: Deployed bcce439fda97c0a91b6ef983221f336a3da0cf99 Wait at least 1 second before pushing into redis wb2-phab
[17:27:44] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL, Master
[17:28:04] <legoktm>	 now someone just needs to batch edit a bunch of stuff again
[17:28:10] <Coren>	 valhallasw`cloud: I'm guessing right now that there is either something wrong with that kernel's attribute cache or with the value of its tunables.  Looking into it.
[17:28:34] <valhallasw`cloud>	 legoktm: no, this part doesn't get deployed automagically :-p
[17:28:44] <valhallasw`cloud>	 but no worries, I can fab that
[17:28:46] <legoktm>	 I just deployed it :P
[17:28:50] <legoktm>	 [10:27:43] <ircnotifier> !log tools.wikibugs legoktm: Deployed bcce439fda97c0a91b6ef983221f336a3da0cf99 Wait at least 1 second before pushing into redis wb2-phab
[17:29:01] <valhallasw`cloud>	 oh wait, that wasn't the automagic one
[17:29:02] <valhallasw`cloud>	 never mind
[17:29:10] <Coren>	 valhallasw`cloud: What it *looks* like right now is that it uses an unitilialized (not really there) entry, and that gets properly filled by an explicit stat() call.
[17:32:15] <valhallasw`cloud>	 Coren: I find it weird we keep bumping into obscure NFS kernel issues :/
[17:32:42] <valhallasw`cloud>	 well, the lock issue was related to the server dieing so that's maybe also an obscure corner case
[17:33:09] <Coren>	 I don't know if it's a kernel issue at all, and the lock thing was a corner case on an error path so I'm not surprised.
[17:33:41] <Coren>	 We can certainly turn off attribute caching on the clients but that has a noticable performance impact for many things.
[17:33:53] <Coren>	 (But would almost certainly help for the -static thing)
[17:35:00] <Coren>	 The issue with Musikbot seems different though as the EACCESS doesn't come from an older version of the file, but rather some invalid entry.
[17:36:12] <Coren>	 I've straced things, and from what I can see /bin/cat's open() fails with EACCESS only until something does a [f]stat() on that file, at which point it starts working properly.
[17:36:53] <Coren>	 But of course, I get to test this exactly once before it fails to not work.
[17:38:33] <wikibugs>	 6Labs, 10Tool-Labs: Permission issues and/or failure to load Ruby environment on trusty - https://phabricator.wikimedia.org/T106170#1645943 (10coren) a:3coren There is definitely an issue, and at this time it appears to be with the NFS attribute cache.  I'm investigating the matter.
[17:42:41] <wikibugs>	 10Wikibugs, 5Patch-For-Review: wikibugs - throttle output, don't get kicked for flooding - https://phabricator.wikimedia.org/T112032#1645976 (10Legoktm) So now someone needs to try a batch edit and we'll see if it floods off? :D
[18:16:10] <wikibugs>	 10Tool-Labs-tools-Other: Geohack should be mobile friendly - https://phabricator.wikimedia.org/T103409#1646149 (10Jdlrobson) p:5Triage>3High
[18:29:07] <krrrit-wm>	 (03CR) 1020after4: "I'm not familiar with forrestbot. Where does it run and how would I test this?" [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/238542 (https://phabricator.wikimedia.org/T107192) (owner: 10Jforrester)
[18:29:37] <James_F>	 twentyafterfour: Labs as an hourly cron-run script, and… good question.
[18:29:49] <James_F>	 twentyafterfour: It connects to a private GMail box to execute.
[18:29:58] <twentyafterfour>	 hah nice
[18:31:13] <wikibugs>	 6Labs, 10Tool-Labs: exec hosts have apache2 running - https://phabricator.wikimedia.org/T105059#1646289 (10scfc) If I try to purge `apache2` on `toolsbeta-exec-202` (`apt-get purge apache2-bin`), `apt-get` instead installs (!) `php5-cgi` and uninstalls `apache2`, `apache2-bin` and `libapache2-mod-php5`.  So I...
[18:37:20] <valhallasw`cloud>	 twentyafterfour: it works with any pop3 box! ;-)
[18:38:18] <twentyafterfour>	 What is the purpose of the mailbox? I thought it was just a phabricator bot
[18:38:45] <valhallasw`cloud>	 Gerrit events
[18:38:55] <valhallasw`cloud>	 it's subscribed to mediawiki-commits
[18:39:34] <valhallasw`cloud>	 it's much more stable than using gerrit stream-events (it's the same system gerrit reviewer bot uses)
[18:39:35] <twentyafterfour>	 oh. And that's better than just tracking the git history?
[18:39:53] <valhallasw`cloud>	 Heh. I guess that would actually also have worked, yes.
[18:40:28] <valhallasw`cloud>	 I hadn't thought of that option, and I had code to parse the gerrit event stream already, so I re-used that
[18:40:32] <twentyafterfour>	 Phabricator's herald system might be the ideal way to do it
[18:40:44] <twentyafterfour>	 valhallasw`cloud: reusing code is always nice
[18:41:01] <valhallasw`cloud>	 Yeah, having it all in Phab would be super cool
[18:43:17] <James_F>	 twentyafterfour: It was a Hackathon project. ;-)
[18:43:51] <twentyafterfour>	 James_F: and a rather cool one. I might look into what it would take to implement the same functionality in a herald commit hook
[18:44:09] * James_F nods.
[18:44:36] * valhallasw`cloud grins thinking back at how confused everyone was by 'Forrestbot'
[18:45:27] <Cyberpower678>	 Coren, ping
[18:46:03] <Coren>	 Cyberpower678: In the middle of something atm, should be free in 30m
[18:46:44] <James_F>	 valhallasw`cloud: If you want to review and work out a way of testing https://gerrit.wikimedia.org/r/238542 that'd be awesome, BTW. :-)
[18:47:34] <valhallasw`cloud>	 James_F: this asks for doctests. I'll look at it in a bit.
[18:47:41] * James_F nods.
[18:47:42] <James_F>	 Thanks!
[18:54:36] <andrewbogott>	 yuvipanda: When you appear, can you help me out with postgres?
[18:55:14] <yuvipanda>	 andrewbogott: ouch yes.
[18:55:41] <yuvipanda>	 Sorry about that. I'm going to head to the office now, I'll try to create an account on the train 
[18:57:21] <andrewbogott>	 thanks
[18:57:43] <andrewbogott>	 also — thoughts about how I should store the creds?  I assume puppet is not the best place
[18:58:39] <yuvipanda>	 andrewbogott: I'll put it on nfs 
[18:58:47] <andrewbogott>	 ok
[18:58:54] <yuvipanda>	 Same place as the mysql creds just different file 
[19:04:01] <andrewbogott>	 btw, that alert storm was me, should be resolved now
[19:04:49] <valhallasw`cloud>	 yuvipanda: btw, have you seen https://phabricator.wikimedia.org/passphrase/ ? I'm not sure if it could be of use in our infra, but it's good to know it exists
[19:05:23] <valhallasw`cloud>	 probably not more practical than just using puppetmaster in a more secure way
[19:05:26] <chasemp>	 yeah that should be disabled it must have snuck in
[19:05:36] <yuvipanda>	 valhallasw`cloud: ah interesting.
[19:06:04] <yuvipanda>	 Hah
[19:06:05] <valhallasw`cloud>	 chasemp: why? I've used pastes before to transfer credential information before, and thought it was pretty practical
[19:06:17] <valhallasw`cloud>	 this seems more to-the-point than abusing pastes ;-)
[19:06:23] <yuvipanda>	 valhallasw`cloud: I'm thinking of making a puppetmaster just for the k8s parts 
[19:06:48] <valhallasw`cloud>	 yuvipanda: I was also thinking maybe k8s should be completely seperate from tools?
[19:07:07] <valhallasw`cloud>	 so we'd just have a labs k8s infra that any project could use
[19:07:33] <yuvipanda>	 valhallasw`cloud: also don't think a comment is needed about the FROM? Like how we don't put comment about source of python module next to import. Dockerhub name is what the FROM refers to...
[19:07:52] <yuvipanda>	 valhallasw`cloud: people need to be able to switch between k8s and gridengine and have nfs access 
[19:07:57] <yuvipanda>	 That's why it has to be in tools...
[19:08:03] <valhallasw`cloud>	 oh, right.
[19:08:27] <yuvipanda>	 I can add explanatory comments in the dockerfile itself about what it does though 
[19:08:30] <chasemp>	 it's only sort of what it intends to be, you are right it's not necessarily better or worse than a secured paste but it encourages things nobody has talked about, especially generating and storing ssh private keys in phab
[19:08:33] <yuvipanda>	 I'll do that on the train 
[19:09:00] <valhallasw`cloud>	 yuvipanda: yeah, it's mostly that I dislike the magic CMD I think
[19:09:09] <chasemp>	 and note all of the passwordd stuffs is plaintext but it grew out of the phab mulihost strategy I believe as a way to hold keys for hosts to communicate securely
[19:09:15] <valhallasw`cloud>	 but it's comparable to an import in some sense
[19:09:15] <yuvipanda>	 valhallasw`cloud: in onbuild?
[19:09:18] <chasemp>	 and was obfuscated into an app
[19:09:30] <yuvipanda>	 Yeah I think documentation is required about what onbuild does 
[19:09:33] <yuvipanda>	 So I'll add that 
[19:09:41] <yuvipanda>	 And explain why that is enough 
[19:10:15] <valhallasw`cloud>	 yuvipanda: I dislike magic :-p
[19:10:25] <yuvipanda>	 Heh
[19:10:27] <valhallasw`cloud>	 well, too much magic, anyway.
[19:10:38] <yuvipanda>	 I think onbuild is OK amounts of magic 
[19:11:04] <valhallasw`cloud>	 I would just copy the onbuild Dockerfile tbh
[19:11:37] <yuvipanda>	 Shuah copying.
[19:11:39] <yuvipanda>	 Why?
[19:11:48] <yuvipanda>	 And miss any updates they do upstream?
[19:11:59] <valhallasw`cloud>	 the -onbuild only has the magic
[19:12:01] <valhallasw`cloud>	 not the actual build
[19:12:27] <yuvipanda>	 I don't know if I'll call it magic realllly
[19:12:37] <valhallasw`cloud>	 https://github.com/nodejs/docker-node/blob/e763a1065077c580aab4d73945597c0b160b4ee2/4.0/onbuild/Dockerfile
[19:12:45] <yuvipanda>	 Ok I should get off the phone now or I'll never get to the office 
[19:12:49] <valhallasw`cloud>	 the magic 'this is where your file is, this command is called'
[19:13:01] <yuvipanda>	 Disagree 
[19:13:05] <valhallasw`cloud>	 let me rephrase it
[19:13:06] <yuvipanda>	 Those are just node conventions 
[19:13:16] <yuvipanda>	 Like how requirements.txt is python convention 
[19:13:19] <valhallasw`cloud>	 I would not expect 'FROM node:...' to actually /run/ something
[19:13:35] <yuvipanda>	 Hmm
[19:13:59] <yuvipanda>	 That is fair. I guess it takes getting some used to.
[19:14:28] <valhallasw`cloud>	 I also agree npm start is the only sane command to run...
[19:14:51] <valhallasw`cloud>	 hence my suggestion 'link to the dockerfile here'
[19:15:04] <yuvipanda>	 Alright. I'll write up comments and link to the file 
[19:15:14] <yuvipanda>	 I wonder if the python onbuild runs thinfs too
[19:15:25] <yuvipanda>	 Or if it is just install requirements.txt
[19:15:29] <valhallasw`cloud>	 or just a comment # this installs dependencies and runs npm start
[19:15:42] <yuvipanda>	 Yup. I'll amend patch soon
[19:15:49] <valhallasw`cloud>	 onbuild just installs deps
[19:15:54] <valhallasw`cloud>	 for python
[19:15:56] <valhallasw`cloud>	 https://github.com/docker-library/python/blob/7663560df7547e69d13b1b548675502f4e0917d1/2.7/onbuild/Dockerfile
[19:15:58] <yuvipanda>	 Right 
[19:16:05] <yuvipanda>	 And you do your own entry point / cmd
[19:16:09] <yuvipanda>	 Which is nice 
[19:16:26] <valhallasw`cloud>	 it's also less standardized than with node, I guess
[19:16:36] <yuvipanda>	 Yeah
[19:17:15] <yuvipanda>	 valhallasw`cloud: BTW look at docker-compose as well
[19:17:29] <yuvipanda>	 Let's you specify a bunch of containers linked together 
[19:17:35] <yuvipanda>	 That's what we should do for wikibugs 
[19:17:46] <valhallasw`cloud>	 is that was k8s uses?
[19:17:47] <yuvipanda>	 A redis, and then two processes. One container each 
[19:17:56] <yuvipanda>	 valhallasw`cloud: no but same philosophy. 
[19:18:05] <yuvipanda>	 And docker compose can run locally 
[19:18:20] <yuvipanda>	 And k8s is a superset of the capabilities docker-compose has 
[19:18:23] <yuvipanda>	 Can has transforms 
[19:18:57] <yuvipanda>	 I'll add a commit with the one for krrrit-wm soon
[19:19:02] <valhallasw`cloud>	 k
[19:19:02] <yuvipanda>	 Now going for realz..
[19:19:04] <yuvipanda>	 Brb in a while 
[19:30:11] <krrrit-wm>	 (03PS2) 10Merlijn van Deen: [WIP] Support wmf branches like '1.23.0-wmf.6' [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/238542 (https://phabricator.wikimedia.org/T107192) (owner: 10Jforrester)
[19:30:28] <krrrit-wm>	 (03PS3) 10Merlijn van Deen: [WIP] Support wmf branches like '1.23.0-wmf.6' [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/238542 (https://phabricator.wikimedia.org/T107192) (owner: 10Jforrester)
[19:31:14] <krrrit-wm>	 (03CR) 10Merlijn van Deen: [C: 04-1] "This doesn't seem to work at the moment:" [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/238542 (https://phabricator.wikimedia.org/T107192) (owner: 10Jforrester)
[19:34:02] <valhallasw`cloud>	 eeeeeh, wait.
[19:34:07] <krrrit-wm>	 (03PS1) 10Merlijn van Deen: Move branch parsing to utils, add tests [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/238837 
[19:34:18] <legoktm>	 HAHAHA
[19:34:23] <legoktm>	 I was about to say "-1, needs tests"
[19:35:06] <valhallasw`cloud>	 also it seems to fail for the existing code as well
[19:35:08] <valhallasw`cloud>	 fun fun fun
[19:35:57] <valhallasw`cloud>	 legoktm: wait, do we parse it 'wmf/1.20wmf1' or just 1.20wmf1?
[19:36:00] <valhallasw`cloud>	 pass*
[19:36:22] <valhallasw`cloud>	 the marker includes wmf, so probably the latter
[19:36:22] <legoktm>	 just 1.20wmf1
[19:36:48] <krrrit-wm>	 (03PS2) 10Merlijn van Deen: Move branch parsing to utils, add tests [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/238837 
[19:36:52] <valhallasw`cloud>	 ok
[19:38:49] <wikibugs>	 10Quarry: Too much whitespace in Quarry - https://phabricator.wikimedia.org/T112803#1646597 (10Spage) 3NEW
[19:40:21] <krrrit-wm>	 (03PS4) 10Merlijn van Deen: Support wmf branches like '1.23.0-wmf.6' [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/238542 (https://phabricator.wikimedia.org/T107192) (owner: 10Jforrester)
[19:41:22] <krrrit-wm>	 (03CR) 10Merlijn van Deen: "That was because I was feeding it nonsensical values. Now:" [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/238542 (https://phabricator.wikimedia.org/T107192) (owner: 10Jforrester)
[19:41:42] <valhallasw`cloud>	 James_F:  if you checkout the latest PS and run 'tox', you should get test output! :-)
[19:41:51] <James_F>	 valhallasw`cloud: Awesome.
[19:41:54] <James_F>	 valhallasw`cloud: Also, thanks.
[19:42:03] <legoktm>	 valhallasw`cloud: I'll set up jenkins for that after lunch
[19:42:05] <valhallasw`cloud>	 I haven't fixed the failures yet, though ;-)
[19:42:07] <valhallasw`cloud>	 legoktm: <3
[19:42:51] <Krinkle>	 krrrit-wm not working in wikimedia-dev?
[19:43:19] <Krinkle>	 ah, it took a while
[19:43:58] <valhallasw`cloud>	 Kermwit
[19:44:10] <valhallasw`cloud>	 Our own personal green frog.
[19:56:35] <yuvipanda>	 valhallasw`cloud: it means we have Krinkle Krenair and krrrit-wm for kr anticomplete
[19:56:37] <James_F>	 valhallasw`cloud: Ha, != 0 and != '0' aren't the same thing. :-)
[19:57:15] <James_F>	 yuvipanda: Next up, kraphtain to remind us about Phabricator tasks?
[19:57:24] <James_F>	 valhallasw`cloud: Fixed.
[19:57:25] <krrrit-wm>	 (03PS5) 10Jforrester: Support wmf branches like '1.23.0-wmf.6' [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/238542 (https://phabricator.wikimedia.org/T107192) 
[19:58:17] <Krenair>	 andrewbogott, did you see the post about the compute service refusing to delete security groups?
[19:58:30] <andrewbogott>	 I saw it but haven’t investigated yet
[19:58:35] <Krenair>	 k
[19:59:48] <krrrit-wm>	 (03CR) 10Jforrester: Move branch parsing to utils, add tests (031 comment) [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/238837 (owner: 10Merlijn van Deen)
[20:00:18] <valhallasw`cloud>	 James_F: oh. probably not, that's a remainder from testing with forrestbot.py itself
[20:00:56] <krrrit-wm>	 (03PS3) 10Merlijn van Deen: Move branch parsing to utils, add tests [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/238837 
[20:00:58] <valhallasw`cloud>	 same for installing requests
[20:01:28] <James_F>	 valhallasw`cloud: Our normal test in CI is `tox-py27-jessie`
[20:02:03] <James_F>	 valhallasw`cloud: But `tox-py34-trusty` works just as well. :-)
[20:39:14] <cscott>	 !log deployment-prep updated OCG to version 4032a596ce6eb442b02cc6ee9b79263b1eb23275
[20:39:19] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL, Master
[20:39:33] <Cyberpower678>	 Coren, ping again.
[20:39:42] <Coren>	 Cyberpower678: What can I do for you?
[20:39:46] <Cyberpower678>	 PM?
[21:00:13] <krrrit-wm>	 (03CR) 10Jforrester: [C: 031] "Good to go once the new CI jobs land to prove the tests are working. :-)" [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/238837 (owner: 10Merlijn van Deen)
[21:12:09] <krrrit-wm>	 (03PS1) 10Jean-Frédéric: Refactor getCategoryFromCommonscat method [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/238966 (https://phabricator.wikimedia.org/T112513) 
[21:14:59] <krrrit-wm>	 (03CR) 10Jean-Frédéric: [C: 032] Refactor getCategoryFromCommonscat method [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/238966 (https://phabricator.wikimedia.org/T112513) (owner: 10Jean-Frédéric)
[21:15:06] <krrrit-wm>	 (03Merged) 10jenkins-bot: Refactor getCategoryFromCommonscat method [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/238966 (https://phabricator.wikimedia.org/T112513) (owner: 10Jean-Frédéric)
[21:37:09] <wikibugs>	 6Labs, 10Tool-Labs, 10Labs-Infrastructure: Can't delete rule in default security group for the Tools project - https://phabricator.wikimedia.org/T112492#1647076 (10Andrew) This is failing deep down in nova code: 2015-09-16 21:34:31.105 3179 TRACE nova.api.openstack Traceback (most recent call last): 2015-09-...
[22:13:17] <wikibugs>	 6Labs, 10Tool-Labs: Kubernetes Beta Signup List - https://phabricator.wikimedia.org/T112824#1647213 (10yuvipanda) 3NEW
[22:14:07] <gifti>	 ooooh
[22:25:32] <Krenair>	 andrewbogott, does deployment-bastion not log ssh auth failures?
[22:27:23] <Krenair>	 -rw-r----- 1 root adm 0 May 17 06:55 auth.log
[22:27:26] <Krenair>	 not incredibly helpful
[22:30:50] <yuvipanda>	 gifti: do sign up if you're interested :)
[22:31:00] * yuvipanda continues writing that email
[22:31:04] <gifti>	 wee, it's working?
[22:31:28] <gifti>	 yuvipanda: can i have webservice and continuous jobs?
[22:31:56] <yuvipanda>	 gifti: not yet, but if you sign up I can put you at the head of the queue :)
[22:32:00] <yuvipanda>	 gifti: krrrit-wm is running on it now
[22:32:06] <yuvipanda>	 I'm sorting out some more security issues
[22:33:18] <wikibugs>	 6Labs, 10Tool-Labs: Kubernetes Beta Signup List - https://phabricator.wikimedia.org/T112824#1647334 (10Giftpflanze) 1. giftbot 2. tcl 3. webservice and 5 continuous jobs
[22:33:20] <yuvipanda>	 gifti: there's https://github.com/efrecon/docker-tcl so you can have a recent version of tcl if you want :)
[22:33:27] <yuvipanda>	 :)
[22:33:28] <yuvipanda>	 nice
[22:33:32] <Cyberpower678>	 anomie, Coren what just happened to the DBs.
[22:33:49] <Cyberpower678>	 I lost connection mid-query and can't execute any more queries.
[22:34:23] <gifti>	 yuvipanda: hm, so it doesn't have the labs tcl?
[22:34:28] <wikibugs>	 6Labs, 10Tool-Labs: Kubernetes Beta Signup List - https://phabricator.wikimedia.org/T112824#1647338 (10yuvipanda)
[22:34:34] <gifti>	 (which is patched)
[22:34:37] <Cyberpower678>	 NVM.  Back up
[22:34:38] <yuvipanda>	 gifti: it can too, sure.
[22:34:54] <yuvipanda>	 gifti: I edied the task with more questions, can you answer those as ewll
[22:34:54] <yuvipanda>	 ?
[22:35:05] <yuvipanda>	 just edit your comment
[23:04:15] <krrrit-wm>	 (03CR) 10Legoktm: "recheck" [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/238837 (owner: 10Merlijn van Deen)
[23:16:50] <krrrit-wm>	 (03CR) 10Jforrester: [C: 032] Move branch parsing to utils, add tests [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/238837 (owner: 10Merlijn van Deen)
[23:16:57] <krrrit-wm>	 (03Merged) 10jenkins-bot: Move branch parsing to utils, add tests [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/238837 (owner: 10Merlijn van Deen)
[23:17:08] <krrrit-wm>	 (03PS6) 10Jforrester: Support wmf branches like '1.23.0-wmf.6' [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/238542 (https://phabricator.wikimedia.org/T107192) 
[23:56:20] <Dispenser>	 I can't seem to write to u2815_p anymore