[01:51:17] * Coren gives up for tonight.
[01:51:25] <Coren>	 andrewbogott: I wouldn't reboot that box yet.
[01:51:36] <Coren>	 (Not that we'd especially want to)
[01:52:09] <andrewbogott>	 Thanks for looking, I'm sorry that it's such a headache
[01:56:02] <Coren>	 No, no, it's important.  I don't like ticking timebombs and a box that may or may not reboot is... not fun.
[04:01:07] <quiddity>	 is there a list of tools, by order of popularity/usage ?
[04:02:15] <quiddity>	 I couldn't see anything obvious at wikitech or wmflabs itself...
[04:46:19] <Carmela>	 quiddity: We can break Labs for a few hours and see which tools people complain about most.
[04:46:30] * quiddity twitches
[04:46:35] <Carmela>	 I guess you'd want visitor stats?
[04:46:38] <Carmela>	 Not sure we have those.
[04:46:42] <Carmela>	 Even though we should.
[04:49:31] <quiddity>	 Mhm.  I'm looking for any other angle than Alphabetical, which helps me get a handle on the hundreds of tools listed at http://tools.wmflabs.org/
[04:49:55] <quiddity>	 any angle other than*
[07:36:55] <yuvipanda>	 !Ping
[07:36:56] <yuvipanda>	 !ping
[07:36:56] <wm-bot>	 !pong
[08:08:49] <multiwiki>	 Trying to get a job running consistently getting libgcc_s.so.1 must be installed for pthread_cancel to work
[08:10:30] <yuvipanda>	 multiwiki: heya! that means it is running out of memory
[08:10:39] <yuvipanda>	 multiwiki: try adding -mem 1G (or some other amount of memory) to your jsub command
[08:13:12] <multiwiki>	 <yuvipanda> thanks also 38408192 Nov 20 08:06 core can I get rid of this binary file its keep growing
[08:14:23] <yuvipanda>	 yup yup you can
[08:14:29] <yuvipanda>	 that just means your code is core dumping
[08:18:51] <multiwiki>	 <yuvipanda> thanks
[08:18:59] <yuvipanda>	 yw!
[11:59:50] <yuvipanda>	 !ping
[11:59:50] <wm-bot>	 !pong
[12:01:25] <matanya>	 quiddity: https://tools.wmflabs.org/hay/directory/#/
[12:02:05] <matanya>	 guillom: i would like to post a blog post next week, handle through you ?
[13:54:54] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): Puppet failures on deployment-bastion - 10https://bugzilla.wikimedia.org/73520#c1 (10Antoine "hashar" Musso (WMF)) 5NEW>3ASSI a:3Ori Livneh Seems to be caused by https://gerrit.wikimedia.org/r/#/c/173353/ keyholder: add /etc/keyholder.d and `keyholder arm` sub...
[15:21:10] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): Beta not picking up merged change - 10https://bugzilla.wikimedia.org/73659 (10Gilles Dubuc) 3NEW p:3Unprio s:3normal a:3None This changeset was merged more than 2 hours ago: https://gerrit.wikimedia.org/r/#/c/172953/3  Beta is still serving the old version o...
[15:24:00] <guillom>	 matanya: yes, you can draft it on Meta: https://meta.wikimedia.org/wiki/Wikimedia_Blog/Drafts and send me a link by email when it's ready :) 2-3 days of notice is ideal. Let me know if you have any questions!
[15:28:05] <Coren>	 multiwiki: Also, that your code dumps core is a very bad symptom you may want to find the underlying cause to - it means that it crashes and that will definitely impact reliability.
[15:37:28] <Coren>	 andrewbogott: Is it possible some of the disks on virt1009 came from virt1008 originally?
[15:38:07] <andrewbogott>	 Coren: I don't know.  I wouldn't think so, but there may have been some shuffling when everything was renamed and shipped from tampa.  Chris might know more
[15:39:34] <Coren>	 andrewbogott: I found the problem, if not the root cause.  Try this on virt1009 and despair:
[15:39:40] <Coren>	 mdadm --examine --scan
[15:40:33] <Coren>	 No wonder the poor thing is confused.
[15:40:33] <andrewbogott>	 there are two different /dev/md/0s?
[15:40:38] <andrewbogott>	 and /dev/md/1?
[15:40:46] <Coren>	 Right, a native one and a foreing one.
[15:40:57] <andrewbogott>	 man
[15:41:16] <andrewbogott>	 so, I guess if those HP servers ever arrive, we should plan to evacuate virt1009 and rebuild it.
[15:41:21] <andrewbogott>	 :(
[15:41:38] <Coren>	 It looks like some of the disks meant to be part of the array actually come from elsewhere.  Note how mdadm --examine /dev/sda2 reports two drives of the array missing?
[15:42:55] <Coren>	 Aha!  /dev/sdh.  It was built on virt1008
[15:43:12] <Coren>	 The disks got mixed up!
[15:43:18] <Coren>	 We're lucky this works at all
[15:43:32] <Coren>	 And this means that 1008 has/will have the same problem.
[15:44:09] <Coren>	 I can reconstruct the physical volumes and force an array rebuild, though.
[15:44:15] <andrewbogott>	 so, I still don't see how this happened… two running servers, and then the drives were hot-swapped?
[15:44:25] <andrewbogott>	 oh?  Without data loss?
[15:44:52] <Coren>	 Yeah; right now we're runing in degraded mode anyways - those drives are disabled because they're part of the wrong array.
[15:45:12] <Coren>	 Perhaps during the move and only the more recent version of the OS notices and complains about the discrepancy?
[15:46:16] <Coren>	 ... oddly enough, I don't see virt1009 disks in 1008's arrays.  Dafu?
[15:49:26] <andrewbogott>	 maybe an old server was scrapped and something else put in its place
[15:49:42] <andrewbogott>	 I'm sure we did some of that
[15:51:58] <Coren>	 But yes, I should be able to offline the drive, recontruct it as part of the right array, and online it again.
[15:54:22] <andrewbogott>	 great!
[15:55:54] <Coren>	 This should make all traces of the virt1008 array disapear, and make grub-probe happy.
[15:57:12] <Coren>	 You may be interested to know that the UUID of the "real" virt1008 array does not match that of the drive on 1009 - it might just be a case of recuperated drives that were never properly reinitialized.
[15:59:27] <andrewbogott>	 Once the drives are feeling more at home… do we need to force grub to reinstall itself?
[15:59:42] <Coren>	 andrewbogott: It's best to.
[15:59:42] <andrewbogott>	 Or do you think grub is fine, just alerting us to the issue?
[16:00:02] <andrewbogott>	 You said the raid is running in degraded mode?  Why doesn't icinga show a drive as having failed then?
[16:00:17] <Coren>	 I'm pretty sure grub has refused to do /anything/ because of the issue - the previous bootloader is probably still there and working, but I don't know if I'd chance it.
[16:01:00] <Coren>	 andrewbogott: They're not failed, they're just *gone*.  They are holes in the array since the last boot and not drives meant to be part of it but dead.
[16:01:44] <Coren>	 andrewbogott: Holey sheets.  We're luckier than we thought - there's indeed /dev/sdh that comes from the wrong array but /dev/sdf is actually dead *too*
[16:01:51] <Coren>	 It was DOA so never part of the array.
[16:02:35] <Coren>	 Yeay redundancy.
[16:03:01] <Coren>	 I can fix sdh, but sdf will need a new drive I think.
[16:04:49] <Coren>	 [4409276.037509] scsi 6:0:5:0: [sdf] Unhandled error code
[16:04:50] <Coren>	 [4409276.037515] scsi 6:0:5:0: [sdf]  Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[16:04:56] <Coren>	 Doesn't look good. 
[16:12:11] <Coren>	 That failed drive shouldn't affect the fix though.
[16:16:59] <Coren>	 andrewbogott: sdh is now part of the "correct" array an in recovery mode.  sdf still quite dead.
[16:36:25] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): Beta not picking up merged change - 10https://bugzilla.wikimedia.org/73659#c1 (10Antoine "hashar" Musso (WMF)) 5NEW>3RESO/DUP The beta cluster Jenkins slave (deployment-bastion) ends up having all its executing slots locked by the Gearman plugin.  The only way t...
[16:59:06] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] Add support for creating tarballs of skins [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/174468 (owner: 10Legoktm)
[16:59:15] <yuvipanda>	 legoktm: want me to merge the ops change?
[16:59:20] <legoktm>	 yuvipanda: no, not yet
[16:59:23] <yuvipanda>	 legoktm: ok
[16:59:29] <yuvipanda>	 legoktm: I'll be around for an hour or so more
[16:59:33] <legoktm>	 yuvipanda: we're gonna do that in an hour
[16:59:35] <legoktm>	 ok :D
[16:59:40] <yuvipanda>	 heh
[16:59:53] <wikibugs>	 3Wikimedia Labs / 3Infrastructure: puppet labsstatus not reported when using role::puppet::self - 10https://bugzilla.wikimedia.org/63296#c7 (10Antoine "hashar" Musso (WMF)) 5NEW>3RESO/WON The reason I filled this bug was to use OpenstackManager as a dashboard of puppet run. Over the last few weeks, Yuvi...
[17:03:07] <Coren>	 andrewbogott: 8913 created to track sdf; you want me to cc?
[17:03:19] <andrewbogott>	 Coren: yes please
[17:04:13] <Coren>	 {{done}}
[17:06:09] <quiddity>	 matanya, that's great, thanks :)
[17:07:23] <Coren>	 andrewbogott: FYI, grub-update worked now that there are no phantom arrays.  We're still in a tenuous position because we have to drives' worth of lost redundancy but that's fixable.
[17:08:04] <andrewbogott>	 So, an RT for chris to replace the drives?
[17:10:55] <legoktm>	 yuvipanda: how do I force a puppet run?
[17:11:03] <yuvipanda>	 legoktm: 'sudo puppet agent -tv'
[17:11:19] <legoktm>	 ty
[17:20:50] <yuvipanda>	 !log fab disabled puppet on fab instance, going to set up redirect
[17:20:50] <labs-morebots>	 fab is not a valid project.
[17:20:55] <yuvipanda>	 !log phabricator disabled puppet on fab instance, going to set up redirect
[17:21:00] <labs-morebots>	 Logged the message, Master
[17:21:11] <yuvipanda>	 !log phabricator (disregard previous message) disabled puppet on fab2 instance, going to set up redirect
[17:21:12] <labs-morebots>	 Logged the message, Master
[18:27:33] <grrrit-wm>	 (03PS1) 10Legoktm: Fix format string [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/174752 
[18:28:00] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] Fix format string [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/174752 (owner: 10Legoktm)
[18:28:14] <grrrit-wm>	 (03CR) 10Legoktm: [V: 032] Fix format string [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/174752 (owner: 10Legoktm)
[18:34:55] <grrrit-wm>	 (03PS1) 10Ori.livneh: add passwords::phabricator [labs/private] - 10https://gerrit.wikimedia.org/r/174758 
[18:35:27] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032 V: 032] add passwords::phabricator [labs/private] - 10https://gerrit.wikimedia.org/r/174758 (owner: 10Ori.livneh)
[18:38:34] <legoktm>	 !log extdist moved all extensions to /srv/dist/extensions, enabled skin dist support and other fun things
[18:38:35] <labs-morebots>	 Logged the message, Master
[18:42:22] <grrrit-wm>	 (03PS1) 10Ori.livneh: add labs mwdeploy_rsa [labs/private] - 10https://gerrit.wikimedia.org/r/174761 
[18:42:38] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032 V: 032] add labs mwdeploy_rsa [labs/private] - 10https://gerrit.wikimedia.org/r/174761 (owner: 10Ori.livneh)
[18:50:07] <yuvipanda>	 andrewbogott: is there a reason icmp pings seem disabled by default inside labs?
[18:50:16] <yuvipanda>	 I see tools has a specific security rule allowing it...
[18:50:37] <yuvipanda>	 legoktm: can you add that to extdist too? port range -1 to -1, source 10.0.0.0/8, protocol icmp
[18:50:37] <andrewbogott>	 yuvipanda: I think it was used for puppet freshness tests back in the day, but no longer.
[18:50:47] <andrewbogott>	 So -- not a good reason that I know of
[18:50:49] <yuvipanda>	 andrewbogott: right, but shinken uses ping to test reachability
[18:50:55] <yuvipanda>	 andrewbogott: is there a way for us to turn it on by default?
[18:51:19] <andrewbogott>	 In what sense is it off?  Blocked by security groups?
[18:51:22] <yuvipanda>	 andrewbogott: yup
[18:51:53] <legoktm>	 !log extdist added port range -1 to -1, source 10.0.0.0/8, protocol icmp to security roles
[18:51:56] <labs-morebots>	 Logged the message, Master
[18:51:59] <andrewbogott>	 Hm, I thought that the default security group for a new project allowed it
[18:52:01] <legoktm>	 yuvipanda: done, and ping works now.
[18:52:10] <legoktm>	 heh
[18:52:11] <legoktm>	 ** RECOVERY alert - extdist2/ is  **
[18:52:26] <yuvipanda>	 legoktm: :)
[18:52:41] <yuvipanda>	 andrewbogott: hmm, extdist was a while back, I guess
[18:53:51] <andrewbogott>	 yuvipanda: looking at a random selection of projects (well, two) and it's enabled in the default group for both
[18:53:55] <yuvipanda>	 hmm
[18:53:58] <yuvipanda>	 wasn't on ext
[18:53:59] <yuvipanda>	 dist
[18:54:05] <yuvipanda>	 I guess it just skipped it or somesuch
[18:54:11] <yuvipanda>	 I'll just enable it if I run into other projects where it's disabled
[18:55:18] <legoktm>	 yuvipanda: so the alerts will just say "PROBLEM" and it's up to me to figure out what's broken? :P
[18:55:32] <yuvipanda>	 legoktm: nope, if an individual 'service' breaks they will say what it is
[18:55:41] <yuvipanda>	 legoktm: only the host notification is like this
[18:55:46] <yuvipanda>	 I'll fix shortly
[18:55:50] <legoktm>	 ok
[18:56:16] <yuvipanda>	 legoktm: but email delivery is kind of shit atm
[18:56:18] <yuvipanda>	 legoktm: http://shinken-test.wmflabs.org/host/extdist2
[18:56:26] <yuvipanda>	 legoktm: you've puppet failures on ext2
[18:56:33] <legoktm>	 yeah
[18:56:38] <legoktm>	 the biglogs thing
[18:56:43] <yuvipanda>	 legoktm: oh, yeah, right
[18:56:47] <legoktm>	 yuvipanda: it wants a login?
[18:56:48] <legoktm>	 ldap?
[18:56:53] <yuvipanda>	 legoktm: no, 'guest/guest'
[18:57:08] <yuvipanda>	 such secure, much wow
[18:57:36] <legoktm>	 lolol
[18:57:55] <yuvipanda>	 legoktm: I couldn't find a way to disable auth
[18:58:10] <yuvipanda>	 legoktm: this is temperoary, of course :) eventually I'll have to write an auth plugin
[18:58:12] <yuvipanda>	 maybe even SUL
[18:58:47] <yuvipanda>	 legoktm: on the plus side, it's no longer just betacluster machines having puppet problems ;)
[18:58:52] <yuvipanda>	 legoktm: can you remove the biglogs role?
[18:58:58] <legoktm>	 >.<
[18:58:59] <legoktm>	 yeah
[18:59:05] <yuvipanda>	 legoktm: remove and force ar un?
[18:59:06] <yuvipanda>	 *a run
[19:01:43] <legoktm>	 !log extdist removed biglogs role from extdist2 and now puppet is happy
[19:01:45] <labs-morebots>	 Logged the message, Master
[19:02:07] <legoktm>	 yuvipanda: ^
[19:02:15] <yuvipanda>	 legoktm: yayyway
[19:04:12] <legoktm>	 yuvipanda: how often does shinken check/re-check?
[19:04:17] <yuvipanda>	 legoktm: every 5mins
[19:04:22] <yuvipanda>	 legoktm: but it looks at data for last 10min
[19:04:28] <yuvipanda>	 also service check emails are busted.
[19:04:29] <yuvipanda>	 let me lok
[19:07:13] <grrrit-wm>	 (03PS1) 10Legoktm: test_skin_repo_list passes now [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/174766 
[19:08:04] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] test_skin_repo_list passes now [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/174766 (owner: 10Legoktm)
[19:15:42] <yuvipanda>	 legoktm: I'm going to mess around with extdist1.eqiad.wmflabs for something.
[19:15:45] <yuvipanda>	 unrelated to extdist
[19:15:51] <legoktm>	 sure
[19:39:16] <grrrit-wm>	 (03PS1) 10Legoktm: Make tarballs a ton smaller by not including .git [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/174777 
[19:39:33] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] Make tarballs a ton smaller by not including .git [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/174777 (owner: 10Legoktm)
[19:44:56] <legoktm>	 !screen
[19:44:56] <wm-bot>	 script /dev/null
[19:51:46] <yuvipanda>	 legoktm: help me test something?
[19:51:49] <yuvipanda>	 legoktm: remove the icmp rule again?
[19:52:11] <legoktm>	 ok
[19:53:04] <legoktm>	 !log extdist removed the icmp security rule thing for yuvipanda
[19:53:06] <labs-morebots>	 Logged the message, Master
[19:53:14] <yuvipanda>	 cool, now let's see if shinken notices
[20:01:24] <yuvipanda>	 legoktm: add that rule again? :)
[20:01:41] <legoktm>	 yuvipanda: can I just make you a admin of extdist? I thought you were...
[20:01:51] <yuvipanda>	 legoktm: i am, but lazy to re-login
[20:02:09] <yuvipanda>	 legoktm: but if you're busy I can do that
[20:02:17] <legoktm>	 !log re-added icmp rule
[20:02:18] <labs-morebots>	 re-added is not a valid project.
[20:02:24] <legoktm>	 !log extdist re-added icmp rule
[20:02:25] <labs-morebots>	 Logged the message, Master
[20:02:41] <legoktm>	 yuvipanda: I just got an email without the hearts
[20:02:55] <yuvipanda>	 fuck yeah :)
[20:02:57] <yuvipanda>	 woot
[20:03:07] <yuvipanda>	 it's just... slow?
[20:07:13] <matanya>	 sure quiddity :)
[20:13:19] <legoktm>	 !log extdist force re-generated all tarballs to get rid of .git directories
[20:13:21] <labs-morebots>	 Logged the message, Master
[20:29:50] <annika_>	 Coren: what do you think about showing dots in webservice (re)start until the webservice has started?
[20:30:35] <Coren>	 annika_: I see no issue with the concept, though that's very 1980's.  :-)
[20:30:44] <yuvipanda>	 tch tch :)
[20:31:06] <Coren>	 andrewbogott: md0 rebuilt, md1 is rebuilding.
[20:32:03] <andrewbogott>	 Coren: what happens then?  Does the raid rebalance onto the new drives?
[20:32:37] <Coren>	 andrewbogott: It's raid 10; the gone drives just meant we had some stripes with no mirror.
[20:33:01] <Coren>	 andrewbogott: In practice, read performance was impacted too and should recover somewhat.
[20:33:16] <andrewbogott>	 Cool.
[20:33:28] <andrewbogott>	 I have one more remaining post-upgrade issue and then we're happily icehoused.
[20:33:31] <Coren>	 andrewbogott: We still have one drive missing from the array; but that'll be returned once Chris has an opportunity to physically intervene.
[20:33:37] <annika_>	 Coren: you already do it when stopping/restarting
[20:33:39] <andrewbogott>	 (I guess the bad drive thing isn't technically an icehouse issue)
[20:34:14] <Coren>	 andrewbogott: No, but it's one I'm really glad the upgrade surfaced - we were in danger of losing virt1009 entirely at any time.
[20:35:47] <Coren>	 I'm still confused how a virt1008 drive ended up in virt1009 and yet was never properly integrated in the array.
[20:36:27] <yuvipanda>	 !ping
[20:36:27] <wm-bot>	 !pong
[21:07:04] <yuvipanda>	 andrewbogott: ping. you were looking earlier to be notified of any/all changes for a particular host. did you find that?
[21:08:06] <andrewbogott>	 yuvipanda: can I get a little more context?
[21:08:26] <yuvipanda>	 andrewbogott: a few days ago, you were looking for a way to have icinga notify you if there were any alerts at all for the virt* machines
[21:08:30] <yuvipanda>	 andrewbogott: did you find a way?
[21:08:38] <andrewbogott>	 Ah, that.  No, I didn't.
[21:08:50] <yuvipanda>	 ah, yeah, looks like there's no way.
[21:08:50] <yuvipanda>	 sigh
[21:19:31] <Coren>	 yuvipanda: I'm receiving many 100s of bounces addressed to you.
[21:19:40] <Coren>	 Over 1000 already
[21:19:40] <yuvipanda>	 bounces?!
[21:19:41] <yuvipanda>	 why
[21:19:50] <yuvipanda>	 why am *I* not getting them? I thought it was shinken being dead
[21:19:52] <yuvipanda>	 let me stop them now
[21:20:15] <Coren>	 yuvipanda: Because they are bounces back to root.  You /should/ be getting them in your opsness.
[21:20:16] <yuvipanda>	 Coren: ok, should've stopped now.
[21:21:04] <yuvipanda>	 Coren: hmm, I'm not getting the originals nor the bounces...
[21:21:07] <yuvipanda>	 Coren: can you forward me one?
[21:21:12] <yuvipanda>	 Coren: and sorry for flooding inboxes.
[21:21:21] <yuvipanda>	 I need to figure out a better way to test email delivery
[21:22:29] <Coren>	 yuvipanda: Sent.
[21:22:54] <Coren>	 I'm not sure what you tried, but I can tell you it didn't work right.  :-)
[21:23:21] <yuvipanda>	 Coren: yeah, *that* one was just me testing on a 'wrong' email address (gmail.org, remnant of replacing wikimedia.org).
[21:23:23] <yuvipanda>	 sorry about that :)
[21:23:42] <yuvipanda>	 service state change emails still not being delivered, and I think I know why, but I'm hoping I'm wrong
[21:23:46] <yuvipanda>	 and verifying now
[21:26:40] <yuvipanda>	 sigh, ok
[21:26:45] <yuvipanda>	 I'll have to fix shinkengen
[21:26:56] <yuvipanda>	 hmm, or not
[21:29:21] <yuvipanda>	 Coren: I'm trying to debug an email delivery problem. I was expecting about 200 emails from shinken, but only got... 4.
[21:29:36] <yuvipanda>	 Coren: any idea where I could / should be looking? it's just using cmdline 'mail' to send emails
[21:50:06] <Coren>	 yuvipanda: You may want to look at the /var/log/exim4/ logs
[21:50:38] <yuvipanda>	 Coren: yeah, it wasn't an exim issue. as I suspected it was a shinken issue and I might have a fix.
[22:00:54] <legoktm>	 yuvipanda: uh, are these alerts for real? ** PROBLEM alert - extdist1/Puppet staleness is CRITICAL **
[22:00:58] <yuvipanda>	 legoktm: nope
[22:01:03] <yuvipanda>	 legoktm: just testing
[22:01:05] <legoktm>	 ok
[22:01:08] <yuvipanda>	 legoktm: AND YAY EMAIL NOTIFICATION WORKS
[22:02:01] <Coren>	 Yuvi; bounce flood again.
[22:02:05] <yuvipanda>	 bah
[22:02:11] <yuvipanda>	 what, to whom?
[22:02:12] <Coren>	 hundreds, for guest@guest
[22:02:19] <yuvipanda>	 oh.
[22:02:42] <yuvipanda>	 hmm, I'll find a way to null that contact out.
[22:02:59] <yuvipanda>	 but email delivery at least seems to work for everyone else
[22:03:27] <yuvipanda>	 Coren: hmm, if it was bouncing for guest@guest it would've been flooding for a month now...
[22:03:48] <Coren>	 And yet, there they go.
[22:03:53] * yuvipanda tries to stop that as well
[23:02:26] <Merlissimo>	 Coren: load on the only available webgrid is rising. is there any reason not to make the other available for new jobs, too?
[23:36:59] <Coren>	 It is, but it runs trusty so you have to start webservices manually (webservice script doesn't yet understand asking for -lrelease=trusty)