[00:52:39] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<44.44%)  
[00:58:54] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]  
[01:01:03] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[01:02:42] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK  
[01:03:48] <shinken-wm>	 PROBLEM - Puppet failure on tools-webproxy is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]  
[01:04:54] <shinken-wm>	 PROBLEM - Puppet failure on tools-redis is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0]  
[01:05:36] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-10 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[01:06:36] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-05 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]  
[01:07:23] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]  
[01:08:01] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-09 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]  
[01:10:04] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-11 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0]  
[01:10:26] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]  
[01:10:50] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-wmt is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[01:11:41] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]  
[01:11:41] <shinken-wm>	 PROBLEM - Puppet failure on tools-trusty is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]  
[01:11:47] <shinken-wm>	 PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]  
[01:11:59] <shinken-wm>	 PROBLEM - Puppet failure on tools-static is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]  
[01:12:34] * YuviPanda|bus pats shinken-wm
[01:12:37] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-06 is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0]  
[01:14:10] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[01:14:46] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-15 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[01:14:46] <shinken-wm>	 PROBLEM - Puppet failure on tools-mail is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]  
[01:14:52] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-tomcat is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[01:15:48] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-generic-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[01:28:47] <shinken-wm>	 RECOVERY - Puppet failure on tools-webproxy is OK: OK: Less than 1.00% above the threshold [0.0]  
[01:28:53] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0]  
[01:30:39] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-10 is OK: OK: Less than 1.00% above the threshold [0.0]  
[01:31:01] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0]  
[01:31:33] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-05 is OK: OK: Less than 1.00% above the threshold [0.0]  
[01:33:03] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-09 is OK: OK: Less than 1.00% above the threshold [0.0]  
[01:34:55] <shinken-wm>	 RECOVERY - Puppet failure on tools-redis is OK: OK: Less than 1.00% above the threshold [0.0]  
[01:35:06] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-11 is OK: OK: Less than 1.00% above the threshold [0.0]  
[01:36:45] <shinken-wm>	 RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0]  
[01:37:23] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-03 is OK: OK: Less than 1.00% above the threshold [0.0]  
[01:37:35] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0]  
[01:39:12] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-05 is OK: OK: Less than 1.00% above the threshold [0.0]  
[01:39:40] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-15 is OK: OK: Less than 1.00% above the threshold [0.0]  
[01:39:40] <shinken-wm>	 RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0]  
[01:39:54] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-tomcat is OK: OK: Less than 1.00% above the threshold [0.0]  
[01:40:24] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0]  
[01:40:48] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-generic-01 is OK: OK: Less than 1.00% above the threshold [0.0]  
[01:40:52] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-wmt is OK: OK: Less than 1.00% above the threshold [0.0]  
[01:41:42] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-02 is OK: OK: Less than 1.00% above the threshold [0.0]  
[01:41:42] <shinken-wm>	 RECOVERY - Puppet failure on tools-trusty is OK: OK: Less than 1.00% above the threshold [0.0]  
[01:42:00] <shinken-wm>	 RECOVERY - Puppet failure on tools-static is OK: OK: Less than 1.00% above the threshold [0.0]  
[01:57:51] <Mjbmr>	 can you start splinetools?
[01:58:56] <YuviPanda|bus>	 In about 5mins
[01:59:03] * YuviPanda|bus is walking home
[01:59:21] <Mjbmr>	 how was the fly?
[03:26:02] <YuviPanda|bus>	 Mjbmr: very tiring, mostly
[03:26:25] <YuviPanda|bus>	 !log tools manually started webservice for splinetools
[03:53:46] <YuviPanda>	 Merlissimo: I see lag reducing
[03:57:33] <wikibugs>	 3Tool-Labs, operations: Replag on labsdb - https://phabricator.wikimedia.org/T88183#1005949 (10yuvipanda) A few hours later! Replag is reducing. I killed a query on labsdb that looked inefficient and had been running for about a hour and a half (and was referencing the rev table, which was also what was implicat...
[04:35:29] <grrrit-wm>	 (03PS1) 10Mattflaschen: Update Phabricator projects for Collaboration team [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/187878 
[04:47:17] <wikibugs>	 3Tool-Labs, operations: Replag on labsdb - https://phabricator.wikimedia.org/T88183#1005975 (10yuvipanda) Lag is 0 everywhere according to tendril.  We really should have lag info public somewhere. Let me try to get it into graphite.
[04:47:19] <grrrit-wm>	 (03PS1) 10Legoktm: Remove no longer useful bugzilla comments [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/187880 
[04:47:21] <grrrit-wm>	 (03PS1) 10Legoktm: Update for project renames [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/187881 
[04:47:23] <grrrit-wm>	 (03PS1) 10Legoktm: Send Blocked-on-MediaWiki-Core to #mediawiki-core [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/187882 
[04:48:01] <grrrit-wm>	 (03PS1) 10Legoktm: Remove some more comments I missed [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/187883 
[04:48:30] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] Remove some more comments I missed [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/187883 (owner: 10Legoktm)
[04:48:34] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] Send Blocked-on-MediaWiki-Core to #mediawiki-core [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/187882 (owner: 10Legoktm)
[04:48:39] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] Update for project renames [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/187881 (owner: 10Legoktm)
[04:48:42] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] Remove no longer useful bugzilla comments [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/187880 (owner: 10Legoktm)
[04:49:01] <grrrit-wm>	 (03Merged) 10jenkins-bot: Remove no longer useful bugzilla comments [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/187880 (owner: 10Legoktm)
[04:49:02] <grrrit-wm>	 (03Merged) 10jenkins-bot: Update for project renames [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/187881 (owner: 10Legoktm)
[04:49:04] <grrrit-wm>	 (03Merged) 10jenkins-bot: Send Blocked-on-MediaWiki-Core to #mediawiki-core [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/187882 (owner: 10Legoktm)
[04:49:12] <grrrit-wm>	 (03Merged) 10jenkins-bot: Remove some more comments I missed [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/187883 (owner: 10Legoktm)
[04:50:51] <legoktm>	 YuviPanda: is ircnotifier not working?
[04:50:52] <legoktm>	 !log tools.wikibugs legoktm: Deployed 2541221f8d609c6d0fe86ade0906c60f02b1a831 Remove some more comments I missed wb2-irc
[04:51:04] <YuviPanda>	 legoktm: I killed it because I didn’t know how to kill your taylor swift bot :P
[04:51:10] <legoktm>	 oh lol
[04:51:24] <YuviPanda>	 legoktm: if you have that safely killed I’ll bring ircnotifier back again
[04:51:32] <legoktm>	 it should be safely killed
[04:52:14] <YuviPanda>	 legoktm: suuuuure? ;)
[04:52:24] <legoktm>	 YuviPanda: I moved the code somewhere else so it should be totally dead.
[04:52:28] <YuviPanda>	 restarting :)
[04:52:29] <YuviPanda>	 then
[04:52:51] <ircnotifier>	 !log tools.wikibugs legoktm: Deployed 6e130fecc19a39a5caed12ca0dda25ad28df62f0 -WikidataRepo is now -WikidataRepository wb2-irc
[04:52:51] <ircnotifier>	 !log tools.wikibugs legoktm: Deployed 2541221f8d609c6d0fe86ade0906c60f02b1a831 Remove some more comments I missed wb2-irc
[04:52:52] <YuviPanda>	 legoktm: back on
[04:53:35] <legoktm>	 heh
[04:53:44] <legoktm>	 I think labs-morebots is missing :P
[04:55:23] <grrrit-wm>	 (03Abandoned) 10Legoktm: Update Phabricator projects for Collaboration team [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/187878 (owner: 10Mattflaschen)
[05:34:03] <YuviPanda>	 !log shinken created shinken-jessie-test to do what the name suggests
[06:35:38] <shinken-wm>	 PROBLEM - Puppet failure on tools-mail is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[06:36:08] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-05 is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0]  
[06:42:52] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]  
[07:00:38] <shinken-wm>	 RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:01:12] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-05 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:07:49] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-04 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:32:14] <grrrit-wm>	 (03PS1) 10Yuvipanda: Move webservice2 to puppet repo [labs/toollabs] - 10https://gerrit.wikimedia.org/r/187890 
[08:24:30] <YuviPanda>	 !log tools cleaned out core dumps on exec-14 and -15
[08:31:48] <shinken-wm>	 RECOVERY - Puppet staleness on tools-submit is OK: OK: Less than 1.00% above the threshold [3600.0]  
[08:33:56] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-exec-14 is OK: OK: All targets OK  
[08:35:21] <shinken-wm>	 RECOVERY - Puppet staleness on tools-exec-07 is OK: OK: Less than 1.00% above the threshold [3600.0]  
[08:37:45] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-exec-15 is OK: OK: All targets OK  
[08:48:35] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] Move webservice2 to puppet repo [labs/toollabs] - 10https://gerrit.wikimedia.org/r/187890 (owner: 10Yuvipanda)
[08:54:05] <PissedPanda>	 !log tools restarted bigbrother2 to pick up webservice2 path changes
[08:58:58] <shinken-wm>	 PROBLEM - Puppet failure on tools-dev is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]  
[09:11:08] <PissedPanda>	 legoktm: ^ py3 support!
[09:11:10] <PissedPanda>	 bah
[09:13:55] <shinken-wm>	 RECOVERY - Puppet failure on tools-dev is OK: OK: Less than 1.00% above the threshold [0.0]  
[12:37:40] <JeroenDeDauw>	 I'd like to have a labs instance for a composer based extension tarball creation tool
[12:37:53] <YuviPanda>	 hi JeroenDeDauw 
[12:38:04] <YuviPanda>	 JeroenDeDauw: I think legoktm’s extdist already does that?
[12:38:15] <JeroenDeDauw>	 YuviPanda: link?
[12:38:50] <YuviPanda>	 JeroenDeDauw: no link, but if you try the ‘download snapshot’ from any extension page, https://www.mediawiki.org/wiki/Extension:ShortUrl you shall get a tarball
[12:39:00] <YuviPanda>	 see https://www.mediawiki.org/wiki/Special:ExtensionDistributor/ShortUrl for example
[12:39:12] <JeroenDeDauw>	 YuviPanda: that is not actually what I want
[12:39:29] <JeroenDeDauw>	 This tool if for selecting a bunch of extensions and their versions and getting a tarball for them
[12:39:31] <YuviPanda>	 it also supports packing in dependencies from composer if your extension has a composer.json file
[12:40:02] <YuviPanda>	 JeroenDeDauw: if you look at https://www.mediawiki.org/wiki/Special:ExtensionDistributor/ShortUrl it lets you pick the version as well
[12:40:04] <JeroenDeDauw>	 Sure, but pulling in the dependencies for extensions individually ultimately is missing the point and can easily break
[12:40:17] <YuviPanda>	 and see also https://www.mediawiki.org/wiki/Special:ExtensionDistributor
[12:40:31] <YuviPanda>	 he’s currently working on moving this service to production from labs.
[12:41:01] <YuviPanda>	 JeroenDeDauw: maybe (I’m thankfully blissfully unaware of the entire issues :)), but I highly reccomend talking to him before deciding to go on your own.
[12:41:07] <YuviPanda>	 JeroenDeDauw: and if you do, I’ve no problems setting up a project.
[12:41:17] <JeroenDeDauw>	 legoktm: ping
[12:41:41] <YuviPanda>	 JeroenDeDauw: but there again, I’d suggest looking at toollabs first. If you get your own project, you’re responsible for it (security updates, making sure it is properly up, etc). Labs admins only can help with the underlying OpenStack infrastructure.
[12:41:52] <YuviPanda>	 with great power (root, in this instance) comes great responsibility, etc
[12:42:08] <JeroenDeDauw>	 YuviPanda: I have the strong suspicion that the stuff I want to have is in some way incompatible with whatever constraints there are for the extension distributor thing
[12:42:34] <JeroenDeDauw>	 YuviPanda: I understand that
[12:42:45] <JeroenDeDauw>	 YuviPanda: what do you mean with security updates?
[12:42:57] <JeroenDeDauw>	 Of the stuff I put on there, or of everything?
[12:43:08] <YuviPanda>	 JeroenDeDauw: of everything on the VMs in your project, yeah. 
[12:43:49] <JeroenDeDauw>	 Yeah sure. If I don't get a labs thing I'd need to do this anyway on a vps of mine
[12:44:28] <YuviPanda>	 https://phabricator.wikimedia.org/T76375 is where you file a project request :) I’m still going to suggest at least *talking* with him, *and* considering toollabs if you need your own thing before setting up your own VM for it
[12:45:05] <wikibugs>	 3Labs: create ORES project in Labs - https://phabricator.wikimedia.org/T87494#1006214 (10yuvipanda) Do you want to re-use the revscoring project or have an ores one? I'm fine with either :)
[12:45:19] <JeroenDeDauw>	 toollabs?
[12:45:58] <YuviPanda>	 yeah, http://tools.wmflabs.org/
[12:46:04] <YuviPanda>	 https://wikitech.wikimedia.org/wiki/Help:Tool_Labs
[12:46:23] <JeroenDeDauw>	 Looking at this now
[12:46:32] <JeroenDeDauw>	 But not directly clear - how is this different from labs?
[12:47:37] <wikibugs>	 3Labs: New Labs project requests (Tracking) - https://phabricator.wikimedia.org/T76375#1006219 (10yuvipanda)
[12:47:40] <wikibugs>	 3Labs: irc bouncer project - https://phabricator.wikimedia.org/T87442#1006215 (10yuvipanda) 5Open>3declined a:3yuvipanda This was extensively discussed in the past and we decided we shouldn't be hosting IRC bouncers, for privacy and other reasons.   So... no :) But if you want to have a bouncer, me or @mar...
[12:47:41] <YuviPanda>	 JeroenDeDauw: you don’t get root on toollabs
[12:47:51] <YuviPanda>	 JeroenDeDauw: and we maintain the underlygin infrastructure
[12:48:02] <YuviPanda>	 it’s more akin to shared hosting in some senses, but of course not as dumbed down
[12:50:03] <JeroenDeDauw>	 YuviPanda: that might actually be fine - assuming no one messed with the tool
[12:50:12] <JeroenDeDauw>	 Then again, if that happens, it can be moved easily enough
[12:50:24] <YuviPanda>	 what do you mean by ‘no one messed with the tool'
[12:50:31] <JeroenDeDauw>	 So no problem with trying
[12:50:55] <YuviPanda>	 heh, cool
[12:51:25] <JeroenDeDauw>	 *messes
[12:52:53] <YuviPanda>	 JeroenDeDauw: anyway, let me know what your labs username is and I’ll add you to the tools project. And then you can create a tool and get on with writing code.
[12:53:38] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<11.11%)  
[13:03:38] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK  
[13:32:11] <YuviPanda>	 !log tools move flickr2commond to trusty
[13:37:01] <YuviPanda>	 !log tools admin moved to trusty and back to precise. It doesn’t seem to work on 5.5
[13:49:38] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<11.11%)  
[13:54:39] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK  
[13:55:16] <YuviPanda>	 !log mediahandler-tests restarted mediahandler-tests-mol instance, was in SHUTOFF state
[14:05:37] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<22.22%)  
[14:07:58] <JeroenDeDauw>	 YuviPanda: jeroendedauw
[14:11:43] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-exec-06 is CRITICAL: CRITICAL: tools.tools-exec-06.diskspace._var.byte_percentfree.value (<11.11%)  
[15:31:01] <wikibugs>	 3Labs: create ORES project in Labs - https://phabricator.wikimedia.org/T87494#1006282 (10Halfak) I think we should stick with 'revscoring'.  Thanks @yuvipanda.  And thanks @danilo.  I had forgotten that Yuvi had already created the 'revscoring' project for us when I talked to @danilo about making this task, so t...
[15:32:04] <wikibugs>	 3Tool-Labs: Provide a status page (list) of all active proxy definitions - https://phabricator.wikimedia.org/T88216#1006283 (10scfc) 3NEW
[15:34:43] <wikibugs>	 3Labs: New Labs project requests (Tracking) - https://phabricator.wikimedia.org/T76375#1006293 (10He7d3r)
[15:34:44] <wikibugs>	 3Labs: create ORES project in Labs - https://phabricator.wikimedia.org/T87494#1006291 (10He7d3r) 5Open>3declined
[17:27:48] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-exec-08 is CRITICAL: CRITICAL: tools.tools-exec-08.diskspace._var.byte_percentfree.value (<11.11%)  
[20:01:37] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<11.11%)  
[20:11:38] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK  
[20:26:21] <shinken-wm>	 PROBLEM - Puppet staleness on tools-exec-07 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [43200.0]  
[21:40:39] <legoktm>	 JeroenDeDauw: pong
[21:43:21] <legoktm>	 JeroenDeDauw: so you're building a tool where people just select all the extensions they want and it generates a tarball for them on the fly? interesting