[00:11:51] hi, I had configured bigbrother according to https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Grid#Bigbrother [00:12:16] but now it keeps restarting a job even after having deleted .bigbrotherrc [00:12:20] how can I stop it? [00:46:12] 6Labs, 6operations, 10wikitech.wikimedia.org: Expand list of people who can create new Labs project - https://phabricator.wikimedia.org/T101688#1390474 (10Legoktm) Do we currently have an issue with projects not being created in a timely manner? [03:58:13] !log deployment-prep stopped OCG on beta; redis 2.8.x is causing the service to crash on startup. [03:58:17] Logged the message, Master [04:24:52] hello [05:20:43] !log deployment-prep updated OCG to version d7c698d5bf730d34057945e912ac75dc542dd788 ; restarted service. [05:20:47] Logged the message, Master [05:33:43] 6Labs, 6Commons: Create labs project for Commons Archive - https://phabricator.wikimedia.org/T103471#1390961 (10zhuyifei1999) 3NEW [05:44:10] 6Labs, 10Labs-Infrastructure, 10OCG-General-or-Unknown, 6operations: salt on deployment-pdf02.deployment-prep.eqiad.wmflabs is wedged - https://phabricator.wikimedia.org/T103473#1390982 (10cscott) 3NEW [05:44:40] !log deployment-prep stopped OCG service on deployment-pdf02, see https://phabricator.wikimedia.org/T103473 [05:44:44] Logged the message, Master [05:46:28] 6Labs, 10Labs-Infrastructure, 10OCG-General-or-Unknown, 6operations: salt on deployment-pdf02.deployment-prep.eqiad.wmflabs is wedged - https://phabricator.wikimedia.org/T103473#1390993 (10cscott) Oh, I forgot to add: ``` cscott@deployment-pdf02:/srv/deployment/ocg/ocg/mw-ocg-service$ git log commit 2b2816... [09:27:40] 6Labs, 10Tool-Labs, 7HTTPS, 5HTTPS-by-default, 5Patch-For-Review: Fix all http-only tools in tools.wmflabs.org - https://phabricator.wikimedia.org/T102457#1391600 (10Magnus) I could add a proxy script on Labs, but my experience is that this does help neither speed nor reliability. [10:08:59] I used to ssh from bastion-01 to login.tools.wmflabs.org for my tool/bot maintenance. Now I am logged in bastion-01 but from there ssh to login.tools.wmflabs.org doesn't work. Something has changed? [10:09:58] rotpunkt: you can ssh directly to login.tools.wmflabs.org from your computer [10:10:03] no need to go through bastion-01 [10:11:01] and when asking do include an error message :) [10:11:02] from my local computer if I ssh to login.tools.wmflabs.org I reach bastion-01 [10:11:23] you would reach tools-bastion-01? [10:11:29] and that is the same as login.tools.wmflabs.org [10:11:34] yes [10:11:41] ok [10:11:41] yes, you have reached your destination :) [10:11:50] tools-bastion-01 is login.tools.wmflabs.org :) [10:12:19] ok, so I can run bot from bastion-01? [10:12:52] through jsub I mean [10:14:20] I ask this because in bastion-01 my crontab is commented out and all the logs are disappeared [10:14:41] rotpunkt: ah yes. that is the result of a labs outage from earlier. [10:15:00] the last run was 18 June... is it correct? [10:15:03] rotpunkt: you can uncomment the crontabs - we explicitly commented them out to not cause damage when running 9 days old code [10:15:07] yes [10:15:33] and please subscribe to the toos/labs announce mailing list :) [10:15:35] ok solved then! [10:16:24] * thedj is one to talk, cause i'm not subscribed either, but whatever :) [10:17:13] heh :) [10:17:19] only one thing... My bot logs have disappeared... [10:18:37] hi, is there a reason that the ~ directory has disapeard... in addition labs-vagrant failed because the file /vagrant/puppet/modules/labs/files/labs_privacy_policy.xml can not be found [10:25:49] 6Labs, 10Tool-Labs: Bring Jurytool back online - https://phabricator.wikimedia.org/T103003#1391800 (10yuvipanda) 5Open>3Resolved a:3yuvipanda All good. Tool Labs is back. [10:33:21] 6Labs, 6Commons: Create labs project for Commons Archive - https://phabricator.wikimedia.org/T103471#1391815 (10yuvipanda) Done. Do remember that you'll be responsible for maintenance / set up on the new project. Debian Jessie is highly reccomended for new installs :) [10:33:59] 6Labs, 7Tracking: New Labs project requests (Tracking) - https://phabricator.wikimedia.org/T76375#1391818 (10yuvipanda) [10:34:01] 6Labs, 6Commons: Create labs project for Commons Archive - https://phabricator.wikimedia.org/T103471#1391816 (10yuvipanda) 5Open>3Resolved a:3yuvipanda [10:37:13] thanks again YuviPanda (no problem for logs) [10:37:22] yw! [10:44:17] does anyone has a problem with the user home directory? [10:44:37] hey physikerwelt [10:44:42] is this for the math project? [10:44:49] yes [10:44:55] at least for my username [10:44:57] physikerwelt: there was a NFS corruption and labs was down for a few days. [10:45:16] physikerwelt: so everything was restored from a 9 day old backup, and NFS was removed from some projects, including the math project. [10:45:25] physikerwelt: I can help recover your files back [10:45:35] and copy them to whichever instance you want [10:45:47] or we could enable NFS again, but then that'll reduce the reliability of your labs instances [10:46:09] YuviPanda: ok thank you for the info [10:46:30] physikerwelt: you should subscribe to the labs-announce / labs-l mailing lists :) [10:46:55] I'm but I thought the problem would be solved [10:47:39] there are tons of emails every day... it's extremly hard to keep track of everything [10:48:09] heh :( [10:48:46] I realized that the labs instances had problems once in a while. I wrote a cronejob that submitted a test job everyminute and notified me on failure [10:48:46] physikerwelt: right, the problems were solved, but someone requested that we bring back the math project before NFS was available (it was down for multiple days) and so I disabled NFS [10:48:56] heh [10:50:14] YuviPanda: Is there a general problem with NFS? Otherwise I would vote for re-enabeling nfs [10:50:23] physikerwelt: yes, there is general problems with NFS. [10:50:34] physikerwelt: basically all our labs related outages in the last few months have been NFS related. [10:51:30] ok. Thank you for the info. I'll discuss that with hcohl and the others involved in the math project [10:51:36] physikerwelt: alright :) [10:52:09] physikerwelt: as labs admin, my preference would be for you guys to *not* use NFS, but I will help re-enable it if that's what you want. [10:52:12] YuviPanda thank you very much for your support and my aplogies for not following the mailinglist in detail [10:52:22] physikerwelt: no worries! [10:53:27] is there an alteranative for database backups and the transfer of larger files [10:53:41] 6Labs, 3Labs-Sprint-102, 3Labs-Sprint-103, 5Patch-For-Review: Disable NFS by default for new projects - https://phabricator.wikimedia.org/T102403#1391845 (10yuvipanda) I'm doing this atm manually by disabling NFS for projects as they get created :) [10:53:58] physikerwelt: right, so transfer of larger files should always be scp / rsync, IMO. [10:54:13] physikerwelt: backups - we can enable /data/project NFS mount only and you can use that for backups. [10:54:23] physikerwelt: a more general backup solution is on the cards sometime in the near future [10:56:32] data project would be really nice [10:57:30] but otherwise I could use scp as well [10:57:34] so don't worry about that [10:59:33] physikerwelt: ok :) [11:25:44] 6Labs, 10Tool-Labs, 7HTTPS, 5HTTPS-by-default, 5Patch-For-Review: Fix all http-only tools in tools.wmflabs.org - https://phabricator.wikimedia.org/T102457#1391880 (10yuvipanda) I could set up an 'official' NFS-independent https -> http proxy on tools (tools-https-proxy.wmflabs.org) that allows you to req... [11:26:14] 6Labs, 10Tool-Labs, 7HTTPS, 5HTTPS-by-default, 5Patch-For-Review: Fix all http-only tools in tools.wmflabs.org - https://phabricator.wikimedia.org/T102457#1391885 (10yuvipanda) I can also help the maintainer of stats.grok.de setup SSL, if so desired. [11:29:26] YuviPanda: isn't loading stuff from stats.grok.se against the privacy policy? [11:29:36] hahahahaaha [11:29:40] oh dear. [11:29:44] it probably is. [11:29:54] the proxy would solve both problems. [11:30:05] yeah [11:30:44] as for the proxy: just make sure to whitelist certain domains :-p [11:44:06] 6Labs, 10Labs-Infrastructure, 10OCG-General-or-Unknown, 6operations: salt on deployment-pdf02.deployment-prep.eqiad.wmflabs is wedged - https://phabricator.wikimedia.org/T103473#1391917 (10Krenair) > Probably some sort of permissions problem on pdf02? I don't have root on the ocg machines, so I can't fix i... [11:47:30] 6Labs, 10Labs-Infrastructure: Unable to see instances on Special:NovaInstance, web proxies on Special:NovaProxy, security groups on Special:NovaSecurityGroup - https://phabricator.wikimedia.org/T103495#1391920 (10Rillke) 3NEW [11:48:59] 6Labs, 10Labs-Infrastructure: Unable to see instances on Special:NovaInstance, web proxies on Special:NovaProxy, security groups on Special:NovaSecurityGroup - https://phabricator.wikimedia.org/T103495#1391928 (10Rillke) 5Open>3Resolved a:3Rillke Logout, login, works. Thanks, yuvi. [12:06:18] (03PS1) 10Sitic: Added namespace filtering [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/220119 (https://phabricator.wikimedia.org/T100157) [12:06:36] (03CR) 10Sitic: [C: 032 V: 032] Added namespace filtering [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/220119 (https://phabricator.wikimedia.org/T100157) (owner: 10Sitic) [12:12:04] (03PS1) 10Sitic: Switch from eventlet to gevent [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/220123 [12:12:18] (03CR) 10Sitic: [C: 032 V: 032] Switch from eventlet to gevent [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/220123 (owner: 10Sitic) [12:56:22] 6Labs, 10Tool-Labs, 3Labs-Sprint-103: Labs: Move tools-shadow off the same host as tool-master - https://phabricator.wikimedia.org/T103390#1392086 (10coren) Testing how gridengine fares if the shadow and master are not matching version in the `versionmismatch` project. [13:02:05] andrewbogott_afk: Having a odd issue you might have insight into: I seem to be unable to create a Precise instance in a new project (versionmismatch) - trusty works fine, but precise just "Failed to create instance.' [13:04:48] 6Labs, 10Labs-Infrastructure, 10OCG-General-or-Unknown, 6operations: salt on deployment-pdf02.deployment-prep.eqiad.wmflabs is wedged - https://phabricator.wikimedia.org/T103473#1392107 (10cscott) ``` ocg-render-admins: gid: 721 description: admins for pdf render (rt 6468) members: [cscott, s... [13:31:28] !log deployment-prep fixed salt on deployment-pdf02, restarted OCG there. [13:31:32] Logged the message, Master [13:35:30] Coren: I’ll have a look. What project? [13:35:42] andrewbogott: versionmismatch [13:35:54] andrewbogott: also new instance creation might be broken for jessie too? quarry-runner-02 stuck trying to contact puppetmaster for about 30 hours now [13:35:56] andrewbogott: I tried to create a small vm 'master' with Precise. [13:36:24] Coren: any chance it’s a name conflict? [13:36:28] Try a different name [13:36:40] andrewbogott: Do we still check for globally unique? [13:36:45] yes [13:36:47] andrewbogott: Ah. [13:37:29] andrewbogott: Oh duh, that was just it. :-/ [13:38:17] great :) [13:39:25] YuviPanda: quarry-runner-02 looks ok to me... [13:40:20] andrewbogott: -01 sorry [13:41:05] btw, YuviPanda, could you take a look at the puppet failure for tools-webproxy-01? that's rpobably a leftover from the outage (puppet disable?), but I'm not sure if you changed anything there [13:41:29] valhallasw: looking [13:42:43] YuviPanda: tools-redis-01 complains about a stale nfs mount; that probably just needs a remount but I'm not 100% how to do that [13:42:50] Error: /Stage[main]/Role::Labs::Instance/File[/data/scratch]: Could not evaluate: Stale file handle - /data/scratch [13:42:58] Coren: ^ [13:42:59] or should maybe just not have nfs at all? [13:43:09] YuviPanda: it looks to me like hiera is messed up for that instance. I created a jessie instance in a different project and it’s fine. [13:43:34] andrewbogott: hmm, so it has NFS disabled except for /data/project. [13:44:16] valhallasw: On it. [13:44:37] YuviPanda: why do you think the failure has to do with nfs? It’s trying to contact the puppetmaster but thinks the master is named “” [13:44:43] which it gets from a hiera lookup. [13:45:01] andrewbogott: oh, hmm. because that's the only bits on https://wikitech.wikimedia.org/wiki/Hiera:Quarry [13:45:26] valhallasw: That was definitely it. Fixed with 'umount /data/scratch; mount /data/scratch' [13:46:00] valhallasw: we should disable NFS on places it isn't required (redis, webproxy, static, etc) [13:46:11] *nod* [13:46:13] thanks, Coren [13:47:09] andrewbogott: transient failure, and then it got stuck? [13:47:14] andrewbogott: should I recreate it? [13:47:18] sure, try. [13:47:45] 6Labs, 10Wikimedia-Extension-setup, 10wikitech.wikimedia.org, 5Patch-For-Review: Re-enable OAuth in Wikitech - https://phabricator.wikimedia.org/T98567#1392312 (10Krenair) 5Open>3Resolved [13:50:23] 6Labs: Find and clean up instances that are unreachable by ssh - https://phabricator.wikimedia.org/T103522#1392343 (10yuvipanda) 3NEW [13:50:30] andrewbogott: ^ as well (unrelated) [13:51:24] 6Labs: Find and clean up instances that are unreachable by ssh - https://phabricator.wikimedia.org/T103522#1392360 (10Andrew) Do you think we should fix them, or just delete? [13:52:00] 6Labs: Find and clean up instances that are unreachable by ssh - https://phabricator.wikimedia.org/T103522#1392366 (10yuvipanda) SHUTOFF should just delete, the other ones I realistically think we have no choice but to just delete - but we should delete with notification, I think [13:53:03] valhallasw: tools-webproxy-01 was just puppet being disabled. shoudl be ok now. [13:53:13] andrewbogott: any easy way to figure out which of those are in SHUTOFF? [13:53:33] nova certainly knows [13:54:37] 6Labs: Find and clean up instances that are unreachable by ssh - https://phabricator.wikimedia.org/T103522#1392387 (10Andrew) Many instances without ssh can still be reached by salt -- so that's a way to double-check things. Also note that some instances are SHUTOFF because the owners legitimately want them sav... [13:54:43] 6Labs, 10wikitech.wikimedia.org, 7Shinken: shinken thinks wikitech and wikitech-static are down. They aren't. - https://phabricator.wikimedia.org/T101517#1392388 (10Krenair) [13:56:10] YuviPanda: I would love it if you would fix ^ and the same for wikitech-static. I hate not being able to trust my alerts. [13:56:22] meanwhile, breakfast... [13:56:27] andrewbogott: I'll take a look once the ssh backport stuff is done [13:56:32] thanks [14:08:27] andrewbogott: would you look on why the other two admins set in Special:NovaProject is not shown in the admin list in https://wikitech.wikimedia.org/wiki/Nova_Resource:Commonsarchive [14:11:32] hello I know maps still remains to be fixed. Is the maps-warper one of those? I am getting permission denied (publickey) from bastion. [14:12:41] (03PS1) 10Sitic: Disable cache for index.html, fix logout method [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/220136 [14:12:59] (03CR) 10Sitic: [C: 032 V: 032] Disable cache for index.html, fix logout method [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/220136 (owner: 10Sitic) [14:13:45] I note that maps-warper is not on the list here: https://phabricator.wikimedia.org/P816 [14:18:52] 6Labs, 6operations: Recover home folders and /data/project from wikimetrics1 - https://phabricator.wikimedia.org/T103530#1392478 (10Krenair) [14:20:50] 6Labs, 10Tool-Labs, 3Labs-Sprint-101, 3Labs-Sprint-102, and 3 others: Puppetize toolserver.org redirect configuration - https://phabricator.wikimedia.org/T85165#1392487 (10coren) [14:24:06] andrewbogott, is https://wikitech.wikimedia.org/wiki/File:Lifeofpuppetpatch.png still mostly up-to-date other than host names? [14:25:45] I think so. [14:26:00] If I have the raw image files I’ll update [14:26:28] andrewbogott: would you look on why the other two admins set in Special:NovaProject is not shown in the admin list in https://wikitech.wikimedia.org/wiki/Nova_Resource:Commonsarchive [14:27:01] manganese -> ytterbium, virt0 -> labcontrol1001?, what about rhodium? [14:28:07] I don’t know what rhodium is — is it another master? [14:28:24] node /^(strontium|rhodium).eqiad.wmnet/ { [14:28:24] include standard [14:28:24] include role::puppetmaster::backend [14:28:59] So I guess that counts as (and others) [14:34:51] Coren, are you available? [14:35:10] Cyberpower678: Depends what for. :-) What can I do for you? [14:35:28] Can I borrow you as a BAGer. It says you are listed as semi-active [14:37:01] Coren, Can you go over Cyberbot II 5 [14:37:14] I believe it's ready for a trial. [14:37:14] 6Labs, 10Labs-Infrastructure, 10Wikimedia-Apache-configuration, 6operations, and 2 others: wikitech-static sync broken - https://phabricator.wikimedia.org/T101803#1392653 (10Dzahn) {F182545} [14:39:56] Cyberpower678: I'll have a look after my work hours. :-) [14:40:38] Coren, thanks. :-) [14:43:28] zhuyifei1999: I’m running a job that might/should update the Nova_Resource pages. In general, though, you shouldn’t believe what you read on those, they tend to drift. [14:43:53] ok [15:00:42] anyone using mosh on labs? i see it's running on the bastion, but since you can't forward auth, wondering if anyone has a solution? hesitant to just stick keys on the bastion. [15:01:28] 6Labs, 6WMF-Legal: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#1392714 (10d33tah) I tried contacting @zhouz privately but he didn't reply. Is there anybody else I should ask? [15:01:49] cwdent: nope, do not stick keys on bastion. there's currently no solution for the general case :( [15:01:53] cwdent: within tool labs, ssh should 'just work'. For other hosts, creating a new private key and saving that on bastion is the only option [15:02:41] (03PS1) 10Sitic: Revert angularjs from 1.4.1 to 1.4.0 [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/220146 [15:02:58] (03CR) 10Sitic: [C: 032 V: 032] Revert angularjs from 1.4.1 to 1.4.0 [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/220146 (owner: 10Sitic) [15:03:28] yes, but I will delete private keys found on the bastion hosts. [15:03:29] :P [15:04:20] ... why, exactly? [15:04:25] it's pretty much host-based auth [15:04:26] ok! i see forwarding auth is on the mosh roadmap, though i don't really understand how it would work with the way it works [15:04:41] and agent forwarding with a key used elsewhere is much more dangerous [15:05:00] valhallasw: because of the number of people who put their primary key in there. [15:05:26] indeed, and I"d disable agent forwarding too if I can, but unfortunately that's not possible at the moment. [15:05:32] but this is something that can be nipped in the bud and should be [15:06:08] what is the ideal solution? passwords? :D [15:06:09] I disagree^1000 [15:06:37] because, with mosh, there /is/ no solution that's not either HBA, agent forwarding or a local PK [15:06:51] and those all carry the same risk profile (a root impersonating as you) [15:07:49] oh wait...just looked at my ssh config and it looks like it's using a tunnel and not auth forwarding? [15:07:57] Seems like the solution is for mosh to support proxycommand [15:08:11] and until then ‘mosh is not supported' [15:08:14] yea [15:08:16] what andrewbogott said [15:08:24] it's a problem with mosh [15:08:32] wat. [15:08:38] what you propose is impossible with mosh [15:08:41] err? [15:08:41] no [15:08:50] well, maybe with hole punching you can get somewhere [15:08:52] but mosh over an ssh tunnel seems pretty useless [15:08:53] there's a bug about it [15:09:17] https://github.com/keithw/mosh/issues/285 [15:09:18] would just die with the tunnel right [15:09:19] open since 2012 [15:09:30] without much response from upstream [15:09:37] it's a nontrivial problem [15:09:47] it would require mosh on the target host, and hole punching between you and that host [15:09:57] i thought "The mosh client logs in to the server via SSH, and users present the same credentials (e.g., password, public key) as before." [15:10:23] either way, it's a missing feature in mosh. [15:10:28] and anyone who says 'people should just use ssh' I laugh at [15:10:33] saying 'just put your private key on the bsation' is not an option [15:10:34] hello latency, how are you doing [15:10:41] YuviPanda: a *NEW* private key [15:10:42] and yes, it is [15:10:49] it is a completely sensible option, fully comparable to HBA [15:11:13] 3 differen tpeople had their primary ssh key on the bastion host [15:11:18] so we need labs in esams/ [15:11:28] or someone could instead help patch mosh :P [15:11:43] YuviPanda: I feel like I'm repeating myself. It's a completely nontrivial problem [15:11:45] anyway, at some point when all of this settles down we want to restrict shell on bash. [15:12:12] valhallasw: sorry, secret keys on bastion is not an acceptable solution to me nor other opsen. You repeating that it is is going to get the exact same response from us, so we are in a loop there, yes. [15:12:22] YuviPanda: then why is HBA acceptable? [15:12:49] because you won't find people putting their primary ssh private keys on a random host due to HBA? [15:12:58] krenair@deployment-mathoid:~$ /usr/lib/nagios/plugins/check_http -H localhost -p 10042 [15:12:58] HTTP WARNING: HTTP/1.1 404 Not Found - 741 bytes in 0.003 second response time |time=0.003405s;;;0.000000 size=741B;;;0 [15:12:59] Weird. [15:13:42] YuviPanda: eh. and how is that different from not using your primary key, but rather a new one? [15:13:55] good luck enforcing that. [15:14:15] anyway, we want to restrict the shell on bastion eventually to only allow very specific things. [15:14:58] YuviPanda: eh, check id_rsa.pub whether it contains 'bastion'? it's not perfect, but solves the 99% case. [15:15:12] * YuviPanda continues loop [15:15:16] no private keys on bastion :) [15:15:35] I like how you don't provide any arguments and just repeat yourself indeed. [15:16:13] YuviPanda: we could provide a programattic way to generate approved, distinctive keys on bastion and exclude them from the ‘no private keys’ rule. [15:16:26] If the only concern is people breaking security by uploading dangerous ones. [15:17:37] Huh. [15:17:40] same 404 in prod. [15:17:56] I wonder why icinga is happy in prod but shinken is warning in labs for that. [15:19:35] valhallasw: I have many times - 'people put their primary key in there' and 'it is terrible security practice to encourage people to maintain private keys on machines not directly under their control'. [15:20:18] mosh not working with bastion based solutions is a known mosh problem. [15:20:59] YuviPanda: 1) I'm not arguing people should put their primary key in there. This is fixable. 2) Maybe, but security-wise, a local pk is completely comparable to HBA. [15:21:08] Krenair: maybe because what "localhost" resolves to is different [15:21:13] indeed, and we *don't* have HBA on bastions [15:21:22] (I had an old puppet repo checked out) [15:21:32] YuviPanda: tools-login is not a bastion? :P [15:21:43] I have no energy to have this argument at the moment. sorry. [15:22:00] but I agree [15:22:02] in my ideal world [15:22:06] we wouldn't have HBA on tools either [15:22:22] YuviPanda: what security risk do you want protecting against? [15:22:28] hey [15:22:31] put your private key there! [15:22:42] put everyone's private keys there! [15:22:42] have fun! [15:23:37] 6Labs, 10Labs-Infrastructure, 3Labs-Sprint-103: Instances without a shared NFS storage suffers from a 3 minutes boot delay - https://phabricator.wikimedia.org/T102544#1392841 (10coren) The root issue, of course, if the risk that puppet runs before the NFS server has been updated with the new exports. A plau... [15:27:06] YuviPanda: Have you ever played with nodejs on Labs? [15:27:21] Tool labs, more specifically. [15:27:39] guillom: as web service or as cli tool? both is possible [15:28:03] guillom: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web#node.js_web_services [15:28:04] valhallasw: CLI at the moment. I'm having trouble connecting to mysql [15:28:09] ah, ok. [15:28:52] I'm using https://github.com/felixge/node-mysql and I keep getting the ECONNREFUSED error. I'm wondering what I might be doing wrong. [15:29:16] valhallasw: Do you know on any tools that successfully use nodejs with the db? [15:29:34] not sure. ECONNREFUSED sounds like you're not connecting to the right server, but e.g. to localhost [15:29:36] If I had a working example, it would help me debug. [15:30:03] I thought the db was on localhost; is it not? [15:30:36] no, it's on nlwiki.labsdb or toolsdb.labsdb, depending on what host you're looking for [15:30:49] Aaah, but of course. [15:31:13] I mixed up database and server. [15:31:19] Silly me. [15:31:31] valhallasw: Thank you so much for unbjocking me :) [15:31:36] yw :-) [15:31:41] unblocking, too. [15:33:21] Weee. It works \o/ [15:47:10] 6Labs: Support connections from bastion to other hosts - https://phabricator.wikimedia.org/T103552#1392968 (10valhallasw) 3NEW [15:47:37] YuviPanda, where would you go to change the check that shinken performs for a service? [15:51:04] 6Labs: Support connections from bastion to other hosts - https://phabricator.wikimedia.org/T103552#1392994 (10valhallasw) The obvious best option is 1), but UDP hole punching is unlikely to become available soon (the bug has been open for two years). As far as I am concerned, 4) is a reasonable security comprom... [15:52:54] 6Labs: Support connections from bastion to other hosts - https://phabricator.wikimedia.org/T103552#1393004 (10yuvipanda) HBA on bastion seems the least bad option that's fully under our control. This requires that we extract the host key of all hosts securely, which *might* be actually possible with nova - I've... [15:54:17] YuviPanda: if we can do hba, that's +100 for me. That also allows one to set rules for their own project (so e.g. it would work for bots but not beta) [15:54:30] valhallasw: yup. [15:57:35] 6Labs: Support connections from bastion to other hosts - https://phabricator.wikimedia.org/T103552#1393015 (10yuvipanda) @MoritzMuehlenhoff can hopefully give more concrete reasons for why we shouldn't let people put manual private keys in their .ssh/ on bastions, but to me it is: # I've had to email at least 3... [15:57:39] valhallasw: we should've also started on the ticket before flaming on IRC :D much easier this way. [15:57:53] maybe =p [15:58:45] YuviPanda: 'making ssh work seamlessly' is the obvious way to prevent people uploading their private keys :-p [15:58:56] it does with proxycommand :P [15:59:13] except for laa. aaa ag [15:59:25] a cron job that cleans them out is also a wonderful way to prevent people from uploading their private keys :P [15:59:30] (not that I'm doing that atm) [15:59:40] maybe. or people will just upload them again and again :P [15:59:47] heh [16:00:23] how about a script that removes the uploaded keys from puppet's allowed list? :) [16:00:37] "this key appears to be insecure, it has been removed for security purposes" [16:01:41] heh [16:01:48] Krenair: depends on for what? [16:01:51] Krenair: for beta? [16:01:59] Krenair: they'll be in the shinken module in operations/puppet [16:02:19] for a service in beta [16:03:01] specifically the mathoid services on deployment-mathoid and deployment-sca02 [16:03:11] YuviPanda: hm, if I read https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Host-based_Authentication correctly, it actually only requires the target host to know the client host, not the other way around. [16:03:20] alex@alex-laptop:~/Development/Wikimedia/Operations-Puppet (production)$ grep math modules/shinken/* -R [16:03:20] alex@alex-laptop:~/Development/Wikimedia/Operations-Puppet (production)$ [16:03:30] valhallasw: hmm, so if that's all it needs we can maintain the bastion fingerprints in hiera. [16:04:27] yeah. I'll fiddle a bit with a toolsbeta host to see if I can then login from tools-login. [16:04:45] valhallasw: cool. [16:05:04] valhallasw: hmm, I also have this in my head that we need both hosts' fingerprints. do you know where we first heard that? [16:05:08] * YuviPanda can't trace the source of that info [16:05:25] YuviPanda: it says "The remote server's host key must be stored on the client, either in the global client configuration file, /etc/ssh/ssh_known_hosts, or the relevant users' configuration files in ~/.ssh/known_hosts." [16:05:30] but that's retrieved on first connect anyway [16:05:49] right [16:05:52] it does prompt you... [16:06:14] but yeah, I think an ideal (outside of mosh adding proxycommand support) would be 1. lock down bastion, 2. HBA [16:06:55] ok [16:10:28] YuviPanda: hrm, toolsbeta-logstash doesn't accept my key? I tried rebooting, but to no avail [16:10:35] valhallasw: oh, let me see [16:10:39] with ProxyCommand ;-) [16:10:50] it might've had really old puppet and be still fucked? [16:10:55] oooh [16:10:57] yeah, could be [16:10:58] valhallasw: did the reboot itself succeed? [16:11:03] https://wikitech.wikimedia.org/w/index.php?title=Special:NovaInstance&action=consoleoutput&instanceid=3aecdded-b4bb-47fc-86ed-8e26a741ba50&project=toolsbeta®ion=eqiad [16:11:08] Cloud-init v. 0.7.5 finished at Tue, 23 Jun 2015 16:09:15 +0000. Datasource DataSourceOpenStack [net,ver=2]. Up 27.61 seconds [16:11:11] looks good to me? [16:11:21] although there's no init output it seems [16:11:32] but sshd is up [16:14:15] valhallasw: I can get in [16:14:43] :/ [16:14:46] debug1: No more authentication methods to try. [16:14:46] Permission denied (publickey). [16:14:54] valhallasw: I mean, with root key [16:15:17] oh :p [16:16:55] Change on 12www.mediawiki.org a page Wikimedia Labs was modified, changed by Gonza ostro link https://www.mediawiki.org/w/index.php?diff=1688561 edit summary: [+147] is very easy [16:19:32] Change on 12www.mediawiki.org a page Wikimedia Labs was modified, changed by Andrewbogott link https://www.mediawiki.org/w/index.php?diff=1688562 edit summary: [-147] Undo revision 1688561 by [[Special:Contributions/Gonza ostro|Gonza ostro]] ([[User talk:Gonza ostro|talk]]) [16:21:18] valhallasw: you should use another host for now. I think all of toolsbeta is basically broken [16:21:54] Change on 12www.mediawiki.org a page Wikimedia Labs was modified, changed by Gonza ostro link https://www.mediawiki.org/w/index.php?diff=1688564 edit summary: [+188] IS VERY COLD [16:22:54] Change on 12www.mediawiki.org a page Wikimedia Labs was modified, changed by Glaisher link https://www.mediawiki.org/w/index.php?diff=1688565 edit summary: [-188] Undo revision 1688564 by [[Special:Contributions/Gonza ostro|Gonza ostro]] ([[User talk:Gonza ostro|talk]]) [16:25:47] valhallasw: puppet fixed on the logstash instance [16:27:55] Thanks [16:28:23] YuviPanda or Coren: any chance https://phabricator.wikimedia.org/T103136 could be looked at sometime soon? [16:28:43] valhallasw: yw. also, I'm thinking we should add toollabs root's keys as root key on the instances in some way. [16:28:52] Nettrom: sure, let me do that now. [16:29:03] YuviPanda: awesome, thanks a bunch! [16:29:09] Nettrom: yw [16:42:33] Nettrom: done. [16:42:38] Nettrom: remember to pass -l release=trusty [16:44:54] YuviPanda: thanks, both for taking care of it and the reminder to specify the release! [16:45:00] yw [16:45:13] Nettrom: it should roll out in the next 20min [16:47:02] YuviPanda: let's see if I can accidentally lock myself out =p [16:47:10] (I'll start a second ssh server) [16:49:52] valhallasw: heh :) [16:50:07] or better, I can test sshd with a seperate config file [16:55:46] 6Labs, 10wikitech.wikimedia.org: Automatically grant shell user right to everyone who signs up on wikitech - https://phabricator.wikimedia.org/T97334#1393243 (10yuvipanda) So now we just need to check when adding to a project and if shell bit isn't set set it. Let me see if I can write up a patch for that. [16:56:37] 6Labs: Support connections from bastion to other hosts - https://phabricator.wikimedia.org/T103552#1393255 (10scfc) I haven't actually used `mosh` more than just now for testing it, but the (only) advantage that was described to me was local echo (and I personally would put "advantage" in quotation marks as on t... [17:07:13] 6Labs, 10wikitech.wikimedia.org, 5Patch-For-Review: Automatically grant shell user right to everyone who signs up on wikitech - https://phabricator.wikimedia.org/T97334#1393279 (10yuvipanda) ^ should fix it. @andrew thoughts / objections? It automatically adds to the shell group people the first time they ge... [17:08:27] YuviPanda: hahahahaha [17:08:28] that was easy [17:09:04] 6Labs, 7Puppet: Fix Puppet timestamp updater for wikitech - https://phabricator.wikimedia.org/T97082#1393289 (10Andrew) 5Open>3Resolved moot due to 220ffc9cef589efa0cde2defdc1e57a5fbf853e2 [17:09:51] valhallasw: ? [17:09:54] valhallasw: which one? [17:09:56] HBA [17:10:01] from tools-login to toolsbeta-logstash [17:10:11] so we just need tools-login's thing? [17:10:37] yes [17:10:43] hahaha [17:10:47] or bastions [17:10:47] oh dear. [17:10:49] right [17:10:52] just bastions, I mean [17:10:54] bastion-01 [17:10:54] and bastion needs to be setup to allow signing [17:10:56] -02 has same key [17:11:01] 'signing'? [17:12:08] the bastion signs the request from the server [17:12:24] right, that's just an sshd option, I guess? [17:12:27] yeah [17:12:55] valhallasw: right, so we can do that for bastions and this all becomes a little moot, I guess. [17:13:14] let me fiddle a bit more to clarify the needed changes [17:13:41] valhallasw: yeah. I do want moritz to say ok before we go ahead with it, but that should be doable. [17:13:47] sure [17:13:54] if it's per-project it should be OK anyway [17:15:47] valhallasw: yeah. we can make it opt-in easily. [17:15:50] via a hiera setting [17:16:27] 6Labs, 10Tool-Labs, 5Patch-For-Review: Install Scipy for Python 3 on Tool Labs - https://phabricator.wikimedia.org/T103136#1393316 (10yuvipanda) 5Open>3Resolved a:3yuvipanda [17:16:37] YuviPanda: https://phabricator.wikimedia.org/P827 [17:17:18] valhallasw: hmm, right. so won't need to 'collect' these or anything. [17:17:36] these hardly change [17:17:38] and can be in hiera [17:18:40] 'collect'? [17:19:10] host key can be in hiera yes [17:19:44] or hard-coded in a puppet manifest, I guess? [17:21:11] 6Labs: Support connections from bastion to other hosts - https://phabricator.wikimedia.org/T103552#1393324 (10cwdent) For my money the big advantage of mosh is it invisibly reconnects and sends history when your laptop wakes up or as one roams among wifi networks, which is a really seamless experience compared t... [17:28:07] 6Labs, 7Puppet: Fix Puppet timestamp updater for wikitech - https://phabricator.wikimedia.org/T97082#1393354 (10scfc) 5Resolved>3Invalid (And here as well: The assignee didn't fix it, so it's not resolved :-).) [17:35:34] 10Tool-Labs-xTools: wikiviewstats rapidly fills error.log - https://phabricator.wikimedia.org/T100572#1393388 (10Andyrom75) I agree with valhallasw. If no one is currently take care about the log error file, it's better to turn it off, as a temporary patch. Once the cause of the rows written in it has been found... [17:40:59] 10Tool-Labs-xTools: wikiviewstats - No db-connection - https://phabricator.wikimedia.org/T91320#1080054 (10Andyrom75) Any progress on the resolution of the wrong DB connection that prevent Wikiviewstats from running? [17:43:24] 10Tool-Labs-xTools: wikiviewstats - No db-connection - https://phabricator.wikimedia.org/T91320#1393441 (10Technical13) a:5Technical13>3None [17:44:39] 10Tool-Labs-xTools: wikiviewstats rapidly fills error.log - https://phabricator.wikimedia.org/T100572#1393443 (10Technical13) The webservice was stopped a couple days ago until it can be worked on by someone. [17:44:52] 10Tool-Labs-xTools: wikiviewstats rapidly fills error.log - https://phabricator.wikimedia.org/T100572#1393446 (10Technical13) p:5Triage>3Normal [17:46:45] 10Tool-Labs-xTools: wikiviewstats - No db-connection - https://phabricator.wikimedia.org/T91320#1393452 (10Cyberpower678) a:3Cyberpower678 [17:56:35] 10Tool-Labs-xTools: wikiviewstats rapidly fills error.log - https://phabricator.wikimedia.org/T100572#1393487 (10Glaisher) https://github.com/wikimedia/labs-tools-wikiviewstats ?? [18:00:07] 6Labs, 6WMF-Legal: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#1393491 (10ZhouZ) Hi @d33tah, Apologies for the late reply. Your comment on the Phabricator ticket came while I was away out of the office without access to email (and very unfortunately), you email below... [18:16:17] 6Labs, 6WMF-Legal: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#1393555 (10d33tah) @ZhouZ: I added a cron job that should handle the purging on the first day of every month, is that okay? Is the rest of the privacy policy good enough? [18:22:18] 10Tool-Labs-xTools: wikiviewstats - No db-connection - https://phabricator.wikimedia.org/T91320#1393574 (10Cyberpower678) OMG, the code. Hedonil certainly has a unique way of writing code. @technical13 we may have to rewrite this too when we move it to xTools. [18:26:21] YuviPanda: dumdiedum this isn't too hard [18:27:14] let me submit a patch [18:31:06] 6Labs, 5Patch-For-Review: Support connections from bastion to other hosts - https://phabricator.wikimedia.org/T103552#1393607 (10valhallasw) After further investigation, HBA is actually fairly trivial to add (see patch, which allows login from tools-login-01 to whichever host that patch is applied to). [18:47:58] 6Labs, 10Tool-Labs, 10Continuous-Integration-Infrastructure: Recover homedir of "ci" tool - https://phabricator.wikimedia.org/T103205#1393713 (10Jdforrester-WMF) p:5Triage>3Normal [18:52:23] 10Tool-Labs-xTools: wikiviewstats - No db-connection - https://phabricator.wikimedia.org/T91320#1393732 (10Technical13) Yep. That's why I killed the webservice for it the other day. It's not something that is going to happen soon based on our current levels of free time. I got midterms in a week, so cramming for... [19:08:21] [13intuition] 15Krinkle pushed 2 new commits to 06master: 02https://github.com/Krinkle/intuition/compare/ced3df196885...a51ca65e1e27 [19:08:21] 13intuition/06master 14925cdb0 15Timo Tijhof: build: Collapse npm-test into composer.json [19:08:22] 13intuition/06master 14a51ca65 15Timo Tijhof: builld: Add HHVM to Travis CI build matrix... [19:08:35] 6Labs, 10Tool-Labs, 3Labs-Sprint-103: Labs: Move tools-shadow off the same host as tool-master - https://phabricator.wikimedia.org/T103390#1393772 (10coren) Gridengine does not seem to suffer from the version mismatch (6.2u5-4 vs 6.2u5-7.3) and the configuration ports without difficulty. Therefore, the plan... [19:19:37] 6Labs, 10Tool-Labs: Move tools-master and tools-shadow to trusty - https://phabricator.wikimedia.org/T94791#1393818 (10Sitic) [19:21:15] 10Tool-Labs-tools-wikibugs-IRC-bot: Bot should use notices in-channel - https://phabricator.wikimedia.org/T72881#1393829 (10valhallasw) a:5valhallasw>3None [19:25:32] 6Labs, 10Tool-Labs: Move tools-master and tools-shadow to trusty - https://phabricator.wikimedia.org/T94791#1393833 (10coren) 5Open>3Resolved a:3coren Doing simultaneously with the parent ticket (that is, the new tools-shadow-01 is constructed with Trusty from the start) [19:25:33] 6Labs, 10Tool-Labs: Phase out precise instances from toollabs - https://phabricator.wikimedia.org/T94790#1393836 (10coren) [19:32:10] 6Labs, 6operations, 7Puppet: Labs puppet breaks for projects without a Hiera: page on wikitech - https://phabricator.wikimedia.org/T101913#1393877 (10RobH) [19:34:58] 10Tool-Labs-xTools: wikiviewstats - No db-connection - https://phabricator.wikimedia.org/T91320#1393887 (10Cyberpower678) I looked and looked, and for the life of me cannot figure out what's causing the problem, or how to even read this. I'd need to open a million files just to even follow along. [19:35:37] Coren, do you have a gun? [19:36:23] Cyberpower678: ... I have a rifle and a shotgun. Why? [19:37:10] Coren, looking at Wikiviewstats and even attempting to find anything in that code makes me want to shoot myself. :/ [19:37:32] Ah, sorry, my firearms are not intended to be pointed at humans. :-) [19:38:02] Coren: are they licensed? :) [19:38:10] Coren, I find the DB connection function and can't figure out why in the world the DB connection is failing. *bangs his head on a metal table. [19:40:11] Boy i can't wait till Coren approves my bot for a trial. :D [19:40:36] At least I'll be staring at code that I can debug. :p [19:48:31] 6Labs, 3Labs-Sprint-103, 5Patch-For-Review: decouple role::labs::instance puppet runs from the rest of puppet - https://phabricator.wikimedia.org/T103357#1393925 (10Andrew) Puppet is really determined to compile the whole catalog even if it's only applying a few things. I don't see a good way to exclude the... [19:50:03] 6Labs, 3Labs-Sprint-103, 5Patch-For-Review: decouple role::labs::instance puppet runs from the rest of puppet - https://phabricator.wikimedia.org/T103357#1393928 (10Andrew) One option is to create an ldap node called 'basicpuppet.node.eqiad.wmflabs' in ldap and have every node use that node definition for th... [19:58:20] 6Labs: Find and clean up instances that are unreachable by ssh - https://phabricator.wikimedia.org/T103522#1393954 (10yuvipanda) Do you think you can produce a list of SHUTOFF instances (I couldn't figure out how to quary it from the nova commandline) and so we can find the list of instances that are on but unre... [19:58:40] 6Labs, 10Labs-Other-Projects, 3Labs-Sprint-103: investigate/clean up 'servermon' project - https://phabricator.wikimedia.org/T103149#1393955 (10Andrew) [19:59:03] 6Labs, 10Labs-Other-Projects, 3Labs-Sprint-103: investigate/clean up 'servermon' project - https://phabricator.wikimedia.org/T103149#1383459 (10Andrew) Alex -- I think this was your project which I deleted... can you confirm that that sounds right? If so I'll delete the orphaned instances as well. [20:01:26] 6Labs, 3Labs-Sprint-102, 3Labs-Sprint-103, 5Patch-For-Review: Replace puppetsigner with a script to clean certificates, puppet's autosign and salt's auto accept - https://phabricator.wikimedia.org/T102504#1393967 (10Andrew) Moritz, can you confirm that salt auto-accept and puppet autosign don't seem danger... [20:07:28] 6Labs: Find and clean up instances that are unreachable by ssh - https://phabricator.wikimedia.org/T103522#1393997 (10Andrew) nova list --all-tenants | grep SHUTOFF [20:20:22] (03PS1) 10Sitic: Added a canary to the coal mine [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/220294 [20:22:58] sitic: why does ^ depend on the tools-shadow thing? [20:23:53] legoktm: I wrote the script in python>=3.3 and the host which currently executes it only has python 3.2 (it's still running precise) [20:24:10] sitic: can't you do -l release=trusty or something? [20:24:52] no, I need a host which can qdel jobs and start them again, I can only do that through jlocal [20:25:01] oh [20:25:07] Is it possible to get shared project storage after instances have been created? I just enabled them on Special:NovaProject&action=configureproject but it doesn't appear to take effect immediately... [20:25:35] sadly the normal exec nodes are not also submission hosts :-( [20:26:35] sudo puppet agent -tv -- Error messages: http://pastebin.com/6HceMvMC [20:27:29] How would I create labstore.svc.eqiad.wmnet:/project/mediahandler-tests/project ? [20:27:49] hey rillke [20:28:00] what do you need shared storage for? [20:28:18] I want to transfer date from one instance to another [20:28:22] scp / rsync? [20:28:42] NFS on an instance basically reduces reliability by orders of magnitude for your instance, and ideally you shouldn't use NFS [20:28:55] we're writing all of this up as we speak as we recover from the major outage last week... [20:29:45] ah, NFS is behind the "Create shared project storage" checkmark [20:29:52] that isn't going to work atm [20:30:11] NFS on new instances is effectively broken right now [20:30:26] "Therefore, we strongly recommend not to save data in spaces that are accessible to individuals only." https://wikitech.wikimedia.org/wiki/Help:Shared_storage#Introduction [20:30:51] 6Labs, 6WMF-Legal: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#1394099 (10ZhouZ) Hi @d33tah, Yes, if you are purging the private end-user information such as IP addresses every month that should comply with the requirement above. Please also update your privacy polic... [20:31:02] https://www.wikidata.org/wiki/Q19610114 [20:31:11] sigh. so much rewriting to do :( [20:31:12] oh sorry, c&p fault [20:31:47] that was the wrong button, I'm sorry [20:33:28] 6Labs, 6WMF-Legal: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#1394101 (10d33tah) @ZhouZ: Thank you! I think I will launch it soon then, I would just like to update my database first. [20:33:38] rillke: if you really need NFS, you can poke Coren and he might be able to setup exports manually. [20:33:58] my reccomendation would be to not use NFS if at all possible - and I"ll work on updating the documentation over the next few days [20:34:55] YuviPanda|brb, do you have a rsync example - I mean where I see which the networking path is [20:35:11] 'which the networking path is'? I don't understand that [20:36:46] rsync [20:37:08] ah, hmm. good question [20:37:12] forget rsync for teh moment if you are new. scp -r :~/ [20:37:16] 6Labs, 10wikitech.wikimedia.org, 7Shinken: shinken thinks wikitech and wikitech-static are down. They aren't. - https://phabricator.wikimedia.org/T101517#1394114 (10Krenair) [20:37:20] or that ^ [20:38:01] 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure: Cant ssh to integration-slave-jessie-1001.eqiad.wmflabs despite reboot - https://phabricator.wikimedia.org/T102592#1394123 (10hashar) [20:38:03] 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure: Cant ssh to integration-slave-jessie-1001.integration.eqiad.wmflabs - https://phabricator.wikimedia.org/T103312#1394124 (10hashar) [20:39:27] 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure: Cant ssh to integration-slave-jessie-1001.integration.eqiad.wmflabs - https://phabricator.wikimedia.org/T103312#1387030 (10hashar) From /var/log/auth.log : ``` Jun 23 20:36:22 integration-slave-jessie-1001 sshd[6109]: Connection from 10.6... [20:40:21] chasemp, so I have to exchange SSH-public keys? [20:40:45] key forward to your first box and your ability to login to the other_box will get you through [20:41:27] YuviPanda|brb: can you unmute shinken? :P [20:41:48] ugh kmlexport again :< [20:42:31] 6Labs, 10Beta-Cluster, 7Shinken: Shinken is showing HTTP 404 warnings for deployment-mathoid/sca02 mathoid services - https://phabricator.wikimedia.org/T103595#1394154 (10Krenair) 3NEW [20:42:48] 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure: Cant ssh to integration-slave-jessie-1001.integration.eqiad.wmflabs - https://phabricator.wikimedia.org/T103312#1394162 (10hashar) So bastion-01.bastion.eqiad.wmflabs has the IP 10.68.17.232 but it is not enabled in the ferm rules! Thus... [20:43:06] rillke: yeah, key forward + scp is the easiest. [20:45:46] 10Quarry: Seeing all of created queries in one page - https://phabricator.wikimedia.org/T98688#1394177 (10matej_suchanek) [20:45:49] 10Quarry: List of users all published queries - https://phabricator.wikimedia.org/T88920#1394178 (10matej_suchanek) [20:45:53] 10Quarry: Provide the ability to view the queries submitted by a particular user - https://phabricator.wikimedia.org/T71174#1394179 (10matej_suchanek) [20:45:57] 10Quarry: Show all published queries in profile - https://phabricator.wikimedia.org/T77948#1394180 (10matej_suchanek) [20:46:35] 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure: Cant ssh to integration-slave-jessie-1001.integration.eqiad.wmflabs - https://phabricator.wikimedia.org/T103312#1394190 (10hashar) Stopped ferm service via salt and I can ssh again. Purged the ferm package, removed /etc/ferm and reran pu... [20:46:58] 10Quarry: Show all published queries in profile - https://phabricator.wikimedia.org/T77948#832247 (10matej_suchanek) [20:49:15] 6Labs, 10Tool-Labs: kmlexport perl script memory usage - https://phabricator.wikimedia.org/T99236#1394197 (10valhallasw) I'm actually wondering whether 4GB is enough. I just killed a few on `tools-webgrid-lighttpd-1401` because puppet couldn't run because of kmlexport. The total memory usage of the processes c... [20:55:22] So I wrote http://heipei.github.io/2015/02/26/SSH-Agent-Forwarding-considered-harmful/ -- is "key forward" what they say is "harmful"? [20:55:27] * I read [20:55:58] rillke: yes [20:56:45] 6Labs, 10Continuous-Integration-Infrastructure: Continuous integration should not depend on labs NFS - https://phabricator.wikimedia.org/T90610#1394220 (10hashar) [20:56:48] 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure: Cant ssh to integration-slave-jessie-1001.integration.eqiad.wmflabs - https://phabricator.wikimedia.org/T103312#1394218 (10hashar) 5Open>3Resolved a:3hashar [20:57:22] Coren: any word on https://phabricator.wikimedia.org/T87870 ? [20:58:21] perhaps I should make an encrypted zip file and just transfer that through public web [20:59:37] that sounds a lot more complicated than just using scp... [21:00:42] but scp requires authentication between the two instances - how would I achieve that without "key forward" [21:01:45] you can scp through your *local* computer? [21:01:54] that works if you have a fast enough network and the files arne't too big... [21:02:00] valhallasw: I guess HBA will help with this as well... [21:02:16] YuviPanda|brb: HBA doesn't allow host-to-host ssh [21:02:28] valhallasw: no, but host -> bast -> host [21:02:31] yeah [21:02:32] 3 way scp [21:02:47] I think you can do that without needing local storage on the bastion too [21:04:40] 6Labs, 10Tool-Labs: Add script_path to meta_p.wiki database - https://phabricator.wikimedia.org/T93483#1394271 (10gpaumier) [21:15:23] matanya: Faidon has a nice solution along a different tack; he's got a deb package to make with it and we'll deploy to test on labstore1001 [21:15:57] that is cool Coren thanks, any time frame on this ? [21:17:12] matanya: "This week" for the tests, but doing the final switch is cutting it close and probably too optimistic. It'd make a reasonable project for the hackathon though. [21:17:33] thank you [21:18:56] 6Labs, 10Tool-Labs, 3Labs-Sprint-103: Labs: Move tools-shadow off the same host as tool-master - https://phabricator.wikimedia.org/T103390#1394351 (10scfc) If the roles of master and shadow are basically identical and they auto-discover who's in charge, couldn't we name them `tools-master-01`, `tools-master-... [21:30:50] 6Labs, 10Tool-Labs, 3Labs-Sprint-103: Labs: Move tools-shadow off the same host as tool-master - https://phabricator.wikimedia.org/T103390#1394408 (10coren) There is a subtle difference, at least structurally, in that the gridengine configuration itself makes the distinction. That is, while the shadows can... [21:51:09] Coren, when do your work hours end? [22:17:23] 6Labs, 10Tool-Labs, 3Labs-Sprint-103: Labs: Move tools-shadow off the same host as tool-master - https://phabricator.wikimedia.org/T103390#1394566 (10scfc) For `tools-redis-01` & Co., @yuvipanda used a scheme where the determination of master and server is done by setting `$active_redis` accordingly. So if... [22:43:40] 6Labs, 6operations, 10wikitech.wikimedia.org: Set IPv6 PTR for wikitech-static - https://phabricator.wikimedia.org/T103621#1394654 (10Krenair) 3NEW [22:44:09] 6Labs, 6operations, 10wikitech.wikimedia.org, 7Ipv6: Set IPv6 PTR for wikitech-static - https://phabricator.wikimedia.org/T103621#1394662 (10Krenair)