[00:43:49] <TParis>	 YuviPanda or Coren : Ping
[00:44:00] <TParis>	 or anyone: does labs support node.js?
[00:44:07] <wikibugs>	 6Labs, 6operations, 10wikitech.wikimedia.org, 7HHVM: Move wikitech (silver) to HHVM - https://phabricator.wikimedia.org/T98813#1426080 (10Krenair)
[00:44:26] <Krenair>	 TParis, sure, you should be able to install node.js in your instance
[00:44:50] <Krenair>	 I know there's a few things using node.js in deployment-prep
[00:45:01] <TParis>	 Okay, thanks, I appreciate it
[00:52:18] <YuviPanda>	 TParis: Krenair I'm technically sleeping but toollabs also has nodejs support 
[00:56:37] <legoktm>	 I'm pretty sure you are not "technically sleeping"
[00:56:56] <legoktm>	 ;)
[01:20:59] <TParis>	 :)
[01:25:47] <Cyberpower678>	 TParis, nice to see you on IRC again.  Can I nag you to join xTools? :p
[01:27:35] * Cyberpower678 goes to bed.  He woke up early this morning, and now is tired.
[03:08:55] <grrrit-wm>	 (03PS1) 10Krinkle: Fix "talk" and "contribs" links in results for prefix search [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/222732 
[03:10:58] <ori>	 !log Promoted Krinkle and Krenair to admin, cloudadmin on wikitech, because duh.
[03:10:58] <labs-morebots>	 Promoted is not a valid project.
[03:11:13] * ori squints.
[03:11:18] <ori>	 Right, that's intuitive and consistent.
[03:11:31] <Reedy>	 do it in -operations?
[03:11:41] <Reedy>	 What've I done wrong? :P
[03:11:46] <Krenair>	 Cloudadmin? Ummm. Okay...
[03:11:50] <grrrit-wm>	 (03PS1) 10Ricordisamoa: Load jQuery and Bootstrap from cdnjs mirror [labs/tools/lists] - 10https://gerrit.wikimedia.org/r/222733 
[03:12:00] <Reedy>	 You can now make and destroy clouds
[03:12:11] <ori>	 Krenair: is my trust misplaced?
[03:12:12] <Krinkle>	 I prefer a bit less heat down here in London
[03:12:26] <Krinkle>	 Make those clouds come Krenair
[03:13:46] <wikibugs>	 10Tool-Labs-tools-Other, 7Epic: Convert all Labs tools to use cdnjs for static libraries - https://phabricator.wikimedia.org/T103934#1426169 (10Ricordisamoa)
[03:15:45] <wikibugs>	 10Tool-Labs-tools-Other, 7Epic: Convert all Labs tools to use cdnjs for static libraries - https://phabricator.wikimedia.org/T103934#1403141 (10Ricordisamoa)
[03:18:27] <grrrit-wm>	 (03CR) 10Krinkle: [C: 032] Fix "talk" and "contribs" links in results for prefix search [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/222732 (owner: 10Krinkle)
[03:28:19] <wikibugs>	 10Tool-Labs-tools-Other, 7Epic: Convert all Labs tools to use cdnjs for static libraries - https://phabricator.wikimedia.org/T103934#1426183 (10Krinkle)
[03:52:28] <grrrit-wm>	 (03CR) 10Krinkle: [V: 032] Fix "talk" and "contribs" links in results for prefix search [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/222732 (owner: 10Krinkle)
[04:13:51] <wikibugs>	 10Tool-Labs-tools-Other, 7Epic: Convert all Labs tools to use cdnjs for static libraries - https://phabricator.wikimedia.org/T103934#1426202 (10Ricordisamoa) //magnustools// in going to be //magna quaestio//...
[06:55:53] <SigmaWP>	 Cyberpower678: Hey, are you here?
[07:16:20] <wikibugs>	 6Labs, 10Tool-Labs: Nagf showing "Project not found" - https://phabricator.wikimedia.org/T104757#1426248 (10zhuyifei1999) 3NEW
[07:36:11] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: Cannot login to wikitech: "Login error" - https://phabricator.wikimedia.org/T104759#1426262 (10zhuyifei1999) 3NEW
[07:46:28] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: Cannot login to wikitech: "Login error" - https://phabricator.wikimedia.org/T104759#1426270 (10Steinsplitter) same here. (=confirmed)
[07:50:35] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: Cannot login to wikitech: "Login error" - https://phabricator.wikimedia.org/T104759#1426271 (10zhuyifei1999)
[07:50:36] <wikibugs>	 6Labs, 10wikitech.wikimedia.org, 5Patch-For-Review: Cannot log into wikitech; works after deleting the cookies - https://phabricator.wikimedia.org/T103939#1426272 (10zhuyifei1999)
[07:55:41] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: Cannot log into wikitech; works after deleting the cookies - https://phabricator.wikimedia.org/T103939#1426281 (10zhuyifei1999)
[08:41:38] <wikibugs>	 10Quarry: SELECT command denied to user for table 'logging_userindex' - https://phabricator.wikimedia.org/T99414#1426333 (10Nemo_bis) Thanks. I got confused by the fact that the two names are equivalent when using `sql`.
[09:49:38] <wikibugs>	 10Tool-Labs-tools-Other, 7Epic: Convert all Labs tools to use cdnjs for static libraries - https://phabricator.wikimedia.org/T103934#1426372 (10Ricordisamoa)
[10:11:51] <Steinsplitter>	 hi, the filearchive table contains sha1 of deleted fies. can the tabel be fully used or it is non public data?
[10:54:53] <valhallasw`cloud>	 YuviPanda: can I just assume dynamicproxy to be tool labs only?
[10:55:06] <valhallasw`cloud>	 making the error page generic enough for other projects is not entirely trivial
[11:09:29] <YuviPanda>	 valhallasw`cloud: urlproxy is toollabs only 
[11:09:34] <valhallasw`cloud>	 ok
[11:09:40] <YuviPanda>	 Domainproxy is general proxy 
[11:09:45] <valhallasw`cloud>	 hm okay
[11:09:53] <YuviPanda>	 Redundant urlproxy is marathon 
[11:09:55] <valhallasw`cloud>	 but there's not specific urlproxy manifest
[11:10:27] <valhallasw`cloud>	 blah, I'm just going to build something, it doesn't matter too much if it isn't clean
[11:11:24] <YuviPanda>	 valhallasw`cloud: yeah. There is toollabs::proxy manifest tho
[11:11:36] <Steinsplitter>	 any iddea about my question above?
[11:11:43] <Steinsplitter>	 can the filarchive table publicy used?
[11:11:47] <valhallasw`cloud>	 it's basically the cleaner version of https://gerrit.wikimedia.org/r/#/c/219191/4/modules/dynamicproxy/templates/urlproxy.conf, but it's already taking too much time :{
[11:12:26] * YuviPanda is still in bed, brb online in a bit
[11:21:12] <valhallasw`cloud>	 Wikitech uses cookies to log in users. You have cookies disabled. Please enable them and try again. *look of disapproval8
[11:21:23] * valhallasw`cloud hits F5 a few times
[11:23:38] <Nemo_bis>	 valhallasw`cloud: same issue as the #wikimedia-tech reporter?
[11:24:46] <valhallasw`cloud>	 I'm not in -tech, but possibly?
[11:42:59] <wikibugs>	 6Labs, 10Tool-Labs, 10wikitech.wikimedia.org: I can't log in to wikitech - https://phabricator.wikimedia.org/T104766#1426421 (10MaxBioHazard) 3NEW
[12:07:13] <wikibugs>	 6Labs, 10Tool-Labs, 10wikitech.wikimedia.org: I can't log in to wikitech - https://phabricator.wikimedia.org/T104766#1426432 (10MaxBioHazard)
[12:31:15] <Cyberpower678>	 Hi SigmaWP.  You caught me while I was asleep.
[12:31:30] <Cyberpower678>	 SigmaWP, if you're not here now, ping me later.
[12:55:45] <YuviPanda>	 valhallasw`cloud: wikitech still giving you trouble?
[12:55:51] <valhallasw`cloud>	 YuviPanda: yep
[12:55:55] <valhallasw`cloud>	 can't login
[12:56:09] <YuviPanda>	 try now?
[12:56:34] <valhallasw`cloud>	 YuviPanda: works
[12:57:58] <valhallasw`cloud>	 !log toolsbeta restarting toolsbeta-webproxy, no response on port 22
[12:58:00] <labs-morebots>	 Logged the message, Master
[13:25:47] <valhallasw`cloud>	 YuviPanda: toolsbeta-webproxy ssh is not coming up at all, it seems, or is blocked by a firewall? any idea what oculd be going on?
[13:26:08] <YuviPanda>	 valhallasw`cloud: not sure.
[13:26:18] <YuviPanda>	 valhallasw`cloud: isn't responding to my root key either.
[13:26:28] <valhallasw`cloud>	 yeah, sshd isn't even listening it seems
[13:26:56] <YuviPanda>	 valhallasw`cloud: anything useful in wikitech console?
[13:27:16] <valhallasw`cloud>	  * Starting OpenSSH server[74G[ OK ]
[13:27:43] <valhallasw`cloud>	 nothing special in there :/
[13:27:51] <valhallasw`cloud>	 bah, bettre re-create it then
[13:28:02] <YuviPanda>	 valhallasw`cloud: so I wonder if I disabled puppet in there at some point in the past (during the outage maybe?) and that's fucked everything up?
[13:28:03] <YuviPanda>	 :'(
[13:28:13] <valhallasw`cloud>	 yeah, could be. oh well.
[13:28:21] <valhallasw`cloud>	 nothing important on there anyway
[13:28:29] <multichill>	 YuviPanda: Already decided on you post Wikimania plans?
[13:29:44] <YuviPanda>	 multichill: yeah, I'm goingback to SF :(
[13:29:57] <YuviPanda>	 multichill: I'm going to CAMP in august in Berlin, so would like a little more time in SF
[13:30:00] <YuviPanda>	 you guys have fun without me
[13:30:14] <YuviPanda>	 anyway, food time! brb
[13:41:09] <multichill>	 YuviPanda: Two things for when you come back. Any idea why http://wdq.wmflabs.org/stats is lagging so much and https://phabricator.wikimedia.org/T104762
[13:43:04] <Steinsplitter>	 there are no replicated tables wiki contain non public data, like filearchive.  i mean, they can be used for tools?
[13:58:04] <valhallasw`cloud>	 Steinsplitter: as far as I know, most data is public, but in general, try not to make things public that couldn't also be recovered from the web interface
[13:58:32] <valhallasw`cloud>	 Steinsplitter: the sha1 of deleted files is probably fine
[14:08:12] <wikibugs>	 6Labs, 10Tool-Labs: Make 'new webproxy' checklist - https://phabricator.wikimedia.org/T104768#1426457 (10valhallasw) 3NEW
[14:11:13] <valhallasw`cloud>	 YuviPanda: what does use_dnsmasq do?
[14:17:29] <YuviPanda>	 valhallasw`cloud: it used to allow people to switch to designate, the new DNS system
[14:17:30] <YuviPanda>	 is a noop now
[14:20:12] <wikibugs>	 6Labs, 10Tool-Labs, 10MediaWiki-User-login-and-signup, 10wikitech.wikimedia.org: I can't log in to wikitech - https://phabricator.wikimedia.org/T104766#1426467 (10MaxBioHazard)
[14:20:43] <wikibugs>	 6Labs, 10Tool-Labs, 10MediaWiki-User-login-and-signup, 10wikitech.wikimedia.org: I can't log in to wikitech - https://phabricator.wikimedia.org/T104766#1426469 (10yuvipanda) Try again now? I restarted nutcracker on silver, and the problem seems to have gone away.
[14:22:29] <sitic>	 YuviPanda: CCC Camp? I'm probably going to be there too
[14:22:33] <YuviPanda>	 multichill: I do not have an answer to the first question however.
[14:22:38] <YuviPanda>	 sitic: woah, cool! 
[14:22:42] <sitic>	 :-)
[14:22:58] <YuviPanda>	 sitic: I've an Schengen Visa now so can travel around europe freely, and hence should! :)
[14:23:09] <sitic>	 yep :-)
[14:27:40] <wikibugs>	 6Labs, 10Tool-Labs, 10MediaWiki-User-login-and-signup, 10wikitech.wikimedia.org: I can't log in to wikitech - https://phabricator.wikimedia.org/T104766#1426471 (10MaxBioHazard) Yes, I log in.
[14:29:02] <wikibugs>	 6Labs, 10Tool-Labs, 10MediaWiki-User-login-and-signup, 10wikitech.wikimedia.org: I can't log in to wikitech - https://phabricator.wikimedia.org/T104766#1426472 (10MaxBioHazard)
[14:29:04] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: Cannot log into wikitech; works after deleting the cookies - https://phabricator.wikimedia.org/T103939#1426473 (10MaxBioHazard)
[14:29:12] <wikibugs>	 6Labs, 10Tool-Labs, 10MediaWiki-User-login-and-signup, 10wikitech.wikimedia.org: I can't log in to wikitech - https://phabricator.wikimedia.org/T104766#1426475 (10zhuyifei1999)
[14:29:13] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: Cannot log into wikitech; works after deleting the cookies - https://phabricator.wikimedia.org/T103939#1426476 (10zhuyifei1999)
[14:32:07] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: Cannot log into wikitech; works after deleting the cookies - https://phabricator.wikimedia.org/T103939#1426478 (10MaxBioHazard) But cookies deleting wasn't work for me. Yuvipanda actions on my task works.
[14:35:53] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: Cannot log into wikitech; works after deleting the cookies - https://phabricator.wikimedia.org/T103939#1426481 (10zhuyifei1999) 5Open>3Resolved a:3zhuyifei1999 Seems fixed by yuvipanda in [[https://phabricator.wikimedia.org/T104766#1426469|T104766#1426469]]
[14:36:40] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: Cannot log into wikitech; works after deleting the cookies - https://phabricator.wikimedia.org/T103939#1426484 (10zhuyifei1999) a:5zhuyifei1999>3None
[14:38:42] <wikibugs>	 6Labs, 10Tool-Labs: Nagf showing "Project not found" - https://phabricator.wikimedia.org/T104757#1426485 (10zhuyifei1999) 5Open>3Resolved a:3zhuyifei1999 Seems fixed (somehow)
[15:03:07] <valhallasw`cloud>	 YuviPanda: uuuggghhh. webproxy doesn't deploy correctly on tools-webproxy-01 for some reason: nginx config is some default value instead of something from puppet :/
[15:03:22] <valhallasw`cloud>	 I don't even get how urlproxy.conf is supposed to be deployed to begin with
[15:04:22] <YuviPanda>	 valhallasw`cloud: it's kind of terrible, I think it's the first puppet code I ever wrote
[15:04:41] <YuviPanda>	 >         luahandler           => 'urlproxy',
[15:04:44] <YuviPanda>	 in toollabs::proxy
[15:04:45] <valhallasw`cloud>	 I can live with that =p but I don't get how it can work on tools at all
[15:04:50] <YuviPanda>	 that's what determines that
[15:04:53] <YuviPanda>	 also tools-webproxy-01?
[15:05:00] <valhallasw`cloud>	 toolsbeta-webproxy-01
[15:05:00] <valhallasw`cloud>	 sorry
[15:05:02] <valhallasw`cloud>	 https://github.com/wikimedia/operations-puppet/blob/acacf97e2df962fef83487a461f3559fa07e4d6f/modules/dynamicproxy/manifests/init.pp#L57
[15:05:14] <valhallasw`cloud>	 ^ that doesn't actually load urlproxy.conf (= the nginx conf that actually calls lua)
[15:05:47] <YuviPanda>	 valhallasw`cloud: indeed, https://github.com/wikimedia/operations-puppet/blob/acacf97e2df962fef83487a461f3559fa07e4d6f/modules/dynamicproxy/manifests/init.pp#L72 loads urlproxy.conf
[15:06:01] <valhallasw`cloud>	 ahhhhh
[15:06:04] <valhallasw`cloud>	 that's why I couldn't find it
[15:06:06] <valhallasw`cloud>	 yes, it's in /etc/nginx/sites-enabled/proxy
[15:06:53] <valhallasw`cloud>	 thanks, that helps
[15:07:11] <YuviPanda>	 yw, sorry it wasn't clearer
[15:42:06] <wikibugs>	 10Quarry: Quarry's indentation function is not completely functional - https://phabricator.wikimedia.org/T101424#1426532 (10yuvipanda) So this is using the sql mode of the codemirror code editor, and this seems to be that one's behavior. I guess possible options are:  # Turn off autoindent for everyone # Make it...
[15:43:11] <wikibugs>	 10Quarry: Suggested Feature:  Put title of the query into the HTML title for page - https://phabricator.wikimedia.org/T99449#1426536 (10yuvipanda) 5Open>3Resolved a:3yuvipanda This was done a while ago :)
[15:45:52] <YuviPanda>	 !log quarry deploying to latest master and hoping!
[15:45:54] <labs-morebots>	 Logged the message, Master
[15:47:48] <valhallasw`cloud>	 grrr. I changed hiera settings, but puppet agent -tv is ignoring them somehow?!
[15:49:09] <valhallasw`cloud>	 YuviPanda: is there some sort of subtlety where hiera settings are not applied to class parameters in some cases? :/
[15:49:25] <valhallasw`cloud>	 oh, no, it's just slow
[15:49:39] <YuviPanda>	 yeah, there's a file cache...
[15:52:47] <valhallasw`cloud>	 YuviPanda: hrm. any idea how to set up /etc/ssl/localcerts/tools-beta.wmflabs.org.chained.crt ?
[15:52:55] <valhallasw`cloud>	 I don't even care about ssl...
[15:52:59] <YuviPanda>	 valhallasw`cloud: ah, nope. turn off ssl instead.
[15:53:08] <YuviPanda>	 valhallasw`cloud: yeah you can set ssl_certificate_name or whatever param to false
[15:53:16] <valhallasw`cloud>	 ah ok
[15:58:08] <valhallasw`cloud>	 YuviPanda: http://tools-beta.wmflabs.org/ \o/
[15:58:19] <YuviPanda>	 w00t!
[15:58:20] <YuviPanda>	 nice
[15:58:21] <valhallasw`cloud>	 something screwed up with the layout though
[15:58:31] <YuviPanda>	 CSS sucks doesn't it
[15:58:44] <YuviPanda>	 also once I get per-host hiera stuff going we can disable NFS entirely on the proxy hosts
[15:58:48] <YuviPanda>	 and the redis hosts, probably
[16:00:41] <wikibugs>	 10Quarry, 7Easy, 5Patch-For-Review: String "Your query is currently executing" should be "This query..." - https://phabricator.wikimedia.org/T103275#1426547 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Fixed! Thank you for reporting it!
[16:00:53] <valhallasw`cloud>	 YuviPanda: it's just a bug in the normal page, I think :-p
[16:01:14] <valhallasw`cloud>	 except it's not working correctly as it's not a 503
[16:01:15] <valhallasw`cloud>	 ugh
[16:04:35] <valhallasw`cloud>	 ok, works now
[16:10:13] <valhallasw`cloud>	 YuviPanda: http://tools-beta.wmflabs.org/ <3 
[16:10:26] <YuviPanda>	 nice!
[16:10:31] <YuviPanda>	 we can put the phabricator link in hiera :P
[16:10:31] <valhallasw`cloud>	 (ignore the nil, that's just me and yaml)
[16:10:39] <valhallasw`cloud>	 yes! https://wikitech.wikimedia.org/wiki/Hiera:Toolsbeta
[16:10:50] <valhallasw`cloud>	 and the message and all the things =p
[16:11:08] <YuviPanda>	 yeah
[16:11:12] <valhallasw`cloud>	 I'm not sure what will happen if a key is unset, though, but we'll see soon(TM)
[16:22:13] <wikibugs>	 10Quarry: Raise query limits - https://phabricator.wikimedia.org/T74342#1426565 (10yuvipanda) Note that current limit is 20mins instead of 10
[16:24:01] <wikibugs>	 10Quarry: Allow published query titles to be searched or filtered by tag - https://phabricator.wikimedia.org/T90509#1426572 (10yuvipanda)
[17:09:02] <wikibugs>	 10Quarry, 5Patch-For-Review: JSON output should be one row per JSON blob - https://phabricator.wikimedia.org/T98492#1426586 (10yuvipanda) 5Open>3Resolved a:3yuvipanda This has been implemented as the 'json lines' output format.
[17:10:29] <wikibugs>	 10Quarry: Make query URLs have a sluggified version of the title in them - https://phabricator.wikimedia.org/T75885#1426595 (10yuvipanda) Yeah, we totally should - don't think there's any other way to do this.
[17:10:36] <wikibugs>	 10Quarry: Make query URLs have a sluggified version of the title in them - https://phabricator.wikimedia.org/T75885#1426596 (10yuvipanda) p:5Triage>3Normal
[17:11:45] <YuviPanda>	 valhallasw`cloud: been doing a bunch of quarry stuff, there's a working vagrant setup for it now :)
[17:11:53] <YuviPanda>	 just vagrant, provisioned with a shell script, no puppet stuff
[17:13:29] <wikibugs>	 10Quarry, 5Patch-For-Review: Remember filters: "All queries", "Published queries", etc. chosen by user in recent queries page - https://phabricator.wikimedia.org/T76084#1426599 (10yuvipanda)
[17:14:12] <wikibugs>	 10Quarry: Build a vagrant box for Quarry dev - https://phabricator.wikimedia.org/T71193#1426605 (10yuvipanda) 5Open>3Resolved There's a Vagrantfile in the repo now :)
[17:15:24] <YuviPanda>	 valhallasw`cloud: next 'big feature' would be celerybeat, which would then enable reports
[17:21:52] <valhallasw`cloud>	 YuviPanda: ah, nice
[17:22:06] <YuviPanda>	 am going through fixing bugs and stuff now
[17:22:15] <valhallasw`cloud>	 YuviPanda: btw, according to something I found, utf-8 with bom should also work with excel. still need to confirm, but that would be much easier
[17:22:25] <YuviPanda>	 valhallasw`cloud: yeah, true.
[17:22:31] <YuviPanda>	 I guess you can force python to output a BOM
[17:22:54] <valhallasw`cloud>	 YuviPanda: yeah, you can just send utf-8-with-bom-encoded binary data to the lower layers
[17:23:02] <valhallasw`cloud>	 "Error: Could not retrieve catalog from remote server: wrong header line format"
[17:24:01] <valhallasw`cloud>	 gah.
[17:25:16] <wikibugs>	 10Quarry, 5Patch-For-Review: Quarry does not respect ORDER BY sort order in result set - https://phabricator.wikimedia.org/T87829#1426614 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Just fixed it :) Thanks for reporting the issue, and apologies for such late resolution.
[17:26:44] <wikibugs>	 10Quarry: Unicode in query results in strange behavior - https://phabricator.wikimedia.org/T71224#742694 (10yuvipanda) Is this still happening?
[17:26:54] <wikibugs>	 10Quarry, 5Patch-For-Review: Excel does not recognize Quarry CSV output as UTF-8 - https://phabricator.wikimedia.org/T76126#1426626 (10valhallasw) UTF-8 with BOM also works fine, and should be easier to implement.
[17:38:34] <wikibugs>	 6Labs, 10Tool-Labs: Document process to run webserver on seperate instance through tools-webproxy - https://phabricator.wikimedia.org/T103056#1426635 (10valhallasw)
[17:39:03] <wikibugs>	 6Labs, 10Wikibugs, 10grrrit-wm: document emergency deployment of grrrit-wm and wikibugs - https://phabricator.wikimedia.org/T102984#1426636 (10valhallasw)
[17:39:29] <wikibugs>	 6Labs, 10Tool-Labs, 5Patch-For-Review: Create process for 'tool labs is down' notifications on tools.wmflabs.org/* - https://phabricator.wikimedia.org/T102971#1426637 (10valhallasw)
[17:46:03] <wikibugs>	 6Labs, 10Tool-Labs: Make 'new webproxy' checklist - https://phabricator.wikimedia.org/T104768#1426656 (10valhallasw)
[17:46:05] <wikibugs>	 6Labs, 10Tool-Labs: [tracking] Tool labs admin guides - https://phabricator.wikimedia.org/T104734#1425793 (10valhallasw)
[17:47:31] <wikibugs>	 6Labs, 10Tool-Labs:  'new webproxy' checklist - https://phabricator.wikimedia.org/T104768#1426658 (10valhallasw)
[18:04:50] <wikibugs>	 10Quarry: Number of queries shown in profile is wrong - https://phabricator.wikimedia.org/T86512#1426673 (10yuvipanda) This seems to be a somewhat strange sqlalchemy issue...
[18:30:40] <Celelibi>	 Hi.
[18:30:59] <Celelibi>	 Is the http://mathoid.testme.wmflabs.org/ unavailability related to the NFS issue?
[18:47:59] <wikibugs>	 6Labs, 10VisualEditor, 10VisualEditor-MediaWiki, 10wikitech.wikimedia.org, 7Regression: Pages can't be saved with VisualEditor on wikitech (Uncaught ReferenceError: attrId is not defined) - https://phabricator.wikimedia.org/T104360#1426705 (10Krenair) a:5Krinkle>3None
[19:51:12] <Nemo_bis>	 Do tools like http://tools.wmflabs.org/mix-n-match/ have a limit on the number of processes they can spawn, or things like that?
[19:51:30] <Nemo_bis>	 The AJAXy interface sometimes waits a really long time on backend requests to complete
[20:25:03] <wikibugs>	 10Quarry: WMF employees use an internal Quarry instance to share data & sample queries - https://phabricator.wikimedia.org/T75142#1426761 (10Krenair) Why do we need an internal WMF-only instance of Quarry?
[20:32:49] <wikibugs>	 10Quarry: WMF employees use an internal Quarry instance to share data & sample queries - https://phabricator.wikimedia.org/T75142#1426765 (10yuvipanda) Labsdb is redacted but internal analytics store isn't.
[20:36:36] <wikibugs>	 10Quarry: WMF employees use an internal Quarry instance to share data & sample queries - https://phabricator.wikimedia.org/T75142#1426769 (10Krenair) So you're going to provide all WMF employees (and presumably contractors etc. as well?) with access to all the research data via some sort of corp Quarry instance?
[20:39:10] <wikibugs>	 10Quarry: WMF employees use an internal Quarry instance to share data & sample queries - https://phabricator.wikimedia.org/T75142#1426770 (10yuvipanda) Such a thing has been proposed, primarily for eventlogging information. It would be for anyone with enough clearance to access it. Currently nobody is spending t...
[20:44:03] <wikibugs>	 10Quarry: WMF employees use an internal Quarry instance to share data & sample queries - https://phabricator.wikimedia.org/T75142#1426778 (10Krenair) So actually it wouldn't be just for WMF employees, it would actually be something in production that some of the groups defined in puppet get access to (via SSH pr...
[20:47:53] <wikibugs>	 10Quarry: WMF employees use an internal Quarry instance to share data & sample queries - https://phabricator.wikimedia.org/T75142#1426780 (10yuvipanda) There already exists such a group (researchers I think?) so this might piggy back off it. Or not - we don't know. Will be determined if / when someone starts wor...
[20:50:21] <wikibugs>	 10Quarry: WMF employees use an internal Quarry instance to share data & sample queries - https://phabricator.wikimedia.org/T75142#1426783 (10Krenair) That was the group I had in mind, I know for a fact it's not 100% WMF employees. Will adjust the title.
[20:52:18] <wikibugs>	 10Quarry: Build an internal Quarry instance to share data & sample queries between researchers (and other analytics users?) - https://phabricator.wikimedia.org/T75142#1426785 (10Krenair)
[20:55:24] <Negative24>	 Do we have anything on labs that drops a binary named core in a working dir?
[21:03:37] <gifti>	 it's called linux ;)
[21:04:38] <YuviPanda>	 hey gifti 
[21:04:52] <YuviPanda>	 gifti: still around?
[21:04:58] <gifti>	 i am
[21:05:24] <YuviPanda>	 gifti: the dwl project is yours, right? https://phabricator.wikimedia.org/T103864
[21:05:29] <gifti>	 it is
[21:06:01] <gifti>	 nfs can go, no problem
[21:07:09] <YuviPanda>	 gifti: w00t, thanks :) can you say so on the ticket?
[21:07:14] <gifti>	 ok
[21:07:33] <wikibugs>	 6Labs: Disable NFS for dwl project - https://phabricator.wikimedia.org/T103864#1426792 (10Giftpflanze) Do it.
[21:07:41] <YuviPanda>	 gifti: thanks
[21:07:44] <gifti>	 np
[21:07:56] <valhallasw`cloud>	 Negative24: core is the coredump of a binary that crashed or was killed
[21:09:41] <gifti>	 YuviPanda: can i get a special permit for long db queries?
[21:09:55] <YuviPanda>	 gifti: I don't think there's such a thing...
[21:10:04] <gifti>	 welp
[21:10:20] <gifti>	 i don't want to use a dump
[21:10:43] <YuviPanda>	 talk to halfak when he's around, and explain your problem to him. he'll probably find a clever solution :0
[21:11:06] <gifti>	 ok, thx
[21:11:08] <Negative24>	 valhallasw`cloud: yea found that out as soon as I asked
[21:20:50] <wikibugs>	 6Labs, 10Tool-Labs: new labs host sends out "mpt raid status change" emails - https://phabricator.wikimedia.org/T104779#1426804 (10valhallasw) 3NEW
[21:23:47] <wikibugs>	 6Labs, 10Tool-Labs: Fix 'unknown's in shinken - https://phabricator.wikimedia.org/T99072#1426819 (10valhallasw)
[21:29:20] <wikibugs>	 6Labs, 10Tool-Labs: can't login to tools-shadow-01 - https://phabricator.wikimedia.org/T104781#1426822 (10valhallasw) 3NEW
[21:31:38] <valhallasw`cloud>	 YuviPanda: ugh, these shinken unknowns annoy me
[21:31:43] <valhallasw`cloud>	 YuviPanda: stupid diamond mess
[21:31:51] <YuviPanda>	 stupid graphite mess, more like
[21:33:53] <valhallasw`cloud>	 nah, it's diamond
[21:33:58] <valhallasw`cloud>	 sudo service diamond restart fixes them
[21:34:12] <YuviPanda>	 oh, i see.
[21:34:30] <valhallasw`cloud>	 http://graphite.wmflabs.org/render/?width=586&height=308&_salt=1436045636.974&target=tools.tools-exec-gift.puppetagent.failed_events&target=tools.tools-exec-1409.puppetagent.failed_events&target=tools.tools-exec-1409.puppetagent.time_since_last_run&from=-24days < no data for most of that period
[21:35:00] <valhallasw`cloud>	 diamond seems to just randomly hang
[21:35:02] <valhallasw`cloud>	 nothing in the log
[21:41:13] <valhallasw`cloud>	 YuviPanda: yeah, all of them are gone now, except shadow-01, where I can't login :<
[21:41:25] <YuviPanda>	 oh, wtf is shadow-01
[21:41:40] <YuviPanda>	 that's the thing coren was playing with
[21:41:53] <YuviPanda>	 valhallasw`cloud: also let's get your root keys into instances!
[21:42:01] <YuviPanda>	 valhallasw`cloud: I can't get in either
[21:42:04] <valhallasw`cloud>	 :D
[21:42:12] <valhallasw`cloud>	 yeah, let me create a new key
[21:42:24] <valhallasw`cloud>	 ExtraSpecialRootKeyForLabs.id_rsa
[21:42:31] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] "Nope, current patch is still key => value, just discards key :)" [labs/private] - 10https://gerrit.wikimedia.org/r/222468 (owner: 10Yuvipanda)
[21:42:37] <grrrit-wm>	 (03CR) 10Yuvipanda: [V: 032] Allow addition of more root keys via hiera [labs/private] - 10https://gerrit.wikimedia.org/r/222468 (owner: 10Yuvipanda)
[21:42:44] <YuviPanda>	 valhallasw`cloud: :) ok.
[21:43:24] <valhallasw`cloud>	 what's the suggestion on RSA vs DSA these days? key length?
[21:43:37] <YuviPanda>	 no idea.
[21:43:41] * YuviPanda is out of the loop
[21:43:45] <YuviPanda>	 mine are 2048 RSAs I think
[21:44:31] <valhallasw`cloud>	 google sez 'never use DSA'
[21:44:48] <valhallasw`cloud>	 and '2048 is good enough'
[21:44:53] <valhallasw`cloud>	 so I'll go with that :-p
[21:44:54] <YuviPanda>	 wtf
[21:44:56] <YuviPanda>	 puppet failures
[21:45:16] <YuviPanda>	 > Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find template 'ssh/root-authorized-keys.erb' at /etc/puppet/private/modules/passwords/manifests/init.pp:22 on node tools-bastion-01.tools.eqiad.wmflabs
[21:45:18] <YuviPanda>	 hmm
[21:46:55] <grrrit-wm>	 (03PS1) 10Yuvipanda: Revert "Allow addition of more root keys via hiera" [labs/private] - 10https://gerrit.wikimedia.org/r/222857 
[21:47:04] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] Revert "Allow addition of more root keys via hiera" [labs/private] - 10https://gerrit.wikimedia.org/r/222857 (owner: 10Yuvipanda)
[21:47:11] <YuviPanda>	 valhallasw`cloud: ^ reverted, I'll come back to this later, I guess :(
[21:47:47] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1409 is CRITICAL 40.00% of data above the critical threshold [0.0]
[21:47:56] <valhallasw`cloud>	 YuviPanda: yeah, you renamed the file
[21:47:59] <shinken-wm>	 PROBLEM - Puppet failure on tools-master is CRITICAL 30.00% of data above the critical threshold [0.0]
[21:48:01] <valhallasw`cloud>	 oh wait
[21:48:03] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1212 is CRITICAL 30.00% of data above the critical threshold [0.0]
[21:48:09] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1407 is CRITICAL 44.44% of data above the critical threshold [0.0]
[21:48:10] <valhallasw`cloud>	 that's the revert
[21:48:32] <YuviPanda>	 valhallasw`cloud: it's named correctly...
[21:48:37] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1207 is CRITICAL 30.00% of data above the critical threshold [0.0]
[21:48:39] <shinken-wm>	 PROBLEM - Puppet failure on tools-services-01 is CRITICAL 40.00% of data above the critical threshold [0.0]
[21:48:45] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-generic-1401 is CRITICAL 40.00% of data above the critical threshold [0.0]
[21:49:13] <shinken-wm>	 PROBLEM - Puppet failure on tools-bastion-01 is CRITICAL 44.44% of data above the critical threshold [0.0]
[21:49:29] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1206 is CRITICAL 50.00% of data above the critical threshold [0.0]
[21:49:35] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1410 is CRITICAL 50.00% of data above the critical threshold [0.0]
[21:49:35] <shinken-wm>	 PROBLEM - Puppet failure on tools-precise-dev is CRITICAL 60.00% of data above the critical threshold [0.0]
[21:49:37] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1406 is CRITICAL 20.00% of data above the critical threshold [0.0]
[21:49:37] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1204 is CRITICAL 60.00% of data above the critical threshold [0.0]
[21:50:01] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL 40.00% of data above the critical threshold [0.0]
[21:50:01] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1406 is CRITICAL 40.00% of data above the critical threshold [0.0]
[21:50:07] <valhallasw`cloud>	 wikitech sessio ndata AAAAAARGH
[21:50:11] <shinken-wm>	 PROBLEM - Puppet failure on tools-webproxy-01 is CRITICAL 55.56% of data above the critical threshold [0.0]
[21:50:19] <valhallasw`cloud>	 I hate wikitech
[21:50:43] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1209 is CRITICAL 40.00% of data above the critical threshold [0.0]
[21:50:52] <valhallasw`cloud>	 YuviPanda: https://wikitech.wikimedia.org/wiki/Hiera:Tools
[21:51:01] <valhallasw`cloud>	 that should be the right format, I think?
[21:51:01] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1403 is CRITICAL 30.00% of data above the critical threshold [0.0]
[21:51:03] <Krenair>	 are you sure that's the right file name? :/
[21:51:23] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-generic-1403 is CRITICAL 55.56% of data above the critical threshold [0.0]
[21:51:41] <YuviPanda>	 valhallasw`cloud: edited
[21:51:43] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1214 is CRITICAL 30.00% of data above the critical threshold [0.0]
[21:51:44] <Krenair>	 hmm
[21:51:47] <YuviPanda>	 Krenair: i think so...
[21:51:56] <YuviPanda>	 valhallasw`cloud: I'm calling join on values() which discards keys, so it can be a hask
[21:52:21] <valhallasw`cloud>	 YuviPanda: the extra line was there for a reason :-p
[21:52:35] <valhallasw`cloud>	 to add a comment to /root/authorized_keys
[21:52:36] <YuviPanda>	 I changed it from an array to a hash
[21:52:37] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1211 is CRITICAL 60.00% of data above the critical threshold [0.0]
[21:52:43] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1401 is CRITICAL 40.00% of data above the critical threshold [0.0]
[21:52:47] <YuviPanda>	 I don't know how values() will react to arrays :)
[21:53:03] <valhallasw`cloud>	 eh. ok
[21:53:05] <shinken-wm>	 PROBLEM - Puppet failure on tools-checker-01 is CRITICAL 55.56% of data above the critical threshold [0.0]
[22:07:56] <Krenair>	 valhallasw`cloud, would that template('ssh/root-authorized-keys.erb') cause it to go looking for templates/ssh/root-authorized-keys.erb or a modules/ssh/templates/root-authorized-keys.erb file?
[22:08:14] <Krenair>	 (I'm not familiar with puppet at all)
[22:09:37] <andrewbogott>	 YuviPanda: was that alert storm something you’re working on?
[22:10:36] <YuviPanda>	 andrewbogott: yes
[22:10:44] <YuviPanda>	 Was reverted sorry to have woken you
[22:10:53] <andrewbogott>	 np —  I’ll go back to the hammock :)
[22:10:57] <YuviPanda>	 Krenair: it should do both theoretically 
[22:11:01] <YuviPanda>	 andrewbogott: please do :)
[22:11:56] <valhallasw`cloud>	 Krenair: it's puppet, so it's a bit like asking where $needle and $haystack are in str_replace
[22:12:17] <Krenair>	 heh
[22:12:57] <valhallasw`cloud>	 Template files should be stored in the templates directory of a Puppet module, which allows the template function to locate them with the simplified path format shown above. For example, the file referenced by template("my_module/mytemplate.erb") would be found on disk at /etc/puppet/modules/my_module/templates/mytemplate.erb (assuming the common modulepath
[22:12:57] <valhallasw`cloud>	 of /etc/puppet/modules).
[22:13:48] <valhallasw`cloud>	 YuviPanda: should be modules/ssh/templates/thingie.erb, then?
[22:14:29] <YuviPanda>	 Oh hmm
[22:14:31] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1206 is OK Less than 1.00% above the threshold [0.0]
[22:14:33] <shinken-wm>	 RECOVERY - Puppet failure on tools-precise-dev is OK Less than 1.00% above the threshold [0.0]
[22:14:34] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1410 is OK Less than 1.00% above the threshold [0.0]
[22:14:37] <YuviPanda>	 But it wasn't in a module
[22:14:38] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1204 is OK Less than 1.00% above the threshold [0.0]
[22:15:17] <shinken-wm>	 RECOVERY - Puppet failure on tools-webproxy-01 is OK Less than 1.00% above the threshold [0.0]
[22:16:13] <valhallasw`cloud>	 YuviPanda: template and file logic are probably not the same
[22:16:16] <JohnFLewis>	 valhallasw`cloud: from looking at another manifest, ssh/root-authorized-keys.erb should work for modules/ssh/templates/root*
[22:17:36] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1211 is OK Less than 1.00% above the threshold [0.0]
[22:17:46] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1409 is OK Less than 1.00% above the threshold [0.0]
[22:17:56] <shinken-wm>	 RECOVERY - Puppet failure on tools-master is OK Less than 1.00% above the threshold [0.0]
[22:18:00] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1212 is OK Less than 1.00% above the threshold [0.0]
[22:18:06] <shinken-wm>	 RECOVERY - Puppet failure on tools-checker-01 is OK Less than 1.00% above the threshold [0.0]
[22:18:12] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1407 is OK Less than 1.00% above the threshold [0.0]
[22:18:38] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1207 is OK Less than 1.00% above the threshold [0.0]
[22:18:40] <shinken-wm>	 RECOVERY - Puppet failure on tools-services-01 is OK Less than 1.00% above the threshold [0.0]
[22:18:46] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-generic-1401 is OK Less than 1.00% above the threshold [0.0]
[22:19:14] <shinken-wm>	 RECOVERY - Puppet failure on tools-bastion-01 is OK Less than 1.00% above the threshold [0.0]
[22:19:38] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1406 is OK Less than 1.00% above the threshold [0.0]
[22:19:58] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1406 is OK Less than 1.00% above the threshold [0.0]
[22:20:00] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-cyberbot is OK Less than 1.00% above the threshold [0.0]
[22:20:44] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1209 is OK Less than 1.00% above the threshold [0.0]
[22:21:00] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1403 is OK Less than 1.00% above the threshold [0.0]
[22:21:22] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-generic-1403 is OK Less than 1.00% above the threshold [0.0]
[22:21:42] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1214 is OK Less than 1.00% above the threshold [0.0]
[22:22:43] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1401 is OK Less than 1.00% above the threshold [0.0]
[23:28:39] <Negative24>	 Happy 4th. You guys must love your job...