[00:41:46] any suggestions for how to forward X11 from a labs instance to my laptop? Trying to setup some browsertests which will eventually run headless, but for testing i would like to forward it locally [00:42:18] i'm using the ProxyCommand setup, that forwards through bastion without making a login shell there [00:44:21] ebernhardson: On the instance configuration page, there's something about X11 forwarding (haven't tested it). [00:45:10] scfc_de: i hadn't noticed that, thanks i'll check it out [00:53:37] scfc_de: thanks, that was exactly the problem. The default sshd_config doesn't include 'X11Forwarding yes' so has to be enabled specifically [01:26:52] !stats [01:26:52] http://tools.wmflabs.org/?status [01:53:11] Hey, any progress on https://bugzilla.wikimedia.org/show_bug.cgi?id=59926 ?? If not, can I have some .htaccess help? [01:56:12] Actually, nvm, IRL is hitting. Ask later. [03:01:30] OK, y'all. Google Analytics? Yea or Nay? [04:59:32] Matthew_: sorry for the delay in responding. In general using Google Analytics with Wikimedia or Labs projects will violate our privacy policies. [04:59:44] andrewbogott: OK. [04:59:54] There are a few narrow exceptions given sufficient disclosure and such. If you have a specific case in mind you can write to the labs list to discuss. [05:00:32] OK. yeah, I was just curious what kind of hits I was getting on my tools. [05:00:46] So it's not a big deal. [05:01:19] * andrewbogott nods [13:57:07] !log tools tools-webserver-02: rm -f /var/log/exim4/paniclog (out of memory errors on 2014-01-10) [13:57:09] Logged the message, Master [13:57:42] can't connect to labs this morning via ssh [13:59:00] looks like bastion1 isn't happy but bastion2 is fine [14:28:27] hi all! i got a noob question about file permissions. I have some files locally, which i want to transfer to the "podt-feed" tool account. [14:28:49] i copied the files to my personal labs account (daniel@tools-login) with rsync. now for some reason they belong to the "svn" group, but the owner is correct (me). [14:29:31] i can copy -r them to ~local-potd-feed, but i want them to be a) editable by local-potd-feed, abd b) accessible via the web (www-data, i guess) [14:30:01] but chown tells me i don't have permission to change the owner or group. so.... now what? [14:31:11] hm, just saw that my user's primary group appears to be "svn"? is that normal? [14:33:01] DanielK_WMDE_: Try using "take" as the tool user (local-potd-feed) to change the ownership. To make files accessible by the web, generally put them in the public_html subdirectory in ~local-potd-feed. See https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help/NewWeb for details on the better way to do that. [14:33:37] anomie: ah, didn't know abotu "take", will have a look. public_html is clear, but the web server needs read permission, too... [14:33:44] i guess world readable is fine for web stuff [14:33:52] thanks! [14:34:32] anomie: take sais: "You need to share a group with the file" [14:35:12] hm, i think i got it now, hold on... [14:37:01] DanielK_WMDE_: Home directories of tools and their sub folders are set "+s" so that new files created there are owned by the group; take requires that group ownership to set user ownership as well. [14:37:54] yea... i had to run chgrp under my personal account, and then run take under the tool account. [14:37:57] that did it [14:38:04] took a while for me to figure this out >_< [14:38:35] thanks anomie and scfc_de [14:58:25] !log tools tools-login: Disabled local-mlwikisource's crontab with explanation [14:58:28] Logged the message, Master [14:59:27] when a cron script on a tool account fails, who gets the email? [15:00:43] IIRC all the maintainers of the tool account, except if there is ~/.forward. [15:01:53] ok, cool [15:02:06] hm... [15:02:08] $ crontab -e [15:02:09] -bash: fork: Cannot allocate memory [15:02:11] what gives? [15:03:25] Someone's overloading tools-login, I assume :-(. [15:03:46] scfc_de, I didn't do it. :p [15:04:28] * Cyberpower678 always remembers to use pkill when he's running his stuff on login [15:04:57] where can i find the grid engien howto? [15:06:04] !help DanielK_WMDE_ [15:06:05] !documentation for labs !wm-bot for bot [15:06:19] I never get those right ... *argl* [15:06:23] !documentation [15:06:36] wm-bot, lies. [15:06:39] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Submitting.2C_managing_and_scheduling_jobs_on_the_grid [15:06:48] DanielK_WMDE_: ^ [15:06:51] thanks :) [15:07:10] HOLY SHIT [15:07:20] oops. Wrong channel. [15:07:32] Cyberpower678: but that comment is appropriate universally. [15:07:44] :D [15:07:57] scfc_de: very roughly, can i just prepend whatever i had in my crontab with qsub, and it should work? [15:09:58] DanielK_WMDE_: The default memory limit might be too low. Also, we recommend *j*sub over *q*sub; you might need "-b y" and other bits with the latter. [15:10:21] ok, thanks [15:10:24] i'll try with jsub [15:10:25] !log tools-login: pkill -u local-mlwikisource: Freed 1 GByte of memory [15:10:25] tools-login: is not a valid project. [15:10:40] !log tools tools-login: pkill -u local-mlwikisource: Freed 1 GByte of memory [15:10:42] Logged the message, Master [15:10:43] re memory limit: i'm scaling potentially big images with imagemagick, so yea... [15:10:47] how would i fix that? [15:11:13] DanielK_WMDE_: If you're sending the job to the grid, you can request specific amounts of vmem. [16:06:08] I can confirm manybubbles report above that ssh to labs instances via bastion1.pmtpa.wmflabs is failing but bastion2.pmtpa.wmflabs succeeds [16:07:50] Actually, there's no "1" in bastion.wmflabs.org :-). But, yeah: "ssh_exchange_identification: Connection closed by remote host". [16:08:38] Connecting to bastion.wmflabs.org:ssh establishes a connection and then closes it immediately. [16:08:55] andrewbogott: ^ [16:09:24] * andrewbogott tries [16:11:27] hm... [16:13:54] andrewbogott: BTW, bastion2 and bastion3 say "*** System restart required ***". As I'm the only user on 3 at the moment, and all users on 2 are in this channel (aude, manybubbles, bd808), it might be a good time for a reboot. [16:14:23] (After their okay.) [16:15:03] A reboot is ok with me [16:16:43] scfc_de: You are correct (no 1 in hostname). My .ssh/config pretends there is one via an alias for some reason (copy-paste from somewhere) [16:17:19] !log bastion rebooting bastion1 [16:17:21] Logged the message, dummy [16:18:11] :( but ok [16:18:22] scfc_de, manybubbles, bd808, is bastion any happier now? [16:18:35] * aude couldn't get into bastion1 [16:18:38] bd808: It's from Help:Access. [16:19:11] andrewbogott: I can log into bastion.wmflabs.org now. [16:19:25] I don't think I'm going to reboot the other bastions just now… they'll get restarted soon enough anyway :) [16:19:32] andrewbogott: Works for me now [16:20:13] andrewbogott: yes [16:20:21] andrewbogott: Do you know when Ubuntu considers a system in a state "*** System restart required ***"? [16:20:34] I think after a kernel patch [16:20:53] I just know that labs instances say that ~ all the time. [16:20:59] s/patch/package upgrade/ [16:34:57] andrewbogott: do you know where/how stuff in "Global groups" in labs puppet config gets defined? [16:37:16] aude: You mean the entries that aren't project-specific? [16:37:29] yeah [16:37:31] At the moment they're configured by hand just like the other entries, but I think you have to be cloudadmin to alter them. [16:37:34] Is there a mistake I should fix? [16:37:49] we're trying to setup a jenkins instance in labs for wikidata [16:37:58] i see a puppet module for that in git [16:38:26] try to see if that's what we need [16:38:32] Can you load https://wikitech.wikimedia.org/wiki/Special:NovaPuppetGroup? [16:39:06] Do you need it to be global? The above link should let you add options for just the wikidata project. [16:39:26] doesn't need to be global [16:39:46] i can see that [16:40:23] That page is pretty confusing but it should do what you want. [16:40:27] ok [16:40:28] great [16:40:52] A 'group' on that page is just an arbitrarily-named organizational group [16:41:02] never would have found that [16:41:20] It's in your sidebar, 'Manage Puppet Groups' [16:41:25] ok :) [16:41:30] Or, should be, under 'projectadmins' [16:41:37] But, yeah, it's not the kind of thing you use every day [17:58:38] Coren, etc., any thoughts on why I can't add another sites-enabled to a labs instance? I can set it up, then visit it, but almost immediately afterwards it gets deleted. [17:58:46] And apparently the webserver gets reset [17:58:49] It's very mysterious [17:59:16] multimedia-alpha in particular [17:59:34] I guess it's managed by puppet, so puppet might be deleting my file for some reason [17:59:38] But that seems really silly [17:59:43] yuvipanda: CC ^^ [18:01:47] marktraceur: it probably does delete your sites-enabled [18:01:49] well, possibly [18:09:41] Well [18:09:43] Helpful [18:09:57] marktraceur: What does /var/log/puppet.log say? [18:11:08] notice: /File[/etc/apache2/sites-enabled/001-limn]/ensure: removed [18:11:10] Sigh. [18:11:17] OK, how do I mess with that? [18:12:33] Put /etc/apache2/sites-enabled/001-limn in Puppet? [18:15:04] marktraceur, scfc_de: ask in #wikimedia-analytics and ask either ottomata or milimetric [18:15:24] drdee: It's not a limn problem, it would delete any file I had there [18:15:24] they wrote the puppet manifest for limn [18:15:36] oh ok, i thought limn was causing havoc [18:15:39] scfc_de: Where is the puppet file? [18:18:46] marktraceur: Are you using a self-hosted puppetmaster? (If ottomata is connected to this project, he would probably be a good person to talk to about that.) [18:19:56] scfc_de: Hilariously I don't even know, yuvipanda pointed puppet at this box and I don't know how [18:22:04] scfc_de: it's not self-hosted puppet [18:22:13] scfc_de: was just using the mediawiki_labs thing [18:22:48] mediawiki_singlenode [18:22:48] I mean [18:22:52] marktraceur: pearl-hammer and multimedia-dragons appear to be self-hosted puppetmasters; you could use the instructions from https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster to set it up for multimedia-alpha as well. Whether that's the Right Thing(TM) for your use case, I don't know. [18:23:20] it's very probably not the 'right thing' [18:23:32] marktraceur: if you want limn, you should just use the limn class that the drdee pointed you to [18:23:49] you should be able to use their puppet class here [18:24:25] marktraceur: Much better if you can do so. [18:25:10] yuvipanda: Bah [18:25:12] Fine [18:25:27] also I dunno how you were gonna get node to talk to apache [18:25:28] yuvipanda: But I still have no bloody idea where you've set up puppet for -alpha [18:25:35] yuvipanda: reverse proxies [18:25:56] marktraceur: -alpha isn't self-hosted, is it? [18:25:57] * yuvipanda looks [18:26:05] I don't knooooooow [18:26:54] marktraceur: looking now [18:27:50] marktraceur: https://wikitech.wikimedia.org/w/index.php?title=Special:NovaInstance&action=configure&instanceid=f004f53b-568a-4085-91ca-b752f15c18ee&project=multimedia®ion=pmtpa [18:27:54] marktraceur: is where it is configured [18:28:34] I don't see limn on that page [18:29:35] You can add the limn with Special:PuppetGroups [18:29:53] https://wikitech.wikimedia.org/wiki/Special:NovaPuppetGroup, I mean. [18:30:11] Ah, christ, that's definitely not intuitive [18:44:28] marktraceur: Yes, that's managed via puppet; you need to add the site /there/ otherwise puppet will clean it up. [18:44:45] marktraceur: Use the configuration management to manage your configuration. :-) [18:45:55] Coren: Now I'm doing that, but without knowing that the thing is managed it was impossible to sort that out [18:51:55] OK, I've added the class, but it doesn't seem to have been applied [18:54:36] marktraceur: Has Puppet run since you changed the configuration? Either wait 30 minutes or "sudo puppetd -tv". [18:55:06] Could not find class misc::limn::instance bah [18:56:49] misc::limn::instance seems to be a "define", not a "class". [18:57:09] Hm [18:57:20] So 'limn::instance' or just 'limn' [18:58:53] marktraceur: Do you want to replicate reportcard.wikimedia.org? [18:59:17] No, I just want a limn instance [18:59:19] I can configure it [18:59:44] This shouldn't be so bloody complicated [19:01:37] marktraceur: I'm not sure there's an easy solution. yuvipanda? [19:04:54] Maybe I should just put the site into puppet somehow [19:10:10] Coren, scfc_de, puppetd -tv is always returning complaints about Could not find class misc::limn::instance even though I've removed that class from the config. [19:10:31] we can chat here too [19:10:38] how are you including misc::limn::instance? [19:10:44] it isn't a class, btw, it is a define [19:10:46] I went to https://wikitech.wikimedia.org/wiki/Special:NovaPuppetGroup [19:10:55] ahh and tried to add it [19:10:56] yeah [19:10:57] I added a group under "multimedia" [19:10:58] you can't, [19:11:09] the labs console interface won't support defines like that [19:11:18] Oh, wait [19:11:22] Maybe this will work. [19:11:28] I think you'll have to set up self hosted puppet master [19:11:30] Aha [19:11:39] and then edit some files in /etc/puppet [19:11:40] :/ [19:11:40] I can include the limn class at least [19:11:57] that will get you node and some directories and users created [19:11:58] thats it [19:12:15] Sigh [19:12:44] set up self hosted puppet [19:12:55] How? [19:13:08] No helpful search results on wikitech [19:13:15] check the so, you'll have to set up self hosted puppet master [19:13:16] oops [19:13:21] check the role::puppet::self box [19:13:21] yeah [19:13:30] https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster [19:13:30] Ah. [19:13:41] yeah, check that box [19:13:42] run puppet [19:13:56] then you will have /etc/puppet directory (which has lots of symlinks to /var/lib/git/operations/puppet) [19:14:12] add a file in /etc/puppet/manifests/role/ [19:14:13] maybe [19:14:16] 'kay [19:14:39] /etc/puppet/manifests/role/limn.pp (or whatever, doesn't matter) [19:14:43] then in that file [19:14:44] add something like [19:15:14] class role::limn::multimedia-metrics { [19:15:14] misc::limn::instance { 'multimedia-metrics': …blablabla params } [19:15:15] } [19:15:32] then you can add the role::limn::multimedia-metrics class to the labsconsole puppet groups interface [19:15:34] and check the box [19:15:40] fingers crossed and that will work :) [19:16:04] Woohoo. [19:21:54] err: /Stage[main]/Apache/Service[httpd]: Failed to call refresh: Could not start Service[httpd]: Execution of '/etc/init.d/apache2 start' returned 1: at /etc/puppet/modules/apache/manifests/init.pp:36 [19:21:58] fuuuuun [19:22:06] I'll try it again I guess [19:23:36] Invalid command 'ProxyRequests', perhaps misspelled or defined by a module not included in the server configuration [19:24:06] Freakin' broken config file [19:32:02] Coren: On tools-login, /home and /data/project seem to be double-mounted, once to labnfs.pmtpa.wmnet:..., and once to /etc/auto.space. Comparing /etc/auto.space on tools-login and tools-exec-09 shows no difference. How could this have happened? [19:32:45] scfc_de: That's only an artefact of automaounter braindamage. [19:33:19] scfc_de: automount, annoyingly, copes *really* badly with configuration changes without an outright reboot. This is a side effect of that. [19:35:06] Coren: Okay, thanks. BTW, I think you wanted to schedule a reboot for the pagecounts issue. [19:36:25] Yeah, I'm still trying to juggle a good time for it that fits in my shedule and is least likely to be disruptive. [21:10:30] If Wikimedia's been dedicated to UTF-8 since 2003-4, why do I have to echo 'AddDefaultCharset utf-8' > ~/public_html/.htaccess ? [21:16:01] Dispenser: dont bother with htaccess, look into newweb [21:16:56] wtf is that? I just want to serve MySQL HTML result. [21:29:35] !newweb [21:29:35] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help/NewWeb [21:29:41] Dispenser: ^ [21:29:55] Apache is being killed [21:30:13] real pain in the ass [21:30:50] So full Toolserver Migration in 2016 I take it. [21:32:07] Dispenser: no, toolserver is being killed soon [21:34:41] I don't think the whole we're killing your tools, so deal with it is going to fly well with the editing community. Especially as resentment and dysfunction become clearer. [21:36:14] So back the basic question: Why do I manually have to enable UTF-8 on the webserver? [21:39:25] Standard setting? http://toolserver.org/~timl/test.txt doesn't give UTF-8, either. [21:41:21] You call browser based heuristic detection 'standard' sure. And it opens the UTF-7 attack. [21:42:25] Feel free to file a bug with Apache. [21:43:23] They'll likely close it for compatibility reason, but we don't use any other character encodings [21:52:16] hi fellas [21:52:20] DNS question [21:52:32] I just added this http://multimedia-metrics.wmflabs.org/ [21:52:37] to point to my limn0 instance [21:52:41] apache is configured fine [21:53:00] curling locally on the port that the VirtualHost points to works fine [21:53:05] milimetric: WFM [21:53:09] the other addresses pointing there works fine [21:53:12] what?! [21:53:14] :) [21:53:20] so i've just got some dns cache? [21:53:23] k, never mind! [21:53:24] Maybe [22:19:06] In memcached randomly dying for anyone else on Labs MW instances? [22:24:19] i've had it get stuck at 100% cpu before, but not die [22:45:32] Coren: familiar with mysql slaves at all? I've got my vagrant role for running multiple mysql servers in vagrant working, but now that i'm reading through mysql's replication i'm not sure if theres any way to actually setup replication from inside a puppet job [22:46:44] basically, mysql replication is amazingly complex and needs a bunch of custom work done to get the master<->slave initial sync done [23:00:26] * ebernhardson sighs ... instead of spending 20 hours building a puppet role should have just spent 2 hours setting it up manually [23:49:50] ebernhardson: The right person to ask is Sean, who is our resident actual expert. :-) [23:56:21] Coren: i'll check with him, thanks