[02:19:39] legoktm: thinking of adding a box that just serves static file content, with varnish and stuff [02:19:41] should be super fast [02:19:44] * YuviPanda|away plots [02:25:58] is public_html in the user directory (not tool directory) accessible from the web somehow? [02:26:23] no [02:36:01] johang: what SigmaWP said. nothing from user directory is available from the web. Use tools [04:16:08] Is the 'text' table available on Tool Labs? [04:17:54] No. And even if it were, it wouldn't be of any use to you because that's not how article text is stored on WMF wikis. [04:18:28] How is it stored then? [04:20:31] [[mw:Manual:External Storage]], I believe [04:21:13] So the only way to get it is though the API? [04:21:34] Yes. [06:44:45] is it possible to change the "Full name" field of my gerrit account? [06:50:44] yes [06:51:01] see https://wikitech.wikimedia.org/wiki/Renaming_users [06:51:08] you'll need to find someone (probably in ops) to do it for you though [06:55:33] legoktm: lisn't the page about changing user name? [06:55:43] ^isn't [06:55:45] yes [06:55:50] its both though I think [06:57:18] https://gerrit-review.googlesource.com/Documentation/cmd-set-account.html this seems to say that admins can issue a command to do that, for the gerrit account. [06:57:51] anyway, that is also someone from ops, probably [07:04:14] whym: gerrit docs in general don't necessarily apply to WMF gerrit [07:04:58] jeremyb: https://gerrit.wikimedia.org/r/Documentation/cmd-set-account.html [07:05:09] hah [07:05:17] however because its also linked to labs + wikitech, other things need to be done too [07:05:29] gerrit docs in general (even if hosted on the wikimedia gerrit instance) don't necessarily apply to WMF gerrit [07:05:35] :P [07:05:56] those docs are essentially just a mirror of the other docs [07:06:45] whym: WMF gerrit shares LDAP with wikitech LDAP and labs machines' LDAP and some prod HTTP basic auth. those docs don't know about all the other pieces. [07:14:53] I see :) [09:44:28] Hi, the server on WMF Tool Labs gives me "503 Service Temporarily Unavailable"! https://tools.wmflabs.org/intersect-contribs/ [09:46:01] PHP problem? Maintenance? [09:46:57] (03PS3) 10Nemo bis: Add gerritfeed-wm for a full feed relayed to IRC [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/87663 [10:19:49] i cannot ssh into instances anymore [10:20:57] ssh_exchange_identification: Connection closed by remote host [10:21:44] is something else broken? [10:33:49] pietrodn: https://bugzilla.wikimedia.org/show_bug.cgi?id=57794 [10:36:13] multichill: thanks. Can I vote for this bug? [10:36:50] Sure, not sure if it makes any difference, but you can always try :-) [10:38:03] multichill: the question is: where is the button to vote on Bugzilla? :P [10:38:14] once upon a time I used to vote bugs :P [10:39:05] pietrodn: Importance -> vote [10:44:19] has anyone called Coren out of bed yet? ;-) [10:45:55] An user on it.wiki asked me abount one of my tool on the Toolserver (my account has expired). I: "Hey! It has now moved to WMF Labs… um… and it's offline" XD [10:58:40] The NewWeb-based sites are still working. [11:00:39] qstat fails w/ error: commlib error: got select error (No route to host) [11:01:08] tools grid status doesn't show up: http://tools.wmflabs.org/?status [11:36:23] My tools are 503 as well, and I'm not able to ssh in anymore [11:48:34] yay [11:48:39] i'm not alone :) [12:47:41] Web seems to be up again [12:47:53] ( http://tools.wmflabs.org/ works for me) [12:48:30] oh, tool accounts are still down [12:51:12] valhallasw: is the grid up? [12:51:43] * YuviPanda considers rebooting the webservers [12:52:36] tools-login seems fine [12:54:08] YuviPanda: qmaster is also down [12:54:13] http://ganglia.wmflabs.org/latest/?r=hour&cs=&ce=&s=by+name&c=tools&tab=m&vn= [12:54:14] yeah [12:54:24] valhallasw: rebooted master [12:56:16] valhallasw: and the webservers too [13:05:16] valhallasw: responded on the bug. this requires higher powers [13:07:48] Hi, can anybody aprove my shell request? https://wikitech.wikimedia.org/wiki/Shell_Request/Lluis_tgn [13:17:14] lluis_tgn: done! [13:17:38] thanks! ;) [13:18:10] yw [13:46:07] WTF is going on? [13:46:27] :0 [13:49:03] :O:O [14:05:10] !log tools underlying virtualization hardware rebooted; tools-master and friends coming back up. [14:24:00] Fix ALL THE THINGS!!!! [14:24:39] the Wikidata synchronisation is still running :) [14:27:01] Coren: enjoy your sunday! :) [14:30:02] Coren: is there a reason you are using top-level scope in the tool labs module? [14:30:11] Thank you Coren, now my tools are up and running :) [14:30:25] matanya: Need moar contexts! [14:30:54] GerardM-: It was the VM lottery. If your tool ran on anything but virt10, it should have survived uninterrupted. :-) [14:31:30] Coren: e.g modules/toollabs/manifests/init.pp line 43 [14:31:50] top-scope variable being used without an explicit namespace on line 43 [14:32:14] it should probably be stuff like ::$ip or something, no? [14:33:48] i still can't log in anywhere [14:33:54] :( [14:34:19] matanya: Hm; those are tidbits coming from facter. I'm not even sure you can do ::$ip [14:34:32] giftpflanze2: What happens when you try? [14:34:52] ssh_exchange_identification: Connection closed by remote host [14:35:15] Where did you try it last so I can look at the logs? [14:35:25] tools-login [14:38:18] giftpflanze2: What's your shell username again? [14:38:22] gifti [14:38:42] ... I don't even see an attempt from you. [14:38:48] … [14:39:11] Try now? [14:39:16] i did [14:39:32] I mean after I asked, while I'm looking at the logging. :-) [14:40:03] do you see anything, i tried a few times now [14:40:27] Nope. As far as I can tell you're not even hitting the server at all. [14:40:34] damn [14:40:52] giftpflanze2: If you wan, PM your current IP so I can tcpdump? [14:41:00] ok [14:42:20] Coren: when you are done with this, mind clear some questions i have regarding the puppet stuff for labs? [14:42:30] matanya: Sure. [14:42:42] giftpflanze2: I'm not even seeing your packets reach tools-login at all. [14:43:01] giftpflanze2: Can you do a ssh -vvv and pastebin the output? [14:43:08] i do [14:45:23] ok, i changed from bastion.wmflabs.org to bastion2.wmflabs.org which works [14:45:50] ... wait, you're logging into bastion? I though you said tools-login! [14:46:05] tools-login [14:46:30] i use the ProxyCommand configuration [14:46:43] See, the reason why I couldn't see your packets reach tools-login was because you weren't actually trying to log in there! [14:46:56] tools-login.wmflabs.org is directly accessible. :-) [14:47:08] it's habit [14:47:22] matanya: What be up? [14:47:57] Coren: you have some declartions of absent that needs to be removed, but i don't no if they can be [14:48:36] matanya: Which? It's entirely possible that they can; much of this was written long before the puppet cleanup. [14:49:53] Coren: do you think bastion is broken? [14:49:55] Coren: modules/toollabs/manifests/init.pp line 68, modules/toollabs/manifests/bastion.pp line 70, modules/toollabs/manifests/webnode.pp line 81 [14:50:13] Coren: I can't start the new webserver on my tools [14:51:06] giftpflanze2: It may be. [14:51:39] matanya: Yep, safe to remove. [14:51:51] matanya: Those were there to remove an other way of doing the tips thing. [14:52:03] pietrodn: What happens why you try? [14:52:18] ok Coren i'll push a patch to remove all [14:52:41] Coren: do you have more insight about the database issues now? [14:53:09] giftpflanze: The dewiki thing? No, this will need the tender loving care of an actual DBA (aka Sean) [14:53:17] Coren: well, "webservice start" succeeds but when I do "webservice status" it says "queue instance "task@tools-exec-01.pmtpa.wmflabs" dropped because it is temporarily not available" [14:53:19] ok [14:53:30] Coren: and then "Your webservice is not running." [14:54:13] pietrodn: What is your tool's name? [14:54:56] It shouldn't be trying to run as a task on the normal grid at all. [14:55:04] Coren: intersect-contribs [14:56:25] That's odd, I don't even see your pending job attempt either. You /are/ trying to start it from the tool user, right? (Just checking) :-) [14:56:52] Coren: I login as pietrodn on tools-login, then "become intersect-contribs", then "webservice start" [14:56:59] * Coren nods. [14:56:59] Coren: Hi, any update on adding SSH keys? [14:57:25] LuaKT: Sadly, the guy who knows how that system work will only be back to work from the long weekend tomorrow. [14:57:53] Coren: Ah ok, thanks [14:59:14] pietrodn: Have you checked your error.log? [14:59:41] Coren: "opening errorlog '/data/project/intersect-contribs/access.log' failed: Permission denied" [15:00:11] Ah, you might have an old-style access.log owned by root still. Make sure you move or delete it. :-) [15:00:26] Yes it's owned by root. Trying... [15:01:40] Coren: no more error entries in error.log, but "Your webservice is scheduled: [15:01:40] queue instance "task@tools-exec-01.pmtpa.wmflabs" dropped because it is temporarily not available" [15:02:07] I'm not sure why it tried running there, but I see it up and running so you can safely ignore that warning. [15:02:33] How can I make sure the new webserver is running? [15:02:36] Do a qstat from the local account. [15:02:51] Perfect. It's up. [15:02:53] You shoudl see the httpd-$tool running on webgrid@ [15:03:21] And it will be permanently running, right? [15:03:44] Yes, but it does not restart automatically on configuration errors. [15:04:17] ok, enabling it on my other projects! [15:06:34] It's running well. Thanks Coren! [15:09:38] Coren: could you perhaps help with the account setting on gerrit? [15:24:30] Coren: did you get my message? [15:35:20] LuaKT: what's your issue with SSH keys? (I may or may not be 'the guy who knows how...') [15:37:19] * andrewbogott missed everything [15:39:26] I was wondering how to change "Full Name" for my gerrit account. [15:39:43] I suppose an admin can do that? [15:42:03] whym: Um… maybe you can change it here? https://gerrit.wikimedia.org/r/#/settings/contact [15:42:06] I haven't tried :) [15:42:27] andrewbogott: that was non-editable for me :( [15:42:54] hm, for me too it seems [15:45:24] I'm not sure full name is managed by LDAP, but can this procedure https://wikitech.wikimedia.org/wiki/Renaming_users be used? [15:46:53] maybe -- there are lots of 'names' and I'm not sure which one corresponds to 'full name'. [15:47:16] I believe that those instructions would rename your account, which isn't what you want. [15:47:35] with vanilla gerrit it can be done with https://gerrit-review.googlesource.com/Documentation/cmd-set-account.html, it seems [15:47:41] I take it your account name in gerrit is 'whym' and your full name is 'Whym' with a capital? [15:48:04] yep, and I want change 'Whym' to 'Yusuke Matsubara' [15:48:37] What is your usename on wikitech? [15:48:50] andrewbogott: that's 'whym' too [15:49:05] yeah, but lowercase or capitalized? [15:49:41] lowercase, but I suspect MediaWiki doesn't care about that. [15:50:20] it doesn't care when you log in, but does it really display back to you as all lowercase? [15:50:43] for instance I can log in as 'andrew bogott' but the wikitech banner shows me as 'Andrew Bogott' [15:50:46] ah I see, it's capitalized, and shell account is in lowercase [15:51:21] ok! [15:51:31] So, I think that your 'full name' on gerrit is the same field as your username on wikitech. [15:51:40] Do you want me to change both? Or neither? [15:52:58] andrewbogott: do you mean they cannot be different? [15:53:23] Correct, I believe they derive from the same ldap field. [15:54:01] I'm not positive, but… when you created your wikitech account the gerrit account just appeared, right? Or did you create the two accounts separately? [15:54:06] * andrewbogott totally can't remember [15:54:45] I don't remember even asked for a 'full name', so it sounds correct [15:55:59] yeah, when I tell gerrit I want to registed it just bounces me to the wikitech signup page. [16:00:39] andrewbogott: hmm I actually don't like to log in a wiki with a long name, so it sounds like I'd have to keep it as it is. [16:00:51] yeah, seems so. Sorry. [16:01:34] thanks for your help anyway [16:41:52] andrewbogott_afk: Sorry, I missed you. Apparently the process for copying keys to the correct place after adding them to Wikimedia Openstack is broken so I can't use any of the keys added. [21:07:44] andrewbogott_afk: i doubt you could login as andrew bogott. andrew Bogott should work but not andrew bogott. [21:11:41] are there still problems with bastion? I've not been able to login since the problems earlier today [21:11:52] ssh: connect to host bastion.wmflabs.org port 22: Network is unreachable [21:13:52] hm, I'm able to login to bastion2 [22:03:50] danmichaelo: did you not just try bastion2 immediately? [22:04:47] no, I had forgot about it [22:04:59] k [22:05:02] $ nc -v bastion.wmflabs.org 22 [22:05:02] Warning: inverse host lookup failed for 208.80.153.207: Unknown host [22:05:02] bastion.wmflabs.org [208.80.153.207] 22 (ssh) : No route to host [22:05:02] ------- [22:05:02] $ nc -v bastion2.wmflabs.org 22 [22:05:04] Warning: inverse host lookup failed for 208.80.153.202: Unknown host [22:05:07] bastion2.wmflabs.org [208.80.153.202] 22 (ssh) open [22:05:09] SSH-2.0-OpenSSH_5.9p1 Debian-5ubuntu1.1 [22:05:20] Coren: andrewbogott_afk: ^ [22:17:56] * jeremyb has caught up [22:18:09] * jeremyb wonders what's wrong with bastion/why it's not fixed yet?? [22:18:37] andrewbogott_afk: the issue for LuaKT was that ldap gets new keys from OSM but they never get updated on /public/keys [22:50:31] https://wikitech.wikimedia.org/w/index.php?title=Special:Ask&p=format%3Dbroadtable%2Flink%3Dall%2Fheaders%3Dshow%2Fsearchlabel%3D%E2%80%A6-20further-20results%2Fclass%3Dsortable-20wikitable-20smwtable&po=%3FInstance+Name%0A%3FInstance+Type%0A%3FProject%0A%3FImage+Id%0A%3FFQDN%0A%3FPublic+IP%0A%3FLaunch+Time%0A%3FPuppet+Class%0A%3FModification+date%0A%3FNumber+of+CPUs%0A%3FRAM+Size%0A%3FAmount+of+Storage%0A&eq=no&q=[[Resource+Type%3A%3Ainstance]] [22:50:45] aka http://ur1.ca/g4mv7 [22:51:12] there's an A record but no machine associated with that IP