[08:18:56] <wm-bot>	 [bz] (8NEW - created by: 2Michelle Grover, priority: 4Highest - 6critical) [Bug 52776] Beta labs getting 503 when you attempt to log in - https://bugzilla.wikimedia.org/show_bug.cgi?id=52776
[09:38:04] <J-Mo>	 hi there. I just submitted a shell request on labs. Can someone add me? https://wikitech.wikimedia.org/wiki/Shell_Request/Jmorgan
[09:43:57] <zhuyifei1999>	 petan, Coren, hasher, YuviPanda: ^^
[09:45:50] <Steinsplitter>	 hashar.... hashar is not online /-(
[09:45:52] <Steinsplitter>	 for days
[09:47:26] <Amir1>	 Steinsplitter: he is in vacation IIRC Someone told me in Wikimania 
[09:48:11] <Steinsplitter>	 ah :-)
[09:48:20] <Steinsplitter>	 thx
[09:48:34] <zhuyifei1999>	 There's no network in Wikimania?
[09:49:13] <Amir1>	 zhuyifei1999: It was but we were busy at the moment 
[09:49:39] <Amir1>	 I had 200 threads in my mailbox unread-ed 
[09:50:00] <Amir1>	 I'm answering them and It's a very boring job
[09:52:18] <zhuyifei1999>	 J-Mo: 17:19, 20 June 2012 User account Jmorgan (Talk | contribs) was created by Sumanah (Talk | contribs) (password sent by e-mail)
[09:52:51] <J-Mo>	 thanks zhuyifei1999!
[09:52:57] <J-Mo>	 (and sumanah)
[09:53:30] <HenriqueCrang>	 yey, J-Mo onboard!
[09:53:50] <J-Mo>	 you say that now… we may all regret it
[09:53:56] <zhuyifei1999>	 sorry I mean you havn't applied until now?
[09:54:37] <J-Mo>	 yep, believe it or not
[09:54:49] <zhuyifei1999>	 I don't believe
[09:54:57] <J-Mo>	 :P
[09:58:36] <J-Mo>	 zhuyifei1999, are there any admins around I can ping directly to get shell access?
[09:59:06] <zhuyifei1999>	 I guess not
[09:59:16] <zhuyifei1999>	 Steinsplitter: Where's petan?
[09:59:41] <J-Mo>	 hmmm. lots of folks vacationing, sleeping or traveling I bet.
[10:03:41] <wm-bot>	 Change on 12mediawiki a page OAuth was Guillom, changed by https://www.mediawiki.org/w/index.php?diff=763613 link fix transclusion edit summary: $6
[10:38:51] <Nirakka>	 hey guys, I can't figure out how to login via public key (ssh)
[10:39:53] <Nirakka>	 when I paste my public key into https://wikitech.wikimedia.org/wiki/Special:Preferences#mw-prefsection-openstack, strange line breaks appear
[10:40:09] <Nirakka>	 and the input field seems to small ...
[10:40:18] <Nirakka>	 I always get pubkey denied. anyone else with that problem?
[10:51:21] <zhuyifei1999>	 Nirakka: OpenSSH?
[10:53:33] <Nirakka>	 zhuyifei1999: y
[10:53:56] <Nirakka>	 $ ssh Nirakka@bastion.wmflabs.org
[10:53:57] <Nirakka>	 Permission denied (publickey).
[10:54:36] <zhuyifei1999>	 you didn't paste privatekey?
[10:55:44] <zhuyifei1999>	 Public key is something like this:
[10:55:45] <Nirakka>	 lol no :D
[10:56:10] <Nirakka>	 ofc I pasted my .ssh/id_rsa.pub
[10:56:57] <zhuyifei1999>	 That's strange, I had no problems with that
[10:57:06] <Nirakka>	 it seems like the input box generates a line break before every slash ('/')
[10:57:12] <Nirakka>	 there are some slashes in my pubkey
[10:57:31] <zhuyifei1999>	 mine too
[10:58:17] <Nirakka>	 hm
[10:58:19] <zhuyifei1999>	 paste the whole file?
[10:58:23] <Nirakka>	 yes
[10:58:47] <Nirakka>	 including 'ssh-rsa' at the beginning, and 'nirakka@x220' at the end
[10:58:51] <zhuyifei1999>	 what if you generate another one?
[10:59:08] <Nirakka>	 well, I can try that
[10:59:23] <Nirakka>	 but what's strange: I use that ssh key successfully with another server
[11:00:50] <zhuyifei1999>	 there should be three lines in https://wikitech.wikimedia.org/wiki/Special:Preferences#mw-prefsection-openstack, one is "ssh-rsa", one is the key, one is "nirakka@x220"
[11:05:58] <Nirakka>	 yeah, that's the problem!
[11:06:03] <Nirakka>	 I got more lines
[11:06:05] <Nirakka>	 there are returns
[11:06:11] <Nirakka>	 but I don't know how to avoid them
[11:07:03] <zhuyifei1999>	 how many?
[11:08:14] <Nirakka>	 three
[11:08:28] <Nirakka>	 3 slashes, and the input field forces a return in front of each
[11:09:35] <zhuyifei1999>	 I have seven slashes
[11:09:48] <Nirakka>	 hm
[11:09:51] <Nirakka>	 it's ridiculous
[11:09:59] <Nirakka>	 I see the returns already in the input field
[11:10:03] <Nirakka>	 but I can't delete them
[11:10:22] <zhuyifei1999>	 space
[11:10:57] <Nirakka>	 zhuyifei1999: ?
[11:11:29] <zhuyifei1999>	 there should be spaces between the lines
[11:12:09] <Nirakka>	 there are no spaces
[11:12:17] <Nirakka>	 it just visually breaks the line
[11:12:34] <Nirakka>	 but there is nothing in between
[11:14:24] <zhuyifei1999>	 there should be about 390 chars
[11:18:50] <zhuyifei1999>	 Nirakka: what if you just copy and paste, without minding ths lines?
[11:23:49] <Steinsplitter>	 Nirakka: do you know hot to gererat a cert?
[11:24:43] <zhuyifei1999>	 Steinsplitter: ?
[11:25:05] <zhuyifei1999>	 generated certs arn't trusted
[11:25:15] <zhuyifei1999>	 *aren't
[11:26:04] <Steinsplitter>	 certs must be generated :P
[11:26:20] <zhuyifei1999>	 *self-generated
[11:26:24] <Steinsplitter>	 ah, jes.
[11:28:04] <zhuyifei1999>	 I've generated a lot of certs, and none of them are trusted
[11:29:19] <Steinsplitter>	 zhuyifei1999: do you know CaCert?
[11:29:39] <zhuyifei1999>	 CaCert?
[11:30:38] <Steinsplitter>	 http://www.cacert.org/
[11:30:53] <zhuyifei1999>	 I just searched it
[11:31:05] <zhuyifei1999>	 !google CaCert
[11:31:05] <wm-bot>	 https://www.google.com/search?q=CaCert
[11:31:38] <zhuyifei1999>	 but unfortunally, I don't really care about that
[11:32:23] <Steinsplitter>	  /-(
[11:33:08] <zhuyifei1999>	 sorry
[11:33:46] <Steinsplitter>	 no. all okay :D
[11:34:44] <addshore>	 petan: around? :>
[11:35:00] <zhuyifei1999>	 addshore: doubt so
[11:35:20] <Nirakka>	 zhuyifei1999: I do just copy and paste but in the input field there appear line breaks
[11:35:35] <zhuyifei1999>	 addshore: (17:38:04) <J-Mo> hi there. I just submitted a shell request on labs. Can someone add me? https://wikitech.wikimedia.org/wiki/Shell_Request/Jmorgan
[11:35:48] <addshore>	 will do in a second :)
[11:36:04] <zhuyifei1999>	 Nirakka: don't care it, save it and try ssh again
[11:36:10] <J-Mo>	 thanks zhuyifei1999 and addshore :)
[11:36:28] <addshore>	 done
[11:36:31] <MartijnH>	 I'm wondering, there is a ganglia view for replag, but I can't really find what unit is used
[11:36:34] <addshore>	 infact it was already done!
[11:36:34] <zhuyifei1999>	 Nirakka: ssh -v
[11:37:52] <MartijnH>	 (this view in case I am unclear: http://ganglia.wmflabs.org/latest/?c=tools&h=tools-login&m=load_one&r=hour&s=by%20name&hc=4&mc=2 )
[11:38:40] <Nirakka>	 zhuyifei1999: do you want to see the whole log?
[11:38:58] <zhuyifei1999>	 addshore: who did it? no user rights log
[11:39:05] <zhuyifei1999>	 Nirakka: maybe
[11:39:32] <addshore>	 hmm *checks again*
[11:40:03] <addshore>	 heh, indeed no rights log, but apparently you have shell, so I have no idea :P
[11:41:09] <zhuyifei1999>	 !pastebin
[11:41:09] <wm-bot>	 http://tools.wmflabs.org/paste/
[11:45:33] <zhuyifei1999>	 Jmorgan (Talk | contribs)‏‎ (shell) (Created on 20 June 2012 at 17:19)
[13:12:43] <Steinsplitter>	 the varnish/apache problem is back,,
[13:13:12] <Steinsplitter>	 manybubbles
[13:13:15] <Steinsplitter>	 *ping*
[13:13:26] <manybubbles>	 steinsplitter: *pong*
[13:13:49] <Steinsplitter>	 this morning a lot of varnish errors and now completly down
[13:16:40] <manybubbles>	 Ryan_Lane: I know I'm probably not high on your list, but did you get to look at that email I sent yesterday about beta?
[13:17:02] <Ryan_Lane>	 what about beta?
[13:17:04] <Ryan_Lane>	 you mean ssl?
[13:17:09] <manybubbles>	 nah - NFS
[13:17:18] <Ryan_Lane>	 I'm technically on vacation...
[13:17:29] <Ryan_Lane>	 NFS is still broken-ish
[13:17:41] <Ryan_Lane>	 we didn't have a chance to look at it at wikimania
[13:17:46] <Steinsplitter>	 http://ganglia.wmflabs.org  oh O_O
[13:17:50] <manybubbles>	 NFS being brokenish is making beta brokenish
[13:18:18] <manybubbles>	 Steinsplitter: It did that yesterday morning too.  I'm not sure why....
[13:18:44] <manybubbles>	 Ryan_Lane: is there someone else who can help with it?  I think there are a few of us this is hurting something fierce.
[13:22:39] <Ryan_Lane>	 manybubbles: there's not much I can really do right now
[13:22:55] <Ryan_Lane>	 coren has a better handle on the situation right now anyway
[13:24:43] <Steinsplitter>	 oh. beta is back o_O
[13:25:15] <manybubbles>	 Ryan_Lane: I'll bother him then.  Have a great vacation.  Where are you having it?
[13:25:43] <Ryan_Lane>	 thailand. in bangkok right now. everything is so cheap and the food is awesome!
[13:26:06] <Steinsplitter>	 oh,  good vacation^^ :)
[13:26:52] <scfc_de>	 Dito.
[13:27:35] <manybubbles>	 Ryan_Lane:  sweet!  have fun.
[13:27:46] <Ryan_Lane>	 Steinsplitter: the NFS issue apparently occurs a couple times an hour
[13:27:56] <Ryan_Lane>	 which will hang beta that often
[13:28:07] <Ryan_Lane>	 manybubbles: so, I recommended beta not use shared storage at all
[13:28:32] <manybubbles>	 Ryan_Lane: amen
[13:28:41] <Ryan_Lane>	 deploy mediawiki and configuration to the instances in /mnt
[13:28:50] <Ryan_Lane>	 hell, use git-deploy
[13:29:06] <Ryan_Lane>	 it's on the schedule for next quarter. may as well get a jump on that
[13:29:45] <Ryan_Lane>	 no shared storage will make things *way* more stable
[13:29:47] <manybubbles>	 I'll send hashar an email about it.
[13:30:17] <manybubbles>	 Ryan_Lane: I've never really been a fan of NFS in particular....
[13:30:26] <Ryan_Lane>	 shared storage has been the only instable thing in like a year and a half or so
[13:33:58] <manybubbles>	 uhg - I'm missing so many permissions on these machines!  on deployment-bastion I can sudo whatever I want.  on deployment-apache32 I'm only allowed to do deployment stuff....
[14:31:04] <wm-bot>	 [bz] (8REOPENED - created by: 2Michelle Grover, priority: 4Highest - 6critical) [Bug 52776] Beta labs getting 503 when you attempt to log in - https://bugzilla.wikimedia.org/show_bug.cgi?id=52776
[16:49:42] <Coren>	 Moo!
[16:49:49] <Coren>	 I live.
[16:50:58] <Coren>	 Ryan_Lane: Ima going to try to isolate the problem now; I'm going to make a rsync/switch to a non-thin provisionned partition so that I can rollback to 3.2.  This is tell us if the issue is in the hardware.
[18:12:14] <scfc_de>	 Coren: BTW, do you know how/where icinga.wmflabs.org is set up?
[18:12:40] <Coren>	 Not really, I haven't looked into it at all.
[18:13:24] <scfc_de>	 k
[18:38:43] <chrismcmahon>	 huh, apaches on deployment-apache32 just melted down. 
[18:38:59] <chrismcmahon>	 33 I mean
[18:39:32] <Coren>	 There does seem to be a stall atm
[18:50:30] <chrismcmahon>	 restarted apaches on 32 and 33
[18:50:35] <chrismcmahon>	 because why not
[19:55:58] <manybubbles>	 I've got another machine that can't be recreated:  solr-mw2.  I waited ten minutes between deleting it and recreating it
[20:00:30] <manybubbles>	 OK!  I just created a new machine (solr-mw3) and now I can't even ssh into it.  It just hangs....
[20:06:05] <scfc_de>	 manybubbles: That was just the minute Coren wanted to restart NFS ...
[20:06:18] <manybubbles>	 ha!
[20:06:36] <manybubbles>	 I suppose I'll wait then
[20:06:42] <Coren>	 Yeah, not gonna happen today.  The copy is *very* long.
[20:06:48] * Coren  posts an update.

[20:13:38] <rschen7754>	 now?
[20:13:46] <rschen7754>	 oh crud, the 15th
[21:31:31] <wm-bot>	 Change on 12mediawiki a page Developer access was Jeremyb, changed by https://www.mediawiki.org/w/index.php?diff=764026 link [-3] rm a redirect hop edit summary: $6
[21:40:06] <wm-bot>	 Change on 12mediawiki a page Developer access was Jeremyb, changed by https://www.mediawiki.org/w/index.php?diff=764044 link [-23] and another edit summary: $6
[21:41:45] <jeremyb>	 hi wm-bot!
[21:42:07] <jeremyb>	 petan: what's up with teh $6?
[21:45:55] <FunPika>	 jeremyb: I think someone with wm-bot admin in this channel needs to fix the recent-changes-template https://meta.wikimedia.org/wiki/Wm-bot#.40configure
[21:46:09] <FunPika>	 @trusted
[21:46:09] <wm-bot>	 I trust:  .*@wikimedia/.* (2trusted), .*@mediawiki/.* (2trusted), .*@wikimedia/Ryan-lane (2admin), .*@wikipedia/.* (2trusted), .*@nightshade.toolserver.org (2trusted), .*@wikimedia/Krinkle (2admin), .*@[Ww]ikimedia/.* (2trusted), .*@wikipedia/Cyberpower678 (2admin), .*@wirenat2\.strw\.leidenuniv\.nl (2trusted), .*@unaffiliated/valhallasw (2trusted), .*@mediawiki/yuvipanda (2admin),
[21:47:14] <jeremyb>	 FunPika: well i'm root :)
[21:47:23] <jeremyb>	 well, maybe not any more. idk if it moved
[21:47:32] <jeremyb>	 anyway...
[21:47:34] <wm-bot>	 You are root identified by name .*@wikimedia/jeremyb
[21:47:46] <jeremyb>	 at least that kind of root if not the other
[21:48:23] <wm-bot>	 Value of recent-changes-template is: <default value>
[21:48:34] <jeremyb>	 welp, that's an upstream bug not a channel bug then!
[21:48:38] <jeremyb>	 petan: ^^
[22:03:22] <scfc_de>	 jeremyb: petan is still on vacation.  Looking at translation/english.txt, the default (for English) seems to be "Change on $1 a page $2 was $3, changed by $4 link $5 edit summary: $6;".  For Chinese, Czech and German, it only uses $5, so something's fishy there.
[22:04:40] <wm-bot>	 [bz] (8NEW - created by: 2Antoine "hashar" Musso, priority: 4Unprioritized - 6normal) [Bug 52867] monitor application server are responding - https://bugzilla.wikimedia.org/show_bug.cgi?id=52867
[22:05:34] <scfc_de>	 Parameters seem to be: "new List<string> { "^C12" + name_url + "^C", "^B" + page + "^B", "created", "^B" + username + "^B", url + "?title=" + page, summary }"
[22:06:41] <scfc_de>	 Which look to me like six; hmmm.
[22:06:45] <jeremyb>	 scfc_de: ok, well it's still his bug :)
[22:08:47] <scfc_de>	 Technical_13: Did you write https://github.com/benapetr/wikimedia-bot/commit/0c5f25de466fc14a29f65f389aee26bc3c6a82fe?  Is that related to $6?
[22:09:14] <Technical_13>	 Probably
[22:10:09] <Technical_13>	 $6 isn't the problem. $3 is what's missing and I pushed something that should fix it, but it needs to be compiled and uploaded.
[22:10:10] <scfc_de>	 Could it that the message templates and the source were updated, but the bot not recompiled?
[22:10:22] <Technical_13>	 ^^^ bingo.
[22:10:23] <scfc_de>	 Okay, that was parallel :-).
[22:11:30] <Technical_13>	 :)
[22:59:42] <alexbrollo>	 Hi all, just to let you know that I successfully run pywikibot from itsource project into Labs, reading and writing a it.source page. It has been far from simple and I have been forced to use rewrite while I'd prefer trunk.... but I'll survive. 
[23:01:20] <alexbrollo>	 A question. Where I can ask for some applications/modules for itsource project? I'd need divulibre-bin, pyquery and PIL. 
[23:04:08] <alexbrollo>	 (this set is what is needed to explore djvu text layer, abbyy.xml OCR output and source images used to get OCR)
[23:08:23] <alexbrollo>	 Opppsss... I see that PIL is already installed, I apologyze. 
[23:13:32] <chrismcmahon>	 greg-g: pretty sure I just knocked over the apaches on beta again.   nice to know *what*, would be good to know *why* for sure.  we assume it's an NFS problem for now I guess. 
[23:14:17] <^d>	 chrismcmahon: Oh fyi, all of the latest Cirrus & core changes for search are live on beta now, plus all the config updates.
[23:14:23] <^d>	 Nothing exploded \o/
[23:14:57] <chrismcmahon>	 thanks ^d, explosions are bad 
[23:15:15] <^d>	 Indeed :)
[23:16:03] <^d>	 So now, if you want to compare the old and new search engine, you can add ?backend=LuceneSearch|CirrusSearch on Special:Search, and for list=search in the API you can add ?srbackend=LuceneSearch|CirrusSearch
[23:16:12] <^d>	 Should make spotting bugs between the systems way easier.
[23:16:26] <chrismcmahon>	 ^d: I'm planning (hoping) to have a cross-browser build of Nik's search tests running vs. beta in Jenkins by the end of the week.  will know more tomorrow. 
[23:16:41] <^d>	 Sweet.
[23:43:16] <alexbrollo>	 \q
[23:43:22] <alexbrollo>	 \quit