[00:00:04] yuvipanda, scfc_de ^? [00:33:27] Raylton: I don't know, you may have to ask on #mediawiki or one of the MediaWiki mailing lists. [00:34:53] ok scfc_de thank you :) [04:36:36] I want to submit a script to job schedule based on the information i found (https://wikitech.wikimedia.org/wiki/User:Legoktm/pywikibot_on_tools_lab) I have to push the code to /data/project does this mean i can't submit a script which is in $home to job schedular to be executed....? [04:45:02] auduwage: You should be able to do exactly the same as your normal user account and your /home/... directory, i. e. without "become tooluser". [04:47:08] scfc_de: Thanks I ll try that and see. [05:26:32] How can I check the status of the job that I submit..? [05:40:05] So here I get a problem [05:40:07] After using ssh-add, I try to "ssh -A justincheng12345@bastion.wmflabs.org" [05:40:09] It says authenticity of host cannot be established, as permission denied(publickey) [05:40:10] What should I do? [07:58:32] Is it possible to change my "Full Name" on gerrit? [09:06:52] (03PS1) 10Yuvipanda: Added README [labs/tools/SuchABot] - 10https://gerrit.wikimedia.org/r/78803 [09:06:55] (03PS1) 10Yuvipanda: Change default password in config sample to something non obvious [labs/tools/SuchABot] - 10https://gerrit.wikimedia.org/r/78804 [09:08:28] (03CR) 10Yuvipanda: [C: 032 V: 032] Added README [labs/tools/SuchABot] - 10https://gerrit.wikimedia.org/r/78803 (owner: 10Yuvipanda) [09:08:43] (03CR) 10Yuvipanda: [C: 032 V: 032] "Whoops" [labs/tools/SuchABot] - 10https://gerrit.wikimedia.org/r/78804 (owner: 10Yuvipanda) [09:27:01] [bz] (8NEW - created by: 2Magnus Manske, priority: 4Unprioritized - 6normal) [Bug 52744] cawiki sql bug - https://bugzilla.wikimedia.org/show_bug.cgi?id=52744 [09:46:01] [bz] (8PATCH_TO_REVIEW - created by: 2Daniel Schwen, priority: 4Unprioritized - 6enhancement) [Bug 52717] libvips-tools, libtiff etc install - https://bugzilla.wikimedia.org/show_bug.cgi?id=52717 [09:58:25] YuviPanda your bot is using some invalid symbols in end of url [09:58:31] (char)003 [09:58:37] irc colors [09:58:42] ask azatoth [09:58:48] it's typicaly terminator of mirc colors [09:58:59] but on place where you have it it's invalid and may break stuff [09:59:04] works for me [09:59:22] also I didn't touch the color code at all, it is all AzaToth [09:59:22] so [09:59:30] for you maybe, but it's still invalid... invalid html works in most browsers too ;) [09:59:44] ok :) [09:59:51] patches welcome! [10:00:01] link? [10:00:21] I don't know javascript I am just reporting a bug [10:00:36] Danny_B: to the bot? [10:00:52] yup [10:00:56] Danny_B: https://git.wikimedia.org/summary/labs%2Ftools%2Fgrrrit [10:01:06] there's a wikitech link too [10:01:16] http://wikitech.wikimedia.org/wiki/Grrrit-wm [10:02:13] petan: mne to funguje normalne, asi je chyba v tvem superduperhyper vlastnim klientovi ;-) [10:02:41] Danny_B me to funguje taky, ale tohle vidim v debug outputu, je to proste spatne vim jak ta syntaxe ma byt [10:03:33] it's like typing hello world in html, it will work in most browsers but every html checker will tell you warning on that [10:03:39] normalne url, ^C [10:14:09] petan: varnish is broen on beta [10:14:31] *broken [10:14:32] can't fix now... ask hashar [10:14:37] ok [13:11:49] Danny_B: *ping* [14:53:48] just found that "russblau" has broken the public_html directory in my tool /data/project/geograph2commons [14:53:57] Anyone know what's up with that? [14:55:40] Anyone here? Got a question related to submitting jobs in tool-labs [14:58:19] auduwage: Waiting for an answer myself. petan? [14:59:04] magnus___: while waiting..? would you know how to check the status of the job? [14:59:11] petan isn't suppose to be back until next week I believe. [15:00:01] petan isn't suppose to be back until next week I believe. [15:01:44] Technical_13: Do you know who labs user russblau is? [15:04:44] I don't [15:21:51] petan: around? [15:24:23] Coren|HKG: [15:24:26] ? :O [15:27:03] when someone comes around check the list :> [15:28:59] addshore: what's up with the list? [15:29:00] * YuviPanda checks [15:29:35] hmm [15:29:38] that's super weird [15:29:48] indeed [15:29:59] im guessing its something to do with the svn group [15:30:39] super odd though O_o [15:30:52] its like svn jsut said.... SCREW YOU to file permissions :P [15:40:15] anyone know why beta labs is mostly down? I don't see anything particularly wrong in the logs e.g. http://en.wikipedia.beta.wmflabs.org/wiki/0.546002634038912 [15:42:36] chrismcmahon: seems to load for me [15:43:14] addshore: yes, sometimes it does, other times I get 503 for any given page. caching I'd guess. [15:46:28] rather odd, ! [15:53:39] Ryan_Lane: around? :) [15:56:37] anyway * points you at the mailing list * [16:16:44] chrismcmahon: upload wizard is broken for me, who can help me? [16:17:08] matanya: broken on what host? [16:17:21] live commons [16:17:39] can't proceed the upload phase [16:19:10] chrismcmahon: more accurate would be the description phase [16:20:45] matanya: I don't think we've updated any production systems in over a week since most of the staff are in Hong Kong for Wikimania [16:21:09] chrismcmahon: i know, i have the issue for about a week [16:21:29] something in recent update maybe [16:21:47] i would like to know how to debug before creating a bz entry [16:22:02] matanya: I'd say go ahead and do the bz entry [16:22:04] so i can help finding the cause (if any) [16:22:32] thanks chrismcmahon [16:50:15] Can someone explain why clicking on the username in https://wikitech.wikimedia.org/w/index.php?title=User:Tarawneh&action=history shows *no* contributions, i. e. not even the one the history lists?! [16:52:50] becous it is a import [16:53:07] ohno O_O [16:55:27] Steinsplitter: Ah, "2007-10-22T22:56:46‎" -- missed that. But shouldn't that show up in the import log? [16:55:27] Ryan_Lane: ^ [16:56:52] no. in teh import log is only schown teh aricel/template... [16:57:13] *the [17:00:24] manybubbles: around? [17:01:46] yeah! [17:03:57] manybubbles: I know you've been doing stuff on beta labs recently. Any idea why pages are mostly 503 there? I just now restarted varnish but that doesn't seem to have helped. [17:04:27] chrismcmahon: no good idea but I'll look around [17:05:29] manybubbles: restarted varnish on deployment-cache-text1 that is [17:06:01] chrismcmahon: fyi: the varnish problem exists ca. one week [17:06:09] Steinsplitter: yes [17:07:56] chrismcmahon: the logs imply that the environment is hanging which is what I've noticed - [17:08:07] stuff works fine for a while then just spins forever and crashes [17:08:12] a reload and it is working again [17:08:58] I see the same kind of problems on my sandbox from time to time. I tend to blame the virtualization infrastructure. [17:09:06] manybubbles: "the environment"? Antoine mentioned in a bug somewhere restarted apache on I think deployment-apache32 [17:10:05] chrismcmahon: by "the environment" I mean the machine running apache rather than php or apache. [17:10:27] chrismcmahon: on my sandbox I sometime can't open files at all [17:11:04] manybubbles: beta labs really shouldn't behave like this :) [17:11:35] chrismcmahon: I'd prefer nothing behave like that :) let me poke around a bit more and then go bother -operations. [17:12:04] manybubbles: yep, after restarting varnish and apache32, I am out of ideas [17:12:25] * ^d notices discussion [17:12:28] * ^d reads scrollback [17:12:34] chrismcmahon: it seems to be working now. [17:12:36] hi ^d [17:13:02] chrismcmahon: just sometimes it flakes out hardcore and everything waits or crashes [17:13:25] manybubbles: from what I see, it sometimes actually dtrt but mostly doesn't [17:13:52] manybubbles: try running your browser tests against beta labs for instance [17:14:14] one mement [17:14:34] ok logging out of deployment-bastion just hung [17:14:53] that is lame. it came back after a few seconds but :( [17:16:13] yeah. I don't even see anything particularly alarming in the logs, just mostly 503 when trying to get pages [17:16:22] chrismcmahon: just started my tests and immediately a request hung [17:17:18] chrismcmahon: apache-error.log has some nasty messages about processes not dying [17:18:06] chrismcmahon: this doesn't look good: [error] server reached MaxClients setting, consider raising the MaxClients setting [17:18:37] * chrismcmahon looks [17:19:46] looks like someone just bounced it [17:21:05] hmm, manybubbles I think that's normal(ish). [17:21:43] crashed again [17:23:13] ^d: any suggestions? beta labs has been just about unusable for a while now. [17:23:13] labs ganglia is broken too [17:23:22] saw that [17:23:25] :-( [17:24:08] Mulilock work quickly, but Special:Nuke is unusable [17:24:18] <^d> I'd check what MaxClients is set to, and compare that to the amount of traffic we're actually serving. [17:25:00] <^d> If that's stuff is setup using ubuntu/debian defaults, we could probably be hitting some those limits. [17:25:06] <^d> That would very likely cause the 503s. [17:25:44] now deployment-bastion has stopped echoing my keystrokes. [17:26:58] I think these freezes correspond with pages crashing [17:28:07] <^d> Ok, so -bastion shouldn't be affected by pages crashing. [17:30:00] ^d: I think we just have rolling freezes. I see them on solr-mw2 from time to time too [17:32:47] If I could get at labs ganglia I might be able to learn more. [17:33:04] yep [17:33:09] deployment-apache32 seems to be coming off a load high. [17:33:56] that does seem odd though. I can't imagine it serves a whole lot of traffic, and it's been behaving like this for a number of days now [17:37:03] the five and 15 minute load average figures were 9 when I first logged in. the one minutes was .5 [17:38:13] <^d> manybubbles: Totally off-topic, but I've just finished reviewing+merging all the open CirrusStuff other than the large regression tests for highlighting patch :) [17:38:29] <^d> *CirrusSearch stuff [17:39:39] ^d manybubbles I mostly want beta labs back to run those search tests (and work on removing some conditionals in them) [17:44:06] scfc_de: seen the mailing list email ? :) [17:44:23] addshore: Responding now. [17:44:26] :) [17:44:31] just checking :) [17:46:17] see yall in the morning [17:47:49] ^d: thanks! [17:50:57] and crashed again... [17:52:57] chrismcmahon: Does beta use NFS? [17:59:31] scfc_de: yes [18:01:07] manybubbles: So the stalls may be related to that? [18:03:32] scfc_de: I've noticed stalls just echoing back keys so I don't think we can blame all of it on nfs. [18:04:31] scfc_de: otoh, /usr/local/apache is in nfs [18:08:04] BTW, has there been any resolution to the NFS issues in the HKG all-ops meeting? [18:12:57] scfc_de: not sure [18:13:14] something just hit that knocked out my sandbox, deployment-bastion, and deployment-apache32 [18:17:52] ouch [18:25:36] ^d: looks like beta isn't pulling new versions of CirrusSearch [18:25:55] the last commit is has was on wednesday [18:26:04] maybe it just hasn't gotten around to it [18:27:54] <^d> aw :( [18:28:32] geez [18:35:32] ^d: do you know how that works? It looks like the clone isn't clean. Do you think it'd hurt anything if I `git clean`ed it? [18:35:53] <^d> Go for it. [18:42:06] manybubbles: it might not be obvious, I'd like CirrusSearch running reliably on beta labs so that I can make a Jenkins build for it and run it for all the browsers, like we have for Mobile and are about to do for Language. [18:42:32] chrismcmahon: cool! I'd like labs reliable at all.... [18:43:10] manybubbles: if we can't figure it out today, I think Antoine is back tomorrow. /me makes a note to wake up early. :-) [18:45:15] !log deployment-prep unstuck CirrusSearch so it'd update. [18:45:18] Logged the message, Master [18:45:28] !log deployment-prep rebuilding all search indecies using updates [18:45:31] Logged the message, Master [18:54:18] strange, i cannot edit the deployment [[Main Page]] o_O [18:54:31] only deleting o_O [18:58:19] now that ganglia's back you can see that apache32 has just frozen [18:58:48] and it came back and ganglia seems to have smoothed over the empty times.... [19:00:29] looks like it freezes so hard it doesn't record its problems in ganglia [19:10:35] !log deployment-prep rebuild search indecies for updates [19:10:38] Logged the message, Master [20:05:09] it is like the whole labs network just locks up every once in a while. [20:05:25] I try `git diff` and it just hangs. [20:05:58] or I ssh to a machine and I get the bastion banner and it sits for many minutes doing nothing [20:47:24] manybubbles: To me, that's an NFS stall (on instances that use NFS). The shell/Git/whatever tries to access various .rc files, etc., and that means they lock up. [23:05:39] ^d: how close do you think https://gerrit.wikimedia.org/r/#/c/78083 is to being able to be deployed on say test2wiki? just curious. [23:06:18] <^d> chrismcmahon: I wasn't aware we were deploying to test2wiki? [23:06:44] <^d> In any case, it needs the core change to be truly useful. [23:07:03] ^d me neither, but that's what I thought of when I read the change. I'm just annoyed that I haven't been able to fix beta. [23:07:28] <^d> This won't affect beta either? :) [23:09:07] ^d: I'd just like to see CirrusSearch working nicely in some shared test env. any shared test env.