[00:00:35] Ryan_Lane: Nah just ProfilerSimpleText [00:00:39] petan: ughhhhh [00:00:40] ah ok [00:00:46] we can switch over to the UDP one later [00:00:54] that's thankfully puppetized, from the daemon perspective [00:00:55] Ryan_Lane, when are you getting virt-1? [00:01:01] Reedy: eh? [00:01:05] * johnduhart runs git reset --soft [00:01:05] ah [00:01:06] heh [00:01:07] :p [00:02:27] Reedy: I'm using virt0 as the controller [00:02:33] and all nodes past that are compute [00:15:17] * Ryan_Lane groans [00:15:24] this is going to be a PITA [00:15:44] * Ryan_Lane checks whois [00:16:03] Name Server:VIRT1.WIKIMEDIA.ORG [00:16:03] Name Server:LABSCONSOLE.WIKIMEDIA.ORG [00:16:05] * Ryan_Lane sighs [00:16:25] fail [00:16:43] well, thankfully the secondary is acceptable [00:32:52] I'm switching the controller.... [00:33:06] there's a possibility during this process I'll fuck up, and we'll have some downtime [00:34:15] no matter what we're going to have some labsconsole downtime [00:34:24] at least from the perspective of modifications [00:48:17] Ryan_Lane: sorry i may have missed a reply yesterday [00:48:25] is there any news about my Labs account? [00:49:12] you didn't provide me with all info [00:49:42] !account-questions | philinje [00:49:42] philinje: I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your SVN account name, or your preferred shell account name, if you do not have SVN access. 3. Your preferred email address. [00:50:59] 1. philinje 2. philinje 3. pchang@wikimedia.org [00:54:38] !initial-login | philinje [00:54:38] philinje: https://labsconsole.wikimedia.org/wiki/Access#Initial_log_in [01:17:53] dear opendj, you just scared the hell out of me: Initializing schema on server virt1.wikimedia.org:4444 with the contents of server virt0.wikimedia.org:4444 ..... Done. [01:17:59] I didn't see "schema" [01:18:07] I thought it was wiping out all the data. heh [01:18:33] well, ldap server is done [01:26:28] Ryan_Lane1: thanks [01:26:33] yw [02:23:22] mysql done [03:04:55] Ryan_Lane: wy no labs? [03:05:02] whats' broken? [03:05:36] hexmode: ?? [03:05:49] Ryan_Lane: I can get to google but not deployment [03:06:05] hm [03:06:07] crap [03:06:10] is dns broken? [03:06:27] "ssh: Could not resolve hostname bastion.wmflabs.org: Name or service not known" [03:06:48] pdns on virt1 isn't responding.... [03:07:20] hmm. wtf [03:08:06] ah. I know [03:08:24] one quick fix. [03:08:31] ok. working now [03:09:32] well. it should be anyway [03:10:07] yeah. it is [03:12:36] ok. did the permanent fix too [03:13:00] and to think I thought I was going to do this with no downtime! [06:17:54] Ryan_Lane: mysql is broke? [06:18:00] * Ryan_Lane groans [06:18:15] I moved all services from virt1 onto virt0 [06:18:32] i saw :) [06:18:46] seems something is wrong with apache or mysql [06:18:51] but that was 3 hours ago. and mysql brokenness was discussed [06:18:59] Ryan_Lane: i think it's missing grant [06:19:03] it was [06:19:22] i mean what i'm getting now is missing grant [06:19:23] it's taking ages to load right now [06:19:26] or wrong password [06:19:35] I'm not anymore. I just added it [06:19:38] but it's timing out now [06:19:52] memcached not running or wrong port/address? [06:20:10] (we had that problem with deployment-prep i think) [06:20:13] did you just connect? [06:20:21] that was DNS a little while ago [06:20:35] for a short period of time. [06:20:56] hmm. It's loading, but it's taking a really long time... [06:21:11] agreed, long time [06:21:53] wtf. [06:23:30] doesn't look like this recovered per my channel buffer ----> 13 02:21:16 <+nagios-wm> PROBLEM - Memcached on virt0 is CRITICAL: Connection refused [06:25:17] I can connect to it on localhost [06:25:27] and that's how mediawiki is accessing it [06:26:14] spence is slow... [06:27:25] found it [06:27:41] seems memcached used a different port on virt1 [06:27:50] mediawiki was improperly configured [06:28:26] this one is using the cluster's port, the other was using the default for some reason. [06:28:27] sooo fast [06:28:41] heh. yeah. way better now [06:29:15] there's some issue with the zones now too [06:29:25] when I added a new scheduler it added a new zone [06:29:32] scheduler? [06:29:37] the correct zone, but a new one [06:29:44] nova-scheduler service [06:30:06] it chooses which compute node is going to run an instance [06:30:18] what's this? word missing? ----> the correct zone, but a new one [06:30:23] zones are different clusters of compute-nodes [06:30:40] now we have nova and pmtpa [06:30:54] what's the diff? [06:31:15] it was only nova before [06:31:28] though I did have the zone listed in the config as pmtpa for all of the nodes [06:31:48] the scheduler we are using is a single zone controller, so it's odd [06:31:53] it's a bug, likely [06:32:10] now to see if an instance will create in both zones and still work [06:32:16] what did it add? [06:32:35] pmtpa [06:32:49] which is what I wanted all along :D [06:32:55] the sidebar's not right on labsconsole [06:33:24] huh, fixed [06:33:34] I'm not sure what the deal with that is. I changed the version of vector to a tagged version for 1.18 [06:41:04] nacht [06:49:07] ugh, for some reason the floating ip for bastion got disassociated [07:29:37] Change on 12mediawiki a page Wikimedia Labs was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=484775 edit summary: /* Proposals */ [07:35:41] Change on 12mediawiki a page Wikimedia Labs was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=484776 edit summary: /* Proposals */ [07:57:07] 01/13/2012 - 07:57:07 - Updating keys for beetstra [07:57:09] 01/13/2012 - 07:57:09 - Updating keys for beetstra [07:58:07] 01/13/2012 - 07:58:07 - Updating keys for beetstra [07:58:09] 01/13/2012 - 07:58:09 - Updating keys for beetstra [07:59:07] 01/13/2012 - 07:59:06 - Updating keys for beetstra [07:59:09] 01/13/2012 - 07:59:08 - Updating keys for beetstra [08:00:14] :-) [08:35:31] Is there someone from the bots-group around? [08:36:01] I need some help about passwords used by a bot [08:42:22] !log bots Installed XLinkBot on bots-2, running properly [08:42:23] Logged the message, Master [08:47:32] PROBLEM host: deployment-test is DOWN address: deployment-test check_ping: Invalid hostname/address - deployment-test [09:18:12] PROBLEM host: deployment-test is DOWN address: deployment-test check_ping: Invalid hostname/address - deployment-test [09:21:44] !logs [09:27:33] !sal [09:27:33] https://labsconsole.wikimedia.org/wiki/Server_Admin_Log see it and you will know all you need [09:28:20] ok. time for bed. way too tired to do anything correctly [11:51:34] !logs [11:52:24] Can someone help setting up mysql for my perl bots on bots-2 ?? [11:52:56] I can't sudo sucessfully on deployment-wmsearch [11:53:06] any suggestions [11:58:24] Beetstra: SQL shouldn't be installed directly on that server [11:58:56] that is fine, so I have to connect to an 'external' server [11:59:27] that is, i.s.o. localhost, something else in the connect-statement [11:59:45] But I would need to set up tables somewhere, and then to connect to them ... don't know how [12:10:59] :o [12:11:02] Beetstra: hey [12:11:12] Hi! [12:11:24] Got the first bots running, perl is running fine there!! [12:11:28] I know [12:11:32] :-) [12:11:35] how many mb you need for your db [12:11:41] hehehehe [12:11:53] hm? [12:12:05] At first, I will start from scratch, just testing, so maybe a Gb to start with [12:12:25] ok I will create you an account on bots-sql1 with create [12:12:27] In the end, when I transfer the existing tables, it will be in the order of 100 Gb already [12:12:30] petan|w: I have permission issues on mwsearch [12:12:44] OrenBo: like? [12:12:53] cant su [12:12:53] Beetstra: want a service account too? [12:12:55] for bot [12:12:59] OrenBo: use sudo [12:13:00] no su [12:13:02] cant make dirs [12:13:05] with sudo [12:13:06] OrenBo: where [12:13:10] sec [12:13:16] OrenBo: because there is no root squash [12:13:19] petan|w .. what do I do with a service account? [12:13:25] Beetstra: for boy [12:13:28] * bot [12:13:41] Beetstra: so bot uses own acc for mysql [12:14:01] ehm .. the bots access each others tables ... [12:14:03] OrenBo: tell me which directory you need to create [12:14:17] But that is a good idea, as long as they can get to each others tables [12:14:25] sure [12:14:43] Beetstra: I will create a root account for you and standart acc [12:14:44] ok? [12:14:48] petan: botnames - linkwatcher, coibot, monitorbot [12:15:00] you want a maria or mysql? [12:15:02] OK, sounds good [12:15:07] oren@deployment-wmsearch:/usr/local/search/ls2$ sudo mkdir bin [12:15:08] mkdir: cannot create directory `bin': Permission denied [12:15:09] the bots are used to mysql [12:15:54] OrenBo: again :) [12:15:57] OrenBo: fixed [12:16:13] Beetstra: maria is same as mysql just a better version [12:16:32] OK, use the better one [12:16:34] :-D [12:16:41] any idea why ? [12:16:49] Enough troubles already with the old MySQL on the box they are running on atm [12:16:50] Beetstra: ok see bots.wmflabs.org/phpmyadmin [12:17:13] OrenBo: yes permissions were wrong [12:17:18] OrenBo: fixed [12:17:26] petan: not found .. [12:17:37] ah, wait [12:18:04] Beetstra: http://bots.wmflabs.org/phpmyadmin/ [12:18:23] OrenBo: anything else? [12:18:25] petan|w is there a jdk or just a jre on mwsearch ? [12:18:33] I don't know [12:18:36] checking [12:18:43] I get can't find tools.jar [12:20:15] petan .. error #1045 Cannot log in to the MySQL server [12:20:27] I presume it uses my (lowercase) username and standard password? [12:20:35] * Beetstra sees PM [12:20:55] no there is no ldap yet [12:21:05] we work on that with Ryan [12:21:29] I was looking at the path you supplied - there was no localsetting.php there [12:21:48] no there is wmf-config/CommonSettings.oho [12:21:49] php [12:21:54] we don't have LS.php on prod [12:21:55] in /usr/local/apache/conf [12:22:24] usr/local/apache/common/wmf-config [12:22:39] why not, how does it work? [12:22:41] before you make any change there let me know [12:22:47] OrenBo: all wikis use same config [12:22:52] check that file [12:22:58] ok [12:23:01] we have almost 1000 wikis on production [12:23:13] using a separate LS.php would be really hard [12:24:57] OrenBo: let me know if you needed me [12:25:04] ok [12:25:09] I am in work right now so probably won't be around quick [12:25:36] I'm trying evrything again [12:25:42] should take about a minute [12:25:46] OrenBo: use etherpad [12:25:47] ok [12:26:58] I stll have issue with tools.jar not found in jdk [12:27:04] ok [12:27:27] JavaHome point to a jre [12:35:40] OrenBo: I installed tomcat [12:35:46] that's probably what you were missing [12:37:09] why tomacat ? [12:38:02] Beetstra: installed [12:38:10] OrenBo: it was on search or not? [12:38:31] OrenBo: what libs are you missing, I thought you want all stuff you had on search [12:39:02] I got libs [12:39:06] ok [12:39:09] this jar is part of the jdk [12:39:10] so it works? [12:39:13] no [12:39:19] hm... I can't find jdk in repos [12:39:24] only jdk emacs [12:39:27] :O [12:39:31] :-( [12:39:52] jeremyb: ping [12:40:01] I remember we fixed it on search-test [12:40:30] !sal [12:40:30] https://labsconsole.wikimedia.org/wiki/Server_Admin_Log see it and you will know all you need [12:41:13] how do I get the value of the JAVA_HOME environment var [12:41:28] OrenBo: there is no log [12:41:34] regarding jeremy's setup [12:42:08] env | grep 'java_home' [12:42:56] * JAVA_HOME [12:43:00] it's case sensitive [12:43:53] env | grep 'JAVA_HOME=' | sed 's/HOME=//' [12:43:59] if you onl need the value [12:44:25] OrenBo: I installed some other try now [12:45:25] OrenBo: hm? [12:45:42] sec [12:45:43] what all tools you need to set up :o [12:46:11] good [12:46:50] thanks [12:47:27] ok [12:47:57] can you get junit 3.x into lob [12:48:00] can you get junit 3.x into lib [12:48:18] hm? [12:49:04] sure [12:49:11] installed [13:01:17] I think the btw jdk is not good :-) I run stuff on wrong machine [13:01:45] i.e. still not tools.jar in jdk [13:04:22] sudo apt-get install openjdk-6-jdk [13:04:28] but it fials with [13:05:06] Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable) [13:05:10] E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it? [13:05:46] installing [13:05:47] done [13:07:06] ok [13:07:34] log that stuff [13:07:50] !log deployment-prep installed jdk on search [13:07:51] Logged the message, Master [13:08:23] ps test now build but 142 test fail [13:08:31] hm? [13:09:01] OrenBo: /usr/local/search/indexes [13:09:06] ?? [13:09:06] can I scp a file from my local machine [13:09:06] what is it [13:09:09] yes [13:09:24] !socks-proxy [13:09:25] see https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances ssh @bastion.wmflabs.org -D [13:10:02] hmm [13:10:05] complex [13:10:11] heh [13:10:23] OrenBo: there is /mnt/index [13:10:25] perhaps you can put the junit3 lib also into search-test instance [13:10:33] ok [13:12:09] petan|w: Can you please not blindly commit stuff https://github.com/johnduhart/deploymentprep-conf/commit/e04f979b5240e4260ba81e0aed4d20ac1b95eea6 [13:12:12] !log search installed junit 3.8 [13:12:12] Logged the message, Master [13:12:47] johnduhart: that wasn't from me [13:12:55] no idea who inserted it there [13:12:59] I probably commit -a [13:13:03] petan|w: I did, you committed it. [13:13:07] ah [13:13:13] petan|w: You need to check what's committed before you commit it [13:13:17] how [13:13:22] I don't know git [13:13:27] petan|w: You never added the file before you committed [13:13:28] git check? [13:13:33] * johnduhart sighs [13:13:36] I probably changed some [13:13:39] ok we have a problem [13:13:45] petan|w: Go read some git docs [13:13:53] if only I have a time [13:13:57] petan|w: http://progit.org/book/ [13:14:01] I know [13:14:04] git status is your friend [13:14:13] OrenBo: yes? [13:15:22] it realy want to find a LocalSettings.php [13:15:40] ok there is a LocalSettings but it doesn't include much [13:15:45] contain [13:15:50] it include CommonSettings [13:15:59] where is it [13:16:01] OrenBo: on prod there is no LS.php [13:16:16] OrenBo: /usr/local/apache/comm*/live [13:16:54] Talk to me about configuration [13:17:09] sec [13:19:57] johnduhart: 404 is working now [13:19:58] is CommonsSetting the same a LocalSettings just a different name [13:20:12] it's a replacement for that [13:20:15] or is it a different file format [13:20:23] it's a configuration using wgCong [13:20:25] Conf [13:20:27] for all wikis [13:20:35] check the source code [13:20:37] SiteConfiguration* [13:20:38] thanks [13:20:53] let me explain the issue [13:20:57] petan|w: Is it done how it is in production? [13:21:05] johnduhart: I don't have access on prod [13:23:52] it looks like it [13:23:56] but I don't know [13:24:39] actually I doubt there is a live folder on prod [13:24:45] just like /usr/local/apache [13:24:52] johnduhart: ^ [13:24:57] so probably it's not same [13:31:51] re [13:32:00] I had some buisneess to take care [13:33:27] search-2 looks for LocalSettings - its hard coded [13:33:58] I can add locations and tell it to look for CommonSettings [13:34:24] but the code for parsing the config is not in search - it is in MWdumper [13:34:30] another project [13:35:33] OrenBo: ask notpeter [13:35:39] production use Common [13:35:41] not Local [13:35:58] OrenBo: there is only one Local Settings and it's same for all wikis [13:36:23] which vars you want to get from it [13:41:31] btw OrenBo I don't want search to use db access which is being used by wiki [13:41:40] I would rather create a new acc [13:41:58] petan I don't follow you [13:42:09] you need to use sql? [13:42:17] I don't know [13:42:18] I don't want search accidentally damage db [13:42:26] so I wanted to create a new account for it [13:42:26] hmm [13:42:34] no need for it to be able to write to db [13:42:39] wiki tables [13:42:57] I am looking through the spagetti code [13:43:01] ok [13:43:54] I also don't have access porduction [13:44:14] I have no idea if it uses dbase access [13:44:19] hm [13:44:25] I don't know what it needs LS for [13:44:25] I'm 95% sure that it does [13:44:34] righjt [13:44:56] I am pretty sure I know what it uses it for [13:45:07] hm? [13:45:12] LS that is [13:45:35] give me a minute to open the source files [13:45:42] ok [13:47:25] hexmode: you still wait with announcement or not? why is that section what needs to be done missing in ep :o [13:48:33] johnduhart: LQT is working [13:48:34] ? [13:48:55] what about uw [13:49:15] ah, that is [13:57:37] the configuration is quite a hack - it has a java method that executes a bash script to call a php script [14:01:51] can I get a map to the maintenance folder? [14:07:27] hm... [14:07:33] map? [14:12:48] Hello all [14:13:11] Anyone awake? [14:17:31] make a NFS share to the mainenece folder [14:24:29] I added a problem report: Uncaught TypeError: Object function (selector,context){return new jQuery.fn.init(selector,context,rootjQuery);} has no method 'cookie' [14:24:44] I'm trying to code a new config class to woek with a common config [14:24:51] I'm trying to code a new config class to work with a common config [14:25:17] what instance is the common config on [14:25:20] ? [14:25:49] enwiki [14:27:59] Edokter: hm... [14:28:09] OrenBo: everywhere [14:28:17] OrenBo: /usr/local/apache/common [14:28:26] Edokter: thanks for report [14:39:19] to paraphrase - which deployment host has mediawiki installation + common.conf [14:39:45] I need to do some checks [14:40:04] deployment-web ? [14:43:42] what should I be looking for the maintence dir with eval.php [14:48:23] hi all, who can help Andre Engels to give him the rights to upload his public ssh key to https://labsconsole.wikimedia.org/wiki/Special:NovaKey [14:48:42] or is this something that only Ryan can help with/ [14:49:51] !logs created databases and accounts for coibot, xlinkbot, monitorbot on bots-sql3 for bots on bots-2 [14:51:40] drdee_: generally he should be able to do that [14:51:53]