[00:03:50] init: nss-ldap: do_open: do_start_tls failed:stat=-1 [00:03:50] [00:03:50] init: nss_ldap: could not search LDAP server - Server is unavailable [00:03:50] [00:03:50] Console output to bots-irc2 [00:04:02] RECOVERY host: bots-irc2 is UP address: bots-irc2 PING OK - Packet loss = 0%, RTA = 4.07 ms [00:04:20] !logging :o [00:04:20] To log a message, use the following format: !log [00:04:22] do that [00:04:23] :D [00:04:30] I have no idea what's going on [00:04:43] methecooldude: ^ [00:05:26] !log bots LDAP issue on bots-irc2 > "init: nss-ldap: do_open: do_start_tls failed:stat=-1" and "init: nss_ldap: could not search LDAP server - Server is unavailable" [00:05:27] Logged the message, Master [00:05:50] meh why it exist? [00:05:54] I thought we deleted irc2? [00:06:00] jeremyb: ^ [00:06:59] methecooldude: I think that best solution for now would be to delete it, but I don't know who created it nor why... [00:07:06] ask jeremy [00:07:30] * jeremyb clicks delete finally [00:07:55] :D [00:08:15] it's too bad when you delete then all the details are gone [00:08:31] !log bots deleted irc2 [00:08:32] Logged the message, Master [00:08:41] would be better to just have a semnatic option to indicate it's deleted [00:08:52] s/option/property/ [00:08:52] hm... [00:08:57] that's true [00:09:19] otoh allow everyone (even anons?) to see deleted pages [00:09:27] also a good idea [01:25:32] petan: Would you mind adding me to the bots project? Interested in seeing what's going on and might want to move some of my stuff over [03:17:53] PROBLEM Disk Space is now: WARNING on nova-production1 nova-production1 output: DISK WARNING - free space: / 565 MB (5% inode=86%): [06:01:07] 01/04/2012 - 06:01:07 - Creating a home directory for johnduhart at /export/home/bots/johnduhart [06:02:08] 01/04/2012 - 06:02:08 - Updating keys for johnduhart [06:17:53] PROBLEM Disk Space is now: CRITICAL on nova-production1 nova-production1 output: DISK CRITICAL - free space: / 279 MB (2% inode=86%): [10:44:03] Change on 12mediawiki a page Wikimedia Labs/Create shared sql service for all projects was modified, changed by Petrb link https://www.mediawiki.org/w/index.php?diff=480900 edit summary: [15:55:13] 01/04/2012 - 15:55:13 - Creating a project directory for deployment-prep [15:56:23] shoulda been here already [15:56:23] !labs [15:56:23] https://labsconsole.wikimedia.org/wiki/$1 [15:56:35] !labs Access [15:56:35] https://labsconsole.wikimedia.org/wiki/Access [15:56:49] that's a console [15:57:24] arr, i thought i was on this channel already:) [15:57:27] wrong logfile now [15:57:33] !log added project deployment-prep for hexmode and petan [15:57:34] added is not a valid project. [15:57:41] !logging [15:57:41] To log a message, use the following format: !log [15:57:49] what is the global project? [15:57:54] no idea [15:57:55] for news like "added project":) [15:58:02] !sal [15:58:02] https://labsconsole.wikimedia.org/wiki/Server_Admin_Log see it and you will know all you need [15:58:10] maybe there is a log where Ryan log this [15:58:31] dunno [15:59:32] btw you didn't insert us to project :) [15:59:39] https://labsconsole.wikimedia.org/wiki/Special:NovaProject [15:59:51] hexmode: you know how to use instances? [16:00:07] !socks-proxy [16:00:07] see https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances ssh @bastion.wmflabs.org -D [16:00:32] petan|work: no... This'll be my first time on labs [16:00:36] ok [16:00:41] there is a login server bastion [16:00:52] from there you can ssh to instances [16:01:05] hexmode: so you need a wiki user on labsconsole.wm [16:01:15] you ssh using key, for sudo you use password, it's not so hard [16:01:21] hexmode: then i can add you to the new project [16:01:54] mutante: MarkAHershberger is on there already ;) [16:01:59] ah [16:02:00] o.0 it's busy in here [16:02:05] hey [16:02:06] :) [16:02:43] mutante: my user is Petrb [16:02:44] !log deployment-prep added new project deployment-prep for hexmode and petan [16:02:46] Logged the message, Master [16:02:58] :) thats the way:) [16:03:00] yes that's probably how to log that :) [16:04:13] 01/04/2012 - 16:04:13 - Creating a home directory for mah at /export/home/deployment-prep/mah [16:04:26] :) [16:04:27] hexmode: I think I will create a web server (small instance) and 80 gb sql instance we would use until we get a better server [16:04:42] that would be enough even for full simple wiki [16:04:50] i was about to say "added users", but the bot already tells you:) [16:04:56] petan|work: sure lets do that [16:04:59] ok [16:05:14] 01/04/2012 - 16:05:13 - Updating keys for mah [16:05:14] 01/04/2012 - 16:05:13 - Creating a home directory for petrb at /export/home/deployment-prep/petrb [16:05:53] I need netadmin too :) [16:05:59] to create instances and Mark too [16:06:16] 01/04/2012 - 16:06:16 - Updating keys for petrb [16:06:39] !log deployment-prep added members MarkAHershberger & Petrb - added them to sysadmin and netadmin roles [16:06:40] Logged the message, Master [16:07:59] !log deployment-prep configured firewall for webserver [16:08:00] Logged the message, Master [16:08:16] OrenBochman: did you get tidy yet? [16:08:38] btw OrenBochman import of simple wiki is nearly done, I will fix db now [16:08:54] ok [16:09:04] can you also put in tidy [16:09:26] it has loose html tags - can you install tidy to clean these up - e.g. bottom of /w/index.php/Italic [16:09:59] btw where can you see the import progress ? [16:10:09] !log deployment-prep create instances for apache and mysql [16:10:10] Logged the message, Master [16:10:15] on [[special:statistics]] [16:10:19] out of 300k [16:10:22] pages [16:10:36] petan|work: the first log line is missing on https://labsconsole.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL though [16:10:44] i'm doing tidy? [16:10:46] uh [16:10:47] petan|work [16:10:59] jeremyb: yes? [16:11:08] < jeremyb> i'm doing tidy?\ [16:11:15] you mean rebuildall.php? [16:11:26] because that's what we have to do now :) [16:11:31] no, wgUseTidy [16:11:33] I will check if import is finished [16:11:36] ah [16:11:40] OrenBochman asked for it [16:11:48] i just wanted to make sure you weren't doing [16:11:48] of course np [16:11:49] it [16:11:59] but I don't know if it's finished yet [16:12:05] import [16:12:55] ok hexmode we have the instances created I will now install all software there and start import of simple wiki and hiwiki latest rev, ok [16:13:02] huh, https://www.mediawiki.org/wiki/Manual:$wgAlwaysUseTidy [16:14:33] PROBLEM Current Users is now: CRITICAL on deployment-web deployment-web output: Connection refused by host [16:14:39] hexmode: https://labsconsole.wikimedia.org/wiki/Nova_Resource:Deployment-prep [16:15:06] mutante: petan|work: tyvm :) :) [16:15:13] PROBLEM Disk Space is now: CRITICAL on deployment-web deployment-web output: Connection refused by host [16:15:45] petan|work: can we do a full simple? [16:15:59] yes [16:16:03] PROBLEM Free ram is now: CRITICAL on deployment-web deployment-web output: Connection refused by host [16:16:23] latest hi would be today [16:16:30] full simple will probably take a day or two [16:16:44] sure, np [16:16:48] ok [16:17:22] hexmode: you can insert more people who are involved to project [16:17:23] if i get other people who want different wikis on there how hard will it be for me to add them? [16:17:23] PROBLEM Total Processes is now: CRITICAL on deployment-web deployment-web output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:17:34] easy :) [16:17:44] awesome [16:17:50] https://labsconsole.wikimedia.org/wiki/Special:NovaProject [16:18:06] there is add a member, you type a wiki name and it would insert them to project [16:18:13] PROBLEM dpkg-check is now: CRITICAL on deployment-web deployment-web output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:18:14] * hexmode imagines a button to "create a dupe of X wiki with X codebase" [16:18:31] ah, that's not so easy yet [16:18:43] but there is a proposal to create it [16:18:46] petan|work: make it so? :) [16:18:53] PROBLEM Current Load is now: CRITICAL on deployment-web deployment-web output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:19:13] http://www.mediawiki.org/wiki/Wikimedia_Labs [16:19:26] that's a summary of all what we are doing right now, and what is going to be done [16:19:58] in proposal on first line is a proposal to create automatic system where dev just pick db, branch and press enter and wiki is up [16:23:52] PROBLEM Current Load is now: CRITICAL on deployment-sql deployment-sql output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:24:32] PROBLEM Current Users is now: CRITICAL on deployment-sql deployment-sql output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:25:12] PROBLEM Disk Space is now: CRITICAL on deployment-sql deployment-sql output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:26:02] PROBLEM Free ram is now: CRITICAL on deployment-sql deployment-sql output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:27:22] PROBLEM Total Processes is now: CRITICAL on deployment-sql deployment-sql output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:28:12] PROBLEM dpkg-check is now: CRITICAL on deployment-sql deployment-sql output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:29:04] OSM needs to be updated [16:29:55] !log sesarch search-test: added Alias /wiki /var/www/w/index.php [16:29:56] sesarch is not a valid project. [16:30:01] !log search search-test: added Alias /wiki /var/www/w/index.php [16:30:02] Logged the message, Master [16:30:58] RECOVERY Free ram is now: OK on deployment-web deployment-web output: OK: 92% free memory [16:32:28] RECOVERY Total Processes is now: OK on deployment-web deployment-web output: PROCS OK: 87 processes [16:33:08] RECOVERY dpkg-check is now: OK on deployment-web deployment-web output: All packages OK [16:33:48] RECOVERY Current Load is now: OK on deployment-web deployment-web output: OK - load average: 0.10, 0.28, 0.26 [16:34:38] RECOVERY Current Users is now: OK on deployment-web deployment-web output: USERS OK - 1 users currently logged in [16:35:18] RECOVERY Disk Space is now: OK on deployment-web deployment-web output: DISK OK [16:35:20] !log search search-test: set $wgArticlePath = "/wiki/$1"; $wgUsePathInfo = true; [16:35:21] Logged the message, Master [16:38:03] !log search search-test: set $wgUseTidy = true; [16:38:04] Logged the message, Master [16:38:07] OrenBochman: ping? [16:38:26] hey [16:38:41] so, when i flip the switch i see a difference in the page [16:38:52] so i think it's done. but let me know what you think [16:39:00] i have to run very shortly [16:39:13] it is good now [16:39:22] thanks! [16:39:36] sure [16:39:40] my vacation is over tommorow [16:39:51] I'll be in touch as time allows [16:43:48] sure. i'll work on it some too [16:44:13] is there some other way that's easier to get you besides IRC? [16:45:38] email [16:46:05] my gmail OrenBochman at Gmail.com [16:47:03] petan|work: What's this deployment-prep stuff we have here? :) [16:47:23] hexmode wants to try 1.19 there on production db [16:48:26] johnduhart: If you have ideas on how to follow up on that wikitech-l post I made about this, I'm all ears! [16:48:42] oh [16:48:51] Didn't get the email, one sec [16:49:13] I go to school in a underground bunker that doens't get cell signal [16:49:38] johnduhart: sent it a couple of days ago [16:50:00] oh that one [16:50:23] well, I thought I did... [16:50:31] * hexmode doesn't see it right off [16:50:37] sounds reasonable until testlabs is working [16:51:05] http://thread.gmane.org/gmane.science.linguistics.wikipedia.technical/57950/focus=57969 [16:51:43] Your organization has chosen to limit viewing of this site (http://thread.gmane.org/gmane.science.linguistics.wikipedia.technical/57950/focus=57969), due to the rating of its content (sharewares download,hobbies/interest,web host). [16:51:46] fsck [16:51:53] hehe [16:52:05] I'm literallly loling [16:52:15] hahahahaha [16:52:19] who dat? [16:52:21] heh [16:52:21] that is pretty lame for mailing list archives [16:52:28] johnduhart: ssh proxy? :P [16:52:52] I have firewalled everything apart of port I need [16:52:54] petan|work: too lazy :p [16:53:04] and that one port is enough for me to get everywhere :D [16:53:21] in work [16:55:13] oh crap I have a CCNA test to do this period [17:01:16] oh, that must have been at school. [17:01:19] * jeremyb runs away [17:42:58] back [18:17:49] hexmode: I will wait for Ryan with import of simple wiki, I need more space [18:18:05] I need temporary 200 gb to extract dump there [18:19:52] moar space! [18:20:19] you know if I can import .18 db there? [18:20:26] I don't know if it's compatible or not [18:20:40] maybe I should install 1.18 and update it? [18:22:30] !rt is http://rt.wikimedia.org/Ticket/Display.html?id=$1 [18:22:30] Key was added! [18:28:50] can $1 be in the middle of the string as well? [18:28:56] yes [18:29:02] do i need to escape it then? [18:29:16] !pxe is http://wikitech.wikimedia.org/view/$1#PXE_booting [18:29:16] Key was added! [18:29:25] !pxe Dell_PowerEdge_R410 [18:29:25] http://wikitech.wikimedia.org/view/Dell_PowerEdge_R410#PXE_booting [18:29:25] !ghsh is $1 $2 gsdfgs$34ghrsh $4gsh _$5 [18:29:25] Key was added! [18:29:32] :) nice, thx [18:29:34] !ghsh gsdgh hs gfhj trjrtj trj [18:29:34] gsdgh hs gsdfgsgfhj4ghrsh trjrtjgsh _trj [18:29:39] !ghsh A B C D [18:29:39] A B gsdfgsC4ghrsh Dgsh _$5 [18:29:41] !ghsh A B C D E [18:29:42] A B gsdfgsC4ghrsh Dgsh _E [18:29:50] ;) [18:31:28] hi petan, [18:31:33] hey [18:31:47] I tried your solution for gaining ssh access to reportcard1 [18:31:52] but it doesn't work :( [18:32:06] right, did you insert it to wiki? [18:32:17] you need to copy the public key [18:32:30] manage your ssh keys [18:32:34] in console [18:32:53] ok [18:32:55] thx [18:33:31] let me know if you didn't know where it is [18:34:05] 01/04/2012 - 18:34:05 - Updating keys for diederik [18:34:08] 01/04/2012 - 18:34:08 - Updating keys for diederik [18:34:10] 01/04/2012 - 18:34:09 - Updating keys for diederik [18:34:11] 01/04/2012 - 18:34:11 - Updating keys for diederik [18:47:12] PROBLEM HTTP is now: CRITICAL on deployment-web deployment-web output: Connection refused [18:52:12] RECOVERY HTTP is now: OK on deployment-web deployment-web output: HTTP OK: HTTP/1.1 200 OK - 452 bytes in 0.003 second response time [19:01:12] PROBLEM Current Load is now: WARNING on bots-cb bots-cb output: WARNING - load average: 0.58, 7.22, 5.70 [19:06:12] RECOVERY Current Load is now: OK on bots-cb bots-cb output: OK - load average: 0.37, 2.97, 4.28 [19:15:33] hexmode: I changed my mind I will start a slow import so I won't need so many mb [19:18:17] maxsem@bastion1:~$ ssh mobile-feeds.pmtpa.wmflabs [19:18:17] Permission denied (publickey). [19:18:22] WTF? [19:18:44] are you using proper key? [19:20:22] PROBLEM Disk Space is now: CRITICAL on pageviews pageviews output: DISK CRITICAL - free space: / 0 MB (0% inode=93%): [19:22:00] lo [19:22:03] l [19:22:16] Reedy: is .19 compatible with 18 dump? [19:22:20] mediawiki [19:22:34] should be [19:22:36] I want to run importDump [19:22:38] ok [19:22:41] i don't think much has changed [19:23:09] 01/04/2012 - 19:23:09 - Updating keys for maxsem [19:25:22] RECOVERY Disk Space is now: OK on pageviews pageviews output: DISK OK [19:27:08] 01/04/2012 - 19:27:08 - Updating keys for maxsem [19:27:53] hexmode: done [19:28:02] import is running for both, but site is working [19:28:08] w00! [19:28:10] 01/04/2012 - 19:28:09 - Updating keys for maxsem [19:28:38] mhm, I tried deleting my key and then reimporting it [19:28:50] MaxSem: do you have a private key on bastion? [19:29:01] or forward it? [19:29:03] now I can't even login into bastion [19:29:08] no [19:29:09] ah [19:29:20] you will need to wait for Ryan then [19:33:39] btw OrenBochman import is done [19:45:03] hexmode: I deployed some extension we use in production and it seems to work as for now :o [19:45:08] * extensions [19:45:15] :) [19:45:21] want to test it? [19:45:31] url? [19:45:37] you need to connect to proxy [19:45:43] !socks-proxy [19:45:44] see https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances ssh @bastion.wmflabs.org -D [19:47:21] hrm... need to find out the proper user [19:47:30] mah [19:47:49] and now... my public key [19:48:00] let me make sure rsa and dsa are both loaded [19:48:07] ok [19:48:12] yep [19:48:17] ok... hrm [19:48:33] ryan gave me a place to put them [19:48:42] need to find it.... [19:49:52] hi wiki is a big piece of vandalism :D [19:59:14] petan: ok, looks like my key matches they key up there... any ideas? [19:59:25] works? [19:59:33] no [19:59:36] ah [19:59:43] ssh mah@b... doesn't work [19:59:50] what does it say [19:59:58] Ryan "improved" it [20:00:08] so that it's more safe but no one can log in :) [20:00:11] heh [20:00:12] "Permission denied (publickey)." [20:00:18] typical message [20:00:19] let me do verbose [20:00:21] :) [20:00:41] do you have a private key in .ssh/ [20:00:42] ? [20:00:47] that's important [20:00:54] also is your key in console hm? [20:01:07] https://labsconsole.wikimedia.org/wiki/Special:NovaKey [20:01:11] console hm? [20:01:11] yes [20:01:13] this [20:01:35] that's weird unfortunatelly I can't do much, Ryan can [20:01:35] without going through every character, it looks like a match [20:01:39] k [20:01:49] isn't he in office [20:01:56] no idea [20:01:57] what time is in SF [20:02:02] you are there? [20:02:13] noon in sf... I'm on the east coast [20:02:16] ah [20:02:27] I thought you are from sf too :D [20:02:52] is actually someone working there XD [20:03:01] no, I'm in Amish country [20:03:06] heh [20:03:10] :) [20:03:13] yes, they are when I visit [20:03:22] heh [20:03:53] Heh, you're right, none of the SF people have been active here in the past few hours [20:04:14] I guess Ryan would normally be here but I think he's traveling today, right? [20:04:19] no [20:04:33] he doesn't [20:04:48] he returned yesterday and said that should be today... [20:04:51] Oh [20:04:59] or I think so... [20:06:10] just confirmed that my pubkey is the right one char by char [20:06:27] * hexmode goes to do something else till Ryan shows [20:06:32] ok [20:07:38] hexmode: another possibility is to ask someone with admin rights to give us a public ip [20:07:43] :o [20:07:55] so that we could open deployment.wmflabs.org [20:08:08] mutante: can you do this? [20:08:11] petan: we'll need that eventually, I think so ok [20:08:36] I can register dns but IP's are restricted [20:08:56] Sara can do this too but I didn't see her a long time [20:09:40] * petan goes to prepare apache for wild public [20:09:56] petan: i can, but how many IPs do you need and please provide a reason why you need them [20:10:06] sure, hexmode^ [20:10:10] we need only one ip [20:10:24] petan: its like RIPE rules, because v4 IPs are getting rare [20:10:29] I don't really know what is this project for but hex does ;) [20:10:33] mutante: one IP needed to test deployment of 1.19 [20:10:58] need something so we can put this on a public ip w/ port 80 open [20:11:31] yep, ok, give me a couple minutes, checking some other puppet change right now [20:11:48] and i need to lookup the commands, but can do in a bit [20:11:55] petan: why doesn't my cable company just switch to IPV6? then we could all just forget about scarcity [20:12:07] heh [20:15:30] !sal [20:15:31] https://labsconsole.wikimedia.org/wiki/Server_Admin_Log see it and you will know all you need [20:15:51] hey Ryan! [20:15:55] howdy [20:15:59] :) [20:16:04] need to talk to you [20:16:16] sup? [20:16:38] first (briefly) there are two wiki clone's one Oren's for search project and second Mark's deployment project [20:16:42] both use huge db [20:16:48] so I wanted to talk with you about db and space [20:17:15] I made a simple proposal of mariadb, feel free to rewrite it http://www.mediawiki.org/wiki/Wikimedia_Labs/Create_shared_sql_service_for_all_projects [20:17:25] why do they need huge databases? [20:17:31] you said you wanted to move sql outside of labs [20:17:35] yep [20:17:38] would like to [20:17:43] because one is clone of simple wiki and second is clone of two wikis [20:17:54] one is 20gb second is 80gb sql [20:17:59] hm. root is hard [20:18:00] re:) [20:18:03] for shared database [20:18:16] hm... maybe, but it would be shared between devs [20:18:22] I don't think we would fight heh [20:18:24] hi Ryan, i created a new project for hexmode [20:18:35] I don't think testing db would contain private data [20:18:46] ideally there'd be an api that would let people create as many databases as they'd like [20:18:47] and now i was about to raise the IP limit and add one public IP [20:18:55]