[00:03:46] * petan found a new storage to expand his /home to [00:03:54] 80tb? sounds good [00:04:01] I'm sure some bots could take up a few hundred gig a day in logs :P [00:04:25] I am sure that if I remotely mount taht 80 tb to my desktop I will get few bits too [00:10:07] Could make a mirror of fosdem videos and still have 79tb to spare! [00:11:01] I'm keeping home directories small [00:11:33] no megavideo by wikimedia :/ [00:11:54] wikivideo [00:12:16] petan: patch: **** Can't rename file includes/Article.php to includes/Article.php.orig : Permission denied [00:12:19] sigh [00:12:25] you haven't got your permissions sorted yet :) [00:13:52] actually I think you need to add me to the depops group [00:14:41] or I can add myself actually [00:14:47] Yeah you need to add yourself [00:15:54] Still hoping you don't the servers then nfs gets turned off :P [00:16:12] It would be terrible to accidentally the servers [00:17:17] ssmollett: if $instanceproject == "testing" { blah } else { blah } [00:17:43] thanks. [00:19:16] yw [00:19:46] Some of the puppet stuff is strange, like in the openstack class where it does stuff based on hostname :P [00:22:47] werdna: sec [00:23:03] werdna: you need to insert yourself to group depops in /etc/group [00:23:13] petan: I already did [00:23:15] thanks [00:23:16] :) [00:23:19] ok [00:24:00] petan: dbdump is really slow :/ [00:24:07] what's load? [00:24:14] storage is slow [00:24:21] instances are usually fast [00:25:03] just typing is slow [00:25:11] feels like a slow connection but everything else is fine [00:25:24] ah... it works fine to me [00:25:30] load is less than 1 [00:25:33] should be okay [00:25:33] hmm [00:25:36] Someone needs to invent a waterproof laptop, that way I can go for a shower. [00:25:36] maybe packet loss or sth [00:25:55] or just stop using shower that would have similar effect [00:26:24] I think if that is your main impediment to having a shower, you might have a problem ;-) [00:27:02] !log deployment-prep switched to HEAD [00:27:03] Logged the message, Master [00:27:10] hexmode requested it [00:27:19] as in SVN HEAD? [00:27:20] werdna: we are using HEAD now, so shouldn't be problem [00:27:22] yes [00:27:46] hexmode decided to switch to that version since more people are using it [00:27:46] hmm [00:28:01] did I really type switched to HEAD... [00:28:03] Tbf whenever I've used HEAD it's been pretty stable. [00:28:07] I meant switching [00:28:08] :P [00:28:08] werdna: problem? [00:28:11] it's running now [00:28:18] that kinda defeats the purpose of going to somewhere with the same setup as wmf production [00:28:24] petan: You should have logged "development-prep got HEAD" [00:28:34] :) [00:28:37] classy [00:28:52] ;) [00:28:55] werdna: being "beta" it was never supposed to be what WMF currently is [00:29:03] but what WMF aims to be [00:29:10] at least, that was my thought [00:29:21] then why bother with trying to replicate the rest of the setup? [00:29:21] Beta is where you run production stuff but kinda have an excuse when stuff breaks, right? [00:29:30] it's for testing with a wmf setup, right? [00:29:41] Damianz: :) [00:30:07] werdna: no, that is OPs job... we do try to simulate it somewhat [00:30:11] I have one project in production that's been in "Beta" for 5 years and will soon be in v3 "Beta" :D [00:30:38] I want a place to find bugs earlier [00:30:39] * Damianz thinks he might update the footer [00:30:48] so we need HEAD [00:31:05] I think petan just gave you HEAD. [00:31:11] :P [00:31:16] working [00:31:23] sotrage is slow [00:31:24] buncha teenagers in here [00:31:27] "Dear HR..." [00:31:37] * hexmode stomps off like an old man [00:31:44] * Damianz sticks with his Master [00:31:50] I am not a teenager :) [00:31:57] my beard even has grey, so you can call me a greybeard [00:32:05] ok [00:32:11] hexmode: well, the point of the exercise of testing on the web rather than just on my local box is that if it will break on production it should break on the testing box as well and I can catch it early [00:32:16] Tbf I don't think your hair is related to your age, just working around IT :P [00:32:29] !hexmode is grey old man [00:32:29] Key was added! [00:33:05] werdna: ok but fwiw, TMH people wanted us to upgrade [00:33:11] and they were there first [00:33:24] fair enough [00:33:26] TMH? [00:33:34] Timed Media Handler [00:34:25] ah [00:34:31] It's a shame you can't run a haproxy box with a bunch of acls that based on a cookie will point you to different installs running different versions (stable, trunk, etc) of mw but I guess the setup and updates for one is enough work. [00:34:49] I don't believe you can write a code which doesn't break the testing site and break the production, testing site is already so broken... [00:35:10] someone should fix squid [00:35:14] yes [00:35:20] some RoanKattouw [00:35:42] I saw the problem with Gadget-defines, but dismissed it b/c of squid [00:35:51] he is asleep [00:35:57] or should be [00:36:03] just like me [00:36:09] because he's in same tz [00:36:23] 1:36 am :o [00:36:48] friday! :) [00:37:03] Rebecca Black had a song about that [00:37:12] don't know her [00:37:24] I am outsider from eu [00:37:28] see youtube in english [00:37:32] :) [00:37:45] all music I see is metal... [00:38:16] https://en.wikipedia.org/wiki/Friday_%28Rebecca_Black_song%29 [00:38:18] probably she is singing different style [00:39:32] how does it come the article about weird song is 20 times longer than article about unix filesystem [00:40:00] specialist area vs. article of general interest [00:40:01] shrug [00:40:37] yeah, you don't want k&r writing your encyclopedia [00:40:40] ;) [00:40:45] :o [00:42:04] It's friday, friday, friday... has been for 42min! [00:42:23] I got it almost 2 h :P [00:42:30] enjoying... [00:42:33] :D [00:43:20] I don't think remove 'artist' from that page would be a COI in regards to music :P [00:43:29] be bold [00:44:19] New patchset: Sara; "Test migration from libnss-ldap to libnss-ldapd." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2632 [00:44:39] Still loving http://www.youtube.com/watch?v=SAIEamakLoY, google really know how to do some stuff right. [00:47:25] hexmode / petan: Reckon we could throw together a het deploy setup? [00:47:37] especially since we have that on WMF [00:47:42] it will help reflect wmf setup more [00:47:55] werdna: can you do that? I don't know anything about het deploy [00:47:55] *and* you can use the site for a few purposes at once [00:48:01] Do you want me to? [00:48:04] werdna: dbdump is fucked up, I will reboot it [00:48:05] I like it! [00:48:13] I just tried to ssh there [00:48:33] yeah, I told you D [00:48:35] :D* [00:48:38] though it was just rebooted [00:48:52] do you have access? [00:48:58] yes he does [00:48:58] yeah, petan gave me access [00:49:43] I love the idea, go for it :) [00:50:09] werdna: reedy is also supposed to have access if you need help [00:50:21] but I think he is pretty busy with 1.19 [00:50:37] Roan Kattouw also have access but he's just busy :| [00:50:49] no squid [00:51:07] !log deployment-prep I broke it! [00:51:08] Logged the message, Master [00:51:11] :D [00:51:13] Roan shouldn't be blamed for squid [00:51:21] I should be blamed for squid ;) [00:51:22] Is the deployment site no longer up? [00:51:23] Ryan_Lane should be [00:51:25] yes [00:51:29] exactly [00:51:31] :) [00:51:33] Sid-G: hi [00:51:34] you guys want storage, or squid? [00:51:38] I just broke it :o [00:51:39] sorry [00:51:39] petan: hey [00:51:44] np [00:51:44] should be up in a min [00:51:45] :) [00:51:48] Ryan_Lane: everything! [00:51:52] we'r switching repo [00:51:52] hexmode: :D [00:52:51] petan: Heh, I just logged in now after pestering you way back on the first day. Did nothing in between... (Just saw the deployment schedule... oops...) [00:55:38] petan: heh, the trusted XFF bug again right now :P [00:55:55] Sid-G: works [00:56:02] or maybe not [00:56:21] yes it does [00:56:22] HTTP 500 Internal Error [00:56:27] try to refresh [00:56:32] it's squid [00:56:43] open another page [00:56:48] ok it does [00:56:50] work [00:57:02] * Sid-G goes to test some stuff [00:59:49] !log deployment-prep if anything is broken, it was me [00:59:50] Logged the message, Master [00:59:56] night [01:00:00] lol [01:02:04] hey, will the deployment site go down after the 1.19 upgrade? Or is it a permanent testing ground? [01:03:37] permenant [01:03:40] Think it will pretty much be running nearish to trunk constantly, the idea being changes ar etesting before prod [01:03:43] in some form or another [01:06:00] :D [01:06:29] I just got a testing ground for twinkle changes before putting them in production :D [01:22:33] awww. only 36TB of space per node formatted [01:22:52] lol [01:22:52] ~72TB will be in the cluster [01:23:02] o.O O.o O.O 36TB!!! [01:23:24] we are planning this per-datacenter too [01:23:28] Sid-G: we're hoping to use it in other departments [01:23:29] right now just in pmtpa [01:23:36] werdna: this is just for labs [01:23:39] Only 700 blurays [01:23:52] * Sid-G is wishing he had 36TBs ;) [01:23:53] everyone else can keep their grubby hands off ;) [01:24:05] Sid-G: well, you do, for wikimedia related activities [01:24:10] I hope you bought SSDS :P [01:24:12] lol [01:24:20] SSDS? that's way too expensive [01:24:38] this is four servers with 24 2TB disks [01:25:05] with each server having two raid 6 arrays of 12 disks [01:25:09] Going for a raid5 styleish for project storage? [01:25:15] uh, gadgets dont work at deployment? [01:25:35] and this is gluster storage, so then raid-1'd [01:25:38] ? [01:26:55] testing twinkle at deployment gave the error message: No valid token. I think that's the edit token the API gives? [01:27:13] nvm [01:27:22] works now [01:31:20] wow [01:31:35] I just saw someones cv... they had 16pages! Talk about boring people to death [01:31:42] hahahaha [01:31:49] a 3 page CV I can understand [01:32:05] I struggle to get under 3 without leaving out all detail.. but 16!? [01:36:52] I just got this twice while trying to undo an edit (not rollback): "AbuseFilterHooks::onArticleSaveComplete" ????? ???? ??? ????????? ?? ?????? ?? "1054: Unknown column 'afl_rev_id' in 'field list' (deployment-sql)" the ???s are hindi which my IRC client wont accept. The undo does happen, but all I get is this message (not the original page) [01:38:27] it's either a column typo [01:38:32] or deployment isn't up to date [01:39:22] yup, a typo [01:39:51] or not [01:41:17] deployment-sql is missing a patch [01:41:24] for some reason this col isn't in the default db schema [01:45:17] bad werdna [01:47:11] hmm? [01:47:17] what I do? [01:47:39] you'd added a column in patches, but not seemingly added it to the core AF table file [01:47:56] well, presumably you [01:47:57] oopsies :) [01:47:59] ;) [01:48:00] which patch [01:48:08] see my last commit [01:48:36] Uh, did the channel get the bug that I just tried to tell about twice? Or did my connection beat me yet again? [01:49:10] afl_rev_id? [01:49:15] yeah [01:49:23] i've fixed in svn [01:49:30] :) [01:49:33] need to get someone to run a db patch on all wikis [01:49:51] ok [01:52:30] !log deployment-prep installing ack (source code search tool) on dbdump [01:52:38] Logged the message, junior [01:53:36] lolol [01:54:18] Ryan_Lane: can you run a sql patch on all the wikis? [01:54:27] in labs? [01:54:35] ya [01:54:43] I don't even have access to that project ;) [01:54:49] lolol [01:54:55] also, you're talking to the wrong guy when it comes to mysql [01:55:07] on the cluster it's foreachwiki sql.php file.sql [01:55:10] much easy [01:55:19] does this not exist in beta? [01:55:32] nfi [01:55:35] i've not even tried to login [01:55:37] :D [01:56:02] I'll do it [01:56:07] i know it's supposed to be like production, but i'm guessing it's not really yet [01:56:13] !log deployment-prep running afl_rev_id patch on all wikis [01:56:14] Logged the message, junior [01:56:24] probably not, yeah [01:58:00] werdna@deployment-dbdump:/usr/local/apache/common$ for wiki in ` pretty easy [01:59:24] heh [01:59:39] taking a while though :/ [02:01:15] I've closed my fenari shell [02:01:22] that was the second time I nearly ran something on the wrong box :p [02:02:06] Ryan_Lane: boxes aren't allowed to access the outside internet by default, right? [02:02:25] no, the opposite [02:03:47] hmm [02:05:44] whys that? [02:06:22] just seeing why I can't from deployment-dbdump [02:06:25] maybe that has a custom policy [02:07:15] what can't you reach? [02:07:23] you *can't* reach anything inside of the production cluster [02:07:26] that's on purpose [02:07:38] BAH damn you sql syntax [02:09:06] Ryan_Lane: ahhh [02:09:07] svn [02:09:12] ah. right [02:09:15] Checking out svn+ssh://svn.wikimedia.org/svnroot/mediawiki/branches/wmf/1.19wmf1 to /usr/local/apache/common/php-1.19... [02:09:18] svn: Network connection closed unexpectedly [02:09:19] we *may* want to open that back up [02:09:26] is that recent? [02:09:29] or just use git :) [02:09:31] I swear I checked something out just today [02:09:36] maybe via http [02:09:37] mm, old script [02:09:40] ahh that'll be it [02:09:40] svn is blocked [02:09:45] well [02:09:46] ssh is [02:10:01] I'll change the script [02:13:21] svn is *really* slow, even between servers in the same DC [02:15:15] yep [02:18:06] like, 20-minutes-to-checkout [02:18:10] wtf [02:20:04] I wonder if ^demon has another Git repack job going [02:20:23] memory and cpu usage on formey are fine [02:34:22] !log deployment-prep Moving /usr/local/apache/common/live to /usr/local/apache/common/live-hom and symlinking live to live-hom [02:34:23] Logged the message, junior [02:39:54] RECOVERY Free ram is now: OK on bots-sql3 bots-sql3 output: OK: 20% free memory [02:40:12] Svn is just slow generally. [02:40:18] it's like cvs with more junk. [02:55:44] RECOVERY Disk Space is now: OK on firstinstance firstinstance output: DISK OK [02:55:44] RECOVERY Current Users is now: OK on firstinstance firstinstance output: USERS OK - 0 users currently logged in [02:56:14] RECOVERY Free ram is now: OK on firstinstance firstinstance output: OK: 88% free memory [02:57:34] RECOVERY Total Processes is now: OK on firstinstance firstinstance output: PROCS OK: 80 processes [02:58:14] RECOVERY dpkg-check is now: OK on firstinstance firstinstance output: All packages OK [02:59:24] RECOVERY Current Load is now: OK on firstinstance firstinstance output: OK - load average: 0.03, 0.09, 0.04 [03:07:54] PROBLEM Free ram is now: WARNING on bots-sql3 bots-sql3 output: Warning: 18% free memory [06:37:05] http://upload.beta.wmflabs.org/wikipedia/commons/e/ea/LetsShar1950.webm returns mime type text/plain is this comming from squid or some apache static file or where is the mimetype set? [06:46:17] Naturally the Apache config [06:47:00] Most servers need some proper coaxing for .webm files. It's not in that many deployed configs yet. [06:48:50] added it to /etc/mime.types but did not reload apache, looks ok now [06:48:56] except for the squid cache [14:57:15] hi, i'm trying to test http://ja.wikipedia.beta.wmflabs.org, and i want MediaWiki: pages imported from ja.wikipedia.org. [14:57:27] is there any way to automatize it? [16:04:40] hi all [18:14:26] New patchset: Sara; "Test migration from libnss-ldap to libnss-ldapd." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2632 [18:17:14] New review: Ryan Lane; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2632 [18:17:39] ssmollett: looks good. I just marked +2 and didn't submit [18:18:12] I think that's our normal operation, now. that way we don't submit something someone elese wasn't necessarily ready to submit :) [18:19:02] that makes sense. [18:19:46] you probably want to create a new instance before submitting, though [18:20:03] otherwise if the manifests fail, your instance won't finish building, and you won't be able to troubleshoot it [18:20:38] ldap configuration sucks to deal with :D [18:20:54] i was planning on just using testing-ldap. [18:21:03] it's already been configured, though [18:21:18] I usually take this approach: http://ryandlane.com/blog/2011/11/02/a-process-for-puppetization-of-a-service-using-nova/ [18:21:25] yes, but i might as well make sure i can revert its config. [18:21:45] true [18:26:33] ssmollett fixed the nss bug? [18:26:35] \o/ [18:27:43] libnss-ldapd (as opposed to libnss-ldap) is bug free. so we just have to migrate everything. [18:27:50] we'll see if it causes a new string of weird bugs :D [18:27:59] I don't think it will. I think this'll fix it [18:28:10] yes, by "bug free", i meant just this bug. there are always bugs. [18:28:16] heh [18:28:30] at least setuid/setgid will work [18:28:54] well that's a start :)) [18:54:24] RECOVERY Current Users is now: OK on testing-ldap testing-ldap output: USERS OK - 3 users currently logged in [18:55:44] RECOVERY Free ram is now: OK on testing-ldap testing-ldap output: OK: 89% free memory [18:56:47] i just ran puppet (before submitting my change) on testing-ldap. [18:57:00] (hi ssmollett) [18:57:04] RECOVERY Total Processes is now: OK on testing-ldap testing-ldap output: PROCS OK: 95 processes [18:57:04] ah. so that it would switch the libraries back? [18:57:09] RECOVERY Disk Space is now: OK on testing-ldap testing-ldap output: DISK OK [18:57:34] RECOVERY dpkg-check is now: OK on testing-ldap testing-ldap output: All packages OK [18:57:34] i tested switching the libraries back manually, and then with puppet. [18:57:40] * Ryan_Lane nods [18:57:45] (hi sumanah) [18:58:03] * sumanah won't distract you, just wanted to wave [18:58:54] RECOVERY Current Load is now: OK on testing-ldap testing-ldap output: OK - load average: 0.01, 0.06, 0.03 [19:09:24] how long does it take for a merge after a change is submitted? [19:09:32] one minute [19:11:37] gerrit shows the status of my change as "Submitted, Merge Pending" for about 10 minutes. is there something else i need to do? [19:12:02] umm [19:12:19] I don't see that [19:12:24] https://gerrit.wikimedia.org/r/#change,2632 [19:12:46] what do you see? [19:12:59] ah. I see where you say that [19:13:08] oh [19:13:11] it depends on another change [19:13:20] you need to rebase... [19:13:30] it's usually best to make a new branch for every change you are doing [19:13:42] then it's less likely one change will end up relying on another unrelated change [19:13:52] it depends on this change: https://gerrit.wikimedia.org/r/#change,2157 [19:14:25] so, rebase, and submit an amended commit [19:14:38] i fail at branching. how do i actually rebase? [19:15:00] heh. I fail at rebasing :) [19:15:06] it's git rebase, though [19:15:28] that said, this should likely work: https://labsconsole.wikimedia.org/wiki/Git#Fixing_a_path_conflict [19:30:23] New patchset: Sara; "Test migration from libnss-ldap to libnss-ldapd." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2632 [19:30:54] no more dependency! :D [19:31:19] New review: Ryan Lane; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2632 [19:32:35] one day i will understand git. that day will not be today. [19:33:32] Change merged: Sara; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2632 [19:37:52] New patchset: Sara; "Fix puppet manifest for testing migration to libnss-ldapd." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2640 [19:38:12] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/2640 [19:38:43] puppet didn't like having two notifies. [19:38:48] ssmollett: git people assume that if you understand how git stores data, you automatically know everything about it [19:39:43] :) i was figuring i'd learn it as i go. which does seem to be the case, but may not be ideal. [19:40:02] andrewbogott_afk: http://openstackgd.wordpress.com/2012/02/17/dns-for-openstack/ [19:41:56] New review: Sara; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2640 [19:41:56] I hate when people throw code over the wall [19:41:56] Change merged: Sara; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2640 [19:52:02] I kind of want to stab the grid dynamics people [19:52:35] they didn't bother to comment on the DNS blueprint, and they also didn't bother to tell anyone they were working on a DNS project [19:54:48] oh well. ours is upstreamed. [19:55:03] they can take the time to merge or have it replaced. [20:03:35] New patchset: Sara; "Migrating to libnss-ldapd: remove libnss-ldap configs/init script." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2643 [20:03:55] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/2643 [20:04:29] New review: Ryan Lane; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2643 [20:04:48] Change merged: Sara; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2643 [20:09:34] * aude waves [20:10:38] howdy [20:11:42] how's the storage? [20:11:54] os is installed on them [20:12:20] nice :) [20:12:25] need to get gluster running, clustered, then write some method of managing the volunes [20:12:27] *volumes [20:12:37] Ryan_Lane: I see you already responded :) [20:12:44] on their blog, yeah [20:12:56] when they mentioned they were going to release this, I posted then too [20:13:01] they haven't responded [20:13:08] they don't seem to sit in the irc channel [20:13:20] maybe they don't understand how the open source world works :D [20:14:18] Ryan_Lane: sounds good [20:15:09]