[00:00:02] <^demon> I pushed the wrong copy of the repo to gerrit, so I'm trying to obliterate the master. [00:00:12] <^demon> I want to destroy all history, somehow. [00:01:01] Yikes! Um... I'm not sure that git permits you to change history. [00:01:02] Force push from the right copy? [00:01:15] <^demon> Hrm, that might work too. [00:01:16] you'd need to rebase, and delete all revisions [00:01:19] You can kinda fake it with some hacky stuff but IIRC there is no way to truly delete the history. [00:01:35] keeping in one empty commit [00:01:48] then force-push to master [00:01:54] <^demon> Like I said, I should be able to just get away with a `git push origin :master` if it wasn't for the fake HEAD pointer. [00:02:13] want me to just do it directly? [00:02:24] <^demon> ! [remote rejected] master (branch is currently checked out) [00:02:39] <^demon> Can't do it directly on the server, it's not a working copy. [00:02:42] <^demon> Can't rebase in it. [00:02:51] <^demon> I tried ;-) [00:02:58] which project is this? [00:03:03] <^demon> mediawiki/core.git [00:03:50] <^demon> I wonder if copying the files to /var/lib/gerrit2/review_site/git/mediawiki/core.git/ it would work. [00:03:57] <^demon> Just overwrite it [00:04:15] hm. dunno [00:07:44] wow. this repo is huge [00:07:59] As big as the linux kernel? (that's a beast to clone) [00:09:00] 164MB [00:09:18] Quite big [00:09:21] yeah [00:10:04] <^demon> Ryan_Lane: That's after repacking. Unrepacked it's ~5G [00:10:49] <^demon> Anyway, I'll do a fresh dump, repack it, confirm the tags are right, then see about sorting gerrit. [00:10:52] I thought mediawiki was in svn with the other developer junk...stuff? [00:10:53] <^demon> Stupid mistake. [00:10:56] * ^demon whacks self [00:11:03] yeah, this is like a billion commits [00:11:07] rebasing isn't going to help [00:11:11] <^demon> Damianz: Yes, and the repo's about 8 years old with ~111k commits. [00:11:27] just make sure not to screw gerrit up too badly ;) [00:11:31] <^demon> Lots of stupid commits over the years too :p [00:11:38] <^demon> Ryan_Lane: Yeah, I'll be careful. [00:11:39] I really need to bring a replica up in eqiad [00:11:44] <^demon> Worst case, mediawiki/core is hosed and I rename it. [00:11:51] * Ryan_Lane nods [00:12:22] <^demon> Anyway, this is giving me a headache. I'm gonna go eat and relax while these dumps do their thing [00:12:26] Ryan_Lane: I think you can configure git to not accept 'crazy' pushes. Only accept patches from fast-forwarded branches, for starters... [00:13:13] Although maybe --force overrides everything [00:13:17] <^demon|away> andrewbogott: Yeah, but while I'm doing these history imports I kinda have to do some crazy pushes that bypass gerrit. [00:13:22] yeah. we don't allow —force, though [00:13:25] <^demon|away> Sadly, it makes it more possible for me to script stuff up. [00:13:52] <^demon|away> s/script/screw/ [00:14:31] Oh, didn't realize that you were doing dramatic/invasive stuff. Nevermind then. [00:25:47] RECOVERY Disk Space is now: OK on test1 test1 output: DISK OK [00:26:17] RECOVERY Free ram is now: OK on test1 test1 output: OK: 83% free memory [00:27:15] Hmm I wonder if the openstack dashboard supports multiple zones. Really need to do a test setup but my ubuntu boxes seem to be borked :s [00:27:27] RECOVERY Total Processes is now: OK on test1 test1 output: PROCS OK: 79 processes [00:28:07] RECOVERY dpkg-check is now: OK on test1 test1 output: All packages OK [00:29:27] RECOVERY Current Load is now: OK on test1 test1 output: OK - load average: 0.00, 0.04, 0.02 [00:29:57] RECOVERY Current Users is now: OK on test1 test1 output: USERS OK - 0 users currently logged in [00:34:11] Damianz: it should [00:34:20] though from what I read zones in nova are totally broken right now [00:37:52] Ah that sucks, I was reading some docs about it the other days. Might end up with just 2 controllers but centralised control would be nice. [00:38:26] why more than one controller anyway? redundancy? [00:38:49] which networking mode are you using? flatdhcp with multi-host? [00:39:18] More the us<>uk network not being the most reliable in the world. Latency is horrid sometimes :( [00:40:12] Most likely the vlan based setup but I'm not totally sure right now, there's a couple of uses I have in mind for it. Probably flat with dhcp for testing stuff and vlan for a production setup. [00:40:37] ah [00:42:13] I really should vlan off the office network but that requires effort to make the phones work properly lol [00:42:22] heh [00:42:34] well, the phones should be on their own vlan anyway, right? [00:44:44] Yeah they should but some of them aren't right now, will be when we knock some walls down in the office and move all the network sockets, sledgehammer fun makes up for hours of re-punching cable and changing the switch stack over. [02:44:57] PROBLEM Free ram is now: WARNING on puppet-lucid puppet-lucid output: Warning: 11% free memory [02:49:57] PROBLEM Free ram is now: CRITICAL on puppet-lucid puppet-lucid output: Critical: 3% free memory [02:59:27] RECOVERY Current Users is now: OK on translation-memory-1 translation-memory-1 output: USERS OK - 1 users currently logged in [03:00:47] RECOVERY Free ram is now: OK on translation-memory-1 translation-memory-1 output: OK: 88% free memory [03:00:47] RECOVERY Disk Space is now: OK on translation-memory-1 translation-memory-1 output: DISK OK [03:01:57] RECOVERY Total Processes is now: OK on translation-memory-1 translation-memory-1 output: PROCS OK: 85 processes [03:02:27] RECOVERY dpkg-check is now: OK on translation-memory-1 translation-memory-1 output: All packages OK [03:03:47] RECOVERY Current Load is now: OK on translation-memory-1 translation-memory-1 output: OK - load average: 0.00, 0.04, 0.01 [03:11:47] petan: do you still need puppet on bots-5 updated? [03:29:57] RECOVERY Free ram is now: OK on puppet-lucid puppet-lucid output: OK: 20% free memory [03:31:43] drdee, Ryan_Lane: hey what's up [03:33:08] abemusic: howdy [03:34:07] i hear there's some issues with content type strings in the udp logger [03:34:28] that's what drdee mentioned [03:34:42] drdee emailed me about it and i think i have a fix for it (basically copied the user agent escaping code) [03:35:43] ah. cool [03:35:50] did you submit it to svn? [03:35:58] going right now [03:36:09] there were a few issues. I'm trying to find the email [03:38:47] the second was: "The path part for API calls is missing: GET [03:38:47] https://fr.wikipedia.org/w/api.php" [03:38:59] excluding the sequence number issue [03:39:08] so I guess there was a bunch of problems. heh [03:39:54] yeah the sequence number issue is *slightly* harder :) [03:39:58] yeah [03:40:20] did you see the thing about using a gcc specific function? [03:40:38] "The fastest way to get a monotonic per-host sequence number is probably with the GCC builtin function __sync_fetch_and_add(): [03:40:38] http://gcc.gnu.org/onlinedocs/gcc-4.4.4/gcc/Atomic-Builtins.html#Atomic-Builtins [03:40:39] It atomically adds a number (say 1) to a memory location and returns the previous value. [03:40:39] If the nginx workers have a separate address space, you'll have to map a small shared memory segment, e.g. using mmap with MAP_SHARED on a regular file in each worker." [03:41:01] yeah that's in an email somewhere. I started on it, but then i got stuck with trying to test it. how can i reproduce that problem you all are seeing? [03:41:24] should be possible by listening, and making a ton of requests [03:41:42] without a patch, you'd see the sequence numbers coming in out of order [03:41:55] with a patch, you'd see them always coming in order [03:42:30] the key is to make enough concurrent requests that you'll hit another worker [03:42:48] so i could start it with 2 workers and then hammer it with several python threads [03:42:48] making the worker count large in the config would likely increase your chances [03:43:13] I'd make like 30 workers, or something like that [03:44:21] alright. i might play around with it and see what i can come up with. the other issue with the request type getting pulled is stemming from what? i'm not sure why the GET would get pulled off [03:44:35] me either [03:44:44] I'll have to see what he's talking about in the logs [03:44:49] I haven't seen this issue yet [03:46:44] ok. i'll look at the sequence number issue [11:36:41] hi Beetstra [11:37:03] Hi sumanah [11:38:21] Beetstra: are you enjoying Labs? [11:38:48] Well, mainly my bots .. they are the ones doing the fun :-) [11:39:15] ha! [11:39:34] But it is a good place for them I think [11:39:37] Beetstra: will you be coming to https://www.mediawiki.org/wiki/Berlin_Hackathon_2012 or https://wikimania2012.wikimedia.org/wiki/Main_Page ? [11:40:08] Small chance - probably not in the neighbourhood [11:40:24] :-/ [11:40:51] Beetstra: on Feb 23 (tentative) we're going to have an IRC chat about using ResourceLoader https://www.mediawiki.org/wiki/ResourceLoader that may interest you [11:40:53] And my location is not the most travel-friendly location, unfortunately - I need permission to leave the country [11:41:10] ! [11:41:15] May I ask what country this is? [11:41:24] Iḿ in Saudi Arabia [11:41:34] you might also be interested in http://lalashan.mcmaster.ca/theobio/worden/index.php/Wikipedia_SF_Hackathon_notes for your extension -- how to implement API actions and Ajax features in a MediaWiki extension [11:41:46] oh yeah, I think we might have talked before [11:42:49] ResourceLoader is more for the front-end .. my bots needs speed at the API end (and at internet-end) [11:42:54] nod. [11:43:09] Beetstra: if you have enhancement requests re the API, please do feel free to file them [11:43:14] as Bugzilla issues [11:43:43] I know. I actually have something that I think is missing in the API [11:44:07] But it is not an issue, just something that 'helps' [11:44:13] tell us! [11:44:25] editcount for IP-users .. ?? [11:44:41] Bugzilla is where we track new feature requests as well as defect reports. That's why there is an "enhancement" category. [11:44:43] seems impossible to get [11:44:55] then please, go ahead and suggest it as a fix, attach your patch, etc [11:45:04] https://www.mediawiki.org/wiki/How_to_become_a_MediaWiki_hacker#Submit_your_changes [11:45:09] more, request the patch [11:45:49] I'm sorry, I don't understand; I am requesting the patch from you. [11:45:50] What are they doing with this friggin' network I am on. I have a 1Gb connection directly into glassfibre, and pages trickle in :-( [11:46:06] :( [11:46:59] Anyway, I am not familiar with the API programming part .. so I can't write a patch that easily [11:47:04] If you have a suggestion, please do tell us about it via a new item at bugzilla.wikimedia.org -- and if you have written a patch.....oh, ok [11:47:27] sorry, when you said "I actually have something" I thought you might have meant that you patched a local MediaWiki installation to do what you wanted. [11:49:41] No probs. I did some work on it quite some time ago, but my php is now rusty [11:50:23] :-) [11:50:29] better than my nonexistent PHP [13:05:33] Hi people [13:07:57] Ryan_Lane: Do you need to talk to me? [15:30:47] Ryan_Lane: re the issues Nikerabbit described (not being able to create instances.). Yes, i created that instance for him, to see if i also get the error and it worked fine for me. I thought it might be related to the fact that Nikerabbit was an existing LDAP user (svn), and already had the sysadmin flag before, without somebody adding him to it via labsconsole. I just added the labs-user attribute to him, and ..woops he was already a sysadmin. I [15:31:18] the global groups are handled automatically now [15:31:28] we figured it out [15:31:34] alright, cool [15:31:45] it's because his user name had a utf8 character [15:31:55] and nova is apparently broken with non-ascii characters [15:32:33] andew opened a nova bug, and is looking at fixing it [15:32:38] *andrew [15:32:59] yep, just read that in backlog.. arr well, we briefly discussed whether we should use realname as wiki user or not, but then decided yes because other users also have full realnames with spaces in git/gerrit/labs. Did not expect the umlaut to break it thoughP [15:33:22] yeah. I had no reason to think the umlaut would do so [15:34:02] so that same problem also explained not being able to ssh into an instance after succesful creation? [15:34:17] i couldnt ssh into his instance myself .. [15:35:26] a number of things were broken :) [15:35:31] ok:) [15:35:36] brewster was broken [15:35:46] so, the apt repo was broken, so instance creation failed [15:36:01] i see.. [15:36:02] also, some projects were hitting quota issues regarding instance number [15:36:15] some projects have 10 instances, and the default quota is 10 [15:37:22] 10? but we have UNLIMITED hard disks! If you work for certain people anyway... [15:37:53] heh [15:37:59] unlimited hard disks? :) [15:38:08] apropos UNLIMITED disk: you saw how /home was almost filled up yesterday by that 10GB nohup.out file :p [15:38:22] yeah. someone's bot is doing that [15:38:28] Beetstra's, I think [15:38:41] Isn't that the bot which also eats all the ram on bots3 [15:38:47] probably [15:38:54] yes, i talked to him.. petan moved it first, then they deleted it [15:39:08] I sent something to labs-l about it a while back [15:39:14] could be that Beetstra isn't on the list [15:39:14] it was all wiki HTML it seemed [15:39:15] no, it was my bot on bots-2 .. [15:39:20] I probably should subscribe to that list :P [15:39:28] it's low traffic, so far [15:39:32] Only added wikitech the other week :( [15:39:43] Yes .. I forgot to comment out a debug-line .. :-/ [15:39:46] heh [15:39:49] it happens [15:39:56] we have the gluster storage in now, thankfully [15:40:03] so we'll be able to have larger home directories [15:40:09] :D [15:40:15] not that people should write data or logs into their home directories [15:40:16] :) [15:40:16] The memory leak one is on bots-3 .. I should have a look what it does to do that [15:40:36] We just using gluster mounts for everything and scrapping nfs? [15:40:44] yep [15:40:44] Sweet [15:40:44] it's more reliable [15:41:00] Yeah you can't restart the server and break my bots >.> [15:41:00] I can bring down gluster nodes for maintenance without breaking everything [15:41:25] I'm happy to move the logs etc somewhere else .. no problem [15:41:25] Beetstra: on the instance itself is likely good [15:41:56] in /mnt or in /tmp [15:41:56] Ryan_Lane: Spoil sport, yo could have made an excuse for a hadoop cluster there :P [15:42:44] will rewrite that in near future [15:42:44] Damianz: a hadoop cluster where? [15:42:44] we'll likely have a hadoop cluster in the future [15:42:44] for analytics [15:43:00] Beetstra: cool. no major rush. just make sure not to kill the home directories :) [15:44:15] It would be nice, tbf 10g of logs isn't at scale enough for hadoop... even rsyslogd can do that [15:44:15] oh. for logs. right [15:45:19] and that 10G was just about 8 hours of running the bot ... :-/ [15:45:19] well, we don't really need to keep most of the logs for long periods of time, right [15:45:19] ? [15:45:19] Most but some could be fun [15:45:20] Really need to re-install the sieve extension for thunderbird... pita having to login to the mailcluster everytime I add a new mailing list subscription. [15:45:20] People are just editing Wikipedia too much .. bots-2 has too much work with it, and it fills up too fast if I log in debug mode .. [15:45:31] Beetstra: You could have debuged in the whole day when there was no edits! [15:45:53] Brilliant plan! [15:46:09] The bot was doing fine that day .. :-p [15:46:28] (and 'no edits' was just on a couple of wikis, does not help too much) [15:46:28] Yeah, shame I didn't get time to review cbngs feed code until the week after :( [15:49:00] who called [15:49:53] Nikerabbit: People were discussing the LDAP/labs issues caused by the non-ASCII character in your name [15:50:02] ah [15:50:11] sorry. gratuitous page :) [15:50:43] still in the phase where I figure out what I can actually do with the labs [15:51:16] back in a while [16:03:59] As a side note, someone remind me it's the irc conference on saturday, possibly on saturday :P [16:04:37] It's on Saturday, at 17:00 UTC IIRC [16:05:59] 6pm gmt I think so yeah... are we still in summer time? [16:06:08] * Damianz shrugs and looks at the topic [16:06:28] 18th of February at 18:00 UTC [16:06:39] No, we're not in summer time yet until March [16:06:44] oh lol [16:06:46] Ah [16:06:46] It's in the topic [16:07:11] US switches to DST on March 11th, Europe on the 25th [16:07:49] I can't keep up with time zones... having half the office working in the us and have in the uk makes for confusion as theres betwean a 6 and 9hour time difference :( [16:08:38] heh [16:08:42] UK - SF is 8 hours [16:08:47] needs to use http://www.timeanddate.com/worldclock/converter.html before every conference call to make sure as well :p [16:09:41] * RoanKattouw knows all timezones in the US and Europe as well as Eastern Australia and New Zealand, but is just weird that way [16:10:22] I should put a clock on the wall.... the wall is reserved though for some aduino based defcon scrollsbars based off our cluster metrics :D [16:10:38] heh [16:10:45] bash timezone lookup by area code in !change 2264 btw ;) [16:10:59] When I don't want to do mental arithmetic, I use FoxClocks [16:11:02] !change 226 [16:11:02] https://gerrit.wikimedia.org/r/226 [16:11:06] !change 2264 [16:11:06] https://gerrit.wikimedia.org/r/2264 [16:11:12] * RoanKattouw glares at his '4' key [16:11:38] That's awesome [16:11:47] http://www.timeanddate.com/worldclock/meeting.html <--- [16:11:57] I like that, personally [16:22:29] You can just put the meeting in google calendar, and mark the entry with a timezone. That works for me even if I change timezones between the scheduling and the event. [16:24:17] Hmm that would work :P [16:28:45] Yes, that works [16:28:57] Google Calendar's timezone support is pretty good [16:29:17] I love the feature where you can create an event that starts and ends in different timezones, it's ideal for adding flights to your calendar [16:37:29] Ryan_Lane: If we use cloud-init to manage puppet, that will mean that we can only control it at instance creation time, right? No adding or removing classes once it's running? [16:37:44] (I may not understand about how cloud-init works. Hard to find much useful documentation about it.) [16:39:17] Oops. Speaking of timezones... Ryan won't be here for hours [16:47:26] He was here earlier [17:08:00] andrewbogott: hm. I think it's possible to change the metadata after instance creation [17:08:12] I'm jet lagged, so have been awake for a while [17:08:47] Ryan_Lane: And you're thinking that we'll have a daemon on the instance that checks the metadata periodically? Rather than a one-off script? [17:09:09] a cron maybe [17:09:26] a daemon would be nice too, though [17:09:37] I wonder if the puppet people would add support for this [17:09:38] heh [17:09:52] Have you used cloud-init to install something like this before? Is there a cloud-init doc page someplace? [17:10:14] the cloud-init documentation sucks [17:10:25] https://help.ubuntu.com/community/CloudInit [17:10:45] cloud init uses the user-data section of the metadata [17:10:54] ok, I will read [17:11:22] I think it's possible to add anything we want into the metadata, outside of the userdata, though [17:11:34] I think it's also possible to read the metadata any time we want [17:11:58] so we could likely write facter scripts to pull from the metadata [17:12:38] I wonder if we could write some other puppet support for reading the node config from the metadata, so that puppet could just be configured to do so [17:12:53] it's too bad the puppet people don't actually show up in the #puppet channel [17:13:54] http://puppetlabs.com/events/triagepuppet/ [17:14:25] that was in January though [17:14:39] maybe next time.."Freenode #puppethack) and the Web. We’re going to review all the open tickets in the Puppet project " [17:16:51] is coren still running a bot on labs somewhere? [18:07:02] 02/14/2012 - 18:07:01 - Updating keys for andrew [18:07:04] 02/14/2012 - 18:07:03 - Updating keys for andrew [18:07:05] 02/14/2012 - 18:07:05 - Updating keys for andrew [18:07:09] 02/14/2012 - 18:07:09 - Updating keys for andrew [18:13:38] o.0 [18:13:47] * Damianz wonders if that was 4 seperate boxes [18:50:17] New patchset: Sara; "Merge branch 'ganglia' into test" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2589 [18:51:57] i'm not sure if that actually picked up my changes (still learning my way around git) [19:01:27] ah. seems that's a new change [19:01:39] rather than an amended patchset to a change [19:01:58] you should amend a change, when wanting to push back into the same change [19:02:01] git commit -a —amend [19:02:42] amend complained that I was in the middle of a merge, so I tried it without the amend. how do I recover from here? [19:03:06] heh. [19:03:08] ummmmm [19:03:20] amend said you were in the middle of a merge? that's weird [19:03:42] seems this is an empty change [19:03:45] so I'm going to abandon it [19:03:50] agreed. [19:04:47] Change abandoned: Ryan Lane; "Empty change, should be submitted as amended patchset to original change." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2589 [19:05:10] so, this is the change you want: https://gerrit.wikimedia.org/r/#change,2157 [19:05:23] how extensive are your changes? [19:06:14] an easy way of handling this is to do a reset back to the original commit [19:06:15] Argh [19:06:22] Why does everyone have to screw up uis. [19:06:24] then re-make your changes, then do an amended commit [19:06:43] moderately. but i also have a tarball of my changes since i had low confidence in my git-ability. [19:07:39] So reset to the original commit, use git review -d 2157 [19:07:43] (if you have git-review installed) [19:09:25] i don't have git review. [19:09:54] it's something new some of us are using [19:09:59] I haven't switched to it yet [19:10:01] it's way easier [19:11:07] git-review is the best thing since sliced bread [19:11:14] sudo pip install git-review [19:16:58] ok. git-review'ed. [19:31:37] New patchset: Sara; "First iteration of adding ganglia for labs." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2157 [19:31:46] \o/ [19:31:50] heh [19:32:32] with actual files this time [19:32:41] indeed [19:34:22] the biggest change was adding a script to generate gmetad.conf. i still need to do something similar for gmond, but was less sure what the conf file should look like, and figured it was easier to do one thing at a time. [19:34:27] * Ryan_Lane nods [19:39:15] ssmollett: ah. earlier had you done a git pull? [19:39:22] that might be why you were in a merge [19:40:30] very likely. [19:40:49] we did a merge from production to test recently [19:40:57] so your git pull may have had some conflicts [19:41:29] and indeed, in your change there is some stuff that changed in production, and is being changed back in your change [19:41:54] the way we are handling this right now kind of sucks [19:42:08] i take it this will require some manual merging to sort out? [19:42:09] we really need branch per project working soon :) [19:42:12] yeah [19:42:51] project branches sounds like a good idea. [19:43:28] yeah. less changes that are possibly conflicting [19:43:56] we haven't been cherry-picking from test to production often enough, so test is borked [19:54:04] Ryan_Lane: Some stuff in test is in wrong anyway. [19:54:10] yep [19:54:24] at some point we need to just delete and recreate test based on production [19:54:38] I'm going to wait until we have per-project branches though [19:55:24] Are you thinking no review per project for project members and review for non project members then review into the test branch and pick into production? [19:56:38] * Damianz thinks we should have some kinda council thing and project leaders so stuff like merging from upstream gets done and life doesn't kick people in the face. [19:56:47] project branches should be able to bypass review [20:02:54] !initial-login [20:02:54] https://labsconsole.wikimedia.org/wiki/Access#Initial_log_in [20:20:31] ssmollett: so, the script looks good [20:20:55] some of the stuff put into the config can be avoided if you use the helper class [20:21:09] it'll parse ldap.conf and return values [20:21:28] and some of the config options hardcoded are options in the file [20:21:42] ok, i can update to use that. [20:23:09] I'd say push forward with getting everything working [20:23:16] and we can go back and clean things up after [20:23:24] ok. [20:23:41] working is better than perfect :) [20:26:58] Perfect is better than breaks in weird ways you can never track down :P [20:34:23] true. I see no reason why this script would do so, though ;) [23:46:21] who did the nagios install ? was that you petan ? [23:47:18] Pretty sure it was. [23:50:11] wondering what all he did :) wanting to make a new nagios host in prod [23:52:06] Labs does some weird puppet class grabby thing where production is exported puppet recourses IIRC?