[03:38:24] Sorry to as again, but I have an account already, but I think I have no instance...is there some way I can create one that I missed? [03:42:29] marktraceur: you need to be a sysadmin in a project [03:42:44] you can use the "Manage Instances" interface, in the sidebar [03:46:23] Ryan_Lane, I'm not entirely sure the project exists yet? [03:46:34] what project did you want to work on? [03:46:47] My own, the org-charts [03:47:35] it exists, and you are a sysadmin [03:47:49] have you tried the "Manage Instances" link in the sidebar? [03:47:55] Ah, OK, missed it [03:48:08] "No Nova credentials found for your account." [03:48:24] log out and back in [03:48:28] it's a bug [03:49:18] All right, thanks a bunch [03:49:28] yw [04:01:56] OK, next up, git repository....do I have authorization to do that as well? [04:02:05] yes [04:02:08] log into gerrit [04:02:13] using your wiki credentials [04:04:38] PROBLEM Current Users is now: CRITICAL on orgcharts-dev orgcharts-dev output: CHECK_NRPE: Error - Could not complete SSL handshake. [04:05:18] PROBLEM Disk Space is now: CRITICAL on orgcharts-dev orgcharts-dev output: CHECK_NRPE: Error - Could not complete SSL handshake. [04:06:08] PROBLEM Free ram is now: CRITICAL on orgcharts-dev orgcharts-dev output: CHECK_NRPE: Error - Could not complete SSL handshake. [04:06:52] Hrm [04:07:19] I clearly did something wrong [04:07:28] PROBLEM Total Processes is now: CRITICAL on orgcharts-dev orgcharts-dev output: CHECK_NRPE: Error - Could not complete SSL handshake. [04:08:18] PROBLEM dpkg-check is now: CRITICAL on orgcharts-dev orgcharts-dev output: CHECK_NRPE: Error - Could not complete SSL handshake. [04:08:58] PROBLEM Current Load is now: CRITICAL on orgcharts-dev orgcharts-dev output: CHECK_NRPE: Error - Could not complete SSL handshake. [04:09:13] OK, none of this is true, what on earth is happening [04:09:27] oh [04:09:29] heh [04:09:33] can you log into it? [04:09:47] there's an issue with the puppet manifests and newly created instances [04:09:55] Yeah, I'm logged in now [04:09:58] you need to force run puppet again on the instance as root [04:10:00] (though it took two tries) [04:10:03] and those errors will go away [04:10:05] check allowed_hosts in the nrpe config [04:10:21] mutante: the nrpe manifests are broken on initial puppet run [04:10:28] gotcha [04:10:28] I'd love for someone to diagnose and fix that :) [04:10:42] OK, service restarted [04:10:50] i am looking at it right now.. but on spence.. it doesnt not allow itself to talk it [04:10:55] because it means new instances spam for 30 mins or so [04:11:07] marktraceur: you need to run: puppetd -tv [04:11:08] as root [04:11:13] Bollocks [04:11:15] you can't run it as sudo [04:11:24] you need to sudo to root, then run it :) [04:11:28] puppet is weird about that [04:12:56] Allll righty then [04:13:58] RECOVERY Current Load is now: OK on orgcharts-dev orgcharts-dev output: OK - load average: 0.41, 0.34, 0.30 [04:14:38] RECOVERY Current Users is now: OK on orgcharts-dev orgcharts-dev output: USERS OK - 1 users currently logged in [04:15:18] RECOVERY Disk Space is now: OK on orgcharts-dev orgcharts-dev output: DISK OK [04:15:53] Is the gerrit documentation right in saying that the best way to create a project is to issue a command over ssh? [04:16:08] RECOVERY Free ram is now: OK on orgcharts-dev orgcharts-dev output: OK: 90% free memory [04:17:28] RECOVERY Total Processes is now: OK on orgcharts-dev orgcharts-dev output: PROCS OK: 76 processes [05:21:33] Ryan_Lane: still up? [05:21:39] yep [05:22:30] i got a problem with iptables_add_service, trying to use 2 services at a time [05:22:46] currently i get value is a required option for Puppet::Parser::Resource::Param at /var/lib/git/operations/puppet/manifests/iptables.pp:146 [05:23:34] i tried to repeat iptables_add_service and iptables_add_exec twice after another..in ne class [05:23:46] wondering if that is the wrong aproach altogether [05:24:39] as in https://gerrit.wikimedia.org/r/#patch,sidebyside,3005,2,manifests/nagios.pp [05:25:20] would you know which "value" it is missing as a required option? [05:26:52] i did add "snmp-trap" in iptables_ports and iptables_protocols.. and nsca worked before [05:27:35] before merging them like this, i would run into duplicate definitions for repeating "iptables_add_service{ "lo_all" etc [05:28:32] or maybe "-" isn't an allowed character in the service name? hrmmm [05:36:39] or it just did not learn about the new protocol/port snmp-trap yet because puppet did not run since i added it. dont worry about it if you were about to go to sleep and you also cant tell off-hand [05:55:21] /redraw/win 2 [06:11:43] whoops. wasn't in this channel [06:12:15] mutante: yeah, don't use - [06:12:22] it's always a bad idea [06:12:36] it works in some places in puppet and doesn't in others [06:12:51] I was *just* talking with ma rk about this today, and he said it's never bit him. :D [06:14:06] ok, thanks! [06:14:14] yw [06:18:06] dddooh... stupid typo "snmap-trap" != "snmp-trap" ..ignore me :p [06:18:26] removing - anyways:) [06:18:33] heh [06:19:52] Cannot write to `lucid-server-cloudimg-i386.tar.gz' (Disk quota exceeded). [06:19:53] \o/ [06:20:00] checking gluster disk quotas :) [06:20:13] mutante: on a labs project, go to /data/project [06:21:14] then: df -h [06:21:15] :) [06:22:01] i got /data but not /data/project [06:22:05] oh? [06:22:07] which project? [06:22:07] on the first i tried [06:22:16] planet, instance "venus" [06:22:36] oh [06:22:37] sorry [06:22:43] /etc/init.d/autofs reload [06:22:45] then do it [06:23:06] puppet needs to not be broken on the instance too. heh [06:23:44] running puppet [06:24:24] did the autofs reload [06:24:34] /data stays empty so far [06:24:43] they are automounts [06:24:45] ah [06:24:50] I need to turn ghosting on [06:24:50] labstore1.pmtpa.wmnet:/planet-project on /data/project [06:25:01] 71T :) [06:25:16] i see them :) nice!! [06:28:06] so now you are looking at quotas so we can set them individually fo reach project, i suppose [06:28:21] already done [06:28:29] +1 [06:28:34] default quota for projects is 300GB [06:28:55] that's plenty [06:29:09] hm. I don't think I can ghost the shares [06:29:13] maybe I can [06:29:56] * mutante sees something about a --ghost option for the automounte? [06:30:08] yeah, i don't think that'll work [06:31:02] I think it's because they are * autofs mounts [06:32:17] hm. lemme try adding the option to automaster [06:34:13] "Note that the wildcard key does not work in conjunction with the -browse option." [06:34:17] welp, that's out [06:38:45] oh well, I'll just document how it works [06:39:07] in the future people will be able to create their own volumes, so we need to use wildcard mounts [06:41:09] yeah, just asking them to do that restart shouldnt be too much [06:42:20] I'm running it for them :) [06:42:34] what I need to document is the location of the mounts, since they aren't ghosted [06:42:45] hm [06:42:48] put it in the default motd? [06:43:03] ah. not a bad idea [06:49:56] nope, just working late [06:50:05] gotcha [06:50:11] whoops [06:50:12] heh [06:50:16] didn't see it was apm [06:50:43] np [07:13:47] bah. the bot can't connect to irc [07:17:01] hm. I could probably syslog it somewhrre [07:20:10] oh well, it's working properly otherwise [07:25:05] PROBLEM dpkg-check is now: CRITICAL on deployment-web deployment-web output: CHECK_NRPE: Socket timeout after 10 seconds. [07:25:45] PROBLEM Current Users is now: CRITICAL on deployment-web deployment-web output: CHECK_NRPE: Socket timeout after 10 seconds. [07:26:05] PROBLEM Free ram is now: CRITICAL on deployment-web deployment-web output: CHECK_NRPE: Socket timeout after 10 seconds. [07:27:45] PROBLEM Current Load is now: CRITICAL on deployment-web deployment-web output: CRITICAL - load average: 131.68, 71.73, 30.53 [07:29:55] RECOVERY dpkg-check is now: OK on deployment-web deployment-web output: All packages OK [07:30:35] RECOVERY Current Users is now: OK on deployment-web deployment-web output: USERS OK - 0 users currently logged in [07:30:55] PROBLEM Free ram is now: WARNING on deployment-web deployment-web output: Warning: 6% free memory [07:35:55] RECOVERY Free ram is now: OK on deployment-web deployment-web output: OK: 71% free memory [07:47:35] PROBLEM Current Load is now: WARNING on deployment-web deployment-web output: WARNING - load average: 0.03, 3.07, 14.83 [08:07:35] RECOVERY Current Load is now: OK on deployment-web deployment-web output: OK - load average: 0.02, 0.11, 4.10 [11:33:05] PROBLEM dpkg-check is now: CRITICAL on deployment-web deployment-web output: CHECK_NRPE: Socket timeout after 10 seconds. [11:34:05] PROBLEM Free ram is now: CRITICAL on deployment-web deployment-web output: CHECK_NRPE: Socket timeout after 10 seconds. [11:34:35] PROBLEM SSH is now: CRITICAL on deployment-web deployment-web output: CRITICAL - Socket timeout after 10 seconds [11:35:35] PROBLEM Current Load is now: CRITICAL on deployment-web deployment-web output: CRITICAL - load average: 29.10, 32.80, 16.23 [11:36:15] PROBLEM Disk Space is now: CRITICAL on deployment-transcoding deployment-transcoding output: DISK CRITICAL - free space: / 21 MB (1% inode=53%): [11:37:55] RECOVERY dpkg-check is now: OK on deployment-web deployment-web output: All packages OK [11:38:55] RECOVERY Free ram is now: OK on deployment-web deployment-web output: OK: 71% free memory [11:39:25] RECOVERY SSH is now: OK on deployment-web deployment-web output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [11:40:35] PROBLEM Current Load is now: WARNING on deployment-web deployment-web output: WARNING - load average: 0.21, 12.05, 11.76 [11:41:15] PROBLEM Disk Space is now: WARNING on deployment-transcoding deployment-transcoding output: DISK WARNING - free space: / 56 MB (4% inode=53%): [11:55:35] RECOVERY Current Load is now: OK on deployment-web deployment-web output: OK - load average: 0.16, 0.75, 4.57 [13:01:55] PROBLEM Free ram is now: CRITICAL on deployment-web deployment-web output: Critical: 3% free memory [13:03:35] PROBLEM Current Load is now: CRITICAL on deployment-web deployment-web output: CRITICAL - load average: 24.11, 34.14, 17.63 [13:34:05] !log deployment-web add ppa:j/timedmediahandler and install ffmpeg on web3 and web5 [13:34:05] deployment-web is not a valid project. [13:34:20] !log deployment-prep add ppa:j/timedmediahandler and install ffmpeg on web3 and web5 [13:34:21] Logged the message, Master [13:36:15] RECOVERY Disk Space is now: OK on deployment-transcoding deployment-transcoding output: DISK OK [13:42:47] petan|wk: is it possible increase wgMaxShellMemory on the web instances? currently ffmpeg is not able to extract frames for HD videos [13:42:51] http://commons.wikipedia.beta.wmflabs.org/w/thumb.php?f=2012-tlr1_h1080p.webm&width=800 [13:43:19] Isn't deploy suppose to mirror production? I thought external repos wern't allowed on production for security purposes. [13:47:13] j^: of course, how much you want [13:47:31] you shouldn't use shell on web server [13:47:43] there is dbdump instance for shell [13:48:23] petan|wk: thumbnail extraction uses ffmpeg(via shell) [13:48:25] j^: you can also log stuff by typing log in shell [13:48:35] ok [13:48:55] whats the current value? [13:49:02] let me check [13:50:55] RECOVERY dpkg-check is now: OK on deployment-web4 deployment-web4 output: All packages OK [13:51:35] RECOVERY Current Load is now: OK on deployment-web4 deployment-web4 output: OK - load average: 1.47, 0.75, 0.28 [13:52:21] !log deployment-prep root: updated apt on webs1 [13:52:22] Logged the message, Master [13:52:35] RECOVERY Current Users is now: OK on deployment-web4 deployment-web4 output: USERS OK - 0 users currently logged in [13:53:15] RECOVERY Disk Space is now: OK on deployment-web4 deployment-web4 output: DISK OK [13:53:55] RECOVERY Free ram is now: OK on deployment-web4 deployment-web4 output: OK: 68% free memory [13:54:25] RECOVERY SSH is now: OK on deployment-web4 deployment-web4 output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:55:15] RECOVERY Total Processes is now: OK on deployment-web4 deployment-web4 output: PROCS OK: 110 processes [13:59:11] petan|wk: ffmpeg needs something like 33MB to decode HD video into a frame [14:01:41] $wgMaxShellMemory = 300000; [14:01:44] j^ [14:03:55] RECOVERY Free ram is now: OK on deployment-web2 deployment-web2 output: OK: 79% free memory [14:04:25] RECOVERY SSH is now: OK on deployment-web2 deployment-web2 output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:05:15] RECOVERY Total Processes is now: OK on deployment-web2 deployment-web2 output: PROCS OK: 110 processes [14:05:35] PROBLEM Current Users is now: CRITICAL on deployment-webs1 deployment-webs1 output: Connection refused by host [14:05:55] RECOVERY dpkg-check is now: OK on deployment-web2 deployment-web2 output: All packages OK [14:05:55] PROBLEM dpkg-check is now: CRITICAL on deployment-webs1 deployment-webs1 output: Connection refused by host [14:06:20] PROBLEM Disk Space is now: CRITICAL on deployment-webs1 deployment-webs1 output: Connection refused by host [14:06:35] RECOVERY Current Load is now: OK on deployment-web2 deployment-web2 output: OK - load average: 0.59, 0.57, 0.25 [14:06:55] PROBLEM Free ram is now: CRITICAL on deployment-webs1 deployment-webs1 output: Connection refused by host [14:07:35] PROBLEM SSH is now: CRITICAL on deployment-webs1 deployment-webs1 output: Connection refused [14:07:35] RECOVERY Current Users is now: OK on deployment-web2 deployment-web2 output: USERS OK - 0 users currently logged in [14:08:20] RECOVERY Disk Space is now: OK on deployment-web2 deployment-web2 output: DISK OK [14:08:45] PROBLEM dpkg-check is now: CRITICAL on deployment-nfs-memc deployment-nfs-memc output: DPKG CRITICAL dpkg reports broken packages [14:09:16] PROBLEM Total Processes is now: CRITICAL on deployment-webs1 deployment-webs1 output: Connection refused by host [14:09:35] PROBLEM Current Load is now: CRITICAL on deployment-webs1 deployment-webs1 output: Connection refused by host [14:12:26] !log deployment-prep petrb: rebooting -nfs [14:12:27] Logged the message, Master [14:15:35] PROBLEM Current Load is now: WARNING on bots-sql3 bots-sql3 output: WARNING - load average: 8.76, 9.01, 5.88 [14:20:35] RECOVERY Current Load is now: OK on bots-sql3 bots-sql3 output: OK - load average: 1.52, 4.67, 4.89 [14:21:38] petan|wk: another question, w/extensions/ is w/extensions-trunk now, can this be symlinked? [14:23:14] yes [14:23:21] some parts of TMH right now expect some web resources to be in ../extensions/TimedMediaHandler/MwEmbedModules [14:24:15] j^: you want to link extensions-trunk to extensions? [14:24:44] petan|wk: yes so that http://commons.wikimedia.beta.wmflabs.org/w/extensions/TimedMediaHandler/MwEmbedModules/EmbedPlayer/resources/skins/kskin/images/ksprite.png is accessible [14:25:04] extensions-trunk -> /usr/local/apache/common/php-trunk/extensions [14:26:06] j^: I don't understand this new system, werdna didn't give us any guide how to use it [14:26:27] I think you need to pick a version you want in .dat file [14:26:47] commons should be trunk [14:26:52] j^: you need to use trunk? [14:27:13] petan|wk: only extensions-trunk contains TimedMediaHandler so yes [14:27:25] but it should be already like that [14:29:34] the png is loaded via css, css is loaded via load.php. load.php extens the url to /w/extensions/TimedMediaHandler/MwEmbe... but the extension is in /w/extensions-trunk/... [14:30:01] so i guess load.php needs to know about the new extensions location [14:30:02] w extensions is a link to extensions-trunk so... [14:30:20] werdna: hey [14:30:24] can you solve it? [15:05:28] 03/09/2012 - 15:05:27 - Creating a home directory for mmovchin at /export/home/hugglewa/mmovchin [15:06:27] 03/09/2012 - 15:06:26 - Updating keys for mmovchin [17:04:45] PROBLEM Current Load is now: CRITICAL on bots-cb bots-cb output: CHECK_NRPE: Socket timeout after 10 seconds. [17:14:36] PROBLEM Current Load is now: WARNING on bots-cb bots-cb output: WARNING - load average: 0.90, 12.59, 16.22 [17:34:36] RECOVERY Current Load is now: OK on bots-cb bots-cb output: OK - load average: 0.29, 0.66, 4.77 [17:35:05] petan or petan|wk you about? [19:24:49] PROBLEM Total Processes is now: CRITICAL on bots-dev bots-dev output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:25:29] PROBLEM dpkg-check is now: CRITICAL on bots-dev bots-dev output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:26:25] PROBLEM Current Load is now: CRITICAL on bots-dev bots-dev output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:26:59] PROBLEM Current Users is now: CRITICAL on bots-dev bots-dev output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:27:39] PROBLEM Disk Space is now: CRITICAL on bots-dev bots-dev output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:28:29] PROBLEM Free ram is now: CRITICAL on bots-dev bots-dev output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:31:46] hi [19:32:02] howdy [19:34:14] so, we can use a project for "Maps" or if you want "OpenStreetMap" [19:34:47] i'm not sure how the instances work, if we should have one for everything or multiple for trying different things? [19:35:55] such as the german wiki / toolserver projects, multilingual maps (germans also working on that), mobile / production test [19:36:30] well, it depends [19:36:39] we can either create different projects for each of those [19:36:45] or everyone can work together in one project [19:36:50] everyone creates their own instances [19:37:01] the default quota for a project is 10 instances [19:37:23] I think it's likely we should just have a single project [19:37:29] we can create more later if we need to [19:37:40] aude2: did you want to be added to the project? [19:37:58] what's your labs account name? [19:38:16] ah. it's aude [19:38:32] ok. I created the maps project and added you to it [19:39:10] let's see if the storage gets created properly. heh [19:39:28] 03/09/2012 - 19:39:28 - Creating a project directory for maps [19:39:28] 03/09/2012 - 19:39:28 - Creating a home directory for aude at /export/home/maps/aude [19:39:28] 03/09/2012 - 19:39:28 - Creating a home directory for laner at /export/home/maps/laner [19:40:10] Ryan_Lane: we might want to try 2 approaches for osm rendering [19:40:13] hm. did not. [19:40:23] yeah, but you can do both in the same project [19:40:24] one with node.js and one with tirex [19:40:27] 03/09/2012 - 19:40:27 - Updating keys for aude [19:40:27] 03/09/2012 - 19:40:27 - Updating keys for laner [19:40:31] ok but different instances? [19:40:52] ah. I see why it didn't work. an instance must exist before it'll create the volumes [19:40:57] want to evaluate how they perform, are flexible and scale [19:40:58] wait. that's not right [19:41:24] well, performance testing isn't going to work amazingly in labs [19:41:29] yeah [19:41:30] i know [19:41:49] at least we can get puppet configurations for both and try them somewhere [19:45:33] yeah [19:45:41] then we can try them on production hardware [19:45:49] that's where we can do performance testing [19:47:09] but having a tested configuration that's puppetized that we can easily switch between will make that possible [19:48:56] yep [19:48:59] that's the plan [19:49:51] i should have more time tonight and over the weekend to work on this [19:50:14] and expect we'll have some additional people join the projects [19:50:15] ok [19:50:18] project [19:50:28] aude2: so i created the project and added you as sysadmin and netadmin [19:50:34] ok, thanks [19:50:45] you can add people to the project, and add them to sysadmin and netadmin [19:50:49] via "Manage projects" in the sidebar [19:51:15] remember to read the docs on instances and security groups before creating instances [19:51:18] !instances [19:51:18] https://labsconsole.wikimedia.org/wiki/Help:Instances [19:51:22] !security [19:51:22] https://labsconsole.wikimedia.org/wiki/Help:Security_Groups [19:53:10] Ryan_Lane: ok [19:53:26] and sure i'll have questions for you (or petan) [19:57:48] * Ryan_Lane nods [19:57:53] thanks [19:58:13] * aude2 back to my other work [19:59:24] ffs [20:03:02] !account-questions [20:03:02] I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your preferred email address. 3. Your SVN account name, or your preferred shell account name, if you do not have SVN access. [20:04:10] Ryan_Lane: asking me? [20:04:31] no, was copy/pasting into an email [20:04:36] ah, okay [20:04:42] i already have git, etc. [20:27:39] PROBLEM Current Load is now: WARNING on bots-cb bots-cb output: WARNING - load average: 2.37, 14.62, 9.93 [20:39:03] petan: have you see oren around? [20:42:39] RECOVERY Current Load is now: OK on bots-cb bots-cb output: OK - load average: 0.35, 1.17, 4.08 [21:20:18] !log globaleducation Added redirect in apache config so /index.php/ goes to /wiki/ [21:20:19] Logged the message, Master [21:25:30] Change on 12mediawiki a page Wikimedia Labs was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=508517 edit summary: /* Roadmap */ [21:27:11] Change on 12mediawiki a page Wikimedia Labs/status was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=508518 edit summary: /* 2012-03-09 */ new section [21:27:12] Change on 12mediawiki a page Wikimedia Labs/status was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=508518 edit summary: /* 2012-03-09 */ new section [22:00:42] Change on 12mediawiki a page Wikimedia Labs was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=508526 edit summary: /* Documents */ [22:03:25] Change on 12mediawiki a page Wikimedia Labs/Terms of use was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=508528 edit summary: [22:05:11] Change on 12mediawiki a page Wikimedia Labs/Terms of use was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=508529 edit summary: [22:05:38] Change on 12mediawiki a page Wikimedia Labs/Terms of use was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=508530 edit summary: [22:10:02] Ryan_Lane: Have you ever had gluster report failure when starting/stopping a volume even though it's actually working? [22:10:13] not yet [22:10:20] which version of gluster are you using? [22:10:24] we're using 2.4.1 [22:10:34] err [22:10:35] sory [22:10:39] 3.2.4 [22:10:41] I was way off [22:10:57] I'm testing on a VM right now, with 3.2.1. [22:11:03] It didn't do this yesterday. [22:12:34] 2.4.1 ewww [22:14:06] andrewbogott: are you using the glusterfs package? [22:14:15] I uploaded a specific version to our custom repo [22:14:25] Damianz: heh, yeah, that would be really really old :) [22:14:48] Ryan_Lane: No, but I'll move all this code to a labs machine eventually. [22:15:08] ahhh ok [22:15:15] it isn't in labs right now. heh [22:15:19] that makes sense then [22:15:20] Back in the 2 branch gluster was REALLY buggy as well [22:15:27] maybe the issue you are having is fixed in the version we are using [22:15:45] because you also had an issue where it would let you delete a volume without stopping it too, right? [22:15:51] that didn't happen to me [22:15:59] it forced me to stop it first [22:16:39] Not exactly... that was just a volume which was deleted when I tried to stop it, but started when I tried to delete it. [22:16:44] And that /was/ on a labs machine. [22:17:02] But I'd twiddled with the brick directories by hand, probably got it into an impossible state. [22:17:23] So I deserved it, even though gluster could be more fault-tolerant. [22:18:22] Change on 12mediawiki a page Wikimedia Labs/Terms of use was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=508535 edit summary: [22:18:33] heh [22:18:47] yeah, gluster seems to be non-fault tolerant of certain things [22:18:53] the _base directory was a perfect example of that [22:18:56] thanks to FUSE [22:45:59] PROBLEM Current Load is now: CRITICAL on bots-cb bots-cb output: CHECK_NRPE: Socket timeout after 10 seconds. [22:47:58] PROBLEM Total Processes is now: CRITICAL on bots-cb bots-cb output: CHECK_NRPE: Socket timeout after 10 seconds. [22:48:03] PROBLEM Current Users is now: CRITICAL on bots-cb bots-cb output: CHECK_NRPE: Socket timeout after 10 seconds. [22:48:03] PROBLEM dpkg-check is now: CRITICAL on bots-cb bots-cb output: CHECK_NRPE: Socket timeout after 10 seconds. [22:48:12] How do I create a new project in gerrit? The gerrit docs say to use an ssh command? [22:52:48] RECOVERY Total Processes is now: OK on bots-cb bots-cb output: PROCS OK: 104 processes [22:52:53] RECOVERY Current Users is now: OK on bots-cb bots-cb output: USERS OK - 0 users currently logged in [22:52:53] RECOVERY dpkg-check is now: OK on bots-cb bots-cb output: All packages OK [23:00:38] PROBLEM Current Load is now: WARNING on bots-cb bots-cb output: WARNING - load average: 0.89, 6.39, 15.50 [23:11:57] !log globaleducation reduced logo size to 155px [23:11:59] Logged the message, Master [23:14:41] !log globaleducation still too big (not same width/height), reducing logo to 130x155px [23:14:42] Logged the message, Master [23:19:06] * ^demon stabs gerrit, again and again [23:20:39] RECOVERY Current Load is now: OK on bots-cb bots-cb output: OK - load average: 0.32, 0.62, 4.70 [23:21:02] <^demon> Ryan_Lane: This is the most annoying bug I've hit in gerrit so far. [23:21:03] <^demon> http://code.google.com/p/gerrit/issues/detail?id=1072 [23:22:07] ah. well, just make sure it has an empty commit :) [23:22:17] though I'm sure it's incredibly annoying [23:24:13] <^demon> Hmmm, let me try something [23:24:57] <^demon> I've been fighting this for the last ~3 hours off and on :\ [23:27:10] <^demon> Ryan_Lane: The part that baffles me is it seems to be pushing master, 2 user branches, and 1.10, 1.11, 1.14, 1.7 and 1.8 release branches [23:27:18] <^demon> But the other release branches give me this bogus error. [23:28:31] oh [23:28:32] weird [23:29:06] <^demon> I've attempted to give myself push merge commits to every possible refspec I can imagine. [23:29:12] <^demon> Since somehow that's a workaround. [23:29:13] <^demon> No luck. [23:29:16] * ^demon kicks gerrit [23:30:35] you tried refs/for/refs/heads/master [23:30:35] ? [23:32:13] <^demon> I tried refs/for/refs/* [23:32:17] <^demon> Lemme try that. [23:36:07] <^demon> Ahhh, it worked. I wonder if my refs/for/* DENY on Registered Users was causing me to hit this. Wow. [23:36:17] <^demon> Ryan_Lane: I love you. I should've come crying to you hours ago [23:37:46] <^demon> Seriously. I owe you like a whole pack of beers.