[03:28:38] Change on 12mediawiki a page Developer access was modified, changed by Hanxue link https://www.mediawiki.org/w/index.php?diff=598444 edit summary: [03:29:30] Change on 12mediawiki a page Developer access was modified, changed by Hanxue link https://www.mediawiki.org/w/index.php?diff=598445 edit summary: /* User:Hanxue */ [03:33:03] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=598446 edit summary: /* User:Hanxue */ done [03:45:17] wm-bot! [03:45:34] hi! ;) [03:46:14] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=598448 edit summary: test [03:46:19] Damianz: :D [03:46:21] luckily i could just ask the engineers about it [03:47:26] welcome back wm-bot :) [03:47:34] woot [03:47:41] !log bots [bots-1 - wm-bot] booted wmib and that didn't fix anything, booted bouncer and then wmib again and that worked. must have been hung post netsplit [03:47:41] Logged the message, Master [03:48:35] paul_irish: are you near some impending doom? aka Sandy [03:53:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:13:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [04:23:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:43:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [04:53:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:13:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [05:24:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:44:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [05:54:45] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:14:16] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:24:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:44:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:55:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:14:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [07:25:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:44:23] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [07:56:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:14:23] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [08:17:52] RECOVERY Free ram is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: OK: 51% free memory [08:26:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:39:04] !ping [08:39:05] pong [08:44:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [08:56:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:14:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [09:22:27] [bz] (8NEW - created by: 2Jan Luca, priority: 4Unprioritized - 6normal) [Bug 41452] !log in IRC does not work - https://bugzilla.wikimedia.org/show_bug.cgi?id=41452 [09:27:35] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:44:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [09:57:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:14:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [10:28:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:44:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [10:59:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:14:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [11:29:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:42:34] PROBLEM Current Users is now: WARNING on wikidata-dev-4 i-000004bb.pmtpa.wmflabs output: USERS WARNING - 6 users currently logged in [11:44:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [11:45:22] !log bots petrb: patch of wm-bot [critical] [11:45:24] Logged the message, Master [11:46:41] @help [11:46:41] Type @commands for list of commands. This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.8.26.1 source code licensed under GPL and located at https://github.com/benapetr/wikimedia-bot [11:46:46] !ping [11:46:46] pong [11:46:53] wow [11:47:19] I would send myself a barnstar haha [11:47:51] rewrote like a half of code, without single test and then moved to production... it works :D [11:48:00] RECOVERY Current Users is now: OK on wikidata-dev-4 i-000004bb.pmtpa.wmflabs output: USERS OK - 5 users currently logged in [11:48:18] @seenrx hashar [11:48:19] petan: hashar is in here, right now (multiple results were found: hashar_, hasharEat, hasharHome, hasharLunch, hasharFood) [11:48:21] there [11:48:35] hashar was just testing :D didn't notice you [11:49:57] lunch brb [12:00:36] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:14:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [12:31:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:31:24] @infobot-detail petan [12:31:24] Info for petan: this key was created at N/A by N/A, this key was displayed 1 time(s), last time at 10/10/2012 11:42:20 AM (19.00:49:04.4357360 ago) [12:44:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [13:01:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:05:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 199 processes [13:10:32] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 106 processes [13:14:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [13:31:17] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:44:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [14:00:36] [bz] (8NEW - created by: 2Antoine "hashar" Musso, priority: 4Normal - 6normal) [Bug 36994] disk I/O in ganglia - https://bugzilla.wikimedia.org/show_bug.cgi?id=36994 [14:01:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:14:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [14:19:42] PROBLEM Free ram is now: WARNING on aggregator2 i-000002c0.pmtpa.wmflabs output: Warning: 8% free memory [14:19:54] RECOVERY SSH is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [14:19:54] RECOVERY Disk Space is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: DISK OK [14:20:53] RECOVERY dpkg-check is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: All packages OK [14:23:38] RECOVERY Current Users is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [14:23:38] RECOVERY Total processes is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: PROCS OK: 227 processes [14:24:23] RECOVERY Current Load is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: OK - load average: 0.03, 0.32, 0.52 [14:25:56] on https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bugtracker I'm not listed under "Members" (mutante has tried earlier already). Could somebody add me (User:Aklapper) there? [14:28:13] !nyan [14:28:42] !log centralauth Updated centralauth-daemons and centralauth-mysql and reboot them [14:28:42] Logged the message, Master [14:31:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:40:41] [bz] (8RESOLVED - created by: 2Jan Luca, priority: 4Unprioritized - 6normal) [Bug 41452] !log in IRC does not work - https://bugzilla.wikimedia.org/show_bug.cgi?id=41452 [14:47:06] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [14:59:42] PROBLEM Free ram is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:01:36] PROBLEM Current Users is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:01:36] PROBLEM Total processes is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:01:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:02:23] PROBLEM Current Load is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:02:53] PROBLEM SSH is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: Server answer: [15:02:53] PROBLEM Disk Space is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:03:53] PROBLEM dpkg-check is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:04:35] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [15:10:45] Silke_WMDE_ is the stuff you just pushed to gerrit working for you, or are you still stumped by the problem you mentioned yesterday? [15:11:13] It's working for me now. [15:11:54] great! [15:16:12] yes. I'm not sure if it's a good idea to provide separate client and repo installation recipes because then I can't provide the client config. But that's a decision we have to make in the team here, I suppose. [15:17:04] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [15:29:33] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 107 processes [15:31:55] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:39:25] Silke_WMDE_: I reviewed your change [15:41:38] Jan_Luca: thx, I'll look into it. I chose the way andrewbogott recommended (no module, but a role). Let's see what he criticizes. [15:43:05] * andrewbogott is reading [15:47:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [15:47:33] Silke_WMDE_: Is it necessary to duplicate everything that mediawiki.pp does in your new class? Can't your new wikidata class just install wikimedia::singlenode and then add in the extension(s) you want? [15:48:25] andrewbogott: I tried this but puppet didn't let me modify LocalSettings.php then. [15:49:15] Ok… let's solve /that/ problem then, rather than duplicating everything. [15:50:37] One option is to put switchable sections into the LocalSettings.php template. So that it includes the bits you need based on the state of a boolean. [15:51:28] Do you need a bunch of extra stuff in LocalSettings.php or just an additional include_once line? [15:52:05] Or we can add a var for additional stuff that can be loaded from another file [15:52:22] several require_once lines and other stuff e.g. configuration of the namespace for the COntenthandler [15:53:40] So we could use erb for this and load the content from another template [15:54:58] I don't know how to add a switchable option to LocalSettings.php [15:55:04] Yeah… probably the original LocalSettings.php should include a variable that's just a list of other files to include. [15:55:16] that sounds good [15:55:39] Then each extension can append its include lines (which can potentially refer to other templates) [15:56:37] Silke_WMDE_: A good baby step would be to refactor the existing extensions in mediawiki::singlenode to do that. Then the path for data would be clear. Do you want to try that or shall I? [15:56:51] andrewbogott: Do you think about something like this: <% addons.each do |addon| -%> [15:57:06] ... [15:57:24] Jan_Luca: I'm not good with erb syntax off the top of my head, but that's roughly what I was thinking. [15:57:50] andrewbogott: I copied this from another erb file ;-) [15:57:57] Then the syntax is probably right! [15:58:08] andrewbogott: Have a go! [15:58:08] 'k [15:59:12] andrewbogott: Should each the parameter a array of files that should be included or raw PHP [15:59:44] filename I think. [16:00:01] ok, then it should be easy :-) [16:02:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:16:37] On https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bugtracker I'm not listed under "Members". Could somebody add me (User:Aklapper) there? [16:17:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [16:18:43] You'll need to ask DanielFriesen or a labs ops person [16:19:40] I had hopes to find such a person here in the #labs channel, but thanks [16:20:52] andre__: You should wait for Ryan_Lane or a admin of the project [16:21:04] ok... [16:21:38] or you ask andrewbogott [16:21:53] (he is there at moment) [16:22:27] andre__: Hang on, lemme see if I know how to fix that. [16:28:58] andrewbogott: Is it allowed for non-ops to change the global puppets groups on labsconsole. I would like to add the php classes when the change is merged [16:29:37] I think it's a site.pp change, and can go through gerrit like any other. [16:29:54] btw, is labsconsole totally busted for you right now? For me the sidebar is wrong, and it won't let me log in. [16:30:16] No it is a change on https://labsconsole.wikimedia.org/wiki/Special:NovaPuppetGroup [16:30:45] Oh, hm. I don't know who can/should change that, maybe best to check with Ryan. [16:31:54] At moment the right to change it only the cloudadmins has but the question is if it other can get the right like the new contentadmins [16:32:28] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:32:44] andrewbogott: You are right, labsconsole is not working... [16:33:46] It's in the middle of a puppet run, that's suspicious [16:34:38] Maybe the good, old reboot helps ;-) [16:36:18] paravoid: virt0 is doing a puppet run and I'm seeing lots of this: [16:36:18] from /var/lib/puppet/templates/ldap/nslcd.conf.erb:26 [16:36:25] /usr/lib/ruby/1.8/puppet/parser/functions/versioncmp.rb:30: warning: multiple values for a block parameter (2 for 1) [16:36:40] paravoid: Which, I think you just modified that file. Any thoughts? [16:37:43] I did? [16:37:48] no I didn't [16:38:11] commit 894cd51fd0c64271dd2047fbcf9d6cb895c1ed13 [16:38:11] Merge: 00858d0 12aee7b [16:38:12] Author: Faidon [16:38:18] Date: Wed Aug 29 14:42:04 2012 +0000 [16:38:18] andrewbogott: I can remember there was a ubuntu-update for nslcd that I installed on my instances [16:38:23] Oh, wait, it's not August is it? [16:38:33] Damn, nevermind, sorry paravoid :) [16:38:54] * andrewbogott goes back to having no idea what's wrong [16:39:38] i wish it was august :( [16:40:00] I guess it's been throwing that error for ages, so not the cause of the problem. [16:40:15] Jens_WMDE: Do you have snow, too? :-) [16:40:32] Pft... I can't log into labs anyways... [16:41:01] DanielFriesen: That's we are talking about [16:42:26] andre__: In theory to get added to a project you would ask a sysadmin of that project. Of course I can't tell you who that is because labsconsole is broken... [16:43:03] haha [16:44:08] paravoid: Could you rereview this change please: https://gerrit.wikimedia.org/r/#/c/29975/ [16:47:00] there you go :) [16:47:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [16:47:24] we can talk about it here too if you'd prefer that [16:47:39] rather than go into review ping-pong on gerrit (but I don't mind that either, whatever works for you best) [16:48:38] The classes should be used in labs so users can install this php-things with the instance-config [16:49:37] why? [16:49:45] I mean, usually you need those php extensions for something [16:50:29] if we go down this road we'll end up with thousands of 3-line classes [16:52:04] But there this should help users with less apt-knowhow to get things installed [16:52:49] I don't see how that's the case, has this happened? [16:52:57] e.g. there are many toolserver-users (which should switch in future to labs) that does not have this kind of linux-knowhow [16:53:19] what do they want exactly? [16:53:33] what would they expect from the system I mean [16:53:49] also, have in mind that labs instances need a sysadmin [16:54:17] Often they want a apache-server with a php and mysql-installation and some php-extensions [16:54:21] each project must have a sysadmin who at least knows very few basic things and we can provide some support to that [16:54:32] it doesn't have to be every /user/ of the project [16:55:02] okay, if there's a real use case for that then we can have a standard lamp labs class that installs apache/php/mysql or something [16:55:47] So they could easy install the webserver (webserver::php5) and mysql and then select the extensions they need and have a installion with all they need [16:56:06] there is one (webserver::php5 + webserver::php5-mysql) [16:56:23] but there are no classes for other extension that can be needed [16:56:46] like curl [16:57:16] I'd be open to have a more full-blown webserver php class that installs a bunch of extensions [16:58:01] I'm not saying no to that, but I'd prefer having an actual pattern of users needing that [16:58:03] I know that are many small classes but I don't know a better way to enable the selection of this extensions [16:58:52] I don't get why this needs puppetization tbh [16:59:14] either there's a certain combination of extensions needed for a project (hence a class) or sysadmins can manually install them [16:59:44] and if there are a bunch of users needing to work with a lot of extensions, maybe we can have a webserver::php5-heavy that installs all of them to the system [17:00:01] to get the config cloneable so you can easy setup a new instance, select the classes and you have the same system-setup [17:02:25] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:02:30] you can say the same for pretty much every package of the system, can't you? :-) [17:03:30] let's see a real pattern first and try to solve the problem then I'd say [17:03:35] (unless there's one already that I'm missing) [17:03:59] until then, I don't think it's too much to ask to ask projects' sysadmins do "apt-get install php5-curl" [17:04:11] I used this page as base for thing Toolserver users needed: http://www.mediawiki.org/wiki/Wikimedia_Labs/Toolserver_features_wanted_in_Tool_Labs [17:05:25] don't get me wrong, I'm grateful for having someone working on those issues [17:05:43] paravoid: Could you take a look at virt0 and see if you can tell why LabsConsole is refusing logins? I'm baffled. [17:05:54] I am also interested in your discussion with Jan_Luca but… distracted :( [17:06:18] andrewbogott: No problem, I can wait [17:06:41] andrewbogott: looking, although I know very little of labsconsole [17:08:07] paravoid: The problem started 30 or so minutes ago. Unfortunately the logs are so full of noise that it's hard to find anything interesting. [17:08:42] I think I know that this is :) [17:09:55] hm. not. [17:10:20] oh wait [17:10:20] works for me [17:10:25] I took the shotgun approach and restarted memcached [17:10:35] there was a memcached alert for virt0 half an hour ago [17:10:44] so it wasn't completely random [17:12:24] Hm, OK. an email alert? Or in wikimedia-operations? [17:12:35] the latter [17:13:01] ok. Thank you! Works for me too, now. [17:13:04] be back in 10' [17:17:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [17:21:34] Anyone around to help me get access to beta labs? [17:24:35] anomie: To get a user or to get a project [17:25:05] Jan_Luca: I think he means the beta project specifically [17:25:10] Jan_Luca- I'm already on Bastion. Robla wants me to get acquainted with Beta. [17:25:20] hi Roan [17:25:35] Good morning [17:26:06] Morning, for me it's got dark outside ;-) [17:27:37] Early afternoon here, aren't timezones fun ;) [17:28:42] anomie: When you need shell access you have to wait for a sysadmin of the Deployment-prep project [17:29:28] Change on 12mediawiki a page Developer access was modified, changed by Arjanizary link https://www.mediawiki.org/w/index.php?diff=598728 edit summary: [17:31:23] Change on 12mediawiki a page Developer access was modified, changed by Arjanizary link https://www.mediawiki.org/w/index.php?diff=598729 edit summary: /* User:Arjanizary */ [17:31:43] ok, I lied, 20 :P [17:32:17] paravoid: A moment please, there is law for this you have broken... [17:32:31] I'm sure there is somewhere [17:32:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:34:10] paravoid: If I unterstand you right, you would like something like a class php::wikimedia for wikimedia-extension and php::db for database extensions, ... [17:34:26] well, not exactly, no [17:34:56] I mean bigger classes so there are not so many small ones [17:35:13] that should be a general rule (with its exceptions), yes [17:35:14] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=598732 edit summary: /* User:Arjanizary */ done [17:35:18] hi wm-bot! [17:35:29] Hi robla. Trying to find out who to talk to to give me permission to log into beta. [17:35:34] but I'd also like to create classes when it makes sense [17:35:56] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: WARNING - load average: 11.12, 10.17, 7.24 [17:35:56] lemme see if I can do it.... [17:36:11] I thought chrismcmahon had the rights to do that [17:36:11] anomie: i haven't read much scrollback but... "access" and "log in" are ambiguous [17:36:32] There is a need for php [17:36:50] jeremyb- What should I say then? [17:36:59] robla: I'm not sure how to grant shell access on the deployment cluster [17:37:05] having a project where the person designated as the sysadmin doesn't know how to run a simple apt-get install is not going to work [17:37:11] and with this classes we can apply wm- or labs-settings for the php things [17:38:00] anomie: well, first of all log in to what? the web interface of the wiki? the shell? [17:38:03] paravoid: Is your argument roughly that the php module is so trivial that such work should just be duplicated in any class that needs those packages? [17:38:56] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 9.22, 8.82, 6.10 [17:39:12] Yes, but then we wouldn't need a class like webserver::apache2 or webserver::php5 because this is not very different from the package "apache2" and "libapache2-mod-php5" [17:39:16] andrewbogott: why is this page blank https://labsconsole.wikimedia.org/w/index.php?title=Special:NovaInstance&action=consoleoutput&project=mobile&instanceid=6e03e8b9-8d22-4483-ba0a-bfb701e72450®ion=pmtpa [17:39:52] PROBLEM host: i-000004f3.pmtpa.wmflabs is DOWN address: i-000004f3.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000004f3.pmtpa.wmflabs) [17:40:32] prielly: I can't see that page without adding myself to your project. But… is it for an instance that you just now created? [17:40:56] andrewbogott: it's for an instance that I created 10 minutes ago or so [17:40:56] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 8.04, 7.75, 5.79 [17:40:56] andrewbogott: is labs down? [17:42:08] No, should be working. I'll create an instance and see if I have any better luck. [17:42:24] preilly, have you had luck creating other instances in that project? (Just wondering if maybe there's still a security problem.) [17:42:26] andrewbogott: basically, yes. I also think that we can't plan for puppet classes to handle the scenario "the project sysadmin doesn't know how to run apt-get" [17:42:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 8.67, 8.19, 5.98 [17:43:10] andrewbogott: yes, I've created other instances in this group…. it's the mobile group [17:43:51] paravoid: apt-get is a different question from having a base class for re-use in roles or higher-level classes. So I'm confused about which of those things we're talking about. [17:43:53] RECOVERY host: i-000004f3.pmtpa.wmflabs is UP address: i-000004f3.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 0.59 ms [17:44:21] andrewbogott: We talk about my php change [17:44:23] PROBLEM Total processes is now: CRITICAL on staging i-000004f3.pmtpa.wmflabs output: Connection refused by host [17:44:37] Jan_Luca: Right, which was a refactor so that you could reuse those classes in a different class, right? [17:44:40] andrewbogott: I'm contesting against dozens of small php classes that they just install the respective package, because users don't know how to run apt-get [17:45:05] not as far as I understood it, Jan wants them to be options in labsconsole [17:45:05] chrismcmahon: visit here: https://labsconsole.wikimedia.org/wiki/Special:NovaProject [17:45:11] I want to use this php-classes on the one side for users to install php extensions and the other side to use it in other classes [17:45:49] (for example see my change: https://gerrit.wikimedia.org/r/#/c/30147/ ) [17:45:52] PROBLEM Current Load is now: CRITICAL on staging i-000004f3.pmtpa.wmflabs output: Connection refused by host [17:45:52] PROBLEM dpkg-check is now: CRITICAL on staging i-000004f3.pmtpa.wmflabs output: Connection refused by host [17:46:32] PROBLEM Current Users is now: CRITICAL on staging i-000004f3.pmtpa.wmflabs output: Connection refused by host [17:47:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [17:47:20] PROBLEM Disk Space is now: CRITICAL on staging i-000004f3.pmtpa.wmflabs output: Connection refused by host [17:48:04] PROBLEM Free ram is now: CRITICAL on staging i-000004f3.pmtpa.wmflabs output: Connection refused by host [17:48:04] chrismcmahon: if you get an error about permissions, log out and back in (session bug) [17:48:04] chrismcmahon: once you have the project list, find "deployment-prep", add Anomie as a member of the project, then add him as a sysadmin and netadmin [17:49:19] !htmllogs [17:49:19] experimental: http://bots.wmflabs.org/~wm-bot/html/%23wikimedia-labs [17:49:20] preilly: I am investigating, will let you know when I know more. [17:50:09] preilly: Actually, I think maybe we were just swamped. Is your console log still empty? [17:50:16] andrewbogott: thanks [17:50:22] 10/29/2012 - 17:50:22 - Updating keys for cmcmahon at /export/keys/cmcmahon [17:50:46] andrewbogott: I've got output in the console log now [17:50:53] PROBLEM Current Load is now: CRITICAL on testlabs-createtest i-000004f4.pmtpa.wmflabs output: Connection refused by host [17:51:03] PROBLEM Free ram is now: CRITICAL on testlabs-createtest i-000004f4.pmtpa.wmflabs output: Connection refused by host [17:51:29] paravoid: What's about making the php-module smaller so there are some classes that can be used in other classes (like php::mysql in the webserver-classes) and some classes for easy setup the instance (like php::mostused or something like that) [17:51:33] PROBLEM Current Users is now: CRITICAL on testlabs-createtest i-000004f4.pmtpa.wmflabs output: Connection refused by host [17:51:51] robla anomie I think I've added you to deployment-prep group, thanks robla [17:52:13] PROBLEM Disk Space is now: CRITICAL on testlabs-createtest i-000004f4.pmtpa.wmflabs output: Connection refused by host [17:52:23] PROBLEM Total processes is now: CRITICAL on testlabs-createtest i-000004f4.pmtpa.wmflabs output: Connection refused by host [17:52:57] PROBLEM dpkg-check is now: CRITICAL on testlabs-createtest i-000004f4.pmtpa.wmflabs output: Connection refused by host [17:55:53] RECOVERY Current Load is now: OK on staging i-000004f3.pmtpa.wmflabs output: OK - load average: 1.49, 1.39, 0.89 [17:55:53] RECOVERY dpkg-check is now: OK on staging i-000004f3.pmtpa.wmflabs output: All packages OK [17:56:32] RECOVERY Current Users is now: OK on staging i-000004f3.pmtpa.wmflabs output: USERS OK - 1 users currently logged in [17:57:12] RECOVERY Disk Space is now: OK on staging i-000004f3.pmtpa.wmflabs output: DISK OK [17:58:02] RECOVERY Free ram is now: OK on staging i-000004f3.pmtpa.wmflabs output: OK: 416% free memory [17:59:23] RECOVERY Total processes is now: OK on staging i-000004f3.pmtpa.wmflabs output: PROCS OK: 90 processes [17:59:45] paravoid: ? [18:00:00] Jan_Luca: We have a team meeting now, everyone will be distracted for a bit. [18:01:35] RECOVERY Current Users is now: OK on testlabs-createtest i-000004f4.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [18:02:12] RECOVERY Disk Space is now: OK on testlabs-createtest i-000004f4.pmtpa.wmflabs output: DISK OK [18:02:21] andrewbogott: Thank you for the info, then I go with the dogs in the mean time [18:02:22] RECOVERY Total processes is now: OK on testlabs-createtest i-000004f4.pmtpa.wmflabs output: PROCS OK: 84 processes [18:02:52] RECOVERY dpkg-check is now: OK on testlabs-createtest i-000004f4.pmtpa.wmflabs output: All packages OK [18:03:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:03:52] PROBLEM dpkg-check is now: CRITICAL on staging i-000004f3.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [18:05:52] RECOVERY Current Load is now: OK on testlabs-createtest i-000004f4.pmtpa.wmflabs output: OK - load average: 0.12, 0.67, 0.71 [18:06:02] RECOVERY Free ram is now: OK on testlabs-createtest i-000004f4.pmtpa.wmflabs output: OK: 1051% free memory [18:14:14] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:17:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [18:33:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:44:09] petan: ping [18:44:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:44:39] how can i recompile wmib? is there a compiler on bots-1 ? [18:44:54] or do you do it locally? [18:47:02] i guess you're not there [18:47:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [18:47:31] wanted to fix metawiki's RC channel on the irc.wikimedia side. which i'm pretty sure is hardcoded and wrong [18:47:38] s/wikipedia/wikimedia/ [18:50:39] petan: also can we mvoe from svn to gerrit [18:50:39] ? [18:54:06] move* [19:04:06] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:07:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [19:07:53] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 4.01, 4.58, 4.92 [19:14:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:17:07] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [19:18:53] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 5.44, 4.91, 4.98 [19:20:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 5.82, 5.36, 5.13 [19:25:52] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 4.39, 4.83, 4.98 [19:34:06] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:46:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:47:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [19:57:57] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 5.06, 5.06, 5.02 [20:02:32] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 293 processes [20:04:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:07:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [20:16:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:18:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [20:22:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.11, 4.58, 4.95 [20:34:46] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:44:22] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 154 processes [20:46:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:48:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [20:50:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 1.01, 2.23, 4.11 [21:05:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:13:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 10.62, 8.97, 6.25 [21:16:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:17:21] took many instances dead :-] [21:19:14] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [21:35:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:46:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:48:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 13.36, 13.32, 12.60 [21:49:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [21:50:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 8.88, 8.82, 8.61 [22:02:32] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 293 processes [22:05:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:11:10] !log deployment-prep Added anomie as a sysadmin (for sudo) and netadmin [22:11:12] Logged the message, Master [22:12:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [22:12:37] anomie: whenever we do something we try to log it in this channel using the !log command [22:12:42] anomie: that ends up on https://labsconsole.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [22:12:51] anomie: syntax is: !log deployment-prep [22:12:56] !log hashar cookies++ [22:12:57] hashar is not a valid project. [22:13:10] hashar gets no cookies [22:13:25] Damianz: ahh ! ;-] [22:13:55] Damianz: since you have some knowledge about nagios, could you make it so that when a host goes down it does not gives the hostname three times? Example: [22:14:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [22:14:03] ;-] [22:14:12] hashar- I'll write that down. Tomorrow we can help me figure out how to make changes correctly (: [22:14:16] yeah [22:14:26] when I can be bothered to login to the server [22:14:31] anomie: yeah I guess the best would be to make you do a configuration change [22:14:32] * Damianz is still doing w0rk [22:14:46] that will cover everything from sending a patch to gerrit, connecting to labs and deploying it [22:14:59] Damianz: just wanted to let you know :-] [22:15:05] no hurry [22:15:08] Sure [22:15:20] I collect things that are in no hurry :) [22:17:02]