[00:00:04] but you'll need to transfer it from bastion to your instance [00:00:11] so in winscp go to 'proxy' under connection, stick bastion in there, under session stick the instance details and if it doesn't work ask google [00:00:17] really ? [00:00:20] https://kb.bluecoat.com/index?page=content&id=KB5282&actp=RSS < totally asked google and it didn't help [00:00:24] well sorta helped [00:00:25] oh. it has a proxy option? [00:00:26] <3 screenshots [00:00:27] do you use it that way ? [00:00:34] Nah I use OSX/Linux [00:00:39] k [00:01:04] http://winscp.net/eng/docs/guide_tunnel#setting_up_tunnel_using_putty_for_sftp_scp_session [00:01:07] HEY [00:01:08] IT'S IN THE DOCS [00:01:29] uh [00:01:41] something for tomorrow [00:01:54] is there a log of this channel? [00:02:08] !logs [00:02:08] logs http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-labs [00:02:16] k [00:04:07] I swaer I start feeling awake at midnight -.- [00:06:05] Ryan_Lane: Vacation9: just to let you know, I am not only on Windows, but also running several (>10) Linux systems [00:06:24] Wikinaut: wow [00:07:57] Irony is I'm on linux running ~4 windows vms [00:19:35] Ryan_Lane: Vacation9 WORKS !!! [00:19:41] no probs [00:19:46] will document [00:19:49] my steps [00:19:58] WinSCP -> bastion -> instance [00:20:20] you can upload directly to the instance? [00:20:23] correct help page is http://winscp.net/eng/docs/ui_login_tunnel [00:21:38] Ryan_Lane: [00:22:34] just to be sure: can I now (or later) install E:OpenID on that Wiki [00:22:49] I suppose, this is what you want me to do [00:25:53] PROBLEM Free ram is now: CRITICAL on lucene-test1.pmtpa.wmflabs 10.4.0.205 output: Connection refused by host [00:25:53] PROBLEM Current Load is now: CRITICAL on lucene-test1.pmtpa.wmflabs 10.4.0.205 output: Connection refused by host [00:26:33] PROBLEM Current Users is now: CRITICAL on lucene-test1.pmtpa.wmflabs 10.4.0.205 output: Connection refused by host [00:27:13] PROBLEM Disk Space is now: CRITICAL on lucene-test1.pmtpa.wmflabs 10.4.0.205 output: Connection refused by host [00:27:23] PROBLEM Total processes is now: CRITICAL on lucene-test1.pmtpa.wmflabs 10.4.0.205 output: Connection refused by host [00:28:18] PROBLEM dpkg-check is now: CRITICAL on lucene-test1.pmtpa.wmflabs 10.4.0.205 output: Connection refused by host [00:29:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 9.83, 8.27, 6.39 [00:33:13] RECOVERY dpkg-check is now: OK on lucene-test1.pmtpa.wmflabs 10.4.0.205 output: All packages OK [00:34:30] !logs [00:34:30] logs http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-labs [00:35:53] RECOVERY Free ram is now: OK on lucene-test1.pmtpa.wmflabs 10.4.0.205 output: OK: 1046% free memory [00:35:53] RECOVERY Current Load is now: OK on lucene-test1.pmtpa.wmflabs 10.4.0.205 output: OK - load average: 0.08, 0.67, 0.57 [00:36:33] RECOVERY Current Users is now: OK on lucene-test1.pmtpa.wmflabs 10.4.0.205 output: USERS OK - 0 users currently logged in [00:37:13] RECOVERY Disk Space is now: OK on lucene-test1.pmtpa.wmflabs 10.4.0.205 output: DISK OK [00:37:23] RECOVERY Total processes is now: OK on lucene-test1.pmtpa.wmflabs 10.4.0.205 output: PROCS OK: 84 processes [00:37:53] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 28% free memory [00:39:26] hey, as I am now "in" my instance, and want to install extensions, I need to become "root". How is this managed? [00:40:05] sudo -i [00:40:12] ok [00:40:13] then enter your ldap password [00:40:19] which is the one you used on labsconsole wiki [00:40:26] k [00:40:37] is this documented, I don't think so [00:40:42] it is somewhere :) [00:40:46] but be VERY careful as root [00:40:49] :-)) [00:40:53] yes [00:40:54] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 8% free memory [00:40:58] I do know what I am doin [00:41:00] g [00:41:04] good :P [00:41:18] this is why I mentioned that I maintain > 10 servers [00:41:24] didn't see that part :) [00:41:32] I literally just now joined the channel [00:43:54] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 7.08, 7.09, 5.68 [00:44:18] fox_wilson: I also mentioned that I will document how to use WinSCP & Tunnel to an instance [00:44:47] oh that's good [00:44:50] but i missed that too :) [00:44:55] it's working w/o problems [00:45:00] not yet documented [00:45:19] docs on that would be nice [00:52:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 6.76, 6.20, 5.37 [00:58:44] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 5.88, 6.16, 5.34 [01:05:52] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 5% free memory [01:10:44] PROBLEM Total processes is now: WARNING on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS WARNING: 167 processes [01:15:43] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 92 processes [01:15:53] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 8% free memory [01:38:43] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 3.72, 3.86, 4.81 [01:39:47] I have good news for the few Windows users of you. [01:39:49] I succeeded in accessing my labsconsole VM using [01:39:50] PuTTY and [01:39:51] WinSCP [01:39:53] with appropriate proxy respectively tunnel settings via the bastion server. [01:39:54] This will be documented in the next days (sorry, not earlier) on [01:39:56] https://labsconsole.wikimedia.org/wiki/Help:Access [01:39:58] and https://labsconsole.wikimedia.org/wiki/Help:Putty . [01:39:59] With the correct settings, a single click connection or file transfer is possible, and no command line input trouble is needed. [01:40:26] I think it's worth to be posted in the wikitech-l [01:41:53] PROBLEM Total processes is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS WARNING: 152 processes [01:42:43] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 5.68, 4.25, 4.89 [01:50:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 3.94, 6.04, 5.73 [01:51:44] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 5.16, 5.44, 5.22 [01:56:53] RECOVERY Total processes is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS OK: 148 processes [02:23:52] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 3.93, 3.87, 4.72 [02:39:55] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 7.85, 7.08, 5.89 [02:55:52] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 6% free memory [03:04:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 4.48, 5.10, 5.50 [03:05:44] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 7.87, 7.16, 6.92 [03:06:04] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 5% free memory [04:04:42] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 4.01, 3.94, 4.92 [04:12:32] PROBLEM dpkg-check is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: CHECK_NRPE: Error - Could not complete SSL handshake. [04:12:42] PROBLEM Disk Space is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: CHECK_NRPE: Error - Could not complete SSL handshake. [04:13:32] PROBLEM SSH is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Server answer: [04:14:32] PROBLEM Total processes is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: CHECK_NRPE: Error - Could not complete SSL handshake. [04:17:23] RECOVERY Free ram is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: OK: 49% free memory [04:17:33] RECOVERY dpkg-check is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: All packages OK [04:17:43] RECOVERY Disk Space is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: DISK OK [04:18:33] RECOVERY SSH is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [04:19:33] RECOVERY Total processes is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: PROCS OK: 144 processes [04:40:24] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 21% free memory [04:40:54] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 8% free memory [04:45:45] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 2.91, 3.41, 4.43 [04:54:53] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 4.63, 4.54, 4.99 [04:58:22] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 15% free memory [05:05:53] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 5% free memory [05:10:53] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 6% free memory [06:10:53] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 5% free memory [06:29:55] PROBLEM Total processes is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS WARNING: 152 processes [06:46:02] PROBLEM Free ram is now: WARNING on spellcheckself-bot.pmtpa.wmflabs 10.4.0.246 output: Warning: 19% free memory [06:49:52] RECOVERY Total processes is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS OK: 148 processes [06:51:02] RECOVERY Free ram is now: OK on spellcheckself-bot.pmtpa.wmflabs 10.4.0.246 output: OK: 33% free memory [06:56:12] PROBLEM dpkg-check is now: CRITICAL on lucene-test1.pmtpa.wmflabs 10.4.0.205 output: DPKG CRITICAL dpkg reports broken packages [07:01:13] RECOVERY dpkg-check is now: OK on lucene-test1.pmtpa.wmflabs 10.4.0.205 output: All packages OK [07:55:52] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 7% free memory [08:10:52] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 5% free memory [08:10:52] !logs [08:10:53] logs http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-labs [08:30:52] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 6% free memory [08:38:22] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 21% free memory [08:40:52] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 20% free memory [08:48:53] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 8% free memory [09:06:23] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 15% free memory [09:55:52] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 5% free memory [10:25:53] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 7% free memory [10:30:52] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 5% free memory [10:39:42] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.58, 4.76, 4.91 [11:30:53] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 6% free memory [11:32:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.10, 5.13, 5.05 [11:42:44] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.71, 4.83, 4.93 [12:30:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.34, 5.30, 5.16 [12:38:52] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 28% free memory [12:41:27] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 20% free memory [12:49:22] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 14% free memory [12:56:53] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 8% free memory [13:15:53] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 6% free memory [13:45:22] !log huggle petrb: inserting es.wikivoyage to global config [13:45:24] Logged the message, Master [14:07:44] PROBLEM Free ram is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Warning: 19% free memory [14:25:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.14, 5.18, 5.07 [14:30:42] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.81, 4.94, 5.00 [14:59:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 4.93, 5.11, 5.05 [16:38:43] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 154 processes [16:39:23] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 23% free memory [16:41:54] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 20% free memory [17:02:23] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 14% free memory [17:09:54] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 8% free memory [17:33:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 4.99, 5.07, 5.04 [17:43:44] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 149 processes [17:56:43] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 154 processes [18:05:39] !log bots Created shared Pywikipedia repositories at /data/project/pywikipedia/trunk and /data/project/pywikipedia/rewrite (you'll still need separate user-config files and so on for each bot) [18:06:50] russblau: is it autoupdated? [18:21:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.01, 5.01, 5.05 [18:25:59] giftpflanze: I'm working on that now; trying to figure out how to get a cron job to run as root... [18:26:42] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 148 processes [18:26:56] sounds like a bad idea [18:46:20] giftplanze: solved; put the update script in /etc/cron.daily, no further sudoing required [18:58:10] heh [19:03:44] !log russblau created /etc/cron.daily/pywikipedia to update /data/project/pywikipedia/{trunk,rewrite} [19:03:45] russblau is not a valid project. [19:03:53] !log bots russblau created /etc/cron.daily/pywikipedia to update /data/project/pywikipedia/{trunk,rewrite} [19:03:55] Logged the message, Master [19:20:30] hey Ryan_Lane, yt? [19:23:14] Hello andrewbogott :) My variables are submitted to gerrit now. [19:23:15] sup? [19:23:23] ori-l: ^^ [19:23:33] Ryan_Lane: just looking at this: http://packages.python.org/GitPython/0.3.2/reference.html#git.remote.RemoteProgress [19:23:50] "Handler providing an interface to parse progress information emitted by git-push and git-fetch and to dispatch callbacks allowing subclasses to react to the progress." [19:23:52] Silke_WMDE: OK, I will read. [19:23:57] Would be great if you could look at them! And have a nice weekend! [19:23:59] cool [19:24:02] specifically: update(), "Called whenever the progress changes" [19:24:03] CU [19:24:16] ori-l: GitPython shells out to git [19:24:26] so it has the same limitations [19:24:26] wat. lame. [19:24:33] dulwich, on the other hand.... [19:24:40] dulwich is native [19:24:54] it may be a solution to the problem, but that's definitely a long term goal ;) [19:25:16] oh, dulwich is cool, didn't know about it [19:25:19] thanks for the pointer! [19:25:29] is fauklner the contact person? [19:25:31] Ryan_Lane, can you explain about the live hack on virt0 that forces showlinks in getGroupList() to true? [19:25:58] Is that pending a real fix to the line you removed, or do we want those links to be present for non-sysadmins, or... [19:26:56] there's a live hack there? [19:27:38] http://dpaste.org/EDw0h/ [19:28:29] oh [19:28:37] I think I pushed that up and merged it [19:28:49] ok… i'll rebase, maybe it'll disappear. [19:29:21] I changed it from the old wya to the new way of handling links [19:29:40] yep, indeed that change seems to be in git. So, nevermind! [19:30:26] I should have rebased that [19:33:41] I just merged a patch that should create automatic shell requests. The shell request queue is empty at the moment, so mike must be taking care of it. [19:34:10] sweet [19:34:19] I've been picking them up, too [19:35:00] Is deleting test accounts on production easy, or do they tend to leave cruft about? [19:35:10] meh, no need to delete them [19:35:19] leaves cruft around [19:35:50] deletion is difficult for a number of reasons. mediawiki is especially painful in this regard [19:36:12] just make sure it doesn't have shell [19:36:38] * Damianz gives mike cookies [19:37:07] I just have to remember not to leave my test accounts password as test on production :P [19:37:42] This is non-critical so rather than create test accounts I can just wait for self-registering people to test it for me [19:39:02] andrewbogott: well, you tested it in the labs instance, so that seems fine to me [19:39:08] yep [19:45:51] Ryan_Lane, *.beta.wmflabs.org points at 208.80.153.219 (which is the beta squid instance) but I don't see an entry for *.beta on the web interface. Any idea where that's being defined? [19:45:57] Does beta have its own DNS server? [19:46:11] no [19:46:45] Hm, I wonder how it knows. [19:47:09] In theory we want *.m.beta to direct to a different host than *.beta. It's unclear how pdns will handle that ambiguity. [19:47:21] Seems to be doing it wrong, although maybe my caches are just stale. [19:48:11] andrewbogott: oh [19:48:13] well.... [19:48:16] the dns code sucks :) [19:48:19] it's probably a bug [19:48:28] that code hasn't been touched in ages [19:48:39] since I was hoping to replace it with something openstack natiev [19:48:42] *native [19:48:46] Sure. [19:48:49] occasionally entries won't show in the interface [19:48:52] it'll be in ldap, though [19:49:24] But at some point it does come down to pdns behavior, right? Would you expect it to check for more specific addresses before less-specific? [19:49:40] (Um, two questions here, you have satisfied me that I shouldn't worry about the missing entry in the web interface.) [19:51:56] it should check for more specific addresses first [19:52:08] so, test.beta.wmflabs.org entry would match before * [19:52:23] great, that's what we want. [19:52:40] I'll give it a few minutes and check back [19:54:13] is it not working currently? [19:54:20] it's possible that it's cached [19:54:40] you can check directly against the server, though [19:54:45] dig @labs-ns0.wikimedia.org [19:55:15] Actually, it just started working. [19:55:20] it's fun when you do that and 0 works and 2 doesn't [19:55:22] s/2/1 [19:55:44] So, all set. (But good to know that about dig, I barely use it but should learn more.) [19:56:15] dig is like the mother fudging tool when it comes to dns [19:58:03] <3 dig [20:07:54] PROBLEM Total processes is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS WARNING: 155 processes [20:08:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 6.79, 5.66, 5.26 [20:32:54] RECOVERY Total processes is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS OK: 135 processes [20:37:23] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 20% free memory [20:37:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 7.30, 7.33, 5.87 [20:38:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 7.70, 7.23, 5.56 [20:39:23] Ryan_Lane how fast are vd devices [20:39:34] are they directly mounted to vm? [20:39:40] from barebone hardware [20:39:53] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 28% free memory [20:40:05] I mean, is it OK to use them for swap? [20:40:42] some machines on bots are having ram utilized on 90% and I would like to prevent OOM crash by making swap [20:47:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 8.83, 7.35, 5.72 [20:50:22] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 19% free memory [20:50:53] PROBLEM Total processes is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS WARNING: 153 processes [20:52:52] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 8% free memory [21:02:53] PROBLEM Current Load is now: WARNING on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: WARNING - load average: 7.79, 8.60, 6.31 [21:06:33] petan: yes [21:06:36] petan: it's fine [21:06:42] petan: they are local disks, same as / [21:06:50] ok [21:08:13] ah ryan!! ;-] [21:08:18] hashar: ? [21:08:24] I managed to deploy some mediawiki-config change on beta using …. wait for it [21:08:28] git-deploy !!!!!!!! [21:08:30] so happy :-] [21:08:49] so hmm yeah kudos [21:09:04] and you / someone got all the links pointing to /srv/deployment/mediawiki [21:09:09] huge improvement [21:09:10] love it [21:09:11] happy [21:09:42] some love their gf's hashar git-deploy [21:10:35] hashar: :D [21:10:43] Ryan_Lane: "User wikinaut is not allowed to run sudo on openid-wiki." I sent you mail asking for root access; can you please set it up ? [21:10:51] Wikinaut: ah. crap. right [21:10:59] I forgot new projects don't have a default sudo policy [21:11:01] no crap [21:11:04] andrewbogott: ^^ [21:11:06] * petan is going to replace git-deploy with "echo hashar, would you marry me?" [21:11:06] it's fully okay [21:11:14] it's normal [21:11:31] better this way [21:11:39] petan: I am still not married mind you :-] [21:11:40] than opening by default [21:11:43] Wikinaut: I am sure you are going to love labs :D [21:11:49] I do love it [21:11:51] andrewbogott: that should be a pretty easy feature to add, if you're not crazy busy with something else [21:12:08] hashar: I already manage > 10 server (VM) elsewhere [21:12:15] much better than a PC at home [21:12:34] "headless" VMs [21:12:41] no head - no trouble [21:12:52] I already manages 10 VM servers in my PC at home :P [21:12:55] * manage [21:13:04] uh [21:13:06] nah kidding :D [21:13:37] the only thing is, that the access is a little bit slow [21:13:42] but I am pretty sure there is a rack full of boxes next to Damianz's bed [21:14:01] boxes with girls ? [21:14:08] boxes with hardware [21:14:09] Nah, there's a load on my work laptop though [21:14:22] petan's just jealous of my huge rack :o [21:14:28] yes [21:14:32] :D [21:14:46] when I grow up I will have such a big rack as well [21:14:58] hashar: I wish you would code-review my latest patchset on E:RSS [21:15:01] Big racks are snuggly [21:15:14] or make me owner [21:15:22] (request is pending) [21:15:47] hashy is awesome and buuusy [21:16:11] Hashar: https://www.mediawiki.org/wiki/Git/Gerrit_project_ownership#Wikinaut_on_Extension:RSS [21:16:12] Wikinaut: hoo the RSS extension :/ [21:16:21] Wikinaut: yeah that need some help :-) [21:16:24] What? It works [21:16:32] no help. It works [21:16:37] was reviewed once with Brion [21:16:55] but of course... [21:17:26] ..any code review is appreciated [21:17:32] I want to have committed [21:17:46] but do not want to be pushy [21:17:51] (uh, I am) [21:18:26] there's a lot of other (shitty) extensions [21:18:31] !log deployment-prep running puppet on cache-bits03 to find out whether role::cache::bits cleanly apply there. [21:18:33] lqt [21:18:34] Logged the message, Master [21:19:44] Wikinaut: is the RSS extension deployed on the cluster? [21:19:50] I mean the wikimedia servers [21:20:24] !log deployment-prep deployment-varnish-t : apt-get upgrade [21:20:25] Logged the message, Master [21:21:07] hashar: ahem, do not understand. The best (my last version changes were reverted during the switchover to git/gerrit) [21:21:36] and I want to have the "pending" changes code-reviewed, so that this rest can be pushed to git [21:21:45] (is this correct wording?) [21:22:19] This one https://gerrit.wikimedia.org/r/#/c/3925/ [21:22:53] RECOVERY Current Load is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: OK - load average: 4.23, 4.16, 4.75 [21:23:20] hashar: just the wording: what do you my by "deployed on the cluster"? [21:23:31] s/my/mean/ [21:23:49] is RSS used on wikimedia servers ? [21:23:54] such as de.wikipedia.org ? :D [21:27:56] hashar: on mediawiki.org [21:28:01] the OLD version [21:28:14] PROBLEM dpkg-check is now: CRITICAL on deployment-varnish-t.pmtpa.wmflabs 10.4.1.74 output: DPKG CRITICAL dpkg reports broken packages [21:28:25] I will also deploy it on my (new :-) openid [21:28:34] openid-wiki@labs [21:29:09] once Ryan_Lane has configured it for me so that I can be root [21:29:19] oh [21:29:20] sorry [21:29:22] I did it [21:29:30] https://labsconsole.wikimedia.org/wiki/Nova_Resource:I-00000589 [21:29:33] (mom) [21:29:37] you can sudo now [21:30:12] confirmed! [21:30:20] was this quick enough? [21:30:29] ty [21:31:06] You know when you where suppose to do something really important but can't remember wtf it was? [21:31:11] currently, I document in screenshots how to single-click PuTTY and WinSCP to an instance [21:31:32] for the Windows-"terminal" users [21:31:39] pls. apologise [21:32:12] yay, totally need more windows docs since a large portion of the people that write tech docs don't use windows [21:35:52] PROBLEM Current Load is now: WARNING on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: WARNING - load average: 2.80, 5.67, 5.86 [21:38:06] hey Damianz :) [21:38:29] * Damianz looks at MJ94 [21:38:41] yeh [21:53:13] RECOVERY dpkg-check is now: OK on deployment-varnish-t.pmtpa.wmflabs 10.4.1.74 output: All packages OK [21:55:57] here's the first part (PuTTY) https://labsconsole.wikimedia.org/wiki/User:Wikinaut/Help:Access_to_instances_with_PuTTY_and_WinSCP [21:57:40] Wikinaut: Thanks for documenting! [21:58:02] :-) [21:58:14] WinSCP follows soon [21:58:21] single-click solutions ! [21:58:45] Wikinaut, does your new page supersede this one? https://labsconsole.wikimedia.org/wiki/Help:Putty [21:58:53] If so, can you coordinate them or link between them or something? [21:59:14] andrewbogott: slowly [21:59:17] yes [21:59:28] but first I want to set up my page [21:59:30] 'k -- as long as you know that other page is there I'm happy [21:59:43] it#s already in teh "references" [21:59:55] probably you have overlooked the section at the end [22:00:17] this isone of the best pages http://mikelococo.com/2008/01/multihop-ssh/ [22:00:23] of course, also linked [22:03:19] Ryan_Lane: I see *** System restart required *** in my PuTTY window [22:03:35] on the instance. shall I reboot [22:03:37] ? [22:03:39] that's because it's applied security patches that require a reboot [22:03:40] yes [22:04:47] reboots? isn't that why we let Ryan do maintaince [22:04:51] (rebooting now) [22:05:02] Damianz: so many machines [22:05:14] Ryan_Lane has to do certainly other things... [22:05:35] He makes everything reboot though :D [22:06:46] Damianz: ;) [22:07:28] grrr windows, stupid stupid system [22:24:18] can someone pls. switch off this stupid captcha when inserting external links on pages https://labsconsole.wikimedia.org/ [22:25:45] et volià : https://labsconsole.wikimedia.org/wiki/User:Wikinaut/Help:Access_to_instances_with_PuTTY_and_WinSCP#Prerequisites [22:32:44] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 4.15, 4.22, 4.66 [22:36:10] has anyone questions to me, otherwise I'll take a break [22:37:34]