[09:07:12] !log deployment-prep Running update.php on all databases. [09:07:15] Logged the message, Master [11:18:29] !log deployment-prep migrating memcached from -mc to the apaches boxes. {{gerrit|49261}} [11:18:31] Logged the message, Master [11:19:20] pff [11:19:25] gluster is unusuable again [11:21:09] [2013-02-18 11:20:59.036164] I [glusterfsd-mgmt.c:65:mgmt_cbk_spec] 0-mgmt: Volume file changed [11:21:10] [2013-02-18 11:20:59.141540] I [glusterfsd-mgmt.c:65:mgmt_cbk_spec] 0-mgmt: Volume file changed [11:21:11] [2013-02-18 11:20:59.176934] I [glusterfsd-mgmt.c:1569:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [11:21:11] [2013-02-18 11:20:59.216349] I [glusterfsd-mgmt.c:1569:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [11:21:12] spppam :_D [11:21:55] !log deployment-prep rebooting -bastion [11:21:57] Logged the message, Master [11:25:42] oh my [11:25:45] seriouslyl [11:26:34] !log deployment-prep got Gluster client upgraded on -bastion [11:26:36] Logged the message, Master [11:27:38] !log deployment-prep rebooting -bastion again [11:27:40] Logged the message, Master [11:35:05] !log deployment-prep Deleting -mc instance, memcached is now on apaches [11:35:07] Logged the message, Master [11:36:45] !ping [11:36:45] pong [11:38:48] [bz] (8NEW - created by: 2Antoine "hashar" Musso, priority: 4Normal - 6enhancement) [Bug 36994] [OPS] Add disk I/O to ganglia reports - https://bugzilla.wikimedia.org/show_bug.cgi?id=36994 [11:45:01] !log nagios petrb: restarting feed [11:45:03] Logged the message, Master [12:13:42] bluh [12:14:42] :o [12:20:57] O_o [12:21:17] Don't tell me that that is the reason my bots have been crashy again lately ... [12:21:21] * Beetstra investigates [12:37:53] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 22% free memory [12:40:12] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [12:53:12] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [13:05:53] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 18% free memory [13:10:11] !log wiktionary-dev locale-gen en_US.UTF-8 [13:10:11] wiktionary-dev is not a valid project. [13:10:27] !log wiktionary-tools locale-gen en_US.UTF-8 [13:11:53] Hm? No log? [13:14:12] !log wiktionary-tools locale-gen en_US.UTF-8 [13:16:06] ...How can I hope to work on this if I can't even log properly :P [13:26:04] !log wiktionary-tools apt-get install source-highlight [13:26:06] Logged the message, Master [13:26:12] aha [13:26:58] !log wiktionary-tools locale-gen en_US.UTF-8 [13:26:59] Logged the message, Master [13:27:16] good bot [14:08:18] !log deployment-prep applying the new squid::redirector class to deployment-squid so we can handle mobile redirects [14:08:20] Logged the message, Master [14:13:40] !log deployment-prep fixed puppet on -squid, it was blocked by attempting to apply a non existent class: generic::package::git-core [14:13:42] Logged the message, Master [14:18:42] [bz] (8NEW - created by: 2Marc A. Pelletier, priority: 4Unprioritized - 6enhancement) [Bug 45119] Add per-project service/role user accounts and groups - https://bugzilla.wikimedia.org/show_bug.cgi?id=45119 [14:22:04] !log deployment-prep wikidatawiki is missing oh no beta dead again [14:22:12] Logged the message, Master [14:23:02] * Coren looks at his usernames. [14:23:47] Gah, I have things named 'Coren', 'marc', 'MPelletier', 'Marc A. Pelletier' and 'Marc-André Pelletier' in different places. [14:24:18] SUL this is not. :-) [14:24:22] PROBLEM Total processes is now: WARNING on deployment-squid.pmtpa.wmflabs 10.4.0.17 output: PROCS WARNING: 199 processes [14:25:30] Coren: na that is a long standing issue :( [14:25:43] Coren: welcome among the WMF contractor btw :-D [14:25:47] <-- Antoine Musso [14:25:50] hashar: Thanks. :-) [14:25:53] I am in charge of continuous integration tools [14:25:57] and fluent in french [14:26:02] well [14:26:09] I really only speak / write french [14:26:13] hashar: So you're the one to blame if Jenkins goes berzerk? :-) [14:26:18] but do write some worldglish [14:26:24] yeah \O/ [14:26:34] though most of the time the issue is in the submitted code, not in Jenkins [14:26:36] luckily [14:26:44] I do maintain the beta cluster too [14:26:57] which is an attempt to reproduce the wiki production cluster off of labs instances [14:27:23] With, if I understand correctly, limited success right now. [14:27:59] yup [14:28:41] I am almost the only one maintaining it and did not really outreached it to other people [14:28:54] though we used the beta cluster a few times to prepare work on the prod cluster [14:29:12] such as adding the new "wikivoyage" wikis a few months ago and more recently to try out a tool named "git-deploy" [14:29:16] ... which is the objective. So it's not a wash. :-) [14:30:15] I'm still wrapping my head around the scope of my own work atm, I have the odd feeling it's quickly grow from "Oy, lots of work" to "Auuuuugh!" [14:30:42] basically [14:30:46] with my experience [14:30:51] you are a brain servicing the community [14:31:01] the community will pull as much as possible on your shoulders [14:31:07] until you collapse / drop stuff :-] [14:31:51] Heh. I'm used to it. I keep being elected to ArbCom on enwp. Difference now is at least I'll be paid to shoulder it. :-) [14:31:51] braaains [14:32:00] kind of :-] [14:32:37] (hi by the way) [14:32:38] !log deployment-prep apaches giving errors cause wikidatawiki is not configured [14:32:40] Logged the message, Master [14:32:52] It's quite the culture shock job-wise. I actually look forward looking in an environment saner than the typical corporative pap [14:34:45] Coren: quite the opposite [14:34:52] o_O [14:35:03] we still have a huge technical debt which comes from when the whole thing was managed by a handful of people around the world [14:35:07] Oh, you were agreeing, not saying we were the opposite. :-) [14:35:10] when most of the time was spent trying to keep the site up :-] [14:35:15] but it slowly improve [14:35:22] most of the configuration is managed in puppet already [14:35:26] we have git! [14:35:32] and nagios :-] [14:35:35] and paging [14:35:42] but yeah, still a lot to improve [14:36:10] the plus side is that you will most probably end up improving something that matter to you :-D since there are so many things to choose from [14:38:33] * hashar should read before writing ton of stuff [14:38:42] so yeah hmm. I was agreeing with you :-] [14:39:03] !log webtools deployed gridengine to webtools-grid-* (POC) [14:39:04] Logged the message, Master [14:50:53] hashar: is beta cluster working better now? [14:50:57] or is it still so slow [14:51:12] I guess it depends on gluster [14:51:24] why we rely on it? [14:51:31] we should just use the local storage [14:51:51] that's how production works [14:52:33] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Unknown [14:52:38] just do it ? :-D [14:52:43] that needs a gazillion of changes around [14:52:49] most of them depending on ops to review/merge [14:53:18] git-deploy was supposed to fix the issue but I have no idea what is git-deploy status nowadays [14:53:20] probably abandonned [14:53:25] but yeah [14:53:32] eventually I will do the changes so we use scap to deploy [14:56:28] hashar I have no access to production so I have no idea how to make it indentical [14:56:38] but ofc I don't see a problem in switching of storage [14:57:12] you can always mount it to same location as it's now [14:57:53] petan: I don't understand what you mean by "mount to same location" [15:02:33] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [15:09:00] hashar location where apache looks to [15:10:04] for example if data are in folder /A which is somewhere in /data/project you can remount local storage as /A and set up automatic sync with /data/project [15:10:17] so that when you change anything on gluster, it's changed in local storage as well [15:11:09] petan: automatically ? What do you mean ? :-D [15:11:20] if it is mounted, that is still hitting Gluster isn't it ? [15:13:09] as for automatic sync [15:13:13] here is how we do it in production [15:13:38] Files are pulled on fenari:/home/wikipedia/common from Gerrit [15:13:46] we run the 'scap' script [15:13:51] which DSH to all mediawiki boxes [15:14:00] and ask them to run a rsync command [15:14:14] of course the IP address is hardcoded in the scap script :-] [15:31:57] !log deployment-prep mobile redirection is more or less in place on beta. Browsing with a mobile agent will redirect to the mobile version. [15:31:58] Logged the message, Master [15:36:05] Hi Coren! [15:36:38] I am setting up a list of tools that are running on the toolserver. [15:37:49] Goal: Actually *have* a list of tools, find out their dependencies and licenses. Anything else you would want to know? [15:38:38] !deployment-prep http://upload.beta.wmflabs.org/ is dead :/ [15:38:38] deployment-prep is a project to test mediawiki at beta.wmflabs.org before putting it to prod [15:45:58] petan: is deployment-feed still in use ? [15:46:52] depends [15:46:55] it is, but it's defunct [15:46:58] [bz] (8NEW - created by: 2Antoine "hashar" Musso, priority: 4Unprioritized - 6normal) [Bug 45122] migrate all beta instance from Lucid to Precise - https://bugzilla.wikimedia.org/show_bug.cgi?id=45122 [15:47:12] now something got fucked in my irc client brb [15:47:15] petan: iirc it has been broken since last summer. Is there any point in keeping it around? [15:53:52] PROBLEM Current Load is now: CRITICAL on deployment-cache-text01.pmtpa.wmflabs 10.4.1.55 output: Connection refused by host [15:53:52] PROBLEM Disk Space is now: CRITICAL on deployment-cache-upload04.pmtpa.wmflabs 10.4.0.81 output: Connection refused by host [15:54:13] ok here [15:54:32] PROBLEM Disk Space is now: CRITICAL on deployment-cache-text01.pmtpa.wmflabs 10.4.1.55 output: Connection refused by host [15:54:32] PROBLEM Free ram is now: CRITICAL on deployment-cache-upload04.pmtpa.wmflabs 10.4.0.81 output: Connection refused by host [15:54:48] hashar the point is that there are some configs we might not want to loose, just as the current setup we would need to recreate should we ever wanted to finish it [15:54:56] in fact the feed should be fully operational [15:55:01] the ircd is running, the bot is up [15:55:13] PROBLEM Free ram is now: CRITICAL on deployment-cache-text01.pmtpa.wmflabs 10.4.1.55 output: Connection refused by host [15:55:17] it just doesn't work and no one knows why, because nobody understand how the feed on production works [15:55:39] the bot is getting the data from wikis [15:55:47] but it doesn't autojoin the channels [15:56:03] PROBLEM Total processes is now: CRITICAL on deployment-cache-upload04.pmtpa.wmflabs 10.4.0.81 output: Connection refused by host [15:56:04] it's some problem in bot which is written in some insane language [15:56:11] I think python or such [15:56:43] PROBLEM Total processes is now: CRITICAL on deployment-cache-text01.pmtpa.wmflabs 10.4.1.55 output: Connection refused by host [15:56:43] PROBLEM dpkg-check is now: CRITICAL on deployment-cache-upload04.pmtpa.wmflabs 10.4.0.81 output: Connection refused by host [15:57:23] PROBLEM dpkg-check is now: CRITICAL on deployment-cache-text01.pmtpa.wmflabs 10.4.1.55 output: Connection refused by host [15:57:53] PROBLEM Current Load is now: CRITICAL on deployment-cache-upload04.pmtpa.wmflabs 10.4.0.81 output: Connection refused by host [15:58:42] out [15:58:49] will set them up this evening [15:58:53] RECOVERY Disk Space is now: OK on deployment-cache-upload04.pmtpa.wmflabs 10.4.0.81 output: DISK OK [15:59:33] RECOVERY Disk Space is now: OK on deployment-cache-text01.pmtpa.wmflabs 10.4.1.55 output: DISK OK [15:59:34] RECOVERY Free ram is now: OK on deployment-cache-upload04.pmtpa.wmflabs 10.4.0.81 output: OK: 89% free memory [16:00:12] RECOVERY Free ram is now: OK on deployment-cache-text01.pmtpa.wmflabs 10.4.1.55 output: OK: 94% free memory [16:01:02] RECOVERY Total processes is now: OK on deployment-cache-upload04.pmtpa.wmflabs 10.4.0.81 output: PROCS OK: 84 processes [16:01:32] PROBLEM Disk Space is now: CRITICAL on deployment-sql04.pmtpa.wmflabs 10.4.1.84 output: Connection refused by host [16:01:42] RECOVERY Total processes is now: OK on deployment-cache-text01.pmtpa.wmflabs 10.4.1.55 output: PROCS OK: 90 processes [16:01:43] RECOVERY dpkg-check is now: OK on deployment-cache-upload04.pmtpa.wmflabs 10.4.0.81 output: All packages OK [16:01:52] PROBLEM Current Load is now: CRITICAL on deployment-sql04.pmtpa.wmflabs 10.4.1.84 output: Connection refused by host [16:02:12] PROBLEM Disk Space is now: CRITICAL on deployment-sql03.pmtpa.wmflabs 10.4.1.81 output: Connection refused by host [16:02:13] PROBLEM Free ram is now: CRITICAL on deployment-sql04.pmtpa.wmflabs 10.4.1.84 output: Connection refused by host [16:02:22] RECOVERY dpkg-check is now: OK on deployment-cache-text01.pmtpa.wmflabs 10.4.1.55 output: All packages OK [16:02:32] PROBLEM Current Load is now: CRITICAL on deployment-sql03.pmtpa.wmflabs 10.4.1.81 output: Connection refused by host [16:02:36] wtf [16:02:52] RECOVERY Current Load is now: OK on deployment-cache-upload04.pmtpa.wmflabs 10.4.0.81 output: OK - load average: 0.03, 0.52, 0.48 [16:03:02] PROBLEM Free ram is now: CRITICAL on deployment-sql03.pmtpa.wmflabs 10.4.1.81 output: Connection refused by host [16:03:42] PROBLEM Total processes is now: CRITICAL on deployment-sql04.pmtpa.wmflabs 10.4.1.84 output: Connection refused by host [16:03:52] RECOVERY Current Load is now: OK on deployment-cache-text01.pmtpa.wmflabs 10.4.1.55 output: OK - load average: 0.02, 0.38, 0.37 [16:04:00] hashar really needs to learn to use logs [16:04:22] PROBLEM Total processes is now: CRITICAL on deployment-sql03.pmtpa.wmflabs 10.4.1.81 output: Connection refused by host [16:04:22] PROBLEM dpkg-check is now: CRITICAL on deployment-sql04.pmtpa.wmflabs 10.4.1.84 output: Connection refused by host [16:05:12] PROBLEM dpkg-check is now: CRITICAL on deployment-sql03.pmtpa.wmflabs 10.4.1.81 output: Connection refused by host [16:07:12] RECOVERY Disk Space is now: OK on deployment-sql03.pmtpa.wmflabs 10.4.1.81 output: DISK OK [16:07:32] RECOVERY Current Load is now: OK on deployment-sql03.pmtpa.wmflabs 10.4.1.81 output: OK - load average: 0.45, 0.74, 0.46 [16:08:02] RECOVERY Free ram is now: OK on deployment-sql03.pmtpa.wmflabs 10.4.1.81 output: OK: 97% free memory [16:08:42] RECOVERY Total processes is now: OK on deployment-sql04.pmtpa.wmflabs 10.4.1.84 output: PROCS OK: 124 processes [16:09:22] RECOVERY Total processes is now: OK on deployment-sql03.pmtpa.wmflabs 10.4.1.81 output: PROCS OK: 121 processes [16:09:22] RECOVERY dpkg-check is now: OK on deployment-sql04.pmtpa.wmflabs 10.4.1.84 output: All packages OK [16:10:12] RECOVERY dpkg-check is now: OK on deployment-sql03.pmtpa.wmflabs 10.4.1.81 output: All packages OK [16:11:33] RECOVERY Disk Space is now: OK on deployment-sql04.pmtpa.wmflabs 10.4.1.84 output: DISK OK [16:11:55] RECOVERY Current Load is now: OK on deployment-sql04.pmtpa.wmflabs 10.4.1.84 output: OK - load average: 0.12, 0.56, 0.45 [16:12:12] RECOVERY Free ram is now: OK on deployment-sql04.pmtpa.wmflabs 10.4.1.84 output: OK: 97% free memory [16:21:52] PROBLEM dpkg-check is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to popen() failed [16:22:12] PROBLEM Disk Space is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to popen() failed [16:22:42] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [16:26:53] RECOVERY dpkg-check is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: All packages OK [16:27:13] RECOVERY Disk Space is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: DISK OK [16:37:42] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [16:38:12] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [16:40:52] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 22% free memory [16:43:30] wm-bot: diaf [16:43:30] Hi Damianz, there is some error, I am a stupid bot and I am not intelligent enough to hold a conversation with you :-) [16:44:51] Coren: SANE!?!?! NO ONE IS SANE [16:45:11] * Damianz goes to drive home... attempting to not get run over [16:45:45] Silke_WMDE_: Sorry, didn't see a ping. [16:47:03] Coren: here's my draft: http://www.mediawiki.org/wiki/User:Silke_WMDE/Documentation/List_of_Tools [16:47:25] Silke_WMDE_: The three primary things I need are (a) software dependencies, (b) data dependencies and (c) use pattern (continuous, webservice, batch runs, etc) [16:47:34] * Coren goes to see. [16:47:43] Have you seen my TODO draft? [16:48:15] yes [16:48:40] ... ! [16:48:54] I had no idea there were that /many/ [16:49:07] Oh, I'm not in it [16:49:15] I keep finding more and more and more tools [16:49:19] Silke_WMDE_: how did you compile it? [16:49:36] If I had to guess, I would have said "gotta be between 150 and 200 IMO" [16:49:56] :D Darkdadaah LOL - not at all, I am searching the wiki and jira and writing down what I find. m) [16:50:54] Coren would people know what we mean by "use pattern"? [16:51:12] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [16:51:40] Silke_WMDE_: I think I can guess if you give examples like you did [16:51:49] Silke_WMDE_: Probably, especially if you enumerate a few of the examples. It's a standard question on the enwp bot approval process, for instance, though I expect different communities use different terminology [16:52:27] ok, I'll just add your explanations from above [16:55:19] I'm going to have a parallell "beta" environment that's meant to replicate the eventual labs setup. Having a couple of volunteers to try to work their bots in with the new arrangement to shake the problems up would also be beneficial. [16:55:37] s/bots/tools/ [16:57:18] I'll keep that in mind [17:03:42] Silke_WMDE_: Should I add my tools to your list? Or I can write a page on the toolserver wiki describing my tools (I never took the time for that, although I created such a page on my home project, fr.wikt). [17:04:10] Darkdadaah: Yeah -> into my list! Just let add one more column [17:04:25] Ok [17:08:52] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 18% free memory [17:09:21] Darkdadaah: There you go. :) [17:09:40] I'm on it [17:09:45] cool [17:24:48] Few, ok, finished [17:25:05] phew* [17:25:39] It took me longer than I thought... as usual. [17:28:32] cool [17:28:42] I should moved it out of my user ns [17:29:47] Silke_WMDE_: Actually I have some other tools, but they are of less importance, and they are typical webpages, so I don't think I need to add all of them (just the "webservice" part of my main tools should suffice). [17:30:01] ok [17:57:42] Silke_WMDE_: That entry by Darkdadaah is exactly what I need. :-) [17:57:52] !log deployment-prep applying role::cache::upload to -cache-upload04 [17:57:54] Logged the message, Master [17:58:17] MaxSem: hey there. I am not there long but I think I got mobile redirect enabled on the beta text squid :-] [17:58:30] MaxSem: according to my lame test (iPhone / android browser) I seem to be properly redirected [18:02:10] !log deployment-prep running apt-get distupgrade on -cache-upload04 [18:02:12] Logged the message, Master [18:03:17] !log deployment-prep running apt-get distupgrade on -cache-text01 , -sql04 and -sql03 [18:03:18] Logged the message, Master [18:03:34] hashar, whee [18:03:37] let me check [18:04:56] hashar, confirmed [18:05:05] dude, you're awesome! [18:05:14] MaxSem: I could not find the report card in mingle though [18:05:36] MaxSem: mark has been very helpful this afternoon :-] [18:05:44] I'm not sure I can either:) [18:06:50] https://mingle.corp.wikimedia.org/projects/mobile/cards/367?referrer%5Bq%5D=redirection&referrer%5Bq_type%5D=&referrer%5Bquery_id%5D=q_65c4e8005&referrer%5Brank%5D=8&referrer%5Bsize%5D=16&referrer%5Bts%5D=02%2F18%2F13+18%3A06%3A07 [18:06:56] my eyes! [18:07:09] mingle, your links suck [18:09:42] PROBLEM dpkg-check is now: CRITICAL on deployment-cache-upload04.pmtpa.wmflabs 10.4.0.81 output: DPKG CRITICAL dpkg reports broken packages [18:10:23] PROBLEM dpkg-check is now: CRITICAL on deployment-cache-text01.pmtpa.wmflabs 10.4.1.55 output: DPKG CRITICAL dpkg reports broken packages [18:12:23] PROBLEM dpkg-check is now: CRITICAL on deployment-sql04.pmtpa.wmflabs 10.4.1.84 output: DPKG CRITICAL dpkg reports broken packages [18:13:53] PROBLEM dpkg-check is now: CRITICAL on deployment-sql03.pmtpa.wmflabs 10.4.1.81 output: DPKG CRITICAL dpkg reports broken packages [18:34:05] MaxSem: will give a quick update on the card [18:34:42] RECOVERY dpkg-check is now: OK on deployment-cache-upload04.pmtpa.wmflabs 10.4.0.81 output: All packages OK [18:35:23] RECOVERY dpkg-check is now: OK on deployment-cache-text01.pmtpa.wmflabs 10.4.1.55 output: All packages OK [18:36:08] MaxSem: the conf is only on the squid box :-D I need to puppetize it. [18:37:23] RECOVERY dpkg-check is now: OK on deployment-sql04.pmtpa.wmflabs 10.4.1.84 output: All packages OK [18:38:53] RECOVERY dpkg-check is now: OK on deployment-sql03.pmtpa.wmflabs 10.4.1.81 output: All packages OK [18:48:47] hashar, I think it's not needed per mark [18:49:01] yeah in prod [18:49:13] but in labs I could just have a copy of the current file [18:57:17] dinnner [19:07:42] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:12:33] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [19:48:52] PROBLEM Current Load is now: CRITICAL on schema-review.pmtpa.wmflabs 10.4.1.77 output: Connection refused by host [19:49:32] PROBLEM Disk Space is now: CRITICAL on schema-review.pmtpa.wmflabs 10.4.1.77 output: Connection refused by host [19:53:52] RECOVERY Current Load is now: OK on schema-review.pmtpa.wmflabs 10.4.1.77 output: OK - load average: 0.03, 0.45, 0.40 [19:54:32] RECOVERY Disk Space is now: OK on schema-review.pmtpa.wmflabs 10.4.1.77 output: DISK OK [20:04:23] PROBLEM dpkg-check is now: CRITICAL on schema-review.pmtpa.wmflabs 10.4.1.77 output: DPKG CRITICAL dpkg reports broken packages [20:09:22] RECOVERY dpkg-check is now: OK on schema-review.pmtpa.wmflabs 10.4.1.77 output: All packages OK [20:11:23] PROBLEM Free ram is now: WARNING on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: Warning: 19% free memory [20:13:11] andrewbogott_afk: ping [20:13:12] Ryan_Lane: ping [20:17:27] preilly: apergos might be up if that help [20:17:31] I think he was having dinner [20:17:41] oh we are on -labs hm [20:18:53] hashar: I need a public IP allocated [20:19:42] preilly: yeah that need a quota to be raised for your project [20:20:00] preilly: a workaround is to use instance .instance-proxy.wmflabs.org [20:20:11] works well for port 80 at least [20:20:15] hashar: yeah I'm aware [20:21:12] :-( [20:21:42] hashar: but thanks for the pointer... [20:22:06] hashar: I need to be able to work with our MySQL slaves from it [20:22:19] hashar: so I'll unfortunately need a public routable IP [20:22:30] hashar: due to ACLs etc [20:22:33] ohh [20:22:36] yeah that make sense [20:22:45] I know they have a project to have slaves in labs too [20:23:04] hashar: that is what I'm working on right now ;-) [20:23:23] Coren might not about the status of setting up mysql slaves in labs :-] [20:23:25] ahh [20:23:26] sorry [20:23:27] lol [20:23:46] I know nothing! [20:23:47] so yeah maybe a pub IP [20:23:57] I have no idea how the networking is handled really [20:24:10] ideally we could use the 10.0.0.0/8 network [20:24:13] no real point in having public address [20:24:18] but that might not be possible [20:24:39] Coren: at least you are in the right IRC channels ahha [20:25:10] preilly: What's your use case? You need to connect to the mysql slaves from what? [20:26:52] Coren: don't worry about I've got it covered [20:27:29] so hmm bed time for me [20:27:36] enjoy hacking the servers/instances [20:36:22] RECOVERY Free ram is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: OK: 20% free memory [20:38:52] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 22% free memory [20:41:12] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [20:49:13] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [21:06:53] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 18% free memory [22:57:54] preilly: yo [22:57:57] preilly: which project? [22:58:09] Ryan_Lane: deployment prep [22:58:12] also, why are you working on a holiday? [22:58:54] Ryan_Lane: that's a great question [22:59:16] upped the quota [22:59:21] what are you using the ip for? [22:59:44] something about mysql? [23:00:23] Ryan_Lane: MySQL labs DB replication [23:00:37] Ryan_Lane: this is just for the dashboard portion of it [23:00:40] ahhh. ok [23:00:46] Ryan_Lane: e.g., http://schema-review.instance-proxy.wmflabs.org/ [23:00:55] ah [23:00:55] cool [23:01:44] Ryan_Lane: basically every column in every table will be able to be toggled on/off for labs DB replication purposes [23:01:58] neat [23:02:20] btw, I migrated openstack's wiki to mediawiki this weekend, and went with the strapping skin: https://wiki.openstack.org/wiki/Main_Page [23:02:28] pushed a number of new fixes up for it [23:04:14] glad to see you guys working on mysql replication for labs [23:04:21] it'll make a lot of users happy [23:04:36] Ryan_Lane: oh that's awesome [23:04:59] Ryan_Lane: yeah I'm sure users will be jazzed [23:37:20] preilly, what're you going to do to stop people from seeing e.g. rows which are suppressed? [23:38:40] Krenair: those rows won't be in the replicated databases