[00:04:37] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [00:11:47] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [00:11:47] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [00:12:37] PROBLEM HTTP is now: CRITICAL on deployment-web i-00000217 output: CRITICAL - Socket timeout after 10 seconds [00:14:37] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [00:17:27] PROBLEM HTTP is now: WARNING on deployment-web i-00000217 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.015 second response time [00:34:37] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [00:41:47] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [00:41:47] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [00:44:37] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [01:04:37] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [01:11:47] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [01:11:47] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [01:14:17] PROBLEM HTTP is now: WARNING on mailman-01 i-00000235 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 498 bytes in 0.020 second response time [01:14:37] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [01:18:33] !log mailman [mailman-01] stopped puppet agent [01:18:34] Logged the message, Master [01:34:37] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [01:41:47] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [01:41:47] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [01:44:37] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [01:59:17] PROBLEM HTTP is now: CRITICAL on mailman-01 i-00000235 output: Connection refused [02:04:17] PROBLEM HTTP is now: WARNING on mailman-01 i-00000235 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 498 bytes in 0.021 second response time [02:04:37] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [02:11:47] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [02:11:47] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [02:14:37] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [02:34:37] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [02:41:47] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [02:41:47] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [02:44:37] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [03:04:37] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [03:11:47] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [03:11:47] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [03:14:37] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [03:30:58] Krinkle: i added you to the group so you can write deployment configs a while ago (whenever you asked about it, it was done like 5 mins after you /quit). and !log'd it. but then i forgot to tell you or double check it worked for you [03:34:37] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [03:41:47] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [03:41:47] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [03:43:47] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 14% free memory [03:44:37] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [03:47:37] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 14% free memory [03:58:16] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 14% free memory [03:58:46] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 5% free memory [04:02:36] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 5% free memory [04:04:06] jeremyb: thx [04:04:46] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [04:05:02] labs-nagios-wm: quiet [04:05:07] Krinkle: works? [04:06:22] I haven't tested [04:06:23] will do later [04:06:25] gotta go now [04:06:31] * Krinkle makes note [04:06:33] k [04:07:36] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 97% free memory [04:08:46] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 97% free memory [04:12:16] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [04:14:36] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [04:14:46] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [04:18:16] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 5% free memory [04:28:16] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 94% free memory [04:34:46] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [04:43:16] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [04:44:36] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [04:44:46] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [04:52:56] PROBLEM Free ram is now: CRITICAL on test3 i-00000093 output: Critical: 2% free memory [04:57:56] RECOVERY Free ram is now: OK on test3 i-00000093 output: OK: 96% free memory [05:04:46] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [05:14:16] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [05:14:36] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [05:14:46] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [05:34:46] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [05:43:46] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 17% free memory [05:44:36] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [05:44:36] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [05:44:46] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [06:04:46] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [06:14:36] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [06:14:36] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [06:14:46] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [06:18:46] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 4% free memory [06:23:46] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 96% free memory [06:34:46] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [06:44:36] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [06:44:36] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [06:44:46] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [07:04:46] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [07:14:06] PROBLEM Puppet freshness is now: CRITICAL on nova-ldap1 i-000000df output: Puppet has not run in last 20 hours [07:14:36] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [07:14:36] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [07:14:46] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [07:25:26] PROBLEM dpkg-check is now: CRITICAL on bots-3 i-000000e5 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:30:16] RECOVERY dpkg-check is now: OK on bots-3 i-000000e5 output: All packages OK [07:34:46] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [07:44:36] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [07:44:36] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [07:44:46] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [08:04:46] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [08:14:36] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [08:14:36] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [08:14:46] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [08:31:06] PROBLEM Puppet freshness is now: CRITICAL on mobile-feeds i-000000c1 output: Puppet has not run in last 20 hours [08:34:46] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [08:44:24] PROBLEM dpkg-check is now: CRITICAL on incubator-common i-00000254 output: Connection refused by host [08:44:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [08:44:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [08:45:24] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [08:45:44] PROBLEM Current Load is now: CRITICAL on incubator-common i-00000254 output: Connection refused by host [08:46:14] PROBLEM Current Users is now: CRITICAL on incubator-common i-00000254 output: Connection refused by host [08:46:54] PROBLEM Disk Space is now: CRITICAL on incubator-common i-00000254 output: Connection refused by host [08:47:34] PROBLEM Free ram is now: CRITICAL on incubator-common i-00000254 output: Connection refused by host [08:48:44] PROBLEM Total Processes is now: CRITICAL on incubator-common i-00000254 output: Connection refused by host [08:58:42] New review: Thehelpfulone; "(no comment)" [operations/puppet] (test) C: 1; - https://gerrit.wikimedia.org/r/6584 [08:59:14] if someone could +2 https://gerrit.wikimedia.org/r/#/c/6727 and https://gerrit.wikimedia.org/r/#/c/6584/... :) [09:02:05] who? [09:05:24] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [09:08:39] jeremyb: I don't know, hence "someone" :P [09:13:44] RECOVERY Total Processes is now: OK on incubator-common i-00000254 output: PROCS OK: 95 processes [09:14:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-web i-00000217 output: Puppet has not run in last 20 hours [09:14:24] RECOVERY dpkg-check is now: OK on incubator-common i-00000254 output: All packages OK [09:14:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [09:14:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [09:15:44] RECOVERY Current Load is now: OK on incubator-common i-00000254 output: OK - load average: 0.05, 0.09, 0.20 [09:16:14] RECOVERY Current Users is now: OK on incubator-common i-00000254 output: USERS OK - 1 users currently logged in [09:16:24] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [09:17:34] RECOVERY Free ram is now: OK on incubator-common i-00000254 output: OK: 94% free memory [09:21:54] RECOVERY Disk Space is now: OK on incubator-common i-00000254 output: DISK OK [09:33:43] !log bots made a little tweak to content.html [09:33:44] Logged the message, Master [09:36:24] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [09:44:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [09:44:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [09:46:44] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [10:00:55] petan petan_: poke [10:06:54] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [10:14:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [10:14:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [10:16:44] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [10:31:42] !log dumps hydriz: Deleting this instance pending reinstall due to broken packages [10:31:43] Logged the message, Master [10:36:54] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [10:36:55] !log dumps hydriz: Make that dumps-2 [10:36:56] Logged the message, Master [10:38:24] PROBLEM host: dumps-2 is DOWN address: i-00000174 check_ping: Invalid hostname/address - i-00000174 [10:40:45] !log dumps [i-00000170] hydriz: Testing to see if it logs the hostname... [10:40:46] Logged the message, Master [10:44:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [10:44:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [10:46:30] !log incubator hydriz: [i-00000211] Starting full import of incubatorwiki dump into devwiki [10:46:32] Logged the message, Master [10:46:44] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [11:06:54] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [11:08:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-apache05 i-0000024a output: Puppet has not run in last 20 hours [11:14:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [11:14:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [11:16:44] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [11:18:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-apache01 i-00000246 output: Puppet has not run in last 20 hours [11:36:54] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [11:44:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [11:44:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [11:46:44] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [11:57:54] PROBLEM HTTP is now: CRITICAL on deployment-web i-00000217 output: CRITICAL - Socket timeout after 10 seconds [12:02:44] PROBLEM HTTP is now: WARNING on deployment-web i-00000217 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.012 second response time [12:06:54] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [12:14:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [12:14:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [12:16:44] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [12:36:54] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [12:44:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [12:44:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [12:46:44] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [12:47:34] RECOVERY host: mobile-enwp is UP address: i-000000ce PING OK - Packet loss = 0%, RTA = 2.24 ms [12:51:44] RECOVERY Current Load is now: OK on mobile-enwp i-000000ce output: OK - load average: 0.12, 0.30, 0.15 [12:51:44] RECOVERY Free ram is now: OK on mobile-enwp i-000000ce output: OK: 71% free memory [13:14:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [13:14:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [13:16:44] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [13:44:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [13:44:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [13:46:44] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [13:51:48] !log incubator hydriz: [i-00000211] Git is just pissing me off [13:51:50] Logged the message, Master [13:52:52] what's the problem with git? [13:54:07] <^demon> What problem? [13:54:26] I just did a git checkout of the 1.20wmf2 branch [13:54:34] and boom all the extensions got deleted [13:54:47] its just pissing me off about why Wikimedia did Git this way [13:55:04] or rather, why Git was so strict about what is inside the repository [13:55:08] *directory [13:55:17] <^demon> You were on 1.20wmf1 and switched to 1.20wmf2? [13:55:22] kinda [13:55:44] I mean, extensions weren't packaged in the mediawiki/core.git repository [13:56:01] so I had to get them myself (at least so I thought how I was supposed to do it) [13:56:34] <^demon> No, they're not packaged on the master branch. They're included as submodules on the deployment branch [13:57:29] any guide about doing that? [13:58:04] man, SVN was so much easier, just svn switch and done [13:59:11] <^demon> Once you've checked out the wmf branch, you do `git submodule update --init` to make them checkout. [14:00:16] eh, why does it use ssh://? [14:00:51] <^demon> Mistake on my part when I wrote make-wmf-branch [14:00:57] <^demon> It'll be fixed for 1.20wmf3 [14:01:25] lol wut [14:01:35] so its now unusable zzz [14:01:48] <^demon> It's usable to everyone who has an account ;-) [14:01:57] <^demon> You're the first anonymous user to use 1.20wmf* [14:02:16] I have always followed the wmf branches lol [14:02:26] cos of stability [14:02:47] <^demon> I mean since we switched to git and started using submodules to pull in extensions. [14:02:52] master, with self-packaging of extensions spells some form of trouble at any time [14:03:02] ohoh [14:03:20] feedback is good :) [14:03:57] but now git cloning is taking longer than svn, significantly [14:04:59] <^demon> That's because you're checking out the full history. It's got ~10 years of history ;-) [14:06:56] wait, ssh:// also affects 1.20wmf1 zzz [14:07:24] that would mean having to publish my private key to labs [14:07:26] zzz [14:08:40] <^demon> It affected 1.20wmf1 and 2. It's fixed for 3. [14:09:07] <^demon> Or, it will be rather. [14:10:08] <^demon> Anyway, the wmf branches only trail master by ~2 weeks now. [14:10:08] so, when will deployments start for 1.20wmf3? [14:10:19] <^demon> In another ~week. We deployed last week. [14:10:43] hmm, got to make a prediction then :) [14:14:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [14:14:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [14:16:44] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [14:38:34] hello :-) [14:42:07] hi hashar [14:42:15] can you +2 things on gerrit? [14:42:22] https://gerrit.wikimedia.org/r/#/c/6727 and https://gerrit.wikimedia.org/r/#/c/6584/ if you can :) [14:46:26] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [14:46:46] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [14:46:46] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [14:59:05] Thehelpfulone: yup I can [14:59:06] I think, [14:59:06] it depends on which project [14:59:43] it's for operations/puppet [15:00:24] so I cant [15:00:24] you need people from ops [15:04:04] [15:16:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [15:18:04] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [15:19:14] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [15:25:08] New review: Pyoungmeister; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/6584 [15:25:11] Change merged: Pyoungmeister; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6584 [15:25:17] New review: Pyoungmeister; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/6727 [15:25:19] Change merged: Pyoungmeister; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6727 [15:44:28] New review: Ottomata; "If there are problems with the exec/class stuff, I'd recommend defining the exec in the lighttpd_con..." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6727 [15:46:43] New patchset: Pyoungmeister; "Revert "notify (+ do a reload) on lighttpd when its config changes"" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6796 [15:46:44] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [15:46:57] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/6796 [15:47:00] New patchset: Pyoungmeister; "Revert "Disable https redirection for labs"" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6797 [15:47:15] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/6797 [15:49:04] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [15:49:44] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [16:03:44] New review: Pyoungmeister; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/6796 [16:03:46] Change merged: Pyoungmeister; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6796 [16:03:52] New review: Pyoungmeister; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/6797 [16:03:55] Change merged: Pyoungmeister; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6797 [16:09:12] 05/07/2012 - 16:09:11 - Updating keys for otto [16:16:49] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [16:19:39] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [16:21:09] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [16:25:11] 05/07/2012 - 16:25:11 - Updating keys for otto [16:26:21] 05/07/2012 - 16:26:20 - Updating keys for otto [16:26:22] 05/07/2012 - 16:26:21 - Updating keys for otto [16:26:40] PROBLEM Free ram is now: WARNING on bots-2 i-0000009c output: Warning: 19% free memory [16:27:12] 05/07/2012 - 16:27:12 - Updating keys for otto [16:27:15] 05/07/2012 - 16:27:15 - Updating keys for otto [16:46:52] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [16:48:25] hm http://archiveteam.org/images/0/05/Rejectedatlogo.jpg [16:49:42] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [16:51:42] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [17:04:22] RECOVERY HTTP is now: OK on mailman-01 i-00000235 output: HTTP OK: HTTP/1.1 301 Moved Permanently - 190 bytes in 0.011 second response time [17:15:12] PROBLEM Puppet freshness is now: CRITICAL on nova-ldap1 i-000000df output: Puppet has not run in last 20 hours [17:16:52] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [17:17:52] PROBLEM HTTP is now: CRITICAL on deployment-web i-00000217 output: CRITICAL - Socket timeout after 10 seconds [17:19:52] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [17:21:42] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [17:22:42] PROBLEM HTTP is now: WARNING on deployment-web i-00000217 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.004 second response time [17:46:52] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [17:49:52] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [17:51:42] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [17:57:52] PROBLEM Current Users is now: CRITICAL on mobile-enwp i-000000ce output: CHECK_NRPE: Socket timeout after 10 seconds. [17:57:52] PROBLEM Disk Space is now: CRITICAL on mobile-enwp i-000000ce output: CHECK_NRPE: Socket timeout after 10 seconds. [17:57:52] PROBLEM Total Processes is now: CRITICAL on mobile-enwp i-000000ce output: CHECK_NRPE: Socket timeout after 10 seconds. [17:57:57] PROBLEM SSH is now: CRITICAL on mobile-enwp i-000000ce output: CRITICAL - Socket timeout after 10 seconds [17:57:57] PROBLEM dpkg-check is now: CRITICAL on mobile-enwp i-000000ce output: CHECK_NRPE: Socket timeout after 10 seconds. [17:59:52] PROBLEM Current Load is now: CRITICAL on mobile-enwp i-000000ce output: Connection refused or timed out [17:59:52] PROBLEM Free ram is now: CRITICAL on mobile-enwp i-000000ce output: Connection refused or timed out [18:06:52] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [18:11:53] RECOVERY host: mobile-enwp is UP address: i-000000ce PING OK - Packet loss = 0%, RTA = 3.29 ms [18:16:53] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [18:19:53] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [18:21:43] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [18:21:43] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [18:32:13] PROBLEM Puppet freshness is now: CRITICAL on mobile-feeds i-000000c1 output: Puppet has not run in last 20 hours [18:39:02] @Ryan_Lane / anyone - can anyone held me get mobile-geo back up on labs? [18:42:13] what's wrong with it? [18:42:17] which instance, in which project? [18:46:26] jdlrobson: ? [18:46:52] mobile-enwp.pmtpa.wmflabs [18:46:53] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [18:47:04] http://mobile-geo.wmflabs.org/ is inaccessible [18:47:14] and when I try to ssh into it I get ssh_exchange_identification: Connection closed by remote host [18:49:53] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [18:51:43] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [18:52:27] <^demon> Ryan_Lane: Would you mind reviewing change 5727? I added you to it almost 2 weeks ago. [18:52:41] !change 5727 [18:52:41] https://gerrit.wikimedia.org/r/5727 [18:52:43] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [18:52:56] * Ryan_Lane twitches [18:53:09] it's gonna take me a bit to review this [18:53:13] but yeah, I'll review it today [18:53:20] <^demon> Ok, thank you. [18:53:29] * Damianz gives Ryan_Lane a cookie [19:15:13] PROBLEM Puppet freshness is now: CRITICAL on deployment-web i-00000217 output: Puppet has not run in last 20 hours [19:16:53] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [19:19:53] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [19:21:43] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [19:22:53] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [19:46:25] notpeter: could you take another pass at 6727/6584? why did you think they were on the production branch? [19:46:53] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [19:49:53] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [19:50:15] 6584 has been merged [19:50:28] as has 6727 from the look of it [19:51:43] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [19:52:53] PROBLEM host: mobile-enwp is DOWN address: i-000000ce CRITICAL - Host Unreachable (i-000000ce) [20:07:53] RECOVERY Free ram is now: OK on mobile-enwp i-000000ce output: OK: 72% free memory [20:07:53] RECOVERY Current Load is now: OK on mobile-enwp i-000000ce output: OK - load average: 1.61, 0.74, 0.27 [20:08:03] RECOVERY host: mobile-enwp is UP address: i-000000ce PING OK - Packet loss = 0%, RTA = 0.67 ms [20:10:43] RECOVERY SSH is now: OK on mobile-enwp i-000000ce output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [20:10:43] RECOVERY Current Users is now: OK on mobile-enwp i-000000ce output: USERS OK - 1 users currently logged in [20:10:43] RECOVERY Disk Space is now: OK on mobile-enwp i-000000ce output: DISK OK [20:10:43] RECOVERY Total Processes is now: OK on mobile-enwp i-000000ce output: PROCS OK: 110 processes [20:10:48] RECOVERY dpkg-check is now: OK on mobile-enwp i-000000ce output: All packages OK [20:16:26] Damianz: look more closely [20:16:53] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [20:19:53] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [20:21:43] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [20:22:18] jeremyb: They've been merged into the branch but from the last puppet run they don't seem to be active [20:22:33] Damianz: look even more closely [20:22:39] hint: look at the review comments [20:46:53] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [20:49:20] <^demon> Ryan_Lane: So the gerrit hackathon is going on at Google this week. Their major focus for the week is a plugin interface [20:49:20] <^demon> :) [20:49:53] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [20:51:43] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [21:09:13] PROBLEM Puppet freshness is now: CRITICAL on deployment-apache05 i-0000024a output: Puppet has not run in last 20 hours [21:16:53] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [21:19:13] PROBLEM Puppet freshness is now: CRITICAL on deployment-apache01 i-00000246 output: Puppet has not run in last 20 hours [21:19:53] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [21:21:46] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [21:25:03] PROBLEM host: deployment-web is DOWN address: i-00000217 CRITICAL - Host Unreachable (i-00000217) [21:25:43] RECOVERY host: deployment-web is UP address: i-00000217 PING OK - Packet loss = 0%, RTA = 1.58 ms [21:28:43] PROBLEM HTTP is now: WARNING on deployment-web i-00000217 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.005 second response time [21:28:58] ^demon: ah. awesome [21:32:58] jeremyb: Ahh I see [21:34:01] Yeah their both in test and where merged into test, though it shouldn't be a problem those being merged into production either. [21:42:46] PowerConnect switches are a pile of crap [21:46:53] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [21:47:06] ^demon: can you make me a new mediawiki extension repo? [21:47:15] OATHAuth [21:47:18] <^demon> I can. [21:47:28] <^demon> Is there existing history we're importing, or a fresh start? [21:47:38] (that's not a typo - this isn't oauth, it's oath) [21:47:40] fresh [21:49:30] <^demon> Done. [21:49:34] thanks [21:49:52] <^demon> No problem. It's so much faster since I can do it via the web now :) [21:49:53] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [21:51:19] <^demon> We need to go ahead and update all our docs regarding refs/for/* [21:51:21] <^demon> refs/for/* is becoming refs/publish/* :p [21:51:53] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [21:52:40] ^demon: how did you used to do it before the web? ssh to 29418? [21:52:48] <^demon> Yep [21:52:54] ugh @ refs/publish. will the old one stop working? [21:53:02] <^demon> Probably not for a long time. [21:53:21] <^demon> But since they added refs/drafts/* they decided refs/publish/* made more sense. [21:53:42] <^demon> Oh. Drafts. I totally want to play with that now. [21:53:44] oh, idk about drafts [21:53:53] I'm about to check something really cool in :) [21:54:02] o.0 Draft like central staff? [21:54:09] stash* [21:54:26] errr, something like that? or just changes that aren't in a review queue [21:54:34] maybe ^demon knows more [21:54:39] Ryan_Lane: Liquid nitrogen? [21:54:44] <^demon> https://gerrit.wikimedia.org/r/#/c/6880/ :) [21:54:51] <^demon> It doesn't show up in the pending revisions for review. [21:55:29] <^demon> They're hidden until you share them with people :) [21:55:36] is per project puppet spec'd out somewhere? or i just have to ask paravoid about it? [21:55:52] just ask :) [21:56:35] well i guess the most interesting question is when's it ready? ;) but that's not what made me bring it up [21:56:40] ^demon: aww, you didn't add my git review file [21:57:08] I have it :) [21:57:19] <^demon> Then I would've had to clone a repo I have no intent on using right now :p [21:57:38] i was wondering about how to do manifest testing or debugging iteratively. is that always going to require a push? or could you sometimes just edit and do a run? [21:57:59] where will the puppetmasters live and who will access to them? (will it even be multiple puppetmasters? [21:58:09] * jeremyb has to go in 1 min but will be back later [21:58:37] I thought we where having no puppetmasters. [22:10:13] PROBLEM Puppet freshness is now: CRITICAL on hugglewiki i-000000aa output: Puppet has not run in last 20 hours [22:16:53] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [22:19:13] 05/07/2012 - 22:19:13 - Creating a home directory for smerritt at /export/home/bastion/smerritt [22:19:33] 05/07/2012 - 22:19:33 - Creating a home directory for smerritt at /export/home/swift/smerritt [22:19:53] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [22:20:13] 05/07/2012 - 22:20:13 - Updating keys for smerritt [22:20:32] 05/07/2012 - 22:20:32 - Updating keys for smerritt [22:21:53] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [22:46:53] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [22:49:53] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [22:51:53] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [23:16:53] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [23:19:53] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [23:21:53] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [23:41:13] PROBLEM Disk Space is now: CRITICAL on labs-nfs1 i-0000005d output: CHECK_NRPE: Socket timeout after 10 seconds. [23:46:06] PROBLEM Disk Space is now: WARNING on labs-nfs1 i-0000005d output: DISK WARNING - free space: /export 868 MB (5% inode=81%): /home/SAVE 868 MB (5% inode=81%): [23:46:53] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [23:49:53] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [23:49:53] PROBLEM Disk Space is now: CRITICAL on bz-dev i-000001db output: DISK CRITICAL - free space: / 26 MB (1% inode=43%): [23:51:53] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [23:58:04] hey andrewbogott are you around ? [23:59:05] i don't remember how to add ip's to labs :( [23:59:51] Magic