[00:01:33] superm401: The first thing to try is logging into the instance and runing 'sudo puppetd -tv' [00:01:44] That will force a puppet refresh and you can watch the output and see what's happening. [00:05:05] andrewbogott, yep, gives a relevant error: http://pastebin.com/mgqhF2cw [00:06:11] Oh, yeah -- maybe you checked two boxes, like webserver::apache2 and webserver::php5? [00:06:21] It's not at all obvious, but webserver::php5 already includes apache. [00:06:51] You called it. [00:06:53] superm401: I'm in the process of adding some doc links to that configure page so that we can explain things like that :) [00:07:02] Well, I'm not actually adding the docs but at least adding a place for the docs. [00:07:09] Uncheck apache2 and try again, maybe things will go better. [00:07:17] Great, that's what I was about to suggest. [00:07:37] You can email me after you add the doc location, and I'll try to fill it out. mflaschen@wikimedia.org [00:07:57] Cool. [00:08:20] The docs will be inline in the code rather than a wiki page. Which means that getting them in will involve git and gerrit :( [00:08:38] Hmm, could they be MediaWiki? [00:08:45] Meaning the namespace. [00:08:59] But if it has to be code, I've now been exposed to the joys of Gerrit, so I'll still do it. [00:09:18] Yeah, the docs could be in a wiki instead. But it's nice to have the docs staring you in the face when you write the code... [00:09:24] So for the moment I want to give inline stuff a try. [00:09:48] The docsite (generated from code) is here: http://doc.wikimedia.org/puppet/ [00:10:05] It's pretty overwhelming like that, though, I'm hoping to add deep links to that site from the 'configure instance' page. [00:10:37] Is puppet treating you better now? [00:12:16] andrewbogott, different error. [00:12:18] "err: Could not retrieve catalog from remote server: Error 400 on SERVER: Exported resource Sshhostkey[abusefilter-global-main.pmtpa.wmflabs] cannot override local resource on node i-0000052f.pmtpa.wmflabs" [00:12:50] hm… I feel like that's one of those errors that goes away on the second try [00:12:56] If not then I have no clue :( [00:13:30] Actually, that should be a frown either way. [00:14:49] Fair point [00:14:53] But it does seem to be working. [00:14:57] woot! [00:15:00] :( [00:15:05] :) [00:15:32] puppet is so full of race conditions that the only way to avoid despair is to adopt a policy of retry-and-smile [00:15:49] yes. we need to change things so that only role classes can be selected [00:16:06] first we actually need to have role classes for everything, though :D [00:19:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:21:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:30:22] RECOVERY Total processes is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS OK: 149 processes [00:49:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:52:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:09:33] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 173 processes [01:14:32] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 97 processes [01:20:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:23:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:51:34] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:53:24] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:21:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:23:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:39:52] RECOVERY Free ram is now: OK on wikistream-1 i-0000016e.pmtpa.wmflabs output: OK: 27% free memory [02:51:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:53:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:58:56] Change on 12mediawiki a page Developer access was modified, changed by Nicjansma link https://www.mediawiki.org/w/index.php?diff=613997 edit summary: [03:00:16] Change on 12mediawiki a page Developer access was modified, changed by Nicjansma link https://www.mediawiki.org/w/index.php?diff=613999 edit summary: /* User:Nicjansma */ [03:07:52] PROBLEM Free ram is now: WARNING on wikistream-1 i-0000016e.pmtpa.wmflabs output: Warning: 14% free memory [03:11:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [03:16:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [03:21:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:24:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:41:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [03:51:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:54:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:22:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:24:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:52:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:55:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:22:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:25:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:53:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:56:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:23:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:26:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:28:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 204 processes [06:30:32] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:30:32] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 153 processes [06:31:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [06:43:33] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [06:53:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:55:33] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 148 processes [06:55:43] PROBLEM Free ram is now: WARNING on swift-be1 i-000001c7.pmtpa.wmflabs output: Warning: 19% free memory [06:56:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:00:32] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 147 processes [07:11:52] PROBLEM dpkg-check is now: CRITICAL on spellcheckself-bot i-0000041e.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [07:16:42] RECOVERY dpkg-check is now: OK on spellcheckself-bot i-0000041e.pmtpa.wmflabs output: All packages OK [07:23:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:26:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:41:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [07:54:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:56:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:24:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:27:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:54:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:55:42] RECOVERY Free ram is now: OK on swift-be1 i-000001c7.pmtpa.wmflabs output: OK: 20% free memory [08:57:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:24:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:28:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:54:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:58:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:19:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 8.94, 8.45, 6.51 [10:24:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:29:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:34:53] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 3.63, 3.90, 4.81 [10:54:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:59:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:24:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:29:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:54:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:00:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:24:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:30:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:36:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 11% free memory [12:56:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:59:26] Change on 12mediawiki a page Developer access was modified, changed by Edinwiki link https://www.mediawiki.org/w/index.php?diff=614119 edit summary: /* User:Edinwiki */ [13:01:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:26:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:32:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:35:42] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 18% free memory [13:55:43] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 48% free memory [13:56:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:03:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:26:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:34:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:37:13] PROBLEM Total processes is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS WARNING: 153 processes [14:56:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:04:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:18:42] !ping [15:18:42] pong [15:27:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:28:51] hey petan, what would i need to do to enable the morebots bot on the #wikimedia-analytics channel? [15:29:16] you would need to ask andrewbogott who is maintaining it [15:29:26] ty! [15:30:06] * andrewbogott doesn't know, but will look in to it. [15:34:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:35:30] andrewbogott I guess it would require to launch another instance of bot [15:35:43] probably. [15:35:44] but I don't understand python + I don't have credentials for bot [15:36:17] is there any documentation about this? [15:38:08] There is but I'm not sure where [15:57:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:04:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:17:13] RECOVERY Total processes is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS OK: 146 processes [16:28:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:34:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:58:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:05:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:29:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:32:23] petan, any idea if drdee wanted a morebot in #wikimedia-analytics that logs to the same place or a different place? [17:35:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:48:53] PROBLEM Current Load is now: CRITICAL on bots-abogott-devel i-00000530.pmtpa.wmflabs output: Connection refused by host [17:49:33] PROBLEM Current Users is now: CRITICAL on bots-abogott-devel i-00000530.pmtpa.wmflabs output: Connection refused by host [17:50:12] PROBLEM Disk Space is now: CRITICAL on bots-abogott-devel i-00000530.pmtpa.wmflabs output: Connection refused by host [17:50:52] PROBLEM Free ram is now: CRITICAL on bots-abogott-devel i-00000530.pmtpa.wmflabs output: Connection refused by host [17:52:22] PROBLEM Total processes is now: CRITICAL on bots-abogott-devel i-00000530.pmtpa.wmflabs output: Connection refused by host [17:53:12] PROBLEM dpkg-check is now: CRITICAL on bots-abogott-devel i-00000530.pmtpa.wmflabs output: Connection refused by host [17:55:13] RECOVERY Disk Space is now: OK on bots-abogott-devel i-00000530.pmtpa.wmflabs output: DISK OK [17:55:53] RECOVERY Free ram is now: OK on bots-abogott-devel i-00000530.pmtpa.wmflabs output: OK: 1198% free memory [17:57:23] RECOVERY Total processes is now: OK on bots-abogott-devel i-00000530.pmtpa.wmflabs output: PROCS OK: 83 processes [17:58:13] RECOVERY dpkg-check is now: OK on bots-abogott-devel i-00000530.pmtpa.wmflabs output: All packages OK [17:58:53] RECOVERY Current Load is now: OK on bots-abogott-devel i-00000530.pmtpa.wmflabs output: OK - load average: 0.08, 0.61, 0.53 [17:59:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:59:33] RECOVERY Current Users is now: OK on bots-abogott-devel i-00000530.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [18:05:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:21:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [18:29:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:31:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [18:36:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:49:00] hi petan [18:51:37] petan: hi [18:52:33] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 155 processes [18:59:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:02:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 10.37, 9.13, 6.58 [19:06:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 8.56, 8.35, 5.98 [19:06:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:10:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 6.96, 7.38, 5.53 [19:11:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 7.48, 7.81, 6.13 [19:21:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 4.25, 5.43, 5.03 [19:22:15] benestar: you might want to email him or labs-l [19:26:52] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 5.16, 4.95, 4.90 [19:26:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [19:29:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:35:53] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 2.34, 3.49, 4.52 [19:36:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:53:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 4.70, 5.82, 5.52 [19:54:20] drdee: You want a separate instance of the bot, or just the same bot with a presence in #wikimedia-analytics? [19:54:32] (In the latter case, presumably it will log to the same place as the labs bot) [20:13:05] andrewbogott, separate instance? [20:14:12] drdee: it looks like I can just add your channel to morebots channel list. But then when you log from that channel it'll go to the same place on labsconsole: https://labsconsole.wikimedia.org/wiki/Server_Admin_Log [20:14:14] Is that what you want? [20:14:45] is there a way to configure a different destination? [20:15:16] That would be a different bot. [20:15:24] And presumably you'd have to set up that destination as well... [20:17:25] drdee: Maybe you could tell me what you actually want to have happen? [20:18:08] i would like to have a !log command for the wikimedia-analytics command and have that logged to an analytics dedicated public log page/wiki [20:18:29] "wikimedia-analytics command" => "wikimedia-analytics channel" [20:18:54] drdee: ok! That's a reasonable thing to want :) I don't know offhand how to set up the log page though. [20:19:05] Do you know what host you'd want it on? [20:19:42] you mean, on what labs instance the bot should run? [20:19:59] For hosting the logs, I mean. [20:20:10] (Although that question is also a reasonable one.) [20:20:38] ohhh mediawiki.org/wiki/Analytics/Server_Admin_Log ? [20:21:33] And, you want the same behavior as in labs, where there are multiple projects and you log to specific ones via !log blahblah? [20:21:44] * Damianz looks around [20:22:26] yes that would be awesome! [20:23:49] ok. I will continue to… look in to this :) [20:24:58] INTO THE ABIS [20:25:15] also youtube is ugly now [20:25:17] sadface [20:31:24] * Damianz waves [20:31:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:31:55] howdy [20:33:37] <^demon> Ryan_Lane: Would you mind looking at a cron for hume for me? [20:33:43] sure. which? [20:33:50] <^demon> https://gerrit.wikimedia.org/r/#/c/37429/ [20:34:08] * Damianz notes it's cold [20:34:50] is there really ~800 wikis? :o [20:34:57] yes [20:35:00] yay for only speaking english [20:35:05] probably more [20:35:46] ^demon: by looking at….. ? [20:35:53] you mean approve? heh [20:36:06] <^demon> I meant that implicitly, yeah ;-) [20:36:29] I remember last time chad asked for a cron to be approved.... repos went bye for a few days [20:36:32] * Damianz trolllll [20:37:17] <^demon> I ask for crons all the time ;-) [20:38:22] running [20:38:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:39:07] hmmmrg [20:43:34] <^demon> Ryan_Lane: Thanks [20:44:27] yw [20:49:05] Ryan_Lane: Is labs-morebots already puppetized someplace? [20:49:18] umm [20:49:21] I think so [20:49:24] it's adminbot [20:49:31] hm [20:49:32] wait [20:49:35] no it probably isn't [20:49:50] <^demon> morebots isn't puppetized, nothing on wikitech is puppetized. [20:50:04] OK. I'm going to work on that a bit. [20:50:21] btw, Ryan_Lane, I'd also like a review of https://gerrit.wikimedia.org/r/#/c/37250/ when you have a moment. [20:50:26] <^demon> Well, wikitech is offsite, it doesn't necessarily need to run puppet. [20:50:33] (Sorry if that's a re-nag, can't remember if I asked already) [20:51:04] ^demon: labs-morebots runs on a labs instance, so should be. And since drdee wants a second instance of the bot, seems like a good reason to work on it today. [20:51:15] oh [20:51:17] <^demon> Well labsmorebots, maybe. [20:51:22] I don't think I've seen this at all [20:51:25] <^demon> But The Original morebots isn't. [20:51:54] ^demon: OK. I'm pretty confused by (more/labs/log)bot terminology. [20:53:13] <^demon> The bot that runs in -operations, "morebots," does not run on labs, it runs on wikitech. "labs-morebots" presumably runs on labs somewhere but I know nothing about it. [20:53:35] <^demon> "logmsgbot" runs on fenari, and relays stuff from like scap to irc. [20:54:12] Ryan_Lane, for an example of what that patch does, click on the [?] next to 'singlenode' on this page: https://i-00000236.pmtpa.wmflabs/w/index.php?title=Special:NovaInstance&action=configure&instanceid=a24c0817-4876-4c19-b58f-c67b40dedf70&project=failtest®ion=nova [20:54:22] (If that link survives translation) [20:55:07] ^demon: Is 'morebots' this? https://gerrit.wikimedia.org/r/#/admin/projects/operations/debs/adminbot [20:55:40] <^demon> Yes. [20:56:00] ok. So, same bot, different site. [20:56:35] is wikitech part of toolserver, or a WMF site, or what? [20:56:37] lol [20:56:46] it's on a linode [20:57:29] So totally unrelated? Someone's private hobby-project? [20:57:31] the bot is rather a mess tbh... not really what you'd call a python package or a nicly developed and tested bit of software imo [20:57:38] <^demon> andrewbogott: No, offsite docs. [20:57:47] Well it's on a vm so if the cluster goes down you have docs [20:57:48] <^demon> So in case you're like "Oh shit, site's down, how do I fix [foo]" [20:57:53] <^demon> The docs aren't down too then :) [20:57:57] is here anybody who would like to explain how I can run my bot at labs? [20:58:08] Ah, so it's /run/ by WMF but just hosted elsewhere because of being smart [20:58:13] benestar: After I finish eating if no one else does in the mean time [20:58:21] <^demon> andrewbogott: Yup. [20:58:29] well no... being smart would be a read only offsite copy with a proper on site one so ssl etc can be used [20:58:32] Damianz: thanks a lot :) [20:58:33] * Damianz growls at lack of ssl [20:58:52] benestar: You'll start by wanting a labs account and probably shell access. Do you have either of those things? [20:59:05] yes, I have got it [20:59:17] have a look here: https://labsconsole.wikimedia.org/wiki/Shell_Request/Bene [20:59:48] OK, in that case I will refer you back to Damianz. Probably you don't need your own private VM and can share with an existing one. [21:00:28] ok, i will wait until he has finished eating [21:01:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:01:47] nom pizza [21:01:57] benestar: In case this isn't obvious… the deal is that Labs itself is largely administered by WMF staff (e.g. me) but the actual governance of particular projects like 'bots' is generally handled by volunteers (e.g. Damianz). [21:02:25] i understand [21:02:31] tbh I dislike the first part of that statement [21:02:35] …although that distinction barely matters because in either case this is the place to ask :) [21:02:53] it should all be administered by everyone - granted we'll have roots for security reasons but puppetize it and accept patches [21:02:54] Damianz: Well, we aspire to have it be less true. But it's still sort of true today, right? [21:03:03] today yes, but lets change the world [21:03:47] yep! [21:05:53] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 2.96, 3.89, 4.81 [21:06:29] Damianz andrewbogott: https://www.youtube.com/watch?v=d8pmCSItNDs&feature=g-user-u Sorry, couldn't resist. OK, back to serious. [21:08:34] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:08:37] lol [21:09:12] I was thinking more https://www.youtube.com/watch?v=ZM9ssz7FiUg [21:09:30] * benestar 's bot will soon make it's 100,000 edit: https://toolserver.org/~tparis/pcount/index.php?name=BeneBot*&lang=de&wiki=wikidata [21:11:16] Damianz: LOL [21:11:43] andrewbogott: my account isn't in the project requested :) [21:12:05] also, wikitech is on a linode server. managed by us [21:12:05] oops, one sec [21:12:25] it's off-cluster because we need the documentation to bring the site up sometimes [21:12:37] and if it was on-cluster it would be down ;) [21:13:05] Ryan_Lane: OK, try now [21:13:16] oh [21:13:18] awesome [21:13:20] that's really cool [21:13:46] yeah - it's fucking freezing [21:13:49] would it be possible to make it pop up in an iframe in a lightbox or something? [21:13:52] * Damianz turns the heating up again [21:13:53] It would be cooler if 90% of the linked pages weren't empty [21:13:57] heh [21:14:09] Ryan_Lane: Probably but I don't know how [21:14:25] I'm sure jquery has something for this. [21:14:26] lemme see [21:15:00] people really use jquery still? [21:15:03] * Damianz hugs his backbone [21:31:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:32:02] Damianz: still eating? [21:34:45] <^demon> Ryan_Lane: Have to shut that cron back off :( https://gerrit.wikimedia.org/r/#/c/37504/ [21:34:57] <^demon> (Cron's fine, script just having problems, and we're calling it a weekend) [21:36:25] -_- [21:36:29] stabby stabby stab [21:36:46] <^demon> The script works perfectly fine when run by hand. [21:36:58] <^demon> Something's breaking when it's run as a cron, and nothing's showing up in the error log. [21:36:59] as mwdeploy? [21:37:02] <^demon> Yeah [21:38:47] <^demon> So yeah, it's almost 5pm on the east coast and even later in Europe, so me and Daniel and Katie are giving up for the weekend :p [21:39:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:39:27] yeah [21:39:34] seems like a good idea ;) [21:56:42] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 4.92, 4.63, 4.98 [21:56:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 4% free memory [21:58:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 4.63, 4.34, 4.77 [22:01:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:09:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:26:42] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 3.85, 4.49, 4.93 [22:31:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:39:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:41:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [23:01:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:09:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:31:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:40:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100%