[00:01:54] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 7.93, 7.96, 6.42 [00:04:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:05:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:08:13] RECOVERY Current Load is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: OK - load average: 0.50, 1.25, 4.97 [00:11:33] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 5.30, 6.11, 5.59 [00:22:22] PROBLEM Disk Space is now: CRITICAL on kubo i-000003dd.pmtpa.wmflabs output: DISK CRITICAL - free space: / 286 MB (2% inode=66%): [00:36:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:36:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:37:08] andrewbogott: did you have any pending OpenStackManager changes that haven't been deployed? [00:37:22] because i'm likely to deploy one really soon [00:37:30] Just the one in gerrit that I linked you to yesterday. [00:38:34] Ryan_Lane: you can merge that if it looks ok and do a 'git reset --hard' in production [00:41:03] ok. sec [00:41:24] ah. this one is already in place? [00:41:38] ah [00:41:38] right [00:41:41] forgot [00:41:46] yeah, this looks fine [00:42:38] thanks [00:42:58] weird. there's a slot0 link in w? heh [00:43:20] * Ryan_Lane removes that [00:43:49] ok. now to see if I broke things [00:44:32] -_- [00:44:33] sure did [00:44:45] ah [00:44:45] right [00:44:51] I need to update ldap extension too [00:45:15] better [00:46:44] that seems to be working [00:46:52] hopefully I just fixed the "No nova credentials" issue [00:46:57] minus when preferences change [00:47:06] and minus when a user's account is first created [00:47:07] oh! [00:47:15] let me see if I can fix the preferences issue [00:47:42] the problem was that memcache would be restarted and the user's token would be gone [00:48:01] but they are still logged in because they had a long-lived token [00:48:04] yeah [00:48:12] I figured it had something to do with memcached being restarted [00:48:39] so, when the user logs in, if they selected to have a long-lived token, I stick the generic openstack token into their user options, like I do with the ldap domain [00:48:57] so, they can re-fetch all of their project specific tokens from the generic one [00:49:19] if they don't have a long-lived token they'll be logged out anyway [00:49:22] so, that should do it [00:49:44] preferences should be a easy fix [00:49:48] * Ryan_Lane gets to it [00:50:09] hm. I say that.... [00:50:17] I may need to make a core change for this [00:50:57] shit, you know, I shouldn't have deployed that change without a review.... [00:51:01] * Ryan_Lane sighs [01:06:33] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 174 processes [01:06:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:06:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:14:44] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 6.85, 6.47, 5.45 [01:21:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 4.68, 5.38, 5.11 [01:26:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.47, 4.60, 4.88 [01:29:42] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 3.28, 4.13, 4.71 [01:34:32] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 4.30, 5.27, 5.56 [01:36:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:36:54] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:47:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 5.21, 5.88, 5.37 [01:56:38] fixed No nova creds when changing preferences [01:57:03] yay [01:57:33] I need to merge those changes [01:58:15] I pushed a change into core to let us change our email addresses from labsconsole, too [01:59:09] ack. the attributes to be updated in the ldap extension are hardcoded [01:59:15] I definitely need to fix that [01:59:29] don't want people changing their CN when they change their real name [02:00:41] shit [02:00:45] I take it back. I didn't fix that [02:00:49] :( [02:02:37] hm [02:06:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:07:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:09:33] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 3.66, 4.36, 4.83 [02:15:16] bah [02:35:23] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 149 processes [02:36:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:37:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:37:43] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 21% free memory [02:57:32] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 5.15, 5.50, 5.19 [03:05:42] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 15% free memory [03:07:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:08:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:12:32] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 4.51, 4.76, 5.00 [03:15:35] Anyone here? [03:38:34] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:39:14] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:49:55] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 7.11, 6.30, 5.48 [03:54:53] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.39, 4.44, 4.90 [03:55:32] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 4.33, 4.79, 5.32 [04:07:02] PROBLEM Current Load is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [04:07:32] PROBLEM Current Users is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [04:08:12] PROBLEM Disk Space is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [04:08:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:09:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:11:43] RECOVERY Free ram is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: OK: 25% free memory [04:12:03] RECOVERY Current Load is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: OK - load average: 0.17, 0.94, 1.04 [04:12:33] RECOVERY Current Users is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [04:13:13] RECOVERY Disk Space is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: DISK OK [04:14:14] bleh. seems I need to add some database code to properly fix the "no nova credentials" problem [04:14:20] in both OSM and ldap extensions [04:14:26] * Ryan_Lane sighs [04:17:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 6.69, 6.08, 5.41 [04:35:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 7.65, 6.22, 5.36 [04:38:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:40:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:40:33] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 3.52, 3.80, 4.71 [04:40:43] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 3.34, 4.76, 5.00 [04:44:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 19% free memory [05:09:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:10:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:10:23] Ryan_Lane: If you get bored coding database fixes you can always look over the new account requests [05:10:28] hint hint [05:10:36] heh [05:26:33] Hi, i posted a request at http://deployment.wikimedia.beta.wmflabs.org/wiki/Global_Requests#Extensions about a week ago [05:26:35] could you please tell me who should i ask additionaly to process this request [05:30:02] probably ^demon [05:30:05] oh [05:30:05] wait [05:30:06] no [05:30:12] maybe hashar [05:32:53] odie5533: done. by email [05:33:26] Change on 12mediawiki a page Developer access was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=607947 edit summary: /* User:Odie5533 */ [05:37:23] Ryan_Lane: thank you! [05:39:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:40:55] I know this isn't the main purpose of WMF Labs, but how far along is the database replication? [05:41:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:47:05] well, it's one of the main purposes [05:47:10] a month or two [05:47:21] odie5533: ^^ [05:47:58] thanks. Until then the http://www.mediawiki.org/wiki/API:Main_page should be used? [05:48:21] yep. we also have dumps available, if you can use the raw dumps [05:48:29] at /public/datasets [05:48:46] is that shared on all servers? [05:48:49] yep [05:48:57] accessible from every instance [05:49:08] neat setup! [05:49:31] I try :) [06:06:08] Labs instances are not hooked up to SUL, correct? [06:09:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:11:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:24:19] StevenW: eh? [06:24:28] in which way? [06:24:35] labs in general is not [06:24:39] ok [06:24:55] we're reworking the landing page you get post-registration, which includes the SUL icons normally, like login [06:25:00] ah [06:25:08] umm [06:25:18] you know those icons log you in, right? :) [06:25:22] yep [06:25:27] ok. just checking. heh [06:25:43] would be awkward for them to be removed in the name of being tidy :D [06:25:53] yeah you should have been in the room when everyone found out the images log you in. It was hilarious. [06:25:58] heh [06:26:09] there's definitely better ways to go about it [06:26:41] It's ok. If we don't want them there we can try making them one pixel so on and for forth. Imma talk to Forrester and maybe Brion about it. [06:26:41] javascript, making the requests via ajax, with a % to tell you when it's complete, for instance [06:26:54] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.49, 4.72, 4.94 [06:26:59] well, you can only do that if you're going to replace it with some other indicator [06:27:07] it's the only way currently for people to know they can leave the page [06:27:19] if they leave it before they all load they won't be logged into whichever one is missing [06:27:35] good point [06:27:52] 1px images is fine if you have some javascript indicator of completeness [06:28:05] and larger images if js isn't available [06:28:24] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 153 processes [06:29:12] thanks for checking about labs :) [06:29:19] of course [06:29:24] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:29:32] I mean, we have to know about that first, since we're building it out on Labs [06:29:53] our an instance, rather [06:29:53] * Ryan_Lane nods [06:32:34] PROBLEM Total processes is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS WARNING: 153 processes [06:33:24] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 148 processes [06:39:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:39:22] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 147 processes [06:40:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 6.50, 5.58, 5.23 [06:41:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:45:53] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.45, 4.82, 5.00 [06:47:33] RECOVERY Total processes is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS OK: 149 processes [07:00:27] Change on 12mediawiki a page OAuth was modified, changed by Smile4ever link https://www.mediawiki.org/w/index.php?diff=607961 edit summary: [07:09:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:11:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:39:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:41:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:09:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:12:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:33:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 4.94, 5.07, 5.12 [08:41:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:42:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:11:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:13:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:43:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:43:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:53:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.33, 4.45, 4.87 [10:13:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:14:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:14:52] PROBLEM Free ram is now: WARNING on dumps-bot3 i-00000503.pmtpa.wmflabs output: Warning: 19% free memory [10:25:06] Change on 12mediawiki a page Developer access was modified, changed by Poornima link https://www.mediawiki.org/w/index.php?diff=608038 edit summary: [10:43:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:44:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:48:41] Ryan_Lane: ping [11:01:31] jeremyb: most definitely sleeping since it is 3am in SF :) [11:13:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:14:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:43:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:44:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:13:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:14:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:19:53] PROBLEM Free ram is now: WARNING on dumps-bot2 i-000003f4.pmtpa.wmflabs output: Warning: 19% free memory [12:22:42] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 18% free memory [12:27:42] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 20% free memory [12:43:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:44:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:46:41] Is Gerrit required to create and operate bots? [13:00:43] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 18% free memory [13:14:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:15:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:14:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:15:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:16:20] all these warnings it looks like the place is falling apart [14:44:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:45:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:49:30] odie5533: they are usually the same instances being spammed every half hour :/ [14:49:37] probably disabled by their admins [14:53:48] ah, that makes sense [15:14:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:15:24] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:44:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:46:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:14:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:16:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:44:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:46:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:03:16] From whom do I request access to the bot project? [17:03:41] hi odie5533 - you can email the labs-l mailing list [17:05:07] sumanah: is there any specific request process? [17:05:28] I don't know, sorry [17:08:08] odie5533: not really [17:08:25] Damianz and petan in this channel can also help you [17:08:29] they manage the project [17:08:53] hi [17:08:57] hey Ryan_Lane got a moment? [17:09:02] sumanah: sure [17:09:06] Hi again petan! [17:09:06] odie5533 need a help? [17:09:13] oh petan is here. :) [17:09:21] yup [17:09:35] Ryan_Lane: ok, it's a few different things, will PM [17:09:40] Yes. I am interested in working on the bots project, possibly by creating my own bots or creating tools access the API but do not edit. [17:09:51] ok, what is your labs username [17:09:57] odie5533 [17:09:59] with a capital O [17:10:09] !botsdocs [17:10:09] https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bots/Documentation [17:10:18] this is a good page to read first :) [17:10:33] I have read it through before [17:10:57] ok [17:11:01] Thought I am not sure how you have the servers divided up [17:11:07] then you know everything you need [17:11:10] that is also there [17:11:17] !log bots added odie5533 to project [17:11:21] I guess new projects would be added somewhere to bots-2 through 4 or something [17:11:34] andrewbogott you really want to fix logbot [17:11:40] it's dead AGAIN [17:11:47] <3 c# [17:11:51] !ping [17:11:51] pong [17:11:52] it's so nice having IRC bots - when they are around [17:11:58] you see that thing didn't die for months ^ [17:12:09] but all it does is respond to pings [17:12:14] nah [17:12:17] it does everything [17:12:35] http://bots.wmflabs.org/~wm-bot/dump/%23wikimedia-labs.htm [17:12:38] look [17:12:42] :D [17:12:55] I have to go now [17:12:56] have fun [17:13:06] !botsdocs read this | odie5533 [17:13:06] odie5533: https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bots/Documentation [17:13:08] petan: The problem last time is that we were trying to make the bot on bots-2 work which wasn't the actual modern version [17:13:41] petan: alright, thank you [17:14:05] andrewbogott ok, in fact bots-2 is not a best place [17:14:10] but I don't really care [17:14:26] it would be ideal to have a specific instance to run these lighweight bots or labs related stuff [17:14:41] because there is a huge bot running on bots-2 that usually take whole machine down [17:14:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:15:55] odie5533: ideal box for you is: bots-4 if you need root or bots-nr1 if you don't [17:16:12] petan: there is, I believe it's bots-labs [17:16:18] I assume I won't need root unless the box is missing dependencies [17:16:28] bots-4 is not really good idea if you have passwords stored in plain text because that's where everyone has root [17:16:43] andrewbogott I don't know, maybe it is [17:16:45] it's bots-labs for the log bot [17:16:48] it's documented [17:16:49] aha [17:16:56] ok [17:17:08] I think it died yesterday when dhcp went crazy for a few minutes [17:17:19] it's supposed to reconnect, but doesn't do so properly [17:17:25] Ryan_Lane maybe implement auto-reconnect? [17:17:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:17:47] btw I have to go [17:17:53] :) [17:17:53] * Ryan_Lane waves [17:17:56] have fun [17:17:57] :D [17:18:00] you too [17:18:28] odie5533: if you needed anything, just send a mail [17:18:30] @search mail [17:18:30] Results (Found 4): osm-bug, new-labsuser, account-questions, mail, [17:18:34] !mail [17:18:34] we have a mailing list labs-l@lists.wikimedia.org feel free to send a message there, don't forget to subscribe [17:18:37] there ^ [17:18:39] :P [17:18:46] alright, thank you [17:21:50] petan: if you have improvements for https://labsconsole.wikimedia.org/wiki/Help:Move_your_bot_to_Labs please do go ahead and edit [17:30:55] sumanah: wow that is a useful article! [17:31:07] I didn't see it linked to from anywhere [17:31:10] odie5533: yes! please feel free to link to it wherever you had been looking for information [17:31:14] please please please link to it [17:36:46] sumanah: I added it to the help contents and to the page about bots [17:37:01] Thank you odie5533 [17:37:07] odie5533: you might also be interested in https://blog.wikimedia.org/2012/11/21/lead-development-process-product-adviser-manager/ [17:37:54] heh, not for me :) [17:38:12] odie5533: maybe you know people who would be interested, though [17:38:27] I don't, but if I meet one I will let them know [17:38:32] Thanks [17:39:01] also odie5533 check out the tech meetings calendar to see if there's one near you - https://www.mediawiki.org/wiki/MediaWiki_developer_meetings [17:39:58] there's only one in my country and it's nowhere near me :( [17:40:43] sumanah: what is your position within WMF/MediaWiki? [17:40:56] or area [17:40:58] odie5533: I am Engineering Community Manager [17:41:01] for Wikimedia Foundation [17:41:13] ah [17:41:13] I help encourage contributors and try to get obstacles out of their way [17:41:24] odie5533: you can always host an event yourself [17:41:29] wow, that's a good position to have [17:41:43] jobs.wikimedia.org -- WMF is hiring :-) and telecommuting is ok [17:41:52] sumanah: I don't know much about mediawiki actually [17:41:56] that's ok [17:42:00] it's a big ecology [17:42:14] Labs, tools, gadgets, bots, MediaWiki, analytics, mobile..... [17:42:29] it is huge [17:42:31] people contribute a lot to Wikimedia technology without knowing much about MediaWiki [17:42:43] I've so far contributed gadgets and outside tools [17:45:48] but hopefully tools and bots are in my future [17:45:57] neat [17:46:13] odie5533: you might be eligible for https://meta.wikimedia.org/wiki/Participation:Support to go to events outside your city [17:46:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:47:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:47:54] uhm, I don't think I'd qualify for that. Also, we have the internet. so we don't really need to travel places just to participate in things [17:48:45] I figure both face-to-face and net have their advantages for different kinds of collaboration [17:48:47] I know, from experience [17:59:11] odie5533: btw you might be interested in watching some videos from our last big event, on Git/Gerrit, Lua for templates, Wikidata, etc. https://commons.wikimedia.org/wiki/Category:Hackathon_Berlin_2012 [18:00:19] thanks. I saw one of them on WMF Labs but didn't realize there were more vids [18:00:48] the one I saw recommended getting a tool server account for bots, but I've heard from other users that the toolserver is being phased out in favor of WMF Labs. [18:01:02] the video you saw might have been from January 2012? [18:01:38] maybe [18:11:07] How does the Webtools differ from the Bots project? Can Tools also be hosted on the Bots project or should I request access to Webtools if I plan to write a tool? [18:11:31] Platonides: petan: Damianz ^ [18:12:03] I'd imagine tools shouldn't be hosted in bots [18:12:12] does your tool need data from the bot/ [18:12:56] Are bots defined as making edits to Wikipedia? And so something that scrapes the API but does not edit is not a bot? [18:13:27] well, it can still be a bot [18:13:44] But let's assume it does not edit. [18:14:02] well, let's take counter-vandalism network [18:14:12] it reads from the API and spits messages out into the channels [18:14:20] it doesn't edit either, but it's a bot [18:14:48] ah, okay. well I think for my first project I'd want to just write a Tool then and not really a bot [18:14:54] * Ryan_Lane nods [18:15:10] the webtools project doesn't really any infrastructure right now [18:15:11] so Webtools would be the right project for that? [18:15:17] yeah [18:15:29] that project really needs a champion of some regard [18:15:39] The Bots project has public web access so I figured that would be a good place. [18:15:40] we are just now starting on toolserver like features [18:15:41] you'll want to be on the labs-l mailing list, odie5533 https://lists.wikimedia.org/mailman/listinfo/labs-l [18:16:03] <^demon> Ryan_Lane: Shawn said we'll try to get my inherit groups thing reviewed + merged next week. [18:16:07] also you may want to look at the schedule at https://www.mediawiki.org/wiki/Wikimedia_Labs [18:16:10] <^demon> (It's a painful schema migration for them) [18:16:11] ^demon: awesome [18:16:16] sumanah: Thanks for the link. I will keep it bookmarked, but I generally like just reading through the archives when I need to rather than getting lots of emails [18:16:23] sure [18:16:24] ok [18:16:41] odie5533: if you are interested in ops-like work, then you could help build webtools :) [18:16:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:16:51] you may want to someday join it and simply turn "receive mail" off in mailman, so you can send mail when you need to [18:17:26] labs-l is relatively low volume [18:17:35] 1-2 emails per week mauybe [18:17:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:17:53] Ryan_Lane: I'm not really sure I am. Webtools probably needs someone to configure a web server on it like nginx and help partition areas for people to work in, but I'm not really up to that kind of task. I could work in my own little instance serving up pages from my own program, but I'm not really a sysadmin or know how to be. [18:18:40] also odie5533 in case you are a woman or know women who are looking for paid internships Jan-March 2013 to contribute to Wikimedia technology, https://blog.wikimedia.org/2012/11/15/apply-for-the-foss-outreach-program-for-women-internships/ [18:18:54] just realized I should be plugging that as well as long as I am spamming :) [18:19:01] Ryan_Lane: So I don't need the infrastructure, but I only have experience building it for my own little projects (e.g. nginx chained to Python or something like that, but not for many people) [18:19:12] * Ryan_Lane nods [18:20:02] Ryan_Lane: Should I still be working on Webtools though, and just request my own instance (very small) until infrastructure is built in that project? [18:20:14] sure [18:20:19] no need to request an instance [18:20:24] you can be made sysadmin [18:20:46] we can lock down roles/permissions when the project gets going, then your stuff can be rolled into the larger solution [18:21:36] * sumanah looks at https://labsconsole.wikimedia.org/wiki/Help:Terminology re "sysadmin" [18:21:49] odie5533: I added you to the project and to sysadmin and netadmin roles [18:21:56] !security [18:21:56] https://labsconsole.wikimedia.org/wiki/Help:Security_Groups [18:21:59] !instances [18:22:00] need help? -> https://labsconsole.wikimedia.org/wiki/Help:Instances want to manage? -> https://labsconsole.wikimedia.org/wiki/Special:NovaInstance want resources? use !resource [18:22:04] make sure to read the docs on that :) [18:22:27] Thank you [18:22:31] yw [18:23:05] sumanah: I'm afraid I don't know any women that would be interested in that, and I am not one either. [18:23:50] OK. We have several applicants for whom this is the first open source project they've ever thought about contributing to -- women who will start in documentation, sysadmin, communications, visual/UX design, or software development [18:24:25] in case that expands the possibilities and makes you think "oh yeah, she might be interested then" [18:25:03] sumanah: have you talked to codechix people in the bay area? [18:25:22]