[00:00:16] https://labsconsole.wikimedia.org/wiki/Help:Contents#Administration < we haz like documentation shit on this [00:00:42] I don't have the ability to add someone to that particular wiki group. [00:00:57] https://labsconsole.wikimedia.org/wiki/Special:ListGroupRights - found that. [00:01:15] when you said it was a wiki group, I thought you meant it was a group of wikis. [00:01:32] I understand better now. [00:02:37] no idea what mw calls them [00:03:06] ok .... to solve the immediate problem, Damianz, can you please add user:mgrover to that MediaWiki group? [00:03:20] I don't have the relevant privileges [00:03:25] nope [00:03:35] I don't have rights on prod to do it [00:04:17] ok. jcmish I suggest that you email the labs-l mailing list to ask to be added to the "shell" group on labsconsole.wikimedia.org [00:04:33] Sumanah will do [00:04:49] you will probably want to subscribe first, if you have not already [00:06:16] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:06:49] sumanah yup just did :) [00:06:58] ok! :) [00:17:05] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:19:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [00:32:13] !log sugarcrm restarting apache to pick up more specific settings for sugar [00:32:13] Logged the message, Master [00:36:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:47:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:49:25] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [01:05:52] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 175 processes [01:06:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:10:52] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 95 processes [01:17:06] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:19:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [01:37:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:47:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:49:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [01:49:22] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 144 processes [01:56:39] !log sugarcrm restarting apache to pick up civi [01:56:45] Logged the message, Master [02:08:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:17:05] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:19:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [02:38:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:47:05] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:49:23] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [03:08:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:17:12] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:19:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [03:39:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:48:14] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:49:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [04:09:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:18:48] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:19:46] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [04:39:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:48:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:50:23] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [05:09:46] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:19:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:21:23] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [05:40:35] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:49:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:52:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:11:27] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:21:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:23:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:41:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:51:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:53:06] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [07:11:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:21:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:24:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [07:32:33] [bz] (8NEW - created by: 2Antoine "hashar" Musso, priority: 4Unprioritized - 6normal) [Bug 41530] setup redis on beta - https://bugzilla.wikimedia.org/show_bug.cgi?id=41530 [07:41:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:51:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:54:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [08:11:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:13:55] PROBLEM Free ram is now: WARNING on dumps-bot2 i-000003f4.pmtpa.wmflabs output: Warning: 19% free memory [08:18:43] jeremyb ping [08:22:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:24:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [08:32:23] PROBLEM Free ram is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: Warning: 17% free memory [08:41:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:47:32] PROBLEM Free ram is now: CRITICAL on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: Critical: 5% free memory [08:52:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:54:23] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [08:57:32] PROBLEM Free ram is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: Warning: 6% free memory [09:05:12] !ping [09:05:12] pong [09:11:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:13:37] petan: hello, I answered you on wikitech-l about using puppet in labs without having the change merged by ops [09:13:37] :) [09:13:50] saw that [09:13:50] cool [09:24:06] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:24:14] !ping [09:24:14] pong [09:24:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [09:30:42] !ping [09:30:42] pong [09:31:01] :/ [09:31:01] meh [09:31:01] I wondering if labs are lagging or just a bot [09:41:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:47:23] PROBLEM Free ram is now: CRITICAL on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: Critical: 2% free memory [09:53:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:54:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [10:10:47] PROBLEM Free ram is now: WARNING on aggregator-test1 i-000002bf.pmtpa.wmflabs output: Warning: 19% free memory [10:11:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:23:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:24:44] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [10:42:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:47:23] PROBLEM Free ram is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: Warning: 18% free memory [10:53:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:57:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [11:02:32] RECOVERY Free ram is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK: 49% free memory [11:12:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:19:03] hey petan [11:23:23] hi [11:23:25] jeremyb what did u want [11:23:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:28:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [11:32:31] jeremyb :D [11:32:33] are u here [11:32:45] you better send me an e-mail next time [11:42:26] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:47:06] petan: brb [11:48:10] petan: how do i recompile wmib? [11:48:18] @help [11:48:18] Type @commands for list of commands. This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.8.26.8 source code licensed under GPL and located at https://github.com/benapetr/wikimedia-bot [11:48:25] get a source, build it :P [11:48:38] petan: can we get a compilation env on labs? or is there one already? [11:48:52] tbh I am using visual studio to build it [11:48:56] ugh [11:48:58] but u can use monodevelop and generate makefile [11:49:06] ok, let's fix that eventually... [11:49:06] for now: [11:49:07] then u would just ./configure [11:49:21] see the latest diff in svn. from june [11:49:26] nah [11:49:34] latest diff is from today [11:49:37] huh? [11:49:40] in svn? [11:50:05] no [11:50:07] @help [11:50:07] Type @commands for list of commands. This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.8.26.8 source code licensed under GPL and located at https://github.com/benapetr/wikimedia-bot [11:50:10] see link [11:50:12] it's in git for ages [11:50:31] svn has ancient version [11:51:06] petan: http://svn.wikimedia.org/viewvc/mediawiki/trunk/tools/wmib/?view=log [11:51:29] petan: let's put a big note in SVN saying where the new place is [11:51:37] also the wiki docs still point to SVN [11:51:43] really? [11:51:46] !wm-bot [11:51:46] http://meta.wikimedia.org/wiki/WM-Bot [11:52:32] yes, really [11:52:47] fixed [11:53:04] I don't really know how to put "notes" to svn :P [11:53:11] danke [11:53:23] petan: make a new commit to README? [11:53:27] right [11:53:38] with a scary commit msg [11:53:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:55:53] @reload [11:55:53] Channel config was reloaded [11:56:00] hrmm [11:56:27] ok, so is there anything you want to fix? [11:56:34] we have a bugzilla category too [11:57:13] https://bugzilla.wikimedia.org/enter_bug.cgi?product=Tools [11:57:13] wm-bot [11:58:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [11:58:04] petan: i found in the source if (File.Exists("sites")) [11:58:10] so then i edited sites [11:58:17] let's see if that works [11:58:24] yes this file contains definition of irc channels [11:58:39] did you change it on bots-1? use !log for that pls [11:58:46] meta was wrong [11:58:53] i will... [11:58:53] i've been logging [11:59:00] check for yourself ;) [11:59:14] ok, so you changed it yesterday? [11:59:39] 2012-10-30 11:55 /mnt/share/wmib/sites [11:59:54] it was changed minutes ago [12:00:07] by me [12:00:10] ok [12:00:33] it worked without a restart [12:00:33] apparently [12:00:33] !log bots root: restarted wm-bot [12:00:40] !ping [12:00:40] pong [12:00:47] done [12:01:39] this page is useful http://bots.wmflabs.org/~petrb/db/systemdata.htm [12:02:12] how often do those pages update? [12:02:32] um, every hour or everytime something except for uptime changes [12:02:56] like if u @add or @part [12:03:11] but these events are in log [12:03:24] see wmib.log [12:12:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:19:25] petan: so... i'm about done. everything's working well i think. (did test edits). do i really need to boot the bot? or did you? (you !log'd it but i didn't see it part and return) [12:19:40] i don't really understand how bouncer works [12:19:40] it never part [12:19:45] there is bouncer [12:19:54] i got that much ;) [12:19:58] i booted bouncer recently [12:20:00] it's another program which handle connection to network [12:20:14] so that bot never needs to reconnect to freenode [12:20:23] unless connectivity dies on side of labs [12:20:25] except when there's a netsplit! [12:20:29] or that [12:20:53] I reboot this bot usually many times a day, just no one notices that [12:20:59] orly [12:21:06] you can see sysdata page for last time of reboot [12:21:09] k [12:21:14] because I do a lot of patches :D [12:21:45] !log bots [bots-1 wm-bot] fixed metawiki's info in ./sites && cleaned up the channels that were watching the original one. tested 2 channels and they both work! [12:21:46] but it rarely crashes itself [12:22:11] jeremyb did it work without rebooting the bot? [12:22:13] I doubt [12:22:24] 30 12:19:24 < jeremyb> petan: so... i'm about done. everything's working well i think. (did test edits). do i really need to boot the bot? or did you? (you !log'd it but i didn't see it part and return) [12:22:24] it should read the sites file only when module is loaded [12:22:29] all i did was @reload [12:22:32] hmm [12:22:38] maybe that does it as well heh [12:22:43] I don't even remember that [12:22:57] I will implement commands to reload modules [12:23:01] that will make it easy [12:23:21] but first I need to separate them from core [12:23:24] working on that now [12:23:36] I would like to make a separate library for each of them [12:23:43] so I would just upload another binary and loaded it [12:23:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:24:54] !log bots [bots-1 wm-bot] fixed metawiki's info in ./sites && cleaned up the channels that were watching the original one. tested 2 channels and they both work! [12:24:56] Logged the message, Master [12:25:15] !log bots [bots-labs labs-morebots] booted again... [12:25:15] Logged the message, Master [12:29:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [12:42:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:54:15] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:59:15] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [13:05:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [13:10:33] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 107 processes [13:12:35] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:24:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:27:36] hashar- Internet is being flaky for me this morning, but should we try this beta stuff? [13:29:26] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [13:40:35] hashar- you here? [13:40:46] anomie: I am [13:41:01] your first ping did not work in my client sorry :/ [13:41:21] hashar- I hope my Internet connection stops being flaky now. Should we get started on the beta thing? [13:42:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:44:19] anomie: suree [13:44:33] anomie: did you get a change to read about https://labsconsole.wikimedia.org/wiki/Deployment/Overview ? [13:44:51] that is supposed to give an overview about the beta cluster [13:44:52] and its setup [13:45:00] though I did not kept it up to date [13:45:38] anyway, are you familiar with the production cluster configuration files for MediaWiki? InitialiseSettings.php / CommonSettings.php [13:46:45] I've looked at that page, and looked at those files in the past, but I wouldn't say I'm intimately familiar with any of it yet. [13:47:00] it is ok [13:47:43] the beta URL are of the format ..beta.wmflabs.org which are handled by a cache frontend running squid [13:48:03] squid in turns load balance the request to two apaches instances named deployment-apache32 and deployment-apache33 [13:48:08] that is where apache and mediawiki are installed [13:49:36] ugh, this is going to be a long day if this internet trouble keeps up. [13:49:42] then it is processed by CommonSettings.php which contains all the MediaWiki configuration [13:49:44] dohh [13:49:50] and I don't see when you hide / part :- [13:49:57] hashar- Ok, the beta URL are of the format ..beta.wmflabs.org [13:49:59] and I don't see when you join/part hehe [13:50:10] squid in turns load balance the request to two apaches instances named deployment-apache32 and deployment-apache33 [13:50:31] and the request is handled by CommonSettings.php / InitialiseSettings.php [13:51:21] anyway, that means that changing a wiki configuration is all about knowing PHP :-] [13:51:39] Good thing I know PHP ;) [13:51:58] files are hosted in a git repository named operations/mediawiki-config.git [13:52:07] git clone ssh://gerrit.wikimedia.org:29418/operations/mediawiki-config.git [13:52:15] ALL changes MUST pass via that repository :-] [13:52:27] and ideally they SHOULD be reviewed by someone else before getting merged :-] [13:52:35] that let us track who is doing what [13:52:41] as well as making sure nobody is going to break something [13:52:53] the configuration files are the same on beta and in production [13:53:07] so a mistake made while configuring beta might kill production wikis :-] [13:53:11] (which is a lot of fun) [13:53:24] hence why we review each other constantly [13:53:47] (doh anomie disappeared again hehe) [13:54:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:54:30] hashar- So if I'm going to make a change, how do I test it first to make sure it won't break something before submitting it to Gerrit so someone else can review it to see if it won't break something? [13:54:47] anomie: we have no test suite yet :/ [13:54:56] so you have to know what you are doing hehe [13:55:00] and/or rely on peer review [13:55:42] When I'm making a change to MediaWiki core, I have it set up so I can test anything at http://localhost/wiki/... [13:56:24] yeah that would be great [13:56:36] anyway once a change is merged in the git repository [13:56:50] when we "git pull" the change in production it is instantly available on test.wikipedia.org [13:56:52] which let us test a few things [13:57:09] but that is a test wiki known as "testwiki" [13:57:19] hence a change made to zhwiki will not reflect there :/ [13:57:33] we would need a test suite for that [13:57:35] on beta things are a bit different [13:57:47] whenever you git pull a change, it is instantly deployed [13:58:01] the apaches servers use an NFS share to read the config [13:58:08] so whenever you write to the NFS share, it is available to apaches [13:59:25] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [14:01:09] anomie: I need to find out a bug that got assigned to you [14:03:12] hashar- So on beta, whenever you git pull a change it is instantly deployed [14:03:33] yup [14:03:33] you can also edit the beta configuration directly [14:03:39] to test it out live [14:03:50] then commit the change and push it to gerrit [14:04:15] though testing live means that whenever there is an error (such as a PHP parser error) that is instantly deployed :-] [14:05:05] So is testing something on beta that way a good idea, or should it be avoided unless absolutely necessary? [14:05:26] I usually do the change directly in beta :/ [14:05:40] well exactly, I do the change locally [14:05:43] copy paste it to beta [14:05:45] test it [14:05:48] eventually alter it [14:05:54] then commit + send to gerrit [14:06:12] ideally I would write a test locally, change the configuration to get the test to pass [14:06:17] and then push to gerrit [14:06:23] (but we don't have a test suite yet) [14:07:24] ahh found the bug [14:07:25] https://bugzilla.wikimedia.org/show_bug.cgi?id=39082 [14:07:34] assigned to bjorsch@wikimedia.org [14:07:46] which is hopefully your bugzilla account [14:08:01] yes, it is [14:08:18] (just changed it yesterday from my personal email) [14:08:24] nice [14:09:11] so on beta we have some specific configuration we do not want to apply in production [14:09:54] you might want to fetch the git repo [14:09:59] git clone ssh://gerrit.wikimedia.org:29418/operations/mediawiki-config.git [14:10:38] Already did ;) [14:10:42] So, it looks like it looks in /etc/wikimedia-realm to decide which set of configuration files to use [14:10:50] indeed [14:10:50] setting that in $cluster [14:10:51] that file comes from puppet [14:10:54] the configuration system [14:11:32] $cluster is used to differentiate between the datacenters [14:11:40] where beta uses the (incorrect) "wmflabs" [14:12:01] beta is hosted in realm = labs [14:12:09] the cluster is either eqiad or pmtpa [14:12:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:13:31] we have some specific bits of configuration split around [14:13:38] such as the database configurat [14:13:38] ion [14:13:53] for example: db.php is for production [14:13:58] db-wmflabs.php for labs [14:14:21] the bug report https://bugzilla.wikimedia.org/show_bug.cgi?id=39082 is about making our configuration support specific configuration for the new DC [14:14:24] eqiad [14:16:57] that is all a bit messy of course :/ [14:18:28] I see support for realms "labs" and "production", which translate to clusters "wmflabs" and "pmtpa". And then the cluster is just sort of used to load different files all over the place based on switches. The simple route, it seems, would be to just make a new realm (and possible rename "production") mapping to cluster 'eqiad', and then update all these switches as necessary, and then make a puppet config for eqiad machines that sets the right re [14:18:28] alm in /etc/wikimedia-realm, and then get that puppet config deployed to the right machines. Does that sound about right? Or if we wanted to change things up some, there are a few suggestions for other ways to do things in the bug. [14:19:26] sounds good :-) [14:19:45] and you might want to add comment at the top of CommonSettings.php to explain what all those variables are for [14:19:47] and possible values [14:20:04] I forgot to ask, are you familiar with gerrit ? [14:20:47] https://gerrit.wikimedia.org/r/#/dashboard/221 is me [14:21:23] nice [14:21:25] saves us a couple days :-] [14:23:50] grmblbl [14:24:01] we have /etc/wikimedia-cluster but I have no idea where it comes from [14:26:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:29:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [14:36:00] !g I47dab6bb3b8b1c7547ac616857007ee1fd248e20 [14:36:00] https://gerrit.wikimedia.org/r/#q,I47dab6bb3b8b1c7547ac616857007ee1fd248e20,n,z [14:42:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:47:47] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 19% free memory [14:56:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:59:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [15:04:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [15:13:25] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:17:43] RECOVERY Free ram is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK: 22% free memory [15:21:08] hashar- Well, I think I have the config switches adjusted to add the possibility for 'eqiad' to $cluster. Did it slightly differently from what I said above based on bug 39082 comment 2. Should I put it in Gerrit for you, or do you want to see it another way first? Then you'll have to show me puppet, I guess. [15:23:33] oh [15:23:38] soryr [15:23:40] anomie: I did the puppet change already [15:23:52] though it takes a few hours to be deployed on all our servers [15:24:02] https://gerrit.wikimedia.org/r/#/c/30784/ [15:24:20] puppet is not that much different than other things [15:24:21] you hack [15:24:23] send to gerrit [15:24:34] ask for review to peer and/or operations team [15:24:42] they are in #wikimedia-operations [15:24:54] a channel you might want to join on connection [15:26:24] That's 6 channels now, plus two for non-work enwiki stuff plus one for something else that no one ever talks in anymore... [15:26:44] ok then... I wonder if I should rename $cluster to $site to match [15:26:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:27:12] yeah ton of channels :/ [15:27:19] I mostly use -labs -operations [15:27:26] hmm, probably not, other config files seem to use that for "wikipedia"/"wiktionary"/etc [15:27:50] yeah site is not really nice [15:27:55] I would prefer datacenter [15:29:27] that is messy [15:29:35] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 107 processes [15:29:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [15:29:44] the problem is that ops and us use different semantics :/ [15:31:14] anomie: I guess we could use another variable [15:31:43] even if its value comes from a file named /etc/wikimedia-site, nothing prevents us from putting that value in something like $datacenter or $wikimedia-dc or something [15:33:16] I'm starting to think it really would be better to break things up into file-$datacenter.php instead of all these switches, except then we'd have to break up CommonSettings.php a bit better [15:33:56] the whole thing started as a quick hack for beta [15:34:15] indeed, feel free to break things up a bit more [15:35:42] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 18% free memory [15:38:17] anomie: will get out soon to go grab my daughter. [15:38:46] anomie: whenever you submit the change, please add me as a reviewer and feel free to talk about it with ^demon / Reedy ;-) [15:38:53] hashar- ok [15:43:22]