[00:50:53] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 157 processes [01:02:35] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 8.64, 7.62, 6.07 [01:09:22] PROBLEM Total processes is now: WARNING on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS WARNING: 174 processes [01:14:23] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 97 processes [01:45:42] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 6.67, 6.33, 5.34 [01:47:32] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 8.44, 7.06, 5.61 [02:39:23] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 21% free memory [02:44:52] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [02:45:28] 12/21/2012 - 02:45:28 - Updating keys for mwang at /export/keys/mwang [03:02:22] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 15% free memory [03:02:32] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 3.18, 4.25, 4.97 [03:05:34] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 3.30, 3.96, 4.64 [03:50:53] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 149 processes [06:28:53] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 154 processes [06:29:53] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 37% free memory [06:32:24] PROBLEM Total processes is now: WARNING on vumi-metrics.pmtpa.wmflabs 10.4.1.13 output: PROCS WARNING: 151 processes [06:38:26] Change on 12mediawiki a page Wikimedia Labs/Toolserver features needed in Tool Labs was modified, changed by Mxn link https://www.mediawiki.org/w/index.php?diff=618960 edit summary: /* Languages */ Replied [06:52:23] RECOVERY Total processes is now: OK on vumi-metrics.pmtpa.wmflabs 10.4.1.13 output: PROCS OK: 148 processes [06:58:53] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 147 processes [07:10:33] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 6.73, 6.32, 5.23 [07:13:33] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 7.27, 7.22, 5.66 [08:18:34] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 4.56, 5.01, 5.05 [08:18:34] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 3.94, 4.28, 4.96 [08:38:33] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 4.83, 4.58, 4.99 [08:46:42] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 8.24, 7.10, 5.90 [09:00:33] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 3.31, 4.10, 4.87 [09:01:43] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 3.41, 3.79, 4.62 [12:49:33] PROBLEM Disk Space is now: WARNING on kubo.pmtpa.wmflabs 10.4.0.19 output: DISK WARNING - free space: / 574 MB (5% inode=61%): [13:09:32] RECOVERY Disk Space is now: OK on kubo.pmtpa.wmflabs 10.4.0.19 output: DISK OK [13:17:32] PROBLEM Disk Space is now: WARNING on kubo.pmtpa.wmflabs 10.4.0.19 output: DISK WARNING - free space: / 574 MB (5% inode=61%): [13:26:06] petan, you there? [13:26:14] sure [13:26:28] I need your help to import the configuration database of a channel [13:26:34] ? [13:26:38] what? [13:26:49] erm, the infobot thing :D [13:26:52] ah [13:26:55] wm-bot [13:27:01] ok, which channel and which db? [13:27:14] #wikimedia-incubator [13:27:23] like, I have the raw file from another bot [13:27:28] oh [13:27:31] kinda a clone of wm-bot :) [13:27:37] how long it is [13:27:46] you know the simplest way is to make a huge copy paste [13:27:53] send me the file to pastebin or mail [13:28:04] @add #wikimedia-incubator-bot [13:28:26] yep, but there are two versions of the infobot [13:28:34] wm-bot has the outdated version [13:28:43] and I was hoping I could do the merge myself [13:28:55] yes you can [13:29:04] no, apparently I need sudo on bots-1 [13:29:18] you don't need to use terminal, in fact you can't [13:29:24] unless you are referring to adding it manually... [13:29:36] manual changes to datafiles can only break the bot [13:30:00] hmm... [13:30:01] just join the second channel, which shares the same db, or send the new db to pastebin and I show you how to do that [13:30:14] is it the .db.xml file? [13:31:01] yes [13:31:03] it is [13:31:40] this is current db -- http://bots.wmflabs.org/~wm-bot/dump/%23wikimedia-incubator.htm [13:31:49] if you send me the new db I show you how to simply override it [13:31:53] hmm [13:31:59] the .db.xml file -> http://incubator.wmflabs.org/db/%23wikimedia-incubator.db.xml [13:32:09] new one? [13:32:11] or old [13:32:20] new [13:32:30] i.e. The latest version [13:32:43] that's the only version of that file you have? [13:32:47] or you just created it? [13:33:06] I just copied from the configuration directory of the clone [13:33:08] OH [13:33:15] you are using same software as wm-bot [13:33:19] now I get it [13:33:23] yep haha [13:33:26] I thought you are using different bot and you need to convert it [13:33:28] but some update broke it [13:33:35] yes ofc... [13:33:40] you needed convert.exe [13:33:48] I moved all config files to .xml [13:33:57] ah I see... [13:34:07] but would it be a hassle to move it? [13:34:13] over to wm-bot? [13:34:16] move what? [13:34:20] this db? no... [13:34:24] yep [13:34:39] ok can you save it to /tmp/db [13:34:40] or something [13:34:44] on bots-1 [13:35:06] erm [13:35:20] how about just directly to /mnt [13:35:26] ok, idc [13:35:30] /mnt/share [13:35:34] wherever... [13:36:14] hmm, what files do you need? [13:36:19] that xml you sent me [13:36:22] that's all [13:36:26] okie [13:37:17] yep its there [13:37:22] /mnt/share/#wikimedia-incubator.db.xml [13:37:39] ok [13:37:47] @rm Infobot core [13:37:53] !ping [13:37:53] pong [13:38:40] @system-rm Infobot core [13:38:40] Unloaded module Infobot core [13:38:44] This module is not currently loaded in core [13:38:49] o.o [13:39:01] what [13:40:05] Loaded module modules/wmib_infobot.bin [13:40:11] !ping [13:40:11] pong [13:55:21] Right, thanks for your help petan :D [14:01:35] Hello labs people! [14:02:10] Can anyone tell me what's in the package wikimedia-task-appserver on labs? [14:12:33] wish we had a documentation for each [14:40:32] PROBLEM Disk Space is now: WARNING on kubo.pmtpa.wmflabs 10.4.0.19 output: DISK WARNING - free space: / 574 MB (5% inode=61%): [14:49:23] Ryan_Lane: what is the status of the wikimedia database replication? [14:51:43] giftpflanze: ryan is definitely sleeping right now. You could ask in labs-l mailing list [14:52:02] thx [14:52:35] @labs-project-info mwreview [14:52:35] The project Mwreview has 2 instances and 6 members, description: The MediaWiki Design Review project, a beta testing area for new testing features to review for design and user experience implications. [14:52:52] just fyi you can send PM to wm-bot now [14:53:06] @labs-project-users mwreview [14:53:06] Following users are in this project (showing all 6 members): Andrew Bogott, Faidon, Eloquence, Jorm, Ottomata, Ryan Lane, [14:53:13] oh, great [14:53:20] petan, oops, true, sorry [14:53:30] it's true like few seconds :P [14:53:35] no need to be sorry for that [14:53:36] :D [15:02:32] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.60, 4.69, 4.94 [15:53:14] !log wikidata-dev set $wgUseLocalMessageCache = true on devrepo and client, re-enabled memcached on client [15:53:16] Logged the message, Master [16:26:52] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 154 processes [16:36:52] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 141 processes [16:59:08] giftpflanze: should happen next month [17:00:37] thx [17:32:53] * Damianz yawn stretch [17:33:53] PROBLEM Current Load is now: CRITICAL on wikidata-puppet-client.pmtpa.wmflabs 10.4.1.53 output: Connection refused by host [17:33:53] PROBLEM Current Load is now: CRITICAL on wikidata-puppet-repo.pmtpa.wmflabs 10.4.1.52 output: Connection refused by host [17:34:33] PROBLEM Current Users is now: CRITICAL on wikidata-puppet-client.pmtpa.wmflabs 10.4.1.53 output: Connection refused by host [17:34:33] PROBLEM Current Users is now: CRITICAL on wikidata-puppet-repo.pmtpa.wmflabs 10.4.1.52 output: Connection refused by host [17:35:13] PROBLEM Disk Space is now: CRITICAL on wikidata-puppet-client.pmtpa.wmflabs 10.4.1.53 output: Connection refused by host [17:35:13] PROBLEM Disk Space is now: CRITICAL on wikidata-puppet-repo.pmtpa.wmflabs 10.4.1.52 output: Connection refused by host [17:35:53] PROBLEM Free ram is now: CRITICAL on wikidata-puppet-client.pmtpa.wmflabs 10.4.1.53 output: Connection refused by host [17:36:03] PROBLEM Free ram is now: CRITICAL on wikidata-puppet-repo.pmtpa.wmflabs 10.4.1.52 output: Connection refused by host [17:37:23] PROBLEM Total processes is now: CRITICAL on wikidata-puppet-client.pmtpa.wmflabs 10.4.1.53 output: Connection refused by host [17:37:23] PROBLEM Total processes is now: CRITICAL on wikidata-puppet-repo.pmtpa.wmflabs 10.4.1.52 output: Connection refused by host [17:39:33] PROBLEM dpkg-check is now: CRITICAL on wikidata-puppet-client.pmtpa.wmflabs 10.4.1.53 output: Connection refused by host [17:40:53] PROBLEM dpkg-check is now: CRITICAL on wikidata-puppet-repo.pmtpa.wmflabs 10.4.1.52 output: Connection refused by host [17:42:23] RECOVERY Total processes is now: OK on wikidata-puppet-client.pmtpa.wmflabs 10.4.1.53 output: PROCS OK: 90 processes [17:43:53] RECOVERY Current Load is now: OK on wikidata-puppet-client.pmtpa.wmflabs 10.4.1.53 output: OK - load average: 0.19, 0.74, 0.57 [17:44:33] RECOVERY Current Users is now: OK on wikidata-puppet-client.pmtpa.wmflabs 10.4.1.53 output: USERS OK - 0 users currently logged in [17:44:33] RECOVERY dpkg-check is now: OK on wikidata-puppet-client.pmtpa.wmflabs 10.4.1.53 output: All packages OK [17:45:12] RECOVERY Disk Space is now: OK on wikidata-puppet-client.pmtpa.wmflabs 10.4.1.53 output: DISK OK [17:45:52] RECOVERY Free ram is now: OK on wikidata-puppet-client.pmtpa.wmflabs 10.4.1.53 output: OK: 1672% free memory [17:47:22] RECOVERY Total processes is now: OK on wikidata-puppet-repo.pmtpa.wmflabs 10.4.1.52 output: PROCS OK: 91 processes [17:48:52] RECOVERY Current Load is now: OK on wikidata-puppet-repo.pmtpa.wmflabs 10.4.1.52 output: OK - load average: 0.36, 1.08, 1.00 [17:49:32] RECOVERY Current Users is now: OK on wikidata-puppet-repo.pmtpa.wmflabs 10.4.1.52 output: USERS OK - 0 users currently logged in [17:50:12] RECOVERY Disk Space is now: OK on wikidata-puppet-repo.pmtpa.wmflabs 10.4.1.52 output: DISK OK [17:50:52] RECOVERY dpkg-check is now: OK on wikidata-puppet-repo.pmtpa.wmflabs 10.4.1.52 output: All packages OK [17:51:02] RECOVERY Free ram is now: OK on wikidata-puppet-repo.pmtpa.wmflabs 10.4.1.52 output: OK: 1667% free memory [17:54:52] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 158 processes [18:45:33] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.06, 5.24, 5.09 [18:55:33] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.49, 4.71, 4.90 [20:10:31] 12/21/2012 - 20:10:30 - Updating keys for mwang at /export/keys/mwang [20:18:14] !beta rebooting some instances so they get the new /home [20:18:14] !log deployment-prep rebooting some instances so they get the new /home [20:18:16] Logged the message, Master [20:42:57] andrewbogott_afk: Beep [22:07:12] Change on 12mediawiki a page Developer access was modified, changed by Qgil link https://www.mediawiki.org/w/index.php?diff=619218 edit summary: Gazillion pages link here. Offering a better transition to signing up. [22:13:30] Change on 12mediawiki a page Developer access was modified, changed by Nemo bis link https://www.mediawiki.org/w/index.php?diff=619219 edit summary: added [[Category:MediaWiki Development]] using [[w:Wikipedia:HotCat|HotCat]] [22:27:40] Change on 12mediawiki a page Developer access was modified, changed by Qgil link https://www.mediawiki.org/w/index.php?diff=619220 edit summary: Logos to pimp up the page. No Gerrit logo found in Commons... [22:33:32] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 4.73, 5.33, 5.24 [22:34:31] Ryan_Lane: Can you think of a good way to document the insanity that is our monitoring? [22:34:40] :D [22:34:46] no. I actually don't know how it works ;) [22:35:03] oh you suck, go read my last email [22:35:25] though I kinda lied... need to setup apache/mysql checks again, because we had the hole 'which fucking class are we using now' issue [22:35:31] so was kinda waiting for 1 role that works [22:36:00] :D [22:36:22] aahhh, right [22:36:39] totally might switch to using file includes for each class actually, that would be neater [22:36:39] it pulls the puppet classes and variables from… smw? [22:36:43] ldap [22:36:44] or ldap? [22:36:45] fuck smw [22:36:47] right [22:36:49] hahaha [22:37:10] it pulls the puppetvariable stuff and parses it into a key/value dict [22:37:16] * Ryan_Lane nods [22:37:21] and uses a mix of that and ldap attrubites to figure everything out [22:38:26] What would be /really/ nice is if a check is assigned to the puppet class it configures nrpe and you don't have to install the scripts on everything [22:40:37] * Damianz waits for his pants to stop vibrating [22:42:33] Change on 12mediawiki a page Developer access was modified, changed by Qgil link https://www.mediawiki.org/w/index.php?diff=619222 edit summary: Bold [22:53:33] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.91, 4.80, 4.97 [23:14:33] Change on 12mediawiki a page Developer access was modified, changed by Krenair link https://www.mediawiki.org/w/index.php?diff=619229 edit summary: