[00:26:13] PROBLEM Disk Space is now: CRITICAL on deployment-apache32 i-0000031a.pmtpa.wmflabs output: DISK CRITICAL - free space: / 286 MB (2% inode=78%): [00:29:23] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [00:59:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [01:08:52] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 172 processes [01:13:53] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 94 processes [01:29:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [01:55:42] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 14% free memory [01:59:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [02:00:42] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 36% free memory [02:11:13] PROBLEM Disk Space is now: WARNING on deployment-apache33 i-0000031b.pmtpa.wmflabs output: DISK WARNING - free space: / 573 MB (5% inode=78%): [02:29:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [02:40:42] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 21% free memory [02:56:12] RECOVERY Disk Space is now: OK on deployment-apache33 i-0000031b.pmtpa.wmflabs output: DISK OK [02:59:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [03:03:43] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 15% free memory [03:04:13] PROBLEM Disk Space is now: WARNING on deployment-apache33 i-0000031b.pmtpa.wmflabs output: DISK WARNING - free space: / 563 MB (5% inode=78%): [03:29:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [03:42:12] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f.pmtpa.wmflabs output: Warning: 15% free memory [03:59:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [04:02:12] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f.pmtpa.wmflabs output: Critical: 5% free memory [04:12:12] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f.pmtpa.wmflabs output: OK: 95% free memory [04:30:52] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [04:42:05] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=593665 edit summary: /* User:Jussi24 */ done [05:00:52] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [05:07:52] PROBLEM Disk Space is now: WARNING on echo-xmpp i-00000351.pmtpa.wmflabs output: DISK WARNING - free space: / 568 MB (5% inode=91%): [05:30:53] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:00:53] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:27:13] RECOVERY Disk Space is now: OK on testing-arky i-0000033b.pmtpa.wmflabs output: DISK OK [06:27:13] RECOVERY Disk Space is now: OK on conventionextension-trial i-000003bf.pmtpa.wmflabs output: DISK OK [06:32:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:36:13] 10/15/2012 - 06:36:12 - Creating a home directory for nx at /export/keys/nx [06:41:15] 10/15/2012 - 06:41:14 - Updating keys for nx at /export/keys/nx [06:42:52] RECOVERY Disk Space is now: OK on echo-xmpp i-00000351.pmtpa.wmflabs output: DISK OK [06:45:13] PROBLEM Disk Space is now: WARNING on conventionextension-trial i-000003bf.pmtpa.wmflabs output: DISK WARNING - free space: / 78 MB (5% inode=51%): [07:02:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [07:20:59] 10/15/2012 - 07:20:59 - Created a home directory for massaf in project(s): editor-engagement [07:25:59] 10/15/2012 - 07:25:59 - User massaf may have been modified in LDAP or locally, updating key in project(s): editor-engagement [07:32:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [07:50:12] PROBLEM Disk Space is now: WARNING on testing-arky i-0000033b.pmtpa.wmflabs output: DISK WARNING - free space: / 78 MB (5% inode=51%): [08:02:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [08:05:48] !nr is https://labsconsole.wikimedia.org/wiki/Nova_Resource:$1 [08:05:48] Key was added [08:06:16] @search Reso [08:06:16] Results (Found 9): instancelist, instance-json, nova-resource, resource, project-discuss, bot, pl, projects, nr, [08:06:24] !nova-resource [08:06:24] https://labsconsole.wikimedia.org/wiki/Nova_Resource:$1 [08:06:26] aha [08:06:30] !nr del [08:06:31] Successfully removed nr [08:06:38] !resource [08:06:38] https://labsconsole.wikimedia.org/wiki/Nova_Resource:$1 [08:06:41] aha [08:06:46] !nova-resource del [08:06:47] Successfully removed nova-resource [08:07:04] !nr alias resource [08:07:05] Created new alias for this key [08:07:14] !pl [08:07:14] https://labsconsole.wikimedia.org/w/index.php?title=Special:Ask&q=[[Resource+Type%3A%3Aproject]]&p=format%3Dbroadtable%2Fheaders%3Dshow%2Flink%3Dall%2Fsearchlabel%3D%E2%80%A6-20further-20results%2Fclass%3Dsortable-20wikitable-20smwtable&po=%3FMember%0A%3FDescription%0A&limit=500&eq=no [08:14:09] petan: What are you doing? [08:15:07] removing duplicates [08:32:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [09:02:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [09:32:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [09:51:15] 10/15/2012 - 09:51:14 - Updating keys for mfarag at /export/keys/mfarag [09:57:26] @seen Ryan_Lane [09:57:26] petan: Last time I saw Ryan_Lane they were quiting the network N/A at 10/15/2012 6:57:52 AM (02:59:33.6859580 ago) [09:57:38] there is problem with /home [09:57:49] 18578176 17351168 283392 99% /home [09:58:17] can someone have a look? mutante [09:58:33] I forgot who is other labs op [09:58:52] PROBLEM Current Load is now: CRITICAL on deployment-video04 i-000004c6.pmtpa.wmflabs output: Connection refused by host [09:59:32] PROBLEM Current Users is now: CRITICAL on deployment-video04 i-000004c6.pmtpa.wmflabs output: Connection refused by host [10:00:12] PROBLEM Disk Space is now: CRITICAL on deployment-video04 i-000004c6.pmtpa.wmflabs output: Connection refused by host [10:00:44] [bz] (8REOPENED - created by: 2Jeff Green, priority: 4High - 6critical) [Bug 36127] labs-nfs1 nfs export for home is full - https://bugzilla.wikimedia.org/show_bug.cgi?id=36127 [10:01:02] PROBLEM Free ram is now: CRITICAL on deployment-video04 i-000004c6.pmtpa.wmflabs output: Connection refused by host [10:02:22] PROBLEM Total processes is now: CRITICAL on deployment-video04 i-000004c6.pmtpa.wmflabs output: Connection refused by host [10:02:52] PROBLEM dpkg-check is now: CRITICAL on deployment-video04 i-000004c6.pmtpa.wmflabs output: Connection refused by host [10:03:52] RECOVERY Current Load is now: OK on deployment-video04 i-000004c6.pmtpa.wmflabs output: OK - load average: 0.95, 0.86, 0.45 [10:04:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [10:04:32] RECOVERY Current Users is now: OK on deployment-video04 i-000004c6.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [10:05:12] RECOVERY Disk Space is now: OK on deployment-video04 i-000004c6.pmtpa.wmflabs output: DISK OK [10:06:02] RECOVERY Free ram is now: OK on deployment-video04 i-000004c6.pmtpa.wmflabs output: OK: 3209% free memory [10:07:22] RECOVERY Total processes is now: OK on deployment-video04 i-000004c6.pmtpa.wmflabs output: PROCS OK: 104 processes [10:07:52] RECOVERY dpkg-check is now: OK on deployment-video04 i-000004c6.pmtpa.wmflabs output: All packages OK [10:10:59] 10/15/2012 - 10:10:59 - Deleting home directory for solr in project(s): wikidata-dev [10:14:02] !log deployment-prep j: create new videoscaler instance deployment-video04 [10:14:05] Logged the message, Master [10:15:57] 10/15/2012 - 10:15:57 - Deleting home directory for solr in project(s): wikidata-dev [10:21:00] 10/15/2012 - 10:20:59 - Deleting home directory for solr in project(s): wikidata-dev [10:26:00] 10/15/2012 - 10:25:59 - Deleting home directory for solr in project(s): wikidata-dev [10:30:58] 10/15/2012 - 10:30:58 - Deleting home directory for solr in project(s): wikidata-dev [10:34:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [10:35:59] 10/15/2012 - 10:35:59 - Deleting home directory for solr in project(s): wikidata-dev [10:40:58] 10/15/2012 - 10:40:58 - Deleting home directory for solr in project(s): wikidata-dev [10:46:00] 10/15/2012 - 10:46:00 - Deleting home directory for solr in project(s): wikidata-dev [10:50:57] 10/15/2012 - 10:50:56 - Deleting home directory for solr in project(s): wikidata-dev [10:55:58] 10/15/2012 - 10:55:58 - Deleting home directory for solr in project(s): wikidata-dev [11:01:00] 10/15/2012 - 11:01:00 - Deleting home directory for solr in project(s): wikidata-dev [11:04:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [11:05:58] 10/15/2012 - 11:05:58 - Deleting home directory for solr in project(s): wikidata-dev [11:11:01] 10/15/2012 - 11:11:01 - Deleting home directory for solr in project(s): wikidata-dev [11:15:59] 10/15/2012 - 11:15:59 - Deleting home directory for solr in project(s): wikidata-dev [11:16:31] idempotent much [11:20:45] PROBLEM dpkg-check is now: CRITICAL on deployment-video04 i-000004c6.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [11:21:01] 10/15/2012 - 11:21:01 - Deleting home directory for solr in project(s): wikidata-dev [11:25:58] 10/15/2012 - 11:25:58 - Deleting home directory for solr in project(s): wikidata-dev [11:30:59] 10/15/2012 - 11:30:59 - Deleting home directory for solr in project(s): wikidata-dev [11:34:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [11:35:58] 10/15/2012 - 11:35:57 - Deleting home directory for solr in project(s): wikidata-dev [11:40:59] 10/15/2012 - 11:40:59 - Deleting home directory for solr in project(s): wikidata-dev [11:45:58] 10/15/2012 - 11:45:58 - Deleting home directory for solr in project(s): wikidata-dev [11:50:59] 10/15/2012 - 11:50:59 - Deleting home directory for solr in project(s): wikidata-dev [11:55:59] 10/15/2012 - 11:55:59 - Deleting home directory for solr in project(s): wikidata-dev [12:00:58] 10/15/2012 - 12:00:58 - Deleting home directory for solr in project(s): wikidata-dev [12:04:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [12:06:00] 10/15/2012 - 12:06:00 - Deleting home directory for solr in project(s): wikidata-dev [12:10:59] 10/15/2012 - 12:10:59 - Deleting home directory for solr in project(s): wikidata-dev [12:15:59] 10/15/2012 - 12:15:58 - Deleting home directory for solr in project(s): wikidata-dev [12:21:00] 10/15/2012 - 12:21:00 - Deleting home directory for solr in project(s): wikidata-dev [12:25:55] 10/15/2012 - 12:25:55 - Deleting home directory for solr in project(s): wikidata-dev [12:30:59] 10/15/2012 - 12:30:58 - Deleting home directory for solr in project(s): wikidata-dev [12:34:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [12:35:58] 10/15/2012 - 12:35:58 - Deleting home directory for solr in project(s): wikidata-dev [12:40:57] 10/15/2012 - 12:40:57 - Deleting home directory for solr in project(s): wikidata-dev [12:46:00] 10/15/2012 - 12:46:00 - Deleting home directory for solr in project(s): wikidata-dev [12:50:59] 10/15/2012 - 12:50:58 - Deleting home directory for solr in project(s): wikidata-dev [12:55:57] Change on 12mediawiki a page Developer access was modified, changed by Matthias M. link https://www.mediawiki.org/w/index.php?diff=593768 edit summary: [12:55:58] 10/15/2012 - 12:55:58 - Deleting home directory for solr in project(s): wikidata-dev [13:01:00] 10/15/2012 - 13:00:59 - Deleting home directory for solr in project(s): wikidata-dev [13:04:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [13:04:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes [13:05:56] 10/15/2012 - 13:05:56 - Deleting home directory for solr in project(s): wikidata-dev [13:09:32] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 97 processes [13:10:59] 10/15/2012 - 13:10:59 - Deleting home directory for solr in project(s): wikidata-dev [13:16:00] 10/15/2012 - 13:16:00 - Deleting home directory for solr in project(s): wikidata-dev [13:21:01] 10/15/2012 - 13:21:00 - Deleting home directory for solr in project(s): wikidata-dev [13:22:15] !log integration manually installing PHPUnit on -jenkins : pear config-set auto_discover 1 && pear install pear.phpunit.de/PHPUnit [13:22:17] Logged the message, Master [13:24:18] !log integration redid PHPUnit install with: `pear install --alldeps pear.phpunit.de/PHPUnit` to bring in PHP_Invoker like in production [13:24:19] Logged the message, Master [13:26:01] 10/15/2012 - 13:26:01 - Deleting home directory for solr in project(s): wikidata-dev [13:30:58] 10/15/2012 - 13:30:58 - Deleting home directory for solr in project(s): wikidata-dev [13:34:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [13:35:59] 10/15/2012 - 13:35:58 - Deleting home directory for solr in project(s): wikidata-dev [13:40:59] 10/15/2012 - 13:40:59 - Deleting home directory for solr in project(s): wikidata-dev [13:45:59] 10/15/2012 - 13:45:58 - Deleting home directory for solr in project(s): wikidata-dev [13:51:00] 10/15/2012 - 13:50:59 - Deleting home directory for solr in project(s): wikidata-dev [13:56:00] 10/15/2012 - 13:56:00 - Deleting home directory for solr in project(s): wikidata-dev [13:58:43] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 10% free memory [14:00:54] 10/15/2012 - 14:00:53 - Deleting home directory for solr in project(s): wikidata-dev [14:03:42] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 28% free memory [14:04:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [14:05:58] 10/15/2012 - 14:05:58 - Deleting home directory for solr in project(s): wikidata-dev [14:10:58] 10/15/2012 - 14:10:58 - Deleting home directory for solr in project(s): wikidata-dev [14:15:58] 10/15/2012 - 14:15:58 - Deleting home directory for solr in project(s): wikidata-dev [14:21:01] 10/15/2012 - 14:21:00 - Deleting home directory for solr in project(s): wikidata-dev [14:22:33] If I delete something, it is gone .. but this apparently comes back every so many minutes [14:22:57] Beetstra because something creates it? [14:22:59] :D [14:23:04] puppet [14:23:09] likely, yes [14:23:17] * Beetstra grabs the salt [14:23:28] Ryan land where puppet is in war with Ryan's army [14:23:29] of bots [14:24:43] 10/15/2012 - 14:25:00 - Deleting home directory for Ryan in projects(s): wikimedia-labs [14:26:00] 10/15/2012 - 14:25:59 - Deleting home directory for solr in project(s): wikidata-dev [14:27:38] maybe if I fill up remaining space in /home the puppet won't be able to create folder [14:28:22] !log bots petrb: rebooting nr1 [14:28:24] Logged the message, Master [14:30:59] 10/15/2012 - 14:30:58 - Deleting home directory for solr in project(s): wikidata-dev [14:32:18] Why is that bot talking about directory deletion all day? [14:32:31] !log deployment-prep root: moving /home/johnduhart/ to /data/project/old/h/johnduhart/ [14:32:33] Logged the message, Master [14:33:22] Silke_WMDE because it's deleting it [14:33:28] and puppet is creating it over again [14:33:41] Silke_WMDE someone from wikidata project fucked it up [14:34:08] no folders are allowed in /home except for folders created by this bot [14:34:21] if you really want to make it a home folder do it somewhere else [14:34:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [14:34:24] yeah, I see, that someone is me. :) [14:34:29] :P [14:34:56] in future it would work but we still use this bot [14:35:58] 10/15/2012 - 14:35:58 - Deleting home directory for solr in project(s): wikidata-dev [14:37:39] !log deployment-prep root: moving /home/wikipedia/logs/ to /data/project/logs [14:37:41] Logged the message, Master [14:39:22] PROBLEM Disk Space is now: WARNING on labs-nfs1 i-0000005d.pmtpa.wmflabs output: DISK WARNING - free space: /export 670 MB (3% inode=58%): /home/SAVE 670 MB (3% inode=58%): [14:40:02] !log deployment-prep root: 4% freed of /home weeeee [14:40:05] Logged the message, Master [14:54:05] [bz] (8RESOLVED - created by: 2Chris McMahon, priority: 4Normal - 6normal) [Bug 35918] no preview thumbnails upon uploading video via UploadWizard - https://bugzilla.wikimedia.org/show_bug.cgi?id=35918 [14:59:23] RECOVERY Disk Space is now: OK on labs-nfs1 i-0000005d.pmtpa.wmflabs output: DISK OK [15:04:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [15:07:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes [15:27:32] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 97 processes [15:34:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [15:44:13] !log wikidata-dev wikidata-dev-3 corrected crontab entry for html validation [15:44:16] Logged the message, Master [16:04:23] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [16:19:07] [bz] (8ASSIGNED - created by: 2Nemo_bis, priority: 4Normal - 6minor) [Bug 36268] List of largest Mediawikis on http://wikistats.wmflabs.org/ displays wrong version number - https://bugzilla.wikimedia.org/show_bug.cgi?id=36268 [16:19:20] [bz] (8ASSIGNED - created by: 2Daniel Zahn, priority: 4Normal - 6minor) [Bug 36290] replace/rewrite largest_html.php / re-add output in other formats for "largest" all-in-one stats table - https://bugzilla.wikimedia.org/show_bug.cgi?id=36290 [16:19:21] [bz] (8ASSIGNED - created by: 2Daniel Zahn, priority: 4Normal - 6minor) [Bug 36287] deploy a replacement for the old "wikistats admin" (WSA) script - https://bugzilla.wikimedia.org/show_bug.cgi?id=36287 [16:19:34] [bz] (8ASSIGNED - created by: 2Daniel Zahn, priority: 4Normal - 6normal) [Bug 36338] do not use old mysql_connect method, avoid client mismatch php/mariadb - https://bugzilla.wikimedia.org/show_bug.cgi?id=36338 [16:20:01] [bz] (8ASSIGNED - created by: 2Daniel Zahn, priority: 4Normal - 6enhancement) [Bug 36570] new possible wikifarms / hives detected - check for lists - https://bugzilla.wikimedia.org/show_bug.cgi?id=36570 [16:34:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [17:04:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [17:06:01] [bz] (8ASSIGNED - created by: 2Robert Hanke, priority: 4Normal - 6normal) [Bug 41023] Make a stats table for 85 W3C wikis - https://bugzilla.wikimedia.org/show_bug.cgi?id=41023 [17:16:51] Unable to parse the feed from https://bugzilla.wikimedia.org/buglist.cgi?chfieldfrom=-4h&chfieldto=Now&list_id=151044&product=Wikimedia%20Labs&query_format=advanced&title=Bug%20List&ctype=atom this url is probably not a valid rss, the feed will be disabled, until you re-enable it by typing @rss+ bugzilla [17:32:38] @rss [17:32:44] !help [17:32:44] !documentation for labs !wm-bot for bot [17:32:57] !wm-bot [17:32:57] http://meta.wikimedia.org/wiki/WM-Bot [17:33:48] @rss+ bugzilla [17:33:48] This item was enabled now [17:34:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [18:05:53] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [18:37:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [19:05:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes [19:07:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [19:15:59] 10/15/2012 - 19:15:59 - Created a home directory for csteipp in project(s): deployment-prep [19:20:58] 10/15/2012 - 19:20:58 - User csteipp may have been modified in LDAP or locally, updating key in project(s): deployment-prep [19:37:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [20:00:34] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 283 processes [20:09:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [20:15:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes [20:39:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [21:00:32] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 293 processes [21:09:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [21:39:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [21:50:44] hey Ops, some people in project editor-engagement can't ssh to bastion. After "debug1: Server accepts key: pkalg ssh-rsa blen 279" they get cut off with "Connection closed by 208.80.153.207" [21:51:25] Do you have to be in the role sysadmin or netadmin in order to ssh to bastion ? [21:51:42] Change on 12mediawiki a page Developer access was modified, changed by Thehansmeet link https://www.mediawiki.org/w/index.php?diff=593930 edit summary: [21:52:04] Change on 12mediawiki a page Developer access was modified, changed by Thehansmeet link https://www.mediawiki.org/w/index.php?diff=593931 edit summary: [21:52:10] (i don't think so) [21:52:15] spagewmf: No, you just have to be a member of the bastion project [21:53:21] Yes, project bastion is probably it. is there any reason to *not* be in that project yet be a member of your team's projects? [21:53:23] Change on 12mediawiki a page Developer access was modified, changed by Thehansmeet link https://www.mediawiki.org/w/index.php?diff=593933 edit summary: [21:54:19] spagewmf: No. Any remotely trustworthy human being who agrees to the labs TOS should be in the bastion project, is my understanding [21:57:46] Change on 12mediawiki a page Developer access was modified, changed by Thehansmeet link https://www.mediawiki.org/w/index.php?diff=593936 edit summary: [22:01:39] Can I request adding Massaf and Swalling to project Bastion here, or do you want them to fill out https://labsconsole.wikimedia.org/wiki/Special:FormEdit/Shell_Access_Request ? [22:03:21] they need to fill out that form [22:03:33] when they get shell access bastion access will be auto granted [22:09:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [22:18:05] Damianz, https://labsconsole.wikimedia.org/wiki/Help:Access didn't mention that form, just "make an appearance on #wikimedia-labs". I updated it, y'all please check https://labsconsole.wikimedia.org/wiki/Help:Access#Accessing_public_and_private_instances [22:18:31] well we're kinda in a transision stage at the moment [22:18:38] The form is really for when we open up registration [22:18:51] But it applies a group permission that you need to be added to projects [22:33:44] Change on 12mediawiki a page Developer access was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=593951 edit summary: removing duplicates [22:39:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [23:09:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [23:27:03] Damianz or anyone, swalling says he filled out https://labsconsole.wikimedia.org/wiki/Special:FormEdit/Shell_Access_Request , yet I don't see anything in https://labsconsole.wikimedia.org/wiki/Help:Contents or its link _Completed requests_. [23:30:45] Hmm, his request exists at https://labsconsole.wikimedia.org/wiki/Shell_Request/Swalling , I guess the SMW queries on Help:Contents aren't quite right or maybe need to be purged [23:34:59] Yup, https://labsconsole.wikimedia.org/w/index.php?title=Help:Contents&action=purge and now they show up. SMW query invalidation is hard. [23:39:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)