[01:08:33] <labs-nagios-wm>	 PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 175 processes
[01:13:33] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 99 processes
[01:29:43] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on conventionextension-trial i-000003bf.pmtpa.wmflabs output: All packages OK
[01:35:22] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on mw1-21beta-lucid i-00000416.pmtpa.wmflabs output: All packages OK
[01:38:52] <labs-nagios-wm>	 PROBLEM Total processes is now: CRITICAL on dumps-incr i-0000035d.pmtpa.wmflabs output: Connection refused by host
[01:39:22] <labs-nagios-wm>	 PROBLEM SSH is now: CRITICAL on dumps-incr i-0000035d.pmtpa.wmflabs output: Connection refused
[01:40:02] <labs-nagios-wm>	 PROBLEM Current Load is now: CRITICAL on dumps-incr i-0000035d.pmtpa.wmflabs output: Connection refused by host
[01:40:12] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on dumps-incr i-0000035d.pmtpa.wmflabs output: Connection refused by host
[01:41:12] <labs-nagios-wm>	 PROBLEM Disk Space is now: CRITICAL on dumps-incr i-0000035d.pmtpa.wmflabs output: Connection refused by host
[01:41:52] <labs-nagios-wm>	 PROBLEM Current Users is now: CRITICAL on dumps-incr i-0000035d.pmtpa.wmflabs output: Connection refused by host
[01:42:02] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on dumps-incr i-0000035d.pmtpa.wmflabs output: Connection refused by host
[01:45:45] <Hydriz>	 andrewbogott: There?
[01:48:52] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on dumps-incr i-0000035d.pmtpa.wmflabs output: PROCS OK: 91 processes
[01:49:22] <labs-nagios-wm>	 RECOVERY SSH is now: OK on dumps-incr i-0000035d.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0)
[01:50:02] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on dumps-incr i-0000035d.pmtpa.wmflabs output: OK - load average: 0.07, 0.05, 0.02
[01:50:13] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on dumps-incr i-0000035d.pmtpa.wmflabs output: All packages OK
[01:50:32] <Hydriz>	 nevermind
[01:51:13] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on dumps-incr i-0000035d.pmtpa.wmflabs output: DISK OK
[01:51:53] <labs-nagios-wm>	 RECOVERY Current Users is now: OK on dumps-incr i-0000035d.pmtpa.wmflabs output: USERS OK - 1 users currently logged in
[01:52:03] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on dumps-incr i-0000035d.pmtpa.wmflabs output: OK: 1581% free memory
[01:57:53] <Ryan_Lane>	 Hydriz: ? need help?
[01:58:57] <Hydriz>	 nah, I was wondering why dumps-incr didn't come back up, but its fixed anyway
[01:59:06] <Hydriz>	 took longer than usual...
[01:59:17] <Ryan_Lane>	 ah, ok. cool
[02:11:13] <labs-nagios-wm>	 PROBLEM Current Load is now: CRITICAL on stackfarm-squid i-0000050c.pmtpa.wmflabs output: Connection refused by host
[02:11:53] <labs-nagios-wm>	 PROBLEM Current Users is now: CRITICAL on stackfarm-squid i-0000050c.pmtpa.wmflabs output: Connection refused by host
[02:12:03] <labs-nagios-wm>	 PROBLEM Total processes is now: CRITICAL on stackfarm-squid i-0000050c.pmtpa.wmflabs output: Connection refused by host
[02:12:23] <labs-nagios-wm>	 PROBLEM SSH is now: CRITICAL on stackfarm-squid i-0000050c.pmtpa.wmflabs output: Connection refused
[02:12:33] <labs-nagios-wm>	 PROBLEM Disk Space is now: CRITICAL on stackfarm-squid i-0000050c.pmtpa.wmflabs output: Connection refused by host
[02:12:33] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on stackfarm-squid i-0000050c.pmtpa.wmflabs output: Connection refused by host
[02:13:53] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on stackfarm-squid i-0000050c.pmtpa.wmflabs output: Connection refused by host
[02:33:53] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on stackfarm-squid i-0000050c.pmtpa.wmflabs output: OK: 1444% free memory
[02:36:12] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on stackfarm-squid i-0000050c.pmtpa.wmflabs output: OK - load average: 0.07, 0.13, 0.06
[02:36:52] <labs-nagios-wm>	 RECOVERY Current Users is now: OK on stackfarm-squid i-0000050c.pmtpa.wmflabs output: USERS OK - 0 users currently logged in
[02:37:02] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on stackfarm-squid i-0000050c.pmtpa.wmflabs output: PROCS OK: 84 processes
[02:37:22] <labs-nagios-wm>	 RECOVERY SSH is now: OK on stackfarm-squid i-0000050c.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0)
[02:37:32] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on stackfarm-squid i-0000050c.pmtpa.wmflabs output: All packages OK
[02:37:32] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on stackfarm-squid i-0000050c.pmtpa.wmflabs output: DISK OK
[02:39:42] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 21% free memory
[02:40:52] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on wikistream-1 i-0000016e.pmtpa.wmflabs output: OK: 24% free memory
[03:02:42] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 15% free memory
[03:08:52] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on wikistream-1 i-0000016e.pmtpa.wmflabs output: Warning: 11% free memory
[06:28:54] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on wikistream-1 i-0000016e.pmtpa.wmflabs output: OK: 38% free memory
[06:29:34] <labs-nagios-wm>	 PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 152 processes
[06:34:33] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 146 processes
[06:48:53] <labs-nagios-wm>	 PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 5.72, 5.56, 5.22
[06:55:33] <labs-nagios-wm>	 PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 156 processes
[07:05:33] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 150 processes
[07:07:43] <labs-nagios-wm>	 PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 9.80, 9.01, 6.36
[07:09:53] <labs-nagios-wm>	 PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 7.82, 8.35, 6.26
[07:18:53] <labs-nagios-wm>	 PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 4.60, 5.46, 5.01
[07:24:02] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 4.07, 4.83, 4.88
[07:47:02] <labs-nagios-wm>	 PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 6.89, 6.35, 5.46
[08:27:54] <petan>	 .
[08:34:08] <petan>	 !log nagios rebooting nagios
[08:34:11] <labs-morebots>	 Logged the message, Master
[09:15:13] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 2.53, 3.74, 4.84
[09:38:03] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.49, 4.68, 5.00
[09:46:35] <hashar>	 !log deployment-prep rebooting -bastion to get the new /home
[09:46:37] <labs-morebots>	 Logged the message, Master
[09:50:50] <hashar>	 !beta updating mediawiki-config
[09:50:50] <wm-bot>	 !log deployment-prep updating mediawiki-config
[09:50:52] <labs-morebots>	 Logged the message, Master
[09:57:57] <petan>	 mutante ping
[09:58:42] <petan>	 paravoid: ping?
[09:58:47] <petan>	 I need help with ssh
[10:05:01] <paravoid>	 petan: pong
[10:05:07] <petan>	 hi
[10:05:11] <petan>	 I can't ssh to nagios-main
[10:05:35] <petan>	 debug1: Offering public key: /home/petrb/.ssh/id_rsa
[10:05:36] <petan>	 debug2: we sent a publickey packet, wait for reply
[10:05:37] <petan>	 debug1: Server accepts key: pkalg ssh-rsa blen 277
[10:05:38] <petan>	 debug2: input_userauth_pk_ok: fp 2c:a3:d9:eb:7c:11:c8:1a:54:e5:0a:75:98:59:da:26
[10:05:39] <petan>	 debug1: read PEM private key done: type RSA
[10:05:40] <petan>	 Connection closed by 10.4.0.120
[10:06:01] <petan>	 can you check why it happens?
[10:06:05] <petan>	 I rebooted the instance twice
[10:06:22] <paravoid>	 logging in
[10:10:56] <hashar>	 !beta rebooting apache32 and apache33 to get new /home
[10:10:56] <wm-bot>	 !log deployment-prep rebooting apache32 and apache33 to get new /home
[10:10:59] <labs-morebots>	 Logged the message, Master
[10:15:01] <petan>	 hashar would be cool to create some more apaches and to keep at least one up all time
[10:15:27] <petan>	 I think the performance of beta is not as good as it used to be
[10:15:33] <hashar>	 might be
[10:15:36] <petan>	 with 8 apaches we had :P
[10:15:37] <paravoid>	 petan: I don't see you as a member of the nagios project
[10:15:38] <hashar>	 haven't looked at beta for like 2 months
[10:15:46] <petan>	 @labs-user Petrb
[10:15:46] <wm-bot>	 Petrb is member of 13 projects: Bastion, Bots, Configtest, Deployment-prep, Deployment-prepbackup, Gareth, Huggle, Hugglewa, Nagios, Openstack, Search, Turnkey-mediawiki, Upload-wizard, 
[10:15:50] <hashar>	 I don't think apaches are the issue anyway
[10:15:52] <paravoid>	 uid=2078(petrb) gid=550(svn) groups=50062(project-bastion),50064(project-bots),50076(project-openstack),50112(project-configtest),50120(project-deployment-prep),50122(project-deployment-prepbackup),50146(project-gareth),50168(project-huggle),50170(project-hugglewa),50276(project-search),50312(project-turnkey-mediawiki),50316(project-upload-wizard),550(svn)
[10:15:56] <petan>	 paravoid ^
[10:16:05] <petan>	 I do in labsconsole
[10:16:08] <paravoid>	 huh
[10:16:18] <petan>	 I am sysadmin and netadmin, in fact I was first member of that projecty
[10:16:22] <petan>	 ryan created it for me
[10:16:26] <paravoid>	 yeah I remember
[10:16:43] <paravoid>	 I didn't say you aren't actually a member, I'm saying that's what the system thinks :-)
[10:16:47] <paravoid>	 hehe
[10:16:51] <petan>	 werid
[10:16:53] <hashar>	 bah http://en.wikipedia.beta.wmflabs.org does not work anymore :-D
[10:16:59] <petan>	 :|
[10:17:16] <paravoid>	 still looking, hang on
[10:18:09] <paravoid>	 oh?
[10:18:11] <paravoid>	 no project-nagios group
[10:18:12] <paravoid>	 at all
[10:18:17] <petan>	 :/
[10:18:21] <petan>	 that suck
[10:18:45] <petan>	 I remember andrewbogott was doing some purge of empty projects
[10:19:16] <petan>	 maybe he removed wrong group
[10:20:00] <paravoid>	 the project is there
[10:20:04] <paravoid>	 i.e. cn=nagios,ou=projects,dc=wikimedia,dc=org exists
[10:20:09] <paravoid>	 the ou=groups one isn't
[10:22:54] <paravoid>	 ok, I'm going to readd it now
[10:22:56] <paravoid>	 that's peculiar
[10:23:44] <paravoid>	 oh wait I have to find the gid dammit :)
[10:27:16] <paravoid>	 hmm?
[10:27:20] <paravoid>	 I didn't do that
[10:27:50] <petan>	 ?
[10:27:56] <paravoid>	 labs-nagios-wm
[10:28:06] <petan>	 that's just bot
[10:28:13] <petan>	 rebooted instance before like 20 min ago
[10:28:20] <petan>	 that's why
[10:31:22] <hashar>	 !beta manually running 'git submodule update --init' under extensions directory to find out what is going on there
[10:31:22] <wm-bot>	 !log deployment-prep manually running 'git submodule update --init' under extensions directory to find out what is going on there
[10:31:25] <labs-morebots>	 Logged the message, Master
[10:32:48] <hashar>	 !beta removing live hack on UserMerge extension (attempted to grant some user right to bureaucrat, that should be done in CommonSettings.php )
[10:32:48] <wm-bot>	 !log deployment-prep removing live hack on UserMerge extension (attempted to grant some user right to bureaucrat, that should be done in CommonSettings.php )
[10:32:50] <labs-morebots>	 Logged the message, Master
[10:35:06] <paravoid>	 petan: sorry for the delay, still looking
[10:35:13] <petan>	 ok
[10:37:03] <hashar>	 damn
[10:37:11] <hashar>	 that Gluster FS  is pissing me off
[10:37:25] <hashar>	 it is sooo slow with git
[10:37:26] <hashar>	 !log deployment-prep /home/wikipedia/common/php-master/extensions/.git/FETCH_HEAD gave I/O error. I have deleted it and reran git pull + git submodule update --init aka : UPDATED ALL EXTENSIONS TO THEIR LATEST master VERSION.
[10:37:28] <labs-morebots>	 Logged the message, Master
[10:59:01] <paravoid>	 changeTime: 20121219002519Z
[10:59:01] <paravoid>	 targetDN: cn=project-nagios,ou=groups,dc=wikimedia,dc=org
[10:59:01] <paravoid>	 changenumber: 365
[10:59:01] <paravoid>	 changeType: delete
[10:59:01] <paravoid>	 changeInitiatorsName: uid=laner,ou=people,dc=wikimedia,dc=org
[11:03:55] <paravoid>	 (I'm trying to find backups to find project-nagios' old uid)
[11:13:35] <paravoid>	 petan: try to login
[11:36:20] <hashar>	 !beta finally had GettingStarted extension installed.
[11:36:20] <wm-bot>	 !log deployment-prep finally had GettingStarted extension installed.
[11:37:17] <labs-morebots>	 Logged the message, Master
[12:27:15] <petan>	 works, but motd is gone
[12:27:20] <petan>	 paravoid ^]
[13:04:19] <petan>	 Damianz ping
[13:24:02] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on testing-arky i-0000033b.pmtpa.wmflabs output: All packages OK
[15:01:32] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 4.97, 4.83, 4.96
[15:10:27] <petan>	 @labs-project-info bots
[15:10:27] <wm-bot>	 The project Bots has 16 instances and 40 members, description: A project for creating and running bots for use on Wikimedia Foundation sites.
[15:10:45] <petan>	 !ping
[15:10:45] <wm-bot>	 pong
[15:25:52] <andre__>	 I accidentially entered http://rl2.wmflabs.org : "ResourceLoader 2.0 prototype wiki farm, Most recent deployment: 3 October 2012"
[15:25:56] <andre__>	 Is that still in use, and if so by whom?
[15:27:14] <^demon>	 andre__: Timo or Roan would know.
[16:20:31] <labs-home-wm_>	 12/19/2012 - 16:20:30 - Creating a home directory for carlb at /export/keys/carlb
[16:25:32] <labs-home-wm_>	 12/19/2012 - 16:25:32 - Updating keys for carlb at /export/keys/carlb
[17:35:42] <petan>	 andre__ see @labs-project info 
[17:35:56] <petan>	 @labs-project-info rl2
[17:35:56] <wm-bot>	 I don't know this project, sorry, try browsing the list by hand, but I can guarantee there is no such project matching this name unless it has been created less than 9 seconds ago
[17:36:02] <petan>	 @labs-project-info rl
[17:36:03] <wm-bot>	 I don't know this project, sorry, try browsing the list by hand, but I can guarantee there is no such project matching this name unless it has been created less than 15 seconds ago
[17:36:15] <petan>	 @labs-project-info resourceloader
[17:36:16] <wm-bot>	 I don't know this project, sorry, try browsing the list by hand, but I can guarantee there is no such project matching this name unless it has been created less than 5 seconds ago
[17:36:21] <petan>	 @labs-project-info resourceloader2
[17:36:21] <wm-bot>	 The project Resourceloader2 has 1 instances and 2 members, description: {unknown}
[17:36:37] <andre__>	 Okay, still no idea how to interpret that though :)
[17:36:42] <petan>	 @labs-project-users resourceloader2
[17:36:43] <wm-bot>	 Following users are in this project (showing all 2 members): Catrope, Krinkle, 
[17:36:56] <petan>	 these 2 know what it is
[17:37:12] <andre__>	 alright
[17:52:18] <andrewbogott>	 petan:  I didn't purge anything, at least not on purpose.
[17:56:26] <petan>	 ok
[17:59:05] <andrewbogott>	 petan, did your instances survive the gluster migration?  I haven't gotten much feedback, which I take to mean we didn't break anything...
[18:01:25] <hashar>	 andrewbogott: that worked for me :-]
[18:01:31] <andrewbogott>	 great!
[18:01:31] <hashar>	 just had to reboot some random instances.
[18:01:44] <andrewbogott>	 Yeah, in theory you have to reboot /every/ instance :(
[18:01:49] <hashar>	 analytics rebooted some of their too
[18:02:01] <andrewbogott>	 But I think it'll be worth it… /home was more and more of a problem.
[18:02:10] <hashar>	 yup
[18:02:48] <hashar>	 anyway, just have to make sure to let people know to reboot their instances.  Maybe drop a mail on engineering list
[18:02:54] <hashar>	 some staffs might not be subscribed to labs-l
[18:03:13] <hashar>	 anyway, dinner time! ;-]  Thanks for the migration!
[19:03:52] <labs-nagios-wm>	 PROBLEM Current Load is now: CRITICAL on mwreview-abogott-dev i-0000053f.pmtpa.wmflabs output: Connection refused by host
[19:04:32] <labs-nagios-wm>	 PROBLEM Current Users is now: CRITICAL on mwreview-abogott-dev i-0000053f.pmtpa.wmflabs output: Connection refused by host
[19:05:12] <labs-nagios-wm>	 PROBLEM Disk Space is now: CRITICAL on mwreview-abogott-dev i-0000053f.pmtpa.wmflabs output: Connection refused by host
[19:06:02] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on mwreview-abogott-dev i-0000053f.pmtpa.wmflabs output: Connection refused by host
[19:07:22] <labs-nagios-wm>	 PROBLEM Total processes is now: CRITICAL on mwreview-abogott-dev i-0000053f.pmtpa.wmflabs output: Connection refused by host
[19:07:52] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on mwreview-abogott-dev i-0000053f.pmtpa.wmflabs output: Connection refused by host
[19:18:52] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on mwreview-abogott-dev i-0000053f.pmtpa.wmflabs output: OK - load average: 1.23, 1.44, 1.09
[19:19:32] <labs-nagios-wm>	 RECOVERY Current Users is now: OK on mwreview-abogott-dev i-0000053f.pmtpa.wmflabs output: USERS OK - 1 users currently logged in
[19:20:12] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on mwreview-abogott-dev i-0000053f.pmtpa.wmflabs output: DISK OK
[19:21:02] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on mwreview-abogott-dev i-0000053f.pmtpa.wmflabs output: OK: 877% free memory
[19:22:22] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on mwreview-abogott-dev i-0000053f.pmtpa.wmflabs output: PROCS OK: 98 processes
[19:22:52] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on mwreview-abogott-dev i-0000053f.pmtpa.wmflabs output: All packages OK
[19:37:04] * Damianz  tickles andrewbogott behind the ear

[20:03:54] <labs-nagios-wm>	 PROBLEM Current Users is now: WARNING on bots-4 i-000000e8.pmtpa.wmflabs output: USERS WARNING - 6 users currently logged in
[20:13:54] <labs-nagios-wm>	 RECOVERY Current Users is now: OK on bots-4 i-000000e8.pmtpa.wmflabs output: USERS OK - 5 users currently logged in
[20:15:53] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on mwreview-abogott-dev i-0000053f.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages
[20:33:52] <labs-nagios-wm>	 PROBLEM Current Load is now: CRITICAL on mwreview-abogott2 i-00000540.pmtpa.wmflabs output: Connection refused by host
[20:34:32] <labs-nagios-wm>	 PROBLEM Current Users is now: CRITICAL on mwreview-abogott2 i-00000540.pmtpa.wmflabs output: Connection refused by host
[20:35:12] <labs-nagios-wm>	 PROBLEM Disk Space is now: CRITICAL on mwreview-abogott2 i-00000540.pmtpa.wmflabs output: Connection refused by host
[20:36:02] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on mwreview-abogott2 i-00000540.pmtpa.wmflabs output: Connection refused by host
[20:37:22] <labs-nagios-wm>	 PROBLEM Total processes is now: CRITICAL on mwreview-abogott2 i-00000540.pmtpa.wmflabs output: Connection refused by host
[20:37:52] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on mwreview-abogott2 i-00000540.pmtpa.wmflabs output: Connection refused by host
[20:39:32] <labs-nagios-wm>	 RECOVERY Current Users is now: OK on mwreview-abogott2 i-00000540.pmtpa.wmflabs output: USERS OK - 0 users currently logged in
[20:40:13] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on mwreview-abogott2 i-00000540.pmtpa.wmflabs output: DISK OK
[20:41:03] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on mwreview-abogott2 i-00000540.pmtpa.wmflabs output: OK: 924% free memory
[20:42:23] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on mwreview-abogott2 i-00000540.pmtpa.wmflabs output: PROCS OK: 84 processes
[20:42:53] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on mwreview-abogott2 i-00000540.pmtpa.wmflabs output: All packages OK
[20:43:53] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on mwreview-abogott2 i-00000540.pmtpa.wmflabs output: OK - load average: 0.04, 0.44, 0.47
[20:52:05] <hashar>	 !log deployment-prep Granted MaxSem and Mgrover sysadmin rights. They are WMF contractors  going to work on setting up MobileFrontend on beta.
[20:52:07] <labs-morebots>	 Logged the message, Master
[21:00:53] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on mwreview-abogott2 i-00000540.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages
[21:09:57] <Ryan_Lane>	 mike_wang: around?
[21:10:54] <labs-nagios-wm>	 PROBLEM Current Load is now: CRITICAL on gerrit-build i-00000541.pmtpa.wmflabs output: Connection refused by host
[21:11:34] <labs-nagios-wm>	 PROBLEM Current Users is now: CRITICAL on gerrit-build i-00000541.pmtpa.wmflabs output: Connection refused by host
[21:12:14] <labs-nagios-wm>	 PROBLEM Disk Space is now: CRITICAL on gerrit-build i-00000541.pmtpa.wmflabs output: Connection refused by host
[21:12:24] <labs-nagios-wm>	 PROBLEM Total processes is now: CRITICAL on gerrit-build i-00000541.pmtpa.wmflabs output: Connection refused by host
[21:12:40] <mike_wang>	 Ryan_Lane: yes. What is up?
[21:12:54] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on gerrit-build i-00000541.pmtpa.wmflabs output: Connection refused by host
[21:12:54] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on gerrit-build i-00000541.pmtpa.wmflabs output: Connection refused by host
[21:13:01] <Ryan_Lane>	 mike_wang: wanted to walk you through some of the normal procedural stuff we need to take care of
[21:13:34] <mike_wang>	 nice.
[21:13:39] <Ryan_Lane>	 https://labsconsole.wikimedia.org/wiki/Help:Contents#Requests
[21:13:56] <Ryan_Lane>	 there's two queues for requests
[21:14:03] <Ryan_Lane>	 shell and project requests
[21:14:14] <Ryan_Lane>	 in general we always accept shell requests
[21:14:56] <Ryan_Lane>	 https://labsconsole.wikimedia.org/wiki/Help:Contents#Administration
[21:15:04] <mike_wang>	 That means if sb request ssh access, we always approve, right?
[21:15:05] <Ryan_Lane>	 the very last item describes how to handle them
[21:15:21] <Ryan_Lane>	 as long as they are giving a valid reason
[21:15:24] <Ryan_Lane>	 which is usually anything
[21:15:36] <Ryan_Lane>	 we have a request process to avoid sockpuppets
[21:16:06] <Ryan_Lane>	 in the sidebar, under "Labs Cloudadmins", there's a "Manage User Rights" link
[21:16:28] <Ryan_Lane>	 when you put in the user's username on that page, it'll bring up a list of permissions to give someone
[21:16:38] <Ryan_Lane>	 you just need to add shell for them
[21:16:47] <Ryan_Lane>	 then go back to the request and "edit with form"
[21:16:52] <Ryan_Lane>	 then mark the request as completed
[21:17:09] <Ryan_Lane>	 the project request queue is different
[21:17:17] <Ryan_Lane>	 we don't always approved projects
[21:17:30] <Ryan_Lane>	 for instance, we don't approve projects specific to users
[21:17:36] <Ryan_Lane>	 there's two of those in there right now
[21:20:08] <Ryan_Lane>	 also, it's usually wise to rename their project name sometimes
[21:20:18] <Ryan_Lane>	 like ext-mapsources should likely be just mapsources
[21:20:52] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on gerrit-build i-00000541.pmtpa.wmflabs output: OK - load average: 0.94, 1.02, 0.63
[21:21:32] <labs-nagios-wm>	 RECOVERY Current Users is now: OK on gerrit-build i-00000541.pmtpa.wmflabs output: USERS OK - 0 users currently logged in
[21:22:12] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on gerrit-build i-00000541.pmtpa.wmflabs output: DISK OK
[21:22:22] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on gerrit-build i-00000541.pmtpa.wmflabs output: PROCS OK: 100 processes
[21:22:52] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on gerrit-build i-00000541.pmtpa.wmflabs output: All packages OK
[21:23:02] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on gerrit-build i-00000541.pmtpa.wmflabs output: OK: 2905% free memory
[21:24:30] <Ryan_Lane>	 mike_wang: so, if you could keep the request queues empty, that would be nice
[21:25:05] <Ryan_Lane>	 I need to add some kind of field for "disapproved" for projects, so they can disappear from the queue
[21:25:31] <labs-home-wm_>	 12/19/2012 - 21:25:30 - Updating keys for mwang at /export/keys/mwang
[21:35:30] <mike_wang>	 Ryan_Lane:  I can not find  "edit with form"
[21:35:40] <Ryan_Lane>	 on the shell request page?
[21:35:47] <Ryan_Lane>	 when you click on the request
[21:36:02] <Ryan_Lane>	 for instance, on this page: https://labsconsole.wikimedia.org/wiki/Shell_Request/Mwalker
[21:36:24] <danmichaelo>	 or this: https://labsconsole.wikimedia.org/wiki/Shell_Request/Danmichaelo ;)
[21:36:40] <Ryan_Lane>	 in some cases they already had shell access and didn't realize it
[21:36:49] <Ryan_Lane>	 if that's the case, ensure they are a member of the bastion project
[21:37:07] <Ryan_Lane>	 via "Manage Projects" under "Labs Users" in the sidebar
[21:37:25] <mike_wang>	 I will add them to bastion project
[21:37:43] <Ryan_Lane>	 they are automatically added to bastion, when you give them shell
[21:37:59] <Ryan_Lane>	 but some users were given shell before we added support for that
[21:38:08] <Ryan_Lane>	 so they have shell, but are missing from bastion
[21:38:36] <Ryan_Lane>	 so, if they are requesting shell and already have it, check bastion, just in case
[21:43:30] <mike_wang>	 Failed to add Jkroll to bastion.
[21:44:41] <mike_wang>	 i give Jkroll shell access 2 mins ago
[21:48:50] <mike_wang>	 sorry. Jkroll is already in bastion project.
[21:53:17] <Ryan_Lane>	 like I told you...
[21:53:27] <Ryan_Lane>	 if you give someone shell, they are automatically added to the bastion project
[21:53:48] <Ryan_Lane>	 if someone is requesting shell, and already have it, you just need to add them into bastion if they aren't already there
[21:54:12] * Damianz  nibbles wang

[21:54:28] <Ryan_Lane>	 if you remove shell from someone, it removes them from all projects, so don't go that route for giving bastion to someone ;)
[21:55:06] <Damianz>	 You changed the config around?
[21:55:11] <Ryan_Lane>	 no?
[21:55:19] <Damianz>	 hmm
[21:55:22] <Ryan_Lane>	 you added that option in
[21:55:38] <Ryan_Lane>	 that's the initial config I went with
[21:55:49] <Damianz>	 I could swear I set bastion by default, projects optionally... maybe it was the other way around though
[21:55:55] <Ryan_Lane>	 yeah, you did
[21:56:19] <Ryan_Lane>	 so, to be technical, I didn't change the config around, because the initial config I went with wasn't default ;)
[21:56:27] <Damianz>	 ^.^
[21:57:17] <Damianz>	 OSM is one of those things that in theory should be useable outside of labs but really never will be :P
[21:57:32] <Ryan_Lane>	 very likely true
[22:02:14] <Damianz>	 Grr don't you hate it when you put paper work down then forget where you left it..
[22:03:24] <Ryan_Lane>	 what's paper?
[22:03:35] <Damianz>	 ^.^
[22:03:50] <mutante>	 locate :)
[22:04:16] <Damianz>	 Well next year I'm just having my insurance company email me the policy cover letter so I can stick it in git and be happy, 40 pages of crap with like 9 details on is around my house /somewhere/
[22:05:07] <Damianz>	 Though I did have it earlier because I was going to ring them, then couldn't be bothered... debating if I can put the parts and swap them out cheaper than the excess... though I have excess protection so technically I'd get it back, just after christmas... and I probably can't get it fixed this week which sucks
[22:05:13] * Damianz  hates christmas

[22:05:55] <mutante>	 oh..that reminds me of another sad story.. so i wanted to become a Verizon customer.. so a company you would expect to be specialized in communications... in the 21st century
[22:06:36] <Ryan_Lane>	 communications companies are terrible at tech and at communications, which is really sad
[22:06:42] <mutante>	 and it failed in the end.. because they had to print(!) a copy of my social security card .. then scan(!) it again immediately.. only to fax it to another person elsewhere 
[22:06:51] <Damianz>	 yeah
[22:07:01] <mutante>	 who said they could not read it.. and they could not just email them
[22:07:13] <mutante>	 or use any feature of my phone i offered to share a file with
[22:07:15] <Damianz>	 Technically to accept our IT policy you have to print it, sign it, scan it, break the email policy by emailing it, then shred it...
[22:08:28] <mutante>	 my German national ID could meanwhile hold my signature file to sign electronically ...
[22:08:50] <mutante>	 if i buy a reader and pay some signature provider ...
[22:09:14] <Damianz>	 Imo a PGP signed document returned should be good enough
[22:09:36] <mutante>	 totally.. and Adobe also has it all in Acrobat Pro 
[22:10:32] <labs-home-wm_>	 12/19/2012 - 22:10:32 - Updating keys for mwang at /export/keys/mwang
[22:11:09] <mike_wang>	 Ryan_Lane: I marked all the 3 request as completed
[22:11:38] <Ryan_Lane>	 thanks
[22:14:22] * Damianz  thinks it would be nice to have rc in here for such st00f

[22:18:56] <danmichaelo>	 thanks as well, should I be able to ssh bastion.wmflabs.org now?
[22:30:08] <giftpflanze>	 danmichaelo: have you uploaded you public key?
[22:31:04] <danmichaelo>	 yup, I did, but currently I get permission denied
[22:33:53] <labs-nagios-wm>	 PROBLEM Current Load is now: CRITICAL on mwreview-abogott4 i-00000543.pmtpa.wmflabs output: Connection refused by host
[22:34:33] <labs-nagios-wm>	 PROBLEM Current Users is now: CRITICAL on mwreview-abogott4 i-00000543.pmtpa.wmflabs output: Connection refused by host
[22:35:13] <labs-nagios-wm>	 PROBLEM Disk Space is now: CRITICAL on mwreview-abogott4 i-00000543.pmtpa.wmflabs output: Connection refused by host
[22:35:53] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on mwreview-abogott4 i-00000543.pmtpa.wmflabs output: Connection refused by host
[22:37:23] <labs-nagios-wm>	 PROBLEM Total processes is now: CRITICAL on mwreview-abogott4 i-00000543.pmtpa.wmflabs output: Connection refused by host
[22:39:33] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on mwreview-abogott4 i-00000543.pmtpa.wmflabs output: Connection refused by host
[22:39:51] <Ryan_Lane>	 danmichaelo: it isn't working, eh?
[22:39:53] <Ryan_Lane>	 ah
[22:39:54] <Ryan_Lane>	 I know why
[22:39:58] <Ryan_Lane>	 nscd negative cache
[22:40:12] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on mwreview-abogott4 i-00000543.pmtpa.wmflabs output: DISK OK
[22:40:19] <Ryan_Lane>	 danmichaelo: try now
[22:40:33] <Ryan_Lane>	 Invalid user danmichael
[22:40:38] <Ryan_Lane>	 is that your shell account name?
[22:40:44] <giftpflanze>	 the documentation is hopefully outdated by now …
[22:40:52] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on mwreview-abogott4 i-00000543.pmtpa.wmflabs output: OK: 876% free memory
[22:40:53] <Ryan_Lane>	 giftpflanze: which documentation is?
[22:40:59] <giftpflanze>	 help:access
[22:41:03] <Ryan_Lane>	 !access
[22:41:03] <wm-bot>	 https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances
[22:41:04] <Ryan_Lane>	 it is?
[22:41:14] <Ryan_Lane>	 how so?
[22:41:14] <giftpflanze>	 we're not beta anymore
[22:41:38] <Ryan_Lane>	 which page is this you're talking about?
[22:41:47] <giftpflanze>	 sigh
[22:42:22] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on mwreview-abogott4 i-00000543.pmtpa.wmflabs output: PROCS OK: 91 processes
[22:42:35] <danmichaelo>	 thanks, had an error in my ssh-config, now it worked!
[22:42:39] <Ryan_Lane>	 cool
[22:42:39] <Ryan_Lane>	 yw
[22:42:54] <Ryan_Lane>	 help access doesn't say we're in beta
[22:43:03] <giftpflanze>	 but it describes it
[22:43:07] <Ryan_Lane>	 ahhh
[22:43:07] <Ryan_Lane>	 Q: Who can give out Labsconsole accounts, and how do they do it?
[22:43:14] <giftpflanze>	 e. g.
[22:43:25] <Ryan_Lane>	 yeah, that page needs to be updated
[22:43:45] <danmichaelo>	 quite beta-ish :)
[22:43:52] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on mwreview-abogott4 i-00000543.pmtpa.wmflabs output: OK - load average: 1.11, 1.00, 0.71
[22:44:32] <labs-nagios-wm>	 RECOVERY Current Users is now: OK on mwreview-abogott4 i-00000543.pmtpa.wmflabs output: USERS OK - 1 users currently logged in
[22:46:25] <danmichaelo>	 are there some fun commands on bastion? to list projects or so?
[22:47:44] <Ryan_Lane>	 giftpflanze: updated: https://labsconsole.wikimedia.org/wiki/Help:Access#Access_FAQ
[22:48:05] <Ryan_Lane>	 danmichaelo: nope, though you can find that info on the wiki
[22:48:07] <Ryan_Lane>	 !projects
[22:48:07] <wm-bot>	 https://labsconsole.wikimedia.org/w/index.php?title=Special:Ask&q=[[Resource+Type%3A%3Aproject]]&p=format%3Dbroadtable%2Fheaders%3Dshow%2Flink%3Dall%2Fsearchlabel%3D%E2%80%A6-20further-20results%2Fclass%3Dsortable-20wikitable-20smwtable&po=%3FMember%0A%3FDescription%0A&limit=500&eq=no
[22:48:57] <giftpflanze>	 Ryan_Lane: Initial log in and password change needs some love as well
[22:49:07] <Ryan_Lane>	 giftpflanze: in what way?
[22:49:38] <giftpflanze>	 If someone has made an account for you, … password reset …
[22:51:47] <Ryan_Lane>	 people don't make accounts for others anymore
[22:51:50] <Ryan_Lane>	 it's self-registration
[22:52:01] <Ryan_Lane>	 you set your password at account creation time
[22:52:36] <apmon_>	 aude: Ping
[22:53:09] <aude>	 hi apmon_ 
[22:53:54] <apmon_>	 Do you know if there has been progress on the osm server for mobile? Or at what state it is?
[22:53:59] <aude>	 https://gerrit.wikimedia.org/r/#/c/36222/
[22:54:19] <aude>	 MaxSem is polishing the puppet stuff
[22:54:55] <apmon_>	 Ah, nice. so there has been some progress :-)
[22:55:30] <aude>	 yes, though not sure what's next after that
[22:56:18] <aude>	 i guess see how this works and then improve / build it out more
[22:58:05] <apmon_>	 Is it at the stage to run an import on the available hardware already?
[22:58:27] <aude>	 i think it's just about ready
[22:59:27] <apmon_>	 nice
[23:00:06] <aude>	 it would be nice to try the carto / mapnik 2.0 stuff at some point, but basic is fine for now
[23:00:44] <aude>	 see how that goes for osm....
[23:00:53] <apmon_>	 yes, the OSM server is about to move over to using the carto style. So it probably would be sensible to do that in wikipedia as well
[23:01:00] <aude>	 yep
[23:01:39] <apmon_>	 but if I am not mistaken carto actually gets compiled by a converter script into standard mapnik XML and then that is what is run
[23:01:56] <aude>	 yes, that's right
[23:02:45] <andrewbogott>	 woosters, make yourself an account here, then tell me your login and I'll set up permissions.
[23:03:15] <aude>	 it's like cascadenik, but with js syntax
[23:08:24] <apmon_>	 Well, if I can do anything to help push things forward, let me know.
[23:09:33] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on mwreview-abogott4 i-00000543.pmtpa.wmflabs output: All packages OK
[23:11:09] <andrewbogott>	 woosters:  Um… somehow I neglected to paste the link:  https://labsconsole.wikimedia.org/w/index.php?title=Special:UserLogin&returnto=Help%3AGetting+Started&type=signup
[23:12:34] <aude>	 apmon_: sure.
[23:25:12] <labs-nagios-wm>	 PROBLEM host: i-00000544.pmtpa.wmflabs is DOWN address: i-00000544.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000544.pmtpa.wmflabs)
[23:30:34] <labs-nagios-wm>	 RECOVERY host: i-00000544.pmtpa.wmflabs is UP address: i-00000544.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 9.85 ms
[23:30:54] <labs-nagios-wm>	 PROBLEM Current Load is now: CRITICAL on mwreview-abogott5 i-00000544.pmtpa.wmflabs output: Connection refused by host
[23:31:34] <labs-nagios-wm>	 PROBLEM Current Users is now: CRITICAL on mwreview-abogott5 i-00000544.pmtpa.wmflabs output: Connection refused by host
[23:32:14] <labs-nagios-wm>	 PROBLEM Disk Space is now: CRITICAL on mwreview-abogott5 i-00000544.pmtpa.wmflabs output: Connection refused by host
[23:33:04] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on mwreview-abogott5 i-00000544.pmtpa.wmflabs output: Connection refused by host
[23:33:54] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on mwreview-abogott5 i-00000544.pmtpa.wmflabs output: Connection refused by host
[23:34:24] <labs-nagios-wm>	 PROBLEM Total processes is now: CRITICAL on mwreview-abogott5 i-00000544.pmtpa.wmflabs output: Connection refused by host
[23:46:32] <labs-nagios-wm>	 RECOVERY Current Users is now: OK on mwreview-abogott5 i-00000544.pmtpa.wmflabs output: USERS OK - 1 users currently logged in
[23:48:02] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on mwreview-abogott5 i-00000544.pmtpa.wmflabs output: OK: 1533% free memory
[23:48:52] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on mwreview-abogott5 i-00000544.pmtpa.wmflabs output: All packages OK
[23:49:22] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on mwreview-abogott5 i-00000544.pmtpa.wmflabs output: PROCS OK: 83 processes
[23:50:52] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on mwreview-abogott5 i-00000544.pmtpa.wmflabs output: OK - load average: 0.10, 0.33, 0.17
[23:52:12] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on mwreview-abogott5 i-00000544.pmtpa.wmflabs output: DISK OK