[00:03:41] !log deployment-prep moved /mnt/upload to /mnt/export/upload6 [00:03:45] come on [00:03:47] BOT [00:03:48] please [00:03:54] ?` [00:03:54] whatever [00:05:14] PROBLEM HTTP is now: CRITICAL on deployment-web4 i-00000214 output: CRITICAL - Socket timeout after 10 seconds [00:05:14] PROBLEM HTTP is now: CRITICAL on deployment-web5 i-00000213 output: CRITICAL - Socket timeout after 10 seconds [00:05:20] hahhhh [00:07:47] PROBLEM HTTP is now: CRITICAL on deployment-web3 i-00000219 output: CRITICAL - Socket timeout after 10 seconds [00:10:17] PROBLEM HTTP is now: CRITICAL on deployment-apache22 i-0000026f output: CRITICAL - Socket timeout after 10 seconds [00:11:47] PROBLEM HTTP is now: CRITICAL on deployment-apache20 i-0000026c output: CRITICAL - Socket timeout after 10 seconds [00:11:47] PROBLEM HTTP is now: CRITICAL on deployment-web i-00000217 output: CRITICAL - Socket timeout after 10 seconds [00:11:47] PROBLEM HTTP is now: CRITICAL on deployment-apache21 i-0000026d output: CRITICAL - Socket timeout after 10 seconds [00:12:17] PROBLEM Free ram is now: WARNING on incubator-bot1 i-00000251 output: Warning: 19% free memory [00:12:47] PROBLEM HTTP is now: CRITICAL on deployment-apache23 i-00000270 output: CRITICAL - Socket timeout after 10 seconds [00:16:37] PROBLEM HTTP is now: WARNING on deployment-apache20 i-0000026c output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.005 second response time [00:16:37] PROBLEM HTTP is now: WARNING on deployment-web i-00000217 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.013 second response time [00:16:37] PROBLEM HTTP is now: WARNING on deployment-apache21 i-0000026d output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.009 second response time [00:17:37] PROBLEM HTTP is now: WARNING on deployment-apache23 i-00000270 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.007 second response time [00:17:37] PROBLEM HTTP is now: WARNING on deployment-web3 i-00000219 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.008 second response time [00:20:07] PROBLEM HTTP is now: WARNING on deployment-apache22 i-0000026f output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.015 second response time [00:20:07] PROBLEM HTTP is now: WARNING on deployment-web4 i-00000214 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.013 second response time [00:20:07] PROBLEM HTTP is now: WARNING on deployment-web5 i-00000213 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.010 second response time [00:27:28] ` [00:29:19] !log [00:41:38] RECOVERY Disk Space is now: OK on deployment-feed i-00000118 output: DISK OK [00:49:37] PROBLEM Disk Space is now: WARNING on deployment-feed i-00000118 output: DISK WARNING - free space: / 76 MB (5% inode=40%): [00:53:07] PROBLEM Puppet freshness is now: CRITICAL on dumps-8 i-0000026e output: Puppet has not run in last 20 hours [00:57:17] RECOVERY Free ram is now: OK on incubator-bot1 i-00000251 output: OK: 27% free memory [02:20:09] PROBLEM Current Load is now: WARNING on bots-cb i-0000009e output: WARNING - load average: 5.15, 11.85, 6.24 [02:25:09] RECOVERY Current Load is now: OK on bots-cb i-0000009e output: OK - load average: 0.25, 4.49, 4.57 [02:31:59] PROBLEM HTTP is now: CRITICAL on deployment-apache20 i-0000026c output: Connection refused [02:39:17] 05/12/2012 - 02:39:17 - Updating keys for laner [02:47:18] 05/12/2012 - 02:47:18 - Updating keys for laner [02:48:18] 05/12/2012 - 02:48:18 - Updating keys for laner [02:50:53] RECOVERY Puppet freshness is now: OK on dumps-8 i-0000026e output: puppet ran at Sat May 12 02:50:46 UTC 2012 [02:51:53] RECOVERY Current Load is now: OK on dumps-8 i-0000026e output: OK - load average: 0.49, 0.26, 0.09 [02:51:53] RECOVERY Free ram is now: OK on dumps-8 i-0000026e output: OK: 88% free memory [02:52:04] PROBLEM HTTP is now: WARNING on deployment-apache20 i-0000026c output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.015 second response time [02:53:33] RECOVERY dpkg-check is now: OK on dumps-8 i-0000026e output: All packages OK [02:53:33] RECOVERY Current Users is now: OK on dumps-8 i-0000026e output: USERS OK - 0 users currently logged in [02:54:23] RECOVERY Total Processes is now: OK on dumps-8 i-0000026e output: PROCS OK: 81 processes [02:54:28] RECOVERY Disk Space is now: OK on dumps-8 i-0000026e output: DISK OK [02:58:19] 05/12/2012 - 02:58:19 - Updating keys for laner [02:59:40] Reedy: stop [02:59:42] Reedy: that [02:59:44] Reedy: comp [02:59:48] Reedy: food! [03:01:09] !log deployment-prep rebooting all deployment-apache2? cause of weird NFS issue [03:02:13] PROBLEM HTTP is now: CRITICAL on deployment-apache20 i-0000026c output: CRITICAL - Socket timeout after 10 seconds [03:02:13] PROBLEM HTTP is now: CRITICAL on deployment-web i-00000217 output: CRITICAL - Socket timeout after 10 seconds [03:02:13] PROBLEM HTTP is now: CRITICAL on deployment-apache21 i-0000026d output: CRITICAL - Socket timeout after 10 seconds [03:03:19] PROBLEM HTTP is now: CRITICAL on deployment-web3 i-00000219 output: CRITICAL - Socket timeout after 10 seconds [03:05:12] PROBLEM HTTP is now: CRITICAL on deployment-apache22 i-0000026f output: Connection refused [03:07:00] PROBLEM HTTP is now: WARNING on deployment-web i-00000217 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.017 second response time [03:07:22] PROBLEM Disk Space is now: CRITICAL on deployment-apache23 i-00000270 output: Connection refused by host [03:07:22] PROBLEM Total Processes is now: CRITICAL on deployment-apache23 i-00000270 output: Connection refused by host [03:07:27] PROBLEM Free ram is now: CRITICAL on deployment-apache23 i-00000270 output: Connection refused by host [03:07:27] PROBLEM Current Load is now: CRITICAL on deployment-apache23 i-00000270 output: Connection refused by host [03:07:50] PROBLEM HTTP is now: WARNING on deployment-web3 i-00000219 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.010 second response time [03:08:00] PROBLEM Current Users is now: CRITICAL on deployment-apache23 i-00000270 output: Connection refused by host [03:10:00] PROBLEM SSH is now: CRITICAL on deployment-apache23 i-00000270 output: Connection refused [03:10:00] PROBLEM Current Load is now: CRITICAL on deployment-apache22 i-0000026f output: Connection refused by host [03:10:00] PROBLEM Current Users is now: CRITICAL on deployment-apache22 i-0000026f output: Connection refused by host [03:10:50] PROBLEM HTTP is now: CRITICAL on deployment-apache23 i-00000270 output: Connection refused [03:11:20] PROBLEM dpkg-check is now: CRITICAL on deployment-apache23 i-00000270 output: Connection refused by host [03:11:30] PROBLEM Total Processes is now: CRITICAL on deployment-apache22 i-0000026f output: Connection refused by host [03:12:07] !log deployment-prep Ok here is the deal. DO NOT USE /data/project , it uses too many layer of abstractions ;--D Reason I did put the log back in /home/wikipedia/ which is directly mounted to some NFS server. [03:12:10] PROBLEM SSH is now: CRITICAL on deployment-apache22 i-0000026f output: Connection refused [03:12:10] PROBLEM Disk Space is now: CRITICAL on deployment-apache22 i-0000026f output: Connection refused by host [03:12:10] PROBLEM Free ram is now: CRITICAL on deployment-apache22 i-0000026f output: Connection refused by host [03:12:10] PROBLEM dpkg-check is now: CRITICAL on deployment-apache22 i-0000026f output: Connection refused by host [03:13:13] !log deployment-prep Apache 20 and 21 rebooted. /mnt/upload6 being used and actually showing something correct. [03:19:54] !log deployment-prep apache 23 seems good. Need to fix 22 [03:20:10] RECOVERY Current Users is now: OK on deployment-apache22 i-0000026f output: USERS OK - 0 users currently logged in [03:20:10] RECOVERY Current Load is now: OK on deployment-apache22 i-0000026f output: OK - load average: 0.65, 0.21, 0.07 [03:21:07] !log deployment-prep Apache 22 works too :-] End of the week [03:21:20] RECOVERY dpkg-check is now: OK on deployment-apache23 i-00000270 output: All packages OK [03:21:30] RECOVERY Total Processes is now: OK on deployment-apache22 i-0000026f output: PROCS OK: 138 processes [03:22:10] RECOVERY SSH is now: OK on deployment-apache22 i-0000026f output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [03:22:10] RECOVERY Disk Space is now: OK on deployment-apache22 i-0000026f output: DISK OK [03:22:10] RECOVERY Free ram is now: OK on deployment-apache22 i-0000026f output: OK: 97% free memory [03:22:10] RECOVERY dpkg-check is now: OK on deployment-apache22 i-0000026f output: All packages OK [03:22:20] RECOVERY Current Load is now: OK on deployment-apache23 i-00000270 output: OK - load average: 0.40, 0.26, 0.11 [03:22:20] RECOVERY Free ram is now: OK on deployment-apache23 i-00000270 output: OK: 97% free memory [03:22:20] RECOVERY Disk Space is now: OK on deployment-apache23 i-00000270 output: DISK OK [03:22:20] RECOVERY Total Processes is now: OK on deployment-apache23 i-00000270 output: PROCS OK: 139 processes [03:23:00] RECOVERY Current Users is now: OK on deployment-apache23 i-00000270 output: USERS OK - 0 users currently logged in [03:25:11] RECOVERY SSH is now: OK on deployment-apache23 i-00000270 output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [03:26:19] 05/12/2012 - 03:26:19 - Updating keys for laner [03:26:59] PROBLEM HTTP is now: WARNING on deployment-apache21 i-0000026d output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.014 second response time [03:30:41] PROBLEM HTTP is now: WARNING on deployment-apache22 i-0000026f output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.013 second response time [03:41:01] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 14% free memory [03:42:21] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 16% free memory [03:44:18] 05/12/2012 - 03:44:18 - Updating keys for laner [03:45:51] PROBLEM HTTP is now: WARNING on deployment-apache23 i-00000270 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.007 second response time [03:55:01] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 17% free memory [03:55:01] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 14% free memory [03:56:01] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 5% free memory [04:02:21] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 4% free memory [04:06:01] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 95% free memory [04:07:46] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 97% free memory [04:14:56] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 4% free memory [04:15:06] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 5% free memory [04:19:56] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 97% free memory [04:25:06] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 94% free memory [04:41:58] PROBLEM HTTP is now: WARNING on deployment-apache20 i-0000026c output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.012 second response time [06:42:47] PROBLEM Current Users is now: CRITICAL on en-wiki-db-precise i-0000023c output: CHECK_NRPE: Socket timeout after 10 seconds. [06:42:47] PROBLEM Disk Space is now: CRITICAL on nova-precise1 i-00000236 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:42:47] PROBLEM Free ram is now: CRITICAL on en-wiki-db-precise i-0000023c output: CHECK_NRPE: Socket timeout after 10 seconds. [06:42:47] PROBLEM Total Processes is now: CRITICAL on nova-precise1 i-00000236 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:43:37] PROBLEM Total Processes is now: CRITICAL on en-wiki-db-precise i-0000023c output: CHECK_NRPE: Socket timeout after 10 seconds. [06:43:46] PROBLEM Current Load is now: CRITICAL on en-wiki-db-precise i-0000023c output: CHECK_NRPE: Socket timeout after 10 seconds. [06:47:09] PROBLEM Disk Space is now: CRITICAL on nova-essex-test i-000001f9 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:47:09] PROBLEM Free ram is now: CRITICAL on nova-essex-test i-000001f9 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:47:44] RECOVERY Current Users is now: OK on en-wiki-db-precise i-0000023c output: USERS OK - 0 users currently logged in [06:47:44] RECOVERY Free ram is now: OK on en-wiki-db-precise i-0000023c output: OK: 77% free memory [06:47:44] RECOVERY Disk Space is now: OK on nova-precise1 i-00000236 output: DISK OK [06:47:44] RECOVERY Total Processes is now: OK on nova-precise1 i-00000236 output: PROCS OK: 130 processes [06:48:34] RECOVERY Current Load is now: OK on en-wiki-db-precise i-0000023c output: OK - load average: 0.14, 1.72, 1.52 [06:48:34] RECOVERY Total Processes is now: OK on en-wiki-db-precise i-0000023c output: PROCS OK: 80 processes [06:48:43] PROBLEM Disk Space is now: CRITICAL on ve-nodejs i-00000245 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:48:44] PROBLEM Total Processes is now: CRITICAL on ve-nodejs i-00000245 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:48:49] PROBLEM Free ram is now: CRITICAL on ve-nodejs i-00000245 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:49:41] PROBLEM dpkg-check is now: CRITICAL on fr-wiki-db-precise i-0000023e output: CHECK_NRPE: Socket timeout after 10 seconds. [06:50:01] PROBLEM Current Users is now: CRITICAL on fr-wiki-db-precise i-0000023e output: CHECK_NRPE: Socket timeout after 10 seconds. [06:50:01] PROBLEM Free ram is now: CRITICAL on fr-wiki-db-precise i-0000023e output: CHECK_NRPE: Socket timeout after 10 seconds. [06:50:31] PROBLEM Current Users is now: CRITICAL on ve-nodejs i-00000245 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:51:41] PROBLEM dpkg-check is now: CRITICAL on ve-nodejs i-00000245 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:52:06] PROBLEM Current Load is now: CRITICAL on ve-nodejs i-00000245 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:54:25] RECOVERY dpkg-check is now: OK on fr-wiki-db-precise i-0000023e output: All packages OK [06:55:05] RECOVERY Current Users is now: OK on fr-wiki-db-precise i-0000023e output: USERS OK - 0 users currently logged in [06:55:05] RECOVERY Free ram is now: OK on fr-wiki-db-precise i-0000023e output: OK: 77% free memory [06:55:15] RECOVERY Current Users is now: OK on ve-nodejs i-00000245 output: USERS OK - 0 users currently logged in [06:56:30] PROBLEM Current Load is now: CRITICAL on pediapress-ocg1 i-00000233 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:56:30] RECOVERY dpkg-check is now: OK on ve-nodejs i-00000245 output: All packages OK [06:56:56] RECOVERY Disk Space is now: OK on nova-essex-test i-000001f9 output: DISK OK [06:56:56] RECOVERY Free ram is now: OK on nova-essex-test i-000001f9 output: OK: 80% free memory [06:56:56] RECOVERY Current Load is now: OK on ve-nodejs i-00000245 output: OK - load average: 0.71, 3.46, 2.68 [06:58:20] RECOVERY Disk Space is now: OK on ve-nodejs i-00000245 output: DISK OK [06:58:20] RECOVERY Total Processes is now: OK on ve-nodejs i-00000245 output: PROCS OK: 81 processes [06:58:25] RECOVERY Free ram is now: OK on ve-nodejs i-00000245 output: OK: 87% free memory [06:58:50] PROBLEM Total Processes is now: CRITICAL on pediapress-ocg1 i-00000233 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:58:55] PROBLEM dpkg-check is now: CRITICAL on pediapress-ocg1 i-00000233 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:01:20] RECOVERY Current Load is now: OK on pediapress-ocg1 i-00000233 output: OK - load average: 1.29, 3.96, 2.70 [07:03:40] RECOVERY Total Processes is now: OK on pediapress-ocg1 i-00000233 output: PROCS OK: 83 processes [07:03:45] RECOVERY dpkg-check is now: OK on pediapress-ocg1 i-00000233 output: All packages OK [07:14:20] PROBLEM Free ram is now: CRITICAL on test3 i-00000093 output: Critical: 4% free memory [07:19:20] RECOVERY Free ram is now: OK on test3 i-00000093 output: OK: 96% free memory [07:22:18] PROBLEM Puppet freshness is now: CRITICAL on nova-ldap1 i-000000df output: Puppet has not run in last 20 hours [09:32:03] 05/12/2012 - 09:32:03 - Creating a home directory for mschon at /export/home/wikistats/mschon [09:33:03] 05/12/2012 - 09:33:03 - Updating keys for mschon [10:17:11] PROBLEM Disk Space is now: CRITICAL on bz-dev i-000001db output: DISK CRITICAL - free space: / 36 MB (2% inode=43%): [10:22:11] PROBLEM Disk Space is now: WARNING on bz-dev i-000001db output: DISK WARNING - free space: / 51 MB (3% inode=43%): [10:42:14] can someone put add my public key in the bastion project/add me to the bastion project [12:14:57] PROBLEM Current Load is now: CRITICAL on incubator-apache i-00000211 output: Connection refused by host [12:14:57] PROBLEM Disk Space is now: CRITICAL on incubator-apache i-00000211 output: Connection refused by host [12:14:57] PROBLEM HTTP is now: CRITICAL on incubator-apache i-00000211 output: Connection refused [12:14:57] PROBLEM Current Users is now: CRITICAL on incubator-apache i-00000211 output: Connection refused by host [12:16:27] PROBLEM SSH is now: CRITICAL on incubator-apache i-00000211 output: Connection refused [12:16:27] PROBLEM Free ram is now: CRITICAL on incubator-apache i-00000211 output: Connection refused by host [12:16:27] PROBLEM Total Processes is now: CRITICAL on incubator-apache i-00000211 output: Connection refused by host [12:16:45] right, I am screwed [12:16:57] PROBLEM dpkg-check is now: CRITICAL on incubator-apache i-00000211 output: Connection refused by host [12:17:56] FARK [12:18:43] huh [12:19:32] incubator-apache has a mount that is showing errors in the console log [12:19:54] unless I can press "S" to the machine [14:12:08] petan|wk: Are you available now? [14:35:53] PROBLEM Free ram is now: WARNING on incubator-bot1 i-00000251 output: Warning: 19% free memory [14:40:03] 05/12/2012 - 14:40:03 - Creating a home directory for mah at /export/home/wikistats/mah [14:41:04] 05/12/2012 - 14:41:03 - Updating keys for mah [14:41:13] PROBLEM Puppet freshness is now: CRITICAL on deployment-syslog i-00000269 output: Puppet has not run in last 20 hours [15:03:17] 05/12/2012 - 15:03:17 - Creating a home directory for mschon at /export/home/bastion/mschon [15:04:15] 05/12/2012 - 15:04:15 - Updating keys for mschon [15:14:53] !log ee-prototype Creating http://ee-prototype.wmflabs.org/echo_test for playing with Echo. [15:14:56] ee-prototype is not a valid project. [15:15:50] !log editor-engagement Creating http://ee-prototype.wmflabs.org/echo_test for playing with Echo. [15:16:04] Logged the message, junior [15:52:16] PROBLEM Disk Space is now: CRITICAL on bz-dev i-000001db output: DISK CRITICAL - free space: / 18 MB (1% inode=43%): [15:55:24] !log incubator Incubator-apache is really down due to NFS issues with incubator-common. Ryan Lane: If you are reading this, please help manually reboot it! [15:55:26] Logged the message, Master [15:57:15] PROBLEM Disk Space is now: WARNING on bz-dev i-000001db output: DISK WARNING - free space: / 51 MB (3% inode=43%): [16:24:26] 05/12/2012 - 16:24:26 - Creating a home directory for soli at /export/home/bastion/soli [16:25:16] 05/12/2012 - 16:25:15 - Updating keys for soli [16:25:32] very soon our homes will be out of space due to the increase of users, and we will need a larger instance for labs-nfs1 [16:29:27] hmm, thats the old docs [16:29:34] maybe there is new ones though [17:23:09] PROBLEM Puppet freshness is now: CRITICAL on nova-ldap1 i-000000df output: Puppet has not run in last 20 hours [17:43:07] [19:33:48] * jeremyb pokes Amgine [19:42:06] * Damianz watches Amgine stab jeremyb with a brick [19:49:33] [19:50:04] Okay, I'm back jeremyb: so, I've been asked to write another mw extension. [19:50:09] RECOVERY HTTP is now: OK on incubator-apache i-00000211 output: HTTP OK: HTTP/1.1 200 OK - 473 bytes in 0.066 second response time [19:50:09] RECOVERY Disk Space is now: OK on incubator-apache i-00000211 output: DISK OK [19:50:09] RECOVERY Current Load is now: OK on incubator-apache i-00000211 output: OK - load average: 0.16, 0.14, 0.05 [19:50:09] RECOVERY Current Users is now: OK on incubator-apache i-00000211 output: USERS OK - 0 users currently logged in [19:50:10] About this labs concept... [19:51:39] RECOVERY SSH is now: OK on incubator-apache i-00000211 output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [19:51:39] RECOVERY Free ram is now: OK on incubator-apache i-00000211 output: OK: 93% free memory [19:51:39] RECOVERY Total Processes is now: OK on incubator-apache i-00000211 output: PROCS OK: 128 processes [19:51:59] RECOVERY dpkg-check is now: OK on incubator-apache i-00000211 output: All packages OK [19:57:30] Amgine: ? [19:58:27] !log incubator fixed incubator-apache1 [19:58:32] Logged the message, Master [19:58:34] Looking for support options, Ryan_Lane, and I've been avoiding learning about wmf labs, so time to learn. [19:58:47] !terms [19:58:47] https://labsconsole.wikimedia.org/wiki/Terminology [19:58:49] !instances [19:58:49] https://labsconsole.wikimedia.org/wiki/Help:Instances [19:58:52] !security [19:58:52] https://labsconsole.wikimedia.org/wiki/Help:Security_Groups [19:59:01] good starting points [19:59:23] Thanks! [20:02:22] Ryan_Lane, I know I can't use proprietary software on labs but can I open a MSSQL server at work and use extensions/MSSQLBackComat from labs to query that sql server? [20:02:38] yes [20:03:12] Great, thanks! [20:04:11] yw [20:13:12] i think a brick ain't too effective for stabbing? [20:27:11] Have you ever tried sharpening a brick? [20:39:55] sort of...