[04:40:53] <bd808>	 !log stashbot Restarted logstash on stashbot-logstash to reattach irc client to freenode
[04:40:56] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Stashbot/SAL, Master
[10:34:50] <wikibugs>	 10Tool-Labs-tools-Other, 7Epic: Convert all Labs tools to use cdnjs for static libraries and fonts - https://phabricator.wikimedia.org/T103934#1835768 (10Ricordisamoa)
[15:03:03] <wikibugs>	 10Tool-Labs-tools-Other, 7Epic: Convert all Labs tools to use cdnjs for static libraries and fonts - https://phabricator.wikimedia.org/T103934#1835980 (10Ricordisamoa)
[15:47:20] <grrrit-wm>	 (03PS3) 10Ricordisamoa: Initial commit [labs/tools/faces] - 10https://gerrit.wikimedia.org/r/192096 
[15:48:51] <grrrit-wm>	 (03CR) 10Ricordisamoa: "PS3 changes tools-static.wmflabs.org/static to tools-static.wmflabs.org/cdnjs" [labs/tools/faces] - 10https://gerrit.wikimedia.org/r/192096 (owner: 10Ricordisamoa)
[15:51:07] <grrrit-wm>	 (03PS4) 10Ricordisamoa: Initial commit [labs/tools/faces] - 10https://gerrit.wikimedia.org/r/192096 
[15:52:08] <grrrit-wm>	 (03CR) 10Ricordisamoa: "PS4 updates jQuery from 2.1.0 to 2.1.4" [labs/tools/faces] - 10https://gerrit.wikimedia.org/r/192096 (owner: 10Ricordisamoa)
[15:53:51] <grrrit-wm>	 (03PS5) 10Ricordisamoa: Initial commit [labs/tools/faces] - 10https://gerrit.wikimedia.org/r/192096 
[15:55:00] <grrrit-wm>	 (03CR) 10Ricordisamoa: "PS5 updates Bootstrap from 3.2.0 to 3.3.6" [labs/tools/faces] - 10https://gerrit.wikimedia.org/r/192096 (owner: 10Ricordisamoa)
[16:33:58] <tonythomas>	 yuvipanda: around ?
[16:35:03] <valhallasw`cloud>	 tonythomas: yuvi was up until late last night, and it's only 8:30 am on the west coast, so probably not yet :-)
[16:35:30] <valhallasw`cloud>	 tonythomas: but maybe someone else can help you?
[16:35:46] <tonythomas>	 I see that /srv is 100% full in my 20 gig instance, https://dpaste.de/uXDw/raw and the redis logs shows up https://dpaste.de/v06M/raw 
[16:35:52] <tonythomas>	 valhallasw`cloud: yeah ! can you please take a look ?
[16:36:10] <tonythomas>	 all started when tinajohnson reported getting [1cc77ed9] /wiki/Special:Newsletter/1/announce JobQueueError from line 719 of /srv/vagrant/mediawiki/includes/jobqueue/JobQueueRedis.php: Redis server error: Could not insert 1 EchoNotificationDeleteJob job(s). 
[16:37:43] <tonythomas>	 and  is that  1.5 gig out of 20 gig is allocated for /srv is normal ?
[16:44:12] <tonythomas>	 valhallasw`cloud: any pointers on what we are looking at ? 
[16:46:59] <valhallasw`cloud>	 tonythomas: the entire instance is 20GB, and I think most of it is in /
[16:47:01] <valhallasw`cloud>	 let me check
[16:47:09] <tonythomas>	 valhallasw`cloud: okey  
[16:47:38] <tonythomas>	 valhallasw`cloud: df -h https://dpaste.de/oAyB/raw
[16:48:15] <valhallasw`cloud>	 tonythomas: huh, that's odd. / is 20GB and then /srv is another 1.5G
[16:48:38] <tonythomas>	 its mapped from somewhere else ?
[16:49:26] <valhallasw`cloud>	 I'm not entirely sure how the openstack mounts are configured
[16:50:08] <valhallasw`cloud>	 is 1.5GB expected for your redis instance? i.e. should we get more space for redis, or should we prune what is in /srv?
[16:51:02] <tonythomas>	 valhallasw`cloud:  we have our wiki there, but should be fine 
[16:51:21] <tonythomas>	 there is a redis folder in /srv/
[16:53:28] <valhallasw`cloud>	 let me rephrase. Should /srv be bigger or should the contents of /srv be smaller? In the first case, you either have to store /srv on / by reconfiguring your mounts (but that only gives you a few GB extra), or you have to move to a larger instance
[16:54:11] <tonythomas>	 I wished we got more out of the 20 gig instance in /srv 
[16:54:17] <valhallasw`cloud>	 in the second case, there are methods to clean up redis a bit, but unless it can be wiped altogether it's a bit more fiddly
[16:54:39] <valhallasw`cloud>	 you can unmount /srv, and then /srv will be on the same partition as /
[16:55:09] <tonythomas>	 root@newsletter-test:/srv/redis# du -h redis-db.rdb 
[16:55:09] <tonythomas>	 9.9M    redis-db.rdb
[16:55:16] <valhallasw`cloud>	 right, so redis is not the issue
[16:55:35] <tonythomas>	 valhallasw`cloud: okey. Just unmounting /srv will do it ? let me try 
[16:55:51] <valhallasw`cloud>	 tonythomas: you'll need to move the data to /, of course, and fix up fstab to prevent it from being auto-mounted
[16:55:58] <valhallasw`cloud>	 which might bring you into a fight with puppet. not sure
[16:56:23] <tonythomas>	 valhallasw`cloud: puppet usually dont give up that easily ( form my previous battle with exim configurations ) 
[16:56:25] <tonythomas>	 :\ 
[16:56:56] <valhallasw`cloud>	 tonythomas: I don't know whether puppet configures fstab or not
[16:57:04] <tonythomas>	 but if there is no other way, and /srv is allocated less than 10% of the total space -- then :\ 
[17:06:54] <tonythomas>	 valhallasw`cloud: and one more thing. the redis-server is not getting restarted 
[17:06:57] <tonythomas>	 just stuck 
[17:08:06] <valhallasw`cloud>	 tonythomas: could be because /srv is full. Try killing it forcefully?
[17:08:39] <tonythomas>	 valhallasw`cloud: I did restart the instance though. Same issue again 
[17:10:10] <valhallasw`cloud>	 this is after unmounting /srv?
[17:10:48] <tonythomas>	 nope. before that 
[17:11:13] <valhallasw`cloud>	 ....
[17:11:27] <valhallasw`cloud>	 but then /srv is still full?
[17:12:35] <tonythomas>	 valhallasw`cloud: yeah. shameless me never unmounted /srv yet 
[17:12:54] <tonythomas>	 let me do that now 
[17:13:14] <tonythomas>	 umount: /srv: device is busy.
[17:14:16] <valhallasw`cloud>	 right, mediawiki/apache and redis are probably still using it, so those would have to be stopped
[17:14:35] <valhallasw`cloud>	 note that you have to copy /srv as well
[17:15:12] <valhallasw`cloud>	 (cp -R /srv /srv_new, then after unmounting mv /srv_new /srv)
[17:15:43] <valhallasw`cloud>	 uh, cp -R -p /srv /srv_new     to preserve ownership and permissions
[18:03:53] <wikibugs>	 10Tool-Labs-tools-Other, 7Epic: Convert all Labs tools to use cdnjs for static libraries and fonts - https://phabricator.wikimedia.org/T103934#1836084 (10Ricordisamoa)
[19:35:18] <rxy>	 !log cvn Setting up Cubbie in cvn-app5.
[19:35:21] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cvn/SAL, Master