[01:02:29] <bd808>	 !log tools.morebots morebots missing from #wikimedia-qa
[01:02:35] <labs-morebots>	 Logged the message, Master
[01:04:43] <bd808>	 jeremyb_: morebots is AWOL from #wikimedia-qa.
[02:01:54] <wikibugs>	 3Tool-Labs: Replicate the Phabricator database to labsdb - https://phabricator.wikimedia.org/T52422#941131 (10Krenair) >>! In T52422#550927, @Aklapper wrote: > Plus I don't see how to sanitize that data (legal).  The same way we do it for MediaWiki databases?
[03:14:24] <andrewbogott>	 Coren: still up?  Ready for me to build a new debian image?
[03:19:40] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:19:44] <shinken-wm>	 PROBLEM - Puppet failure on tools-uwsgi-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:19:46] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:19:58] <shinken-wm>	 PROBLEM - Puppet failure on tools-dev is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:20:00] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-08 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:20:04] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:20:10] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:20:20] <shinken-wm>	 PROBLEM - Puppet failure on tools-shadow is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:20:20] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-tomcat is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:20:21] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-07 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:20:42] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:20:58] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-11 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:21:02] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-15 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:21:06] <shinken-wm>	 PROBLEM - Puppet failure on tools-submit is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:21:16] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-wmt is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:21:18] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:21:54] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-09 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:22:09] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:22:11] <shinken-wm>	 PROBLEM - Puppet failure on tools-redis is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:22:25] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:22:29] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-10 is CRITICAL: CRITICAL: 88.89% of data above the critical threshold [0.0]  
[03:22:31] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:22:35] <shinken-wm>	 PROBLEM - Puppet failure on tools-trusty is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:22:41] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0]  
[03:23:07] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:23:07] <shinken-wm>	 PROBLEM - Puppet failure on tools-static is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:23:25] <shinken-wm>	 PROBLEM - Puppet failure on tools-webproxy is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:23:53] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-12 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:24:09] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-06 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:24:31] <shinken-wm>	 PROBLEM - Puppet failure on tools-mail is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:24:39] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:24:41] <shinken-wm>	 PROBLEM - Puppet failure on tools-login is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:24:49] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-05 is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [0.0]  
[03:24:49] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-catscan is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:24:59] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-13 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:25:34] <shinken-wm>	 PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[03:29:03] <andrewbogott>	 Oddly, ^ is the sound of the puppetmaster being fixed.  Will clear up shortly.
[03:31:08] <shinken-wm>	 RECOVERY - Puppet failure on tools-submit is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:32:11] <shinken-wm>	 RECOVERY - Puppet failure on tools-redis is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:32:29] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-10 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:32:41] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:34:11] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:34:49] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-05 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:35:57] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-11 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:36:15] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-03 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:36:58] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-09 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:38:05] <shinken-wm>	 RECOVERY - Puppet failure on tools-static is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:39:30] <shinken-wm>	 RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:40:22] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-tomcat is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:40:34] <shinken-wm>	 RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:40:46] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-05 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:41:06] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-15 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:41:16] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-wmt is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:42:26] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-02 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:42:38] <shinken-wm>	 RECOVERY - Puppet failure on tools-trusty is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:43:08] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:43:50] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-12 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:44:40] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:44:40] <shinken-wm>	 RECOVERY - Puppet failure on tools-login is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:44:46] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-02 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:44:46] <shinken-wm>	 RECOVERY - Puppet failure on tools-uwsgi-01 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:44:56] <shinken-wm>	 RECOVERY - Puppet failure on tools-dev is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:44:59] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-13 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:44:59] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-08 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:45:09] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-01 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:45:15] <shinken-wm>	 RECOVERY - Puppet failure on tools-shadow is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:45:19] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-07 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:47:10] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:47:30] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-04 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:48:26] <shinken-wm>	 RECOVERY - Puppet failure on tools-webproxy is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:49:38] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-04 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:49:48] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-catscan is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:50:04] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-03 is OK: OK: Less than 1.00% above the threshold [0.0]  
[05:51:32] <shinken-wm>	 PROBLEM - Host tools-uwsgi-01 is DOWN: CRITICAL - Host Unreachable (10.68.16.64)  
[05:56:38] <YuviPanda>	 wat
[05:57:13] <YuviPanda>	 andrewbogott: ^
[05:58:11] <YuviPanda>	 andrewbogott: this was also on one of the newer hosts, I think
[05:58:41] <YuviPanda>	 it’s in SHUTOFF state
[05:58:52] * YuviPanda goes to do a ‘hard’ reboot
[06:00:04] <YuviPanda>	 !log tools tools-uwsgi-01 randomly went to SHUTOFF state, rebooting from virt1000
[06:00:10] <labs-morebots>	 Logged the message, Master
[06:01:34] <shinken-wm>	 RECOVERY - Host tools-uwsgi-01 is UP: PING OK - Packet loss = 0%, RTA = 1.18 ms  
[06:22:36] <grrrit-wm1>	 (03PS2) 10Yuvipanda: Add support for python uwsgi webservices [labs/toollabs] - 10https://gerrit.wikimedia.org/r/181418 
[06:43:14] <andrewbogott>	 YuviPanda: is there anything in the log for that instance explaining why it might have shut down?  Do you know what time it turned off?
[06:43:52] <YuviPanda>	 andrewbogott: shinken-wm’s is timestamp is pretty accurate, I think.
[06:44:17] <andrewbogott>	 So it turned off just an hour ago when no one was doing anything, huh?
[06:44:17] <YuviPanda>	 andrewbogott: nothing in the logs *in* that instance tho
[06:44:20] <YuviPanda>	 andrewbogott: yup
[06:44:36] <andrewbogott>	 Are there a bunch of uwsgi nodes with similar load?
[06:44:54] <andrewbogott>	 Is this the same one that's been SHUTOFF before or is that happening to everything on virt1010?
[06:45:12] <andrewbogott>	 (sorry, I have to ask all the obvious questions while I think about this)
[06:45:21] <YuviPanda>	 andrewbogott: nope, this one is new. created yesterday.
[06:45:32] <YuviPanda>	 andrewbogott: another machine shut off like this a few days ago (wdq-mm-02, I think)
[06:45:52] <andrewbogott>	 dang
[06:45:58] <YuviPanda>	 yeah
[06:47:02] <andrewbogott>	 2014-12-23 05:45:57.239 5932 WARNING nova.compute.manager [-] [instance: 6f8ead40-b3c9-4abe-86a1-407a410f9843] Instance shutdown by itself. Calling the stop API.
[06:47:15] <andrewbogott>	 2014-12-23 05:45:56.922 5932 INFO nova.compute.manager [-] [instance: 6f8ead40-b3c9-4abe-86a1-407a410f9843] VM Stopped (Lifecycle Event)
[06:47:20] <andrewbogott>	 So nova thinks that it turned itself off
[06:47:28] <YuviPanda>	 hmmm
[06:47:37] <andrewbogott>	 lemme see if that's happening anywhere else
[06:47:54] <andrewbogott>	 It could be that… well, it's a long shot but maybe the latest Trusty image just does that
[06:48:02] <YuviPanda>	 that’s possible
[06:48:11] <YuviPanda>	 nothing in dmesg...
[06:50:33] <andrewbogott>	 I don't see that same log message anywhere about anything save that one instance
[06:50:47] <YuviPanda>	 andrewbogott: on the other virt hosts?
[06:50:55] <andrewbogott>	 I looked on 1003 and 1006 and 1011
[06:51:04] <YuviPanda>	 https://wikitech.wikimedia.org/wiki/Nova_Resource:I-000007a1.eqiad.wmflabs was on 11
[06:51:05] <YuviPanda>	 hmm
[06:51:10] <andrewbogott>	 I don't see that message for wdq-mm-02 either
[06:51:17] <YuviPanda>	 andrewbogott: if it happens again should I let it be for you to investigate?
[06:51:22] <YuviPanda>	 (before your holidays, that is)
[06:53:32] <andrewbogott>	 I don't think I'll be any better at investigating than you.  Would be nice to create another similar image on a different (cisco) virt host and see if there's any difference
[06:54:42] <andrewbogott>	 YuviPanda: so, this is the shutdown and restart, right?  https://dpaste.de/mN4j
[06:55:19] <wikibugs>	 3Release-Engineering, Wikimedia-Labs-wikitech-interface: add [[wikitech:Release Engineering/SAL]] to [[wikitech:mediawiki:sidebar]] - https://phabricator.wikimedia.org/T73165#941281 (10jeremyb)
[06:55:42] <YuviPanda>	 andrewbogott: you mean in terms of timing?
[06:55:45] <andrewbogott>	 yeah
[06:55:55] <andrewbogott>	 And there's a 15-minute gap in the log, probably the downtime
[06:55:58] <andrewbogott>	 Does that look right to you?
[06:56:12] <YuviPanda>	 yeah, that does.
[06:56:18] <YuviPanda>	 wait, doing TZ math
[06:57:18] <andrewbogott>	 That instance says that right now it is Tue Dec 23 06:57:09 UTC 2014
[06:58:04] <YuviPanda>	 yeah, that matches up
[06:58:12] <andrewbogott>	 So… nothing useful in that log at all.
[06:58:38] <andrewbogott>	 We don't have metrics of any sort, do we?  To see if it was running out of memory or handles or something?
[06:58:52] <andrewbogott>	 (again, I am asking the obvious questions because clueless)
[06:59:04] <YuviPanda>	 andrewbogott: we do for memory, let me get ‘em
[06:59:18] <YuviPanda>	 andrewbogott: not handles or anything, just memory
[06:59:26] <andrewbogott>	 ok, well, we might get lucky
[06:59:33] <andrewbogott>	 (brb)
[06:59:51] <YuviPanda>	 https://tools.wmflabs.org/nagf/?project=tools
[06:59:55] <wikibugs>	 3Wikimedia-Labs-General, MediaWiki-extensions-OpenStackManager: adding/removing project members updates project page, changing projectadmins does not - https://phabricator.wikimedia.org/T73164#941283 (10jeremyb)
[07:00:02] <YuviPanda>	 andrewbogott: nothing about CPU or memory overusage
[07:02:49] <andrewbogott>	 dang
[07:03:16] <andrewbogott>	 So, Nova /thinks/ that it's off and is marking it as off.  If nova is mistaken about it being off, it might be shutting it down itself
[07:03:31] <andrewbogott>	 But I don't really know how it decides if an instance is up or down.  Presumably not via IP
[07:03:36] <andrewbogott>	 lemme look at the virt logs
[07:04:37] <andrewbogott>	 2014-12-23 05:45:53.476+0000: 64000: error : virProcessGetAffinity:433 : cannot get CPU affinity of process 2000: No such process
[07:04:38] <andrewbogott>	 2014-12-23 05:45:54.460+0000: 63995: error : qemuMonitorIO:656 : internal error: End of file from monitor
[07:05:38] <YuviPanda>	 that doesn’t sound so good. maybe the qemu process running it crashed?
[07:06:35] <andrewbogott>	 qemu-system-x86_64: /build/buildd/qemu-2.0.0+dfsg/kvm-all.c:984: kvm_irqchip_commit_routes: Assertion `ret == 0' failed.
[07:06:35] <andrewbogott>	 2014-12-23 05:45:54.460+0000: shutting down
[07:06:41] <andrewbogott>	 That's the qemu log
[07:06:57] <andrewbogott>	 I can't tell if that thing about kvm_irqchip_commit_routes is related or not, it doesn't have a timestamp
[07:07:30] <YuviPanda>	 brbbmf
[07:07:33] <YuviPanda>	 sigh
[07:11:04] <YuviPanda>	 andrewbogott: https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1029743 has at least the same error message, might be for a different reason
[07:11:08] <YuviPanda>	 related to… hotplug?
[07:11:58] <andrewbogott>	 what is hotplug?
[07:12:18] <YuviPanda>	 I’ve no idea.
[07:12:28] <YuviPanda>	 hotplugging in devices while the system is still running?!
[07:12:35] <YuviPanda>	 might be a red herring
[07:14:48] <YuviPanda>	 legoktm: valhallasw`cloud why is there a webservice running for wikibugs?
[07:16:29] <andrewbogott>	 YuviPanda: here's all I know so far:  https://wikitech.wikimedia.org/wiki/HPshutdowns
[07:16:36] <andrewbogott>	 I guess we'll add to that page if this happens more, and see if things repeat?
[07:17:04] <YuviPanda>	 andrewbogott: yeah, makes sense
[07:17:27] <andrewbogott>	 And now I'm going to lunch, which will ensure it happens again /right now/ :)
[07:17:56] <YuviPanda>	 andrewbogott: hah! to increase chances, let me also go take a shower :)
[07:28:30] <grrrit-wm1>	 (03PS3) 10Yuvipanda: Add support for python uwsgi webservices [labs/toollabs] - 10https://gerrit.wikimedia.org/r/181418 
[07:29:45] <andrewbogott>	 YuviPanda: confirmed that we're running the same qemu/libvirt packages on old and new virt boxes.
[07:30:06] <YuviPanda>	 andrewbogott: hmm, this used to happen on the old hosts but we presumed they were just OOM
[07:30:09] <YuviPanda>	 but maybe they weren’t?
[07:30:20] <andrewbogott>	 oh!  I didn't know this was ever happening before :(
[07:30:22] <andrewbogott>	 Anyway -- lunch!
[07:38:49] <YuviPanda>	 valhallasw`cloud: btw, there’s a uwsgi.ini now in www/python on g-p-u
[07:45:53] <DrSkyLizard>	 So, as I understand from the recent blog about the migration to Phabricator, for code review WMF will migrate from Gerrit to Phabricator's Diffusion ?
[07:46:50] <grrrit-wm1>	 (03PS4) 10Yuvipanda: Add support for python uwsgi webservices [labs/toollabs] - 10https://gerrit.wikimedia.org/r/181418 
[07:47:06] <YuviPanda>	 DrSkyLizard: at some point in the future, possibly. there will be discussions and evaluations about it before then.
[08:35:36] <wikibugs>	 3Tool-Labs: Enable OpenJDK 8 - https://phabricator.wikimedia.org/T68171#941350 (10yuvipanda) We have a trusty package thanks to T78267, but it might or might not be maintained depending on wether we end up using Titan or not. I shall -2 my patch until that is determined.
[09:03:00] <grrrit-wm1>	 (03CR) 10Yuvipanda: [C: 032] Add support for python uwsgi webservices [labs/toollabs] - 10https://gerrit.wikimedia.org/r/181418 (owner: 10Yuvipanda)
[09:08:24] <wikibugs>	 3Tool-Labs: Support uwsg web servers directly on toollabs - https://phabricator.wikimedia.org/T85202#941357 (10yuvipanda) 3NEW
[09:14:51] <wikibugs>	 3Tool-Labs: Support uwsgi web servers directly on toollabs - https://phabricator.wikimedia.org/T85202#941363 (10yuvipanda)
[09:26:27] <valhallasw`cloud>	 YuviPanda: redirect...
[09:59:02] <YuviPanda>	 valhallasw`cloud: hmm?
[10:00:01] <valhallasw`cloud>	 YuviPanda: wikibugs' lighttpd provides a redirect
[10:00:22] <YuviPanda>	 valhallasw`cloud: oh? To what
[10:00:45] <valhallasw`cloud>	 Wikitech, i think
[10:01:09] <valhallasw`cloud>	 Some outdated doc page, probably :-P
[10:08:26] <YuviPanda>	 Heh
[10:50:17] <wikibugs>	 3Tool-Labs: Make http (404, 302, 301 etc) statistics for toolserver.org - https://phabricator.wikimedia.org/T85167#941434 (10TTO) I suspect this is the reason: http://wiki.openstreetmap.org/wiki/Hike_%26_Bike_Map (the connection to `hikebike` is obvious, and `hill` is possibly short for hill-shading).  Probably...
[11:52:30] <wikibugs>	 3Labs-Team, operations, Beta-Cluster: Core dumps fill up /var on labs instances - https://phabricator.wikimedia.org/T1259#941500 (10fgiunchedi) p:5Triage>3High looks like we're close to resolution for this?
[11:54:56] <wikibugs>	 3Labs-Team, operations, Beta-Cluster: Core dumps fill up /var on labs instances - https://phabricator.wikimedia.org/T1259#941506 (10yuvipanda) More like we have a bandaid that works :)  'Solution' I would say is have some way to make the kernel *not* do core dumps at all - bsd does this, but linux I can't find a...
[12:50:15] <YuviPanda>	 Coren: bigbrother doesn’t seem to work with -l release=trusty?
[12:50:18] <YuviPanda>	 for normal services...
[12:51:06] <valhallasw`cloud>	 YuviPanda, are you trying to kill bigbrother again? :-p
[12:51:12] <valhallasw`cloud>	 I think that should just work
[12:51:23] <YuviPanda>	 valhallasw`cloud: gerrit-to-redis, I’m trying to get it to run on trusty.
[12:51:29] <valhallasw`cloud>	 YuviPanda: I see.
[12:51:31] <YuviPanda>	 valhallasw`cloud: keeps getting scheduled on exec-01
[12:51:35] <valhallasw`cloud>	 :/
[12:51:42] <YuviPanda>	 so either jsub -l doesn’t work, or something else is wrong?
[12:52:30] <valhallasw`cloud>	 YuviPanda: use jstart -l ...
[12:52:41] <valhallasw`cloud>	 specifically
[12:52:50] <valhallasw`cloud>	 jstart -N <jobname> -l release=trusty
[12:52:58] <valhallasw`cloud>	 aka 'rtfm': https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Grid#Bigbrother :-p
[12:53:24] <YuviPanda>	 valhallasw`cloud: wait
[12:53:33] <YuviPanda>	 -l release=trusty has to be after jobname?
[12:53:33] <YuviPanda>	 err
[12:53:37] <YuviPanda>	 that hsouldn’t matter
[12:53:55] <YuviPanda>	 valhallasw`cloud: right, so jstart is what’s on bigbrotherrc.
[12:53:55] <valhallasw`cloud>	 YuviPanda: except where bigbrother parses the of the command to find the job
[12:54:01] <YuviPanda>	 ewwwwwww
[12:54:27] <valhallasw`cloud>	 YuviPanda:  jstart -N lolrrit-wm -l release=trusty should be fine?
[12:54:46] <YuviPanda>	 valhallasw`cloud: err, you need to give it path of something too
[12:54:52] <YuviPanda>	 jstart -l release=trusty -N gerrit-to-redis ~/gerrit-to-redis/src/stream-receiver.bash is what I have now
[12:55:10] <valhallasw`cloud>	 YuviPanda: chech .bigbrotherrc.disabled...
[12:55:11] <valhallasw`cloud>	 jstart -N lolrrit-wm -l release=trusty -mem 1G node /data/project/lolrrit-wm/lolrrit-wm/src/relay.js
[12:55:23] <valhallasw`cloud>	 oh, other project :-p
[12:55:27] <YuviPanda>	 valhallasw`cloud: oh, I’m trying to move gerrit-to-redis
[12:55:39] <YuviPanda>	 can move lolrrit-wm after
[12:56:02] <valhallasw`cloud>	 YuviPanda: anyway, -N first, rest later
[12:57:25] <valhallasw`cloud>	 <queue_name>continuous@tools-exec-12.eqiad.wmflabs</queue_name>
[12:57:26] <YuviPanda>	 valhallasw`cloud: works now...
[12:57:28] <valhallasw`cloud>	 ta-ta-taaaa
[12:57:35] <YuviPanda>	 thanks
[12:58:52] <YuviPanda>	 valhallasw`cloud: I suppose I should make lolrrit-wm move now :)
[13:02:36] <YuviPanda>	 valhallasw`cloud: migrated :)
[13:02:41] <valhallasw`cloud>	 yay.
[13:08:14] <valhallasw`cloud>	 YuviPanda: but I think you mean labs-morebots et al, not logmsgbot
[13:08:27] <YuviPanda>	 valhallasw`cloud: hmm, I’ve managed to kill lolrrit-wm
[13:09:00] <valhallasw`cloud>	 *slow clap*
[13:09:29] <valhallasw`cloud>	 it's here, though?
[13:10:23] <valhallasw`cloud>	 YuviPanda: yeah, it's the other one that's broken
[13:10:30] <valhallasw`cloud>	 gerrit-to-redis
[13:10:37] <valhallasw`cloud>	 running, but paramiko is not connecting
[13:10:38] <YuviPanda>	 yeah
[13:10:43] <valhallasw`cloud>	     import _io
[13:10:43] <valhallasw`cloud>	 ImportError: No module named _io
[13:10:44] <valhallasw`cloud>	 wat.
[13:10:56] <YuviPanda>	 valhallasw`cloud: ah, venv corruption
[13:11:17] <valhallasw`cloud>	 ohhhhhh
[13:11:19] <valhallasw`cloud>	 I'm guessing
[13:11:25] <valhallasw`cloud>	 ubuntu changed the python search path
[13:11:33] <YuviPanda>	 recreating...
[13:21:35] <YuviPanda>	 valhallasw`cloud: is back
[13:21:39] <YuviPanda>	 valhallasw`cloud: I left comments
[13:23:56] <valhallasw`cloud>	 I love comments!
[13:28:13] <wikibugs>	 3Labs-Team, operations, Beta-Cluster: Core dumps fill up /var on labs instances - https://phabricator.wikimedia.org/T1259#941548 (10fgiunchedi) under linux it should be enough to set the process' rlimit_core to 0 to disable core dumps, unless I'm missing something?
[14:00:27] <wikibugs>	 3Labs-Team, operations: Setup per-instance configurability for labs in hiera - https://phabricator.wikimedia.org/T1356#941593 (10fgiunchedi)
[14:14:57] <YuviPanda>	 Coren: hmm, we’ve someone vainly attempting to shut down tools-login with sudo shutdown -h now...
[14:15:42] <Coren>	 Two question:  (a) what happens when you try and (b) why?  :-)
[14:16:04] <Coren>	 But also, I've had no problem accidentally rebooting it a few days ago, I don't see what could be the issue.
[14:16:08] <YuviPanda>	 Coren: nothing happens, of course, because they don’t have sudo. as to (b), I’ve no idea.
[14:16:23] <YuviPanda>	 Coren: root just gets a sudo failure email
[14:16:29] <YuviPanda>	 Coren: looks like a newish account
[14:16:47] <Coren>	 Oh, *someone*.  I misread "somehow"
[14:16:53] <YuviPanda>	 oh, yeah. not me :D
[14:17:04] <valhallasw`cloud>	 YuviPanda: probably the typical 'I'm trying to shut down my computer without noticing I'm logged in to tools-login' :-p
[14:17:14] <YuviPanda>	 valhallasw`cloud: two tries so far… :)
[14:17:30] <YuviPanda>	 but yeah, if nothing else we’ll let it be I guess
[14:17:41] <Coren>	 YuviPanda: What valhallasw said.  That would have happened to me a couple days ago if I wasn't rootlike.  :-)
[14:17:47] <YuviPanda>	 heh
[14:18:08] <YuviPanda>	 Coren: you know I’ve almost rm -rf / my laptop once...
[14:18:15] <valhallasw`cloud>	 YuviPanda: I repliieeeeeeed on your commeeeeeeents :>
[14:18:28] <YuviPanda>	 valhallasw`cloud: you diiid :) I’ll look in a minute
[14:18:29] <valhallasw`cloud>	 YuviPanda: --really-delete-root-yes-please-I-do-mean-that
[14:18:43] <YuviPanda>	 valhallasw`cloud: YES EXCEPT THAT THAT DOES NOT FUCKING EXIST ON OS X WHICH HAS BSD RM
[14:18:44] <YuviPanda>	 sigh
[14:18:47] <Coren>	 I don't think I know about that specific incident, but it's part of life as a sysadmin.  :-)
[14:19:07] <YuviPanda>	 valhallasw`cloud: thankfully I interrupted it when I realized it hadn’t errored out immediately
[14:19:34] <valhallasw`cloud>	 YuviPanda: brrr.
[14:19:41] <valhallasw`cloud>	 YuviPanda: windows ftw! ;D
[14:19:45] <YuviPanda>	 heh
[14:19:59] <valhallasw`cloud>	 I don't know /what/  / is in my windows bash, but it sure isn't c:/
[14:20:26] <Coren>	 About 10 years ago I had the "pleasure" to learn that a (then) modern Linux system /can/ be recuperated (if with some effort) so long as /sbin remains intact but with /bin and /etc all gone.
[14:20:59] <YuviPanda>	 heh
[14:21:28] <Coren>	 On a production, live system no less.  The attempt to fix was made because the actual DB server that lived there was still running -- but its executables were gone so we couldn't stop-and-restart.  Fun times.  :-)
[16:06:51] <grrrit-wm>	 (03PS1) 10Merlijn van Deen: + taxonomy script [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/181583 
[16:07:05] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] + taxonomy script [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/181583 (owner: 10Merlijn van Deen)
[16:07:21] <bd808>	 YuviPanda: If you get a chance, qa-morebots is missing from #wikimedia-qa
[16:08:23] <grrrit-wm>	 (03PS2) 10Merlijn van Deen: + taxonomy script [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/181583 
[16:08:35] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] + taxonomy script [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/181583 (owner: 10Merlijn van Deen)
[16:10:48] <grrrit-wm>	 (03PS3) 10Merlijn van Deen: + taxonomy script [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/181583 
[16:13:19] <grrrit-wm>	 (03PS1) 10Merlijn van Deen: Make sure to reset to origin/master, and show current sha1 before doing so [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/181584 
[16:13:35] <wikibugs>	 3Tool-Labs: Replicate the Phabricator database to labsdb - https://phabricator.wikimedia.org/T52422#941897 (10Aklapper) >>! In T52422#941131, @Krenair wrote: >>>! In T52422#550927, @Aklapper wrote: >> Plus I don't see how to sanitize that data (legal). >  > The same way we do it for MediaWiki databases?  Feel fr...
[16:59:21] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: Add passwords::openldap::corp [labs/private] - 10https://gerrit.wikimedia.org/r/181594 
[17:06:58] <grrrit-wm>	 (03PS4) 10Merlijn van Deen: + taxonomy script [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/181583 
[17:10:06] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Add passwords::openldap::corp [labs/private] - 10https://gerrit.wikimedia.org/r/181594 (owner: 10Alexandros Kosiaris)
[17:29:54] <wikibugs>	 3Labs-Team, operations: labs VMs install with outdated packages - https://phabricator.wikimedia.org/T85024#941980 (10fgiunchedi) yep golden master, https://wikitech.wikimedia.org/wiki/OpenStack#Building_new_images
[18:01:44] <legoktm>	 YuviPanda: wikibugs has a redirect to mw.o
[18:04:18] <valhallasw`cloud>	 legoktm: https://github.com/gawel/irc3/commit/da5a525b029a3e852cd5d11cef2afed55a18e983
[18:10:25] <legoktm>	 valhallasw`cloud: nice!
[18:10:43] <legoktm>	 valhallasw`cloud: what abour \r?
[18:10:49] <valhallasw`cloud>	 legoktm: just submitted a PR for that
[18:10:56] <legoktm>	 ok
[19:20:17] <wikibugs>	 3Labs-Team, operations: labs VMs install with outdated packages - https://phabricator.wikimedia.org/T85024#942165 (10coren) 5Open>3declined a:3coren Because of the way OpenStack manages images, making a new one obsoletes the previous ones and makes recovery in case of failure more difficult (if not impossi...
[19:31:15] <wikibugs>	 3Tool-Labs: program created by proprietary compiler allowed on labs? - https://phabricator.wikimedia.org/T74253#942187 (10coren) >>! In T74253#852391, @LuisV_WMF wrote: > is there a reason why these have to reside on labs? Data access? Something else?  I think only the OP can answer that definitively.  @Inkowik?
[19:32:21] <wikibugs>	 3Tool-Labs: Make bigbrother work with webservice2 - https://phabricator.wikimedia.org/T76162#942188 (10coren) 5Open>3Resolved Bigbrother now knows to invoke webservice2 as apropriate.
[19:36:41] <wikibugs>	 3Tool-Labs, Wikidata: Lost connection to MariaDB server during query - https://phabricator.wikimedia.org/T76699#942200 (10coren) a:5coren>3Springle I see the problem occuring intermitently, but I am unable to reproduce it myself.  @springle: do you have insight on what could be going on / has changed?
[19:38:38] <wikibugs>	 3Tool-Labs: Webservice restarts should be faster - https://phabricator.wikimedia.org/T85010#942206 (10coren) p:5Triage>3Low
[19:40:57] <wikibugs>	 3Tool-Labs: Tools labs exec host issue - https://phabricator.wikimedia.org/T78010#942210 (10coren) p:5Triage>3Normal Aborts are normally failed assertions in running code.  In order to diagnose it, we'd need (a) any error messages and (b) a backtrace from the core dump to show where the abort() took place.
[19:41:49] <wikibugs>	 3Tool-Labs: PHP abort()s during execution of a script on an exec node - https://phabricator.wikimedia.org/T78010#942213 (10coren)
[19:42:12] <wikibugs>	 3Tool-Labs: program created by proprietary compiler allowed on labs? - https://phabricator.wikimedia.org/T74253#942216 (10coren) p:5Triage>3Low
[19:42:30] <wikibugs>	 3Tool-Labs: Dumps not updating again. - https://phabricator.wikimedia.org/T74154#942218 (10coren) p:5Triage>3Normal
[19:42:42] <wikibugs>	 3Labs-Team, Tool-Labs: Setup postgres credentials for all tool users - https://phabricator.wikimedia.org/T76167#942219 (10coren) p:5Triage>3Low
[19:43:54] <wikibugs>	 3Tool-Labs: Wikimedia Labs system admin (sysadmin) documentation sucks - https://phabricator.wikimedia.org/T57946#942223 (10coren) p:5High>3Normal
[19:45:43] <wikibugs>	 3Tool-Labs: Transfer domain toolserver.org to WMF - https://phabricator.wikimedia.org/T62864#942229 (10coren) p:5Triage>3Normal
[19:47:22] <wikibugs>	 3Tool-Labs: Toolserver migration to Tools (tracking) - https://phabricator.wikimedia.org/T60788#942237 (10coren)
[19:47:24] <wikibugs>	 3Tool-Labs: Transfer domain toolserver.org to WMF - https://phabricator.wikimedia.org/T62864#942235 (10coren) 5Open>3stalled
[19:47:25] <wikibugs>	 3Tool-Labs: Set up redirect webserver for toolserver.org - https://phabricator.wikimedia.org/T62238#942236 (10coren)
[19:52:42] <wikibugs>	 3Tool-Labs: Moving toolserver domain, mail and redirects - https://phabricator.wikimedia.org/T68113#942249 (10coren) 5Open>3Resolved This is complete.  There are a few kinks leftover but those are tracked separately.
[19:52:43] <wikibugs>	 3Tool-Labs: Toolserver migration to Tools (tracking) - https://phabricator.wikimedia.org/T60788#942251 (10coren)
[20:35:54] <wikibugs>	 3Tool-Labs: Replicate the Phabricator database to labsdb - https://phabricator.wikimedia.org/T52422#942309 (10RobLa-WMF) >>! In T52422#941897, @Aklapper wrote: > Feel free to provide links to how it's done for MW DBs if it can be done "the same way".  A couple of code links: * https://github.com/wikimedia/operat...
[20:46:04] <wikibugs>	 3Tool-Labs: watchlist table not available on labs - https://phabricator.wikimedia.org/T59617#942314 (10coren) One of the issues is the exposure of tables that have few or no watchers, providing valuable vectors for vandalism attacks.  I'm pretty sure that @luisv_WMF would be entirely okay with a version that pro...
[21:46:15] <Betacommand>	 anyone around?
[21:49:10] <Betacommand>	 Im getting an SSL error trying to do an SVN update :/
[21:56:49] <bd808>	 Betacommand: Did the error say "dude! are you still using svn?" ;)
[21:57:21] <Betacommand>	 bd808: I find it a hell of a lot better than GIT
[21:58:23] <valhallasw`cloud>	 Betacommand: svn update of what? :-p
[21:58:34] <Betacommand>	 bd808: GIT stands for Going Insane and To hell
[21:58:55] <Betacommand>	 valhallasw`cloud: A private repo of my code
[21:59:21] <valhallasw`cloud>	 Betacommand: If you want help, it helps if you provide relevant information.
[21:59:55] <valhallasw`cloud>	 where is it located?
[22:00:09] <valhallasw`cloud>	 what kind of clone did you use?
[22:00:25] <valhallasw`cloud>	 or whatever that was called in the svn-sphere
[22:00:27] <Betacommand>	 valhallasw`cloud: http://pastebin.com/gA95KvDp
[22:00:41] <Betacommand>	 valhallasw`cloud: its been working for quite a while
[22:01:41] <Betacommand>	 site had an outage recently and Im trying to update my code to the most recent \
[22:02:04] <valhallasw`cloud>	 mmm.
[22:02:06] <valhallasw`cloud>	 curl seems to work
[22:02:20] <Betacommand>	 committed the changes on my end no issues
[22:02:37] <Betacommand>	 went to update on labs and got an SSl error
[22:03:40] <valhallasw`cloud>	 I'm not sure how to make svn show requests
[22:04:26] <valhallasw`cloud>	 but it works on my local ubuntu
[22:04:36] <valhallasw`cloud>	 weeiiiirrrd
[22:06:37] <Betacommand>	 thats why Im here
[22:07:16] <valhallasw`cloud>	 svn --config-option servers:global:neon-debug-mask=511 checkout https://riouxsvn.com/svn/tspywiki gives some more info, but nothing relevant
[22:08:32] <valhallasw`cloud>	 * Server certificate:
[22:08:33] <valhallasw`cloud>	 *        subject: OU=Domain Control Validated; OU=PositiveSSL Multi-Domain; CN=sni66364.cloudflaressl.com
[22:08:36] <valhallasw`cloud>	 they're doing SNI
[22:08:41] <valhallasw`cloud>	 I'm guessing SVN doesn't like that
[22:08:44] <bd808>	 I bet is has something to do with svn being v1.6.17 on tools-dev.
[22:08:56] <valhallasw`cloud>	 yeah
[22:09:06] <bd808>	 libneon is probably similarly ancient (Ubuntu LTS)
[22:09:14] <valhallasw`cloud>	 try the 14.04 login host
[22:09:37] <valhallasw`cloud>	 which is something with trusty and tools :-p
[22:09:41] <valhallasw`cloud>	 tools-trusty
[22:10:01] <valhallasw`cloud>	 yep, no issues there
[22:10:13] <valhallasw`cloud>	 that's v1.8.8
[22:10:17] <valhallasw`cloud>	 Betacommand: ^
[22:10:55] <Betacommand>	 let me see if I can connect
[22:12:34] <Betacommand>	 valhallasw`cloud: whats the remote connection host name?
[22:12:45] <valhallasw`cloud>	 Betacommand: tools-trusty.wmflabs.org, I think
[22:13:22] <Betacommand>	 Host does not exist
[22:13:26] <valhallasw`cloud>	 on a sidenote, why are you checking out a private repo on tool labs?
[22:13:31] <valhallasw`cloud>	 then ssh internally to tools-trusty
[22:14:14] <valhallasw`cloud>	 trusty.tools.wmflabs.org is the external name ,apparently
[22:16:15] <Betacommand>	 valhallasw`cloud: Its my personal code, Im the only one with access to the repo. Makes things easier
[22:17:08] <valhallasw`cloud>	 Betacommand: dunno, there's also free public svn hosting I think? :P it's supposed to be open source, after all
[22:17:41] <Betacommand>	 valhallasw`cloud: I dont want to have to deal with leaking personal info in a public repo
[22:18:07] <valhallasw`cloud>	 then don't put it in =p
[22:19:41] <Betacommand>	 valhallasw`cloud: Its a pain in the ass sometimes to deal with stuff both in repo and not, and managing private info needed in scripts
[22:19:55] <valhallasw`cloud>	 I suppose.
[22:20:55] <Betacommand>	 valhallasw`cloud: I can think of several remote APIs that I use that require authing (google books is the largest) 
[22:21:18] <valhallasw`cloud>	 Betacommand: right, so one repo with passwords and one repo with code?
[22:21:27] <valhallasw`cloud>	 that would be the somewhat standard solution, at least
[22:21:50] <Betacommand>	 valhallasw`cloud: Its a pain in the ass, Better solution, one repo
[22:22:15] <Betacommand>	 If people want my code they can talk to me
[22:22:48] <valhallasw`cloud>	 that's not really in the spirit of open source, but whatever
[22:26:07] <Betacommand>	 valhallasw`cloud: It may not be. But I dont hand guns to 5 year olds just because they have a right to bare arms
[22:26:31] <valhallasw`cloud>	 that's... an insane comparison
[22:28:26] <Betacommand>	 valhallasw`cloud: given some of the code I have sitting around, its not
[22:29:54] <Coren>	 Betacommand: Dude, anyone can fire up pywikipedia (or whichever framework of choice) and cause tons of damage without needing your receipes for it.  :-)
[22:33:17] <wm-bot>	 testing
[22:33:21] <valhallasw`cloud>	 mmmm.
[22:33:43] <wm-bot>	 testing
[22:34:11] <wm-bot>	 last test
[22:34:34] * valhallasw`cloud hands wm-bot a prize for not being affected by IRC injection