[00:00:34] <bd808>	 andrewbogott: looks like it did, yes
[00:00:50] <andrewbogott>	 great, that’s at least one part of this puzzle that I understand then
[00:01:33] <bd808>	 knowledge is power
[00:03:16] <^d>	 reboot 1, no dice
[00:08:47] <andrewbogott>	 hm really?
[00:08:52] <andrewbogott>	 worked for bd808 :p
[00:13:41] <^d>	 I wonder if something's stuck because these are replacement instances for ones of the same name?
[00:13:56] <^d>	 ie: I deleted eleastic05 and made a new elastic05
[00:14:20] <andrewbogott>	 Shouldn’t matter...
[00:14:34] <andrewbogott>	 I’ll look at that particular instance once I have a better grasp on what’s happening
[00:42:08] <Krinkle>	 !log Jenkins is alerting for integration-slave1006, integration-slave1007 and integration-slave1008 having low /tmp space free (< 0.8GB)
[00:42:08] <labs-morebots>	 Jenkins is not a valid project.
[01:28:27] <andrewbogott>	 ^demon|away: ok, I know a little bit more.  There were two issues, one of which (on the NFS export side) is resolved.
[01:28:50] <andrewbogott>	 But the other is worse — sometimes new instances aren’t getting a DNS record entered in ldap.  That causes NFS to not see them.
[01:29:12] <andrewbogott>	 I think this is another symptom of virt1000 being overloaded, so I’m inclined to hold off until I migrate to new hardware on Monday
[01:29:29] <andrewbogott>	 in the meantime… if you recreate the instances, that should prompt a new attempt at making a DNS entry.  Maybe :/
[01:31:37] <tfinc>	 can someone point me to where i can modify the following file http://unicorn.wmflabs.org/techtask/discussion.json ?
[02:48:22] <Kaare>	 I just migrated my home-run/grown pywikibot to wmflabs, and that turned out to be amazingly easy - thanks for creating this nice environment!
[06:39:55] <shinken-wm>	 PROBLEM - Puppet staleness on tools-exec-15 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [43200.0]  
[06:47:54] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-05 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]  
[06:49:20] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-03 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]  
[06:54:38] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-07 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]  
[07:12:56] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-05 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:14:22] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-03 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:24:34] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-07 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:45:22] <GerardM->	 the catscan tool died
[07:45:50] <GerardM->	 it affect many other tools like AutoList
[12:24:39] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<12.50%)  
[12:29:35] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK  
[13:40:40] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<66.67%)  
[13:49:53] <canaar>	 hey guys, how should i start contributing on gerrit
[13:55:19] <canaar>	 Hey guys how do i start contributin on gerrit
[13:55:35] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK  
[13:56:04] <JohnLewis>	 canaar: have you read https://www.mediawiki.org/wiki/Gerrit/Tutorial ?
[14:00:47] <canaar_>	 Okay i will read it . Thank you mr. lewis
[14:06:35] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<55.56%)  
[15:01:37] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK  
[16:02:51] <Nemo_bis>	 I didn't know quarry had a precedent in https://data.stackexchange.com/stackoverflow/query/new
[16:04:29] <Nemo_bis>	 Silly me, I had even read the announcement http://blog.stackexchange.com/category/cc-wiki-dump/
[17:27:05] <^demon|away>	 andrewbogott_afk: Thanks for looking into it. I'll just recreate again :)
[18:47:38] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<11.11%)  
[18:57:43] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK  
[19:34:30] <Cyberpower678>	 Coren, ping
[20:37:55] <wikibugs>	 3Wikibugs: Add 'team' (purple) to output on IRC - https://phabricator.wikimedia.org/T1209#1022992 (10valhallasw)
[20:37:56] <wikibugs>	 3Wikibugs: Team (and other non-project) tags should be available for filtering - https://phabricator.wikimedia.org/T1316#1022989 (10valhallasw) 5Open>3Resolved a:3valhallasw {icon umbrella}
[20:37:57] <wikibugs>	 3Wikibugs: "Phabricator" project tasks should be reported by phawkes in #wikimedia-devtools IRC - https://phabricator.wikimedia.org/T1317#1022993 (10valhallasw)
[20:40:59] <wikibugs>	 3Wikibugs: Match on usage of Additional Hashtags, so that project renames don't break the bot - https://phabricator.wikimedia.org/T87825#1022996 (10valhallasw) p:5Triage>3Normal After {T1176} is implemented, this is fairly trivial to add, but in the current setup, it's hard. We currently screen scrape the li...
[20:41:06] <wikibugs>	 3Wikibugs: Match on usage of Additional Hashtags, so that project renames don't break the bot - https://phabricator.wikimedia.org/T87825#1023000 (10valhallasw)
[20:41:08] <wikibugs>	 3Wikibugs: Get icon and color from API instead of screen scraping - https://phabricator.wikimedia.org/T1176#20337 (10valhallasw)
[20:46:28] <grrrit-wm>	 (03PS1) 10Merlijn van Deen: Provide more CTCP responses [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189257 (https://phabricator.wikimedia.org/T88070) 
[20:46:42] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Provide more CTCP responses [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189257 (https://phabricator.wikimedia.org/T88070) (owner: 10Merlijn van Deen)
[20:48:59] <grrrit-wm>	 (03PS1) 10Merlijn van Deen: pep8-ify (E402) [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189276 
[20:49:25] <grrrit-wm>	 (03PS2) 10Merlijn van Deen: Provide more CTCP responses [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189257 (https://phabricator.wikimedia.org/T88070) 
[20:49:41] <grrrit-wm>	 (03PS1) 10Merlijn van Deen: Provide more CTCP responses [labs/tools/wikibugs2] (refs/changes/76/189276/1) - 10https://gerrit.wikimedia.org/r/189281 (https://phabricator.wikimedia.org/T88070) 
[20:49:43] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Provide more CTCP responses [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189257 (https://phabricator.wikimedia.org/T88070) (owner: 10Merlijn van Deen)
[20:49:45] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Provide more CTCP responses [labs/tools/wikibugs2] (refs/changes/76/189276/1) - 10https://gerrit.wikimedia.org/r/189281 (https://phabricator.wikimedia.org/T88070) (owner: 10Merlijn van Deen)
[20:50:04] <grrrit-wm>	 (03Abandoned) 10Merlijn van Deen: Provide more CTCP responses [labs/tools/wikibugs2] (refs/changes/76/189276/1) - 10https://gerrit.wikimedia.org/r/189281 (https://phabricator.wikimedia.org/T88070) (owner: 10Merlijn van Deen)
[20:50:45] <grrrit-wm>	 (03PS3) 10Merlijn van Deen: Provide more CTCP responses [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189257 (https://phabricator.wikimedia.org/T88070) 
[20:52:30] <grrrit-wm>	 (03CR) 10Merlijn van Deen: "Tested using wikibugs-dev, returns" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189257 (https://phabricator.wikimedia.org/T88070) (owner: 10Merlijn van Deen)
[20:52:54] <wikibugs>	 3Wikibugs: Provide more / more detailed CTCP responses. - https://phabricator.wikimedia.org/T88070#1023009 (10valhallasw) p:5Triage>3Low
[20:53:10] <wikibugs>	 3Wikibugs: Always show matched project - https://phabricator.wikimedia.org/T86759#1023011 (10valhallasw) p:5Triage>3Normal
[20:54:01] <wikibugs>	 3Wikibugs: Philosophical question: should bugs where *any* projects match be removed from -dev, or projects where *all* projects match? - https://phabricator.wikimedia.org/T86525#1023012 (10valhallasw) 5Open>3declined a:3valhallasw Unless we get complaints, I think we can ignore this issue for now. Adding...
[20:54:13] <wikibugs>	 3Wikibugs: Add sane project ordering - https://phabricator.wikimedia.org/T86448#1023015 (10valhallasw) p:5Triage>3Normal
[21:02:09] <wikibugs>	 3Wikibugs: wikibugs should notify on dependency changes - https://phabricator.wikimedia.org/T77006#1023017 (10valhallasw)
[21:02:38] <wikibugs>	 3Wikibugs: wikibugs should notify on dependency changes - https://phabricator.wikimedia.org/T77006#1023019 (10valhallasw) 5Open>3stalled p:5Triage>3Normal
[21:07:26] <wikibugs>	 3Wikibugs: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1023022 (10valhallasw)
[21:09:28] <wikibugs>	 3Wikibugs: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1023025 (10valhallasw) fdsafdsaf
[21:09:39] * valhallasw`cloud prods wikibugs_
[21:09:51] <wikibugs_>	 3Wikibugs: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1023026 (10valhallasw) another test
[21:10:22] <valhallasw`cloud>	 stupid irc3 :<
[21:14:02] <valhallasw`cloud>	 legoktm: I don't get what irc3 is doing to my logging :S even if I set my logger manually it somehow gets overwritten
[21:29:41] <legoktm>	 valhallasw`cloud: are you setting a logger after the irc3 import?
[21:30:02] <valhallasw`cloud>	 legoktm: yeah. I found the issue; irc3 calls logging.config.dictConfig itself >_<
[21:30:08] <legoktm>	 >.>
[21:30:12] <valhallasw`cloud>	 but I found how I can make it pass an empty config instead
[21:30:17] <legoktm>	 file a bug?
[21:30:20] <valhallasw`cloud>	 now I just need to set up a rotating file hander and we're golden
[21:39:01] <valhallasw`cloud>	 \o/ this is going to be awesome
[21:53:12] <wikibugs>	 3Wikibugs: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1023063 (10valhallasw) lalala, geld.
[21:55:38] <wikibugs>	 3Wikibugs: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1023064 (10valhallasw) another test
[21:55:48] <valhallasw`cloud>	 yeeehaw
[22:10:30] <grrrit-wm>	 (03PS1) 10Merlijn van Deen: Rework logging infrastructure [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189290 (https://phabricator.wikimedia.org/T1232) 
[22:10:41] <valhallasw`cloud>	 legoktm: ^ :-)
[22:10:41] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Rework logging infrastructure [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189290 (https://phabricator.wikimedia.org/T1232) (owner: 10Merlijn van Deen)
[22:16:59] <valhallasw`cloud>	 >:(
[22:17:03] <valhallasw`cloud>	 my flake8 passed!
[22:17:39] <grrrit-wm>	 (03PS2) 10Merlijn van Deen: Rework logging infrastructure [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189290 (https://phabricator.wikimedia.org/T1232) 
[22:17:50] <legoktm>	 WOOT
[22:17:57] <legoktm>	 oh, caps lock was accidental
[22:19:25] <valhallasw`cloud>	 hmm
[22:19:28] <valhallasw`cloud>	 let me check file permissions
[22:19:37] <valhallasw`cloud>	 because it probably also dumps the PASS command
[22:21:59] <valhallasw`cloud>	 hrm.
[22:24:02] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] pep8-ify (E402) [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189276 (owner: 10Merlijn van Deen)
[22:24:13] <grrrit-wm>	 (03Merged) 10jenkins-bot: pep8-ify (E402) [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189276 (owner: 10Merlijn van Deen)
[22:26:05] <valhallasw`cloud>	 legoktm: I think I'm just going to set an umask
[22:26:14] <valhallasw`cloud>	 not the cleanest solution, but certainly the easiest
[22:26:19] <legoktm>	 on the log file?
[22:26:28] <valhallasw`cloud>	 no, on the process
[22:26:50] <valhallasw`cloud>	 so every file will be rw-rw----
[22:26:55] <valhallasw`cloud>	 *written from python
[22:27:03] <valhallasw`cloud>	 or rw-r----- or something
[22:27:29] <valhallasw`cloud>	 the file creation part of the handler is deep down, and creating a file with the correct mask is nontrivial
[22:27:46] <legoktm>	 :o
[22:27:47] <legoktm>	 okay
[22:28:14] <valhallasw`cloud>	 mm let me try something
[22:28:58] <valhallasw`cloud>	 oh, it's not too bad
[22:34:18] <grrrit-wm>	 (03PS3) 10Merlijn van Deen: Rework logging infrastructure [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189290 (https://phabricator.wikimedia.org/T1232) 
[22:34:26] <valhallasw`cloud>	 legoktm: ^ this one writes 600 log files
[22:34:46] <legoktm>	 er wat
[22:35:12] <valhallasw`cloud>	 0o600
[22:36:42] <legoktm>	 oh
[22:36:48] <legoktm>	 not 600 invidiual log files
[22:36:59] <valhallasw`cloud>	 yeah, that would have been so much cooler
[22:37:11] <valhallasw`cloud>	 but no, just 7, one for each day, and then repeating :-p
[22:37:45] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] Rework logging infrastructure [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189290 (https://phabricator.wikimedia.org/T1232) (owner: 10Merlijn van Deen)
[22:37:56] <grrrit-wm>	 (03Merged) 10jenkins-bot: Rework logging infrastructure [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189290 (https://phabricator.wikimedia.org/T1232) (owner: 10Merlijn van Deen)
[22:38:12] <valhallasw`cloud>	 ok, let's see if it works :-)
[22:38:51] <ircnotifier>	 !log tools.wikibugs valhallasw: Deployed 6d78d47f1eae25b63f4cd322a6737db58b8d5c7a Rework logging infrastructure wb2-phab, wb2-irc
[22:38:55] <labs-morebots>	 Logged the message, Master
[22:39:51] <valhallasw`cloud>	 oh, wait
[22:42:47] <valhallasw`cloud>	 ARGH
[22:44:41] <valhallasw`cloud>	 cool, seems to work
[22:44:59] <grrrit-wm>	 (03PS1) 10Merlijn van Deen: Quickfix: pass 'bot' to redisrunner [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189292 
[22:45:00] <valhallasw`cloud>	 after quickfixing :<
[22:45:14] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] Quickfix: pass 'bot' to redisrunner [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189292 (owner: 10Merlijn van Deen)
[22:45:15] <legoktm>	 oops
[22:45:27] <grrrit-wm>	 (03Merged) 10jenkins-bot: Quickfix: pass 'bot' to redisrunner [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189292 (owner: 10Merlijn van Deen)
[22:45:35] <valhallasw`cloud>	 but it works, super cool \o/
[22:45:53] <valhallasw`cloud>	 so we now have /data/project/wikibugs/wikibugs.log and /data/project/wikibugs/redis2irc.log
[22:49:16] <legoktm>	 yay
[22:52:28] <wikibugs>	 3Wikibugs: wikibugs should notify on dependency changes - https://phabricator.wikimedia.org/T77006#1023126 (10valhallasw)
[22:52:29] <wikibugs>	 3Wikibugs: Implement logging so we can debug unclear IRC reports more easily - https://phabricator.wikimedia.org/T1232#1023124 (10valhallasw) 5Open>3Resolved a:3valhallasw
[22:59:45] <wikibugs>	 3Wikibugs: Clean up /data/project/wikibugs/ on tool labs - https://phabricator.wikimedia.org/T75814#1023138 (10valhallasw) 5Open>3Resolved a:3valhallasw Cleaned out most of the home dir. It's still a bit of a mess due to the large number of log files, but at least the large number of useless directories ha...
[23:01:17] <wikibugs>	 3Wikibugs: wikibugs should notify on dependency changes - https://phabricator.wikimedia.org/T77006#1023141 (10valhallasw)
[23:01:19] <wikibugs>	 3Wikibugs: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1023142 (10valhallasw)
[23:02:20] <wikibugs>	 3Wikibugs: wikibugs should notify on dependency changes - https://phabricator.wikimedia.org/T77006#1023148 (10valhallasw) 5stalled>3Invalid a:3valhallasw The change above /was/ reported, with one note for each bug:  ``` 00:01 <wikibugs> Wikibugs: wikibugs should notify on dependency changes - https://phabr...
[23:06:30] <PiRSquared>	 what's the command to use dpkg/apt-get without root?
[23:07:47] <valhallasw`cloud>	 PiRSquared: to do what?
[23:08:02] <PiRSquared>	 like normal apt-get install but to some other directory
[23:08:29] <PiRSquared>	 not /opt or whatever it uses
[23:08:41] <valhallasw`cloud>	 PiRSquared: I don't think that is typically supported, but you could get the source with apt-something and compile that
[23:08:54] <valhallasw`cloud>	 just getting the project source directly is probably more effective
[23:09:17] <valhallasw`cloud>	 in theory, you might be able to unpack the .deb to $HOME as well, but again, I wouldn't bet on it working
[23:11:00] <PiRSquared>	 thanks