[00:05:52] <bd808>	 !log tools.stashbot Restarted bot to clear busted phabricator session
[00:05:54] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL, Master
[00:06:05] <bd808>	 !log tools.stashbot Added Greg Grossmeier as co-maintainer
[00:06:08] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL, Master
[00:06:24] <bd808>	 greg-g: you has the power!
[00:07:44] <greg-g>	 yay
[00:08:11] <greg-g>	 huh, I don't get an Echo notification about that
[00:08:26] <bd808>	 it's all LDAP I think
[00:08:37] <bd808>	 also echo and wikitech aren't the best of friends
[00:08:53] <greg-g>	 yeah yeah
[00:10:03] <wikibugs>	 6Labs, 6Operations, 10wikitech.wikimedia.org, 13Patch-For-Review: Update wikitech-static OS/PHP version - https://phabricator.wikimedia.org/T126385#2101565 (10Krenair) a:3Krenair * Copied my usual files across (basically what I have in prod puppet modules/admin/files/home/krenair), installed git (and bas...
[00:11:42] <greg-g>	 testing the log tailing T123
[00:11:52] <greg-g>	 not in here eh
[00:12:14] <bd808>	 it should work everywhere
[00:12:30] <bd808>	 the errors take a while to show up 'cause NFS
[00:12:35] * greg-g nods
[00:12:36] <greg-g>	 gotcha
[00:12:41] <greg-g>	 'tis working
[00:13:14] <bd808>	 hmm.. T123
[00:13:29] <bd808>	 weird
[00:13:55] <bd808>	 the bot is answering in ##stashbot
[00:14:00] <bd808>	 but not here
[00:28:29] <greg-g>	 and in devtools
[00:29:24] <bd808>	 greg-g: hmm.. that seems suspicious
[00:29:59] <bd808>	 !log tools.stashbot Checking to see if bot is really listening in #wikimedia-labs
[00:30:02] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL, Master
[00:30:34] <bd808>	 that worked -- https://tools.wmflabs.org/sal/log/AVNYx8A4RVcO3hhfZoUt
[00:33:47] <wikibugs>	 10PAWS, 6Research-and-Data: Build a pool of beta testers for PAWS - https://phabricator.wikimedia.org/T129297#2101483 (10jayvdb) What is CSCW?
[00:34:28] <bd808>	 T129283
[00:37:01] <wikibugs>	 10Wikibugs: Get icon and color from API instead of screen scraping - https://phabricator.wikimedia.org/T1176#2101622 (10mmodell)
[00:37:29] <wikibugs>	 10Wikibugs: Get icon and color from API instead of screen scraping - https://phabricator.wikimedia.org/T1176#20337 (10mmodell) Can this be closed now that 1b6bbd391ad1f23a merged?
[00:38:22] <bd808>	 greg-g: are you an op in here? I wonder if somebody +b'd stashbot?
[00:38:42] <greg-g>	 I'm not
[00:38:57] <wikibugs>	 10Wikibugs: Get icon and color from API instead of screen scraping - https://phabricator.wikimedia.org/T1176#20337 (10Legoktm) No? That patch didn't make it stop screen-scraping.
[00:39:48] <wikibugs>	 10Wikibugs: Wikibugs links sometimes to the creation event, not to the mentioned comment - https://phabricator.wikimedia.org/T129246#2101648 (10mmodell) FYI: I want to build a unified conduit api so that you wouldn't have to screenscrape anything, and hopefully it won't even be necessary to make multiple api cal...
[00:49:30] <bd808>	 !bash Just a test
[00:50:15] <bd808>	 greg-g: stashbot must be muted by channel flags here somehow
[00:56:02] <greg-g>	 bd808: :(
[00:56:41] <shinken-wm>	 RECOVERY - Puppet failure on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[01:22:33] <wikibugs>	 6Labs, 6Operations, 10wikitech.wikimedia.org, 13Patch-For-Review: Update wikitech-static OS/PHP version - https://phabricator.wikimedia.org/T126385#2101749 (10Krenair) * Copied the wikitech import cron across * Copied MW config * Set up new copy of MediaWiki * Installed memcached and mysql-server-core-5.5/...
[01:52:19] <wikibugs>	 10Quarry: Quarry should preserve the protocol when logging in through OAuth - https://phabricator.wikimedia.org/T92600#2101766 (10Huji) 5Invalid>3Open
[02:25:32] <wikibugs>	 10Quarry: Quarry should preserve the protocol on redirects - https://phabricator.wikimedia.org/T92600#2101895 (10Huji)
[02:49:56] <wikibugs>	 6Labs, 6Operations, 10wikitech.wikimedia.org, 13Patch-For-Review: Update wikitech-static OS/PHP version - https://phabricator.wikimedia.org/T126385#2101906 (10Dzahn) once we are ready to switch i was going to do this https://gerrit.wikimedia.org/r/#/c/276088/
[02:56:22] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1406 is OK: OK: Less than 1.00% above the threshold [0.0]
[03:08:18] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0]
[03:15:10] <wikibugs>	 10PAWS: PAWS 404 for users with special characters in their names - https://phabricator.wikimedia.org/T120066#2101919 (10yuvipanda) After messing around and debugging, I believe https://github.com/jupyter/jupyterhub/pull/472 fixes it! \o/  The terminal does not work yet, which I believe is fixed by https://githu...
[03:15:37] <yuvipanda>	 /buff/buffer 28
[03:15:38] <yuvipanda>	 bah
[03:16:08] <bd808>	 yuvipanda: I'm more used to seeing "w/in 23"
[03:17:37] <wikibugs>	 10PAWS, 6Research-and-Data: Build a pool of beta testers for PAWS - https://phabricator.wikimedia.org/T129297#2101920 (10yuvipanda) @jayvdb CSCW was a conference where we ran a workshop on using PAWS to do wikimedia related research - you can see more info at https://meta.wikimedia.org/wiki/Research:Breaking_i...
[03:17:39] <yuvipanda>	 bd808: heh
[03:17:45] <yuvipanda>	 bd808: I switched to weechat rather than irssi
[03:17:52] <yuvipanda>	 hopefully someday I can stop using IRC completely
[03:18:01] * bd808 will miss yuvipanda 
[03:18:33] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0]
[03:19:15] <yuvipanda>	 bd808: hopefully the new place will also be open and free and just better technology and contain commandline clients too for people who prefer that
[03:19:22] <yuvipanda>	 unfortunately no such place exists atm
[03:19:31] <bd808>	 yeah. I get the hope
[03:19:53] <bd808>	 I just kinda don't believe that it is coming any time soon
[03:20:10] <yuvipanda>	 I think it might exist in the next 10 years
[03:20:23] <bd808>	 but maybe all the l33t users of slack will start to work on it after microsfot buys them
[03:20:27] * yuvipanda just got added to another slack 'team' today
[03:20:31] <bd808>	 *microsoft
[03:37:48] <wikibugs>	 6Labs, 6Operations, 10wikitech.wikimedia.org, 13Patch-For-Review: Wikitechwiki has 4xx responses to requests for some static assets inc. poweredby_mediawiki_88x31.png and WikiEditor's button-sprite.svg - https://phabricator.wikimedia.org/T128747#2101935 (10Krenair) 5Open>3Resolved a:5Krenair>3None
[03:45:35] <wikibugs>	 6Labs, 6Operations, 10wikitech.wikimedia.org, 13Patch-For-Review: Update wikitech-static OS/PHP version - https://phabricator.wikimedia.org/T126385#2101940 (10Krenair) * Fiddled with apache config some more (`Require all granted`) to make this work under Apache 2.4 * Installed php5-mysql * `mysqldump` on o...
[03:51:33] <wikibugs>	 6Labs, 6Operations, 10wikitech.wikimedia.org, 13Patch-For-Review: Update wikitech-static OS/PHP version - https://phabricator.wikimedia.org/T126385#2101941 (10Krenair) a:5Krenair>3Andrew @Andrew, do you think this is OK to switch now?
[04:22:39] <wikibugs>	 10PAWS, 13Patch-For-Review: Setup an icinga check for PAWS - https://phabricator.wikimedia.org/T129209#2101955 (10yuvipanda) Had to revert with https://gerrit.wikimedia.org/r/#/c/276093/ since icinga wouldn't recognize the host at all :|  @Dzahn any idea what I did wrong?
[04:24:21] <wikibugs>	 6Labs, 10Tool-Labs: Goal: Allow using k8s instead of GridEngine as a backend for webservices (Tracking) - https://phabricator.wikimedia.org/T129309#2101957 (10yuvipanda)
[04:26:05] <wikibugs>	 6Labs, 10Labs-Sprint-100, 10Tool-Labs: Unify / simplify webservice code - https://phabricator.wikimedia.org/T98440#2101972 (10yuvipanda)
[04:26:06] <wikibugs>	 6Labs, 10Tool-Labs: Goal: Allow using k8s instead of GridEngine as a backend for webservices (Tracking) - https://phabricator.wikimedia.org/T129309#2101971 (10yuvipanda)
[04:27:32] <wikibugs>	 6Labs, 10Tool-Labs: Setup a proper deployment strategy for Kubernetes - https://phabricator.wikimedia.org/T129311#2101987 (10yuvipanda)
[04:28:33] <wikibugs>	 6Labs, 6Operations, 13Patch-For-Review: Setup private docker registry with authentication support in tools - https://phabricator.wikimedia.org/T118758#2102004 (10yuvipanda)
[04:28:35] <wikibugs>	 6Labs, 10Tool-Labs: Goal: Allow using k8s instead of GridEngine as a backend for webservices (Tracking) - https://phabricator.wikimedia.org/T129309#2102003 (10yuvipanda)
[04:28:37] <wikibugs>	 6Labs, 10Tool-Labs: Setup DNS for kubernetes services - https://phabricator.wikimedia.org/T111914#2102005 (10yuvipanda)
[04:28:48] <wikibugs>	 6Labs, 10Tool-Labs: Define base Wikimedia Docker container - https://phabricator.wikimedia.org/T118446#2102009 (10yuvipanda)
[04:28:50] <wikibugs>	 6Labs, 10Tool-Labs: Goal: Allow using k8s instead of GridEngine as a backend for webservices (Tracking) - https://phabricator.wikimedia.org/T129309#2101957 (10yuvipanda)
[04:29:15] <wikibugs>	 6Labs, 10Tool-Labs: Define base Wikimedia Docker container - https://phabricator.wikimedia.org/T118446#1800488 (10yuvipanda) https://github.com/wikimedia/operations-docker-images-debian is where it's at, btw.
[04:31:35] <wikibugs>	 6Labs, 10Tool-Labs: Setup a supported HTTP Ingress solution for Kubernetes - https://phabricator.wikimedia.org/T129312#2102015 (10yuvipanda)
[05:13:49] <wikibugs>	 6Labs, 10Tool-Labs: Setup a supported HTTP Ingress solution for Kubernetes - https://phabricator.wikimedia.org/T129312#2102061 (10yuvipanda)
[05:14:42] <wikibugs>	 6Labs, 10Tool-Labs: Setup a supported HTTP Ingress solution for Kubernetes - https://phabricator.wikimedia.org/T129312#2102015 (10yuvipanda) We could also totally decide to not do this for the goal - this is probably a fair chunk of work - and just use our current setup (which will work just as well) for this...
[08:34:53] <shinken-wm>	 PROBLEM - Host tools-bastion-01 is DOWN: CRITICAL - Host Unreachable (10.68.17.228)
[10:11:16] <wikibugs>	 6Labs, 6Operations, 10wikitech.wikimedia.org, 13Patch-For-Review: Update wikitech-static OS/PHP version - https://phabricator.wikimedia.org/T126385#2102371 (10Southparkfan) Not very important, but is it possible to enable OPcache in PHP? Even if you only allow 64MB of cache usage, this should make the wiki...
[14:48:36] <hashar>	 andrewbogott: I am all happy with Horizon :-}
[14:48:48] <hashar>	 I can spawn/delete instances, look at boot logs etc
[14:49:39] <andrewbogott>	 hashar: great!  DNS is going to be messy :(
[14:49:58] <hashar>	 andrewbogott: I think a couple instances lost DNS entries
[14:50:04] <hashar>	 havent investigated further though
[14:50:15] <andrewbogott>	 newish instances?
[14:50:49] <hashar>	 a few months old maybe
[14:50:54] <hashar>	 will try to find the affected ones
[14:51:03] <hashar>	 that prevent ssh access to it iirc
[14:51:09] <hashar>	 but we do most commands via salt nowadays
[14:57:27] <andrewbogott>	 hashar: hm… I purged a bunch of leaked/duplicate DNS entries a few months ago.  And I must’ve hit an awful lot of valid entries by mistake because this issue keeps coming up :(
[15:03:14] <wikibugs>	 6Labs, 6Operations, 10wikitech.wikimedia.org, 13Patch-For-Review: Update wikitech-static OS/PHP version - https://phabricator.wikimedia.org/T126385#2103178 (10Andrew) I'm happy to switch over, as soon as daniel removes the WIP from https://gerrit.wikimedia.org/r/#/c/276088
[15:36:43] <grrrit-wm>	 (03PS1) 10MarcoAurelio: CSS and Elections [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/276184 (https://phabricator.wikimedia.org/T128742) 
[15:42:40] <grrrit-wm>	 (03PS1) 10Youni Verciti: Stage 1 - Cleaning code, using explict names [labs/tools/vocabulary-index] - 10https://gerrit.wikimedia.org/r/276187 
[15:44:40] <grrrit-wm>	 (03CR) 10MarcoAurelio: [C: 032] CSS and Elections [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/276184 (https://phabricator.wikimedia.org/T128742) (owner: 10MarcoAurelio)
[15:45:12] <grrrit-wm>	 (03Merged) 10jenkins-bot: CSS and Elections [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/276184 (https://phabricator.wikimedia.org/T128742) (owner: 10MarcoAurelio)
[15:46:48] <mafk>	 !log tools.stewardbots Merged https://gerrit.wikimedia.org/r/276184
[15:46:51] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL, Master
[16:12:52] <wikibugs>	 6Labs, 10Horizon: Horizon dashboard for managing http proxies for labs instances - https://phabricator.wikimedia.org/T129245#2103360 (10Andrew) Having thought about this for 30 seconds, I'm remembering that the SpecialNovaProxy interface actually manages /two/ things:  the actual proxy (via invisible unicorn)...
[16:18:09] <hashar>	 andrewbogott: would it be worth running a script to verify all instances are properly in DNS backend (ldap?) and can resolve fine?
[16:20:09] <andrewbogott>	 hashar: if you are offering to write that script then absolutely!
[16:20:13] <andrewbogott>	 (It’s not ldap, mercifully)
[16:21:37] * hashar ducks
[16:21:52] <chasemp>	 hashar: we have a ticket for this somewhere if you threw a note in there on teh diea
[16:21:54] <chasemp>	 idea even
[16:21:57] <hashar>	 andrewbogott: if you get the raw data, I dont mind handling the compare
[16:22:29] <hashar>	 which is really all about: colordiff -u <(sort -f instances.txt) <(sort -f dns.txt)
[16:22:38] <hashar>	 taking a break before next meeting
[16:35:48] <wikibugs>	 6Labs, 10Tool-Labs: Linux Error: libgcc_s.so.1 - https://phabricator.wikimedia.org/T129361#2103417 (10doctaxon)
[16:36:13] <wikibugs>	 6Labs, 10Tool-Labs: Linux Error: libgcc_s.so.1 - https://phabricator.wikimedia.org/T129361#2103431 (10doctaxon) p:5Triage>3High
[16:39:30] <wikibugs>	 6Labs, 10Tool-Labs: Linux Error: libgcc_s.so.1 - https://phabricator.wikimedia.org/T129361#2103417 (10valhallasw) You're not requesting enough memory for the job, so malloc() fails, causing this error. Pass e.g. -mem 512M (or a higher number) to jsub to request more memory than the default.
[16:42:57] <wikibugs>	 6Labs, 10Tool-Labs: Linux Error: libgcc_s.so.1 - https://phabricator.wikimedia.org/T129361#2103449 (10doctaxon) I gave it -mem 1g  jsub -once -j y -quiet -v LC_ALL=en_US.UTF-8 -mem 1g -l release=trusty ldw.tcl
[16:57:09] <grrrit-wm>	 (03PS1) 10Glaisher: Centralize db credentials config file [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/276205 
[17:02:01] <grrrit-wm>	 (03PS1) 10MarcoAurelio: General overhaul of CSS/JS content [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/276206 
[17:05:17] <wikibugs>	 10Tool-Labs-tools-stewardbots, 13Patch-For-Review: Make elections.php work again - https://phabricator.wikimedia.org/T128742#2103537 (10MarcoAurelio) CSS/JS imported, working again. But some files are also missing which I think this change (https://gerrit.wikimedia.org/r/#/c/276206/) will solve.
[17:19:01] <Cyberpower678>	 kaldari, ping
[17:19:09] <kaldari>	 Howdy
[17:19:59] <kaldari>	 Cyberpower678: BTW, we have the logging API for Cyberbot finished: http://tools.wmflabs.org/deadlinks/ https://meta.wikimedia.org/wiki/Fixing_dead_links/Deadlink_logging_app
[17:20:01] <Cyberpower678>	 kaldari, I don't want to put a damper on things, but how's the checkIfDead class coming.  It seems the current focus is the central API.
[17:20:41] <kaldari>	 Cyberpower678: The checkifDead improvements will probably be done this week
[17:20:48] <Cyberpower678>	 Yay. :-)
[17:21:01] <Cyberpower678>	 Then I can probably launch another BRFA.
[17:21:13] <Cyberpower678>	 Cyberbot is almost finished for enwiki. :D
[17:21:31] <Cyberpower678>	 Then I will move it to InternetArchiveBot
[17:23:20] <kaldari>	 Cool. Let me know when you have a chance to look at the deadlink logging API. If you have suggestions for improving it, feel free to ping me. The next step on that is to provide some output APIs: 1 for finding the date that an article was last processed, and 1 for finding out the last article that a bot worked on.
[17:24:38] <kaldari>	 Cyberpower678: We could also add the code to Cyberbot to interface with the API or we can leave that to you if you want.
[17:25:17] <grrrit-wm>	 (03PS1) 10MarcoAurelio: Adding .lighttpd.conf file [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/276209 
[17:25:18] <kaldari>	 Hopefully this will provide a better view into Cyberbot's progress than the tagging system
[17:26:09] <Cyberpower678>	 As long as the response time from the API is almost instantaneous, I should be fine.
[17:27:06] <Cyberpower678>	 But then again, if the bot runs on tools and so does the API, you could create an internal API for the bot to access too.  So in case of a common webservice failure, Cyberbot can continue to log.
[17:27:36] <grrrit-wm>	 (03PS2) 10MarcoAurelio: Adding .lighttpd.conf file [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/276209 
[17:28:01] <Cyberpower678>	 kaldari, ^
[17:29:43] <kaldari>	 Cyberpower678: That's not a bad idea
[17:30:30] <kaldari>	 The API should be super fast. It doesn't have to query anything to do the logging, so it should be nearly instant
[17:30:39] <Cyberpower678>	 kaldari, but get the checkIfDead finished first.  That's a little higher on my priority list at the moment.  I've been itching to turn that feature on and see Cyberbot will handle
[17:31:14] <kaldari>	 Cyberpower678: Will do. I'll see if we can have it ready by tomorrow actually
[17:31:38] <Cyberpower678>	 kaldari, I just deployed a serious bug. :/
[17:31:47] <kaldari>	 uh oh
[17:32:23] <Cyberpower678>	 Some URL's query string has stuff in that needs to be escaped before I can feed that into a cite template.
[17:32:34] <Cyberpower678>	 But I ended up escaping the entire URL
[17:32:46] <Cyberpower678>	 Now all the links it converts, are broken. :/
[17:33:12] <kaldari>	 oops
[17:33:29] <kaldari>	 This is why you need unit tests ;)
[17:33:51] <Cyberpower678>	 I thought it was a simple enough fix to not need any tests. 
[17:33:58] <Cyberpower678>	 Shows what I know. :p
[17:35:29] <grrrit-wm>	 (03PS3) 10MarcoAurelio: Adding .lighttpd.conf file [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/276209 
[17:38:42] <grrrit-wm>	 (03PS4) 10MarcoAurelio: Adding .lighttpd.conf file [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/276209 
[17:42:42] <bd808>	 andrewbogott: are you an op in this channel? I think that stashbot got muted here somehow. The bot isn't acking T1234 phab references anymore, but it works in all the other channels I've tested in.
[17:43:09] <bd808>	 The bot is here and seeing new messages and not logging any errors about sending things
[17:43:17] <wikibugs>	 10Wikibugs: Get icon and color from API instead of screen scraping - https://phabricator.wikimedia.org/T1176#2103733 (10mmodell) project.search and maniphest.search return the needed info... Though it'd be nice to have it combined into one api call
[18:16:54] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1215 is OK: OK: Less than 1.00% above the threshold [0.0]
[18:17:53] <Luke081515>	 bd808: There are only two quiets: one at *!*@198.199.82.216 and one at *!*@wikimedia/bot/helpmebot
[18:18:15] <bd808>	 yeah those were the only ones I could see too.
[18:18:48] <bd808>	 But it think there are ways to +q that don't show to non-ops (might be wrong on that)
[18:19:12] <Luke081515>	 I don't there are non visible quiets? I don't know such a possibilty
[18:19:24] <Luke081515>	 btw, is the topic still up to date?
[18:19:56] <yuvipanda>	 it isn't
[18:20:05] <yuvipanda>	 'some tools may need to be manually restarted' is not true
[18:20:22] <yuvipanda>	 however my client sucks and I can't copy paste the title to just remove that bit, can someone else do it?
[18:20:24] <Luke081515>	 Ok, I will remove it
[18:20:45] <yuvipanda>	 thanks Luke081515 
[18:20:58] <Luke081515>	 (I'm using a second client for that too ;))
[18:21:23] <yuvipanda>	 heh :D
[18:21:54] <Luke081515>	 This client gots a lot of "extra functions", but normally I don'T need them ;)
[18:36:15] <halfak>	 yuvipanda, once when I was looking over your shoulder, I saw you type "labs <instance name>" and that was interpreted as a "ssh <instance name>.eqiad.wmflabs"
[18:36:27] <halfak>	 Was wondering how you did that.  Alias line?  Bash script? 
[18:36:59] <yuvipanda>	 halfak: alias line :)
[18:37:08] <halfak>	 Care to share your line? :) 
[18:38:35] <yuvipanda>	 I don't have it anymore unfortunately
[18:38:36] <yuvipanda>	 but let me see
[18:38:57] <halfak>	 Oh!  I made it work with a function 
[18:39:06] * halfak pastes his function and alias
[18:39:19] <yuvipanda>	 cool :D
[18:39:28] <chasemp>	 aliases usually don't take args I wonder how you did it?
[18:39:28] <halfak>	 https://gist.github.com/halfak/c29724704fc99f6c2241
[18:39:34] <halfak>	 A function!  
[18:39:40] <halfak>	 http://stackoverflow.com/questions/7131670/make-bash-alias-that-takes-parameter
[18:39:43] <halfak>	 Thanks SO :D
[18:40:38] <halfak>	 I typo "eqiad.wmflabs" as "eqiad.wmnet" so often.  I'm looking forward to not doing that anymore. 
[18:40:50] <yuvipanda>	 :D
[18:41:06] <halfak>	 Sorry to bug you yuvipanda.  Do you think this would be a nice email to labs-l?
[18:42:04] <yuvipanda>	 halfak: indeed! 
[18:47:46] <legoktm>	 woaaah
[18:47:50] <legoktm>	 that would save me so much time
[18:49:01] <halfak>	 OK.  Email sent :) 
[18:49:19] <halfak>	 I just made one for accessing my virtualenvs too. :) 
[18:49:36] <halfak>	 This is a useful trick :) 
[18:51:13] <legoktm>	 I have `py2` and `py3` aliased to different virtualenvs so its easy to switch
[18:51:51] <halfak>	 I just set up an alias that handles "venv <envname>"
[18:52:06] <halfak>	 so "venv 3.4" starts up my general 3.4 env
[18:52:07] <legoktm>	 Only downside to `labs host` is loss of tab complete...
[18:52:15] <halfak>	 But I have some specific ones -- like 'ores"
[18:52:23] <halfak>	 legoktm, good point
[18:53:02] <halfak>	 I usually don't list out big sets of instances in my .ssh/config, but I suppose that if you did, tab complete could be really powerful.
[18:53:12] <halfak>	 You just have to maintain the list
[18:53:14] <yuvipanda>	 woah,  there's tab complete?
[18:54:58] <legoktm>	 oh, I have mine set up to remember any host I've ever ssh'd into using .ssh/known_hosts
[18:55:22] <chasemp>	 you could pretty easily do bash completion for labs <foo>
[18:55:35] <halfak>	 chasemp, cool.  how do you set something like that up?
[18:55:36] <chasemp>	 copy the normal ssh one almost and put in bash.completion.d I would imagine
[18:56:15] <halfak>	 Woah.  
[18:56:20] * halfak did not know this was here :) 
[18:56:45] <chasemp>	 bash just uses bash to do tab completion, even contextual like certain options for a command
[18:56:56] <chasemp>	 so the magic is usually just a shell script in the .d with a package
[18:58:48] <chasemp>	 I think w/ homebrew on osx you can peek in ls /usr/local/etc/bash_completion.d/
[18:58:50] <chasemp>	 to kind of get an idea
[19:02:23] <halfak>	  /etc/bash_completion.d in Ubuntu Trusty :) 
[19:02:30] <chasemp>	 heh you win
[19:02:46] <chasemp>	 there may also be logic to parse like ~/.bash_completion.d I think 
[19:02:50] <chasemp>	 to do local overrides
[19:05:37] * halfak digs into that for a minute and finds a bit of a mess
[19:05:46] <halfak>	 Looks like the ubuntu default .bashrc
[19:05:51] <halfak>	 doesn't have that. 
[19:06:07] <halfak>	 But if you dig into the lines it does have about loading completion vars, you could probably figure out how to load your own. 
[19:06:43] <halfak>	 'cause there are a few lines about executing /etc/bash_completion if it exists
[19:06:55] * halfak waits for model to build
[19:07:14] <halfak>	 A lot like waiting for something to compile. :) 
[19:07:28] <chasemp>	 I fell down this hole writing my prompt line a few years ago :)
[19:07:34] <chasemp>	 bash completion that is
[19:11:34] <paravoid>	 yuvipanda: ssh does have tab completion by default, but ssh hashes known_hosts by default so it doesn't work
[19:11:49] <paravoid>	 you can do "HashKnownHosts no" in your .ssh/config
[19:11:50] <yuvipanda>	 ah
[19:12:10] <paravoid>	 and/or you can use a more authoritative list for your known_hosts
[19:12:17] <paravoid>	 than typing yes when prompted with fingerprints
[19:12:39] <paravoid>	 for production, I copy known_hosts from bast1001
[19:12:50] <paravoid>	 so I have a chain of trust, essentially
[19:13:01] <paravoid>	 (which unfortunately right now involves puppet, but ok)
[19:13:13] <yuvipanda>	 yeah
[19:13:16] <yuvipanda>	 a bit harder for labs tho
[19:14:50] <paravoid>	 oh and halfak, including /etc/bash_completion should suffice
[19:21:17] <wikibugs>	 6Labs, 6Operations, 10wikitech.wikimedia.org, 13Patch-For-Review: Update wikitech-static OS/PHP version - https://phabricator.wikimedia.org/T126385#2104213 (10Dzahn) Done, removed the WIP and merged DNS switch.
[19:22:09] <Krenair>	 yuvipanda, hey
[19:22:10] <wikibugs>	 6Labs, 6Operations, 10wikitech.wikimedia.org, 13Patch-For-Review: Update wikitech-static OS/PHP version - https://phabricator.wikimedia.org/T126385#2104214 (10Dzahn) the sync between wikitech and wikitech-static .. does that need any change because the IP changed? or all just hostname based?
[19:22:17] <yuvipanda>	 hey Krenair 
[19:22:35] <Krenair>	 I set up an instance called horizon-proxy-dashboard.openstack.eqiad.wmflabs yesterday
[19:22:47] <wikibugs>	 6Labs, 6Operations, 10wikitech.wikimedia.org: Update wikitech-static OS/PHP version - https://phabricator.wikimedia.org/T126385#2104215 (10Dzahn)
[19:22:54] <Krenair>	 but it's now decided to prompt me for a password before allowing me to sudo
[19:23:22] <wikibugs>	 6Labs, 6Operations, 10wikitech.wikimedia.org: Update wikitech-static OS/PHP version - https://phabricator.wikimedia.org/T126385#2012874 (10Dzahn) a:5Andrew>3Krenair
[19:23:54] <wikibugs>	 6Labs, 6Operations, 10wikitech.wikimedia.org: Update wikitech-static OS/PHP version - https://phabricator.wikimedia.org/T126385#2104219 (10Krenair) I don't think anything needs to be changed - the access restriction was removed recently and the dumps became public
[19:24:03] <yuvipanda>	 Krenair: ah
[19:24:10] <yuvipanda>	 Krenair: andrewbogott would have a better idea about that, probably
[19:24:15] <Krenair>	 ok
[19:24:36] <Krenair>	 I suppose I could always add myself to the root authorized keys for the project and see what's up
[19:25:18] <wikibugs>	 6Labs, 6Operations, 10wikitech.wikimedia.org: Update wikitech-static OS/PHP version - https://phabricator.wikimedia.org/T126385#2104222 (10Dzahn) 5Open>3Resolved eh, true :) i just did that in T54170   i will just claim you resolved this
[19:30:35] <wikibugs>	 6Labs, 6Operations, 10wikitech.wikimedia.org: decom old wikitech-static machine - https://phabricator.wikimedia.org/T129391#2104237 (10Dzahn)
[19:31:27] <wikibugs>	 6Labs, 6Operations, 10wikitech.wikimedia.org: decom old wikitech-static machine - https://phabricator.wikimedia.org/T129391#2104237 (10Dzahn)
[19:34:25] <Krenair>	 yuvipanda, mind running puppet there?
[19:36:04] <yuvipanda>	 Krenair: sure
[19:36:31] <yuvipanda>	 > The last Puppet run was at Tue Mar  8 21:05:42 UTC 2016 (1350 minutes ago). 
[19:36:33] <yuvipanda>	 hmm
[19:36:44] <yuvipanda>	 Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to determined $::labsproject at /etc/puppet/manifests/realm.pp:24 on node horizon-proxy-dashboard.openstack.eqiad.wmflabs
[19:36:46] <yuvipanda>	 Warning: Not using cache on failed catalog
[19:36:48] <yuvipanda>	 Error: Could not retrieve catalog; skipping run
[19:36:50] <yuvipanda>	 Krenair: ^
[19:37:29] <shinken-wm>	 PROBLEM - Puppet failure on tools-worker-1009 is CRITICAL: CRITICAL: 16.67% of data above the critical threshold [0.0]
[19:40:22] <Krenair>	 labsprojectfrommetadata looks... interesting
[19:41:32] <Krenair>	       metadata = Facter::Util::Resolution.exec("curl -f http://169.254.169.254/openstack/2013-10-17/meta_data.json/ 2> /dev/null").chomp
[19:41:33] <Krenair>	 what.
[19:42:35] <shinken-wm>	 RECOVERY - Puppet failure on tools-worker-1009 is OK: OK: Less than 1.00% above the threshold [0.0]
[19:42:46] <Krenair>	 okay
[19:42:57] <Krenair>	 krenair@horizon-proxy-dashboard:~$ hostname -d
[19:42:57] <Krenair>	 krenair@horizon-proxy-dashboard:~$ 
[19:43:12] <yuvipanda>	 fun
[19:43:33] <Krenair>	 other instances print something like bastion.eqiad.wmflabs
[19:44:22] <Krenair>	 of course, I can't change anything useful because no root
[19:45:20] <yuvipanda>	 Krenair: I just ran a 
[19:45:27] <yuvipanda>	 usermod -a -G sudo krenair
[19:45:31] <Krenair>	 ty
[19:45:35] <yuvipanda>	 Krenair: so logout and log back in and try again?
[19:45:58] <Krenair>	 no luck
[19:46:04] <Krenair>	 am in the group but sudo still prompts for password
[19:46:40] <Krenair>	 could you just apply https://wikitech.wikimedia.org/w/index.php?title=Hiera:Openstack&diff=356786&oldid=191986 manually?
[19:47:29] <yuvipanda>	 Krenair: just did
[19:47:50] <Krenair>	 works, thanks
[19:48:08] <yuvipanda>	 np
[19:54:20] <dschwen>	 Hi all, I keep getting warning emails about failed puppet runs on one of my instances
[19:54:31] <dschwen>	 I think, however, that I fixed the issue
[19:54:44] <dschwen>	 manual runs work without any problems
[19:54:59] <dschwen>	 is there something that needs to be reset for the emails to stop coming?
[19:57:56] <Krenair>	 dschwen, you get no errors when running puppet, but you still get emails about failures?
[20:03:30] <shinken-wm>	 PROBLEM - Puppet failure on tools-worker-1009 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[20:22:14] <Krenair>	 yuvipanda, got it working again
[20:22:29] <Krenair>	 yuvipanda, had to re-set the hostname and remove that IP from the loopback interface
[20:22:37] <yuvipanda>	 fun
[20:23:55] <wikibugs>	 10PAWS: PAWS 404 for users with special characters in their names - https://phabricator.wikimedia.org/T120066#2104474 (10yuvipanda) The notebook PR got merged upstream too.
[20:27:42] <dschwen>	 Krenair, that's correct
[20:27:49] <dschwen>	 no errors and $? is 0
[20:31:06] <yuvipanda>	 dschwen: can you file a bug?
[20:31:12] <yuvipanda>	 dschwen: and cc andrewbogott :D
[20:32:23] <andrewbogott>	 dschwen: if it works manually it really shouldn’t be sending you emails.  Do please file a bug.
[20:33:31] <shinken-wm>	 RECOVERY - Puppet failure on tools-worker-1009 is OK: OK: Less than 1.00% above the threshold [0.0]
[20:40:47] <dschwen>	 k, will do
[20:45:53] <dschwen>	 done
[20:46:54] <dschwen>	 assigned to andrewbogott (I assume that is what you meant by CC, yuvipanda?)
[20:48:08] <yuvipanda>	 ah, there's a 'subscribers' field, which is what I meant :)
[20:48:16] <yuvipanda>	 but andrewbogott can probably unassign/reassign as he sees fit
[20:48:26] <yuvipanda>	 you should also usually associate the 'Labs' project 
[20:48:33] <yuvipanda>	 thanks for the bug, dschwen!
[20:56:53] <wikibugs>	 6Labs, 10Tool-Labs: Linux Error: libgcc_s.so.1 - https://phabricator.wikimedia.org/T129361#2103417 (10yuvipanda) give it more memory?
[20:59:39] <wikibugs>	 10PAWS: Implement a sane way to access mysql replicas from PAWS - https://phabricator.wikimedia.org/T120471#2104639 (10yuvipanda) a:3yuvipanda
[21:00:06] <yuvipanda>	 ^ is what I'm going to work on today!
[21:06:41] <wikibugs>	 6Labs, 7Puppet: Receiving puppet run failure alert for instance where manual puppet runs complete fine - https://phabricator.wikimedia.org/T129403#2104657 (10dschwen) a:5Andrew>3yuvipanda
[21:07:10] <dschwen>	 Oh, I guess that's necessary to have wikibugs post the announcements here :-)
[21:07:47] <yuvipanda>	 dschwen: yup
[21:08:07] <yuvipanda>	 dschwen: also in general, you don't assign things to people. people pick and assign things to themselves
[21:08:16] <wikibugs>	 6Labs, 7Puppet: Receiving puppet run failure alert for instance where manual puppet runs complete fine - https://phabricator.wikimedia.org/T129403#2104660 (10yuvipanda) a:5yuvipanda>3None
[21:21:09] <wikibugs>	 6Labs, 10Tool-Labs, 10pywikibot-core: Tool Labs Pywikibot does not work with new shared Pywikibot config files - https://phabricator.wikimedia.org/T129406#2104686 (10Ato_01)
[21:21:56] <Luke081515>	 Hello, can someone tell me, how to add a new wiki to wikidata, if you're using the wikidata role at vagrant?
[21:34:57] <wikibugs>	 10PAWS, 6Research-and-Data: Build a pool of beta testers for PAWS - https://phabricator.wikimedia.org/T129297#2104733 (10DarTar)
[21:35:05] <Luke081515>	 !log rcm deleted rcm-4 (was unused)
[21:35:14] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL, Master
[21:35:28] <Luke081515>	 labs-morebots is slow today
[21:35:28] <labs-morebots>	 I am a logbot running on tools-exec-1221.
[21:35:28] <labs-morebots>	 Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log.
[21:35:29] <labs-morebots>	 To log a message, type !log <msg>.
[21:35:42] <wikibugs>	 6Labs, 10Tool-Labs, 10pywikibot-core: Tool Labs Pywikibot does not work with new shared Pywikibot config files - https://phabricator.wikimedia.org/T129406#2104734 (10Ato_01) p:5Triage>3Normal
[21:40:34] <paladox>	 Luke081515: Hi please try in the #wikidata channel you may get a faster response. You can also ping the admins there and they are fast at replying.
[21:40:56] <Luke081515>	 ok, I will try it
[21:42:19] <paladox>	 Luke081515: Ok thanks.
[22:23:14] <yuvipanda>	 !ping
[22:23:14] <wm-bot>	 !pong
[22:23:16] <yuvipanda>	 ok
[22:37:04] <wikibugs>	 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2105003 (10kaldari)
[23:06:59] <wikibugs>	 6Labs: Sort out labs user privs in Horizon vs. Wikitech - https://phabricator.wikimedia.org/T91830#2105124 (10Andrew) 5Open>3Resolved a:3Andrew I think this is reasonably well-address now, but for odd corner cases.
[23:07:01] <wikibugs>	 6Labs, 7Tracking: Make OpenStack Horizon useful for production labs - https://phabricator.wikimedia.org/T87279#2105127 (10Andrew)
[23:09:51] <wikibugs>	 6Labs: Switch to using Horizon/Designate for labs public dns - https://phabricator.wikimedia.org/T124184#2105132 (10Andrew) There are a few conflicts between our security model and designates.  Specifically, domains are 'owned' by a project, and subdomains cannot be created in other projects.  E.g. wmflabs.org w...
[23:12:26] <shinken-wm>	 PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:13:13] <ori>	 yuvipanda: ^
[23:13:40] <yuvipanda>	 ori: yup, load is going back down now
[23:13:50] <yuvipanda>	 we should also get rid of the shinken check, since we've a paging icinga one
[23:13:53] <yuvipanda>	 I'll try to do that later
[23:17:22] <shinken-wm>	 RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 801037 bytes in 5.241 second response time
[23:35:56] <wikibugs>	 10PAWS, 13Patch-For-Review: Setup an icinga check for PAWS - https://phabricator.wikimedia.org/T129209#2105209 (10Dzahn) works now, after moving it into the icinga module  service: https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=paws.wmflabs.org&nostatusheader  host: https://icinga.wikimedia.org/cg...
[23:36:07] <wikibugs>	 10PAWS, 13Patch-For-Review: Setup an icinga check for PAWS - https://phabricator.wikimedia.org/T129209#2105210 (10Dzahn) 5Open>3Resolved
[23:42:08] <wikibugs>	 10PAWS, 13Patch-For-Review: Setup an icinga check for PAWS - https://phabricator.wikimedia.org/T129209#2105217 (10Dzahn) So yea, when using @monitoring::host to create  a virtual host in Icinga, that needs to happen within the icinga module or something that is applied on neon.  we have this structure that wor...