[03:00:42] <Niharika>	 Labs magicians! I need some help. I setup an instance yesterday and it works fine, it boots up and all. I setup a proxy for it using https://wikitech.wikimedia.org/wiki/Help:Proxy as my bible, but it still doesn't work. http://commtech.wmflabs.org gives a 504 gateway timeout. :( 
[03:01:48] <Niharika>	 YuviPanda: ^ If you're awake and at work. 
[03:02:12] <YuviPanda>	 complex question, Niharika!
[03:02:17] <YuviPanda>	 (awake, and at work, that is :))
[03:02:24] <YuviPanda>	 Niharika: you need to open up your security groups
[03:02:55] <YuviPanda>	 Niharika: got o 'manage security groups' on the left sidebar, and add a rule to the default group opening up whatever port (8000?) for tcp, 10.0.0.0/8
[03:03:41] <Niharika>	 YuviPanda: Okay!
[03:05:45] <YuviPanda>	 Niharika: yw! there might be a note in Help:Proxy already if not can you edit it to add info?
[03:05:50] * YuviPanda feels pretty bad about the state of our docs
[03:06:19] <Niharika>	 YuviPanda: I could add it, but I don't understand what rule I should be adding. It seems there already exist a bunch of rules...
[03:06:45] <YuviPanda>	 Niharika: is there one for the port you want to proxy?
[03:07:01] <YuviPanda>	 Niharika: these are just firewall rules. so you need to open the port you want to expose the proxy with.
[03:07:16] <YuviPanda>	 Niharika: is this mediawiki-vagrant? if so the mw vagrant on labs page will have a port number...
[03:07:41] <Niharika>	 YuviPanda: Yes, this is mediawiki-vagrant. I see rules for port 22 and 5666.
[03:08:01] <YuviPanda>	 Niharika: right, so you need to add a rule for... port 8000 I think?
[03:08:37] <Niharika>	 YuviPanda: Okay. 
[03:09:18] <Niharika>	 YuviPanda: Protocol? 
[03:09:25] <YuviPanda>	 Niharika: 'tcp'
[03:10:37] <Niharika>	 YuviPanda: Do I need to add any CIDR changes? 
[03:10:54] <YuviPanda>	 Niharika: yes, 10.0.0.0/8
[03:11:48] <Niharika>	 YuviPanda: Okay, rule added. Will I be able to access my instance instantly or will this take a while? 
[03:11:53] * Niharika goes to update the docs
[03:12:01] <YuviPanda>	 Niharika: should be instant
[03:14:28] <Niharika>	 Hmm. Nah, it's not up yet. 
[03:15:03] <YuviPanda>	 Niharika: what's the instance name and the project name? I can take a look in about 5-10minutes
[03:15:07] <Niharika>	 I added a rule for 8080 too, that seems right, I think. But instance not showing up yet. 
[03:15:11] * YuviPanda hates our security group stuff
[03:15:31] <Niharika>	 YuviPanda: commtech-1 on commtech project.  
[03:15:56] * Niharika is GLAD to have YuviPanda taking care Labs <3
[03:23:22] <YuviPanda>	 Niharika: ugh, this is an underlying Openstack fault that's causing new security rules to not be applied
[03:23:29] <YuviPanda>	 Niharika: I gotta go home now, I'll look at it in like 40mins
[03:23:32] <YuviPanda>	 sorry!
[03:23:34] <YuviPanda>	 brb
[03:23:48] <Niharika>	 YuviPanda: No worries. I'm happy it was not something I did wrong. 
[03:23:52] <Niharika>	 Thanks! :) 
[04:21:25] <YuviPanda>	 Niharika: baccck
[04:21:27] <YuviPanda>	 looking at it now
[04:22:21] <Niharika>	 YuviPanda: Awesome. Also take a look at what i added in the docs https://wikitech.wikimedia.org/wiki/Help:Proxy and verify if that won't blow up somebody's computer. :P 
[04:31:16] <YuviPanda>	 Niharika: http://commtech.wmflabs.org/wiki/Main_Page bam
[04:31:47] <Niharika>	 YuviPanda: Wowie! Thanks! :D 
[04:31:56] <wikibugs>	 6Labs, 6operations: Security setting changes are not applied - https://phabricator.wikimedia.org/T118936#1816325 (10yuvipanda)
[04:32:07] * Niharika owes YuviPanda his choice of treat from delhi :P
[04:32:27] <wikibugs>	 6Labs, 6operations: Security setting changes are not applied - https://phabricator.wikimedia.org/T118936#1813172 (10yuvipanda) Happened again to @niharika with commtech-1, which was also on labvirt1010. Restarting nova-compute fixed it again.
[04:32:52] <wikibugs>	 6Labs, 6operations: Security setting changes are not applied - https://phabricator.wikimedia.org/T118936#1816327 (10yuvipanda) p:5Triage>3High
[04:32:59] <YuviPanda>	 Niharika: :D thanks!
[04:33:05] <YuviPanda>	 Niharika: ^ was the bug that bit you
[04:33:26] <Niharika>	 Right. 
[04:33:51] * YuviPanda was almost attacked by a very drunk person when he was in Delhi by a beer bottle
[04:34:00] <Niharika>	 :D 
[04:34:29] <Niharika>	 That's sad though. :/ 
[04:34:56] <YuviPanda>	 yeah
[04:35:02] <YuviPanda>	 fun times though
[04:35:23] * YuviPanda feels old now
[04:35:54] <Niharika>	 YuviPanda: I don't think a person with flaming red hair can EVER be called "old"
[04:36:09] <YuviPanda>	 haha :D
[04:36:28] <YuviPanda>	 it's going to be bright green soon
[04:37:56] <Niharika>	 Interesting. Just go for the rainbow already. 
[04:38:46] <YuviPanda>	 haha
[04:38:58] <YuviPanda>	 Niharika: i think fhocutt already has that covered
[04:39:39] <Niharika>	 Haha, yeah, probably. 
[04:43:05] * YuviPanda kills another old role
[04:43:09] <YuviPanda>	 so much code deletion!
[04:46:46] <wikibugs>	 6Labs, 5Patch-For-Review: Move all instances using role::lamp::labs to role::simplelamp - https://phabricator.wikimedia.org/T118784#1816331 (10yuvipanda) 5Open>3Resolved I KILLED THEM ALL.
[04:49:28] <Niharika>	 MURDERER! 
[04:50:04] <YuviPanda>	 :D
[04:50:07] * YuviPanda is killing another file now
[04:50:17] <YuviPanda>	 others have killed faaar more than I have!
[04:50:21] <YuviPanda>	 killing code is the best
[04:59:20] <Earwig>	 flaming red hair, eh?
[05:00:26] <YuviPanda>	 Earwig: https://twitter.com/yuvipanda/status/630151634560024576
[05:00:47] <YuviPanda>	 (it's black/red now)
[05:00:53] <Earwig>	 wow, that's more extreme than I thought
[05:01:05] <YuviPanda>	 heh
[05:01:32] <YuviPanda>	 the black grew out, so just the top half of my hair is read now and the roots are all black
[05:02:28] <YuviPanda>	 I'm debating if I should go full green or black+green
[05:19:18] <Niharika>	 Hi Earwig. Frances had a few questions for you yesterday. Are you going to be around for a while?
[05:19:27] <Earwig>	 right now?
[05:19:29] <Earwig>	 yeah
[05:20:29] <Niharika>	 Earwig: I think she'll be online in ~3-4 hours. She's in London. What timezone are you in?
[05:21:04] <Earwig>	 uhh, UTC-6 (Illinois)
[05:21:16] <Earwig>	 I'll be sleeping by then
[05:21:39] <Earwig>	 unless you wanna make me an insomniac :P  it'll be 3am
[05:21:58] <Niharika>	 Earwig: Uh oh. That's okay then. :) 
[05:23:16] <Niharika>	 I'll try to catch you when you wake up. 
[05:24:20] <Earwig>	 I probably won't be available tomorrow until... erm... 18 hours from now
[05:24:52] <YuviPanda>	 aren't timezones fun
[05:24:59] <YuviPanda>	 Earwig: I should check if the next ubuntu has mwparserfromhell package
[05:25:18] <YuviPanda>	 Earwig: it does! http://packages.ubuntu.com/xenial/python/python-mwparserfromhell
[05:25:39] <Earwig>	 that's insane
[05:26:00] <Earwig>	 xenial, really?
[05:26:05] <Earwig>	 ubuntu you silly thing
[05:26:06] <YuviPanda>	 there's a python3 version too http://packages.ubuntu.com/xenial/python/python3-mwparserfromhell
[05:26:09] <YuviPanda>	 haha
[05:26:11] <Earwig>	 what happens in a year and a half??
[05:26:12] <YuviPanda>	 I wonder what happens when they run out of it
[05:26:14] <YuviPanda>	 heh
[05:26:58] <Earwig>	 well assuming they follow ascii it'd be {
[05:27:06] <YuviPanda>	 I hope not
[05:28:20] <Earwig>	 "Outrageously powerful parser for MediaWiki wikicode"
[05:28:27] <Earwig>	 y'know this kind of self-advertising makes me feel weird
[05:31:20] <YuviPanda>	 heh
[05:31:49] <YuviPanda>	 Earwig: let me know if there are other packages/self-advertising-code that you think can end up in debian/ubuntu
[05:32:06] <Earwig>	 I don't even know if mwparserfromhell belongs there :P
[05:32:20] <YuviPanda>	 hehe :P
[05:32:23] <YuviPanda>	 well they accepted it...
[05:32:25] <Earwig>	 why is it there?
[05:32:32] <Earwig>	 yeah, who knows
[05:32:49] <YuviPanda>	 mostly because it's going to go into production with the revscoring service inside wikimedia soon
[05:32:52] <YuviPanda>	 so I needed a package
[05:32:57] <Earwig>	 wat
[05:33:08] <YuviPanda>	 and then already had a package and so legoktm suggested I push to get it into debian and so I did...
[05:33:18] <YuviPanda>	 Earwig: oh yeah, so ores.wmflabs.org uses mwparserfromhell
[05:33:34] <Earwig>	 super informative webpage, that is :P
[05:33:42] <YuviPanda>	 yeah :)
[05:33:49] <YuviPanda>	 https://meta.wikimedia.org/wiki/Objective_Revision_Evaluation_Service
[05:33:51] <YuviPanda>	 is better
[05:34:05] <Earwig>	 ohhhh
[05:34:08] <Earwig>	 boy
[05:34:12] <Earwig>	 that's something
[05:34:17] <Earwig>	 gonna replace cluebot?
[05:34:28] <YuviPanda>	 eventually hopefully. there's a scoredrevisions gadget you can check out
[05:48:49] <wikibugs>	 6Labs, 6operations: Security setting changes are not applied - https://phabricator.wikimedia.org/T118936#1816397 (10chasemp) Do we think this affects labvirt1010 specifically then?
[05:50:52] <wikibugs>	 6Labs, 6operations: Security setting changes are not applied - https://phabricator.wikimedia.org/T118936#1816398 (10yuvipanda) Not sure - it just so happened that the new instances I ran into were on 1010, but that could just be because that's where new instances are going now.
[06:24:01] <wikibugs>	 6Labs, 5Patch-For-Review: Convert all ldap globals into hiera variables instead - https://phabricator.wikimedia.org/T101447#1816424 (10yuvipanda) role::labs::instance class and realm variable are gone now too. The former is included via node.pp and the latter via hiera.
[07:33:55] <wikibugs>	 6Labs, 10Tool-Labs, 6operations, 5Patch-For-Review: Write a diamond collector to collect active ssh sessions - https://phabricator.wikimedia.org/T118827#1816444 (10yuvipanda) We should be collecting active list of open ssh (and mosh) sessions now. This should enable us to see how active individual instance...
[08:47:33] <wikibugs>	 6Labs, 10Tool-Labs, 6operations, 5Patch-For-Review: Write a diamond collector to collect active ssh sessions - https://phabricator.wikimedia.org/T118827#1816505 (10yuvipanda) 5Open>3Resolved
[13:00:50] <wikibugs>	 6Labs, 7Tracking: create labs project for better support of ContentTranslation for Korea - https://phabricator.wikimedia.org/T119063#1816879 (10Ryuch) 3NEW a:3yuvipanda
[15:04:44] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-proxy-01 is CRITICAL: CRITICAL: tools.tools-proxy-01.diskspace.root.byte_percentfree (<44.44%)
[15:24:46] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-proxy-01 is OK: OK: All targets OK
[15:35:14] <wikibugs>	 6Labs, 7Database: Database replicas: replicate user.user_touched - https://phabricator.wikimedia.org/T92841#1817295 (10jcrespo) The filters have been dropped (changed to allow that field). Now I have to backfill that column.
[15:42:21] <wikibugs>	 6Labs, 7Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#1817323 (10Ryuch)
[15:42:23] <wikibugs>	 6Labs, 7Tracking: create labs project for better support of ContentTranslation for Korea - https://phabricator.wikimedia.org/T119063#1817321 (10Ryuch) 5Open>3Invalid
[15:44:28] <wikibugs>	 6Labs, 6operations: Security setting changes are not applied - https://phabricator.wikimedia.org/T118936#1817325 (10Andrew) I see the same behavior on labvirt1005.  And, indeed, restarting nova-compute helps, whereas restarting nova-network does not.
[15:44:38] <wikibugs>	 6Labs, 6operations: Security setting changes are not applied - https://phabricator.wikimedia.org/T118936#1817326 (10Andrew) a:3Andrew
[16:08:52] <wikibugs>	 6Labs, 6operations: Security setting changes are not applied - https://phabricator.wikimedia.org/T118936#1817406 (10Andrew) 2015-11-19 16:07:59.145 56644 ERROR oslo_messaging.rpc.dispatcher [req-83c2d563-4f10-4f78-927d-f3e8482edade andrew testlabs - - -] Exception during message handling: 'metadata' 2015-11-19...
[16:18:01] <wikibugs>	 6Labs, 6operations: Security setting changes are not applied - https://phabricator.wikimedia.org/T118936#1817431 (10Andrew) This looks to be fixed by https://review.openstack.org/#/c/222023/
[16:18:21] <Nikerabbit>	 why is there no log of group membership changes to projects? or is it somewhere but I just can't find it?
[16:23:42] <wikibugs>	 6Labs, 6operations: Security setting changes are not applied - https://phabricator.wikimedia.org/T118936#1817456 (10Andrew) and... fixed in 2015.1.2.  I will upgrade and we'll see what we get.   https://bugs.launchpad.net/nova/+bug/1484738
[16:25:37] <chasemp>	 andrewbogott: weird man, so why did this crop up now I wonder? https://review.openstack.org/#/c/222023/
[16:27:21] <andrewbogott>	 It’s a thing we do fairly infrequently, and it’s only been broken for a couple of months...
[16:27:26] <andrewbogott>	 But yeah, weird we didn’t notice before
[16:35:54] <wikibugs>	 6Labs, 6operations: Security setting changes are not applied - https://phabricator.wikimedia.org/T118936#1817492 (10Andrew) 5Open>3Resolved I've updated nova-compute to 2015.1.2 on all labvirt nodes.  Seems fixed.
[19:44:53] <chasemp>	 andrewbogott: about? I'm going to roll w/ this hiera thing, but by freezing puppet across and slowly rolling out
[19:45:04] <chasemp>	 it's a bit hard to test via puppet compiler as it has no hiera private context it seems
[19:45:14] <chasemp>	 but looks good from waht I see
[19:45:20] <chasemp>	 http://puppet-compiler.wmflabs.org/1327/
[19:45:22] <andrewbogott>	 I’m here, how can I help?
[19:45:39] <chasemp>	 if things go boom help put them back together?
[19:46:00] <andrewbogott>	 ok :)
[19:47:35] * YuviPanda waves too
[19:47:52] <YuviPanda>	 andrewbogott: got any time to look at the designate stuff? :)
[19:54:09] <andrewbogott>	 YuviPanda: link?
[19:54:40] <YuviPanda>	 andrewbogott: https://gerrit.wikimedia.org/r/#/c/253807/
[19:55:32] <andrewbogott>	 YuviPanda: that should be fine… I’ll merge as soon as chasemp is done with holmium
[19:56:02] <chasemp>	 bit of turbulence started on labvirt1008 and some vars didn't translate
[19:56:16] <chasemp>	 so teh nova.conf there is stale atm...looking at what to do 
[19:56:49] <chasemp>	 andrewbogott: will this cause immediate concern? (labvirt1008?)
[19:57:01] <andrewbogott>	 maybe :)  It depends on what changed
[19:57:05] <andrewbogott>	 if puppet is just broken, that’s fine
[19:57:11] <chasemp>	 it def changed the config
[19:57:18] <andrewbogott>	 1011 is a good place to start since it doesn’t host any actual vms
[19:57:20] <andrewbogott>	 other than my tests
[19:57:23] <chasemp>	 I have an idea of one thing 
[19:57:29] <chasemp>	 ah well, I should have asked first
[19:57:53] <andrewbogott>	 If you want to paste the diff I can have a look… but probably it’s fine :)
[19:58:30] <chasemp>	 ok yeah one of them is I didn't translate the keystone vs keystoneconfig to the private repo
[19:58:31] <chasemp>	 fixing taht now
[19:58:37] <chasemp>	 but fyi this is inflight
[19:59:19] <andrewbogott>	 ok, I don’t think the virt nodes talk to keystone at all, so probably harmless
[19:59:50] <chasemp>	 in etc/nova/api-paste.ini
[20:07:23] <multichill>	 YuviPanda: Did you give both wdq boxes a kick http://wdq.wmflabs.org/stats is now even worse :-(
[20:07:40] <YuviPanda>	 I did multichill
[20:07:55] <multichill>	 Weird, it went to the 17th and now back to the 16th
[20:08:08] <YuviPanda>	 I can do that again but I don't know if I can keep doing that every day :(
[20:08:12] <YuviPanda>	 multichill: let me kick it again now
[20:08:48] <YuviPanda>	 multichill: is there a wdq syntax to wdqs translator?
[20:08:59] <multichill>	 Not yet afaik.
[20:09:03] <YuviPanda>	 !log wdq-mm bounce wdq-mm on wdq-mm-01
[20:09:09] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wdq-mm/SAL, Master
[20:09:10] <multichill>	 I need it for the Listeria bot
[20:09:23] <multichill>	 I make reports like https://www.wikidata.org/wiki/User:Multichill/Paintings_check_creator
[20:09:34] <YuviPanda>	 I'll bounce the other host in a bit
[20:09:45] <multichill>	 https://www.wikidata.org/w/index.php?title=User:Multichill/Paintings_check_creator&action=history <- looks like it did update
[20:10:15] <multichill>	 YuviPanda: Now 502
[20:11:05] <YuviPanda>	 multichill: now?
[20:11:28] <multichill>	 Seems to be loading something....
[20:12:01] <chasemp>	 andrewbogott: I think https://gerrit.wikimedia.org/r/#/c/254187/ fixes the stuff I see
[20:17:47] <wikibugs>	 6Labs, 10Tool-Labs: toolsbeta: set up puppet-compiler / temporary-apply - https://phabricator.wikimedia.org/T97081#1818438 (10valhallasw) OK, so a bit more on puppet compiler. I tried adding a fake 'tools bastion host' to site.pp to run puppet-compiler on that:  ``` node 'bastion.toollabs.compiler' {     inclu...
[20:19:07] <andrewbogott>	 chasemp: fixed?
[20:19:29] <chasemp>	 all except hte admin_token in the /etc/nova/api-paste.ini file
[20:19:37] <chasemp>	 I think maybe the hieradata stuff gets weird in private
[20:19:42] <chasemp>	 I'm looking at the end state and why it's not picking it up
[20:21:28] <multichill>	 YuviPanda: I think it's now completely broken :-(
[20:22:19] <YuviPanda>	 multichill: it's still reading from a file
[20:22:22] <YuviPanda>	 I've no idea how long that'll take
[20:22:29] <multichill>	 Both hosts?
[20:22:34] <YuviPanda>	 I think this is the last time I'm going to touch it :(
[20:22:40] <multichill>	 Don't be scared
[20:22:40] <YuviPanda>	 I don't understand the code at all
[20:23:07] <YuviPanda>	 nah, not 'scared' as much as 'overwhelmed by so many different things I'm technically responsible for'
[20:23:28] <YuviPanda>	 'responsible'
[20:24:17] <YuviPanda>	 multichill: it is too reading from something and I've no idea why - I didn't even restart it
[20:25:20] <valhallasw`cloud>	 YuviPanda: bah, of course puppetcompiler also needs hiera data from somewhere, so this is never going to work like this :/
[20:27:38] <chasemp>	 andrewbogott: ran noop on labvirt1009 so I'm doing the labvirts now
[20:27:47] <andrewbogott>	 cool
[20:31:52] <multichill>	 YuviPanda: Maybe still from yesterday?
[20:32:09] <YuviPanda>	 multichill: that's possible
[20:32:16] <YuviPanda>	 and if so then this is all screwed I guess?
[20:32:21] <multichill>	 Do you monitor the real servers and the virtual load balanced server?
[20:32:24] <YuviPanda>	 if it takes more than a few days...
[20:32:55] <YuviPanda>	 nope
[20:33:03] <YuviPanda>	 I setup an autorestarter in the beginning
[20:33:22] <YuviPanda>	 but that didn't work out quite well
[20:33:37] <YuviPanda>	 and labs still doesn't really have any good monitoring setup due to lack of stored config
[20:34:22] <chasemp>	 it's 5th on teh agends to solve in 2017 tho :D
[20:34:28] <chasemp>	 but seriously it's on my mind
[20:35:53] <multichill>	 That's a shame YuviPanda. I now even have both are sms boxes and the VIP for that in monitoring. Not sure how to notify the 24x7 of a problem if the VIP is down though ;-)
[20:36:25] <multichill>	 YuviPanda: Maybe we can setup a custom 502 to point people to the sparql service? That's the least we can do.....
[20:36:42] <YuviPanda>	 probably can
[20:36:50] <YuviPanda>	 if you write some text I can put it in
[20:41:00] <multichill>	 YuviPanda: "<big>502 WikiDataQuery Down</big><br />Looks like WikiDataQuery is down. We're sorry about that! You might want to try https://query.wikidata.org/" ?
[20:44:11] <chasemp>	 andrewbogott: there is a weird puppet error on labcontrol2001
[20:44:21] <chasemp>	 I imagine it's related but maybe not...and not sure how
[20:44:23] <YuviPanda>	 multichill: yeah, looks good I'll do that in a bit (doing some interview prep)
[20:44:24] <chasemp>	 mind looking at it?
[20:44:28] <andrewbogott>	 ok, looking
[20:44:28] <YuviPanda>	 so man yinterviews
[20:44:30] <YuviPanda>	 valhallasw`cloud: ugh yeah...
[20:44:58] <YuviPanda>	 valhallasw`cloud: around to do fastapt or already bedtime?
[20:45:18] <valhallasw`cloud>	 YuviPanda: fixing up diamond, but should be done in 10 mins or so
[20:46:08] <YuviPanda>	 oh ok
[20:46:18] <andrewbogott>	 chasemp: looks to me like it’s trying to ipresolve(“”)
[20:46:29] <chasemp>	 hm
[20:46:34] <chasemp>	 what value is meant to be there?
[20:47:07] <andrewbogott>	 I haven’t found where it’s happening yet
[20:47:10] <chasemp>	 pre-resolve I mean
[20:47:11] <chasemp>	 k
[20:47:46] <chasemp>	 it must be something defined in eqiad but not codfw then
[20:47:55] <andrewbogott>	 @remote_cert_cleaner
[20:48:08] <valhallasw`cloud>	 YuviPanda: ok, am here now
[20:49:11] <andrewbogott>	 chasemp: the hiera value is labs_certmanager_hostname
[20:49:15] <chasemp>	 andrewbogott: remote_cert_cleaner => hiera('labs_certmanager_hostname'), yeah...
[20:49:18] <chasemp>	 seems to exist though
[20:49:23] <andrewbogott>	 yeah
[20:49:52] <chasemp>	 ok I'm running through puppet on eqiad hosts and I"ll circle back
[20:50:00] <chasemp>	 not sure why that would have changed 
[20:50:22] <valhallasw`cloud>	 YuviPanda: I have an idea. We can make a fastapt role!
[20:50:31] <valhallasw`cloud>	 and apply it via wikitech initially
[20:50:36] <valhallasw`cloud>	 or hiera, still
[20:50:38] <valhallasw`cloud>	 maybe hiera :P
[20:50:52] <YuviPanda>	 we can make a $provider param
[20:50:54] <YuviPanda>	 to exec_environ
[20:51:00] <YuviPanda>	 or something
[20:51:05] <YuviPanda>	 and set that to fastapt / just apt
[20:51:07] <YuviPanda>	 and test
[20:52:01] <valhallasw`cloud>	 or just to toollabs?
[20:52:15] <valhallasw`cloud>	 if we want to do per-environ, we should probably take dev_environ instead
[20:52:26] <YuviPanda>	 valhallasw`cloud: hmm, if we put params in toollabs we can't kill the inheritance eventually
[20:52:36] <valhallasw`cloud>	 doesn't need inheritance
[20:52:47] <valhallasw`cloud>	 it's a global change, not just on a single class
[20:52:49] <YuviPanda>	 actually we can since you can just do hiera('toollabs::package_provider')
[20:53:45] <YuviPanda>	 valhallasw`cloud: but yeah, they all work since we can: isolate it to just toollabs and flip it on and off via hiera
[20:53:49] <YuviPanda>	 valhallasw`cloud: wanna make that change? :D
[20:53:51] <valhallasw`cloud>	 yep
[20:54:13] <andrewbogott>	 chasemp: this should fix it:  https://gerrit.wikimedia.org/r/#/c/254289/
[20:54:13] <YuviPanda>	 \o/
[20:54:39] <chasemp>	 andrewbogott: sweet
[20:54:40] <valhallasw`cloud>	 YuviPanda: eh, actually
[20:55:01] <andrewbogott>	 chasemp: it’s kind of dumb that that’s needed, but it’s easier than refactoring down the bottom.  Want me to merge it?
[20:55:02] <valhallasw`cloud>	 YuviPanda: weren't we juts going to kill the require => 'latest' for apt-get dist-upgrade?
[20:55:17] <valhallasw`cloud>	 ensure => 'latest'
[20:55:27] <valhallasw`cloud>	 and if we just ensure => 'present', puppet will be quick as well!
[20:55:38] <chasemp>	 andrewbogott: sure
[20:55:46] <chasemp>	 we can circle back (in fact I'm sure we will)
[20:56:58] <YuviPanda>	 valhallasw`cloud: wait
[20:57:04] <YuviPanda>	 valhallasw`cloud: we already have ensure => present now
[20:57:08] <YuviPanda>	 valhallasw`cloud: because we moved to require_package
[20:57:12] <valhallasw`cloud>	 not everything
[20:57:14] <valhallasw`cloud>	 just python
[20:57:16] <YuviPanda>	 oh
[20:57:18] <YuviPanda>	 ofc
[20:57:21] <YuviPanda>	 valhallasw`cloud: well let's try that I guess
[20:57:21] <valhallasw`cloud>	 buuut
[20:57:28] <valhallasw`cloud>	 there is a patch from me waiting
[20:57:32] <valhallasw`cloud>	 needs rebase probably
[20:57:37] <YuviPanda>	 where?
[20:57:48] <valhallasw`cloud>	 we couldn't use it because we still needed .system/deb-*, but we have aptly now
[20:58:01] <valhallasw`cloud>	 https://gerrit.wikimedia.org/r/#/c/236616/
[20:58:17] <YuviPanda>	 https://gerrit.wikimedia.org/r/#/c/236616/
[20:58:20] <YuviPanda>	 hah
[20:58:58] <YuviPanda>	 valhallasw`cloud: hmm I think we should keep them for things like tools-webservice
[20:59:02] <YuviPanda>	 and our 'infrastructure' packages
[20:59:06] <valhallasw`cloud>	 why?
[20:59:17] <valhallasw`cloud>	 or rather
[20:59:24] <valhallasw`cloud>	 the dist-upgrade should run on every puppet run
[20:59:41] <valhallasw`cloud>	 apt-get update, apt-get dist-upgrade, puppet agent -tv
[21:01:19] <YuviPanda>	 hmmmm
[21:01:39] <YuviPanda>	 dist-upgrade on every puppet run sounds... idk, dangerous
[21:01:53] <valhallasw`cloud>	 why?
[21:01:55] <YuviPanda>	 also there's the debian specific unattended upgrades system we can setup
[21:01:57] <YuviPanda>	 we already use it
[21:02:03] <YuviPanda>	 for security updates
[21:02:08] <YuviPanda>	 and can just turn it on for everything else too
[21:02:17] <valhallasw`cloud>	 yeah, but then it won't run in tandem with puppet
[21:02:21] <YuviPanda>	 yeah but that's ok
[21:02:41] <valhallasw`cloud>	 also, we effectively already do dist-upgrade on every puppet run
[21:02:47] <valhallasw`cloud>	 just not on unpuppetized packages
[21:03:05] <YuviPanda>	 how so?
[21:03:13] <YuviPanda>	 let me read up on what exactly dist-upgrade does :D
[21:03:22] <YuviPanda>	 and how it is dfiferent from just upgrade
[21:03:28] <valhallasw`cloud>	 because of the ensure => 'latest'
[21:03:34] <valhallasw`cloud>	 yeah we probably want upgrade instead
[21:04:31] <valhallasw`cloud>	 apt-get upgrade is restricted to the case where packages are to be replaced by newer versions, but no package needs to be added or removed.
[21:05:51] <YuviPanda>	 ah]
[21:05:57] <YuviPanda>	 well that seems like what we want
[21:07:01] <valhallasw`cloud>	 I'll hack up a patch
[21:07:05] <valhallasw`cloud>	 hiera-configurable!
[21:07:28] <YuviPanda>	 valhallasw`cloud: ok, so: 1. move to ensure => present for all packages and 2. either do unattended upgrades or do apt-get upgrade (I prefer former mostly because apt-get upgrade will never fly in production and we need a realm branch and ugh)
[21:07:45] <valhallasw`cloud>	 realm branch? why?
[21:08:18] <YuviPanda>	 'run apt-get upgrade before puppet runs' -> 'hack the puppet-run script'
[21:08:54] <andrewbogott>	 chasemp: now that puppet will actually run, here’s the diff:  https://dpaste.de/eLhq
[21:08:56] <valhallasw`cloud>	 yes. puppet erb templates, yo.
[21:09:06] <andrewbogott>	 Want me to pick over that or would you like to?
[21:09:25] <YuviPanda>	 valhallasw`cloud: yeah, let's not do that :)
[21:09:26] <chasemp>	 andrewbogott: oh right the stuff I had to port from common as $::site doesn't populate
[21:09:31] <chasemp>	 I got it
[21:09:34] <valhallasw`cloud>	 YuviPanda: ...?
[21:09:37] <valhallasw`cloud>	 we do that for *everything*
[21:09:50] <YuviPanda>	 well, let's not put an apt-get upgrade on every puppet run, actually.
[21:09:54] <YuviPanda>	 let me collect thoughts
[21:10:24] <YuviPanda>	 ok
[21:10:34] <YuviPanda>	 so multiple reasons:
[21:11:08] <YuviPanda>	 1. debian's unattended upgrade is a thing that does basically exactly what we want (assuming what we want is 'all packages must be up to date')
[21:11:46] <valhallasw`cloud>	 1. no it does not, because what we want is 'all packages must be up to date *on a ~20 minute schedule* with messages if something breaks'
[21:11:53] <YuviPanda>	 hmmmm
[21:12:05] <YuviPanda>	 do we really care about the 20 minute schedule?
[21:12:33] <valhallasw`cloud>	 at least for infra upgrades we do
[21:12:36] <YuviPanda>	 also I wonder if apt-get upgrade also does things like upgrade kernels
[21:12:50] <YuviPanda>	 and basically *everything* and I dunno if we want that
[21:13:01] <valhallasw`cloud>	 I'm pretty sure we get new kernels as security updates as well
[21:13:22] <valhallasw`cloud>	 and keep in mind all new hosts automatically get new packages as well
[21:13:59] <chasemp>	 andrewbogott: question
[21:14:06] <chasemp>	 dhcp_domain=${::site}.wmflabs in codfw...should still technically be eqiad right
[21:14:15] <chasemp>	 as labcontrol2001 is the bckup for eqiad
[21:14:17] <YuviPanda>	 can I say that running apt-get upgrade unconditionally makes me feel uneasy and I am not sure why? it could be cargo cult
[21:14:25] <valhallasw`cloud>	 YuviPanda: yes, you can
[21:14:36] <YuviPanda>	 let me see what apt-get upgrade would do on tools-bastion-01
[21:15:10] <Coren>	 YuviPanda: It's delicate; I've seen it fill up /boot but that was years ago.  My understanding is that it's been trustworthy for quite some time.
[21:15:17] <andrewbogott>	 chasemp: I probably need to destroy labcontrol2001 because it’s just confusing.  I don’t think there’s any point in special-casing that setting for now
[21:15:44] <chasemp>	 andrewbogott: your call man, want to leave it for now?
[21:15:53] <chasemp>	 needs looking into before it could handle anything anyways idk
[21:16:43] <YuviPanda>	 valhallasw`cloud: ok, so I can't actually think of any objections to it outside of (just blindly upgrading everything feels wrong), but I also feel uneasy doing it without asking moritz or paravoid or godog (apt-get upgrade)
[21:16:54] <valhallasw`cloud>	 YuviPanda: sure. 
[21:17:00] <YuviPanda>	 one of the things I've heard before is 'what if there's a broken ssh package?!'
[21:17:01] <valhallasw`cloud>	 I have a patch thta should be in -ops in a few mins
[21:17:15] <YuviPanda>	 although I think the answer to that is 'really?'
[21:17:31] <valhallasw`cloud>	 YuviPanda: how did you install the new ssh version?
[21:17:32] <YuviPanda>	 followed by 'it will unbreak when the fixed package rolls out, right?'
[21:17:43] <YuviPanda>	 valhallasw`cloud: we switched it to => latest for labs, and salt for prod
[21:17:57] <YuviPanda>	 so that argument came up when folks wanted to revert that => latest
[21:18:01] <valhallasw`cloud>	 ah
[21:18:12] <valhallasw`cloud>	 buuut
[21:18:29] <valhallasw`cloud>	 maybe we should have an extra ssh server on aptly? :P
[21:18:32] <valhallasw`cloud>	 dunno.
[21:18:54] <YuviPanda>	 also we have a non-redundant aptly server :(
[21:21:47] <YuviPanda>	 so many things to do :(
[21:21:51] <valhallasw`cloud>	 yup
[21:22:02] <YuviPanda>	 valhallasw`cloud: yeah, so I'll wait for your patch and put these comments there and poke the debian folks.
[21:26:14] <chasemp>	 andrewbogott: https://gerrit.wikimedia.org/r/#/c/254297/
[21:45:54] <wikibugs>	 6Labs: Add a note to wdq-mm error page about query.wikidata.org - https://phabricator.wikimedia.org/T119123#1818689 (10yuvipanda) 3NEW a:3yuvipanda
[22:20:58] * YuviPanda bugs andrewbogott about the designate patch
[22:21:29] <andrewbogott>	 chasemp: are you standing clear enough that we can clearly assign blame if I merge that patch?
[22:21:42] * andrewbogott makes sure that instance creation still works
[22:21:45] <chasemp>	 yep no changes from me in the near ftuure
[22:21:47] <chasemp>	 future even
[22:21:55] <chasemp>	 staging my bit for teh morning at this point
[22:33:54] <andrewbogott>	 YuviPanda: no dice, because my code is lame.
[22:33:55] <andrewbogott>	 I will fix...
[22:34:05] <YuviPanda>	 oh :(
[22:34:07] <YuviPanda>	 yay! :)
[22:38:31] <andrewbogott>	 YuviPanda: https://gerrit.wikimedia.org/r/#/c/254312/
[22:39:00] <andrewbogott>	 See, I knew that 0-is-a-special-case bug would bite us, I was just wrong about when
[22:39:20] <YuviPanda>	 andrewbogott: it already bit us for shinkengen and I fixed it
[22:39:29] <YuviPanda>	 andrewbogott: also this project should move to operations/software/something I guess
[22:39:40] <andrewbogott>	 probably
[22:43:47] <YuviPanda>	 andrewbogott: I'm going to go afk for food for a bit, anything you want me to do before that?
[22:43:59] <andrewbogott>	 nope, I think everything should be fixed
[22:45:38] <spagewmf>	 YuviPanda or any CTO candidate: https://www.mediawiki.org/wiki/Software_bundles#Extension_packages has a broken link "Unofficial bundles of the extensions in the Wikimedia SVN repository can sometimes be found on the toolserver." What's the replacement for this?
[22:46:13] <YuviPanda>	 I've no idea. legoktm might know
[22:47:23] <legoktm>	 spagewmf: ~daniel is DanielK I think, but I don't believe anyone has recreated extension bundles. There's an open task for ExtensionDistributor to do it IIRC...
[22:47:38] <andrewbogott>	 YuviPanda: everything looks good, I created a couple of instances without problems.
[22:47:43] <andrewbogott>	 I’m about to go in a few minutes too
[22:47:44] <YuviPanda>	 andrewbogott: \o/ awesome
[22:47:54] <YuviPanda>	 andrewbogott: ok! I guess that kills role::labs::instance and realm
[22:48:02] <YuviPanda>	 I shall hunt and kill other ldap variables next week.
[22:48:08] * YuviPanda devotes rest of this week to kubernetes
[22:48:36] <spagewmf>	 legoktm: thanks as always, O presumptive CTO. I'll replace with link to ExtensionDistributor
[22:50:00] <SMalyshev>	 YuviPanda: hey! do you have a minute for a quick (hopefully) question?
[22:50:27] <YuviPanda>	 SMalyshev: hey
[22:50:29] <YuviPanda>	 go on
[22:50:35] <YuviPanda>	 I gotta go in about 5mins but can do till then
[22:58:25] <YuviPanda>	 Coren: are you around?