[00:03:15] <bd808>	 !log commtech Added MusikAnimal as project admin
[00:03:18] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Commtech/SAL, Master
[00:03:55] <musikanimal>	 cool!
[00:04:05] <musikanimal>	 does that mean I can block you
[00:08:51] <Krenair>	 means you can remove him as a project admin, but that's about it. also he can add himself back with his permissions iirc
[00:10:34] <Krenair>	 more importantly you can create, edit and delete instances in the project, as well as manage dns and other things related to the project
[00:12:53] <bd808>	 musikanimal: mostly you can help Niharika keep the demo wiki there running :)
[00:17:08] <musikanimal>	 hehe alright
[00:18:15] <musikanimal>	 so we just have a demo wiki on there? how do I get to it
[00:20:04] <Krenair>	 I'd look through horizon's domain and web proxy lists
[00:20:18] <Krenair>	 or log into commtech-1.commtech.eqiad.wmflabs and see if anything is there
[00:25:52] <shinken-wm>	 PROBLEM - SSH on tools-grid-master is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[00:41:52] <yuvipanda>	 ooooho
[00:41:54] <yuvipanda>	 that's not good
[00:44:17] <MaxSem>	 errors dropping
[00:44:38] <yuvipanda>	 the grid master is still working
[00:45:03] <MaxSem>	 I will probably write incident report tomorrow, after I get drunk and get dry
[00:53:56] <wikibugs>	 06Labs: Creating new instances goes into ERROR state - https://phabricator.wikimedia.org/T141966#2538871 (10yuvipanda) 05Open>03Invalid Doesn't anymore, and this ticket doesn't have any useful info, I'll open another if it happens again.
[00:54:32] <wikibugs>	 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Copy labmon data to new SSDs - https://phabricator.wikimedia.org/T137924#2538875 (10yuvipanda) a:05yuvipanda>03None
[00:54:55] <wikibugs>	 06Labs, 10Tool-Labs: Make jsub / qsub default to trusty instances - https://phabricator.wikimedia.org/T94792#2538876 (10yuvipanda) a:05yuvipanda>03None un cookie licking right now
[01:13:09] <Niharika>	 musikanimal: You can see it in...I was gonna say "action" at commtech.wmflabs.org but it seems down. You won't be able to ssh in, I guess. 
[01:13:53] <musikanimal>	 yeah that's what I tried hah
[01:14:55] <Krenair>	 you won't be able to ssh in?
[01:15:06] <Niharika>	 musikanimal: This instance is jinxed. :P It goes down every other day. 
[01:15:19] <Niharika>	 Krenair: Yep, can't ssh in until you reboot it. I just rebooted it. 
[01:15:32] <musikanimal>	 but what about our poor demo wiki users!
[01:15:50] <Krenair>	 huh
[01:16:06] <Krenair>	 anything in console when that happens?
[01:16:06] <Niharika>	 musikanimal: Thankfully there aren't many. 
[01:16:43] <yuvipanda>	 niharika can you let it be next time it happens and ping me?
[01:17:34] <Niharika>	 Krenair: Not really. bd80.8 looked into it before once and thought it might have anything to do with runaway processes. Although this instance is just a plain MW vagrant with nothing weird.  
[01:17:40] <Niharika>	 yuvipanda: Sure!
[01:17:59] <yuvipanda>	 thanks niharika
[01:18:04] <Niharika>	 musikanimal: You should be able to login now (reboot finished).
[01:20:01] <musikanimal>	 kewl
[01:20:26] <musikanimal>	 very slowwwww
[01:26:00] <wikibugs>	 06Labs, 10Labs-Infrastructure: Track labs instances hanging - https://phabricator.wikimedia.org/T141673#2538890 (10yuvipanda)
[01:29:41] <wikibugs>	 06Labs, 10Labs-Infrastructure: Track labs instances hanging - https://phabricator.wikimedia.org/T141673#2538892 (10yuvipanda) tools-grid-master, first trusty instance to be dead! It also saw super spikes in load and iowait before it died, *but* the processes on it are still functioning, with the grid still ope...
[03:30:08] <wikibugs>	 10Tool-Labs-tools-Pageviews: Redirect to massviews if more than 10 articles are entered in URL params - https://phabricator.wikimedia.org/T142325#2538947 (10MusikAnimal) 05Open>03Resolved Done with https://github.com/MusikAnimal/pageviews/releases/tag/2016.08.10T03.24
[04:38:13] <wikibugs>	 10Tool-Labs-tools-Xtools: Xtools API hits error and returns 'maintenance' - https://phabricator.wikimedia.org/T136482#2538997 (10MusikAnimal) The xtools gadget should now be fixed, thanks for figuring that bit out @Alfa80 ! I also updated the links/paths to go to the right place. Is anyone using the API outside...
[05:03:52] <wikibugs>	 06Labs: Don't set instance root passwords if using a local puppetmaster - https://phabricator.wikimedia.org/T142531#2539011 (10yuvipanda) The cleanest way to do this is:  1. Introduce a custom fact that reports on what's the puppetmaster 2. Apply this when that's the labs puppetmaster.
[06:27:11] <shinken-wm>	 PROBLEM - Puppet staleness on tools-exec-1204 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [43200.0]
[06:27:23] <shinken-wm>	 PROBLEM - Puppet staleness on tools-webgrid-lighttpd-1208 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [43200.0]
[06:30:16] <shinken-wm>	 PROBLEM - Puppet staleness on tools-webgrid-lighttpd-1207 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [43200.0]
[06:31:58] <shinken-wm>	 PROBLEM - Puppet staleness on tools-exec-1213 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [43200.0]
[06:32:56] <shinken-wm>	 PROBLEM - Puppet staleness on tools-exec-1211 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [43200.0]
[06:55:16] <shinken-wm>	 PROBLEM - Puppet staleness on tools-proxy-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0]
[07:04:38] <wikibugs>	 10Tool-Labs-tools-Xtools: Xtools API hits error and returns 'maintenance' - https://phabricator.wikimedia.org/T136482#2539103 (10Alfa80) @MusikAnimal Thanks a lot for the official update. I guess by the links you mean the internal links in php for new location of xtools-articleinfo instead of xtools/article-info...
[07:59:51] <grrrit-wm>	 (03PS1) 10Legoktm: Send #commit-message-validator to `#wikimedia-releng` [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/303989 
[08:00:12] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] Send #commit-message-validator to `#wikimedia-releng` [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/303989 (owner: 10Legoktm)
[08:00:29] <grrrit-wm>	 (03Merged) 10jenkins-bot: Send #commit-message-validator to `#wikimedia-releng` [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/303989 (owner: 10Legoktm)
[08:57:09] <grrrit-wm>	 (03CR) 10Lokal Profil: "Would you mind adding some info about what this does? Possibly tie it to a phabricator task where it can be discussed?" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/303933 (owner: 10EdouardHue)
[09:02:09] <grrrit-wm>	 (03CR) 10EdouardHue: "> Would you mind adding some info about what this does? Possibly tie" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/303933 (owner: 10EdouardHue)
[11:06:40] <FNDE>	 hey friends, I have question: how I permit all users on tool labs to access my own database?
[11:07:09] <gifti>	 name it with _p at the end
[11:08:00] <FNDE>	 thats all?! stupid simple ;) 
[11:08:10] <FNDE>	 thank u gifti!
[11:08:15] <gifti>	 np
[11:34:14] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Tbscho was created, changed by Tbscho link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Tbscho edit summary: Created page with "{{Tools Access Request |Justification=Hello Wikimedia Team,  I'm looking forward to work on a pyWikiBot based project to process sister cities form wikipedia to the wikidata p..."
[11:51:38] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Tbscho was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=816900 edit summary: 
[12:13:13] <grrrit-wm>	 (03CR) 10Jean-Frédéric: "Thanks for this Édouard!" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/303933 (owner: 10EdouardHue)
[16:16:52] <grrrit-wm>	 (03Abandoned) 10Paladox: Update some packages [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/302918 (owner: 10Paladox)
[16:39:33] <wikibugs>	 06Labs, 10Labs-Infrastructure: Default source group (security group) allowances do not work post Liberty upgrade - https://phabricator.wikimedia.org/T142165#2540359 (10Andrew)
[16:43:23] <wikibugs>	 06Labs, 10Labs-Infrastructure: Default source group (security group) allowances do not work post Liberty upgrade - https://phabricator.wikimedia.org/T142165#2540417 (10Andrew) The specific failure causing this problem appears to be  https://phabricator.wikimedia.org/P3805  I don't see any such timeout when /re...
[16:57:57] <wikibugs>	 10Tool-Labs-tools-Wikidata-Periodic-Table, 10Wikidata: ptable app is broken again! - https://phabricator.wikimedia.org/T142432#2540518 (10ArthurPSmith) Still broken (at least 3 days now). I can't see the error messages but I tried running my own copy and ran into:  https://lists.wikimedia.org/pipermail/mediawi...
[17:05:54] <wikibugs>	 10Tool-Labs-tools-Wikidata-Periodic-Table, 10Wikidata: ptable app is broken again! - https://phabricator.wikimedia.org/T142432#2540542 (10ArthurPSmith) p:05Triage>03High So I updated to https in my local copy and that definitely fixed the problem. Not sure if @Ricordisamoa is around? I don't have permissio...
[17:08:10] <wikibugs>	 06Labs, 10Labs-Infrastructure: Default source group (security group) allowances do not work post Liberty upgrade - https://phabricator.wikimedia.org/T142165#2540566 (10Andrew) Increasing rpc_response_timeout in nova.conf (section DEFAULT) from 60 to 300 resolves the problem.  That's a pretty stupid fix, but ma...
[17:17:38] <wikibugs>	 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Copy labmon data to new SSDs - https://phabricator.wikimedia.org/T137924#2540602 (10yuvipanda)
[17:17:48] <wikibugs>	 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Copy labmon data to new SSDs - https://phabricator.wikimedia.org/T137924#2540604 (10RobH) 05Open>03Resolved a:03RobH
[17:17:52] <wikibugs>	 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: graphite.wmflabs.org is very slow / flaky - https://phabricator.wikimedia.org/T127957#2540606 (10RobH)
[17:32:34] <grrrit-wm>	 (03PS1) 10Greg Grossmeier: -releng: remove #browser-tests, fix -infra [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/304041 
[17:33:50] <wikibugs>	 06Labs, 10Labs-Infrastructure: Default source group (security group) allowances do not work post Liberty upgrade - https://phabricator.wikimedia.org/T142165#2540686 (10Andrew) Upstream bug:  https://bugs.launchpad.net/nova/+bug/1611871
[17:38:38] <wikibugs>	 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 13Patch-For-Review: Kubernetes worker nodes hanging - https://phabricator.wikimedia.org/T141017#2540713 (10yuvipanda) I'm going to close this just now since T141673 covers the general case, and we moved to direct-lvm.
[17:42:34] <wikibugs>	 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 13Patch-For-Review: Kubernetes worker nodes hanging - https://phabricator.wikimedia.org/T141017#2540771 (10yuvipanda) 05Open>03Resolved
[17:42:36] <wikibugs>	 06Labs, 10Labs-Infrastructure: Default source group (security group) allowances do not work post Liberty upgrade - https://phabricator.wikimedia.org/T142165#2540772 (10Andrew) A timout of 120 seems to work ok, so I'll get that change in place shortly.  Meanwhile, some nova devs (mriedem and dansmith) seem to c...
[17:54:29] <shinken-wm>	 PROBLEM - Puppet run on tools-prometheus-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[18:04:28] <shinken-wm>	 RECOVERY - Puppet run on tools-prometheus-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[18:13:45] <wikibugs>	 06Labs, 10Labs-Infrastructure: Default source group (security group) allowances do not work post Liberty upgrade - https://phabricator.wikimedia.org/T142165#2541001 (10Andrew) There's a candidate patch for this here which seems correct:  https://review.openstack.org/#/c/288548/3
[18:28:23] <wikibugs>	 06Labs: nova-network deprecasted, for real this time - https://phabricator.wikimedia.org/T142615#2541058 (10Andrew)
[18:28:44] <wikibugs>	 06Labs, 13Patch-For-Review: promethium.wikitextexp.eqiad.wmflabs (10.68.16.2, labs baremetal host) has strange DNS A record result, and missing PTR - https://phabricator.wikimedia.org/T139438#2541070 (10AlexMonk-WMF) 05Open>03Resolved Thanks for your help and patience everybody, especially Brandon. I belie...
[18:29:00] <wikibugs>	 06Labs: nova-network deprecated, for real this time, as of Openstack N - https://phabricator.wikimedia.org/T142615#2541072 (10Andrew)
[18:30:51] <shinken-wm>	 PROBLEM - Host ToolLabs is DOWN: PING CRITICAL - Packet loss = 100%
[18:33:28] <andrewbogott>	 um… what the heck is ^ 
[18:33:57] <andrewbogott>	 oh, it's just what I thought it was… but it works for me
[18:34:04] <andrewbogott>	 yuvipanda: monitoring hiccup?
[18:34:32] <yuvipanda>	 not entirely sure.
[18:34:34] <yuvipanda>	 works for me too 
[18:35:52] <yuvipanda>	 andrewbogott interesting, it's because shinken can't hit tools.wmflabs.org
[18:36:03] <andrewbogott>	 I blame service groups
[18:36:05] <yuvipanda>	 neither icmp nor curl
[18:36:12] <yuvipanda>	 andrewbogott service groups?
[18:36:23] <yuvipanda>	 andrewbogott nope, because it resolves to 208.80.155.131
[18:36:33] <yuvipanda>	 uh oh
[18:36:34] <chasemp>	 shoudn't we purged anything suspect at the end of last testing
[18:36:44] <chasemp>	 ?
[18:36:47] <andrewbogott>	 chasemp: yeah, I'll delete my rules and instances shortly
[18:36:51] <andrewbogott>	 yuvipanda: dns?
[18:36:53] <yuvipanda>	 chasemp no, this is unrelated, I think this is related to the dns rewriting of public floating IPs to internal ones
[18:36:57] <yuvipanda>	 and that's not working
[18:37:00] <yuvipanda>	 I can't hit any of them 
[18:37:12] <chasemp>	 hm I merged a fix to this yesterday for krenair
[18:37:13] <yuvipanda>	 krenair didn't one of your changes to this just get merged?
[18:37:20] <andrewbogott>	 well, it's been like 2 days since that broke, so we're due
[18:37:21] <chasemp>	 or reltaed to metaldns anyway
[18:37:26] <yuvipanda>	 ah, wah I see
[18:37:37] <chasemp>	 yuvipanda: so what's exactly not working? example?
[18:37:55] <yuvipanda>	 chasemp i'm on shinken-01
[18:38:05] <yuvipanda>	 chasemp dig tools.wmflabs.org
[18:38:09] <yuvipanda>	 shows the floating IP
[18:38:13] <chasemp>	 should be internal?
[18:38:15] <yuvipanda>	 it should show the 10.x.x.x ip instead
[18:38:18] <yuvipanda>	 yeah
[18:38:35] <chasemp>	 I see the same from bastion
[18:38:35] <chasemp>	 ok
[18:39:11] <chasemp>	 only two things I have doen remotely related are
[18:39:11] <chasemp>	 https://gerrit.wikimedia.org/r/#/c/299903/
[18:39:23] <chasemp>	 and brandon merged https://gerrit.wikimedia.org/r/#/c/303833/
[18:39:40] <chasemp>	 but not really related
[18:39:56] <chasemp>	 also did this just stop working?
[18:40:38] <shinken-wm>	 PROBLEM - Host secgroup-lag-101 is DOWN: CRITICAL - Host Unreachable (10.68.19.205)
[18:40:42] <yuvipanda>	 not entirely sure.
[18:40:45] <yuvipanda>	 we just saw the notice
[18:41:01] <chasemp>	 this fails
[18:41:01] <chasemp>	 dig tools.wmflabs.org @208.80.155.118
[18:41:08] <shinken-wm>	 PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218)
[18:41:27] <chasemp>	 succeeds but is wrong
[18:41:28] <chasemp>	 dig tools.wmflabs.org @208.80.154.20
[18:41:39] <chasemp>	 tools.wmflabs.org.	64	IN	A	208.80.155.131
[18:41:50] <Krenair>	 the labsaliaser script is broken?
[18:42:03] <yuvipanda>	 something along those lines
[18:42:06] <chasemp>	 Krenair: seems that way
[18:42:11] <chasemp>	 possibly among others not sure
[18:42:24] <Krenair>	 check /var/log/syslog for issues with pdns-recursor
[18:42:28] <Krenair>	 I'd look but no access
[18:42:35] <Krenair>	 maybe revert my metaldns change
[18:42:44] <yuvipanda>	 I just rm'd that lua file, no difference
[18:42:50] <yuvipanda>	 (put it back)
[18:42:53] <andrewbogott>	 things like 'secgroup-lag-102' alert are just me cleaning up testing cruft
[18:42:59] <Krenair>	 I'm trying to get away from the computer for dinner
[18:43:18] <yuvipanda>	 chasemp rming the lua file (metaldns) caused pdns-recursor to not respond at all, which is why you didn't see it respond
[18:43:20] <yuvipanda>	 it responds now
[18:43:23] <yuvipanda>	 albeit wrong
[18:43:28] <chasemp>	 yuvipanda: ah
[18:43:40] <chasemp>	 yuvipanda: you would have to comment out of the config file as well
[18:43:48] <Krenair>	 Yes there's another file that tries to load it
[18:43:48] <yuvipanda>	 ah, bam
[18:43:50] <yuvipanda>	 yes
[18:43:50] <chasemp>	 but yes it's responding now just...wrong in that case
[18:43:54] <Krenair>	 Deleting the file = almost certainly going to break things
[18:43:56] <Krenair>	 I would not have done that.
[18:43:58] <yuvipanda>	 I'm going to try that now
[18:44:05] <yuvipanda>	 yeah
[18:44:09] <chasemp>	 yuvipanda: what do you hope will happen?
[18:44:09] <andrewbogott>	 the recursor is sad indeed
[18:44:14] <andrewbogott>	 let me try to restart some things and get them in shape
[18:44:21] <yuvipanda>	 I shouldn't have either, been working with prometheus that loads things based on filenames and implicitly assumed that's true for this
[18:44:58] <chasemp>	 yeah it's just wrong in this case
[18:45:04] <yuvipanda>	 chasemp pdns-recursor lua filtering is weird - I've broken it in the past trying to deal with nxdomain stuff since adding new functions at points seem to change behavior of other functions / what they are called with
[18:45:34] <chasemp>	 andrewbogott: any changes to this that you know of recently?
[18:45:46] <andrewbogott>	 nope
[18:45:51] <Krenair>	 So if you disable puppet, remove the line loading metaldns, and restart pdns-recursor, what happens?
[18:45:54] <yuvipanda>	 ok, removed it properly this time.
[18:46:13] <yuvipanda>	 chasemp krenair it works fine now
[18:46:16] <chasemp>	 I think that's what yuvi did
[18:46:17] <Krenair>	 okay
[18:46:17] <yuvipanda>	 with the metaldns removed
[18:46:18] <chasemp>	 yeah
[18:46:24] <chasemp>	 let's reverst those two patches
[18:46:27] <andrewbogott>	 there's kind of a lot of pdns[20944]: message repeated 26 times: [ TCP Connection Thread died because of network error: Reading data: Connection reset by peer] but iirc that's a red herring
[18:46:28] <Krenair>	 two patches?
[18:46:35] <Krenair>	 what's the second?
[18:46:37] <andrewbogott>	 ah, great, so diagnosed?
[18:46:42] <chasemp>	 https://gerrit.wikimedia.org/r/#/c/299903/
[18:46:42] <chasemp>	 https://gerrit.wikimedia.org/r/#/c/303833/
[18:46:54] <yuvipanda>	 andrewbogott yup
[18:47:03] <andrewbogott>	 shit, up, I just ran puppet on labservices, maybe undid your hack :(
[18:47:16] <yuvipanda>	 andrewbogott i disabled puppet
[18:47:18] * andrewbogott logs out of the labservices box to avoid breaking more
[18:47:19] <Krenair>	 One was merged 28 hours ago
[18:47:23] <yuvipanda>	 ah, right
[18:47:24] <Krenair>	 I wouldn't revert that
[18:47:31] <yuvipanda>	 let me disable again
[18:47:35] <chasemp>	 do we know when it was tested as working?
[18:47:48] <chasemp>	 ah is this from today then
[18:47:49] <chasemp>	 https://gerrit.wikimedia.org/r/#/c/303833/2
[18:47:53] <Krenair>	 yes
[18:47:58] <chasemp>	 that seems like it yeah
[18:48:35] <Krenair>	 I really have to go
[18:48:47] <Krenair>	 is it working again with puppet disabled?
[18:48:51] <yuvipanda>	 yeah
[18:48:58] <chasemp>	 yep
[18:49:02] <yuvipanda>	 andrewbogott what's the other DNS host?
[18:49:05] <andrewbogott>	 seems fine, we can just revert and let Krenair sort it out later right?
[18:49:11] <andrewbogott>	 labservices1001 and labservices1002
[18:49:20] <Krenair>	 Okay, when I'm back I'll look into it. No promises though, I have no access to those prod-labs machines
[18:49:43] <yuvipanda>	 andrewbogott oooh, we renamed them? nice
[18:50:16] <andrewbogott>	 it was a long, slow crawl away from holmium :)
[18:50:43] <shinken-wm>	 RECOVERY - Host ToolLabs is UP: PING OK - Packet loss = 0%, RTA = 0.53 ms
[18:52:12] <chasemp>	 yuvipanda: when it comes to the terribleness that is gerrits new interface you have not exagerated at all
[18:52:19] <chasemp>	 Krenair: no worries we got it
[18:52:41] <paladox>	 chasemp have you seen polygerrit?
[18:52:43] <yuvipanda>	 I think the old interface was slightly less terrible, but we got used to it
[18:52:59] <yuvipanda>	 chasemp running puppet on labservices1002 now to see if that unbreaks it
[18:53:07] <chasemp>	 kk
[18:53:24] <paladox>	 yuvipanda have you seen polygerrit?
[18:53:30] <yuvipanda>	 nope paladox
[18:53:37] <paladox>	 gerrit-review.googlesource.com/?polygerrit=1
[18:53:45] <chasemp>	 kk
[18:53:52] <paladox>	 https://gerrit-review.googlesource.com/?polygerrit=1
[18:53:53] <paladox>	 https://gerrit-review.googlesource.com/?polygerrit=0
[18:54:10] <paladox>	 it is the new gerrit ui there working on
[18:54:20] <yuvipanda>	 ah nice
[18:54:24] <paladox>	 yep
[18:54:35] <yuvipanda>	 much better than current one, but I do think it's fundamentally broken in many ways
[18:54:40] <chasemp>	 yuvipanda: it's fixed
[18:54:44] <paladox>	 Yep
[18:54:54] <yuvipanda>	 chasemp yup, running on 1001 now too
[18:55:00] <chasemp>	 I have no faith in mocks as it's before the wrong ppl have touched things
[18:55:03] <paladox>	 Still needs alot of improvements included adding all the features from the current gerrit gui
[18:55:37] <paladox>	 but that will be the new gerrit ui when ever there finished.
[19:00:13] <paladox>	 Hi does anyone know when we can start creating instances again please>
[19:00:17] <paladox>	 > = ?
[19:07:31] <shinken-wm>	 PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22)
[19:25:54] <jaufrecht>	 phlogiston-1 is hung beyond my ability to reboot from the web interface; could somebody kick it harder?
[19:29:38] <chasemp>	 paladox: if you follow https://phabricator.wikimedia.org/T142165 potentially this week
[19:29:53] <paladox>	 Ok thanks
[19:30:08] <chasemp>	 jaufrecht: I'll try
[19:31:25] <chasemp>	 jaufrecht: it says it's already rebooting
[19:31:46] <jaufrecht>	 it's been that way since last week
[19:32:08] <chasemp>	 ah well that state seems to preempt any command I give it hm
[19:33:06] <jaufrecht>	 this happened before and there was some other way to restart it, maybe by controlling the virtual host?
[19:33:08] <chasemp>	 Cannot 'stop' instance ab9a7a7b-709f-4bb6-9f33-c56942f30ab5 while it is in task_state rebooting (HTTP 409)
[19:33:57] <chasemp>	 jaufrecht: there probably is but I'm not sure how, andrewbogott is at lunch currently and is busy today I know, try pinging us on task?  it may take a bit of looking
[19:34:51] <jaufrecht>	 sure
[19:35:03] <jaufrecht>	 thx
[19:37:20] <shinken-wm>	 PROBLEM - Puppet run on tools-docker-builder-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[19:59:39] <wikibugs>	 06Labs, 10Wikimedia-Mailing-lists: Create temporary test mailman mailing list to test synchronization with https://discourse.wmflabs.org/ - https://phabricator.wikimedia.org/T126547#2016945 (10Dzahn) You have successfully created the mailing list discourse and notification has been sent to the list owner ad@hu...
[20:00:26] <wikibugs>	 10Labs-project-other: Successful pilot of Discourse on https://discourse.wmflabs.org/ as an alternative to wikimedia-l mailinglist - https://phabricator.wikimedia.org/T124690#2541434 (10Dzahn)
[20:00:29] <wikibugs>	 06Labs, 10Wikimedia-Mailing-lists: Create temporary test mailman mailing list to test synchronization with https://discourse.wmflabs.org/ - https://phabricator.wikimedia.org/T126547#2541431 (10Dzahn) 05Open>03Resolved a:03Dzahn @AdHuikeshoven you should have recevied automatic mail from mailman. The list...
[20:01:14] <wikibugs>	 10Labs-project-other: Successful pilot of Discourse on https://discourse.wmflabs.org/ as an alternative to wikimedia-l mailinglist - https://phabricator.wikimedia.org/T124690#1962887 (10Dzahn) You now have the mailman list called "discourse" to test syncing.
[20:12:06] <wikibugs>	 10Labs-project-other, 10MediaWiki-extensions-OAuth: Add OAuth 2.0 support to MediaWiki - https://phabricator.wikimedia.org/T125337#2541459 (10Paladox)
[20:45:43] <andrewbogott>	 jaufrecht: looking now
[20:49:02] <andrewbogott>	 jaufrecht: should be up now
[20:49:14] <andrewbogott>	 A 'hard reboot' might have fixed the problem, it's unclear.  I also did a reset-state which isn't available in the web ui
[20:49:29] <jaufrecht>	 thanks
[20:49:50] <jaufrecht>	 I'm going to disable its jobs so that it doesn't crash again, and wait for new instance creation.
[20:51:41] <wikibugs>	 10Tool-Labs-tools-Pageviews: Script/bot rapidly hitting Pageviews tool - https://phabricator.wikimedia.org/T142607#2541617 (10MusikAnimal)
[20:54:16] <chasemp>	 andrewbogott: ah ok 'nova reset-state uuid'?
[20:54:27] <andrewbogott>	 nova reset-state —active uuid
[20:54:59] <andrewbogott>	 it doesn't do anything except set the state representation to 'active' which means then you can issue any commands you want like 'stop' or 'reboot'
[20:55:07] <chasemp>	 gotcha
[21:03:47] <wikibugs>	 10Tool-Labs-tools-Erwin's-tools: 502 Bad Gateway - https://phabricator.wikimedia.org/T142637#2541662 (10Supernino)
[21:04:24] <wikibugs>	 10Tool-Labs-tools-Erwin's-tools: 502 Bad Gateway - https://phabricator.wikimedia.org/T142637#2541677 (10Supernino) p:05Triage>03High
[21:17:29] <shinken-wm>	 PROBLEM - Host secgroup-lag-1002 is DOWN: CRITICAL - Host Unreachable (10.68.22.15)
[21:19:39] <shinken-wm>	 PROBLEM - Host secgroup-lag-1003 is DOWN: CRITICAL - Host Unreachable (10.68.21.237)
[21:41:49] <thcipriani>	 hrm. Seeing a lot of Forbidden: Quota exceeded for instances: Requested 1, but already used 10 of 10 instances (HTTP 403) in the nodepool logs, but I only see 6 instances when I run nodepool list I'm unclear why that's happening?
[21:42:10] <paladox>	 Oh wait that happened to hashar
[21:42:13] <paladox>	 on sunday
[21:46:11] <wikibugs>	 06Labs, 10Labs-Infrastructure: Default source group (security group) allowances do not update properly - https://phabricator.wikimedia.org/T142165#2541912 (10Andrew)
[22:19:51] <bd808>	 thcipriani: still happening?
[22:20:26] <thcipriani>	 bd808: slowly recovering I think. Kicked nodepool.
[22:21:25] <bd808>	 cool. when you have something like that, I'd recommend pinging an.drew. Sadly we don't have much other deep OpenStack debugging knowledge yet
[22:23:29] <OLAPUTOTECARIOS>	 OLA JEM
[22:23:36] <OLAPUTOTECARIOS>	 TE AMO 💝💝💝
[22:23:40] <OLAPUTOTECARIOS>	 ERES LIND
[22:23:41] <OLAPUTOTECARIOS>	 O
[22:23:44] <OLAPUTOTECARIOS>	 💜💜💝💝
[22:23:50] <OLAPUTOTECARIOS>	 ERES PRECIOSO
[22:24:10] <OLAPUTOTECARIOS>	 ERES LINDO Y PRECIOSO
[22:24:17] <OLAPUTOTECARIOS>	 JEM TE QUIERO 💝💝💝
[22:25:06] <Platonides>	 ping Barras
[22:26:06] <OLAPUTOTECARIOS>	 CALLATE PLATONIDES
[22:26:13] <OLAPUTOTECARIOS>	 NO LLAMES A NADIE O TE.....
[22:26:35] <OLAPUTOTECARIOS>	 ASI Q QUEDATE CALLADO PLATONICOS
[22:27:20] <Platonides>	 yuvipanda: could you expel this troll?
[22:27:38] <OLAPUTOTECARIOS>	 CALLATE PLATONICOS 
[22:27:51] <OLAPUTOTECARIOS>	 TE ORDENO Q NO LLAMES A NADIE.. AKI SE HACE LO Q YO DIGO
[22:27:53] <OLAPUTOTECARIOS>	 OVEDECE
[22:28:00] <Platonides>	 or andrewbogott perhaps
[22:28:04] <yuvipanda>	 uh, it's slightly complicated
[22:28:07] <yuvipanda>	 let me do this
[22:28:07] <mafk>	 or Krenair
[22:28:09] <OLAPUTOTECARIOS>	 CALLATE PLATONIDES
[22:28:14] <Platonides>	 yuvipanda: you do have op :P
[22:28:21] <OLAPUTOTECARIOS>	 YUVIPANDA LARGATE DE AKI
[22:28:30] <OLAPUTOTECARIOS>	 CALLATEEE PLATONIDES.. SUNA ODEN
[22:28:35] <OLAPUTOTECARIOS>	 MIERDA
[22:28:40] <yuvipanda>	 bd808 can you kick? you have op now
[22:28:55] <Platonides>	 he left
[22:29:01] <andrewbogott>	 some people ask 'why is there evil in the world?'  And some people ask 'What can I do to make sure there is more evil in the world?'
[22:29:06] <Platonides>	 :)
[22:29:10] <yuvipanda>	 I gave you op as well
[22:29:20] <yuvipanda>	 I haven't figured out how to do IRC commands properly yet with my current IRC setup
[22:29:21] <Platonides>	 thanks
[22:29:34] <Platonides>	 we should review this channel access list
[22:29:38] <mafk>	 yuvipanda: /ban nick should work
[22:29:43] <mafk>	 generaly
[22:29:57] <Platonides>	 too few people, and many old users
[22:30:36] <Platonides>	 now he's on #wikimedia-mobile 
[22:31:24] <Platonides>	 even less ops ther :(
[22:32:07] <robh>	 preventively banned from -operations ;D
[22:32:18] <Platonides>	 :D
[22:32:40] <bd808>	 is this channel using the shared ban list stuff?
[22:32:51] * bd808 is really an irc n00b
[22:32:53] <Platonides>	 robh: I didn't see any mode change on operations
[22:33:02] <robh>	 i set the flag via chanserv
[22:33:07] <Platonides>	 bd808: yes
[22:33:14] <wikibugs>	 06Labs, 13Patch-For-Review: promethium.wikitextexp.eqiad.wmflabs (10.68.16.2, labs baremetal host) has strange DNS A record result, and missing PTR - https://phabricator.wikimedia.org/T139438#2542101 (10AlexMonk-WMF) 05Resolved>03Open Sigh. It had to be reverted because my patch ran into a pretty nasty got...
[22:33:16] <robh>	 cuz i already had an open window for flag setting on another access reuquest
[22:33:19] <Platonides>	 it is fetching them from #wikimedia-bans
[22:34:10] <Platonides>	 heh, operations has a longer op list 
[22:34:21] <robh>	 i suppose i could have set it via the ban list, not chanserv flags.  yeah -operations now has a very large op list.
[22:34:24] <Platonides>	 I hope he doesn't annoy you there
[22:34:35] <Platonides>	 that's not bad
[22:34:51] <Platonides>	 it's annoying finding out that the trol went to channel X
[22:34:59] <robh>	 we've become pretty liberal with our handing out the ability to kickban folks over the past few months
[22:35:04] <Platonides>	 and only a handful people have op there
[22:35:08] <Platonides>	 because it was never needed
[22:35:49] <Platonides>	 what's the criteria for operations?
[22:36:35] <robh>	 file a task noting what other wikimedia channels you have the right to do it in, the usual
[22:36:39] <robh>	 https://phabricator.wikimedia.org/T142270 is example i just resolved
[22:37:00] <robh>	 handing out +Aiotv wont allow setting of other non ban flags, so seems safe enough
[22:37:10] <robh>	 (so no ability to take over a channel that i can see)
[22:37:44] <robh>	 having a large list of trusted users to kickban trolls just makes channel management easier. 
[22:38:20] <robh>	 Platonides: also the 3 day wait i think, that one was the most recent one i handled personally.
[22:38:31] <robh>	 this one had gone over 3 days before alex pinged me via task
[22:39:24] <Platonides>	 I have recently been getting op on many channels on which I have been for years
[22:39:33] <Platonides>	 "thanks" to this guy
[22:50:10] <shinken-wm>	 PROBLEM - Puppet staleness on tools-k8s-etcd-03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0]
[23:19:29] <grrrit-wm>	 (03CR) 10EdouardHue: "- Thanks for creating the task. Will amend my commit message in time." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/303933 (owner: 10EdouardHue)
[23:20:51] <shinken-wm>	 PROBLEM - Host fuckfuckfuck is DOWN: CRITICAL - Host Unreachable (10.68.22.106)
[23:22:34] <yuvipanda>	 ^ heh
[23:22:46] <paladox>	 LOLOL
[23:23:14] <paladox>	 so funny
[23:23:30] <paladox>	 someone actually named a host called that
[23:24:06] <yuvipanda>	 yeah, 'twas me :)
[23:24:12] <mafk>	 rotfl
[23:24:32] <yuvipanda>	 it was my 3rd instance after the first two failed I think
[23:25:56] <paladox>	 lol
[23:28:42] <greg-g>	 yuvipanda: status of instance creation?
[23:28:53] <greg-g>	 we're still having issues with nodepool creating instances, i believe
[23:29:12] <greg-g>	 also, where is the conversation happening during this diagnosis? it seems to not be in any channel I'm in