[01:43:24] <wikibugs>	 3Wikimedia Labs / 3wikitech-interface: Thumbnails for Commons images are not displayed - 10https://bugzilla.wikimedia.org/72245 (10Tim Landscheidt) 3NEW p:3Unprio s:3normal a:3None On https://wikitech.wikimedia.org/w/index.php?title=User_talk:Automatik&oldid=131482, the thumbnail isn't displayed (htt...
[02:33:53] <icinga-wm>	 PROBLEM - ToolLabs: Low disk space on / on labmon1001 is CRITICAL: CRITICAL: tools.tools-webgrid-01.diskspace.root.byte_avail.value (10.00%)  
[06:10:24] <wikibugs>	 3Wikimedia Labs: wildcard release parameter fails with jsub - 10https://bugzilla.wikimedia.org/72248 (10merl) 3NEW p:3Unprio s:3major a:3None jobs with wildcard release resource parameters are not interpret correctly if i am using jsub instead of qsub  My testfile foo.sh: #$ -l release='*' #$ -w v echo...
[07:47:30] <kart_>	 !log deployment-prep Converted I-000006ad.eqiad.wmflabs to use local puppet & salt masters
[07:47:43] <kart_>	 not really :)
[08:34:49] <closedmouth>	 no morebots?
[08:39:25] <kart_>	 hashar: can you check if I-000006ad.eqiad.wmflabs is okay for local puppet & salt masters? It doesn't get patch I cherry pick at deployment-salt
[08:39:38] <kart_>	 if possible, let me know how to debug :)
[08:40:35] <hashar>	 kart_: morning :D
[08:41:12] <kart_>	 hashar: Good Morning!
[08:41:19] <hashar>	 kart_: in wikitech when looking at the list of instance at https://wikitech.wikimedia.org/wiki/Special:NovaInstance  there is a "puppet status" column
[08:41:42] <hashar>	 if the instance has a puppet status of "OK"  it stills point to the labs puppetmaster instead of the  beta cluster puppet master
[08:42:04] <kart_>	 Noted.
[08:42:22] <hashar>	 does puppet pass on deployment-apertium02   ?
[08:42:55] <hashar>	 when ever you save the configuration of an instance, it is saved (in LDAP I think) instantly
[08:43:06] <hashar>	 so the next run of puppet should reflect the changes made in the web interface
[08:43:31] <hashar>	 another thing is that both puppet and salt uses certificates between the client and servers
[08:43:46] <kart_>	 hashar: it passes, but definately not from deployment-salt.
[08:43:50] <hashar>	 so when switching an instance to use the beta cluster puppet/salt masters,  you have to manually confirm the client keys
[08:44:01] <hashar>	 hehe
[08:44:12] <kart_>	 hashar: yes. I did as mentioned on wikitech page.
[08:44:17] <hashar>	 great
[08:44:21] <kart_>	 (did that fine with cxserver03)
[08:44:23] <hashar>	 so puppet conf is in /etc/puppet/puppet.conf
[08:44:33] <hashar>	 and it shows  agent.server = deployment-salt.eqiad.wmflabs
[08:44:36] <hashar>	 so that seems fine
[08:45:07] <kart_>	 yes
[08:45:24] <hashar>	 what is the patch you cherry picked on the puppet master ?
[08:45:35] <kart_>	 https://gerrit.wikimedia.org/r/#/c/165485/
[08:45:49] <kart_>	 I expect something wrong with it now :)
[08:45:56] <hashar>	 have you applied role::apertium::beta   on the apertium02 instance ?
[08:46:39] <kart_>	 another thing, it doesn't appear in config!
[08:46:43] <hashar>	 hehe
[08:46:56] <kart_>	 so, obviously, patch has issue.
[08:46:58] <hashar>	 so if the role class is not applied on the instance, it is never going to be realized by puppet :D
[08:47:20] <kart_>	 right.
[08:47:44] <hashar>	 in OpenStack manager, you would have to add the puppet class to the list of classes that can be applied on an instance
[08:47:46] <hashar>	 that is done at https://wikitech.wikimedia.org/wiki/Special:NovaPuppetGroup
[08:48:04] <hashar>	 under 'deployment-prep' section there is:  Classes [Add class]
[08:48:37] <hashar>	 that would make "role::apertium::beta" available in the list of classes to apply on an instance
[08:48:46] <hashar>	 then configure the instance and check that newly available role::apertium::beta class
[08:48:48] <hashar>	 run puppet
[08:48:50] <hashar>	 solved :D
[08:48:51] <hashar>	 (hopefully)
[08:50:11] <kart_>	 :)
[08:50:16] <kart_>	 Thanks. Testing.
[08:53:27] <kart_>	 hashar: error on puppet run, I guess I will wait from Alex for confirming all packages availability.
[08:53:46] <kart_>	 hashar: Thanks for helping as usual ;)
[08:53:57] <kart_>	 (NovaPuppetGroup was new learning!)
[08:54:29] <kart_>	 hashar: if you're bored, feel free to review Apertium patch ;)
[08:54:46] <YuviPanda>	 !ping
[08:54:46] <wm-bot>	 !pong
[08:54:49] <YuviPanda>	 ok
[08:58:05] <hashar>	 kart_: make sure Alexandros is aware of the packaging need s:D
[09:02:52] <kart_>	 he is :)
[09:15:15] <YuviPanda>	 !log tools cleaned out .svg files from /tmp on tools-webgrid-01
[09:24:48] <icinga-wm>	 RECOVERY - ToolLabs: Low disk space on / on labmon1001 is OK: OK: All targets OK  
[10:19:40] <icinga-wm>	 PROBLEM - ToolLabs: Low disk space on / on labmon1001 is CRITICAL: CRITICAL: tools.tools-webgrid-01.diskspace.root.byte_avail.value (10.00%)  
[11:42:56] <icinga-wm>	 PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools.diskspace._var.byte_avail.value (10.00%)  
[12:01:04] <icinga-wm>	 RECOVERY - ToolLabs: Low disk space on / on labmon1001 is OK: OK: All targets OK  
[12:17:31] <icinga-wm>	 RECOVERY - ToolLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK  
[12:45:10] <wikibugs>	 3Wikimedia Labs / 3tools: program created by proprietary compiler allowed on labs? - 10https://bugzilla.wikimedia.org/72253 (10Inkowik) 3NEW p:3Unprio s:3normal a:3Marc A. Pelletier Hi, I am using a self-written framework (in https://en.wikipedia.org/wiki/PureBasic) for my bot running on my local mac...
[12:47:07] <wikibugs>	 3Wikimedia Labs / 3tools: program created by proprietary compiler allowed on labs? - 10https://bugzilla.wikimedia.org/72253#c1 (10Yuvi Panda) I think it would be, as long as your code itself is open source. I'll wait for Coren to confirm, however.
[13:04:40] <wikibugs>	 3Wikimedia Labs / 3tools: program created by proprietary compiler allowed on labs? - 10https://bugzilla.wikimedia.org/72253#c2 (10Marc A. Pelletier) Legal was pointed towards this bug, and their opinion on the matter is what we need so I am not going to express an opinion either way.  :-)
[15:07:24] <matanya>	 andrewbogott: hello, lvs question for labs ? :)
[15:07:32] <andrewbogott>	 matanya: what's up?
[15:08:01] <matanya>	 modules/lvs/manifests/configuration.pp refers quite a lot to pmtpa for labs
[15:08:33] <matanya>	 i guess it is moot by now, should i remove the entire ref to tamp/replace with eqiad/don't touch ?
[15:09:35] <andrewbogott>	 matanya: That'd be fine, although we'll need to keep the switching logic in place for codfw.
[15:09:52] <andrewbogott>	 Well… ideally it'll move change to using heira but having the switch there is a good reminder that it needs to switch
[15:10:09] <matanya>	 andrewbogott: that = what opetion ? one ?
[15:10:45] <andrewbogott>	 sorry, I don't understand the question
[15:11:03] <matanya>	 I gave three options with what to do with that config
[15:11:42] <andrewbogott>	 Oh… hm.
[15:11:48] <andrewbogott>	 I guess don't touch, since that's easiest :)
[15:12:11] <andrewbogott>	 I'm puzzled that the eqiad option for a lot of these settings is 'undef'.  Does that mean that a lot of that config isn't used at all anymore?
[15:14:16] <YuviPanda>	 andrewbogott: can you create a gerrit project for me when you have the time? operations/software/shinkengen
[15:14:47] <andrewbogott>	 YuviPanda: with an empty commit, or are you going to import?
[15:14:56] <YuviPanda>	 I'm going to import
[15:15:40] <andrewbogott>	 ok.  And… 'rights inherited from'?
[15:16:18] <matanya>	 andrewbogott: i'll leave it for now. thanks!
[15:17:00] <andrewbogott>	 YuviPanda: rights?
[15:17:09] <YuviPanda>	 andrewbogott: ah, hmm. I'd say operations/puppet
[15:17:14] <YuviPanda>	 andrewbogott: but I can't merge then :|
[15:17:26] <andrewbogott>	 yeah, I almost did that and then though, wait, that's not so good for yuvi
[15:17:32] <YuviPanda>	 andrewbogott: can we make it ops/puppet but also add me?
[15:17:38] <andrewbogott>	 Yeah, i think so
[15:17:47] <YuviPanda>	 cool, can you do that?
[15:17:58] <YuviPanda>	 two more weeks!
[15:18:35] <andrewbogott>	 Oh, dammit, except that's done by group...
[15:18:57] <YuviPanda>	 well, just add mediawiki/core as well? we can remove it later...
[15:19:54] <andrewbogott>	 ok… mediawiki has push and submit.  I think that's what you need to import...
[15:19:59] <YuviPanda>	 ok..
[15:20:01] <YuviPanda>	 let me tr
[15:20:01] <YuviPanda>	 y
[15:21:52] <YuviPanda>	 andrewbogott:  ! [remote rejected] master -> master (prohibited by Gerrit) :(
[15:22:23] <andrewbogott>	 try again?
[15:22:35] <YuviPanda>	 andrewbogott: same thing
[15:23:03] <andrewbogott>	 you're in group 'mediawiki' right?
[15:23:32] <YuviPanda>	 andrewbogott: yeah, I can merge there..
[15:23:36] <YuviPanda>	 so I should be?
[15:23:42] <YuviPanda>	 Is there any other way to check?
[15:25:04] <andrewbogott>	 ok, one more time?
[15:25:19] <YuviPanda>	 doing
[15:25:56] <YuviPanda>	 andrewbogott: yup, done!
[15:25:57] <YuviPanda>	 thanks
[15:26:07] <andrewbogott>	 cool.  I'm going to remove some of these weirder permissions now
[15:26:28] <YuviPanda>	 andrewbogott: cool
[15:26:39] <andrewbogott>	 you should still have 'submit'
[15:27:34] <YuviPanda>	 trying now
[15:27:47] <YuviPanda>	 andrewbogott: nope, can't +2
[15:27:52] <andrewbogott>	 dammit
[15:28:09] <andrewbogott>	 I guess +2 is different from submit?
[15:28:13] <apergos>	 !log updated salt master on virt1000 to 2014.1.11 (this is labs salt master)
[15:28:15] <YuviPanda>	 yeah
[15:28:16] <YuviPanda>	 probably
[15:29:21] <andrewbogott>	 ok, I think I found +2
[15:29:57] <YuviPanda>	 yay
[15:31:00] <YuviPanda>	 andrewbogott: so I can C+2, but can't V+2 or submit
[15:31:18] <andrewbogott>	 Ah, not using jenkins I suppose
[15:31:24] <YuviPanda>	 yeah
[15:31:26] <YuviPanda>	 not yet
[15:32:00] <andrewbogott>	 should be better now
[15:32:19] <YuviPanda>	 andrewbogott: yay! thanks
[15:46:31] <YuviPanda>	 andrewbogott: wanna merge https://gerrit.wikimedia.org/r/#/c/167591/? Tested, and affects shinken only
[15:57:02] <apergos>	 andrewbogott: what do you think bout the salt minions in the lab instances? should I mass upgrade them or not? it's not required
[15:57:10] <apergos>	 i.e. the master is backwards compat
[15:57:36] <andrewbogott>	 apergos: Let me try on a test instance...
[15:57:46] <andrewbogott>	 It'll be hard to do everywhere in any case, as salt is broken on many many instances
[15:58:28] <apergos>	 make sure it's running precise
[15:58:42] <apergos>	 I hven't pushed the lucid or trusty packages yet, that will be in another 20 mins or so
[15:59:03] <apergos>	 beta is running upgraded on all instances though
[16:03:10] <andrewbogott>	 apergos: so, I think I don't care :)  Probably fine to let it alone unless there are specific benefits we'll get from an upgrade.
[16:03:31] <apergos>	 bug fixes, oh... hmm... there's some grain related stuff
[16:03:39] <apergos>	 but if you'renot using trebuchet in there you won't care either
[16:04:22] <icinga-wm>	 PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools.diskspace._var.byte_avail.value (10.00%)  
[16:05:05] <andrewbogott>	 apergos: if you're feeling super ambitious… I have a report that detects which instances are salty and which aren't.  So the full project would be to upgrade, and to investigate (and maybe fix) those that aren't responding.
[16:05:20] <apergos>	 eeerrrggghh
[16:05:22] <andrewbogott>	 YuviPanda: Can you take care of ^^?  You have root on labmon1001, right?
[16:05:30] <apergos>	 well I would be fine upgrading those that are responsive
[16:05:34] <YuviPanda>	 andrewbogott: yeah, that's not on labmon, it's on tools-webproxy
[16:05:44] <apergos>	 I can do that by a test.ping and shovleling the names in there
[16:05:47] <andrewbogott>	 YuviPanda: ?  how so?
[16:06:02] <andrewbogott>	 apergos: can't you just tell salt "upgrade everything" and what fails, fails?
[16:06:13] <apergos>	 well I could I guess
[16:06:14] <YuviPanda>	 andrewbogott: it checks graphite, just is picked up by icinga from labmon. see the message - tools.<hostname> is specified
[16:06:41] <andrewbogott>	 YuviPanda: oh… that's confusing :)
[16:06:43] <apergos>	 but I'd rather just have failures from non responsive hosts to look at
[16:06:48] <YuviPanda>	 andrewbogott: yeah, shinken should fix things up :)
[16:06:51] <apergos>	 instead of possibly fails from hosts that were working
[16:07:01] <apergos>	 and not know which ones were which
[16:07:31] <andrewbogott>	 apergos: I think you should go ahead.  If you notice any patterns (like, e.g. 'salt is broken on every instance created since August') then please let me know
[16:07:40] <apergos>	 yep
[16:07:51] <apergos>	 ok I'll get to that after I finsih up the prod cluster
[16:08:04] <apergos>	 then I'll have all the debs out there
[16:08:07] <andrewbogott>	 apergos: you'll need to set up a trusty package first, lots of labs trusty instances.
[16:08:14] <andrewbogott>	 ah, yeah, what you said :)
[16:08:25] <apergos>	 I stole these (just like ryan) from the salt ppa
[16:13:05] <YuviPanda>	 BAM! http://shinken.wmflabs.org/host/tools-login
[16:13:09] <YuviPanda>	 yay
[16:13:45] <YuviPanda>	 andrewbogott: https://gerrit.wikimedia.org/r/#/c/167597/ and https://gerrit.wikimedia.org/r/#/c/167600/
[16:13:57] <YuviPanda>	 andrewbogott: already deployed, and things work! (see previous link)
[16:14:54] <andrewbogott>	 YuviPanda: what is my login on shinken?
[16:15:03] <icinga-wm>	 RECOVERY - ToolLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK  
[16:15:05] <andrewbogott>	 oh, nm :)
[16:15:07] <YuviPanda>	 andrewbogott: guest/guest, we don't have working login yet
[16:15:50] <andrewbogott>	 should there be a red line for /var on that page someplace?
[16:16:05] <YuviPanda>	 andrewbogott: I don't have service checks yet.
[16:16:10] <andrewbogott>	 ok
[16:16:28] <YuviPanda>	 andrewbogott: shinkengen generates hosts.cfg now, will need to make it generate services
[16:30:13] <icinga-wm>	 PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools.diskspace._var.byte_avail.value (11.11%)  
[16:30:47] <YuviPanda>	 andrewbogott: ^ is flapping, since it's very close to the threshold when it rotates logs, I think. I'll fix in a while, shinken is close anywa
[16:48:18] <YuviPanda>	 hmm, need to have shinken spam here
[16:48:28] <YuviPanda>	 could either to irccecho or replace it properly...
[16:48:29] <YuviPanda>	 hmm
[16:53:33] <icinga-wm>	 RECOVERY - ToolLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK  
[17:15:28] <icinga-wm>	 PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools.diskspace._var.byte_avail.value (10.00%)  
[17:16:28] <icinga-wm>	 PROBLEM - ToolLabs: Puppet freshness check on labmon1001 is CRITICAL: CRITICAL: tools.tools-webgrid-tomcat.puppetagent.time_since_last_run.value (100.00%) tools.tools-webgrid-04.puppetagent.time_since_last_run.value (100.00%) tools.tools-webgrid-03.puppetagent.time_since_last_run.value (100.00%) tools.tools-webgrid-01.puppetagent.time_since_last_run.value (100.00%)  
[17:16:39] <Coren>	 Hmm.  Odd.
[17:17:54] <Coren>	 Ah.  Manifest error.  Will be corrected when I'm ready for the next patch.
[17:21:09] <icinga-wm>	 RECOVERY - ToolLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK  
[17:22:46] <icinga-wm>	 ACKNOWLEDGEMENT - ToolLabs: Puppet freshness check on labmon1001 is CRITICAL: CRITICAL: tools.tools-webgrid-tomcat.puppetagent.time_since_last_run.value (100.00%) tools.tools-webgrid-04.puppetagent.time_since_last_run.value (100.00%) tools.tools-webgrid-03.puppetagent.time_since_last_run.value (100.00%) tools.tools-webgrid-01.puppetagent.time_since_last_run.value (100.00%) Coren Known issue in the manifest  will be cleaned up by an upcoming
[17:35:58] <icinga-wm>	 PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools.diskspace._var.byte_avail.value (10.00%)  
[17:38:14] <andrewbogott>	 Coren: https://gerrit.wikimedia.org/r/#/c/167623/  <- if this entirely blocks instance creation then I'll raise it back up a bit.  But, something like this might prevent future crashes like virt1005 last week
[17:38:55] <Coren>	 Ah-ha.  So /that/'s what happened to 1005?
[17:39:17] <andrewbogott>	 maybe?
[17:39:33] <andrewbogott>	 OOM seems like the most likely explanation.  And that's the only tool I have to prevent such things :)
[17:40:19] <andrewbogott>	 In other news, this looks so crazy that it almost has to be wrong:  https://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&m=mem_free&s=by+name&c=Virtualization+cluster+eqiad&h=&host_regex=&max_graphs=0&tab=m&vn=&hide-hf=false&sh=1&z=small&hc=4
[17:40:23] <andrewbogott>	 well, related news
[17:40:24] <Coren>	 I always feel a bit uneasy with any overcomitment, personally, but we don't have infinite hardware and budget.
[17:41:22] <Coren>	 I dunno, it looks pretty consistent to me.
[17:41:47] <andrewbogott>	 Coren: for one thing, why is virt1009 red and virt1007 green?
[17:41:56] <andrewbogott>	 But also, really, only 10M free?  That seems… alarming
[17:48:25] <Coren>	 Wait, I must be looking at the wrong thing because I see over 1T free over the set.
[17:49:25] <Coren>	 150-ish on each.
[17:49:50] <icinga-wm>	 RECOVERY - ToolLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK  
[17:53:28] <andrewbogott>	 Coren, if you scroll down and look at the individual graphs...
[17:53:41] <andrewbogott>	 I bet there's a mistake, and some of the axis should be 'G' rather than 'M'
[17:54:00] <andrewbogott>	 well, even that wouldn't explain the colors
[17:54:32] <Coren>	 Well, there's a bug indeed because clicking on any of them leads you to the per-server details and the numbers there are much more sane.
[17:54:44] * YuviPanda shakes head in general direction of ganglia
[17:57:09] <andrewbogott>	 Coren: for example?  I just clicked through several and see the same numbers
[17:57:21] <icinga-wm>	 PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools.diskspace._var.byte_avail.value (10.00%)  
[17:57:50] <Coren>	 https://ganglia.wikimedia.org/latest/?c=Virtualization%20cluster%20eqiad&h=virt1002.eqiad.wmnet&m=mem_free&r=hour&s=by%20name&hc=4&mc=2
[17:58:00] <Coren>	 Shows 168 some G of free
[17:58:35] <Coren>	 Ah, but the *bottom* graph shows the tiny number.
[17:58:44] <andrewbogott>	 Ah, I'm looking at memory graphs, you're looking at disk space I think?
[17:59:00] <andrewbogott>	 Oh, no, I see what you mean
[17:59:24] <andrewbogott>	 wtf
[17:59:27] <Coren>	 It looks like it's calcularing free_memory differently than how it shows memory usage.  Dumb dumb dumb.
[18:02:21] <Coren>	 In fact, that free_memory number seems to have no correlation to the actual numbers; free on virt1002 reports the same as the usage graph.
[18:02:38] <andrewbogott>	 well… that's good news I guess
[18:04:12] <Coren>	 Bad metrics is bad news, but less bad than "omg no ram left!!one!!1"
[18:05:05] <icinga-wm>	 RECOVERY - ToolLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK  
[18:05:11] <apergos>	 is there an easy way (cli) to poll with a list of instanecs and ask 'has this been deleted'?
[18:05:38] <apergos>	 I can do it the slow way, by going to the nove page for the resource but meh
[18:06:53] <apergos>	 I've got 1900 of them to check so I really don't want to do it manually... 
[18:08:18] <andrewbogott>	 apergos: this may help:  https://wikitech.wikimedia.org/wiki/OpenStack#novastats.py
[18:08:28] <andrewbogott>	 Will require a bit of python to customize, but it's pretty straightforward
[18:09:31] <YuviPanda>	 apergos: andrewbogott I *kinda* did something similar yesterday
[18:09:50] <apergos>	 I was considering polling for entries in the archive table for wikitech
[18:10:03] <YuviPanda>	 apergos: https://gerrit.wikimedia.org/r/#/c/166902/ has python code, used to archive deleted instance metrics from graphite
[18:10:08] <andrewbogott>	 apergos: note that the initial stage of that, gathering up the big dict with all instance stats, takes forever to run
[18:10:28] <YuviPanda>	 apergos: I added an API call to wikitech to list current instances, so you can call that and check if the instance you're looking for is there
[18:11:03] <apergos>	 oohhh  get_deleted_instances():
[18:11:07] <apergos>	 that looks like the ticket
[18:11:19] <apergos>	 it can take an hour, I wouldn't care
[18:11:55] <andrewbogott>	 apergos: get_deleted_instances() is in yuvi's code, right?
[18:12:08] <apergos>	 uh huh
[18:12:10] <andrewbogott>	 YuviPanda: might be nice to merge your query stuff with mine, I bet yours is better :)
[18:17:57] <YuviPanda>	 apergos: andrewbogott note that get_deleted_instances looks at *graphite* as well, and works only in that context. You need some list of 'current instances I am tracking', and then you can have a 'current instances on wikitech' and compare to find deleted ones
[18:18:12] <YuviPanda>	 apergos: andrewbogott the other code to fetch them might be more useful :)
[18:18:24] <apergos>	 hm I want to avoid a graphite dependence actually
[18:18:24] <andrewbogott>	 There's also just 'nova list --all-tenants'
[18:18:39] <andrewbogott>	 and then some sed
[18:18:48] <YuviPanda>	 andrewbogott: I guess the wikitech API is just the replacement for nova list
[18:18:50] <apergos>	 I"ll have a look at that output, maybe that's quicker
[18:18:57] <andrewbogott>	 YuviPanda: yeah :)
[18:19:08] <YuviPanda>	 apergos: https://wikitech.wikimedia.org/w/api.php?action=query&list=novainstances&niproject=deployment-prep&niregion=eqiad&format=json is the equivalent output from wikitech.
[18:19:20] <apergos>	 if I get this going it can be turned into a cron job and clean up once a week or so
[18:19:38] <YuviPanda>	 andrewbogott: I still need to have it output: 1. list of images + ids (so I can map those), 2. list of puppet vars + classes per instance. Will dig into the PHP later
[18:19:48] <apergos>	 got it
[18:20:02] <apergos>	 I'm tempted to use wikitech but having a reliance only on two things (nova and puppet) is likely best
[18:20:08] <apergos>	 if I can do it quickly
[18:20:24] <YuviPanda>	 apergos: nova requires some auth as well, I think. andrewbogott?
[18:20:31] <YuviPanda>	 that's why I wrote the wikitech thing rather than depend on nova
[18:20:38] <wikibugs>	 3Wikimedia Labs / 3Infrastructure: mariadb10 s2/s4/s5 unreachable - 10https://bugzilla.wikimedia.org/69144#c4 (10Sean Pringle) 5NEW>3RESO/FIX This hasn't recurred, plus the box has been since upgraded to MariaDB 10.0.14 which is listed as including the MDEV 6455 fix. Worth watching carefully, but no need...
[18:21:20] <andrewbogott>	 YuviPanda: for a one-off on the commandline, https://wikitech.wikimedia.org/wiki/OpenStack#novaenv.sh
[18:21:39] <andrewbogott>	 if apergos needs a recurring reliable process, then… I would have to think about that while not in a meeting :)
[18:22:02] <YuviPanda>	 andrewbogott: pffft, meetings are best places to think :) although maybe I've been going to different kinds of meetings.
[18:22:14] <apergos>	 yeah I'm mostly listening also
[18:23:14] * YuviPanda probably won't make them until Nov
[18:27:47] <apergos>	 ot puppet keys.  salt keys. grrr
[18:27:52] <apergos>	 anyways...
[18:29:25] <apergos>	 how is the instance name derived from anything?  it's going to be annoying if I have to run hostname on each of them via salt to do the mapping
[18:32:29] <andrewbogott>	 apergos: you're asking about correlating the ec2 name with the human-readable name?
[18:32:37] <apergos>	 yep
[18:32:53] <apergos>	 nov seems to list the instance by hostname and a long id that is unrelated to the instance name
[18:33:07] <apergos>	 i-00000220 = dpeloyment-db1  r whatever
[18:33:08] <andrewbogott>	 That's pretty much what I wrote that nova stats tool for.
[18:33:13] <apergos>	 i need to be able to get at that
[18:33:21] <andrewbogott>	 There are actually three ids, the nova id, the ec2 id, the hostname.
[18:33:32] <andrewbogott>	 nova id and ec2 are unique, hostname not necessarily unique.
[18:33:42] <apergos>	 I guess the i-xxx is the ec2 name?
[18:33:45] <andrewbogott>	 yeah
[18:34:04] <apergos>	 guess I'd better look at nova-stats then :-D
[18:34:11] <andrewbogott>	 Maybe
[18:34:19] <YuviPanda>	 (wikitech api lists them too :) )
[18:37:10] <Coren>	 andrewbogott: Oh, that reminds me - if we tweak the images to accept the new_install root key before its first puppet run, we can have the nova rescue mode working.  I've been testing it when mls died and that's the only thing we need to have it work.
[18:38:16] <andrewbogott>	 Coren:  That's probably fine.  A year or so ago we discussed the merits/demerits of including keys in new images.  The disadvantage is it means we can pretty much never revoke access (since many many labs instances have broken puppet)
[18:38:24] <andrewbogott>	 But I don't know if that's a necessary paranoia
[18:38:43] <Coren>	 Well, it only needs to exist until a succesful puppet run so the manifest can remove it.
[18:39:10] <andrewbogott>	 oh?  Then what is it good for?  Maybe I don't understand what 'nova rescue' is
[18:39:15] <andrewbogott>	 It doesn't need the key ongoingly, just to set up?
[18:40:14] <Coren>	 nova rescue creates a new empty image with just the OS, and puts the "real" one on the second virtual device.  Because of keys, puppet can't run in the rescue image (and we wouldn't want it to) but if we can log into it we can then mount and fix the real one.
[18:43:03] <apergos>	 ohh   Coren,  virt1006: known dead?  or ..?  (it's the only host salt knows that doesn't respond in prod)
[18:44:27] <Coren>	 apergos: I know it's not in actual use, though andrewbogott probably can tell you why.
[18:44:59] <andrewbogott>	 virt1006 was misbehaving -- we migrated everything off of it and shut it down and now I'm not sure what the right thing to do is.
[18:45:14] <andrewbogott>	 But, yeah, not in use for the moment.
[18:45:22] <Coren>	 Oh, right, it's that one.
[18:45:51] <apergos>	 ah ha
[18:45:58] <andrewbogott>	 The frustrating thing is that it's really hard to limit the scheduler to particular hosts.  So I'd like to re-image it and keep it in a probationary state...
[18:46:10] <andrewbogott>	 but that feature in nova either doesn't work or I don't understand it :(
[18:46:11] <apergos>	 well if it comes on line, with the same name and install, it will need a manual salt minion upgrade
[18:46:20] <apergos>	 if it's repurposed then there's no worries
[18:46:27] <andrewbogott>	 apergos: if it comes online it'll be with a fresh OS install
[18:46:32] <apergos>	 excellent!
[19:07:13] <YuviPanda>	 !ping
[19:07:13] <wm-bot>	 !pong
[19:15:23] <wikibugs>	 3Wikimedia Labs: wildcard release parameter fails with jsub - 10https://bugzilla.wikimedia.org/72248 (10merl)
[19:15:23] <wikibugs>	 3Tool Labs tools / 3[other]: merl tools (tracking) - 10https://bugzilla.wikimedia.org/67556 (10merl)
[19:34:47] <YuviPanda>	 anyone know what deployment-soa-cache is for? is down
[19:40:40] <russblau>	 Coren: I just sent a message to labs-l with details of possible database corruption in enwiki_p
[19:41:51] <ottomata>	 hm, heya, i created 3 new instances in the services project about an hour ago
[19:42:01] <ottomata>	 and they are still on BUILD (scheduling) Instance state
[19:48:23] <YuviPanda>	 andrewbogott: merge https://gerrit.wikimedia.org/r/#/c/167646/?
[19:48:35] <YuviPanda>	 I've all of toollabs and deployment-prep on shinken now :)
[19:48:48] <YuviPanda>	 now to clean up the generator code a bit, make it into a package, and deploy that with cron...
[19:49:03] <YuviPanda>	 hmm, not cron, I can just trigger it from puppet
[19:49:38] <YuviPanda>	 andrewbogott: ty
[19:53:44] <milimetric>	 is there a known issue about indices on enwiki_p?
[19:54:01] <milimetric>	 i'm seeing queries taking forever that used to complete sub-second
[19:54:05] <milimetric>	 like select rev_user, count(*) from revision_userindex where rev_timestamp between '20141001000000' and '20141002000000' group by rev_user;
[20:06:06] <Betacommand>	 Coren: have you seen https://bugzilla.wikimedia.org/show_bug.cgi?id=72226
[20:07:57] <russblau>	 Betacommand: looks like that's the same issue I reported independently
[20:08:15] <Betacommand>	 russblau: yeah, emailed you about it
[20:08:25] <russblau>	 this really ought to be a "red alert" level bug!
[20:08:37] <Betacommand>	 which prompted me to nag Coren :P
[20:09:01] <Coren>	 Noted, ima poke Sean.
[20:17:55] <ottomata>	 heya, andrewbogott, it looks like new labs instances aren't building
[20:18:04] <andrewbogott>	 ottomata: ok, I will look
[20:18:26] <ottomata>	 3 in the services project aren't building, and i just double checked in the analytics project.  same deal.
[20:18:27] <ottomata>	 thanks
[20:23:39] <russblau>	 Betacommand: at least this is a better response than we used to get back in the good old Toolserver days :-)
[20:25:46] <Coren>	 russblau: Betacommand: Last I heard about this is that there are cases where OOM can cause some "skipped beats" in replication; I know Sean is working on a reliability fix there but I don't know how close he is to a solution.
[20:30:56] <Betacommand>	 Coren: ouch, this a a common issue then
[20:31:33] <Coren>	 Thankfully not that common, OOMs are rare and fairly well guarded against, but we expect there are little inconsistencies that crept in over time because of it.
[20:34:00] <andrewbogott>	 ottomata: reluctantly fixed
[20:34:15] <Coren>	 "reluctantly"?
[20:34:26] <Coren>	 Oh.  Overcommited?
[20:34:50] <russblau>	 Coren: does this require re-initializing the replica to fix it?
[20:34:59] <ottomata>	 thank you andrewbogott, 2/3 active now
[20:35:25] <Coren>	 andrewbogott: I think we need to put "moar powar" higher on our priorities.  I've been keeping an eye on CPU when it's memory I needed to be concerned about.
[20:35:25] <cscott>	 !log deployment-prep updated OCG to version ea10c93aca9bc1cae34f284fd74bb05d4b6a8cc6
[20:35:43] <Merlissimo>	 russblau: i have the same problem and opened a bugreport more than a month ago: https://bugzilla.wikimedia.org/show_bug.cgi?id=70711
[20:35:53] <andrewbogott>	 Coren:  You mean buying new hardware?  I'm waiting on a bid from HP and will order hardware as soon as they respond.
[20:36:02] <andrewbogott>	 ah, should've cc'd you...
[20:36:13] <Coren>	 russblau: I don't know.  I expect that if it turns out to be needed, we'll do the databases in rotation so it shouldn't be causing issues except for 2/3 of the available power.
[20:36:39] <Coren>	 andrewbogott: Ah.  I knew you were looking into it, I didn't know we were that far along in the req process.
[20:37:09] <Coren>	 russblau: (It's easier now that the databases are functionally identical)
[20:42:30] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): no log in deployment-bastion:/data/project/logs from "503 server unavailable" on beta labs - 10https://bugzilla.wikimedia.org/72275 (10spage) 3NEW p:3Unprio s:3normal a:3None I'm getting 503 errors from beta labs when I visit http://en.wikipedia.beta.wmflabs...
[20:42:53] <Merlissimo>	 Coren: why can sge resources defined in script head and not as argument be differently interpreted by qsub and jsub?
[20:45:18] <ottomata>	 andrewbogott: sorry to bug ya but it looks like that 3rd instance is in the same state
[20:45:31] <ottomata>	 also, nfs home not mounting?  ( i don't really care about that right now, just fyi)
[20:45:46] <andrewbogott>	 ottomata: I'd say just trash the third instance and try again
[20:46:37] <andrewbogott>	 ottomata: what project/instance is having trouble with nfs?
[20:46:51] <ottomata>	 services, otto-cass1 and otto-cass2
[20:47:45] <ottomata>	 mount... failed, reason given by server: No such file or directory
[20:48:04] <ottomata>	 that's just via puppet though
[20:48:09] <ottomata>	 haven't actually checked if it is failed
[20:48:12] <ottomata>	 could just be puppet erroring
[20:48:19] <andrewbogott>	 how long has this project existed?  
[20:48:36] <andrewbogott>	 Have other instances in this project access nfs properly?
[20:49:17] <ottomata>	 andrewbogott: don't know, this is gwicke
[20:49:19] <ottomata>	 's project
[20:49:26] <ottomata>	 this is my first time creating instances there
[20:49:34] <ottomata>	 don't worry about it, unless he complains, or I do later
[20:49:40] <ottomata>	 i really don't need the shared NFS there 
[20:49:42] <andrewbogott>	 ok.  That project has shared directories turned off.  So, those volumes can't be found because they don't exist.
[20:49:46] <ottomata>	 ah!
[20:49:47] <ottomata>	 ok!
[20:49:49] <andrewbogott>	 You can turn them on via the project config if you like.
[20:50:03] <ottomata>	 naw, its ok, but we should probably make puppet ok with that if they are turned off
[20:51:00] <andrewbogott>	 that turns out to be hard
[20:51:06] <andrewbogott>	 and it's on by default in new projects.
[20:51:23] <gwicke>	 ottomata: you should have full rights to change anything in the project
[20:51:50] <gwicke>	 I don't need nfs normally, so didn't notice so far
[20:52:14] <Coren>	 Merlissimo: jsub does not support qsub-like script headers - they are considered as normal executable files.  If you need that functionality, you'll need to use qsub itself.
[20:52:18] <ottomata>	 yeah i don't need it either, its ok
[20:52:22] <ottomata>	 i just noticed in the puppet output
[20:53:54] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): no log in deployment-bastion:/data/project/logs from "503 server unavailable" on beta labs - 10https://bugzilla.wikimedia.org/72275 (10Greg Grossmeier) p:5Unprio>3Normal a:3Sam Reed (reedy)
[20:55:24] <Merlissimo>	 Coren: sge script header are respected by jsub in generell. it only overrides some like queue name
[20:56:14] <Coren>	 ... they're really not supposed to since they get invoked with '-b y'.  I'll have to look into it.
[21:56:06] <^d>	 Who can adjust quotas?
[21:56:31] * ^d has some ideas, but doesn't want to if there's someone capable about :)
[21:56:36] <^d>	 *want to ping
[21:58:46] <andrewbogott>	 ^d: what do you need?
[21:59:18] <^d>	 We filed https://bugzilla.wikimedia.org/show_bug.cgi?id=71886 for deployment-prep. 43 instances seems to be bit tight now.
[22:00:21] <andrewbogott>	 I can raise the quota a bit.   Most of this is stalled, though, pending new hardware purchases.
[22:00:49] <^d>	 Yeah, understand. Just a little more wiggle room is what we're needing, not a ton of space for expansion :)
[22:01:16] <andrewbogott>	 Your current quota is 45 -- you've hit that already as well, I take it?
[22:01:47] <^d>	 Ah, we're at 43/45.
[22:01:58] <andrewbogott>	 so -- wiggle room?
[22:02:29] <^d>	 I can probably get my current task done :)
[22:23:47] <^d>	 Did gmond disappear in beta or something?
[22:23:59] <^d>	 puppet's all broken on deployment-elasticNN boxes :\
[22:25:24] <mutante>	 cough.. the 'uptime' of puppet not being broken 
[22:25:42] <mutante>	 i think i saw something about gmond .. wait
[22:54:36] <notconfusing_>	 qdel is still not working properly for me ,and im having to go onto the xgrid and use kill
[22:54:49] <notconfusing_>	 is there a bug open for this, should I file one?