[00:00:22] <Krinkle>	 !log cvn Migrated web services from pmtpa to eqiad. cvn-apache2.pmtpa.wmflabs can be deleted.
[00:00:24] <labs-morebots>	 Logged the message, Master
[00:00:36] <Krinkle>	 cvn-apache4 *
[00:00:40] <Krinkle>	 oh well
[00:01:06] <andrewbogott>	 ok, I erased the proxy… everything still working?
[00:01:36] <Krinkle>	 andrewbogott: Yep, looking good here
[00:01:54] <andrewbogott>	 great.  Thanks for migrating!
[00:02:02] <andrewbogott>	 That's it for you and labs migrations, right?
[00:02:28] <Krinkle>	 Yep, I think so.
[00:02:45] <cajoel>	 out of curiousity -- how far along is the labs move?
[00:16:06] <andrewbogott>	 cajoel: it is this far along:  https://wikitech.wikimedia.org/wiki/Labs_Eqiad_Migration/Progress
[00:16:25] <andrewbogott>	 Which is quite a bit behind, since I was hoping to shut things down tomorrow.
[00:16:55] <cajoel>	 nice
[00:17:01] <cajoel>	 I moved my project over
[00:17:06] <cajoel>	 would you like me to update this page?
[00:18:16] <andrewbogott>	 sure… which project is it?
[00:18:32] <cajoel>	 netflow
[00:18:40] <cajoel>	 current listed as mothballed
[00:18:44] <cajoel>	 actually - moved
[00:18:45] <cajoel>	 fixing
[00:19:27] <andrewbogott>	 cajoel: and, more than that, I would like you to subscribe to labs-l (if you are not already):  https://lists.wikimedia.org/mailman/listinfo/labs-l
[00:20:03] <cajoel>	 you betcha
[00:20:29] <cajoel>	 I got your original note to move...
[00:21:14] <cajoel>	 subscription request sent
[00:21:33] <andrewbogott>	 hanks
[00:21:35] <andrewbogott>	 *t
[00:22:02] <cajoel>	 http://en.wikipedia.org/wiki/Tom_hanks
[00:23:11] <bd808>	 !log deployment-prep `sudo chmod -R a+rwx /data/project/upload7`; We need to get this file permissions thing figured out
[00:23:13] <labs-morebots>	 Logged the message, Master
[00:38:04] <Betacommand>	 Coren: are you around?
[00:38:14] <Coren>	 Betacommand: What's up?
[00:38:29] <Betacommand>	 I cant access global user table on centralauth
[00:38:58] <Betacommand>	 Access denied 
[00:43:40] <Betacommand>	 Coren: any ideas?
[00:44:04] <Coren>	 Hm.  Connected to centralauth_p, right?
[00:44:11] <Coren>	 (and not centralauth)
[00:44:36] <Betacommand>	 I cant connect via that shortcut
[00:44:41] <Betacommand>	 I connected to s7
[00:44:49] <Betacommand>	 switched dbs
[00:44:52] <Betacommand>	 did a show tables
[00:45:09] <Betacommand>	 then did a select * from globaluser LIMIT 5;
[00:45:24] <Betacommand>	 and got a access denied error message
[00:45:52] <Coren>	 'sql centralauth' then that select works for me.  How odd, because as long as the database name ends with _p things should work.
[00:46:31] <Coren>	 (Everyone has select on '%_p')
[00:47:00] <Betacommand>	 Coren: that worked for me
[00:47:20] <Coren>	 You probably connected accidentally to 'centralauth' rather than 'centralauth_p' then.
[00:47:49] <Betacommand>	 Coren: odd
[00:48:29] <Betacommand>	 I used the sql shortcut and connected to both centralauth and centralauth_p and it worked
[00:49:01] <Betacommand>	 The first time I connected to enwiki, changed dbs to meta
[00:49:22] <Betacommand>	 looked up what slice centralauth was on, and another wiki on the same slice
[00:49:46] <Betacommand>	 in this case it was arwiki
[00:49:52] <Betacommand>	 connected to arwiki
[00:50:01] <Betacommand>	 did a use centralauth;
[00:50:09] <Betacommand>	 did a show tables
[00:50:20] <Betacommand>	 and then attempted the select where I got the error
[00:50:24] <Coren>	 Right, you wanted 'use centralauth_p' instead.
[00:50:28] <Betacommand>	 ah
[00:51:00] <Coren>	 But also, you can connect to 'centralauth.labsdb' directly; it's an alias for s7
[00:51:02] <Betacommand>	 Coren: it should be possible to do a sql meta_p
[00:51:39] <Betacommand>	 Coren: I was cheating using the sql shortcut command as I am more familiar with that :)
[00:52:04] <Betacommand>	 For some reason I was trying to connect to centralauthwiki_p
[00:52:05] <Coren>	 'sql centralauth' also works.  :-)
[00:52:11] <Coren>	 Ah!
[00:52:12] <Coren>	 :-)
[00:52:24] <Betacommand>	 thus couldnt connect directly
[00:52:53] <Betacommand>	 CA is the only wiki that doesnt use the *wiki_p format
[00:53:06] <Coren>	 Understandable, pretty much every other DB uses ...wiki
[00:53:10] <Coren>	 Right.
[00:53:23] <Coren>	 I don't think it's the /only/ one, is it?
[00:53:48] <Betacommand>	 there might be a few non-primary wikis
[00:54:11] <Betacommand>	 IE chapter dbs or some such
[00:54:40] <Betacommand>	 Coren: you sould make sql meta_p work
[00:54:42] <Coren>	 Well, also all the xxwithionary.  :-)
[00:54:51] <Coren>	 wiktionary*
[00:54:55] <Betacommand>	 let me double check that
[00:57:50] <Betacommand>	 Coren: guess there are more than I remember
[00:58:09] <Betacommand>	 Coren: any reason that meta_p doesnt work?
[00:59:01] <Coren>	 Because there isn't just the one place you can get meta_p, unlike every other one, and the poor sql script doesn't know what to do with that.  :-)  I'll add some code to recognize and connect to a random shard.
[01:00:15] <Betacommand>	 Coren: makes trying to look something up about a database a pain :P
[01:02:22] <Betacommand>	 Coren: all that to try and help answer a question on a mailing list :)
[01:10:31] <Betacommand>	 Coren: did we have any issues with the mail servers today?
[01:11:10] <Coren>	 Betacommand: Not that I have heard of, though I've unclogged some mail to root.  There was an issue fixed last night however that may be causing some backlog to catch up.
[01:11:51] <Betacommand>	 I got about 15 emails at 1310EST about failed mail delivery 
[01:13:17] <Coren>	 Those are probably bounces that were stuck in the queue.
[01:45:34] <Reedy>	 "That tool might not have a web interface, or it may currently be disabled"
[04:26:56] <superm401>	 andrewbogott_afk, Growth team ones are officially done.  Marked at https://wikitech.wikimedia.org/wiki/Labs_Eqiad_Migration/Progress#Pending_Migration
[04:39:36] <Coren>	 Reedy: Webservice not started.
[05:41:48] <physikerwelt_>	 I justed tested mathoid on the new labs cluster. Locally everything works fine, the proxy did not work correctly
[05:41:50] <physikerwelt_>	 https://wikitech.wikimedia.org/wiki/Nova_Resource:I-000002d5.eqiad.wmflabs
[05:42:25] <physikerwelt_>	 Can I forward to adresses other than 80? The line in the web interface reads mathoid.testme.wmflabs.org --> http://mathoid.eqiad.wmflabs:10042
[08:43:51] <hashar>	 !log deployment-prep Created integration-slave-pbuilder on eqiad to replace pmtpa instance integration-debian-builder
[08:43:54] <labs-morebots>	 Logged the message, Master
[08:48:38] <hashar>	 !log deployment-prep deleting integration-slave-pbuilder. Unneeded (i need a coffee)
[08:48:41] <labs-morebots>	 Logged the message, Master
[10:42:58] <Jelte>	 Coren: It looks like my database (p50380g50947__jeltebot / s51522__jeltebot) for my tool jeltebot is missing from tools-db
[10:47:02] <Jelte>	 I have run finish-migration, but the only output (after confirmation) was ". restoring crontab", nothing about the database
[10:54:18] <hashar>	 !log deployment-prep deleting instance integration-debian-builder . That is breaking all debian-glue jobs.  Will revisit later next week to get pbuilder/cowbuilder set up on the other eqiad slaves
[10:54:21] <labs-morebots>	 Logged the message, Master
[11:13:59] <Reedy>	 http://tools.wmflabs.org/awb
[11:14:03] <Reedy>	 How do I fix the no web service?
[11:17:58] <petan>	 Reedy: login as tool (become awb)
[11:18:05] <petan>	 then type "webservice start"
[11:20:39] * Reedy  adds it to https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Creating_a_tool_web_page

[11:21:14] <Reedy>	 thanks
[11:23:36] <hashar>	 !log integration creating integration-puppetself attended to be a puppet master for the integration project
[11:23:39] <labs-morebots>	 Logged the message, Master
[11:36:54] <hashar>	 !log integration converted integration-puppetmaster as a self puppet master \O/
[11:36:56] <labs-morebots>	 Logged the message, Master
[11:40:39] <petan>	 @notify andrewbogott_afk
[11:40:39] <wm-bot>	 This user is now online in #wikimedia-tech. I'll let you know when they show some activity (talk, etc.)
[11:49:06] <wm-bot>	 !log tools petrb: disabled default vimrc which everybody hates on -login
[11:49:09] <labs-morebots>	 Logged the message, Master
[11:51:35] <wm-bot>	 !log tools petrb: removed content of /var/log/exim4/paniclog
[11:51:37] <labs-morebots>	 Logged the message, Master
[11:53:18] <wm-bot>	 !log tools petrb: did the same on -mail server (removed /var/log/exim4/paniclog) so that we don't get spam every day
[11:53:21] <labs-morebots>	 Logged the message, Master
[13:05:14] <Coren>	 Jelte: Give me a minute, I'll go look for it.
[13:14:12] <Coren>	 Jelte: Not quite sure why it didn't copy over as expected, but I've placed a dump of it in your tool's home.
[13:54:02] <physikerwelt>	 I have a question on  webproxies to forward adresses other than 80 01https://wikitech.wikimedia.org/wiki/Nova_Resource:I-000002d5.eqiad.wmflabs01
[13:54:49] <physikerwelt>	 I created a proxy mathoid.testme.wmflabs.org --> http://mathoid.eqiad.wmflabs:10042
[14:05:18] <scfc_de>	 physikerwelt: I don't think NovaProxy can handle anything but :80.  Can't you make your server listen on :80?  Worst case you could set up iptables rules that translate the port on your machine.  Another solution is of course public IP, but if you already have a web server ...
[14:06:59] <physikerwelt>	 scfc_de: ok I'll change the port
[14:07:56] <scfc_de>	 Oh, sorry, I should have looked at https://wikitech.wikimedia.org/w/index.php?title=Special:NovaProxy first.  Don't you have a field "Instance port" on the "create new proxy" page?  Doesn't that work?
[14:14:33] <physikerwelt>	 Yes, I specified the port
[14:14:51] <physikerwelt>	 but it still did not work
[14:14:52] <scfc_de>	 And what's the error?
[14:15:06] <physikerwelt>	 I don't get an response
[14:18:38] <andrewbogott>	 physikerwelt: I'll take a look at your proxy issue now.  Worst case we can just give that instance a public IP.
[14:19:14] <andrewbogott>	 mutante: can you please update me about the 'wikistats' labs project?
[14:19:30] <scfc_de>	 physikerwelt: Allow your security groups access to that port?
[14:19:49] <physikerwelt>	 ok that might be the problem
[14:19:59] <andrewbogott>	 physikerwelt: it's definitely part of the problem
[14:21:02] <andrewbogott>	 spagewmf: Are you done with your labs migration?  Can I mark the 'editor-engagement' project as finished?
[14:25:15] <physikerwelt>	 andrewbogott: I don't know what to specify in the CIDR ranges filed
[14:25:47] <andrewbogott>	 physikerwelt: 10.0.0.0/8 is safe, that's basically all wmf internal IPs
[14:25:54] <andrewbogott>	 Or if you want to do just eqiad labs, that's, um...
[14:25:56] * andrewbogott  looks it up

[14:26:22] <andrewbogott>	 10.68.16.0/21
[14:28:23] <physikerwelt>	 The problem is that whatever I specify I get the error Failed to add rule.
[14:29:12] <andrewbogott>	 Are you specifying tcp?
[14:29:21] <physikerwelt>	 yes
[14:29:30] <andrewbogott>	 You should also leave 'source group' empty
[14:30:06] <physikerwelt>	 still the same problem
[14:30:58] <andrewbogott>	 ok, hang on, I'll do it...
[14:31:02] <andrewbogott>	 what port #?
[14:31:15] <physikerwelt>	 10042
[14:31:52] <andrewbogott>	 Does that look right to you?
[14:33:05] <physikerwelt>	 yes
[14:33:15] <physikerwelt>	 great
[14:33:17] <physikerwelt>	 everyst
[14:33:22] <physikerwelt>	 everything works now
[14:33:56] <andrewbogott>	 cool
[14:34:04] <andrewbogott>	 Do you have other migration tasks?  Or is that everything?
[14:34:40] <physikerwelt>	 no I decided not to migrate anything
[14:35:51] <physikerwelt>	 I'm just setting a new test instance on the new cluster
[14:36:28] <andrewbogott>	 ok.  So can I mark your project as finished?  Is there anything in pmtpa that you need to salvage?
[14:37:00] <physikerwelt>	 Yes it can be marked as finished
[14:37:18] <andrewbogott>	 perfect, thank you.
[14:37:37] <physikerwelt>	 I have to thank you
[14:40:16] <scfc_de>	 Also no shared storage to migrate?
[14:59:23] <mutante>	 andrewbogott: shutting down in a few minutes
[14:59:25] <mutante>	 on it
[14:59:50] <mutante>	 i think i just need to ask how to switch proxy over..looks
[15:15:16] <mutante>	 how do i change yuviproxy config plz
[15:15:35] <mutante>	 to point my existing foo.wmflabs.org to eqiad instance
[15:22:21] <bd808>	 mutante: With https://wikitech.wikimedia.org/wiki/Special:NovaProxy ? That's how I changed the ones I was using.
[15:24:41] <mutante>	 bd808: thanks, too easy:)
[15:29:53] <andrewbogott>	 hashar, how are things going?
[15:30:07] <hashar>	 andrewbogott: craziness since I woke p
[15:30:08] <hashar>	 up
[15:30:18] <hashar>	 I have ended up installing a puppet master self on the integration project
[15:30:31] <hashar>	 to pile up a bunch of patches.  Got puppet running green a few minutes ago :-]
[15:30:41] <andrewbogott>	 OK, need me to review anything?
[15:30:50] <hashar>	 and after hours and hours, I finally found a very trivial solution to all my problems :-(((
[15:31:15] <hashar>	 since /var/cache/pbuilder doesn't have enough disk space, I should just make it a symlink to /mnt/pbuilder :D
[15:34:08] <hashar>	 anyway
[15:35:03] <hashar>	 andrewbogott: if you have some bandwidth I got a few simple ones 
[15:35:13] <andrewbogott>	 sure, I have a few minutes
[15:35:34] <hashar>	 some craziness to avoid duplicate package definitions in puppet
[15:35:35] <hashar>	 https://gerrit.wikimedia.org/r/#/c/121659/6/modules/elasticsearch/manifests/packages.pp
[15:35:40] <hashar>	 in the elastic search module
[15:35:57] <hashar>	 I am including it on the jenkins slaves which already have curl and openjdk-7-jdk
[15:36:28] <andrewbogott>	 package { 'elasticsearch': } is that really normal syntax?
[15:36:33] <hashar>	 no clue
[15:36:50] <hashar>	 not much different from what was in previously:
[15:36:52] <hashar>	 package { [ 'openjdk-7-jdk', 'elasticsearch', 'curl' ]: }
[15:37:00] <hashar>	 I guess puppet  ensure => present by default
[15:37:38] <hashar>	 got another crazy change that fix a few issues on the labs instance
[15:37:41] <hashar>	 no impact on prod https://gerrit.wikimedia.org/r/#/c/121645/
[15:38:01] <hashar>	 it does some defined() as well and move a few includes around
[15:39:05] <hashar>	 !log deployment-prep deleting instance ntegration-selenium-driver  no more needed.  browsertests jobs should now be runnable on integration-slave1001 and integration-slave1002 (in eqiad)
[15:39:07] <labs-morebots>	 Logged the message, Master
[15:39:15] <hashar>	 grr
[15:39:32] <hashar>	 !log deployment-prep Last log made to wrong project
[15:39:34] <labs-morebots>	 Logged the message, Master
[15:39:39] <hashar>	 !log integration deleting instance ntegration-selenium-driver  no more needed.  browsertests jobs should now be runnable on integration-slave1001 and integration-slave1002 (in eqiad)
[15:39:41] <labs-morebots>	 Logged the message, Master
[15:39:48] <hashar>	 !log integration integration project fully migrated to eqiad \O/
[15:39:50] <labs-morebots>	 Logged the message, Master
[15:40:32] <andrewbogott>	 mutante: any luck with the proxy?
[15:40:52] <andrewbogott>	 hashar, I can mark Integration as finished?
[15:40:56] <hashar>	 yes sir!
[15:41:05] <hashar>	 was going to ask you where the wikipage is
[15:42:01] <andrewbogott>	 It's here:  https://wikitech.wikimedia.org/wiki/Labs_Eqiad_Migration/Progress  but I moved it already
[15:44:00] <andrewbogott>	 hashar, I may switch bastion over later today.  that shouldn't break anything for you but you will have to start fully.qualifying.pmtpa.wmflabs instance names when connecting.
[15:44:18] <andrewbogott>	 Hm, might affect security groups as well I suppose.
[15:44:20] <hashar>	 sounds good to me
[15:44:25] <hashar>	 ahh
[15:44:49] <andrewbogott>	 Anyway, it won't be for 4-5 hours or so, you'll most likely be out for the day by then.
[15:44:51] <hashar>	 I think we are always using either the IP address or the fqdn
[15:45:31] <mutante>	 andrewbogott: yes
[15:45:43] <mutante>	 !log wikistats - deleted instance wikistats-history-01
[15:45:45] <labs-morebots>	 Logged the message, Master
[15:45:45] <andrewbogott>	 mutante: can I mark that project as finished?
[15:45:46] <mutante>	 !log wikistats - deleted instance wikistats-mod
[15:45:48] <labs-morebots>	 Logged the message, Master
[15:46:13] <mutante>	 andrewbogott: one last instance, almost
[15:46:19] <andrewbogott>	 'k
[15:49:01] <hashar>	 andrewbogott: I got a few more changes :D  A directory that needs to be created https://gerrit.wikimedia.org/r/#/c/121663/3/manifests/misc/package-builder.pp
[15:49:41] <hashar>	 it is in a define, I tried to use a file { 'blala' : ensure => directory }  but since the define is called twice or more that causes duplicate File[] resources: /
[15:51:04] <hashar>	 andrewbogott: and a duplicate variable : ( https://gerrit.wikimedia.org/r/#/c/121631/5/manifests/misc/package-builder.pp
[15:51:21] <hashar>	 then a stupid parameter ordering fix https://gerrit.wikimedia.org/r/#/c/121671/2/manifests/misc/package-builder.pp
[15:51:22] <hashar>	 :(
[15:51:26] <hashar>	 sorry made a ton of tiny commits
[15:51:40] <andrewbogott>	 hashar: I don't understand that last one.
[15:51:49] <andrewbogott>	 How is the define getting the build place?
[15:51:51] <andrewbogott>	 without that?
[15:52:02] <^d>	 hashar: Has beta mostly moved to eqiad now?
[15:52:09] * ^d  wants to run maintenance scripts

[15:52:14] <hashar>	 ^d: run them on pmtpa
[15:52:27] <hashar>	 ^d: eqiad has the elastic search populated and a DB but it is not serving traffic yet
[15:53:00] <^d>	 Ah, I'll wait then.
[15:53:12] <hashar>	 andrewbogott: the change basically move the --create parameter at the beginning of the command.  pbuilder requires --create to be the first (it is the action)
[15:53:19] <hashar>	 ^d: you can still run your scripts on both :]
[15:53:26] <hashar>	 ^d: would be a nice way to test out eqiad
[15:53:37] <^d>	 What's left to move the rest to eqiad and shut down pmtpa?
[15:53:46] <hashar>	 gotta test it
[15:54:00] <hashar>	 I wanted to do it today but took most of the day to complete the migration of the integration labs project
[15:54:36] <andrewbogott>	 hashar: oh, I see now
[15:57:28] <hashar>	 andrewbogott: that is a bit of a mess :D
[15:57:49] <andrewbogott>	 hashar:  It's ok, matanya is looking forward to refactoring it :)
[15:58:00] <hashar>	 we should use a shell script
[15:58:24] <hashar>	 or even better a python script wrapper :-]
[16:00:47] <hashar>	 so https://gerrit.wikimedia.org/r/121671  is the last of them
[16:02:31] <andrewbogott>	 hashar:  Ok, I'm about to vanish for a while… are you unblocked for the moment?
[16:02:55] <hashar>	 well both my projects are using puppetmaster now :-]  So there is no blocking
[16:03:03] <hashar>	 beta I will get it switched monday I guess
[16:03:21] <hashar>	 not tonight for sure sorry :-(
[16:03:43] <andrewbogott>	 Monday should be OK.  Feel free to email over the weekend if you need anything.
[16:04:00] <hashar>	 will probably just test out beta and figure out if it works fine
[16:04:26] <hashar>	 I guess it will just need a few tweaks in the mediawiki configuration
[16:06:28] <hashar>	 andrewbogott: enjoy )your week-end!
[16:06:39] <andrewbogott>	 Coren: I'm about to take a weird, extra-long lunch break… will be back (briefly) in the afternoon.  Email me if anything urgent comes up.
[16:07:25] <Coren>	 andrewbogott: How weird.  :-)
[16:07:50] <andrewbogott>	 I mean, like, four-hours long extra-long.
[16:07:55] <andrewbogott>	 But, ok, not that weird :)
[16:08:09] <NativeForeigner>	 lol
[16:14:54] <mutante>	 !log wikistats - migrated over to eqiad instance
[16:14:57] <labs-morebots>	 Logged the message, Master
[16:15:32] <NativeForeigner>	 !help
[16:15:32] <wm-bot>	 !documentation for labs !wm-bot for bot
[16:15:38] <NativeForeigner>	 darn, nvm
[16:18:29] <hashar>	 !log deployment-prep rebased puppet on deployment-salt 
[16:18:31] <labs-morebots>	 Logged the message, Master
[16:27:11] <asad1>	 Hi everyone.... I need help... Is there any way to get the number of searches of specific article of specific day from enwiki_p database ????
[16:29:25] <asad1>	 Hi everyone.... I need help... Is there any way to get the number of searches on specific article of specific day from enwiki_p database ????
[16:30:42] <mutante>	 andrewbogott_afk: https://wikitech.wikimedia.org/w/index.php?title=Labs_Eqiad_Migration%2FProgress&diff=107972&oldid=107964
[16:30:45] <mutante>	 done
[16:44:29] <hashar>	 and I am off , have a good weekend
[18:21:54] <grrrit-wm>	 (03PS3) 10Tim Landscheidt: Fix build and run-time dependencies [labs/toollabs] - 10https://gerrit.wikimedia.org/r/106281 
[18:26:29] <marktraceur>	 Ugh
[18:26:39] <marktraceur>	 Having some trouble getting mongodb up and running with a custom db path
[18:26:40] <marktraceur>	 marktraceur@orgchart:/data/project$ sudo chown mongodb mongodb/
[18:26:40] <marktraceur>	 chown: changing ownership of `mongodb/': Invalid argument
[18:26:52] <marktraceur>	 Start currently fails with a "cannot create lock file" message
[18:33:31] <grrrit-wm>	 (03PS3) 10Tim Landscheidt: Work around pbuilder not properly setting $USER [labs/toollabs] - 10https://gerrit.wikimedia.org/r/106283 
[18:56:16] <cajoel>	 puppet Q: are all classes in modules/BLAH/manifests/init.pp auto imported?
[18:56:28] <cajoel>	 or is there an explicit list somewhere?
[18:57:28] <paravoid>	 oh hey cajoel 
[18:57:33] <paravoid>	 you may find this useful http://docs.puppetlabs.com/puppet/latest/reference/lang_namespaces.html
[18:58:36] <cajoel>	 figured it out -- puppetmaster (local) doesn't follow symlinks
[19:01:42] <marktraceur>	 Anyone have thoughts on the above? I guess it may have to do with the /data/project setup we have for this project but I'm not sure
[19:01:55] <grrrit-wm>	 (03PS4) 10Tim Landscheidt: Fix build and run-time dependencies [labs/toollabs] - 10https://gerrit.wikimedia.org/r/106281 
[19:23:40] <bd808|LUNCH>	 marktraceur: In eqiad the NFS server requires that the user is known on the NFS server side. YOu need to ask Coren or andrewbogott_afk to create the mongodb system user in LDAP
[19:24:22] <marktraceur>	 Fuuuuck.
[19:24:31] <Coren>	 Well, that depends really.  If there's no particular reason for the user to have a specific name, it's easier to just create a service user.
[19:24:31] <marktraceur>	 Well, I guess I'll file that bug and then lunch.
[19:24:44] <marktraceur>	 Coren: Oh, can we do that faster/now?
[19:25:19] <Coren>	 marktraceur: It takes all of 10 seconds.  Just go on wikitech and pick 'manage service groups' from the sidebar.
[19:25:52] <Coren>	 That adds a group /and/ a user, both named the same.
[19:26:07] <marktraceur>	 Oooh.
[19:26:21] <Coren>	 The interface lies atm, it will cal it 'local-whatever', but the real name is now '$project.whatever'
[19:26:43] <Coren>	 (So that they are globally unique)
[19:27:13] <marktraceur>	 Coren: Manage service groups, or security groups?
[19:27:17] <Coren>	 service.
[19:27:20] <marktraceur>	 Oh, never mind
[19:27:23] <marktraceur>	 Wrong dropdown
[19:27:54] <Coren>	 But also, are you putting a DB on NFS?  That's... rarely a good idea.  :-)
[19:30:21] <marktraceur>	 Coren: It doesn't really have to be fast, but I'd be up for changing
[19:30:34] <marktraceur>	 Maybe I will post-lunch
[19:31:00] <Coren>	 kk.  Poke me then if you want.  Hint: labs instance have available lvm instance-local storage that might be better.
[20:18:23] <andrewbogott>	 spagewmf: Are you all done migrating your editor-engagement stuff?
[20:19:41] <marktraceur>	 Coren: So...the service group didn't work
[20:19:51] <marktraceur>	 I've tried changing the group and the owner and neither seem to be right
[20:20:36] <Coren>	 Wait, what?  That's a serious bug if that's the case.  The instance is orgchart, right?  What's the name of the group to created?
[20:20:37] <marktraceur>	 The answer here may be "stop using NFS for databases"
[20:20:45] <marktraceur>	 The name of the group is mongodb
[20:20:53] * Coren  goes to check, hang thight.

[20:21:55] <Coren>	 That's odd, it seems to have been created okay and all.  Where are you getting the issue?
[20:51:36] <marktraceur>	 Coren: Sorry - I try to do "chown -R mongodb mongodb" in /data/project and get "chown: changing ownership of `mongodb/': Invalid argument"
[20:51:54] <marktraceur>	 I assume because the user doesn't exist but I'm not sure
[20:53:06] <chippy>	 marktraceur, chown user:group /directory
[20:53:25] <chippy>	 so it's looking for a mongodb directory 
[20:53:41] <marktraceur>	 Which exists.
[20:53:48] <chippy>	 ahh okay then :-)
[20:53:49] <marktraceur>	 marktraceur@orgchart:/data/project$ ls
[20:53:49] <marktraceur>	 backup  glustercopy  journal  mongodb  olddb  orgchart  orgcharts-dev.tar.gz  spool
[20:55:24] <marktraceur>	 Coren: Do I need to restart something after creating the service group, maybe?
[20:55:42] <Coren>	 marktraceur: Sorry, was getting food.
[20:55:47] <marktraceur>	 's OK
[20:56:02] <Coren>	 And your user/group is named orgcharts.mongodb not just mongodb.  :-)
[20:56:08] <marktraceur>	 Agh
[20:56:13] <marktraceur>	 Misread
[20:56:19] <Coren>	 This is why I said "unless it's important that the name be a specific one"  :-)
[20:56:32] <marktraceur>	 It sort of is?
[20:56:46] <marktraceur>	 Maybe I should just move it to the actual disk and stop mucking around
[20:56:57] <Coren>	 Aah; in which case yeah -- I could create the user in LDAP (I do it for reasonably frequent system users)
[20:57:17] <Coren>	 But honestly, I'd put DBs on the local disk unless you actually do need to access it through several NFS clients.
[20:57:23] <marktraceur>	 I'm doing that now
[20:57:33] <marktraceur>	 I can just keep the backups on NFS
[20:57:41] <Coren>	 That works.  :-)
[20:59:17] <Coren>	 marktraceur: Erm, two things though.  (a) your /var/ is full, and (b) you haven't actually set up a local volume.  :-)
[20:59:40] <marktraceur>	 ...agh
[21:01:10] <Coren>	 You need to 'include labs_lvm', add add a labs_lvm::volume.
[21:01:11] <marktraceur>	 I guess because some of my libraries got installed into /var/lib when I installed mongo?
[21:01:35] <marktraceur>	 Coren: Where, in "configure instance"?
[21:01:41] <Coren>	 ... probably.  Although the package is a bit insane if it put libraries in /var/lib really.
[21:02:01] <marktraceur>	 "probably" this is exciting
[21:02:32] <marktraceur>	 I don't see any labs_lvm in the instance configuration page
[21:02:39] <andrewbogott>	 Mardetanha: if you check the 'role::labs::lvm::mnt' box on configure instance, that'll just make one giant volume in /mnt
[21:02:45] <marktraceur>	 Am I going to have to do the self-hosting puppet thing again? -.-
[21:02:46] <andrewbogott>	 um… sorry Mardetanha, typo
[21:02:50] <andrewbogott>	 marktraceur: that was for you
[21:02:53] <marktraceur>	 Aha.
[21:02:59] <Coren>	 No need.
[21:03:07] * marktraceur  does

[21:03:21] <Coren>	 role::labs::lvm::mnt in configure instance will give you a default "all the rest of the disk space on /mnt"
[21:03:26] <marktraceur>	 Done
[21:03:37] <marktraceur>	 But, /var is still full
[21:03:52] <Coren>	 marktraceur: Lemme see where the space has gone.
[21:04:14] <Coren>	 Yeah, it's mongodb.
[21:04:30] <marktraceur>	 Balls.
[21:04:56] * Coren  growls at the bad package.

[21:05:28] <marktraceur>	 I guess I can just rm -rf it, I have the data in /mnt now
[21:05:40] <Coren>	 LFS is clear that /var should only contain stuff that is meant to be modified at runtime; not general package crap.
[21:06:51] <Coren>	 It's a pet peeve of mine.  Almost as bad as packages putting stuff straight in / (I'm looking at *you* Bernstein!)
[21:07:17] <Coren>	 The good part is that a full /var pretty much can't hose your system.
[21:07:22] <marktraceur>	 Cool
[21:07:33] <Coren>	 (Which is the point of insulating it from /; this way runaway logs can't break the box)
[21:07:50] <Coren>	 Or spools, or mail, etc.
[21:10:53] <marktraceur>	 Now just need to figure out what's wrong with the HTTP listener...but that's my problem :)
[21:26:10] <cajoel>	 can someone walk me through sending a new patch set against a gerrit?
[21:26:32] <cajoel>	 I've made all the changes I want, but I've long lost the original clone..  
[21:26:49] <cajoel>	 so I've fetched the previous patch set
[21:26:54] <cajoel>	 and applied my changes on top.
[21:26:56] <cajoel>	 now what?
[21:28:06] <marktraceur>	 Now it looks like the bloody proxy is broken
[21:28:25] <marktraceur>	 I can access localhost:8888 fine from the instance, but orgchart.wmflabs.org is no good
[21:39:14] <scfc_de>	 marktraceur: Do your security groups allow access to :8888?
[21:41:10] <^d>	 Does bastion point to eqiad now?
[21:42:00] <Damianz>	 yes
[21:42:31] <^d>	 Ah. So if I need to ssh to tampa...I'll have to use the old IP
[21:43:29] <Damianz>	 bastion-pmtpa should work
[21:43:57] <Damianz>	 I wonder if all these offline instances are in the same project hmmm
[21:44:04] * Damianz  figures out how to use semantic queries

[21:44:26] <^d>	 Bleh, I can hit the bastion.
[21:44:41] <^d>	 Old beta instance in tampa won't let me in though :\
[21:46:35] <Damianz>	 I wonder if the broken gluster issue was just abandond in tampa
[21:55:26] <petan>	 @replag
[21:55:26] <wm-bota>	 Replication lag is approximately 00:00:01.6564550
[22:02:36] <petan>	 @info
[22:02:36] <wm-bot>	 http://bots.wmflabs.org/~wm-bot/dump/%23wikimedia-labs.htm
[22:02:41] <petan>	 Steinsplitter ^
[22:04:42] <petan>	 Steinsplitter: of course there is example...
[22:05:09] <petan>	 mhm there was
[22:05:14] <petan>	 something is broken
[22:05:44] <petan>	 however it's pretty simple, you just generate RSS you want to watch in bugzilla and insert it using @rss+
[22:05:46] <petan>	 @rss+
[22:05:48] <petan>	 @rss+ bla\
[22:05:48] <wm-bot>	 There is no such item, if you want to define new item, please use 2 parameters
[22:05:55] <petan>	 yes that is the command :P
[22:07:38] <Steinsplitter>	 thx :)
[22:08:46] <marktraceur>	 Uhhhh
[22:19:11] <marktraceur>	 scfc_de: Yeah, security groups let 8888 through just fine
[22:32:50] <PiRSquared>	 hi
[22:33:03] <PiRSquared>	 Can someone start the webservice for a tool whose maintainer is away?
[22:35:55] <scfc_de>	 PiRSquared: Was the tool automatically migrated?  If so, it would be very nice if the maintainer could check beforehand if everything's fine.
[22:36:12] <PiRSquared>	 The tool is searchsbl
[22:36:57] <PiRSquared>	 the maintainer is seth
[22:40:44] <scfc_de>	 Hmmm.  According to his user page, he will be away for another week.  Let me see if the tool is simple enough.
[22:56:19] <PiRSquared>	 scfc_de: I know tools-webproxy, but how can I access other instances?
[23:00:48] <scfc_de>	 In the same way?
[23:01:02] <scfc_de>	 Doesn't that work for you?
[23:25:08] <scfc_de>	 *argl*  Spent half an hour getting frustrated why lighttpd didn't react to the config changes, until I noticed that I was creating ~/.lighttpd.
[23:48:36] <scfc_de>	 PiRSquared: I think I've mastered searchsbl now.  I'll leave a note a Seth's talk page on what I've done.
[23:48:58] <PiRSquared>	 scfc_de: wow, thanks
[23:49:09] <PiRSquared>	 :)