[00:07:32] <wm-bot>	 [bz] (8NEW - created by: 2Jackmcbarn, priority: 4Unprioritized - 6normal) [Bug 54302] http://deployment.wikimedia.beta.wmflabs.org/ yields "No wiki found" - https://bugzilla.wikimedia.org/show_bug.cgi?id=54302
[05:29:36] <Reedy>	  https://wikitech.wikimedia.org/wiki/User:Socialbizteam is spam
[05:29:57] <Reedy>	 https://wikitech.wikimedia.org/wiki/Shell_Request/Socialbizteam is also crap
[12:35:16] <addshore>	 petan: Coren around?
[12:35:28] <petan>	 yes
[12:35:36] <petan>	 addshore: quick, what's up?
[12:35:45] <addshore>	 could you please add https://wikitech.wikimedia.org/wiki/User:Denny_Vrandecic to tools :)
[12:36:03] <petan>	 !tr Denny Vrandecic
[12:36:03] <wm-bot>	 goes fishing, catches a trout, and smack Denny with it, over and over.
[12:36:07] <petan>	 wtf
[12:36:37] <addshore>	 .. xD
[12:37:10] <petan>	 done
[12:37:16] <addshore>	 cheers :)
[13:39:40] <manybubbles>	 !log deployment-prep rebuilt most search indecies in beta but commonswiki crashed late last night so it is half rebuilt.  filing bug.
[13:39:43] <labs-morebots>	 Logged the message, Master
[17:11:08] <Elph>	 Hi all
[17:11:13] <Elph>	 I have a Q
[17:11:41] <Elph>	 I want to run a query in wikiquote
[17:12:09] <Elph>	 Is "arwikiquote_p.labsdb" true?
[17:12:40] <Elph>	 	Is "arwikiquote_p.labsdb" correct?
[17:51:04] <Nettrom>	 Elph: it seems you're wodnering what server to connect to?
[17:52:34] <Elph>	 yes it is
[17:53:07] <Nettrom>	 the server is "arwikiquote.labsdb"
[17:53:15] <Nettrom>	 I do not find a "arwikiquote_p" database on it though
[17:53:18] <Nettrom>	 that sounds like a bug
[17:53:51] <liangent>	 Elph: arwikiquote.labsdb
[17:54:06] <Nettrom>	 I'll file a bug about the missing database
[17:54:07] <liangent>	 Nettrom: views are missing
[17:54:12] <liangent>	 databases are not
[17:54:18] <liangent>	 Coren: ^
[17:54:32] <Nettrom>	 there's no "arwikiquote_p" database with the views in it
[17:54:40] <Nettrom>	 there's just "arwikiquote" which we don't have access to
[17:55:27] <Nettrom>	 or is the database technically a view too?
[17:56:53] <liangent>	 Nettrom: arwikiquote is the database containing data
[17:57:17] <liangent>	 arwikiquote_p has only views allowing you to access insensitive data in arwikiquote
[17:57:32] <Elph>	 this is the error msg (_mysql_exceptions.OperationalError: (1049, "Unknown database 'arwiki_p'"))
[17:58:23] <Nettrom>	 Elph: the database you want to use is "arwikiquote_p", but currently it doesn't exist
[17:58:35] <Elph>	 oh
[17:59:05] <Elph>	 when it will be existed? 
[18:00:15] <liangent>	 Elph: Coren said "that will complete withing an hour or two.", two days ago
[18:00:38] <liangent>	 and no more updates
[18:01:07] <Elph>	 thanks a lot for your help
[18:03:20] <wm-bot>	 [bz] (8UNCONFIRMED - created by: 2Morten Wang, priority: 4Unprioritized - 6normal) [Bug 54390] Add arwikiquote_p database view - https://bugzilla.wikimedia.org/show_bug.cgi?id=54390
[18:06:00] <liangent>	 that's not arwq only
[18:08:51] <Nettrom>	 liangent: feel free to update the bug then :)
[18:19:51] <Nettrom>	 liangent: ah, I see, it was a result of the DB outage... thanks for clarifying, didn't catch that! :)
[19:20:36] <hashar>	 !log integration on integration-selenium-driver installing openjdk-7-jre-headless and qa/browsertests.git packages dependencies (see also {{bug:54385}} and {{gerrit|85264}}
[19:20:37] <labs-morebots>	 Logged the message, Master
[19:51:26] <kaldari>	 looks like http://en.wikipedia.beta.wmflabs.org is down
[19:51:43] <kaldari>	 Uncaught exception 'Exception' with message 'WikibaseLib depends on the ValueView extension.' in /data/project/apache/common-local/php-master/extensions/Wikibase/lib/WikibaseLib.php:105
[19:53:19] <kaldari>	 Coren, Ryan_Lane: ^
[19:53:39] <kaldari>	 looks like it needs the ValueView extension enabled
[19:54:00] <Coren>	 kaldari: I think someone is currently messing with it.  I'll ask around.
[19:54:07] <kaldari>	 thanks
[20:05:16] <anomie>	 Coren: Labs FUD alert at https://en.wikipedia.org/w/index.php?title=Wikipedia:Village_pump_%28technical%29&diff=573815193&oldid=573813402
[20:13:06] <greg-g>	 kaldari: Coren so, aude fixed it, but beta hasn't updated yet...
[20:16:13] <Coren>	 greg-g: If they are relying on puppet to push those changes (I'm not entirely sure how beta is set up), then that runs every 30 minutes.
[20:17:54] <aude>	 Coren: ok
[20:18:08] <aude>	 i can see how to update it 
[20:18:27] <Coren>	 aude: If you are project admin, you can do a 'sudo puppetd -tv' to force a puppet run
[20:18:32] <aude>	 ok
[20:21:48] <cscott>	 sigh.  another day, another irc channel.
[20:21:51] <cscott>	 i'm here, YuviPanda
[20:21:57] <YuviPanda>	 hi cscott :)
[20:22:05] <hashar>	 most beta updates are handled via Jenkins https://integration.wikimedia.org/dashboard/
[20:22:14] <hashar>	 and http://integration.wikimedia.org/ci/view/Beta/ :-]
[20:22:24] <hashar>	 any one in the wmf group can retrigger the jobs if needed
[20:22:46] <YuviPanda>	 cscott: if you can give ma admin access to your project and I'll setup NFS for you.
[20:23:00] <cscott>	 hm, i wonder how i do that?
[20:23:30] <hashar>	 social engineering !
[20:23:37] <YuviPanda>	 cscott: login to wikitech.wikimedia.org
[20:24:15] <YuviPanda>	 cscott: then go to https://wikitech.wikimedia.org/wiki/Special:NovaProject
[20:24:26] <hashar>	 have a good weekend, I am off
[20:24:31] <aude>	 hashar: ok!
[20:24:32] <YuviPanda>	 cscott: select your project from the 'project filter' thingy
[20:24:38] <YuviPanda>	 cscott: click 'set filter'
[20:25:02] <aude>	 it is in progress?
[20:25:14] <YuviPanda>	 cscott: you should see 'actions', with 'add member' in it. first do that, pick my name on it
[20:27:21] <cscott>	 YuviPanda: at what point do i have to tell you my social security number and credit card CCV?
[20:27:45] <YuviPanda>	 cscott: right after you give me your house key
[20:28:19] <cscott>	 "There are no members to add to this group."
[20:28:30] <cscott>	 it's in the visualeditor project, i bet you're already a member.
[20:28:37] <YuviPanda>	 cscott: oh
[20:28:38] <YuviPanda>	 let me check
[20:28:45] <YuviPanda>	 cscott: no, i'm not :(
[20:28:46] <cscott>	 no, wait, hang on. maybe i clicked the wrong 'add member'
[20:29:09] <YuviPanda>	 cscott: yeah, probably. you need the one under 'actions' by itself, not actions under roles
[20:29:30] <cscott>	 ok, you're a member now.  i assume you'll want to be a projectadmin, as well?
[20:30:53] <cscott>	 pinklake is https://wikitech.wikimedia.org/wiki/Nova_Resource:I-000008d0 fwiw
[20:31:51] <YuviPanda>	 cscott: yeah, i'd need projectadmin
[20:32:05] <cscott>	 done
[20:32:18] <YuviPanda>	 okay!
[20:34:59] * cscott  checks his bank account balances

[20:38:55] <YuviPanda>	 cscott: i'm making a backup of your home on it right now
[20:39:03] <YuviPanda>	 cscott: jesus christ gluster is so fucking slow
[20:39:13] <YuviPanda>	 cscott: do you need a backup?
[20:39:15] <cscott>	 YuviPanda: yes indeed.
[20:39:25] <YuviPanda>	 is that a 'yes' for the 'do you need a backup'?
[20:39:25] <cscott>	 YuviPanda: no, it's just checked out code that i could recreate if needed.
[20:39:29] <YuviPanda>	 sweet
[20:39:35] <cscott>	 yes == yes gluster is slow
[20:39:37] <YuviPanda>	 restarting
[20:42:00] <YuviPanda>	 cscott: should be faster now
[20:42:02] <YuviPanda>	 is on NFS
[20:42:05] <YuviPanda>	 cscott: check it out?
[20:42:46] <cscott>	 seems to work
[20:42:53] <YuviPanda>	 cscott: faster?
[20:43:25] <cscott>	 maybe?  it's still working on https://pinklake.wmflabs.org/wiki/Boston but maybe that's just caching
[20:43:45] <cscott>	 yes, reloading is much faster
[20:44:03] <YuviPanda>	 cscott: sweeeet :D
[20:44:30] <YuviPanda>	 cscott: that'll be 400$, I'll mail you an invoice :P
[20:44:38] <cscott>	 sent it to WMF ;)
[20:44:46] <YuviPanda>	 hehe
[20:44:58] <cscott>	 presumably gluster is good for *something*?
[20:45:11] <YuviPanda>	 cscott: not that i know of, at least in our setup
[20:45:20] <YuviPanda>	 cscott: i don't even know why it is still default
[20:51:26] <cscott>	 YuviPanda: how are you doing with that url-based proxying?
[20:51:46] <YuviPanda>	 cscott: figured out the 'right' way to do it while still being fast. just needs writing. next week, probably.
[20:52:11] <YuviPanda>	 cscott: in the meantime, if you are okay with something like bluelake.wmflabs.org that points to port 80 on the same instance, I can set that up
[20:52:42] <cscott>	 i think VE wants a public address for parsoid, so i really need three urls/ports: one for MW, one for parsoid, and one for the websocket hub.
[20:53:12] <cscott>	 they can be differently colored lakes, if you like.  parsoid is running on port 8001 at the moment.
[21:17:46] <J-Mo>	 hi Coren: I had a bot user db on the replica cluster that went down on Wednesday: p50380g50729__grantsbot. Looks like most of the dbs on the cluser are back up (arwiki, metawiki), but my bot db is nowhere to be found. Any ideas where it went?
[21:19:11] <Coren>	 J-Mo: It's probably gone for good; databases on the replicas are explicitly not permanent (though we try our best to not randomly destroy them), but we can /try/ to see if we can recover it from the broken image if you can't regenerate it and have no backup.
[21:20:42] <J-Mo>	 Coren: oh. okay. in future, what's the best way for me to backup these databases? I didn't have much data on this one yet, but there will be tables I can't afford to lose.
[21:22:27] <Coren>	 J-Mo: A scheduled dump to the shared filesystem is probably your best bet; that has redundancy up the wazoo.  If you want to be extra paranoid, a second copy in /home would be extra fun safe.
[21:23:32] <Coren>	 J-Mo: (you might want to look ay mysqldump for this)
[21:23:38] <J-Mo>	 so (excuse the n00b question): I should set up a .sql dump on cron? 
[21:24:25] <Coren>	 J-Mo: That's the easiest possible way to do it, yes.  If the data is generally very static (or is only added to over time) it might even be extra nice if you then stuff that .sql to a source control of some kind.
[21:24:50] <Coren>	 J-Mo: But that's probably overkill and overcomplicated for most things.
[21:24:54] <J-Mo>	 cool. I'll work on that. Thanks for the info!
[21:26:20] <YuviPanda>	 Coren: is this on tools-help?
[21:26:32] <YuviPanda>	 Coren: also we should tell people to create dbs on tools-db unless they have very specific needs
[21:27:41] <Coren>	 YuviPanda: I don't think so.  And you're correct that we should tell people to prefer tools-db; "wipe and restart replication" is our default recovery system for the replicas.
[21:27:59] <YuviPanda>	 Coren: yeah, you should add it :)
[21:28:14] <Coren>	 Although we're going to be adding a fourth box to serve as hot swap so that the delay is shorter if it happens again.
[21:28:56] <Coren>	 YuviPanda: I've just added this to my TODO for once I'm back home next week.  :-)
[21:28:59] <YuviPanda>	 sure, but documentation is good! :)
[21:29:18] <YuviPanda>	 Coren: plus writes to tools-db are a lot faster than writes to the replica
[21:29:25] <YuviPanda>	 Coren: the 26ms ping should also be mentioned
[21:29:53] <Coren>	 YuviPanda: True.
[21:36:22] <wm-bot>	 [bz] (8NEW - created by: 2Morten Wang, priority: 4Unprioritized - 6normal) [Bug 54390] Re-create database views lost in the outage such as arwikiquote_p - https://bugzilla.wikimedia.org/show_bug.cgi?id=54390
[21:36:23] <wm-bot>	 [bz] (8UNCONFIRMED - created by: 2Nicolas Raoul, priority: 4Unprioritized - 6critical) [Bug 54059] LAMP instance becomes 404 a few hours after spawn (reproducible) - https://bugzilla.wikimedia.org/show_bug.cgi?id=54059
[21:36:24] <J-Mo>	 YuviPanda and Coren: can I run joins on replica tables from tools-db? The documentation made it seems as though that was not possible. 
[21:37:16] <Coren>	 J-Mo: You can't; that's the case where having your DBs on the replicas DB is the way to go.;
[21:37:55] <J-Mo>	 Coren: that's a pretty common use case for bots though, isn't it?
[21:38:12] <Coren>	 J-Mo: It is, which is why it's explicitly supported.  :-)
[21:39:33] <J-Mo>	 yeah, but if wipe-and-restart is the default for replica clusters, that can throw bots out of whack pretty easily even if they're on a daily backup.
[21:39:50] <YuviPanda>	 +1
[21:40:20] <J-Mo>	 so, I guess I'm asking: is it possible to support dbs on replicas as fully as dbs on tools-db?
[21:40:56] <J-Mo>	 (assuming, again, that there are better backups in place on tools-db: I'm shooting in the dark here… ;)
[21:43:59] <J-Mo>	 Coren ^^ . Also, I can file a bug report for this if you think that will help move the discussion forward.
[21:44:33] <YuviPanda>	 +1 to bug report, J-Mo
[21:44:49] <J-Mo>	 on it!
[21:45:39] <Coren>	 Go ahead and file the bugzilla; but I don't know how realistically we can support that scenario in practice.  I can certainly try to get the resources for it.
[22:00:26] <wm-bot>	 [bz] (8NEW - created by: 2Jonathan Morgan, priority: 4Unprioritized - 6normal) [Bug 54401] Backups systems for userDBs hosted on replicas - https://bugzilla.wikimedia.org/show_bug.cgi?id=54401
[22:02:01] <J-Mo>	 Coren: bug filed! I hope we can get resources for this, if other bot owners agree that it's an issue.
[22:08:26] <jgrivolla>	 hello, could someone add https://wikitech.wikimedia.org/wiki/User:Jgrivolla to tools? I've requested it via the wiki a few days ago, but no reaction so far...
[22:11:20] <jgrivolla>	 I mean tool labs, i.e. ssh etc.
[22:19:57] <YuviPanda>	 !requests
[22:19:57] <wm-bot>	 https://labsconsole.wikimedia.org/wiki/Help:Contents#Requests
[22:26:31] <spagewmf>	 Hey, ee-prototype labs instance is long in the tooth (2011 or earlier). Old Ubuntu 10.04, old packages.  Is there a way to rebuild it? (e.g. delete instance, recreate with same name)
[22:27:39] <YuviPanda>	 spagewmf: theoretically should work, but i know that labs is having DNS problems so I'm not toooo sure
[22:27:42] <YuviPanda>	 spagewmf: poke Ryan_Lane
[22:45:23] <spagewmf>	 thx YuviPanda (and for helping Arthur Richards )
[22:45:33] <YuviPanda>	 yw, spagewmf :)
[22:58:15] <grrrit-wm>	 (03PS1) 10Yuvipanda: Stop sending SMW messages to #wikimedia-dev [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/85355 
[22:58:38] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] Stop sending SMW messages to #wikimedia-dev [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/85355 (owner: 10Yuvipanda)
[23:03:36] <Ryan_Lane>	 gwicke: restarted
[23:03:38] <Ryan_Lane>	 so....