[00:36:03] <mutante>	 !log deployment-prep raised instance quota to 43
[00:36:07] <labs-morebots>	 Logged the message, Master
[01:30:08] <wikibugs>	 3Wikimedia Labs / 3Infrastructure: Replica MySQL:  Views completely missing from some wiki's - 10https://bugzilla.wikimedia.org/71041#c4 (10Sean Pringle) labsdb1003 hasn't upgraded to a single MariaDB 10 instance yet.
[03:12:21] <Svetlana>	 apparently theres a near-live copy of db on labs but i don't see docs about it yet
[03:18:12] <Svetlana>	 someone knows what its called?
[03:21:56] <Coren>	 You mean the replicas?  They're essentially live (<1s lag most of the time).  They are called <projectname>.labsdb
[03:22:05] <Svetlana>	 ok
[03:22:14] <Coren>	 Like, "enwiki.labsdb"
[03:22:15] <Svetlana>	 'replicas' is good enough, i hope to find some docs from there
[03:22:49] <Coren>	 https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Database_access
[03:23:06] <Svetlana>	 do i have to request the access or do i have it by default?
[03:23:23] <Coren>	 Svetlana: If you are a member of the tools project in general, you have access.
[03:23:40] <Svetlana>	 how do i check?
[03:24:51] <Svetlana>	 uid=3319(gryllida) gid=500(wikidev) groups=500(wikidev),50062(project-bastion),50380(project-tools),51221(tools.gpy),51254(tools.wmtran)
[03:25:00] <Svetlana>	 i guess thats sufficient, it says project-tools
[03:25:06] <Coren>	 Yep.
[03:25:51] <Coren>	 There's an easy shortcut if you want to just run a few queries, "sql projectname" from tools-login will give you an sql prompt.
[03:25:55] <Coren>	 Like, "sql enwiki"
[03:26:26] <Svetlana>	 i wonder how to access them programmatically but i'll take time to read the docs and play with it, even a prompt is fascinating
[03:26:35] <Svetlana>	 thanks :)
[03:26:47] <Svetlana>	 btw they're readonly right?
[03:27:30] <Coren>	 Well, the project DBs are.  You can write to your own though.
[03:27:52] <Svetlana>	 Ok, good, thanks.
[03:36:25] <jimmyxu>	 Hi, are we still doing / able to do renameuser on the wikitech.wikimedia.org wiki?
[03:59:35] <Coren>	 jimmyxu: Ostensibly, yes.  In practice, you'd have to have a *really* good reason for it because so many systems are tied into this it takes over an hour of manual work and almost always causes trouble later.  :-)
[04:01:05] <jimmyxu>	 Coren: That's reasonable.  On second thought, can I like delete my labs user and link the username to a new account on wikitech?
[04:02:05] <Coren>	 ... can't delete users in mediawiki, you know.  :-)  What's the issue with your name atm?
[04:03:04] <jimmyxu>	 Coren: I meant delete the user inside tools labs (ldap?), register a new user on mediawiki, and give the new user that name
[04:03:34] <jimmyxu>	 Coren: Not a big issue, just it being inconsistent with me on rest of the wmfwikis bugs me... :(
[04:03:57] <Coren>	 Hmmm.  Probably not, but me might be able to _swap_ them with some LDAP trickery.  Open a bugzilla about it, and I'll discuss it with Andrew see if we can do something.
[04:07:50] <jimmyxu>	 Coren: Will do.  Do I need to request access to tools from the new user?
[04:08:20] <Coren>	 Not if we manage to swap them.
[04:14:08] <wikibugs>	 3Wikimedia Labs: Swap two shell names - 10https://bugzilla.wikimedia.org/71083 (10Jimmy Xu) 3NEW p:3Unprio s:3enhanc a:3None I'm hoping to rename my username on wikitech from User:Jimmyxu to User:Jimmy_Xu, and Coren told me this is a bad idea.  So instead I now have User:Jimmyxu = shell `jimmy`, and Us...
[04:14:12] <jimmyxu>	 Coren: filed bug 71083.  I realize the bug summary is bad...
[04:15:31] <Coren>	 Meh, should be clear enough.
[04:29:32] <Svetlana>	 right so I found a query which gives me the results I like ... hm...
[05:09:27] <Pathoschild>	 All tables on s7.labsdb seem to have disappeared. Is that expected?
[05:38:35] <Coren>	 No, it is not.  Use c1 or c2 for now.
[05:38:43] <Coren>	 (Both of them have all the databases)
[06:37:22] <icinga-wm>	 PROBLEM - ToolLabs: Puppet failure events on labmon1001 is CRITICAL: CRITICAL: tools.tools-webgrid-01.puppetagent.failed_events.value (33.33%)  
[07:05:56] <icinga-wm>	 RECOVERY - ToolLabs: Puppet failure events on labmon1001 is OK: OK: All targets OK  
[10:43:11] <wikibugs>	 3Wikimedia Labs / 3tools: Rows missing from enwiki_p on s1 - 10https://bugzilla.wikimedia.org/71084 (10Brad Jorsch) 3NEW p:3Unprio s:3normal a:3Marc A. Pelletier As can be seen from the following, at least the row for page_id=43871050 is missing from enwiki_p.page on s1 while it's present on s2, s4,...
[11:43:37] <tonythomas>	 configuring instance on wikitech labs shows a page - The specified resource does not exist. :\
[11:43:49] <tonythomas>	 broke ?
[11:44:24] <grrrit-wm>	 (03Abandoned) 10Yuvipanda: Remove duplication of -multimedia extensions from -dev [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/134154 (owner: 10Yuvipanda)
[11:49:31] <Reza1615>	 When i type sql fawiki
[11:49:36] <Reza1615>	 describe page;
[11:49:42] <Reza1615>	 it shows this error
[11:49:50] <Reza1615>	 Table 'fawiki_p.page' doesn't exist
[11:49:57] <Reza1615>	 what should i do?
[11:50:18] <Reza1615>	 before this i used sql for fawiki
[11:50:27] <Reza1615>	 today it shows this error
[11:51:03] <tonythomas>	 Reza1615: happened to me yesterday. I did it like  $sql loginwiki
[11:51:03] <tonythomas>	 > use mywiki
[11:53:15] <Reza1615>	 at mywiki how can i connect to fawiki?
[11:53:43] <Reza1615>	 tonythomas: I typed sql loginwiki
[11:53:48] <Reza1615>	 use mywiki
[11:53:51] <tonythomas>	 no no 
[11:53:53] <Reza1615>	 it went to mywiki
[11:53:58] <tonythomas>	 can you try. $sql loginwiki
[11:54:04] <tonythomas>	 >use fawiki
[11:54:31] <Reza1615>	 ERROR 1049 (42000): Unknown database 'fawiki'
[11:54:33] <anomie>	 You'll probably need "use fawiki_p"
[11:55:01] <Reza1615>	 ERROR 1146 (42S02): Table 'fawiki_p.page' doesn't exist
[11:55:36] <tonythomas>	 anomie: any idea why I am not able to configure instances in wikitech ?
[11:55:47] <anomie>	 tonythomas: No idea, sorry
[11:55:59] <tonythomas>	 ok. 
[11:56:34] <Reza1615>	 anomie:  after use fawiki_p it shows the same erro
[11:56:48] <tonythomas>	 Reza1615: might be you are in a wrong host ?
[11:57:18] <Reza1615>	 about one year i used this putty and account!
[11:57:27] <Reza1615>	 to it happend :(
[11:57:41] <Reza1615>	 *today
[11:57:53] <anomie>	 Reza1615: Apparently some of the servers are missing it. Try 'sql dewiki', which uses s5, then the 'use fawiki_p'.
[11:58:52] <Reza1615>	 :) now it works
[11:59:07] <Reza1615>	 thanks how can i config tools to use s5?
[12:08:28] <Reza1615>	 do you know how can i save sql output from MariaDB  commoand line?
[12:11:24] <yuvipanda>	 Reza1615: if you're just trying to run SQL commands and get output, try quarry.wmflabs.org
[12:14:45] <Reza1615>	 yuvipanda: quarry.wmflabs.org shows time out!
[12:14:59] <yuvipanda>	 Reza1615: oh? when?
[12:15:04] <Reza1615>	 now
[12:15:05] <yuvipanda>	 try again? i just tried and seems to work
[12:15:20] <Reza1615>	 port is 22?
[12:15:22] <yuvipanda>	 Reza1615: try a hard refresh?
[12:15:30] <yuvipanda>	 Reza1615: oh no, you use it from a web browser
[12:15:33] <yuvipanda>	 Reza1615: http://quarry.wmflabs.org
[12:15:34] <yuvipanda>	 not ssh
[12:19:18] <tonythomas>	 yuvipanda: any idea why I am not able to configure labs instances ?
[12:19:35] <tonythomas>	 I had to check in my database, and it looks like I dont have the mysql role :( 
[12:19:43] <tonythomas>	 oh. wait - I can add it manually too right ?
[12:23:12] <yuvipanda>	 tonythomas: hmm, not sure :( you need andrewbogott_afk for that usually
[12:23:55] <tonythomas>	 yuvipanda: ok. I will not be able to access my webserver instance db without the mysql role right ?
[12:24:04] <yuvipanda>	 tonythomas: probably, yeah
[12:24:19] <tonythomas>	 yuvipanda: okey. let me try to figure that one out 
[12:24:25] <yuvipanda>	 ok
[12:24:29] <yuvipanda>	 tonythomas: try logging out and in
[12:24:33] <yuvipanda>	 that sometimes fixes things
[12:24:36] <tonythomas>	 ha :) 
[12:24:40] <tonythomas>	 let me try that one 
[12:25:19] <tonythomas>	 I cant believe it 
[12:25:24] <tonythomas>	 its alright :) yay ! 
[12:25:47] * tonythomas never thought logging in and out had these effects :)
[12:28:24] <yuvipanda>	 tonythomas: :D
[12:28:29] <yuvipanda>	 tonythomas: wikitech  is sometimes like that
[12:34:58] <reza1615>	 yuvipanda: how can i get a file when i click on submit Query it does not do any thing! my query is
[12:35:26] <yuvipanda>	 reza1615: is it this one? http://quarry.wmflabs.org/query/573
[12:38:59] <reza1615_>	 yuvipanda: I wrote a simple query and clicked on submit but it doesn't do !
[12:39:18] <reza1615_>	 my query:
[12:39:20] <reza1615_>	 use fawiki_p; SELECT page_title FROM page WHERE page_namespace = 0 AND page_is_redirect = 1;
[12:40:39] <yuvipanda>	 reza1615_: under the box it says 'running'
[12:41:29] <yuvipanda>	 reza1615_: that query will probably produce a *lot* of output
[12:41:35] <yuvipanda>	 you should refine it to produce lesser output
[12:41:46] <yuvipanda>	 assuming there are a *lot* of redirects in fawiki
[12:42:03] <reza1615>	 yes i want the output as file
[12:42:38] <yuvipanda>	 reza1615_: hmm, quarry.wmflabs.org will allow you to download the output as CSV or other formats once it completes
[12:42:47] <yuvipanda>	 assuming it completes within 10 minutes
[12:43:43] <reza1615>	 use fawiki_p; SELECT page_title FROM page WHERE page_namespace = 0 AND page_is_redirect = 1 lIMIT 10;
[12:44:30] <reza1615>	 would you please to add output file without printing?
[12:44:46] <reza1615>	 like os.system(u'sql fawiki "SELECT page_title FROM page WHERE page_namespace = 0 AND page_is_redirect = 1 LIMIT 100;" > up02.txt')
[12:45:19] <reza1615>	 it is so fast but now because of servers it does not work
[12:45:50] <reza1615>	 also the last query LIMIT 10 shows this error Can't connect to MySQL server on 'enwiki.labsdb' ([Errno 113] No route to host)
[12:45:59] <yuvipanda>	 yeah, am investigating that error
[12:46:57] <Zhaofeng_Li>	 reza1615, that happened to me during yesterday's outage
[12:47:42] <reza1615>	 Zhaofeng_Li: may be some serevrs are down!
[12:48:53] <Zhaofeng_Li>	 The server cats aren't feeling well... :(
[12:49:30] <Zhaofeng_Li>	 Ah, yesterday it was: ERROR 2003 (HY000): Can't connect to MySQL server on 'enwiki.labsdb' (110)
[13:09:13] <Zhaofeng_Li>	 reza1615, get a real client :)
[13:10:19] <yuvipanda>	 I have to go now
[13:10:27] <yuvipanda>	 I'll try fixing quarry latter. sorry!
[13:29:05] <Coren>	 yuvipanda: slice 1 has changed IP; you need to update your hosts file.
[13:32:13] <Pathoschild>	 Coren: thanks, that works. What is c1.labsb?
[13:33:03] <Coren>	 It's not 10.64.4.11
[13:33:05] <Coren>	 now*
[13:39:42] <Pathoschild>	 Coren: I mean is it a permanent server? What are the disadvantages of continuing to use it after the DB issues are fixed? I can't find any documentation about it.
[13:40:55] <Coren>	 Oh!  Yes, it's a permanent server, all three are normally in use - the disadvantage of using it indefinitely is just that if everyone does that we'll have two servers with 150% usage and one that sits idle.
[13:41:56] <Coren>	 But they are otherwise identical.
[13:44:32] <Pathoschild>	 That makes sense. :)
[13:59:58] <andrewbogott>	 !log account-creation-assistance moving  accounts-mwoauth to virt1005
[14:00:01] <labs-morebots>	 Logged the message, dummy
[14:01:52] <andrewbogott>	 !log analytics moving wikimetrics1 to virt1005
[14:01:53] <labs-morebots>	 Logged the message, dummy
[14:07:23] <wikibugs>	 3Wikimedia Labs / 3Infrastructure: Replica MySQL:  Views completely missing from some wiki's - 10https://bugzilla.wikimedia.org/71041#c5 (10metatron) (In reply to Sean Pringle from comment #4) > labsdb1003 hasn't upgraded to a single MariaDB 10 instance yet.  What does that mean? Upgrade hasn't started yet?...
[14:43:15] <andrewbogott>	 !log cvn moved cvn-apache5 to virt1007
[14:43:16] <labs-morebots>	 Logged the message, dummy
[14:43:28] <andrewbogott>	 !log deployment-prep movingdeployment-pdf02 to virt1009
[14:43:31] <labs-morebots>	 Logged the message, dummy
[14:45:03] <andrewbogott>	 !log full-text-reference-tool moved fulltext-pup to virt1009
[14:45:06] <labs-morebots>	 Logged the message, dummy
[14:46:03] <andrewbogott>	 !log glam moved gwtoolset to virt1009
[14:46:05] <labs-morebots>	 Logged the message, dummy
[14:49:56] <andrewbogott>	 !log language moved language-browsertests to virt1009
[14:49:57] <labs-morebots>	 Logged the message, dummy
[14:53:47] <andrewbogott>	 !log language moved language-lcmd to virt1009
[14:53:48] <labs-morebots>	 Logged the message, dummy
[14:57:17] <andrewbogott>	 !log language moved language-mleb-legacy to virt1009
[14:57:19] <labs-morebots>	 Logged the message, dummy
[15:17:35] <andrewbogott>	 !log maps moved maps-dj to virt1001
[15:17:37] <labs-morebots>	 Logged the message, dummy
[15:19:18] <andrewbogott>	 !log maps moved maps-tiles3 to virt1001
[15:19:19] <labs-morebots>	 Logged the message, dummy
[15:23:58] <andrewbogott>	 !log maps moved maps-wma1 to virt1001
[15:24:00] <labs-morebots>	 Logged the message, dummy
[15:25:58] <andrewbogott>	 !log mediawiki-core-team moving contactpage-test to virt1001
[15:25:59] <labs-morebots>	 Logged the message, dummy
[15:37:56] <multichill>	 andrewbogott: Openstack doesn't have a VMotion like feature, right? :-S
[15:38:10] <andrewbogott>	 multichill: I don't know what that is
[15:38:33] <multichill>	 The ability to move around VM's from one physical server to another
[15:38:42] <multichill>	 Without downtime for the VM
[15:39:01] <andrewbogott>	 Oh -- 
[15:39:13] <multichill>	 ( http://www.vmware.com/products/vsphere/features/vmotion )
[15:39:14] <andrewbogott>	 yes, that's available in openstack, but in that case the instance storage would have to be on a shared filesystem.
[15:39:24] <andrewbogott>	 Which we've had terrible luck with
[15:39:37] <multichill>	 VMware needs to have the same luns too
[15:39:50] <andrewbogott>	 I'm modestly tempted to give another go in our new datacenter but it's a generally unpopular idea
[15:40:11] <multichill>	 It's very nice when you need to clear out a server or rebalance load
[15:40:33] <andrewbogott>	 Yes, but not so nice when the fileserver gets into a race and every instance stops working at once
[15:40:51] <multichill>	 You can have multiple luns/fileservers
[15:40:55] <multichill>	 To spread the risk
[15:52:50] <grrrit-wm>	 (03PS1) 10Multichill: (bug 70806): Fix id length for sk so it wont get truncated [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/161672 
[15:53:40] <grrrit-wm>	 (03CR) 10Multichill: [C: 032 V: 032] "Deploy" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/161672 (owner: 10Multichill)
[15:56:17] <multichill>	 !log tools.heritage Fixed https://bugzilla.wikimedia.org/show_bug.cgi?id=70806 and deployed new 2 new sk tables
[15:56:20] <labs-morebots>	 Logged the message, Master
[18:14:19] <gifti>	 Coren: are you there?
[19:42:54] <andrewbogott_afk>	 !log multimedia moved multimedia-perf off of virt1006
[19:42:57] <labs-morebots>	 Logged the message, Master
[19:47:00] <andrewbogott_afk>	 !log otrs moved otrs-test and otrs-puppetisation off of virt1006
[19:47:03] <labs-morebots>	 Logged the message, Master
[19:54:19] <andrewbogott_afk>	 !log netflow moved flow-localpuppet off of virt1006
[19:54:22] <labs-morebots>	 Logged the message, Master
[20:00:32] <andrewbogott_afk>	 !log reportcard moved reportcard1 off of virt1006
[20:00:35] <labs-morebots>	 Logged the message, Master
[20:02:08] <andrewbogott_afk>	 !log pubsubhubbub moved suggester2 off of virt1006
[20:02:11] <labs-morebots>	 Logged the message, Master
[20:08:14] <andrewbogott_afk>	 !log structured-wikiquote moved structured-wikiquote-beta off of virt1006
[20:08:17] <labs-morebots>	 Logged the message, Master
[20:09:25] <andrewbogott_afk>	 !log toolsbeta moved toolsbeta-exec-01 and toolsbeta-scfc-icinga-test off of virt1006
[20:09:28] <labs-morebots>	 Logged the message, Master
[20:09:54] <andrewbogott_afk>	 !log visualeditor moved parsoid-square off of virt1006
[20:09:56] <labs-morebots>	 Logged the message, Master
[20:15:01] <andrewbogott_afk>	 !log wikisource-dev moved wikisource-dev off of virt1006
[20:15:04] <labs-morebots>	 Logged the message, Master
[20:17:55] <andrewbogott_afk>	 !log wikisource-tools moved wsexport off of virt1006
[20:17:58] <labs-morebots>	 Logged the message, Master
[20:19:08] <gifti>	 spam ;)
[20:19:51] <andrewbogott_afk>	 only one left I think
[20:22:03] <andrewbogott_afk>	 !log wikidata-topicmaps moved topicmaps-beta-launch off virt1006
[20:22:06] <labs-morebots>	 Logged the message, Master
[20:43:12] <hedonil>	 Coren: Can you give an ETA when the massively mising views and user!databases are back online again
[20:43:54] <hedonil>	 Coren: I'm unable to make head or tail of springle 's comment.  https://bugzilla.wikimedia.org/show_bug.cgi?id=71041
[20:44:19] <hedonil>	 Coren: would be neat
[22:33:06] <wikibugs>	 3Tool Labs tools / 3Commons Delinker: COMMONSDELINKER-19 CommonsDelinker should not replace images if a different version of the target exists locally - 10https://bugzilla.wikimedia.org/61656#c2 (10Stefan2) For the record: There is a complaint about this at [[c:COM:AN#Bot User:CommonsDelinker and Voting boot...
[22:42:22] <wikibugs>	 3Wikimedia Labs: Swap two shell usernames - 10https://bugzilla.wikimedia.org/71083 (10Andre Klapper)