[07:13:52] 10DBA, 06Operations, 10ops-codfw: db2011 disk media errors - https://phabricator.wikimedia.org/T149099#2762273 (10Marostegui) I am going to try to fix db2011 today. This server belongs to m2 shard. This is what I am going to do, in order to roll back if this box happens to fail. First, I am planning on stop... [07:59:50] 10DBA, 06Labs, 10Labs-Infrastructure, 10MassMessage, and 3 others: mysqld process hang in db1069 - S2 mysql instance - https://phabricator.wikimedia.org/T145077#2762301 (10Marostegui) Interesting update on the percona bug: ``` The stack trace shows TokuDB waiting for a row lock. There is a pull request... [08:24:58] 10DBA, 06Operations, 10ops-codfw: install new disks into dbstore2001 - https://phabricator.wikimedia.org/T149457#2762328 (10Marostegui) @jcrespo yo ok if I delete all the content of `dbstore2002:/srv/sqldata` today and move the snapshot from dbstore2001, start dbstore2002 and once that is done, we can contin... [08:36:44] I may have found the cause of db1073's lag [08:36:56] A disk failed yesterday when I pooled it [08:37:27] and iops fall suddenly when it is close to failure [08:38:16] Ah [08:38:24] The raid wasn't degraded? [08:38:31] no [08:38:33] now it is [08:38:35] So it was in that weird state when the controller doesn't catch it [08:38:38] until it gets hit [08:39:20] and look now at the graph: https://grafana.wikimedia.org/dashboard/db/mysql?var-dc=eqiad%20prometheus%2Fops&var-server=db1073 [08:39:29] (last 24 hours) [08:39:57] it makes sense now, if it is degraded it should go back to the "normal" performance [08:40:00] very interesting [08:40:09] I wonder if it would have failed at some pooint if you didn't pool it [08:40:21] I am going to depool it to confirm it [08:40:49] cool, can you check this (not now, whenever you have time): https://phabricator.wikimedia.org/T149457#2762328 [08:41:23] that is a yes [08:41:29] I actually suggested that [08:41:43] \o/ [08:41:47] Cool I will start now [08:41:50] Thanks [08:42:20] I will not backup the content of dbstore2002 [08:42:24] (it is broken anyways) [08:42:28] correct [08:42:39] thanks [08:46:11] 10DBA, 06Operations, 10ops-codfw: Degraded RAID on db2052 - https://phabricator.wikimedia.org/T149377#2762366 (10jcrespo) This is the command output: ``` $ hpssacli ctrl all show config Smart Array P420i in Slot 0 (Embedded) (sn: 0014380337E0DB0) Gen8 ServBP 12+2 at Port 1I, Box 1, OK array A (S... [08:46:49] 10DBA, 06Operations, 10ops-codfw: Degraded RAID on db2052 - https://phabricator.wikimedia.org/T149377#2762367 (10jcrespo) Sorry, wrong ticket- I will remove my previous comment. [08:49:50] jynus: I think you meant db2047, papaul mentioned you, but as I was online earlier I did it myself, hope that is fine [08:50:32] yes [08:50:42] I just got confused with the emails [08:50:53] I didn't read your answer until now [08:52:09] Ah :) [08:54:11] 10DBA, 06Operations, 10ops-codfw: install new disks into dbstore2001 - https://phabricator.wikimedia.org/T149457#2762437 (10Marostegui) The transfer is now happening between dbstore2001 and dbstore2002 [09:08:02] 10DBA, 10Cognate, 10Wikidata, 15User-Addshore, 03WMDE-QWERTY-Team-Board: Cognate DB review - https://phabricator.wikimedia.org/T148988#2762557 (10jcrespo) ``` DROP TABLE cognate_sites; DROP TABLE cognate_titles; DROP TABLE cognate_normalizations; /* *Create the tables */ CREATE TABLE IF NOT EXISTS... [09:08:05] 10DBA, 06Operations: Review Icinga alarms with disabled notifications - https://phabricator.wikimedia.org/T149643#2762558 (10Volans) @MoritzMuehlenhoff: done :wink: [09:11:18] jynus: I can chat about that ticket again if you would like! [09:11:28] not now [09:11:35] okay! I'll just comment on the ticket! [09:11:38] please answer on ticket and we can schedule a char at some time [09:14:46] 10DBA, 06Operations: Review Icinga alarms with disabled notifications - https://phabricator.wikimedia.org/T149643#2758740 (10jcrespo) A number of hosts have been decommissioned. As per new instructions, they are not removed from icinga (puppet) nor stopped, and role::spare doesn't do that either. > Leaving th... [09:22:40] 10DBA, 06Labs, 10Labs-Infrastructure, 10MassMessage, and 3 others: mysqld process hang in db1069 - S2 mysql instance - https://phabricator.wikimedia.org/T145077#2762576 (10jcrespo) That should not be it, because replication changes are commited in order, or in a single thread. However, if the issue happens... [09:27:38] 10DBA, 10Cognate, 10Wikidata, 15User-Addshore, 03WMDE-QWERTY-Team-Board: Cognate DB review - https://phabricator.wikimedia.org/T148988#2762577 (10Addshore) >>! In T148988#2762557, @jcrespo wrote: > ``` > CREATE TABLE IF NOT EXISTS cognate_titles ( > ``` > > You separate the trio site, namespace, title-... [09:42:50] 10DBA, 06Operations, 10ops-codfw: db2011 disk media errors - https://phabricator.wikimedia.org/T149099#2762624 (10Marostegui) The backup finished, and I was able to extract it, so proceeding now. Clearing the foreign config ``` root@db2011:~# megacli -CfgForeign -Scan -aALL There are 1 foreign configurati... [09:49:16] 10DBA, 10Cognate, 10Wikidata, 15User-Addshore, 03WMDE-QWERTY-Team-Board: Cognate DB review - https://phabricator.wikimedia.org/T148988#2762644 (10jcrespo) > I think the key here that may have been missed in our previous chat is that the normalization step is the same for all wikis. Ok, assuming that, wh... [09:49:30] 10DBA, 06Operations: Review Icinga alarms with disabled notifications - https://phabricator.wikimedia.org/T149643#2762648 (10Volans) >>! In T149643#2762562, @jcrespo wrote: > Disabling notifications is the only way to not alert. @jcrespo: I know, and there are also other reasons why make sense to disable noti... [09:53:18] 10DBA, 06Operations: Review Icinga alarms with disabled notifications - https://phabricator.wikimedia.org/T149643#2762654 (10jcrespo) But they have it, see: ``` 2016-09-21 09:01:06 Marostegui This server is going to be decommissioned - T146265 ``` This is for db1019. It is just not shown on the link you share... [09:54:19] 10DBA, 06Operations: Review Icinga alarms with disabled notifications - https://phabricator.wikimedia.org/T149643#2758740 (10Marostegui) >>! In T149643#2762654, @jcrespo wrote: > But they have it, see: > ``` > 2016-09-21 09:01:06 Marostegui This server is going to be decommissioned - T146265 > ``` > > This is... [10:06:07] 10DBA, 06Operations: Review Icinga alarms with disabled notifications - https://phabricator.wikimedia.org/T149643#2762679 (10Volans) @jcrespo @Marostegui: `db1019` it's fine, it has a scheduled downtime with a related comment and you can see it directly from the link I put in the description that it has one.... [10:07:16] 10DBA, 10Cognate, 10Wikidata, 15User-Addshore, 03WMDE-QWERTY-Team-Board: Cognate DB review - https://phabricator.wikimedia.org/T148988#2762681 (10Addshore) >>! In T148988#2762644, @jcrespo wrote: >> I think the key here that may have been missed in our previous chat is that the normalization step is the... [10:17:26] 10DBA, 06Operations, 10ops-codfw: db2011 disk media errors - https://phabricator.wikimedia.org/T149099#2762715 (10Marostegui) 32:4 finished the rebuild correctly Starting 32:7 ``` root@db2011:~# megacli -PdReplaceMissing -PhysDrv[32:7] -array3 -row1 -a0 Adapter: 0: Missing PD at Array 3, Row 1 is replaced.... [10:18:44] 10DBA, 06Operations: Review Icinga alarms with disabled notifications - https://phabricator.wikimedia.org/T149643#2762717 (10jcrespo) @Volans db1019 scheduled downtime will eventually expire (where the comments is), the comment I was referring to is not shown on the link. [10:19:04] 10DBA, 10Cognate, 10Wikidata, 15User-Addshore, 03WMDE-QWERTY-Team-Board: Cognate DB review - https://phabricator.wikimedia.org/T148988#2762718 (10Addshore) ``` SET @cognateNormalizedTitle = 'Apostrophe'''; SELECT ct_title AS title, ct_site AS site, cs_interwiki AS interwiki FROM cognate_normalizations R... [10:21:52] 10DBA, 10Cognate, 10Wikidata, 15User-Addshore, 03WMDE-QWERTY-Team-Board: Cognate DB review - https://phabricator.wikimedia.org/T148988#2762720 (10jcrespo) ``` RIGHT JOIN cognate_titles ON cn_raw = ct_title LEFT JOIN cognate_sites ON ct_site = cs_dbname ``` No thanks, please continue with your original p... [10:26:19] 10DBA, 06Operations: Review Icinga alarms with disabled notifications - https://phabricator.wikimedia.org/T149643#2758740 (10Joe) If you assign `role::spare` to a server and run puppet on the host and the icinga host, you should remove all alerts there. If that's not the case, it is a bug in puppet and we sho... [10:32:43] 10DBA, 06Operations: Review Icinga alarms with disabled notifications - https://phabricator.wikimedia.org/T149643#2762731 (10jcrespo) @Joe Apparently it removes it, if it didn't, it would show mysql alertss. But it keeps the common ones, which we do not want to show (plus potentially any running process that c... [10:35:38] 10DBA, 10Cognate, 10Wikidata, 15User-Addshore, 03WMDE-QWERTY-Team-Board: Cognate DB review - https://phabricator.wikimedia.org/T148988#2762746 (10Addshore) >>! In T148988#2762720, @jcrespo wrote: > ``` > RIGHT JOIN cognate_titles ON cn_raw = ct_title > LEFT JOIN cognate_sites ON ct_site = cs_dbname > ```... [10:36:13] 10DBA, 06Operations: Review Icinga alarms with disabled notifications - https://phabricator.wikimedia.org/T149643#2762748 (10Volans) >>! In T149643#2762717, @jcrespo wrote: > @Volans db1019 scheduled downtime will eventually expire (where the comments is), the comment I was referring to is not shown on the lin... [10:46:23] 10DBA, 06Operations, 10ops-codfw: db2011 disk media errors - https://phabricator.wikimedia.org/T149099#2762769 (10Marostegui) 32:7 finished fine. Starting 32:11 ``` root@db2011:~# megacli -PdReplaceMissing -PhysDrv[32:11] -array5 -row1 -a0 Adapter: 0: Missing PD at Array 5, Row 1 is replaced. Exit Code: 0... [10:46:48] 10DBA, 10Cognate, 10Wikidata, 15User-Addshore, 03WMDE-QWERTY-Team-Board: Cognate DB review - https://phabricator.wikimedia.org/T148988#2762770 (10jcrespo) Is there a reason why T148988#2744793, with some fixes based on your latest feedback could not work? I do not know what you are trying to do achieve... [10:58:21] 10DBA, 06Operations: Review Icinga alarms with disabled notifications - https://phabricator.wikimedia.org/T149643#2762823 (10jcrespo) >>! In T149643#2762748, @Volans wrote: >>>! In T149643#2762717, @jcrespo wrote: >> @Volans db1019 scheduled downtime will eventually expire (where the comments is), the comment... [11:07:57] 10DBA, 06Operations: Review Icinga alarms with disabled notifications - https://phabricator.wikimedia.org/T149643#2762856 (10MoritzMuehlenhoff) >>! In T149643#2762731, @jcrespo wrote: > Anyway, I would like to put down completely a host until it has been formatted, *then* it can be added to role:spare and be u... [11:30:46] 10DBA: Decommission db1042 - https://phabricator.wikimedia.org/T149793#2762930 (10jcrespo) [11:31:03] 10DBA: Decommission db1042 - https://phabricator.wikimedia.org/T149793#2762942 (10jcrespo) https://gerrit.wikimedia.org/r/319298 [12:09:44] 10DBA, 06Operations, 10ops-codfw: db2011 disk media errors - https://phabricator.wikimedia.org/T149099#2763089 (10Marostegui) The last disk finished fine and the RAID is now Optimal ``` Device Present ================ Virtual Drives : 1 Degraded : 0 Offline... [12:10:00] 10DBA, 06Operations: Puppetize tendril web user creation - https://phabricator.wikimedia.org/T148955#2763092 (10MoritzMuehlenhoff) p:05Triage>03Normal [12:30:38] 10DBA: Test InnoDB compression - https://phabricator.wikimedia.org/T139055#2763184 (10Marostegui) dbstore2001 has enwiki now compressed and the total dataset is `490G` ``` root@dbstore2001:/srv/sqldata/enwiki# mysql --skip-ssl information_schema -e "select TABLE_NAME,ROW_FORMAT,ENGINE,TABLE_ROWS,DATA_LENGTH fro... [12:34:57] remember the partitioning issue yourself mentioned [12:35:08] Re: db2034 [12:35:21] yep [12:35:25] also, as a reminder for you: triggers, for me replication [12:35:26] that is a good one [12:35:39] the triggers were removed [12:35:46] from all those hosts [12:35:47] (I think) [12:35:49] the replication was not restored :-) [12:35:50] Let me double check [12:36:37] yes, there are no triggers [12:36:56] then remind me replication next time, just in case :-) [12:37:04] I think you enabled it no? [12:37:16] I do not think so, or cannot remember [12:37:21] let me check [12:37:32] re: T139055 [12:37:33] T139055: Test InnoDB compression - https://phabricator.wikimedia.org/T139055 [12:37:42] Indeed, you didn't [12:37:48] note some of those tables are on the "to be dropped" list [12:38:05] in particular, non-innodb ones [12:38:20] That is good, didn't think of that [12:38:34] do not do anything as part of that [12:38:39] but just a heads up [12:39:24] yes yes, thanks :) [12:39:47] updates is large [12:39:58] and I think it is one of the to-be-dropped ones [12:43:22] it is 2.5G now (compressed) [12:44:19] 5gb * ~100 hosts adds up [12:45:00] although to be fair, I am more worried about: profiling | Fixed | MEMORY [16:08:54] 10DBA, 06Operations, 13Patch-For-Review: dbtree broken - https://phabricator.wikimedia.org/T149357#2763895 (10Joe) 05Open>03Resolved p:05Triage>03High [16:33:52] 10DBA, 10Cognate, 10Wikidata, 15User-Addshore, 03WMDE-QWERTY-Team-Board: Cognate DB review - https://phabricator.wikimedia.org/T148988#2763922 (10Addshore) When discussing with people here in the office having the relation to a second table as in T148988#2744793 seems odd, as said above, normalization on... [17:06:21] 10DBA, 10Cognate, 10Wikidata, 15User-Addshore, 03WMDE-QWERTY-Team-Board: Cognate DB review - https://phabricator.wikimedia.org/T148988#2764019 (10jcrespo) > A strictly unique key (possible PK) in the table would be a combination of all 4 fields, Do you see this as being too much? Would the id vs string f... [17:17:52] 10DBA, 10Cognate, 10Wikidata, 15User-Addshore, 03WMDE-QWERTY-Team-Board: Cognate DB review - https://phabricator.wikimedia.org/T148988#2764062 (10jcrespo) a:05jcrespo>03None [17:22:20] 10DBA, 06Collaboration-Team-Triage, 10Flow, 07Schema-change: Add primary keys to remaining Flow tables - https://phabricator.wikimedia.org/T149819#2764089 (10Catrope) [17:23:03] 10DBA, 06Collaboration-Team-Triage, 10Flow, 13Patch-For-Review, 07Schema-change: Add primary keys to remaining Flow tables - https://phabricator.wikimedia.org/T149819#2764108 (10Catrope) SQL: https://gerrit.wikimedia.org/r/#/c/319358/1/db_patches/patch-primary-keys.sql [17:32:01] 10DBA, 10Cognate, 10Wikidata, 15User-Addshore, 03WMDE-QWERTY-Team-Board: Cognate DB review - https://phabricator.wikimedia.org/T148988#2764155 (10Addshore) >>! In T148988#2764019, @jcrespo wrote: > I have no idea what you are doing (this is not a complain, I just do not understand it). Most of what the... [17:44:47] 10DBA, 10Cognate, 10Wikidata, 15User-Addshore, 03WMDE-QWERTY-Team-Board: Cognate DB review - https://phabricator.wikimedia.org/T148988#2764212 (10jcrespo) > the NULL values in the title field where the title is the same as the normalized title (to cut back on duplicated data) This is the kind of thing w... [17:56:01] 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations: Prepare and check production and labs-side filtering for olowiki - https://phabricator.wikimedia.org/T147302#2764229 (10jcrespo) [17:56:05] 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations: Create maintain-views user for labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T148560#2764226 (10jcrespo) 05Open>03Resolved a:03jcrespo I am assuming this as resolved because I think it is done and saw no complain it is not working fr... [18:07:31] 10DBA: Exceptional SQL time-out limits for volunteers doing special analysis - https://phabricator.wikimedia.org/T149478#2764269 (10jcrespo) 05Open>03Invalid I am closing this as invalid, as I didn't have more feedback from the original reporter. Please reopen a new one with something like "My tool has db is... [18:11:03] jynus: about? [18:11:31] sorry, what? [18:12:26] jynus: I put a questoin here (too late I think yesterday) re: maintainviews and permissions, the script is failing with [18:12:26] [08:54] pymysql.err.OperationalError: (1045, "Access denied for user 'maintainviews'@'::1' (using password: YES)") [18:12:42] we saw that in barcelona I recall and it was something you knew how to sort out perms wise [18:13:07] you are trying to connect to the ipv6 address 'maintainviews'@'::1' [18:13:14] you need to use the socket [18:13:21] which language is that? [18:13:39] it seems pymysql does not have the ability? https://github.com/PyMySQL/PyMySQL/issues/509 [18:13:41] sorry, I have not the code handy [18:13:42] python [18:13:57] it has, let me see [18:14:27] pymysql, strange option [18:14:29] but anyway [18:15:48] unix_socket="/tmp/mysql.sock" should be based on non-official manuals [18:15:50] 10DBA, 06Operations, 10ops-codfw: Degraded RAID on db2052 - https://phabricator.wikimedia.org/T149377#2764317 (10Papaul) a:05Papaul>03jcrespo Dick replacement complete. [18:16:12] ok trying it [18:16:17] set a user, pass and socket only [18:16:20] no host [18:17:13] yes, I can confirm it on the source code of the api [18:17:22] 'unix_socket=None' [18:17:48] let me confirm the socket location [18:18:23] yes, confirmed, at least on 1008 [18:18:26] that must be right I'm getting somehwere now :) at least a new error [18:18:29] pymysql.err.OperationalError: (1142, "SELECT command denied to user 'maintainviews'@'localhost' for table 'user'") [18:18:50] ah! [18:18:54] you are selecting user [18:18:55] that is new [18:19:12] there is a method valid the definer exists [18:19:15] pre view creation [18:19:23] SELECT 1 [18:19:23] FROM `mysql`.`user` [18:19:23] WHERE `user`=%s; [18:19:25] that is good [18:19:27] actually [18:19:41] but i on purpose took away that [18:19:54] thinking it would be dangerous if there was no reason [18:19:59] let me add it back [18:20:03] kk [18:20:18] are you testing on 8? [18:20:23] yes [18:20:23] to add it there first [18:20:42] BTW, local access is a bit broken there [18:20:47] for unrelated reasons [18:20:55] soon it will be, if you are root, you have access [18:21:09] no passwords needed [18:24:34] ok, try again, chasemp [18:24:54] seems like it all worked now :) [18:24:57] nice [18:25:05] I'll do a bit more validation but at the moment -- sweet [18:25:05] I will apply to the rest and document it [18:25:09] sure [18:25:11] thanks man, owe you an ice cream sunday [18:25:12] one last thing [18:25:30] we realize recently we need a temporary sanitarium [18:25:36] to move forward [18:25:46] (no argument but --) why? [18:25:52] as we have no other host, we were thinking on moving labsdb1008 to production [18:25:56] and use that [18:26:01] which means [18:26:16] starting using e.g. labsdb1009 for testing [18:26:22] heh ok [18:26:24] ok with that? [18:26:29] sure what's teh timeline? [18:26:32] I will notify it when we start doing that [18:26:39] no worries then [18:26:45] well in advance [18:26:59] not this week, probably next week [18:27:03] cool [18:27:06] we will load 9 with the same thing as 8 [18:27:08] I'm out most of next week anyway fyi [18:27:15] great all good to me [18:27:19] thanks! [18:27:32] and do not thank me when you are doing all the hard work [18:28:13] I feel the same way about you :D [18:28:19] good times dude [18:31:19] 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations: Create maintain-views user for labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T148560#2764404 (10jcrespo) Actually there was one issue with entry point (using the socket instead of the hostname), and another one with a check which required... [18:34:28] 10DBA, 06Operations, 10ops-codfw: Degraded RAID on db2052 - https://phabricator.wikimedia.org/T149377#2764412 (10jcrespo) [18:35:56] 10DBA, 06Operations, 10ops-codfw: Degraded RAID on db2052 - https://phabricator.wikimedia.org/T149377#2750453 (10jcrespo) Waiting for a complete rebuild to close this https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=db2052&service=HP+RAID [18:43:49] 10DBA, 13Patch-For-Review: Reimage dbstore2001 as jessie - https://phabricator.wikimedia.org/T146261#2764435 (10jcrespo) Manuel, I mentioned the labsdb1008 plan to Chase. No issue on doing that next week. We will have to ping him in advance, and I promised to move things there to labsdb1009 to continue the tes... [18:53:13] chasemp, I am blocking this on when you give the ok: https://phabricator.wikimedia.org/T149829 [18:54:00] gotcha [18:54:19] I will setup tomorrow labsdb1009 [18:54:46] for you to continue as normal (we will reimage when testing is finished) [18:55:35] FYI, we are starting to move and sanitize enwiki, commons and s3 soon [18:56:01] nice [19:00:58] 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, and 2 others: Move dbproxy1010 and dbproxy1011 to labs-support network, rename them to labsdbproxy1001 and labsdbproxy1002 - https://phabricator.wikimedia.org/T149170#2764508 (10jcrespo) Labs and DBAs agree this should go on. @robh @Cmjohnson Is this som... [19:18:24] jynus: on labsdb1001 trying to fix up olowiki_p I see: [19:18:25] pymysql.err.OperationalError: (1044, "Access denied for user 'maintainviews'@'localhost' to database 'olowiki_p'") [19:18:44] mmm [19:18:49] failed to create the database? [19:18:52] that should not happen [19:19:00] can you show the traceback? [19:19:03] let me check [19:19:40] https://phabricator.wikimedia.org/P4354 [19:19:56] GRANT ALL PRIVILEGES ON `%wik%\_p`.* TO 'maintainviews'@'localhost' [19:20:12] yeah so it's the CREATE DATABASE line [19:20:12] all should incude create database [19:20:19] but I may be wrong [19:20:34] jynus: you can run 'maintain-views --databases olowiki --debug' there to see [19:20:38] labsdb1001 that is [19:20:53] I believe you [19:20:57] I am just confused [19:21:24] I'm not actually sure whether I'd expect 'all privileges on ' to allow creation of new databases [19:21:32] actually, it does [19:21:34] probably worth double checking that behaves as we're expecting [19:21:35] okay [19:21:39] so it is something else [19:21:52] so that's the "first thing" it tries to do so to speak [19:21:54] dry run output would be [19:21:55] https://phabricator.wikimedia.org/P4354#21433 [19:22:40] "Removing 0 dbs as sensitive" [19:22:44] that just means [19:22:47] I only specified one DB [19:22:48] so it can do DROP DATABASE [19:22:49] and it wasn't sensitive [19:22:50] on all [19:22:56] or only _p? [19:22:58] jynus, what? [19:23:06] it doesn't DROP DATABASEs [19:23:14] jynus: that's in relation ot loading from mediawiki-config [19:23:22] not in relation to db operations at that stage [19:23:25] ok [19:23:35] I'll reword that debug [19:23:40] it is ok [19:23:50] I just wanted to avoid another error [19:24:00] I think it can drop _p databases [19:24:06] but not regular ones [19:24:17] I would expect yeah, no clue why the create fails tho [19:24:40] let me check a couple of things [19:24:46] and see if I can see why [19:25:14] BTW, DEFINER=viewmaster [19:25:25] I think will create it as viewmaster@% [19:25:35] I created viewmaster@localhost [19:25:53] hm, that part I didn't change from pre-existing [19:25:54] the other probably exist too, but I cannot guarantee it will work [19:26:03] or [19:26:05] guys [19:26:15] I can change the grants for @% [19:26:22] can we focus on the error please? [19:47:08] Krenair: I think all it took was a flush privs, care to poke at olowiki_p to see if it seems right to you? [19:47:09] https://phabricator.wikimedia.org/P4354#21439 [19:47:43] chasemp, that was my fault, I was the last day doing millions of test until i got it right [19:48:37] here million may be 5 or more [19:48:41] it happens :) after all this time tho, views actually maintainable? we all deserve a drink [19:49:03] my rule w/ my wife is if you give me a list of more than 3 things write it down :) [19:49:09] yeah looks good chasemp [19:49:54] Krenair: sweeet [19:53:15] so just gotta run this on the other couple of missing wikis [19:53:39] yep [19:53:41] maybe a pass over the existing ones? [19:53:44] document it on Add_a_wiki [19:54:07] and then I think we can call this a success? [19:54:29] then we should write down any other things we came across during these last few months that should be changed [19:54:35] I'm debating on running across pre-existing but I have no specific reason not too I think but yes agreed [19:54:42] gonig to hit the few outstanding and kind of see how it goes [19:54:46] at least a dry run [19:54:52] yes that's a good idea [19:56:48] 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations: Prepare and check production and labs-side filtering for olowiki - https://phabricator.wikimedia.org/T147302#2764842 (10chasemp) 05Open>03Resolved olowiki_p exists [19:59:50] the others were [19:59:50] https://phabricator.wikimedia.org/T126096 [20:00:17] https://phabricator.wikimedia.org/T142223 [20:00:42] Krenair: https://wikitech.wikimedia.org/w/index.php?title=Add_a_wiki&type=revision&diff=948302&oldid=892686 [20:00:50] first iteration [20:01:19] thanks [20:01:53] and hey I guess you can set up new labsdb servers with this now [20:01:56] once that changes to a whitelist [20:01:59] I have planned it [20:02:14] we will not be even blockers [20:02:33] not sure about the existing mysql users, they'll need to be copied in somehow I suppose [20:02:51] 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations: Enable access to Wikipedia Tulu (tcywiki) on labs replicas - https://phabricator.wikimedia.org/T142223#2764883 (10chasemp) 05Open>03Resolved a:03chasemp _p view variant should be good to go [20:02:55] 10DBA, 06Labs, 06Operations, 07Tracking: Database replication services (tracking) - https://phabricator.wikimedia.org/T50930#2764887 (10chasemp) [20:03:05] yes, do not worry, Krenair, that is part of the process, but at very late stages [20:03:15] 10DBA, 06Labs, 06Operations, 07Tracking: Database replication services (tracking) - https://phabricator.wikimedia.org/T50930#530760 (10chasemp) [20:03:19] 10DBA, 06Labs, 06Operations, 10Tool-Labs: Replicate wikimania2017wiki to labs - https://phabricator.wikimedia.org/T126096#2764890 (10chasemp) 05Open>03Resolved a:03chasemp _p view variant should be good to go [20:03:31] a few weeks ago we were not even able to copy wikis between hosts [20:04:03] you can see the status here: https://phabricator.wikimedia.org/T140788 [20:04:44] Krenair: afaik orphaned views are caught up, let me know if you see any others [20:04:54] chasemp, those new dbs look correct, thanks [20:05:17] chasemp, orphaned views? [20:05:28] I just meant, db's w/o views [20:05:42] where w/o is without :) [20:05:54] I apparently can't help myself when it comes to shorthand [20:05:56] do we have any databases without views exposed to labs users? [20:06:05] I know what w/o is [20:06:08] not to my knowledge [20:06:42] maybe you mean views referencing tables that no longer exist? [20:08:08] no I was just being confusing :) I meant I don't know of any remaining db's replicated to labsdb that still need a _p created [20:08:16] ah [20:08:23] there's some inconsistencies in the databases but I think it's covered in https://phabricator.wikimedia.org/T59617 [20:08:27] I can check [20:08:52] yeah jynus I'm not sure if you saw https://phabricator.wikimedia.org/T59617#2745255 [20:08:54] chasemp, oh, yeah, I don't know any either. Unless 'labswiki' exists on the labsdb servers? [20:09:00] hm [20:09:03] IIRC that one is waiting for replication [20:09:20] along with it's testing counterpart, theoretically [20:09:30] chasemp, that would need legal ok [20:09:35] but it's wikitech so not a priority [20:09:44] I may be wrong [20:10:00] but I belive the current watchlist_count definition was agreed with them [20:10:03] jynus, there are like 4 things there, which need legal ok? all of them? [20:10:12] so changing it I assume needs that [20:10:16] that is for the ok [20:10:24] then there is the technical way [20:10:25] I was mostly referring to MZMcBride's comment above [20:10:35] where somehow the table was created on enwiki but not others [20:10:43] when I mentioned inconsistencies [20:10:44] ah [20:10:45] the table is created [20:10:49] everywhere [20:10:58] I just created the view on enwiki as a test [20:11:02] manually [20:11:13] it should be dropped and all created [20:11:19] with this new script [20:11:25] there is an (empty) view in wikimania2017wiki_p [20:11:57] however, we should only make the current version available [20:12:08] and make new changes as part of a separate ticket [20:12:14] it doesn't appear to exist for tcywiki [20:12:31] also doesn't exist for olowiki [20:12:38] the view or the table? [20:12:43] gotta afk for a few gents [20:17:17] jynus, the view exists which implies the underlying table exists [20:18:01] what I mean is that I created it manually on enwiki [20:18:03] (as it was just created by a script which should've checked that) [20:18:04] as a test [20:18:12] the view, yes [20:18:17] but not anywhere else [20:18:30] the table is created automatically on all wikis [20:18:55] please double check that against tcywiki and olowiki [20:19:14] and fawiki, and probably the majority of them really [20:20:09] it uses /srv/mediawiki/dblists/s${shard}.dblist [20:20:13] from terbium [20:20:22] so if terbium is updated, it should be updated [20:20:29] there's a script that creates those tables? [20:20:32] yes [20:20:52] has it created olowiki.watchlist_count? [20:21:14] if olowiki is on s3, it should [20:21:26] indeed, I'm asking if it *has* [20:21:32] other thing is if there is a bug [20:22:15] because I see that wiki in s3.dblist [20:22:19] ERROR 1146 (42S02) at line 1: Table 'tcywiki.watchlist_count' doesn't exist [20:22:46] the problem is, I think it doesn't exist on production [20:22:58] tcywiki is the same [20:23:07] (in this respect) [20:23:10] 10DBA, 10MediaWiki-Database, 13Patch-For-Review, 07PostgreSQL, 07Schema-change: Some tables lack unique or primary keys, may allow confusing duplicate data - https://phabricator.wikimedia.org/T17441#2765623 (10Jdforrester-WMF) Migrating from the old tracking task to a tag for PostgreSQL-related tasks. [20:23:32] actually, ignoring labs, it indeed doesn't exist in production [20:23:36] so the script failed? [20:23:53] it succeeded to fail? [20:26:26] ERROR 1146 (42S02) at line 1: Table 'tcywiki.watchlist_count' doesn't exist [20:26:32] if fails every time [20:26:35] yeah [20:26:56] ah "CREATE TABLE IF NOT EXISTS watchlist_count" [20:27:08] I know the error [20:27:26] ^that like has a # at the beginning, for some reason [20:32:36] Generating table tcywiki.watchlist_count... / Saving table to db1069.eqiad.wmnet... [20:32:39] now it works [20:32:57] is this script puppetised somewhere? [20:33:03] no [20:33:14] ok [20:34:31] 10DBA, 06Labs, 10Labs-Infrastructure, 07Availability: Decide between proxysql and haproxy for labsdbproxy service - https://phabricator.wikimedia.org/T149844#2766141 (10jcrespo)