[07:13:12] 10DBA, 13Patch-For-Review: Import S2,S6,S7,m3 and x1 to dbstore2001 and dbstore2002 - https://phabricator.wikimedia.org/T151552#2864163 (10Marostegui) a:03Marostegui [08:33:44] 10DBA, 13Patch-For-Review: Import S2,S6,S7,m3 and x1 to dbstore2001 and dbstore2002 - https://phabricator.wikimedia.org/T151552#2864212 (10Marostegui) I have now started to import S2 from db2064 into dbstore2001. [09:06:16] I am about to deploy an alter table on db1049 (s5 master) [09:19:10] 10DBA, 13Patch-For-Review: Fix PK on S5 dewiki.revision - https://phabricator.wikimedia.org/T148967#2864260 (10Marostegui) I am about to run the following ALTER on db1049, which is the master: ``` ./software/dbtools/osc_host.sh --host=db1049 --port=3306 --db=dewiki --table=revision --method=ddl --no-replicat... [09:23:41] jynus ^ I am still doubting whether it is worth the risk or not [09:24:22] all points out that it is safe, but it is the master [09:43:31] 10DBA, 13Patch-For-Review: dbstore1001 duplicate entry on s4 commonswiki.recentchanges - https://phabricator.wikimedia.org/T152766#2864285 (10Marostegui) I have deployed: https://gerrit.wikimedia.org/r/#/c/326124/ which fixes the small display issue we saw on Icinga. [09:47:20] 'lo DBAs, there's a disk space warning for dbstore1001 FYI [09:48:22] ema: I will take care of it :) [09:49:14] thanks! [09:53:25] that is waiting for new disks [09:53:48] I have cleaned some old stuff, which gave us 50G back only [09:53:57] We can always extend the volume if we like [09:54:03] to remove the alert I mean [13:58:05] jynus: just to make sure you have this in your radar: https://gerrit.wikimedia.org/r/#/c/326086/ [13:58:12] no rush to review it [14:06:47] 10DBA, 13Patch-For-Review: Fix PK on S5 dewiki.revision - https://phabricator.wikimedia.org/T148967#2864725 (10Marostegui) I will be running the above ALTER tomorrow morning around 7:45-8AM (UTC+1) as looks like it is the best time to avoid long running slow queries. [15:29:09] 10DBA, 13Patch-For-Review: Deploy gtid_domain_id flag in our mysql hosts - https://phabricator.wikimedia.org/T149418#2864919 (10Marostegui) I have deployed how we calculate gtid_domain_id (which is the same way as we calculate server_id for now, but they are now independent from each other) This is the changel... [15:54:12] 10DBA, 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Migrate existing labs users from the old servers, if possible using roles and start maintaining users on the new database servers, too - https://phabricator.wikimedia.org/T149933#2864970 (10jcrespo) Fixed. It turned out that you need GRANT OPTION an... [15:57:03] ^ that is interesting [16:04:24] you also need super if servers are in read only [16:04:33] will keep it read-write for now [16:04:52] if they are in read only you need super?? [16:06:01] yes, after all you need to edit the grant tables [16:06:10] ah right, that is true [16:06:30] the thing I learned [16:06:39] is that even if you have create user [16:06:56] you need grant option to edit things like max_user_connectios [16:07:15] on *.* [16:07:45] isn't enough with yourdb.* ?? [16:08:05] well, you could grant those, but not touch the global privileges [16:08:18] like change a password [16:08:46] so you need 3 things, create user, grant option on *.* and edit on mysql.* (or super) [16:09:07] I could, however [16:09:29] grant a role with admin option to several accounts [16:09:36] without needing an extra grant [16:09:48] GRANT role to user WITH ADMIN OPTION; [16:10:04] and now that user can grant the role to other users [16:10:26] I would assume grant role with grant option would have been confusing [16:10:28] is that the labsadmin role we were discussing earlier? [16:10:39] yes, I did not create it [16:11:03] I just left the labsdbuser, and let labsdbuser with admin for the admin accounts [16:11:25] labsdbuser I think now it is empty [16:11:43] but the good thing about roles is that you only have to edit it once to apply it to all users [16:11:59] yes, that is a great feature [16:12:09] specially if you handle dozens of users like labs :-) [16:12:12] we can add enwiki\_p SELECT later [16:26:48] 10DBA, 10Edit-Review-Improvements-RC-Page, 06Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016), 13Patch-For-Review: Implement functionality for RC page 'Experience level' filters - https://phabricator.wikimedia.org/T149637#2865081 (10Trizek-WMF) I agree with @tarlocesilion. Fewer than 10 edits and 4... [17:00:58] 10DBA, 10Edit-Review-Improvements-RC-Page, 06Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016), 13Patch-For-Review: Implement functionality for RC page 'Experience level' filters - https://phabricator.wikimedia.org/T149637#2865225 (10tarlocesilion) @Trizek-WMF, yes, you're right as for that list. Bu... [17:18:23] 10DBA, 10Edit-Review-Improvements-RC-Page, 06Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016), 13Patch-For-Review: Implement functionality for RC page 'Experience level' filters - https://phabricator.wikimedia.org/T149637#2865284 (10SBisson) These terms are used in code and URL parameters only. Inte... [18:01:43] 10DBA, 13Patch-For-Review: Import S2,S6,S7,m3 and x1 to dbstore2001 and dbstore2002 - https://phabricator.wikimedia.org/T151552#2865474 (10Marostegui) The import is about to finish in a couple of hours or so I believe. So db2064 will remain with replication stopped until tomorrow morning (it is depooled) [18:14:17] 10DBA, 13Patch-For-Review: db2034: investigate its crash and reimage - https://phabricator.wikimedia.org/T149553#2865533 (10Marostegui) We have inserted a new disk on slot 2 to check: 1) if the controller marks it as a predictive failure for the second time - which can be suspicious 2) we will do the crash te... [18:31:15] 10DBA, 13Patch-For-Review: dbstore1001 duplicate entry on s4 commonswiki.recentchanges - https://phabricator.wikimedia.org/T152766#2865588 (10jcrespo) 05Open>03Resolved Let's resolve this, reload commonswiki fresh from production when the new disks arrive. [18:32:22] 10DBA, 13Patch-For-Review: dbstore1001 duplicate entry on s4 commonswiki.recentchanges - https://phabricator.wikimedia.org/T152766#2865603 (10jcrespo) BTW, T152766#2860145 got fixed itself with time. [20:57:27] 10DBA, 13Patch-For-Review: db2034: investigate its crash and reimage - https://phabricator.wikimedia.org/T149553#2866283 (10Marostegui) The RAID finished and the disk is showing again predictive failure: ``` logicaldrive 1 (3.3 TB, RAID 1+0, OK) physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 600...