[07:08:05] 10DBA, 13Patch-For-Review: Rampant differences in indexes on enwiki.revision across the DB cluster - https://phabricator.wikimedia.org/T132416#3042144 (10Marostegui) db2048 finished correctly in less than 24 hours, so that is great news. I am going to go ahead and keep altering some more codfw hosts: ``` root... [07:59:24] 10DBA: Install and reimage dbstore1001 as jessie - https://phabricator.wikimedia.org/T153768#3042174 (10Marostegui) >>! In T153768#3040773, @Marostegui wrote: > In order to backup `dbstore1001:/srv/sqldata` we'd need 4.3T available (probably less as it would be compressed) but still, quite a big amount of space.... [08:06:03] FYI https://github.com/prometheus/mysqld_exporter/pull/183 [08:06:46] oh nice! [08:06:50] jynus will like that [08:08:02] yeah! that'll be nice indeed, not yet merged heh but I don't see why it couldn't [10:00:45] 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#2124305 (10Marostegui) Hello, I have been testing parallel replication + gtid_domain_id to run the alter table in... [10:16:26] 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3042418 (10jcrespo) > TL;DR: Looks like altering a big table with a different gtid_domain_id for that specific tran... [10:22:08] 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3042419 (10jcrespo) Probably the only other blocker would be to test that in our production, adding a primary key i... [11:59:00] jynus, marostegui: FYI, I'm upgrading openssl on database servers next, this includes various restarts of SSL-dependant system services (like openssh e.g), but ofc no updates for mysqld, these can trickle in via regular DBA maintenance [12:01:02] yes, no problem [12:01:13] only if there was like a binary incompatibility [12:01:19] which probably isn't [12:01:32] thanks for the heads up! [12:01:50] we can restart a host on codfw for a quick check if it was necessary [12:07:15] did you reimage any servers last week? then this change is already live [12:07:25] (on those set of servers) [12:08:00] the update is entirely binary-compatible, most of the other systems are upgraded already and no problems were found [12:08:50] yes, that is why there is no issue [12:10:28] ok :-) [13:22:06] 10DBA, 10Gerrit, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#3042690 (10Paladox) We can still do https://gerrit.wikimedia.org/r/#/c/336002/ since I doint see an urgency to ha... [15:03:12] https://phabricator.wikimedia.org/P4957 [15:03:28] :) [15:03:36] I have been using the neodymium one today for some checks :) [15:03:45] still need to get used to --no-dry-run [15:03:46] :) [15:04:07] should I make it run by default? [15:04:10] no [15:04:11] XD [15:04:13] too dangeours [15:04:17] maybe in the end [15:04:22] but we can wait for now [15:04:23] are you generating that table from the the dbhost files? [15:04:54] that is a real table, which should subtitute the need for files [15:05:29] ah db1011! missed that it was tendril [15:20:12] FWIW the mysqld-exporter pt-heartbeat support pull request I mentioned this morning got merged \o/ [15:20:37] that was fast! [15:21:45] aye, not sure when there'll be the next release but we can test it anyways [15:23:35] godog, do you have 1 minute? [15:23:59] it is about unix_auth, I will do the commits, but want your ok [15:25:51] jynus: sure, what changed wrt the configuration we have now in non-labs ? [15:26:55] well, the idea is that the unix socket is configured per server [15:27:01] and it will change with time [15:27:34] but I do not like the idea of setting it on hiera, because it will be out of sync with changes on mysql [15:28:32] either mariadb should write the port somewhere and prometheus read it from there or from /etc/my.cnf [15:28:37] *socket [15:29:00] the other part, is to eliminate the need for passwords- assuming prometheus executes with its own separate uid [15:29:33] mhh how the my.cnf is generated? we could generate a similar prometheus-specific one too like we sort of do now [15:30:01] the problem is that both clases are separate- and I do not want to change things twice [15:30:42] let me see how prometheus connects [15:31:14] ok! we could think about sharing one setting though, namely prometheus reading mariadb's setting [15:31:54] ideally that would be on hiera, but to share it between classes it has to be global [15:32:04] which is a problem we found in the past [15:32:50] maybe role mysql reads confing, and includes role mysql exporter? [15:33:17] that'd work too I think yeah [15:33:26] let me see where the exporter is added to the dbs [15:33:35] so $client_socket = '/tmp/mysql.sock' [15:33:46] would be set my a mariadb role [15:33:56] and client_password will be nullified [15:35:46] yes, I think that works, we get rid of "if os_version('debian >= jessie || ubuntu >= trusty') {" [15:36:08] and I pass socket as a parameter, which I also have to add to config [15:36:28] I think that is all, godog, thanks, I will send some patches and will ask you for the ok [15:37:12] jynus: fantastic, thanks! see also my comment re: labsdb1004/5 in the code review [15:37:27] yes, let's deploy that now, even if it will fail [15:37:42] and we can fix it on the review I mentioned [15:37:58] I may also get rid of those yaml files soon [15:40:26] there is one last catcha- and that is that we need to take care of multi-instance hosts [15:40:56] running several instances of the exporter, on different ports [15:42:24] yeah I'm sure that'll require more puppet-fu, hopefully not much more [15:42:55] let me start with the easies parts [16:11:22] 10DBA, 06Operations, 13Patch-For-Review: Followup for TLS MariaDB server roll-out - https://phabricator.wikimedia.org/T157702#3043273 (10jcrespo) [16:11:26] 10DBA, 10Monitoring, 06Operations: Create a check/calendar alert for MariaDB TLS certs - https://phabricator.wikimedia.org/T152427#3043272 (10jcrespo) [16:11:29] 10DBA, 06Operations, 13Patch-For-Review: Set up TLS for MariaDB replication - https://phabricator.wikimedia.org/T111654#3043274 (10jcrespo) [16:38:25] 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3043334 (10Marostegui) >>! In T130067#3042419, @jcrespo wrote: > Probably the only other blocker would be to test t... [16:58:46] 10DBA, 13Patch-For-Review: Deploy gtid_domain_id flag in our mysql hosts - https://phabricator.wikimedia.org/T149418#3043527 (10JAllemandou) @Marostegui : Hi, trying to keep an eye on that task - Do you think the analytics team plan on being able to use the new labsdb infra soon for testing and productionizati... [17:00:04] 10DBA, 06Labs, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3043530 (10JAllemandou) @Marostegui : Quick update on that one as well - How many shards are still to be done? Thanks ! [17:06:41] 10DBA, 06Labs, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3043562 (10Marostegui) >>! In T153743#3043530, @JAllemandou wrote: > @Marostegui : Quick update on that one as well - How many shards are still to be done?... [17:09:28] 10DBA, 13Patch-For-Review: Deploy gtid_domain_id flag in our mysql hosts - https://phabricator.wikimedia.org/T149418#3043566 (10Marostegui) Hey @JAllemandou, We will not be able to enable GTID soon as we would like to - we have found a bug with mariadb implementation: https://jira.mariadb.org/browse/MDEV-120... [17:09:43] 10DBA: enwiki database on db1065 missing rev_timestamp index - https://phabricator.wikimedia.org/T158454#3043572 (10EBernhardson) [17:09:46] 10DBA, 13Patch-For-Review: Rampant differences in indexes on enwiki.revision across the DB cluster - https://phabricator.wikimedia.org/T132416#3043570 (10EBernhardson) [17:11:49] 10DBA, 13Patch-For-Review: Rampant differences in indexes on enwiki.revision across the DB cluster - https://phabricator.wikimedia.org/T132416#3043576 (10EBernhardson) T158454 is about the rev_timestamp index on db1065's (vslow) enwiki database. Basically the CirrusSearch forceSearchIndex maintenance script, u... [17:19:14] 10DBA, 13Patch-For-Review: Deploy gtid_domain_id flag in our mysql hosts - https://phabricator.wikimedia.org/T149418#3043610 (10JAllemandou) @Marostegui : Thanks for the update, I'll continue monitoring in here and ask for headsup evey now and then :) Thanks [17:20:02] 10DBA, 06Labs, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3043611 (10JAllemandou) Thanks for the headsup @Marostegui :) [17:24:19] 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3043617 (10jcrespo) There is one thing that worries me more- in your #1 case, you have setup the slave by replicati... [19:00:23] 10DBA, 10fundraising-tech-ops: fundraising database tuning - https://phabricator.wikimedia.org/T158446#3044038 (10Jgreen) innodb_buffer_pool_size yep, I've increased that to 75-80% of system RAM. The tables are all innodb but the config dates back before we were fully converted. Looks like we should reduce ke... [21:19:43] Hi bd808! Are you here? [21:20:07] kind of :) [21:21:16] please open https://de.wikipedia.org/w/index.php?title=Wikipedia:Importwünsche&action=info [21:21:48] edit count: 17.323 edits [21:23:00] bd808: can you delete this page and restore the last 100 revisions? [22:21:52] doctaxon: I'm not a dewiki admin or a dba [22:22:34] it are too many revisions that a dewiki admin can handle it [22:22:47] yes, it needs a dba [22:24:05] doctaxon: it sounds like something that should go in Phabricator as a site request and/or dba issue. I have never messed with cleaning up pages with too many revs. [22:27:08] phabricator? [22:27:09] okay [23:18:04] We shouldn't muck about in the DB, nor does it need a DBA [23:18:09] Should use 'bigdelete' permission [23:22:12] RainbowSprinkles: where can I get bigdelete permission [23:23:36] *where -> how [23:23:36] Hmm, no group has it by default I don't think [23:24:12] Yeah, I don't see it granted other than on ckbwiki [23:24:27] But yeah, we'd want a group with bigdelete to perform the deletion then, doing it in the DB is risky/error-prone [23:24:42] (I thought the global group sysop had it for some reason) [23:24:53] Eh, sysadmins, not sysop [23:25:28] what about the permission group limit excepted [23:31:09] noratelimit isn't the same permission [23:31:30] That's about rate limiting on API actions, account creations, that sorta thing