[05:12:33] 10DBA, 10Community-Tech, 10Expiring-Watchlist-Items: Watchlist Expiry: Release plan [rough schedule] - https://phabricator.wikimedia.org/T261005 (10Marostegui) Let's use this task for communication, that works for me [05:25:48] 10DBA, 10Cloud-VPS, 10cloud-services-team (Kanban): Drop openstack databases from m5-master - https://phabricator.wikimedia.org/T261152 (10Marostegui) p:05Triage→03Medium Thanks Andrew for cleaning this up. Some of those databases are empty. The other ones, indeed are showing no activity for around a mon... [05:45:02] 10DBA, 10Patch-For-Review: Upgrade m5 to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T260324 (10Marostegui) [06:24:31] 10DBA, 10Patch-For-Review, 10cloud-services-team (Kanban): Upgrade m5 to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T260324 (10Marostegui) @Andrew and #cloud-services-team **I would like to failover this master maybe Monday 31st at 14:00 UTC? Would that work for you?** The procedure for m5... [06:44:04] 10DBA, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service, 10Patch-For-Review, 10User-Marostegui: DBA review for Echo push notification subscription tables - https://phabricator.wikimedia.org/T246716 (10Marostegui) >>! In T246716#6389990, @Mholloway wrote: > Hi @Marostegui, thanks for the... [07:39:15] 10DBA, 10Cloud-VPS, 10cloud-services-team (Kanban): Drop openstack databases from m5-master - https://phabricator.wikimedia.org/T261152 (10jcrespo) I will remove now the backup grants from the multiinstances hosts. FYI, the following backups will continue running: ` labsdbaccounts labswiki nodepooldb nova... [07:42:41] 10DBA, 10Cloud-VPS, 10cloud-services-team (Kanban): Drop openstack databases from m5-master - https://phabricator.wikimedia.org/T261152 (10jcrespo) I will also remove grants for: ` nodepooldb nova ` As those seem to no longer exist. [08:01:42] I've just done db1117:~$ systemctl restart prometheus-mysqld-exporter@m5.service [08:01:48] thanks [08:02:12] I thought I did it yesterday after recloning db1128 from it [08:03:01] are you working on dbstore1004? [08:03:25] it seems scraping is failing there too [08:03:43] I see yes, on the log [08:06:02] yes, I am with it [08:15:41] 10DBA, 10Patch-For-Review, 10User-Kormat, 10cloud-services-team (Kanban): Upgrade m5 to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T260324 (10Kormat) [09:06:11] 10DBA, 10ContentTranslation, 10Language-Team (Language-2020-July-September): Create notification-log table in Production (wikishared) - https://phabricator.wikimedia.org/T261190 (10KartikMistry) [09:14:01] 10DBA, 10ContentTranslation, 10Language-Team (Language-2020-July-September): Create notification-log table in Production (wikishared) - https://phabricator.wikimedia.org/T261190 (10Marostegui) p:05Triage→03Medium Please note that normally DBAs do not create tables, as this should be done during normal de... [09:33:47] 10DBA, 10ContentTranslation, 10Language-Team (Language-2020-July-September): Create notification-log table in Production (wikishared) - https://phabricator.wikimedia.org/T261190 (10Nikerabbit) The table will contain just a few timestamps. I can't think of any privacy implications to necessitate keeping it pr... [09:45:32] 10DBA, 10ContentTranslation, 10Language-Team (Language-2020-July-September): Create notification-log table in Production (wikishared) - https://phabricator.wikimedia.org/T261190 (10Marostegui) >>! In T261190#6408526, @Nikerabbit wrote: > The table will contain just a few timestamps. I can't think of any priv... [09:50:04] 10DBA, 10ContentTranslation, 10Language-Team (Language-2020-July-September): Create notification-log table in Production (wikishared) - https://phabricator.wikimedia.org/T261190 (10Marostegui) 05Open→03Resolved a:03Marostegui Table created in x1: ` ./section x1 | while read host port; do echo "$host:$... [09:58:24] 10DBA, 10Operations, 10serviceops, 10Goal, 10Patch-For-Review: Strengthen backup infrastructure and support - https://phabricator.wikimedia.org/T229209 (10jcrespo) [11:00:16] 10DBA, 10ContentTranslation, 10Language-Team (Language-2020-July-September): Create notification-log table in Production (wikishared) - https://phabricator.wikimedia.org/T261190 (10KartikMistry) Thanks a lot @Marostegui [11:10:07] 10DBA, 10Platform Engineering, 10Wikidata, 10mariadb-optimizer-bug: SpecialWhatLinksHere::showIndirectLinks on wikidatawiki needs optimizing - https://phabricator.wikimedia.org/T254688 (10Marostegui) 05Open→03Stalled From what I can see the optimizer is chosen either `PRIMARY` or `page_len` depending o... [11:10:25] 10DBA, 10Epic, 10Patch-For-Review: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10Marostegui) [11:10:27] 10DBA, 10Platform Engineering, 10Wikidata, 10mariadb-optimizer-bug: SpecialWhatLinksHere::showIndirectLinks on wikidatawiki needs optimizing - https://phabricator.wikimedia.org/T254688 (10Marostegui) [11:11:32] 10DBA, 10Platform Engineering, 10Wikidata, 10mariadb-optimizer-bug: SpecialWhatLinksHere::showIndirectLinks on wikidatawiki needs optimizing - https://phabricator.wikimedia.org/T254688 (10Marostegui) a:05aaron→03Marostegui [11:42:24] 10DBA, 10MediaWiki-Special-pages, 10Platform Engineering, 10Wikidata, and 2 others: SpecialWhatLinksHere::showIndirectLinks on wikidatawiki needs optimizing - https://phabricator.wikimedia.org/T254688 (10Reedy) [12:42:23] 10Blocked-on-schema-change, 10DBA: Extend sites.site_global_key on WMF production - https://phabricator.wikimedia.org/T260476 (10Marostegui) a:03Marostegui [13:39:19] 10DBA, 10Operations, 10Patch-For-Review, 10User-Kormat: DBA python layout - https://phabricator.wikimedia.org/T259516 (10Kormat) [13:39:59] 10DBA, 10Operations, 10Patch-For-Review, 10User-Kormat: DBA python layout - https://phabricator.wikimedia.org/T259516 (10Kormat) 05Open→03Resolved Mission accomplished. [13:41:18] https://binside.typepad.com/.a/6a00d834fd7f7353ef010536ccc557970c-pi [13:49:24] :D [14:16:41] 10DBA, 10Goal: Expand database provisioning/backup service to accomodate for growing capacity and high availability needs - https://phabricator.wikimedia.org/T257551 (10jcrespo) Most urgent hardware, dbprov1003 and dbprov2003 has been configured. New backups from dbprov1003 are running right now to validate th... [15:07:10] so what I would like to try is [15:07:20] stop slave; set replicate-wild-ignore-table = superset_staging.%; start slave; [15:07:21] full doc is at https://dev.mysql.com/doc/refman/5.7/en/replication-options-replica.html#option_mysqld_replicate-wild-do-table [15:07:31] before that, set session sql_log_bin=0 [15:07:33] do set GLOBAL [15:07:41] ack [15:07:58] that should fix everthing automatically [15:08:17] if the issue is only skipping data updates from a db [15:08:30] not need to sql_log_bin [15:09:32] then check the filter is correct, and we should send a patch so it persists after a restart [15:14:18] Slave_IO_Running: Yes [15:14:49] and I have [15:14:50] | replicate_wild_ignore_table | superset_staging.% | [15:14:59] that should do it [15:15:02] is sql running too? [15:15:32] what do you mean sorry? [15:15:48] (be patient with me :) [15:15:55] BTW, I am not sure if it should be superset_staging or superset\_staging, but it should work in both cases (_) could be interpreted as a wildcard [15:16:05] is Slave_SQL_running: Yes? [15:16:10] that is the one that broke [15:16:26] i see it on the alerts, so it worked [15:16:46] would you send a patch with that? [15:17:49] IO is "it is receiving data" SQL is "it is applying data" this broke the SQL thread (now fixed) for background [15:17:50] sure! So I think I need to create a copy of profile/mariadb/mysqld_config/misc_multiinstance.my.cnf.erb since I am using it for db1108 (I mean a analytics copy) [15:17:58] I was about to ask [15:18:05] it uses the same file? [15:18:43] it does yes, I didn't need anything different so I used it but it was not the wisest choice [15:18:46] going to amend it [15:18:47] yeah, creating one with the filter added would be the best option for now [15:19:03] in the future we may change how we handle .cnfs [15:19:10] but that should work for now [15:19:20] and just add that line as a config [15:19:34] yep going to do it now [15:20:05] as a suggestion, restarting the server to check it is applied correctly is what we do on other hosts with filters [15:20:11] as it is easy to make mistakes [15:20:14] but up to you [15:21:32] let me check if the filter should be \_ rather than _ [15:24:31] it should be \_, but it is a minor issue, I hope you never create a superset0staging db :-) [15:25:39] ahahah [15:32:53] checking [15:37:47] I am back, anything I can help with? I see you guys fixed it already <3 :) [15:38:15] 10DBA, 10Community-Tech, 10Expiring-Watchlist-Items: Watchlist Expiry: Release plan [rough schedule] - https://phabricator.wikimedia.org/T261005 (10ifried) Great, thanks! In that case, I'll post an update when we begin the process of deploying on March 14th. Thank you! [15:39:23] marostegui: all good! Going to stop again the slave, restart the replica and check the filter [15:40:04] great! :* [15:40:09] thanks jaime for taking over! [15:47:54] all good I see | replicate_wild_ignore_table | superset\_staging.% | after the restart [15:47:57] thanks a lot! [15:48:10] (and slave running fine) [15:48:20] cool [15:48:31] better test after a deploy, avoids headaches later [15:48:34] thanks elukey [15:48:46] I wasn't aware there was an extra db on the replica [15:48:54] *on the primary [16:41:42] 10DBA, 10Cloud-Services, 10Platform Team Initiatives (API Gateway): Prepare and check storage layer for api.wikimedia.org - https://phabricator.wikimedia.org/T246946 (10hnowlan) Just a heads-up - The parent task is going to be un-stalled soon hopefully. The wiki will be visible to the public, with some restr... [16:56:34] 10DBA, 10Community-Tech, 10Expiring-Watchlist-Items: Watchlist Expiry: Release plan [rough schedule] - https://phabricator.wikimedia.org/T261005 (10Marostegui) March? :-) [17:12:17] 10DBA, 10Community-Tech, 10Expiring-Watchlist-Items: Watchlist Expiry: Release plan [rough schedule] - https://phabricator.wikimedia.org/T261005 (10ifried) Aaaah, typo (the days sure do blend together lately)! I have corrected it to be September 14 :) [20:20:58] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-09-15) rack/setup/install db1150 (see note on hostname) - https://phabricator.wikimedia.org/T260817 (10Jclark-ctr) [20:22:06] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-09-15) rack/setup/install db1150 (see note on hostname) - https://phabricator.wikimedia.org/T260817 (10Jclark-ctr) [20:41:21] 10DBA, 10MediaWiki-Special-pages, 10Wikidata, 10Performance Issue, 10mariadb-optimizer-bug: SpecialWhatLinksHere::showIndirectLinks on wikidatawiki needs optimizing - https://phabricator.wikimedia.org/T254688 (10eprodromou) This seems like we're just waiting for the upgrades to happen, so there's not a r... [20:47:42] 10DBA, 10Cloud-Services, 10MW-1.35-notes (1.35.0-wmf.36; 2020-06-09), 10Platform Team Initiatives (MCR Schema Migration), and 2 others: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10daniel)