[07:02:37] 10DBA, 10Patch-For-Review: Migrate some s4 hosts to file per table - https://phabricator.wikimedia.org/T161088#3564939 (10Marostegui) db1097 is done [07:02:44] 10DBA, 10Patch-For-Review: Migrate some s4 hosts to file per table - https://phabricator.wikimedia.org/T161088#3564940 (10Marostegui) [07:06:18] 10DBA, 10Operations, 10ops-eqiad: BBU issues on db1055, RAID cache on WriteThrough - https://phabricator.wikimedia.org/T174265#3564954 (10Marostegui) As this is looking good after the whole night, I am going to start slowly repooling it back [08:46:38] 10Blocked-on-schema-change, 10DBA: Make several mediawiki table fields unsigned ints on wmf databases - https://phabricator.wikimedia.org/T89737#3565059 (10jcrespo) [08:47:28] 10Blocked-on-schema-change, 10DBA: Make several mediawiki table fields unsigned ints on wmf databases - https://phabricator.wikimedia.org/T89737#1044134 (10jcrespo) We can probably deploy this at the same time than T174509. [08:48:01] I have added some stuff to the backups documented, rephrased some sentences and things like that [08:48:13] *document [09:17:56] 10DBA, 10Analytics-Kanban, 10Patch-For-Review: Inconsistent default charset for analytics slaves - https://phabricator.wikimedia.org/T170952#3565122 (10Marostegui) >>! In T170952#3562297, @Ottomata wrote: > Hm, for now I think this is good, so that the slaves match each other. If we want everything to be a... [09:22:45] 10DBA, 10Data-Services: Design a method for keeping user-created tables in sync across labsDBs - https://phabricator.wikimedia.org/T156869#3565126 (10Marostegui) I agree with @bd808 too - let's continue the discussion on T173511. [10:07:35] I think db1048 complained yesterday about BBU [10:07:48] did it? :( [10:07:50] and labsdb1003 about temperature, but it could have been a false positive [10:08:13] I will try to retire db1048 now that it is 100% passive [10:08:21] and upgrade it to stretch [10:08:39] sounds good [10:22:41] 10DBA, 10Operations, 10Phabricator, 10ops-eqiad, 10Patch-For-Review: db1048 BBU Faulty - slave lagging - https://phabricator.wikimedia.org/T160731#3565221 (10jcrespo) 05Resolved>03Open I think this happened again yesterday- I will just modify this task into a decommissioning one. [10:24:02] 10DBA, 10Operations, 10Phabricator: Decom db1048 (BBU Faulty - slave lagging) - https://phabricator.wikimedia.org/T160731#3565225 (10jcrespo) [11:06:32] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: BBU issues on db1055, RAID cache on WriteThrough - https://phabricator.wikimedia.org/T174265#3565301 (10Marostegui) 05Open>03Resolved a:03Cmjohnson The original weight values have been set now. I will close this for now Thanks @Cmjohnson for help... [11:12:15] so this is what I am thinking- remove db1059 from s4 T161088 and use it as m3 T160731 [11:12:16] T161088: Migrate some s4 hosts to file per table - https://phabricator.wikimedia.org/T161088 [11:12:17] T160731: Decom db1048 (BBU Faulty - slave lagging) - https://phabricator.wikimedia.org/T160731 [11:12:42] jynus: and use db1097 as api? [11:13:11] anything similar ot https://gerrit.wikimedia.org/r/#/c/338996/1/wmf-config/db-eqiad.php [11:13:46] 1.5 for api, 2.5 for main [11:13:49] yeah [11:13:51] that sounds good [11:13:58] we can use db1097 then and it would be similar to that [11:14:09] but instead of db1091, db1097 [11:14:55] but for that, I have to fix s4 for pt-table-checksum [11:15:36] https://phabricator.wikimedia.org/T162593#3175313 [11:15:42] yeah, those are the ones found [11:15:48] but at the time we were only checking the tables with PK at the time [11:16:31] what did we clone db1068 from? [11:16:49] I think it was db1081 [11:16:57] let me check gerrit [11:17:06] yep [11:17:08] db1081: https://gerrit.wikimedia.org/r/#/c/349218/ [11:17:10] nice memory [11:19:10] 10DBA, 10Operations, 10Scoring-platform-team, 10cloud-services-team: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3369055 (10faidon) JFTR, since I didn't see it mentioned neither here nor in T142807, how impending is that decomm? Days/weeks/months? [11:25:34] 10DBA, 10Analytics-Kanban, 10Patch-For-Review: Inconsistent default charset for analytics slaves - https://phabricator.wikimedia.org/T170952#3565375 (10Marostegui) 05Open>03Resolved Closing this as per my chat with Luca Thank you! [14:14:23] 10DBA: Run pt-table-checksum on s4 (commonswiki) - https://phabricator.wikimedia.org/T162593#3565877 (10jcrespo) I am performing actual replaces on almost all hosts (the ones mentioned above, using db1053 as reference. The main issue (archive) are not missing or different updates, just different ids for the same... [15:13:48] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Platform-Team: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3566137 (10Anomie) [15:16:07] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Platform-Team: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3566174 (10Anomie) Regarding the row visibility in the `comment` table, something like this might work: ``` WHERE EXISTS(SELECT 1 FROM revision JOIN revision_... [15:31:22] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Platform-Team: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3566290 (10jcrespo) This looks like it requires much more work to maintain #data-services and #dumps-generation than #DBA, which seems mostly trivial. [15:32:30] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3566292 (10jcrespo) I am going to assume that dumps generation is backwards compatible, but adding it here as a heads... [15:41:22] 10DBA, 10Data-Services: LabsDB infrastructure pending work - https://phabricator.wikimedia.org/T153058#3566345 (10bd808) [15:41:25] 10DBA, 10Data-Services: Design a method for keeping user-created tables in sync across labsDBs - https://phabricator.wikimedia.org/T156869#3566343 (10bd808) 05Open>03declined Per rationale explained in T156869#3559945, this feature request is being denied. Write access will **not** be granted to end-user a... [16:39:48] 10DBA: Run pt-table-checksum on s4 (commonswiki) - https://phabricator.wikimedia.org/T162593#3566680 (10jcrespo) Sorry, I meant to say that I used the current master/older master/newer hosts as references (which were the ones that didn't fail last time). Archive has been uniformized on all s4 hosts, continuing n... [16:43:03] 10DBA, 10Operations, 10Scoring-platform-team, 10cloud-services-team: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3369055 (10chasemp) No definite date has been set as we are working on T173511 as the precursor to moving over quarry (and probably PAWS). I think we are tal... [17:24:05] 10DBA: Remove reader_feedback tables from wikis - https://phabricator.wikimedia.org/T174586#3566888 (10Reedy) [17:24:23] 10DBA: Remove ReaderFeedback tables from wikis - https://phabricator.wikimedia.org/T174586#3566903 (10Reedy) [17:25:55] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#2867068 (10Reedy) [17:26:09] 10DBA: Remove ReaderFeedback tables from wikis - https://phabricator.wikimedia.org/T174586#3566888 (10Reedy) [18:40:52] 10DBA, 10Data-Services, 10cloud-services-team, 10wikitech.wikimedia.org: move wikitech and labstestwiki to s3 (needs discussion) - https://phabricator.wikimedia.org/T167973#3351594 (10Krenair) > I cannot think any reason why wikitech's database (and the similar labstestwiki) is not part of s3 Historically,... [18:41:27] 10DBA, 10cloud-services-team, 10wikitech.wikimedia.org: Rename database labswiki to wikitech - https://phabricator.wikimedia.org/T171570#3469261 (10Krenair) Surely it'd be wikitechwiki anyway :p [18:56:50] rofl [19:15:59] 10DBA, 10Cloud-Services, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Replicate CentralNotice tables to Labs - https://phabricator.wikimedia.org/T135405#3567484 (10Base) Any news? [19:27:29] 10DBA, 10Data-Services, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Replicate CentralNotice tables to Labs - https://phabricator.wikimedia.org/T135405#3567557 (10Reedy) [19:28:26] 10DBA, 10Data-Services, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Replicate CentralNotice tables to Labs - https://phabricator.wikimedia.org/T135405#2297843 (10Reedy) [19:37:43] 10DBA, 10Data-Services: enwiki_p logging vs logging_userindex returning dramatically different results - https://phabricator.wikimedia.org/T168349#3567619 (10Krenair) [19:57:09] Reedy, ? [19:57:19] wikitechwiki