[05:46:21] 10DBA, 10MediaWiki-User-management, 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Rename ipb_address index on ipb_address to ipb_address_unique - https://phabricator.wikimedia.org/T250071 (10Marostegui) Cool - thank you @daniel I will standby to change the above wikis till @Ladsgrou... [05:48:47] 10DBA: tl_namespace index on templatelinks is unique only in s8 - https://phabricator.wikimedia.org/T250060 (10Marostegui) a:03Marostegui [05:51:18] 10DBA: tl_namespace index on templatelinks is unique only in s8 - https://phabricator.wikimedia.org/T250060 (10Marostegui) [05:51:37] 10DBA: tl_namespace index on templatelinks is unique only in s8 - https://phabricator.wikimedia.org/T250060 (10Marostegui) codfw was done and looks like: ` root@db2079.codfw.wmnet[wikidatawiki]> show create table templatelinks\G *************************** 1. row *************************** Table: templat... [05:59:42] 10DBA: tl_namespace index on templatelinks is unique only in s8 - https://phabricator.wikimedia.org/T250060 (10Marostegui) [06:23:20] 10DBA: tl_namespace index on templatelinks is unique only in s8 - https://phabricator.wikimedia.org/T250060 (10Marostegui) [07:01:48] 10DBA: tl_namespace index on templatelinks is unique only in s8 - https://phabricator.wikimedia.org/T250060 (10Marostegui) [07:12:47] Amir1: around? [07:13:22] Amir1: we've got an issue with T250060 [07:13:22] T250060: tl_namespace index on templatelinks is unique only in s8 - https://phabricator.wikimedia.org/T250060 [07:13:27] I'm always around [07:13:36] Looks like we do have a query that forces it: SELECT /*! STRAIGHT_JOIN */ tl_from,tl_namespace AS `bl_namespace`,tl_title AS `bl_title`,page_title,page_namespace FROM `templatelinks` FORCE INDEX (tl_namespace),`page` WHERE (tl_from = page_id) AND ((tl_namespace = 4 AND tl_title = 'Database_reports/Identified_duplicates/2')) AND page_is_redirect = 0 ORDER BY tl_from LIMIT 501 [07:13:58] Let me check [07:14:29] so that needs changing to: tl_backlinks_namespace [07:17:08] I can add the index back if needed - the table is small, so shouldn't take long [07:17:15] but maybe it is easy to patch? [07:17:35] is the issue with db1096.mgmt known? [07:17:44] nope [07:18:09] jynus: would you mind taking a look? [07:18:25] CRITICAL - Socket timeout after 10 seconds [07:19:31] Amir1: I have added it to db1092 (api) it took 18 seconds [07:19:46] going to add it to db1104, the other api host [07:19:51] as the query seems only API related [07:20:04] so we can at least clear out the errors we are getting [07:20:23] done [07:20:28] marostegui: I will find it, it shouldn't be too hard [07:20:38] cool, let me know when done and I can kill it again on those two [07:22:50] db1096: local ipmi works, but remote one doesn't work either (in additiona to ssh) [07:23:09] 10DBA: tl_namespace index on templatelinks is unique only in s8 - https://phabricator.wikimedia.org/T250060 (10Marostegui) Index re-added to the API hosts db1092 and db1104 as they were the ones giving errors: ` root@db1104.eqiad.wmnet[wikidatawiki]> set session sql_log_bin=0; alter table templatelinks add index... [07:23:17] Amir1: actually we should remove the force [07:23:43] the FORCE INDEX (tl_backlinks_namespace) would be bad [07:23:47] it would do a full scan [07:24:00] mm [07:24:09] let me check something [07:24:24] trying a cold reset [07:25:36] Amir1: no, looks like forcing tl_backlinks_namespace is still slower than the older index, but not that bad [07:25:43] the reset worked, but still no connection- most likely a link issue [07:26:05] Amir1: but looks like the old index was there for a performance reason, at least on wikidata [07:27:00] I think I should remove the force index bit [07:27:05] wait a sec [07:27:30] Amir1: I misunderstood your task, and I didn't realise you only asked to remove the UNIQUE, not the index itself [07:27:31] :) [07:27:39] oh yeah [07:27:46] yeah, let me fix that [07:28:17] 10DBA: tl_namespace index on templatelinks is unique only in s8 - https://phabricator.wikimedia.org/T250060 (10Marostegui) [07:30:26] Amir1: but yeah, from what I can see the optimizer chooses the right index now, so the force might not be necessary, but let me fix this and then I can create a task for that, with a bit more of an analysis [07:33:25] marostegui: sorry for not being clear :( [07:33:53] Amir1: No worries at all, it was my fault [07:34:00] at least it was spotted fast and fixed :) [07:34:09] the only two hosts giving errors were API ones [07:34:14] marostegui: https://phabricator.wikimedia.org/T250652 [07:34:15] I am now fixing the rest which receive no queries like that [07:34:23] jynus: thanks will check in a sec [07:38:41] 10DBA: tl_namespace index on templatelinks is unique only in s8 - https://phabricator.wikimedia.org/T250060 (10Marostegui) 05Open→03Resolved Index re-added everywhere. There was a misunderstanding on whether the index had to remain and only the UNIQUE was to be removed. My bad!. Fixed it. All done On the ot... [07:38:43] 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, 10Wikimedia-Incident, 10WorkType-NewFunctionality: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459 (10Marostegui) [07:38:44] Amir1: all done ^ going to check the FORCE index removal now - will create a task if neeed in the end. [07:38:58] oh thanks <3 [07:39:05] We are below 500 drifts now \o/ [07:43:55] Amir1: not sure which tags to place on this new task [07:44:01] CPT? [07:44:16] I guess wikimedia-rdbms? [07:44:29] both I think [07:44:36] ok! [07:45:04] https://phabricator.wikimedia.org/T250654 [07:45:05] done [07:46:06] jynus: I have ack'ed the alert for db1096 [07:47:09] 10DBA, 10Operations, 10ops-eqiad: db1096 management interface unresposive remotelly, likely connectivity issue - https://phabricator.wikimedia.org/T250652 (10Marostegui) [07:47:22] follow up with dc-ops in case they need downtime [07:47:32] my bet is only on a loose cable, happened other times [07:47:36] yep, I guess we need to wait for the next scheduled DC visit [07:47:42] I added the DBA tag so I don't forget about this task [07:48:23] here is a review for you: https://gerrit.wikimedia.org/r/c/operations/puppet/+/590995 [07:53:09] And that's a +1 for you [07:53:43] I should have those backups working in less than an hour :-) [07:54:31] when that's done, will keep experimenting with mysqlbinlog --read-from-remote-server --stop-never --raw [07:54:39] btw, I set the priority to high just because it is a backup source, up to you to change it once you have moved them :) [07:54:51] but i guess that server needs a new mainboard or power supply or something [07:54:58] seems reasonable to me [07:56:13] let me add a disable notifications to that patch and then deploy [07:56:43] I disabled them for db1140 [07:56:53] yeah, but that won't work for new ones [07:57:00] ah you mean to db1095? [07:57:07] yes [07:57:23] ah ok cool [08:05:52] I see your change, sadly it will never apply as it needs a puppet run for icinga to see it :-( [08:07:29] ah yes, on both icinga and the host itself [08:07:29] hehe [08:07:43] I acke'ed them manually anyways yesterday [08:13:20] 10DBA, 10Schema-change: Remove image.img_deleted column from production - https://phabricator.wikimedia.org/T250055 (10Marostegui) I am going to delete this column from a s1 (enwiki), s4 (commonswiki) and s8 (wikidatawiki) host only and leave it like that for a week before going ahead for the rest. Just in case. [08:21:16] 10DBA, 10Schema-change: Remove image.img_deleted column from production - https://phabricator.wikimedia.org/T250055 (10Marostegui) db1089 (enwiki): ` mysql.py -hdb1089 enwiki -A -e "show create table image\G" *************************** 1. row *************************** Table: image Create Table: CREAT... [08:21:52] 10DBA, 10Schema-change: Remove image.img_deleted column from production - https://phabricator.wikimedia.org/T250055 (10Marostegui) [08:33:38] mw1311.mgmt down, too, will check if on the same rack [08:34:13] A6 both [08:36:40] Just those two hosts? [08:36:45] We recently had a switch mgmt crashing [08:36:47] Let me check where it was [08:36:57] https://phabricator.wikimedia.org/T249309 C6 apparently [08:37:50] 3 hosts on A6 flopping [08:38:03] ganeti1006 too [08:38:55] now back up [08:39:45] although still cannot ssh to db1096 [08:41:09] that's weird [08:42:17] it works now [08:43:48] did you do anything? [08:44:21] nope [08:44:26] it comes and goes [08:44:33] maybe worth pinging arzhel [08:44:36] to review switch logs? [08:44:45] not sure if he is around yet [08:44:59] I will add him to the task [08:45:00] he is [08:45:08] he is in EU these days [08:45:47] i don't see him on irc [08:46:06] ah, saw him [08:46:10] I was blind [08:49:17] 10DBA, 10Operations, 10ops-eqiad: db1096 management interface unresposive remotelly, likely connectivity issue - https://phabricator.wikimedia.org/T250652 (10jcrespo) I saw a few hosts flop down and up its host up status on icinga, all on A6: in additiona to db1096.mgmt, mw1311.mgmt and ganeti1006.mgmt CC @a... [08:49:39] almost sure it is network now [08:53:17] Re: T234826 just to be clear, it was just an idea, but the loop is easily closable by exporting the tables to text format before upgrade [08:53:17] T234826: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 [08:53:24] 0:-D [08:53:50] for example, it could be tested on a test host (db2098?) to check size differences [08:54:06] as a test to check performance only [08:54:59] yep [08:55:21] but we still need to know whether that host can be offline for a few hours or whatever [08:55:30] anyways, I don't think we have the time to do that as now to be honest [08:56:07] I do have the time- for a test [08:56:21] I am waiting for db1095 to be fully setup [08:56:23] That's up to you and Luca I would say then [08:57:52] I enabled rocksdb on db2114 [08:58:11] not db2098 [08:58:19] which is the test host on codfw? [08:58:37] I don't know [08:58:41] db2102 [08:58:58] I don't know if that runs 10.4 [08:59:00] I don't think so [08:59:18] so where did you test 10.4 replication, wasn't that? [08:59:24] on codfw [08:59:45] No, we tested it on eqiad's test host [09:02:51] can I reimage to buster db2102? (technically, it is one of "my" hosts, if that is a "thing") [09:03:01] :-P [09:03:08] sure [09:06:03] marostegui: btw we should drop the beat [09:06:08] the beast I mean [09:14:27] was there an epic task about buster upgrade of database-related hosts? If not, can I create one? [09:15:21] Amir1: Yeah, we can do a couple of hosts [09:15:33] Amir1: but considering that this is a very short week, we shouldn't do it all I think [09:15:43] even if the rename effectively works as a drop anyways [09:15:55] you've not seen any errors, right? (I haven't) [09:16:59] marostegui: no, I haven't seen any errors, yeah, I think only a couple of hosts would be okay. Why this week is short? [09:17:19] I know it is but I don't know why [09:17:33] Amir1: WMF staff holiday [09:18:35] Amir1: check the calendar for this week https://wikitech.wikimedia.org/wiki/Deployments [09:19:21] 10DBA, 10Patch-For-Review: Test MySQL 8.0 with production data and evaluate its fit for WMF databases - https://phabricator.wikimedia.org/T193226 (10jcrespo) 05Stalled→03Resolved a:03jcrespo There was limited testing on db1114- replication worked for a long time, and in general data storage. It will requ... [09:19:23] 10DBA, 10MediaWiki-General, 10Operations: Evaluate and decide the future of relational datastore at WMF after the upgrade of MariaDB 10.1 is finished - https://phabricator.wikimedia.org/T193224 (10jcrespo) [09:20:35] thanks [09:23:49] 10DBA, 10Epic: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10jcrespo) [09:23:58] 10DBA, 10Epic: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10jcrespo) p:05Triage→03High [09:24:38] 10DBA, 10MediaWiki-General, 10Operations: Evaluate and decide the future of relational datastore at WMF after the upgrade of MariaDB 10.1 is finished - https://phabricator.wikimedia.org/T193224 (10jcrespo) 05Open→03Resolved a:03Marostegui As per meetings, the decision was: no decision. Closing this in... [09:24:46] 10DBA, 10Operations, 10ops-eqiad: db1096 management interface unresposive remotelly, likely connectivity issue - https://phabricator.wikimedia.org/T250652 (10ayounsi) p:05Medium→03High [09:25:35] 10DBA: Test MariaDB 10.4 in production - https://phabricator.wikimedia.org/T242702 (10jcrespo) [09:25:37] 10DBA: Remove deprecated status options from grafana in mariadb 10.4 - https://phabricator.wikimedia.org/T244696 (10jcrespo) [09:25:39] 10DBA, 10Epic: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10jcrespo) [09:25:41] 10DBA, 10MediaWiki-General, 10Operations: Evaluate and decide the future of relational datastore at WMF after the upgrade of MariaDB 10.1 is finished - https://phabricator.wikimedia.org/T193224 (10jcrespo) [09:26:50] 10DBA, 10Epic: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10jcrespo) [09:26:52] 10DBA: Test MariaDB 10.4 in production - https://phabricator.wikimedia.org/T242702 (10jcrespo) [09:27:35] 10DBA, 10Upstream: Possibly disable optimizer flag: rowid_filter on 10.4 - https://phabricator.wikimedia.org/T245489 (10jcrespo) [09:27:37] 10DBA, 10Epic: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10jcrespo) [09:27:39] 10DBA: Test MariaDB 10.4 in production - https://phabricator.wikimedia.org/T242702 (10jcrespo) [09:27:43] 10DBA, 10Upstream, 10mariadb-optimizer-bug: Slow query on 10.4: SpecialRecentChanges::doMainQuery - https://phabricator.wikimedia.org/T246069 (10jcrespo) [09:27:45] 10DBA: Test MariaDB 10.4 in production - https://phabricator.wikimedia.org/T242702 (10jcrespo) [09:27:55] 10DBA, 10Upstream: Events set to SLAVESIDE_DISABLED when upgrading from 10.1 to 10.4 - https://phabricator.wikimedia.org/T247728 (10jcrespo) [09:27:57] 10DBA: Test MariaDB 10.4 in production - https://phabricator.wikimedia.org/T242702 (10jcrespo) [09:27:59] 10DBA, 10Epic: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10jcrespo) [09:28:48] 10DBA, 10Operations, 10ops-eqiad: db1096 management interface unresposive remotelly, likely connectivity issue - https://phabricator.wikimedia.org/T250652 (10ayounsi) As the mgmt switch is 9yo and there are no errors on the msw1-eqiad side I'd say let's replace it. [09:28:50] 10DBA: Reimage labsdb1011 to Buster and 10.4 - https://phabricator.wikimedia.org/T249188 (10jcrespo) [09:28:52] 10DBA, 10Epic: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10jcrespo) [09:28:54] 10DBA, 10MediaWiki-General, 10Operations: Evaluate and decide the future of relational datastore at WMF after the upgrade of MariaDB 10.1 is finished - https://phabricator.wikimedia.org/T193224 (10jcrespo) [09:30:18] 10DBA, 10Data-Services, 10Packaging, 10Patch-For-Review: Upgrade wmf-pt-kill package for Debian Buster - https://phabricator.wikimedia.org/T248843 (10jcrespo) [09:41:34] ^will comment on all this on meeting :-D [09:42:39] 10DBA, 10Operations, 10ops-eqiad: msw1-a6-eqiad flopping up and down mgmt connections on A6 - https://phabricator.wikimedia.org/T250652 (10jcrespo) [09:42:51] 10DBA, 10Operations, 10ops-eqiad: msw1-a6-eqiad flopping up and down mgmt connections on A6 - https://phabricator.wikimedia.org/T250652 (10jcrespo) [09:48:02] I've added some updates to etherpad, as I am not sure the doc is safe to udpate [09:49:00] yeah, I don't know if we need to them there or not [09:49:10] yeah :-( [09:49:12] apparently it should've been reseted [09:49:17] From what I read on other channels [09:49:28] But maybe we can just add all the stuff there [09:49:35] as we are not going to bold anything anyways [09:49:35] we can do etherpad for now until we know [09:49:39] ah [09:49:41] ok [09:49:50] in any case, we will talk soon :-D [09:49:53] I am not planning to bold anything at least [09:50:06] I wanted to bold the ES setup [09:50:14] it is a nice milestone [09:50:22] ES set up? [09:50:26] ES backups [09:50:29] ah [09:50:32] that yes of course [09:50:39] although again [09:50:45] not sure if it really belongs there [09:50:48] beacuse it is a goal [09:50:56] all too confusing :-D [09:50:56] worth saying anyhow [09:51:47] 10DBA, 10Epic, 10Patch-For-Review: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10jcrespo) [12:36:05] 10DBA, 10Epic, 10Patch-For-Review: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10Marostegui) [12:42:28] 10DBA, 10Schema-change: Remove image.img_deleted column from production - https://phabricator.wikimedia.org/T250055 (10Marostegui) db1101 (wikidatawiki): ` root@db1101.eqiad.wmnet[wikidatawiki]> show create table image\G *************************** 1. row *************************** Table: image Create... [12:42:59] 10DBA, 10Schema-change: Remove image.img_deleted column from production - https://phabricator.wikimedia.org/T250055 (10Marostegui) [13:06:09] running the 2 missed (monday) backups now, other than that, T250602 should be mitigated and handed over to dcops [13:06:10] T250602: db1140 (backup source) crashed - https://phabricator.wikimedia.org/T250602 [13:06:16] cool [13:06:23] I have bolded it on today's doc [13:06:30] yeah [13:06:38] you talk about or I do? [13:06:38] bold == blocked on others [13:06:52] you can talk about that and I can talk about es? [13:06:58] sounds good [13:14:32] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: db1140 (backup source) crashed - https://phabricator.wikimedia.org/T250602 (10jcrespo) a:05jcrespo→03wiki_willy Service failover done, this is now 100% on #dc-ops side for vendor handling, as per description and initial triage by Manuel. No production service... [13:16:18] 10DBA, 10Schema-change: Remove image.img_deleted column from production - https://phabricator.wikimedia.org/T250055 (10Marostegui) db1081 (commonswiki): ` root@cumin1001:~# mysql.py -hdb1081 -e "show create table commonswiki.image\G" *************************** 1. row *************************** Table:... [13:16:54] 10DBA, 10Schema-change: Remove image.img_deleted column from production - https://phabricator.wikimedia.org/T250055 (10Marostegui) [13:19:32] also we may want to discuss also previous week's incidents, I don't know if those were done already [13:19:56] They were there on the previous docs [13:20:12] I know [13:20:13] The IR are being written [13:20:21] pc actually is in review already [13:20:23] not sure if people will want to comment [13:20:45] I am going to add the table one [13:24:55] have you copied from etherpad yet? [13:25:03] I was about to [13:25:20] but I am confused, because there is last week things that no longer apply [13:25:41] should we separate the ones from last week? [13:27:17] check m4rk's comments on -private [13:27:23] just add the new ones [13:27:33] yes [13:27:40] but if we should separate them [13:27:49] within the section [13:28:06] "problems with backup2002" and "backup2002 finished" may be confusing :-D [13:28:19] reword it as you wish [13:28:20] if you know what I mean :-D [13:32:27] I've added between brackets the name of people we are blocked on, if you are ok with taht [13:32:51] sure [13:37:17] 10DBA, 10Operations, 10ops-eqiad: msw1-a6-eqiad flopping up and down mgmt connections on A6 - https://phabricator.wikimedia.org/T250652 (10Cmjohnson) I have a spare Netgear switch already in storage, I will request access to the cage and complete this on Tuesday 4/21 @wiki_willy can we order a replacement t... [13:50:24] 10DBA, 10Schema-change: Remove image.img_deleted column from production - https://phabricator.wikimedia.org/T250055 (10Marostegui) [13:51:14] 10DBA, 10Schema-change: Remove image.img_deleted column from production - https://phabricator.wikimedia.org/T250055 (10Marostegui) s8 eqiad (empty table): [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1005 [] db1126 [] db1124 [] db1116 [] db1114 [] db1111 [] db1109 [] db1104 [x] db1101 []... [13:55:46] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1140 (backup source) crashed - https://phabricator.wikimedia.org/T250602 (10Cmjohnson) a:05wiki_willy→03Cmjohnson @Marostegui @jynus Before I can start a ticket with HPE some local troubleshooting has to be done. [14:01:50] 10DBA, 10Schema-change: Remove image.img_deleted column from production - https://phabricator.wikimedia.org/T250055 (10Marostegui) [14:02:12] 10DBA, 10Schema-change: Remove image.img_deleted column from production - https://phabricator.wikimedia.org/T250055 (10Marostegui) [14:50:48] 10DBA, 10Cognate, 10ContentTranslation, 10Growth-Team, and 10 others: Restart extension1 (x1) database primary master (db1120) - https://phabricator.wikimedia.org/T250701 (10Marostegui) [14:51:44] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb daemons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [15:26:00] 10DBA, 10Cloud-Services: Prepare and check storage layer for gomwiktionary - https://phabricator.wikimedia.org/T250706 (10Urbanecm) [15:37:00] 10DBA, 10Cloud-Services: Prepare and check storage layer for gomwiktionary - https://phabricator.wikimedia.org/T250706 (10Marostegui) p:05Triage→03Medium Thanks for the heads up - let us know when the database is created so we can sanitize it [15:54:03] 10DBA, 10Operations, 10ops-eqiad: msw1-a6-eqiad flopping up and down mgmt connections on A6 - https://phabricator.wikimedia.org/T250652 (10wiki_willy) a:03Cmjohnson [15:55:33] 10DBA, 10Operations, 10ops-eqiad: msw1-a6-eqiad flopping up and down mgmt connections on A6 - https://phabricator.wikimedia.org/T250652 (10wiki_willy) @cmjohnson - we have a refresh for the eqiad management switches scheduled to be ordered this quarter, so I'll check with Rob to see when those are coming in.... [15:57:00] 10DBA, 10Operations, 10ops-eqiad: msw1-a6-eqiad flopping up and down mgmt connections on A6 - https://phabricator.wikimedia.org/T250652 (10RobH) T249048 was approved last Friday (today being Monday), and my plan is to place the info into Coupa later today for ordering. I don't think we'll need another task... [16:28:20] 10DBA: Drop (and archive?) aft_feedback - https://phabricator.wikimedia.org/T250715 (10Reedy) [16:40:41] 10DBA: Drop (and archive?) aft_feedback - https://phabricator.wikimedia.org/T250715 (10Marostegui) p:05Triage→03Medium And it has indeed not being written for a long time: ` root@db1120:/srv/sqldata/enwiki# ls -lh aft_feedback.ibd -rw-rw---- 1 mysql mysql 848M Jul 25 2018 aft_feedback.ibd ` [16:42:03] 10DBA: Drop (and archive?) aft_feedback - https://phabricator.wikimedia.org/T250715 (10Marostegui) [16:42:07] 10DBA, 10Epic, 10Tracking-Neverending: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Marostegui) [16:47:56] 10DBA: Drop (and archive?) aft_feedback - https://phabricator.wikimedia.org/T250715 (10Marostegui) @jcrespo where would you like to get these tables archived at? ` root@db1120:/srv/sqldata# find . -name aft_feedback.ibd | xargs ls -lh -rw-rw---- 1 mysql mysql 60M Jul 25 2018 ./dewiki/aft_feedback.ibd -rw-rw-... [16:58:39] 10DBA: Drop (and archive?) aft_feedback - https://phabricator.wikimedia.org/T250715 (10jcrespo) If public, on dumps, if private, on archive backup system. If mixed, someone to sanitize them, then a copy to each place. [17:44:07] 10DBA: Drop (and archive?) aft_feedback - https://phabricator.wikimedia.org/T250715 (10Reedy) I can't find the task for dropping of the aft tables.... Only {T92739} [17:57:22] 10DBA, 10Security-Team: Drop (and archive?) aft_feedback - https://phabricator.wikimedia.org/T250715 (10Marostegui) #security-team advise on whether the content of this table can be public or it needs to be redacted? Thanks! [19:12:01] 10DBA, 10Security-Team: Drop (and archive?) aft_feedback - https://phabricator.wikimedia.org/T250715 (10Reedy) >>! In T250715#6072786, @Marostegui wrote: > #security-team advise on whether the content of this table can be public or it needs to be redacted? Thanks! Not sure offhand. https://github.com/wikimedi... [19:30:25] marostegui: Did we archive the older/other AFT tables? Or only privately? [19:50:50] btw marostegui we had a ~6 minute interval where db1091 was absolutely maxed out on network (even usual monitoring shows 1Gbps transmit sustained) and other s4 hosts were saturating some too