[00:25:33] 10DBA, 10wikitech.wikimedia.org: Move database for wikitech (labswiki) to a main cluster section - https://phabricator.wikimedia.org/T167973 (10Andrew) @LSobanski, I would very much like to see movement on this task (mostly because it will unblock T237773 which will remove a substantial thorn from my side). M... [06:18:14] 10DBA, 10wikitech.wikimedia.org: Move database for wikitech (labswiki) to a main cluster section - https://phabricator.wikimedia.org/T167973 (10Marostegui) @Andrew I am not sure this is a fairly small task, even if the process can looks simple (RO on wikitech + mysqldump + importing the dump on the destination... [06:35:46] 10DBA, 10Add-Link, 10Growth-Structured-Tasks, 10Growth-Team (Current Sprint), 10Patch-For-Review: Add Link engineering: Provide a mechanism for storing data about which link recommendations were rejected by the user - https://phabricator.wikimedia.org/T266446 (10Marostegui) >>! In T266446#6742393, @Tgr w... [06:56:52] 10DBA, 10Add-Link, 10Growth-Structured-Tasks, 10Growth-Team (Current Sprint), 10Patch-For-Review: Add Link engineering: Provide a mechanism for storing data about which link recommendations were rejected by the user - https://phabricator.wikimedia.org/T266446 (10Tgr) >>! In T266446#6746674, @Marostegui w... [07:15:23] 10DBA, 10Add-Link, 10Growth-Structured-Tasks, 10Growth-Team (Current Sprint), 10Patch-For-Review: Add Link engineering: Provide a mechanism for storing data about which link recommendations were rejected by the user - https://phabricator.wikimedia.org/T266446 (10Marostegui) Thanks @Tgr for the detailed e... [07:18:56] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Marostegui) [07:58:27] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Marostegui) [08:07:52] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Marostegui) [08:12:03] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Marostegui) [08:12:54] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Marostegui) [08:23:52] marostegui: I just replied to your email. Drop me a PM if anything needs clarifying as I'd rather not in a publicly logged channel. [08:24:42] RhinosF1: reading it now [08:25:28] RhinosF1: thanks - going to talk to you in private [08:30:45] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in logging table - https://phabricator.wikimedia.org/T270620 (10Marostegui) @RhinosF1 and myself just sync and it is fine to proceed as examknow extension isn't enabled, as Amir says, and it is likely it won't be in the future anyway... [08:31:37] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Marostegui) [08:43:59] 10DBA, 10SRE, 10ops-codfw, 10Patch-For-Review: db2140 crashed due to HW memory errors - https://phabricator.wikimedia.org/T271084 (10Marostegui) 05Open→03Resolved a:05Marostegui→03Papaul Data check was ok. Notifications enabled and host repooled. Thanks Papaul for replacing its memory. [09:00:14] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Marostegui) [09:05:41] 10DBA, 10cloud-services-team (Kanban): Move wikireplicas under the new sanitarium hosts (db1154, db1155) - https://phabricator.wikimedia.org/T272008 (10Marostegui) [09:05:53] 10DBA, 10cloud-services-team (Kanban): Move wikireplicas under the new sanitarium hosts (db1154, db1155) - https://phabricator.wikimedia.org/T272008 (10Marostegui) p:05Triage→03Medium [09:42:37] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Marostegui) [10:21:57] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Marostegui) [10:30:26] I'm going to do some restarts of backup hosts, this may momentarily generate some global alerts/metrics lost that are difficult to downtime individually [10:33:44] no worries [10:52:35] 10DBA, 10Data-Services: Prepare and check storage layer for altwiki - https://phabricator.wikimedia.org/T271982 (10LSobanski) p:05Triage→03Medium Thanks, let us know when the database is created, so we can sanitize it. [11:28:51] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in sites table - https://phabricator.wikimedia.org/T270621 (10Marostegui) [12:28:24] 10DBA, 10Patch-For-Review: Test upgrading sanitarium hosts to Buster + 10.4 - https://phabricator.wikimedia.org/T268742 (10Marostegui) [12:38:07] 10Blocked-on-schema-change, 10DBA: Drop default of revactor_timestamp - https://phabricator.wikimedia.org/T267767 (10Kormat) [12:48:58] 10Blocked-on-schema-change, 10DBA: Drop default of revactor_timestamp - https://phabricator.wikimedia.org/T267767 (10Kormat) s6 codfw progress: [] db2076 [] db2087:s6 [] db2089:s6 [] db2095:s6 sanitarium [] db2097:s6 dbstore [] db2114 candidate master [] db2117 [] db2124 [] db2129 master [] db2141 dbstore [12:49:37] 10Blocked-on-schema-change, 10DBA: Drop default of revactor_timestamp - https://phabricator.wikimedia.org/T267767 (10Kormat) [15:17:47] 10Data-Persistence-Backup, 10SRE, 10SRE-swift-storage: Depool codfw swift cluster - https://phabricator.wikimedia.org/T267338 (10jcrespo) [15:20:18] 10Data-Persistence-Backup, 10SRE, 10SRE-swift-storage, 10Traffic, 10netops: Depool codfw swift cluster - https://phabricator.wikimedia.org/T267338 (10jcrespo) Adding bblack (traffic) and ayounsi, as we have a tentative date for codfw, proposed by Filippo and Jaime: week of 15th. It may change as persiste... [15:20:32] 10Data-Persistence-Backup, 10SRE, 10SRE-swift-storage, 10Traffic, 10netops: Depool codfw swift cluster - https://phabricator.wikimedia.org/T267338 (10jcrespo) a:05jcrespo→03None [15:22:03] 10Data-Persistence-Backup, 10SRE, 10SRE-swift-storage, 10Traffic, 10netops: Depool codfw swift cluster - https://phabricator.wikimedia.org/T267338 (10jcrespo) [15:22:33] 10Data-Persistence-Backup, 10SRE, 10SRE-swift-storage, 10Traffic, 10netops: Depool codfw swift cluster - https://phabricator.wikimedia.org/T267338 (10jcrespo) [15:31:24] 10Data-Persistence-Backup, 10SRE, 10SRE-swift-storage, 10Epic, 10Goal: WMF media storage must be adequately backed up in a remote location - https://phabricator.wikimedia.org/T262668 (10jcrespo) [15:31:27] 10Data-Persistence-Backup, 10SRE, 10SRE-swift-storage, 10Goal: Research storage solutions for media backups - https://phabricator.wikimedia.org/T264190 (10jcrespo) 05Resolved→03Open p:05Medium→03High [17:39:49] 10DBA, 10Analytics-Radar: mariadb on dbstore hosts, and specifically dbstore1004, possible memory leaking - https://phabricator.wikimedia.org/T270112 (10razzi) [19:12:41] 10DBA, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10Cmjohnson) I ran into some issues along the way, these are taking a little longer to get the idrac's setup. I am sorry for the delay. [21:24:56] 10DBA, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10Jclark-ctr) @RobH @Cmjohnson DYV8773 is the ST not in netbox right now [22:43:32] 10DBA, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10wiki_willy) Thanks @Jclark-ctr, I just sent an email to Dell to figure out what's going on. >>! In T267043#6749435, @Jclark-ctr wrote: > @RobH @Cmjohnson DYV8773 is the...