[06:44:02] 10DBA, 10DC-Ops, 10Operations, 10ops-codfw: (Need By: 2020-11-29) rack/setup/install db214[234] - https://phabricator.wikimedia.org/T267041 (10Marostegui) [06:46:22] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad, 10Patch-For-Review: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10Marostegui) I have merged the puppet changes needed for the initial installation (puppet for `insetup` and the partman recipe). Pending merg... [06:47:57] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad, 10Patch-For-Review: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10Marostegui) [06:53:21] 10DBA, 10DC-Ops, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need By: 2020-11-29) rack/setup/install db214[234] - https://phabricator.wikimedia.org/T267041 (10Marostegui) I have merged the puppet changes needed for the initial installation (puppet for insetup and the partman recipe). Pending merges f... [06:53:25] 10DBA, 10DC-Ops, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need By: 2020-11-29) rack/setup/install db214[234] - https://phabricator.wikimedia.org/T267041 (10Marostegui) [06:57:55] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Productionize clouddb10[13-20] - https://phabricator.wikimedia.org/T267090 (10Marostegui) 05Open→03Stalled The data population is blocked on {T260843} - stalling this [08:30:07] I am going to break replication on pc2007 to see what orchestrator does with the slave [09:35:34] 10DBA: Productionize db114[1-9] - https://phabricator.wikimedia.org/T252512 (10Marostegui) [09:35:49] 10DBA, 10Patch-For-Review: Relocate "old" s4 hosts - https://phabricator.wikimedia.org/T253217 (10Marostegui) 05Open→03Stalled Stalling as we are blocked on the migration to 10.4, the pending hosts are either masters or candidate masters. [09:54:58] 10DBA, 10Operations, 10ops-eqiad: db1139 memory errors on boot 2020-08-27 - https://phabricator.wikimedia.org/T261405 (10jcrespo) 05Resolved→03Open Is it possible one of the memory stick needs reseating? ` 462 - Uncorrectable Memory Error Threshold Exceeded (Processor 2, DIMM 6). The DIMM is mapped out... [09:57:22] 10DBA, 10Orchestrator: Investigate moving replicas around with Orchestrator doesn't result on skipped transactions - https://phabricator.wikimedia.org/T267133 (10Marostegui) Testing what happens if an intermediate master goes down. pc2007 had pc2010 as a slave. I stopped mysql on pc2010 and issued a manual re... [10:01:25] 10DBA, 10Orchestrator: Orchestrator doesn't use FQDN when manipulating replicas - https://phabricator.wikimedia.org/T267389 (10Marostegui) [10:01:37] 10DBA, 10Orchestrator: Orchestrator doesn't use FQDN when manipulating replicas - https://phabricator.wikimedia.org/T267389 (10Marostegui) p:05Triage→03Medium [10:22:49] 10DBA, 10Orchestrator: Orchestrator doesn't use FQDN when manipulating replicas - https://phabricator.wikimedia.org/T267389 (10Marostegui) This is the timeline: Stops mysql on pc2007 (pc2010's master): ` Nov 06 09:40:18 pc2007 mysqld[12415]: 2020-11-06 9:40:18 0 [Note] InnoDB: Starting shutdown... Nov 06 09:4... [10:38:31] 10DBA, 10Orchestrator: Orchestrator: Create basic documentation - https://phabricator.wikimedia.org/T266428 (10Marostegui) Self note: Make sure to add that `move-up` and `move-down` are NOT preferred ways of moving replicas around, but better to use `relocate`. Move-up and move-down use file:pos and so far I a... [10:38:39] 10DBA, 10Orchestrator: Orchestrator doesn't use FQDN when manipulating replicas - https://phabricator.wikimedia.org/T267389 (10Kormat) From looking at the code, it seems there's a `hostname_unresolve` table in the orchestrator db that could map from the bare hostname to "something else" (not clear to me yet).... [10:41:05] 10DBA, 10Orchestrator: Orchestrator doesn't use FQDN when manipulating replicas - https://phabricator.wikimedia.org/T267389 (10Kormat) ` case registerCliCommand("register-hostname-unresolve", "Instance, meta", `Assigns the given instance a virtual (aka "unresolved") name`): ` It looks like this is somethi... [10:46:02] 10DBA, 10Orchestrator: Orchestrator doesn't use FQDN when manipulating replicas - https://phabricator.wikimedia.org/T267389 (10Marostegui) It can maybe be related to: ` MySQLHostnameResolveMethod string // Method by which to "normalize" hostname via MySQL server. ("none"/"@@hostname"/"@@r... [12:05:18] 10Blocked-on-schema-change, 10DBA: Drop default of protected_titles.pt_expiry - https://phabricator.wikimedia.org/T267335 (10Marostegui) p:05Triage→03Medium [13:10:36] 10Blocked-on-schema-change, 10DBA: Drop default of ip_changes.ipc_rev_timestamp - https://phabricator.wikimedia.org/T267399 (10Ladsgroup) [13:10:48] marostegui: please don't hate me :D [13:11:40] on the bright side and thanks to Ammarpad, 2/3 of core tables have been migrated [13:18:30] 10DBA, 10Operations, 10Orchestrator, 10CAS-SSO, 10User-Kormat: orchestrator: Support SSO - https://phabricator.wikimedia.org/T266106 (10Marostegui) [13:18:47] Amir1: Even if I wanted, I couldn't hate you <3 [13:19:07] 10DBA, 10Operations, 10Orchestrator: orchestrator: Use ssl for talking to db servers - https://phabricator.wikimedia.org/T267401 (10Kormat) [13:21:34] 10DBA, 10Operations, 10Orchestrator, 10User-Kormat: orchestrator: Add service monitoring - https://phabricator.wikimedia.org/T266338 (10Marostegui) Thank you Daniel! This looks good for now, so far we are going to keep notifications disabled on the host as we are doing many changes still, some of which inv... [13:21:59] <3 [13:22:09] Let me know if I can be of any service [13:25:24] 10DBA, 10Operations, 10Orchestrator: orchestrator: Use ssl for talking to db servers - https://phabricator.wikimedia.org/T267401 (10Kormat) Looking at the code, it looks like this is what happens: - if MySQLTopologyUseMixedTLS is set, check if the host 'requires' ssl - if it can auth to the db host without s... [13:33:18] 10DBA, 10Operations, 10ops-eqiad: db1139 memory errors on boot (Issues continues after board change) 2020-08-27 - https://phabricator.wikimedia.org/T261405 (10jcrespo) [13:33:33] 10DBA, 10Operations, 10ops-eqiad: db1139 memory errors on boot (issue continues after board change) 2020-08-27 - https://phabricator.wikimedia.org/T261405 (10jcrespo)