[02:11:04] 10DBA, 10Operations, 10ops-codfw: (Needed By 31st January) codfw: rack/setup/install es202[0-5].codfw.wmnet - https://phabricator.wikimedia.org/T241336 (10Papaul) [05:58:36] 10DBA: Remove partitions from revision table - https://phabricator.wikimedia.org/T239453 (10Marostegui) [06:13:22] 10Blocked-on-schema-change, 10Notifications, 10Patch-For-Review, 10Schema-change: Remove unused bundling DB fields - https://phabricator.wikimedia.org/T143763 (10Marostegui) Thanks @Reedy That index is existing on all the wikis as that index wasn't included on the original files of this task and hence it w... [06:13:52] 10Blocked-on-schema-change, 10DBA: Extend flow_ext_ref.ref_src_wiki - https://phabricator.wikimedia.org/T242688 (10Marostegui) p:05Triage→03Normal a:03Marostegui [06:22:37] 10Blocked-on-schema-change, 10DBA: Extend flow_ext_ref.ref_src_wiki - https://phabricator.wikimedia.org/T242688 (10Marostegui) ` officewiki *************************** 1. row *************************** Table: flow_ext_ref Create Table: CREATE TABLE `flow_ext_ref` ( `ref_src_object_id` binary(11) NOT... [06:22:46] 10Blocked-on-schema-change, 10DBA: Extend flow_ext_ref.ref_src_wiki - https://phabricator.wikimedia.org/T242688 (10Marostegui) [06:23:35] 10Blocked-on-schema-change, 10DBA: Extend flow_ext_ref.ref_src_wiki - https://phabricator.wikimedia.org/T242688 (10Marostegui) [06:26:49] 10Blocked-on-schema-change, 10DBA: Extend flow_ext_ref.ref_src_wiki - https://phabricator.wikimedia.org/T242688 (10Marostegui) [06:27:02] 10Blocked-on-schema-change, 10DBA: Extend flow_ext_ref.ref_src_wiki - https://phabricator.wikimedia.org/T242688 (10Marostegui) 05Open→03Resolved All done! [06:49:38] 10DBA: Test MariaDB 10.4 in production - https://phabricator.wikimedia.org/T242702 (10Marostegui) [06:51:13] 10DBA, 10Operations, 10ops-codfw: Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC - https://phabricator.wikimedia.org/T242481 (10Marostegui) p:05Triage→03High [06:51:39] 10DBA, 10Operations, 10ops-eqiad: (Needed by 31st January) eqiad: rack/setup/install es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10Marostegui) p:05Triage→03Normal [12:16:56] FYI we are upgrading openstack (uses m5-master database) and it involves database scripts for schema changes as usual [12:28:57] andrew just let me know that we may not need many schema changes this time [13:41:22] 10DBA, 10Operations, 10ops-codfw: Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC - https://phabricator.wikimedia.org/T242481 (10Marostegui) @Papaul can you double check (maybe even with the vendor) if there is a way to disable the 10G port for now? [13:41:38] 10DBA: Drop `echo_notification_user_hash_timestamp` index where it exists - https://phabricator.wikimedia.org/T242749 (10Reedy) [13:44:24] o/ the recent changes slaves have some different indexes correct? [13:50:19] addshore: correct, but we are in process of fixing that [13:50:42] addshore: https://phabricator.wikimedia.org/T239453 [13:50:45] it is only for the revision table [13:50:51] logging table was fixed already [13:51:10] ooh, okay [13:52:20] and the recentchanges table never had anything special? [13:52:44] Maybe a long time ago [13:52:48] But I cannot really remember [13:53:07] I mean, based on previous experience... [13:53:18] I'd be surprised if there wasn't some random indexes added somewhere on some hosts [13:53:32] yeah, that for sure XD [13:53:35] we have indexes called tmp [13:53:36] XD [13:53:51] tmp4lyf [13:53:54] xD [13:53:56] What I meant is that it doesn't have a different PK like logging or revision do now [13:54:02] okay [13:54:23] how can I find one of the hosts that is currently in the "recentchnages" group? I used to look in mw-config, but i see it is not there any more! [13:54:40] https://noc.wikimedia.org/dbconfig/eqiad.json [13:54:44] look for recentchanges [13:55:48] aaaah, json :) [13:56:01] for context, im staring at https://phabricator.wikimedia.org/T239192 trying to figure out what we might be able to do [13:56:22] only keep 1 days worth of RC data [13:57:53] heh, indeed [13:58:23] it looks like maybe it should be using a different index [14:00:25] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [14:05:04] yup, 2 minutes query down to a few seconds, https://phabricator.wikimedia.org/T239192 [14:05:20] im guessing we don't like doing FORCE INDEX anywhere though [14:05:29] especially on something called tmp_3 ;) [14:16:37] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1100 - https://phabricator.wikimedia.org/T241506 (10Jclark-ctr) Replaced Disk #0 [14:19:53] addshore: yeah we try to avoid that if possible, if it fixes something critical, then I guess we can go for it [14:20:24] looking at that ticket can you see any way to make mysql make the better decision itself? :) [14:22:21] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1100 - https://phabricator.wikimedia.org/T241506 (10Marostegui) Thanks - it is now rebuilding. I will close the task once it is finished ` PD: 0 Information Enclosure Device ID: 32 Slot Number: 0 Drive's position: DiskGroup: 0, Span: 0, Arm: 0 Enclosure po... [14:42:16] 10DBA, 10Operations, 10ops-codfw: Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC - https://phabricator.wikimedia.org/T242481 (10Marostegui) >>! In T242481#5801763, @MoritzMuehlenhoff wrote: > So, I digged into this a little: Interface auto setup not working if on... [14:49:49] 10DBA, 10Patch-For-Review: Test MariaDB 10.4 in production - https://phabricator.wikimedia.org/T242702 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` db1107.eqiad.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/202001141449_marostegui_... [15:07:30] 10DBA, 10Patch-For-Review: Test MariaDB 10.4 in production - https://phabricator.wikimedia.org/T242702 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db1107.eqiad.wmnet'] ` and were **ALL** successful. [15:11:02] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1100 - https://phabricator.wikimedia.org/T241506 (10Marostegui) 05Open→03Resolved All good - thank you! ` Number of Virtual Disks: 1 Virtual Drive: 0 (Target Id: 0) Name : RAID Level : Primary-1, Secondary-0, RAID Level Qualifie... [15:17:02] 10DBA, 10Operations, 10ops-codfw: Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC - https://phabricator.wikimedia.org/T242481 (10Marostegui) So adding the `01-` did the trick and es2020 is installing: ` append initrd=debian-installer/amd64/initrd.gz vga=normal aut... [15:18:00] 10DBA, 10Operations, 10ops-codfw: Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC - https://phabricator.wikimedia.org/T242481 (10Marostegui) Forgot to thank @MoritzMuehlenhoff for all the help and time with the troubleshooting [15:30:39] 10DBA, 10Operations, 10ops-codfw: Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC - https://phabricator.wikimedia.org/T242481 (10Papaul) On the phone with Dell support. [15:55:07] 10DBA, 10Operations, 10ops-codfw: Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC - https://phabricator.wikimedia.org/T242481 (10Papaul) Dell said that it is not possible to disable the 10Gb interface. [15:57:39] 10DBA, 10Operations, 10ops-codfw: Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC - https://phabricator.wikimedia.org/T242481 (10ayounsi) >>! In T242481#5798551, @Marostegui wrote: > - Enable to 10G even though it will go to a 1G switch port? Is that even possible... [15:59:20] 10DBA, 10Operations, 10ops-codfw: Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC - https://phabricator.wikimedia.org/T242481 (10MoritzMuehlenhoff) >>! In T242481#5802185, @ayounsi wrote: >>>! In T242481#5798551, @Marostegui wrote: >> - Enable to 10G even though i... [16:02:09] 10DBA, 10Operations, 10ops-codfw: Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC - https://phabricator.wikimedia.org/T242481 (10ayounsi) >>! In T242481#5802186, @MoritzMuehlenhoff wrote: > Is that because of different cables/connectors? Indeed, 1G switch ports ar... [16:05:12] 10DBA, 10Operations, 10ops-codfw: Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC - https://phabricator.wikimedia.org/T242481 (10Papaul) So I asked Dell if it was possible to replace the NIC card we have now with 2 separate NiC cards ( 1 x10Gb NiC and 1 x1GB NIC).... [16:05:38] marostegui: FYI I'm about to deploy debmonitor that will perform those add indexes to m2 master, lmk if this is not a good time [16:05:43] I'll keep an eye on the DB ofc [16:05:56] good from my side [16:06:00] thanks for the heads up [16:06:56] great, proceeding [16:08:44] {done} [16:18:05] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC - https://phabricator.wikimedia.org/T242481 (10MoritzMuehlenhoff) pxelinux has a generic option to pass the MAC in place when receiving the boot image as BOOTIF... [16:27:59] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC - https://phabricator.wikimedia.org/T242481 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` es20... [16:34:45] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC - https://phabricator.wikimedia.org/T242481 (10Marostegui) >>! In T242481#5802260, @MoritzMuehlenhoff wrote: > pxelinux has a generic option to pass the MAC in... [16:53:02] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC - https://phabricator.wikimedia.org/T242481 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['es2020.codfw.wmnet'] ` and were **ALL** successful. [16:54:20] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: (Needed By 31st January) codfw: rack/setup/install es202[0-5].codfw.wmnet - https://phabricator.wikimedia.org/T241336 (10Marostegui) [21:01:25] 10DBA, 10Schema-change: Drop `echo_notification_user_hash_timestamp` index where it exists - https://phabricator.wikimedia.org/T242749 (10Zoranzoki21)