[02:02:10] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Andrew) >>! In T239791#5770128, @Marostegui wrote: > @Andrew @Bstorm who in WMCS would be responsible for restarting mysql on these hosts? > ` >... [02:07:41] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Andrew) >>! In T234052#5785015, @Marostegui wrote: > @Andrew I have executed this alter on `labtestwiki` on clouddb2001-dev Thank you! I was on vacation :) [05:49:49] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) >>! In T239791#5796099, @Andrew wrote: >>>! In T239791#5770128, @Marostegui wrote: >> @Andrew @Bstorm who in WMCS would be responsible... [05:50:02] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [05:52:39] 10DBA: Compress new Wikibase tables - https://phabricator.wikimedia.org/T232446 (10Marostegui) [05:54:54] 10DBA: Remove partitions from revision table - https://phabricator.wikimedia.org/T239453 (10Marostegui) [05:56:45] 10Blocked-on-schema-change, 10DBA, 10User-RhinosF1: Extend flow_wiki_ref.ref_src_wiki - https://phabricator.wikimedia.org/T241387 (10Marostegui) [05:56:53] 10Blocked-on-schema-change, 10DBA, 10User-RhinosF1: Extend flow_wiki_ref.ref_src_wiki - https://phabricator.wikimedia.org/T241387 (10Marostegui) 05Open→03Resolved All done [06:05:32] 10DBA: Automatically populate tendril/zarcillo with the list of databases in the infrastructure - https://phabricator.wikimedia.org/T242571 (10Marostegui) [06:05:41] 10DBA: Automatically populate tendril/zarcillo with the list of databases in the infrastructure - https://phabricator.wikimedia.org/T242571 (10Marostegui) p:05Triage→03Normal [06:33:52] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) [06:52:00] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [08:10:09] 10DBA, 10Operations, 10ops-codfw: Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC - https://phabricator.wikimedia.org/T242481 (10Marostegui) >>! In T242481#5794886, @Papaul wrote: > @Marostegui I will focus more on troubleshooting this on the NIC level on Monday.... [09:37:46] 10DBA, 10Operations, 10ops-codfw: Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC - https://phabricator.wikimedia.org/T242481 (10Marostegui) I have tried to look thru the BIOS to find a way to disable the 10G capability but I have found nothing. On the installer... [09:38:53] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) [09:39:15] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) 05Open→03Resolved All done [09:54:44] 10DBA, 10Operations, 10ops-codfw: Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC - https://phabricator.wikimedia.org/T242481 (10MoritzMuehlenhoff) For a test maybe disable Puppet on the install* servers and add ` d-i netcfg/choose_interface select eno3 ` to th... [10:18:23] 10DBA, 10Operations, 10ops-codfw: Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC - https://phabricator.wikimedia.org/T242481 (10Marostegui) Thanks @MoritzMuehlenhoff - I have merged https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/563975/ to make sure we... [10:37:58] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Daimona) [10:51:54] 10DBA, 10Operations, 10ops-codfw: Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC - https://phabricator.wikimedia.org/T242481 (10Marostegui) The hack didn't work. It keeps choosing a different NIC than eno3, which unfortunately looks like the 10G one. I tried a di... [11:13:53] 10Blocked-on-schema-change, 10DBA, 10Wikidata: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language - https://phabricator.wikimedia.org/T237120 (10Marostegui) We might be doing a DC switchover in Q3, so we can wait till eqiad is passive to address the final alter on the current... [11:17:32] 10DBA: Switchover s8 primary database master db1109 -> db1104 - Date TBD - https://phabricator.wikimedia.org/T239238 (10Ladsgroup) [11:19:46] 10DBA: Switchover s8 primary database master db1109 -> db1104 - Date TBD - https://phabricator.wikimedia.org/T239238 (10Marostegui) [14:02:13] 10DBA, 10Operations, 10ops-codfw: Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC - https://phabricator.wikimedia.org/T242481 (10Marostegui) Last test done with @ayounsi: I have removed the drivers `tg3` and `bnxt_en` from the OS: ` /bin # ip addr 1: lo: 10DBA, 10Operations, 10ops-codfw: Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC - https://phabricator.wikimedia.org/T242481 (10Marostegui) Some more notes. The only way to ping the gateway seems to be using the 1G ports, and making sure the 10G aren't there. The... [16:47:00] 10DBA, 10Operations, 10ops-codfw: Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC - https://phabricator.wikimedia.org/T242481 (10MoritzMuehlenhoff) The 10G card is identical to what we have running fine on Stretch in e.g. ms-be2050 and I also validated there are n... [17:01:48] 10DBA, 10Operations, 10ops-codfw: Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC - https://phabricator.wikimedia.org/T242481 (10Marostegui) Thanks @MoritzMuehlenhoff for your tests. So recap: - this host has 1G and 10G: we bought the 10G because once we start do... [17:34:50] 10DBA, 10Operations, 10ops-codfw: Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC - https://phabricator.wikimedia.org/T242481 (10Marostegui) For what is worth, exactly the same behaviour is happening on es2020. And what was done at T242481#5797643 for es2024 also... [17:41:31] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1100 - https://phabricator.wikimedia.org/T241506 (10Marostegui) @Jclark-ctr feel free to replace the disk once you get to the DC, disk #0 is the one. [21:02:04] 10Blocked-on-schema-change, 10Notifications, 10Patch-For-Review, 10Schema-change: Remove unused bundling DB fields - https://phabricator.wikimedia.org/T143763 (10Reedy) @Marostegui https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Echo/+/563543/ has just been made and merged... I think there might b... [23:47:16] 10Blocked-on-schema-change, 10DBA: Extend flow_ext_ref.ref_src_wiki - https://phabricator.wikimedia.org/T242688 (10Reedy)