[05:49:41] 10DBA, 10Operations, 10ops-codfw: Reboot, upgrade firmware and kernel of db1096-db1106, db2071-db2092 - https://phabricator.wikimedia.org/T216240 (10Marostegui) @Papaul can we upgrade firmware and BIOS on db2080?, I was bitten by this today. [08:42:43] 10DBA, 10Wikidata, 10Wikidata wb_terms Trailblazing, 10User-Addshore: Send the schema to DBAs and follow up with their review - https://phabricator.wikimedia.org/T219142 (10Ladsgroup) 05Open→03Resolved Yup [10:31:23] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/deploy eqiad dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T219399 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jynus on cumin1001.eqiad.wmnet for hosts: ` ['dbprov1001.eqiad.wmnet'] ` The... [10:32:04] 10DBA, 10MediaWiki-Database, 10Core Platform Team (Multi-DC (TEC1)), 10Core Platform Team Backlog (Watching / External), and 4 others: FY18/19 TEC1.6 Q4: Replace the usage of GTID_WAIT with pt-heartbeat in MW - https://phabricator.wikimedia.org/T221159 (10Marostegui) Thanks @mobrovac for the summary! I wou... [10:51:50] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/deploy eqiad dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T219399 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jynus on cumin1001.eqiad.wmnet for hosts: ` ['dbprov1002.eqiad.wmnet'] ` The... [11:02:47] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/deploy eqiad dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T219399 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbprov1002.eqiad.wmnet'] ` Of which those **FAILED**: ` ['dbprov1002.eqiad.wmne... [11:03:29] Hey, is https://tendril.wikimedia.org/report/sampled_queries?host=^db&user=wikiuser&schema=wik&hours=1 disabled? [11:19:53] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/deploy eqiad dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T219399 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jynus on cumin1001.eqiad.wmnet for hosts: ` ['dbprov1002.eqiad.wmnet'] ` The... [11:20:04] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/deploy eqiad dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T219399 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbprov1002.eqiad.wmnet'] ` Of which those **FAILED**: ` ['dbprov1002.eqiad.wmne... [11:21:56] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/deploy eqiad dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T219399 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jynus on cumin1001.eqiad.wmnet for hosts: ` ['dbprov1002.eqiad.wmnet'] ` The... [11:22:05] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/deploy eqiad dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T219399 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbprov1002.eqiad.wmnet'] ` Of which those **FAILED**: ` ['dbprov1002.eqiad.wmne... [11:22:23] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/deploy eqiad dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T219399 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jynus on cumin1001.eqiad.wmnet for hosts: ` ['dbprov1002.eqiad.wmnet'] ` The... [11:22:34] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/deploy eqiad dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T219399 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbprov1002.eqiad.wmnet'] ` Of which those **FAILED**: ` ['dbprov1002.eqiad.wmne... [14:39:50] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2047 - https://phabricator.wikimedia.org/T221481 (10Papaul) a:05Papaul→03Marostegui complete [14:45:38] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2037 - https://phabricator.wikimedia.org/T221512 (10Papaul) a:05Papaul→03Marostegui Complete [15:01:47] 10DBA, 10Operations, 10decommission, 10ops-eqiad: Decommission parsercache hosts: pc1004.eqiad.wmnet pc1005.eqiad.wmnet pc1006.eqiad.wmnet - https://phabricator.wikimedia.org/T210969 (10Cmjohnson) 05Open→03Resolved All the disk were securely wiped and server reset to server defaults [15:23:07] 10DBA: Fix revision special indexes and partitions on db1103:3314 and db1113:3316 - https://phabricator.wikimedia.org/T221782 (10Marostegui) [15:23:22] 10DBA: Fix revision special indexes and partitions on db1103:3314 and db1113:3316 - https://phabricator.wikimedia.org/T221782 (10Marostegui) p:05Triage→03Normal [15:24:59] 10DBA, 10Operations, 10ops-codfw: Reboot, upgrade firmware and kernel of db1096-db1106, db2071-db2092 - https://phabricator.wikimedia.org/T216240 (10Papaul) Before: BIOS Version 2.4.3 Firmware Version 2.40.40.40 IP Address(es) 10.193.1.75 iDRAC MAC Address 84:7B:EB:F6:99:B2 DNS Domain Name Lifecyc... [15:33:01] 10DBA, 10Operations, 10ops-codfw: Reboot, upgrade firmware and kernel of db1096-db1106, db2071-db2092 - https://phabricator.wikimedia.org/T216240 (10Marostegui) Thanks @Papaul I am rebooting the server a few times to confirm it is indeed solved! [15:50:24] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/deploy eqiad dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T219399 (10jcrespo) [15:59:49] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2037 - https://phabricator.wikimedia.org/T221512 (10Marostegui) 05Open→03Resolved Thanks! All good! ` root@db2037:~# hpssacli controller all show config Smart Array P420i in Slot 0 (Embedded) (sn: 0014380312088E0) Port Name: 1I Port Name:... [16:02:36] 10DBA, 10Operations, 10ops-codfw: Reboot, upgrade firmware and kernel of db1096-db1106, db2071-db2092 - https://phabricator.wikimedia.org/T216240 (10Marostegui) [16:30:18] 10DBA, 10Operations, 10ops-codfw, 10Goal, 10Patch-For-Review: rack/setup/install db2[103-120].codfw.wmnet (18 hosts) - https://phabricator.wikimedia.org/T221532 (10Papaul) [17:04:28] 10DBA, 10Operations, 10ops-codfw, 10Goal, 10Patch-For-Review: rack/setup/install db2[103-120].codfw.wmnet (18 hosts) - https://phabricator.wikimedia.org/T221532 (10Papaul)