[02:22:00] thank you cdanis [05:04:32] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: hw troubleshooting: power supply for db1129 - https://phabricator.wikimedia.org/T230458 (10Marostegui) [05:47:42] 10DBA, 10Phabricator, 10Release-Engineering-Team-TODO, 10Documentation, and 2 others: Prepare a disaster recovery plan for failing over from phab1001 to phab2001 (or phab2001 to 1001) - https://phabricator.wikimedia.org/T190572 (10Marostegui) The proxy at codfw is now provisioned. It obviously points to th... [06:02:05] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) [07:02:48] 10DBA: Replace db2044 with db2063 - https://phabricator.wikimedia.org/T230459 (10Marostegui) [07:03:17] 10DBA: Replace db2044 with db2063 - https://phabricator.wikimedia.org/T230459 (10Marostegui) p:05Triage→03Normal [07:18:07] 10DBA, 10Patch-For-Review: Replace db2044 with db2063 - https://phabricator.wikimedia.org/T230459 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db2063.codfw.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/201908140717_marostegui_7... [07:32:18] 10DBA, 10Goal: Productionize db21[21-31} - https://phabricator.wikimedia.org/T228969 (10Marostegui) [07:32:30] 10DBA, 10Goal: Productionize db21[21-31} - https://phabricator.wikimedia.org/T228969 (10Marostegui) 05Open→03Resolved All these hosts have been populated and are provisioned [07:32:33] 10DBA, 10Operations, 10ops-codfw: (2019-08-31)rack/setup/install db2131.codfw.wmnet - https://phabricator.wikimedia.org/T229251 (10Marostegui) [07:32:38] 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10Marostegui) [08:18:44] 10DBA: Replace db2044 with db2063 - https://phabricator.wikimedia.org/T230459 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db2063.codfw.wmnet'] ` Of which those **FAILED**: ` ['db2063.codfw.wmnet'] ` [08:49:36] 10DBA: Replace db2044 with db2063 - https://phabricator.wikimedia.org/T230459 (10Marostegui) I have been trying to PXE boot this host but it has been impossible. Even though I have manually set the PXE from the ipmitool locally it is still not working: ` root@db2063:~# ipmitool chassis bootparam get 5 Boot para... [10:37:57] 10DBA: Replace db2044 with db2063 - https://phabricator.wikimedia.org/T230459 (10Marostegui) Forgot to mention that this host is not in use and it is downtimed, so this onsite maintenance can be done anytime without heads-up to the DBAs [10:39:59] are you touching the backup hosts? [10:40:06] nope [10:40:11] I will [10:40:16] cool [13:59:11] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: hw troubleshooting: power supply for db1129 - https://phabricator.wikimedia.org/T230458 (10Cmjohnson) a:05Cmjohnson→03Jclark-ctr John, Please check to make sure that the power cables are fully seated. Update the task and let me know if I need to order a new... [14:10:30] 10DBA, 10Cloud-Services, 10cloud-services-team, 10User-Banyek: Prepare and check storage layer for nap.wikisource - https://phabricator.wikimedia.org/T210762 (10Reedy) It's created [14:32:06] 10DBA, 10Cloud-Services, 10cloud-services-team, 10User-Banyek: Prepare and check storage layer for nap.wikisource - https://phabricator.wikimedia.org/T210762 (10Marostegui) a:03Marostegui Thanks - I will take care of this next week. Assigning this to me so it is known it is blocked on me before creating... [15:23:11] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: hw troubleshooting: power supply for db1129 - https://phabricator.wikimedia.org/T230458 (10Jclark-ctr) 05Open→03Resolved Found Power cable not fully seated . Reseated cable.