[00:31:02] 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10Papaul) [04:42:36] 10DBA, 10Operations, 10ops-codfw: pc2010 possibly broken memory - https://phabricator.wikimedia.org/T227552 (10Marostegui) This host crashed again, this time it was totally frozen and I had to reset it via idrac. These are the HW logs, same issue: ` ----------------------------------------------------------... [04:43:29] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) [04:44:43] 10DBA, 10Operations, 10ops-eqiad: Upgrade db1100 firmware and BIOS - https://phabricator.wikimedia.org/T228732 (10Marostegui) >>! In T228732#5362544, @Cmjohnson wrote: > @Marostegui This can be done any day...Let's plan 8/6 @1000EDT /1400UTC Great! Thank you. I have made a note on my calendar so the host wi... [05:03:59] 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10Marostegui) [05:04:23] 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10Marostegui) db2121-db2125 looking good! Thanks [05:06:10] 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10Marostegui) [05:33:17] 10DBA, 10Operations, 10Phabricator, 10User-notice: Switchover m3 (phabricator) master db1072 to db1128 - https://phabricator.wikimedia.org/T228243 (10Marostegui) [05:41:39] 10DBA, 10Operations, 10Phabricator, 10User-notice: Switchover m3 (phabricator) master db1072 to db1128 - https://phabricator.wikimedia.org/T228243 (10Marostegui) 05Open→03Resolved This has been done. Phabricator read only start: 05:30:44 Phabricator read only stop: 05:31:37 Total read-only time: 53 se... [05:41:43] 10DBA, 10Operations: Decommission db1061-db1073 - https://phabricator.wikimedia.org/T217396 (10Marostegui) [05:48:25] 10DBA, 10Operations, 10decommission: decommission db1072.eqiad.wmnet - https://phabricator.wikimedia.org/T228956 (10Marostegui) [05:48:32] 10DBA, 10Operations, 10decommission: decommission db1072.eqiad.wmnet - https://phabricator.wikimedia.org/T228956 (10Marostegui) p:05Triage→03Normal [07:14:19] 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db2126.codfw.wmnet'] ` The log can be found in `/var/log/wmf-aut... [07:29:26] 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db2126.codfw.wmnet'] ` Of which those **FAILED**: ` ['db2126.codfw.wmnet'] ` [07:38:03] 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10Marostegui) @Papaul I tried to install db2126 myself to advance on the task, but looks like it keeps rebooting on PXE forever :-) I think it needs your on-site magic, as with... [07:48:10] 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db2126.codfw.wmnet'] ` The log can be found in `/var/log/wmf-aut... [08:12:22] 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db2126.codfw.wmnet'] ` and were **ALL** successful. [08:13:30] 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10Marostegui) >>! In T227113#5364053, @Marostegui wrote: > @Papaul I tried to install db2126 myself to advance on the task, but looks like it keeps rebooting on PXE forever :-)... [08:14:34] 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10Marostegui) [08:38:28] 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db2128.codfw.wmnet'] ` The log can be found in `/var/log/wmf-aut... [08:48:03] 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db2129.codfw.wmnet'] ` The log can be found in `/var/log/wmf-aut... [09:03:40] 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db2128.codfw.wmnet'] ` and were **ALL** successful. [09:03:53] 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db2130.codfw.wmnet'] ` The log can be found in `/var/log/wmf-aut... [09:04:48] 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10Marostegui) [09:12:40] 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db2129.codfw.wmnet'] ` and were **ALL** successful. [09:13:30] 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10Marostegui) [09:20:17] 10DBA, 10Operations, 10decommission: decommission db1072.eqiad.wmnet - https://phabricator.wikimedia.org/T228956 (10Marostegui) [09:27:25] 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db2130.codfw.wmnet'] ` and were **ALL** successful. [09:27:53] 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10Marostegui) [09:29:00] 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10Marostegui) @Papaul all hosts but db2127 have been installed. I have managed to get into the BIOS of all of them and change the boot settings, however, db2127's idrac password... [09:29:41] 10DBA, 10Goal: Productionize db21[21-30} - https://phabricator.wikimedia.org/T228969 (10Marostegui) [09:29:58] 10DBA, 10Goal: Productionize db21[21-30} - https://phabricator.wikimedia.org/T228969 (10Marostegui) p:05Triage→03Normal [10:05:16] 10DBA, 10Data-Services: Compress and defragment tables on labsdb hosts - https://phabricator.wikimedia.org/T222978 (10Marostegui) labsdb1009 is fully done. [10:05:24] 10DBA, 10Data-Services: Compress and defragment tables on labsdb hosts - https://phabricator.wikimedia.org/T222978 (10Marostegui) [11:49:51] 10DBA, 10Data-Services: Compress and defragment tables on labsdb hosts - https://phabricator.wikimedia.org/T222978 (10Marostegui) [12:57:53] 10DBA, 10AbuseFilter: Drop abuse_filter_log.afl_log_id in production - https://phabricator.wikimedia.org/T226851 (10Marostegui) [13:20:02] 10DBA, 10AbuseFilter: Drop abuse_filter_log.afl_log_id in production - https://phabricator.wikimedia.org/T226851 (10Marostegui) s4 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1004 [] db1138 [] db1125 [] db1121 [x] db1103 [x] db1102 [x] db1097 [] db1091 [] db1084 [] db1081 [13:20:39] 10DBA, 10AbuseFilter: Drop abuse_filter_log.afl_log_id in production - https://phabricator.wikimedia.org/T226851 (10Marostegui) [13:40:06] 10DBA, 10AbuseFilter: Drop abuse_filter_log.afl_log_id in production - https://phabricator.wikimedia.org/T226851 (10Marostegui) [13:49:55] 10DBA, 10Analytics, 10Analytics-EventLogging, 10Operations, and 2 others: Decommission dbproxy1004 and dbproxy1009 - https://phabricator.wikimedia.org/T228768 (10Marostegui) [14:14:34] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10RobH) [14:54:35] 10DBA, 10Analytics, 10Analytics-EventLogging, 10Operations, and 2 others: Decommission dbproxy1004 and dbproxy1009 - https://phabricator.wikimedia.org/T228768 (10Marostegui) [15:35:36] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Cmjohnson) [16:41:15] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) [21:15:27] 10DBA, 10AbuseFilter: Drop abuse_filter_log.afl_log_id in production - https://phabricator.wikimedia.org/T226851 (10MusikAnimal) The `abuse_filter_log` table is apparently missing on the replicas, presumably because of this. Maybe the view definition needs to be updated? ` MariaDB [enwiki_p]> DESCRIBE itwiki_... [23:00:40] 10DBA, 10serviceops: phased rollout of dbctl, etcd-backed database configuration in Mediawiki - https://phabricator.wikimedia.org/T229070 (10CDanis) [23:01:00] 10DBA, 10serviceops: phased rollout of dbctl, etcd-backed database configuration in Mediawiki - https://phabricator.wikimedia.org/T229070 (10CDanis) [23:01:08] 10DBA, 10MediaWiki-Configuration, 10Operations, 10Patch-For-Review, and 2 others: Create tool to handle the state of database configuration in MediaWiki in etcd - https://phabricator.wikimedia.org/T197126 (10CDanis) [23:01:45] 10DBA, 10serviceops: phased rollout of dbctl, etcd-backed database configuration in Mediawiki - https://phabricator.wikimedia.org/T229070 (10CDanis) [23:03:15] 10DBA, 10MediaWiki-Configuration, 10Operations, 10Patch-For-Review, and 3 others: Create tool to handle the state of database configuration in MediaWiki in etcd - https://phabricator.wikimedia.org/T197126 (10Krinkle) [23:03:25] 10DBA, 10serviceops, 10Performance-Team (Radar): phased rollout of dbctl, etcd-backed database configuration in Mediawiki - https://phabricator.wikimedia.org/T229070 (10Krinkle)