[04:57:15] 10DBA, 10Operations: db2084 temporary correctable hardware errors - https://phabricator.wikimedia.org/T225884 (10Marostegui) 05Open→03Resolved And it finally cleared up ` 23:38:30 <+icinga-wm> RECOVERY - EDAC syslog messages on db2084 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dash... [05:00:25] 10DBA, 10Operations: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['dbproxy1018.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/201906... [05:26:12] 10DBA, 10Operations: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['dbproxy1020.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/201906... [05:34:12] 10DBA, 10Operations: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['dbproxy1019.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/201906... [05:34:37] 10DBA, 10Operations: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['dbproxy1021.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/201906... [06:00:42] 10DBA, 10Operations, 10Patch-For-Review: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbproxy1018.eqiad.wmnet'] ` Of which those **FAILED**: ` ['dbproxy1018.eqiad.wmnet'] ` [06:23:26] 10DBA, 10Operations, 10Patch-For-Review: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['dbproxy1018.eqiad.wmnet'] ` The log can be found in `/var/log/w... [06:26:32] 10DBA, 10Operations, 10Patch-For-Review: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbproxy1020.eqiad.wmnet'] ` Of which those **FAILED**: ` ['dbproxy1020.eqiad.wmnet'] ` [06:30:24] 10DBA, 10Operations, 10Patch-For-Review: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['dbproxy1020.eqiad.wmnet'] ` The log can be found in `/var/log/w... [06:34:36] 10DBA, 10Operations, 10Patch-For-Review: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbproxy1019.eqiad.wmnet'] ` Of which those **FAILED**: ` ['dbproxy1019.eqiad.wmnet'] ` [06:35:04] 10DBA, 10Operations, 10Patch-For-Review: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbproxy1021.eqiad.wmnet'] ` Of which those **FAILED**: ` ['dbproxy1021.eqiad.wmnet'] ` [06:40:18] 10DBA, 10Operations, 10Patch-For-Review: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['dbproxy1021.eqiad.wmnet'] ` The log can be found in `/var/log/w... [06:47:06] 10DBA, 10Operations, 10Patch-For-Review: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbproxy1020.eqiad.wmnet'] ` and were **ALL** successful. [06:48:41] 10DBA, 10Operations, 10Patch-For-Review: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10Marostegui) [06:58:34] 10DBA, 10Operations, 10Patch-For-Review: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbproxy1021.eqiad.wmnet'] ` and were **ALL** successful. [07:23:43] 10DBA, 10Operations: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbproxy1018.eqiad.wmnet'] ` Of which those **FAILED**: ` ['dbproxy1018.eqiad.wmnet'] ` [07:29:55] 10DBA, 10Operations: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10Marostegui) [07:30:15] 10DBA, 10Operations: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10Marostegui) a:05Marostegui→03Cmjohnson @Cmjohnson @ayounsi is there anything special with dbproxy1018 and dbproxy1019 VLAN's and PXE? None of the seems to be booting up from PXE, despite that... [07:30:41] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) [07:53:43] 10DBA, 10Operations: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['dbproxy1018.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/201906... [08:08:13] 10DBA, 10Operations, 10Patch-For-Review: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10Marostegui) a:05Cmjohnson→03Marostegui While debugging we Arzhel we have noticed that the DNS entries for dbproxy1018 and dbproxy1019 didn't belong to the cloud network,... [08:29:14] 10DBA, 10Growth-Team, 10PageCuration, 10Community-Tech (Kanban), 10Spike: [4 hours] Investigate whether it's efficient to order by tag value (DBA input requested) - https://phabricator.wikimedia.org/T225169 (10Marostegui) I have been checking this query on `enwiki` and it doesn't seem to be too bad: ` ro... [08:51:07] 10DBA, 10Operations, 10Patch-For-Review: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbproxy1018.eqiad.wmnet'] ` Of which those **FAILED**: ` ['dbproxy1018.eqiad.wmnet'] ` [08:55:43] 10DBA, 10Operations, 10Patch-For-Review: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['dbproxy1018.eqiad.wmnet'] ` The log can be found in `/var/log/w... [09:11:47] 10DBA, 10Operations, 10Patch-For-Review: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['dbproxy1019.eqiad.wmnet'] ` The log can be found in `/var/log/w... [09:12:30] 10DBA, 10Operations, 10Patch-For-Review: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbproxy1018.eqiad.wmnet'] ` and were **ALL** successful. [09:14:01] 10DBA, 10Operations, 10Patch-For-Review: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10Marostegui) [09:14:31] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) [09:14:56] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) [09:28:29] 10DBA, 10Operations, 10Patch-For-Review: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbproxy1019.eqiad.wmnet'] ` and were **ALL** successful. [09:30:02] 10DBA, 10Operations, 10Patch-For-Review: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10Marostegui) [09:30:11] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) [09:30:13] 10DBA, 10Operations, 10Patch-For-Review: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10Marostegui) 05Open→03Resolved All hosts installed [09:30:52] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui)