[14:37:17] 10DBA, 10Operations, 10ops-eqiad: check db1054 power supply redundancy - https://phabricator.wikimedia.org/T177628#3667792 (10Cmjohnson) 05Open>03Resolved a:03Cmjohnson The PSU was indeed dead, swapped it with a spare from a decom'd server. Both power supplies are working. [14:45:12] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1092 - https://phabricator.wikimedia.org/T177264#3667812 (10Cmjohnson) A support ticket has been placed with HP. Sent them the AHS log. [14:47:17] 10DBA, 10Operations, 10ops-eqiad: check db1052 power supply redundancy - https://phabricator.wikimedia.org/T177627#3667816 (10Cmjohnson) 05Open>03Resolved a:03Cmjohnson db1052 has a bad PSU, swapped with a spare from a decom'd server [14:48:29] 10DBA, 10Operations, 10ops-eqiad: check db1080 power supply redundancy - https://phabricator.wikimedia.org/T177630#3667819 (10Cmjohnson) 05Open>03Resolved a:03Cmjohnson Loose cable, fixed [14:58:14] 10DBA, 10Operations, 10ops-eqiad: check db1052 power supply redundancy - https://phabricator.wikimedia.org/T177627#3667827 (10Marostegui) Thanks a lot @Cmjohnson - confirmed they are looking good now ``` /admin1/system1/logs1/log1-> show record1 properties CreationTimestamp = 20171008144224.000000-300... [15:11:51] 10DBA, 10Operations, 10ops-eqiad: check db1054 power supply redundancy - https://phabricator.wikimedia.org/T177628#3667832 (10Marostegui) The new PSU looks broken too, it didn't last long until it complained again: ``` reset /admin1/system1/logs1/log1-> show record1 properties CreationTimestamp = 20171... [15:16:16] 10DBA, 10Operations, 10ops-eqiad: check db1080 power supply redundancy - https://phabricator.wikimedia.org/T177630#3667833 (10Marostegui) Looks good now - thanks!: ``` hpiLO-> show powersupply1 status=0 status_tag=COMMAND COMPLETED Sun Oct 8 14:58:22 2017 /system1/powersupply1 Targets Prop... [15:21:30] 10DBA, 10Data-Services, 10XTools, 10cloud-services-team: Request to increase active connection quota for user s51187 on analytics.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T177570#3667834 (10Marostegui) >>! In T177570#3664959, @MusikAnimal wrote: >>>! In T177570#3664528, @jcrespo wrote: >> N... [15:25:16] 10DBA, 10Operations, 10ops-codfw: db2038 two disks with predictive failure - https://phabricator.wikimedia.org/T177720#3667836 (10Marostegui) [23:39:26] 10DBA, 10Data-Services, 10XTools, 10cloud-services-team: Request to increase active connection quota for user s51187 on analytics.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T177570#3668086 (10MusikAnimal) >>! In T177570#3667834, @Marostegui wrote: > How many you think you'd need on *.web? I'...