[06:23:36] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2044 - https://phabricator.wikimedia.org/T210049 (10Marostegui) 05Open>03Resolved The disk got rebuilt but now it shows predictive failure: `  root@db2044:~# hpssacli controller all show config  Smart Array P420i in Slot 0 (Embedded)    (sn: 0014380264...
[06:23:45] <wikibugs>	 10DBA, 10Operations: Predictive failures on disk S.M.A.R.T. status - https://phabricator.wikimedia.org/T208323 (10Marostegui)
[06:24:22] <wikibugs>	 10DBA, 10Operations: Predictive failures on disk S.M.A.R.T. status - https://phabricator.wikimedia.org/T208323 (10Marostegui) db2044 got its disk replaced but came up with predictive failure (T210049#4767169)
[06:37:15] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Marostegui) Make sure to check whether wikitech (labswiki and labtestwiki) need this schema change
[08:05:03] <wikibugs>	 10DBA, 10JADE, 10Operations, 10TechCom-RFC, 10Scoring-platform-team (Current): Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10kchapman) TechCom hosted an IRC meeting on this today: * Log: https://tools.wmflabs.org/meetbot/wikimedia...
[08:10:48] <wikibugs>	 10DBA, 10JADE, 10Operations, 10TechCom-RFC, 10Scoring-platform-team (Current): Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) Points I took away from the TechCom meeting, which should be addressed before deployment: * The s...
[08:20:20] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping site_stats.ss_total_views on wmf databases - https://phabricator.wikimedia.org/T86339 (10Marostegui)
[08:24:09] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping site_stats.ss_total_views on wmf databases - https://phabricator.wikimedia.org/T86339 (10Marostegui) s1 eqiad progress  [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1002 [x] dbstore1001 [] db1124 [] db1119 [] db1114 []...
[08:24:25] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping site_stats.ss_total_views on wmf databases - https://phabricator.wikimedia.org/T86339 (10Marostegui)
[09:44:47] <jynus>	 I am going to restart the dbstores
[09:44:54] <jynus>	 minus dbstore1002
[09:46:02] <marostegui>	 restart as in reboot?
[09:46:28] <marostegui>	 can you leave dbstore1001 for the last one? I am finishing some things there
[09:46:29] <jynus>	 yes
[09:46:35] <jynus>	 ok
[09:46:42] <marostegui>	 ping me when ready for dbstore1001
[09:46:43] <jynus>	 will ping before doing that
[09:46:48] <marostegui>	 thank you
[10:16:31] <jynus>	 because dbstore2002 is messy, I will wait to do that after our meeting
[10:16:41] <marostegui>	 ok
[10:16:45] <jynus>	 that way I can see dbstore2001 catching up
[10:16:46] <marostegui>	 you can go with dbstore1001 if you like
[10:17:06] <jynus>	 not touching it anymore?
[10:17:15] <marostegui>	 nope
[12:12:55] <jynus>	 I am restarting dbstore2002 next
[12:13:05] <marostegui>	 ok
[12:38:10] <jynus>	 s1 and s4 are taking a long time to stop, "flushing buffer pool"
[13:33:25] <wikibugs>	 10DBA, 10Anti-Harassment, 10Operations: Error Unknown column  ipb_sitewide in field list on query - https://phabricator.wikimedia.org/T208462 (10Marostegui) 05Open>03Resolved
[14:20:48] <jynus>	 things looking good on the dbstore front
[14:46:04] <marostegui>	 ≤o/