[05:08:06] <wikibugs>	 10DBA, 10AbuseFilter, 10Patch-For-Review: Drop abuse_filter_log.afl_log_id in production - https://phabricator.wikimedia.org/T226851 (10Marostegui) >>! In T226851#5367028, @MusikAnimal wrote: > The `abuse_filter_log` table is apparently missing on the replicas, presumably because of this. Maybe the view defi...
[05:13:40] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: pc2010 possibly broken memory - https://phabricator.wikimedia.org/T227552 (10Marostegui) p:05Normal→03High MySQL crashed again: ` [Thu Jul 25 16:10:27 2019] mce: Uncorrected hardware memory error in user-access at 336c902080 [Thu Jul 25 16:10:27 2019] {1}Hardware error d...
[05:15:17] <wikibugs>	 10DBA, 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar): phased rollout of dbctl, etcd-backed database configuration in Mediawiki - https://phabricator.wikimedia.org/T229070 (10Marostegui) \o/
[05:26:30] <wikibugs>	 10DBA, 10Operations, 10Phabricator, 10User-notice: Switchover m3 (phabricator) master db1072 to db1128 - https://phabricator.wikimedia.org/T228243 (10Marostegui)
[05:26:33] <wikibugs>	 10DBA, 10Goal: Address Database infrastructure blockers on datacenter switchover & multi-dc deployment - https://phabricator.wikimedia.org/T220170 (10Marostegui)
[05:38:34] <wikibugs>	 10DBA, 10Data-Services: Compress and defragment tables on labsdb hosts - https://phabricator.wikimedia.org/T222978 (10Marostegui)
[05:40:27] <wikibugs>	 10DBA, 10Operations, 10decommission: decommission db1072.eqiad.wmnet - https://phabricator.wikimedia.org/T228956 (10Marostegui)
[05:43:15] <wikibugs>	 10DBA, 10Operations, 10decommission: decommission db1072.eqiad.wmnet - https://phabricator.wikimedia.org/T228956 (10Marostegui)
[06:18:46] <wikibugs>	 10DBA, 10Goal: Productionize db21[21-30} - https://phabricator.wikimedia.org/T228969 (10Marostegui)
[06:19:03] <wikibugs>	 10DBA, 10Goal: Productionize db21[21-30} - https://phabricator.wikimedia.org/T228969 (10Marostegui)
[06:19:51] <wikibugs>	 10DBA, 10Goal: Productionize db21[21-30} - https://phabricator.wikimedia.org/T228969 (10Marostegui)
[09:31:52] <wikibugs>	 10DBA, 10Goal, 10Patch-For-Review: Productionize db21[21-30} - https://phabricator.wikimedia.org/T228969 (10Marostegui)
[09:34:30] <wikibugs>	 10DBA, 10Goal, 10Patch-For-Review: Productionize db21[21-30} - https://phabricator.wikimedia.org/T228969 (10Marostegui)
[10:26:22] <wikibugs>	 10DBA, 10serviceops-radar, 10Patch-For-Review, 10Performance-Team (Radar): phased rollout of dbctl, etcd-backed database configuration in Mediawiki - https://phabricator.wikimedia.org/T229070 (10jijiki) p:05Triage→03Normal
[12:36:09] <jeh>	 The analytics replica labsdb1012 needs to be rebooted for a kernel update T224228. 
[12:36:49] <jeh>	 marostegui if I reboot that host, do you happen to have time to take a look at it afterwords and make sure all is well?
[12:36:50] <marostegui>	 jeh: You can coordinate with elukey, labsdb1012 is owned by them :)
[12:37:13] <jeh>	 I noticed that mariadb is not configured to start on boot
[12:37:22] <jeh>	 is it OK to just start it?
[12:38:12] <marostegui>	 jeh: Probably better to leave it off, so it needs to be started manually
[12:38:28] <marostegui>	 jeh: Also, replication better to be also stopped before stopping mariadb
[12:38:52] <marostegui>	 jeh: did you get the green light from luca?
[12:38:56] <jeh>	 ok, I'll check with elukey. I coordinated with Nuria and milimetric
[12:39:11] <marostegui>	 Ah then that's ok
[12:39:16] <marostegui>	 So they are not using it at the moment?
[12:39:29] <jeh>	 they said anytime before the end of the month
[12:39:33] <jeh>	 is OK
[12:39:37] <marostegui>	 So what I would do: log into mysql and stop replication: (stop all slaves;)
[12:39:46] <marostegui>	 and once it is stopped, then stop mariadb
[12:40:10] <marostegui>	 I can do all that if you want, or else, you can too! :)
[12:40:27] <jeh>	 I don't mind doing it, just wanted to make sure I did the right things :)
[12:40:56] <marostegui>	 jeh: Sure, so: stop all slaves
[12:41:01] <marostegui>	 and then stop mariadb
[12:41:34] <jeh>	 on boot is it OK to just do the reverse? start without the slaves, wait, start slaves?
[12:42:29] <marostegui>	 yeah, just run this before starting mariadb: systemctl set-environment MYSQLD_OPTS="--skip-slave-start"
[12:43:36] <jeh>	 OK, thanks 
[12:57:35] <jeh>	 labsdb1012 is back online, mariadb is up and slaves are replicating. I also started the wmf-pt-kil service
[13:01:36] <marostegui>	 lovely
[13:01:41] <marostegui>	 thanks a lot
[13:02:21] <jeh>	 thank you for the start/stop info :)
[13:11:57] <marostegui>	 np!