[04:40:23] 10DBA, 10Operations: Predictive failures on disk S.M.A.R.T. status - https://phabricator.wikimedia.org/T208323 (10Marostegui) [04:43:17] 10DBA, 10Wikimedia-Site-requests: Global rename of Fiona B. → Fiona*: supervision needed - https://phabricator.wikimedia.org/T224348 (10Marostegui) @Anomie might be able to give us more information of the timeline for the deployment. [04:58:59] 10Blocked-on-schema-change, 10DBA: Drop abuse_filter_log.afl_log_id in production - https://phabricator.wikimedia.org/T226851 (10Marostegui) @Daimona as per https://tools.wmflabs.org/versions/ we are on .11 everywhere, this is safe to proceed then? [04:59:53] 10Blocked-on-schema-change, 10DBA: Drop abuse_filter_log.afl_log_id in production - https://phabricator.wikimedia.org/T226851 (10Marostegui) p:05Triage→03Normal [05:03:26] 10DBA, 10Core Platform Team, 10MediaWiki-General-or-Unknown: Investigate query planning in MariaDB 10 - https://phabricator.wikimedia.org/T85000 (10Marostegui) Not sure what's the actionable here for the DBAs. This looks like another case of the optimizer not doing what expected. I have tested the original t... [05:17:09] 10DBA, 10MediaWiki-extensions-OATHAuth, 10Schema-change: Drop old oathauth_users columns - https://phabricator.wikimedia.org/T226826 (10Marostegui) I have altered this table on db2054 on `centralauth` and will leave the columns renamed for a few days to make sure nothing uses it. ` root@db2054.codfw.wmnet[ce... [05:18:30] 10DBA, 10MediaWiki-extensions-OATHAuth, 10Schema-change: Drop old oathauth_users columns - https://phabricator.wikimedia.org/T226826 (10Marostegui) p:05Triage→03Normal [05:19:13] 10DBA, 10MediaWiki-extensions-OATHAuth, 10Schema-change: Drop old oathauth_users columns - https://phabricator.wikimedia.org/T226826 (10Marostegui) a:03Marostegui [05:35:30] 10DBA: Move more wikis from s3 to s5 - https://phabricator.wikimedia.org/T226950 (10Marostegui) [05:35:46] 10DBA: Move more wikis from s3 to s5 - https://phabricator.wikimedia.org/T226950 (10Marostegui) p:05Triage→03Normal [06:16:39] 10DBA, 10OTRS, 10Operations, 10Operations-Software-Development, 10Recommendation-API: Failover m2 master db1065 to db1132 - https://phabricator.wikimedia.org/T226952 (10Marostegui) [06:16:49] 10DBA, 10OTRS, 10Operations, 10Operations-Software-Development, 10Recommendation-API: Failover m2 master db1065 to db1132 - https://phabricator.wikimedia.org/T226952 (10Marostegui) p:05Triage→03Normal [07:05:27] 10DBA, 10MediaWiki-extensions-OATHAuth, 10Schema-change: Drop old oathauth_users columns - https://phabricator.wikimedia.org/T226826 (10Marostegui) Also changed on an eqiad host (so we can check if there is something reading from those): ` root@db1094.eqiad.wmnet[centralauth]> set session sql_log_bin=0; Que... [07:25:27] 10DBA, 10OTRS, 10Operations, 10Operations-Software-Development, 10Recommendation-API: Failover m2 master db1065 to db1132 - https://phabricator.wikimedia.org/T226952 (10MoritzMuehlenhoff) debmonitor readonly time is not an issue, the debmonitor clients will simply retry the next time. [07:52:02] 10DBA, 10OTRS, 10Operations, 10Operations-Software-Development, 10Recommendation-API: Failover m2 master db1065 to db1132 - https://phabricator.wikimedia.org/T226952 (10jcrespo) Because of the TTL mention, are you planning a failover of proxy at the same time? [07:53:49] 10DBA, 10OTRS, 10Operations, 10Operations-Software-Development, 10Recommendation-API: Failover m2 master db1065 to db1132 - https://phabricator.wikimedia.org/T226952 (10Marostegui) >>! In T226952#5295201, @jcrespo wrote: > Because of the TTL mention, are you planning a failover of proxy at the same time?... [07:54:09] 10DBA, 10OTRS, 10Operations, 10Operations-Software-Development, 10Recommendation-API: Failover m2 master db1065 to db1132 - https://phabricator.wikimedia.org/T226952 (10Marostegui) [08:17:28] 10DBA, 10OTRS, 10Operations, 10Operations-Software-Development, 10Recommendation-API: Failover m2 master db1065 to db1132 - https://phabricator.wikimedia.org/T226952 (10jcrespo) I am actually proposing to maybe do it, but it needs more work. [08:20:12] 10DBA, 10OTRS, 10Operations, 10Operations-Software-Development, 10Recommendation-API: Failover m2 master db1065 to db1132 - https://phabricator.wikimedia.org/T226952 (10Marostegui) Let's leave it aside for now :-) [08:28:56] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Drop abuse_filter_log.afl_log_id in production - https://phabricator.wikimedia.org/T226851 (10Daimona) [08:29:18] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Drop abuse_filter_log.afl_log_id in production - https://phabricator.wikimedia.org/T226851 (10Daimona) [08:30:44] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Drop abuse_filter_log.afl_log_id in production - https://phabricator.wikimedia.org/T226851 (10Daimona) >>! In T226851#5294936, @Marostegui wrote: > @Daimona as per https://tools.wmflabs.org/versions/ we are on .11 everywhere, this is safe to proceed then? Ac... [08:31:37] 10DBA, 10OTRS, 10Operations, 10Operations-Software-Development, 10Recommendation-API: Failover m2 master db1065 to db1132 - https://phabricator.wikimedia.org/T226952 (10Volans) For `debmonitor` it connects to `m2-master.eqiad.wmnet` and I'm not sure if Django's connection pooling would be smart enough to... [08:35:06] 10DBA, 10OTRS, 10Operations, 10Operations-Software-Development, and 2 others: Failover m2 master db1065 to db1132 - https://phabricator.wikimedia.org/T226952 (10Marostegui) >>! In T226952#5295258, @Volans wrote: > For `debmonitor` it connects to `m2-master.eqiad.wmnet` and I'm not sure if Django's connecti... [08:36:01] 10DBA, 10OTRS, 10Operations, 10Operations-Software-Development, and 2 others: Failover m2 master db1065 to db1132 - https://phabricator.wikimedia.org/T226952 (10Marostegui) [08:40:23] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Drop abuse_filter_log.afl_log_id in production - https://phabricator.wikimedia.org/T226851 (10Marostegui) 05Open→03Stalled Thanks, I am going to stall this until then. [09:09:26] 10DBA, 10OTRS, 10Operations, 10Operations-Software-Development, and 2 others: Failover m2 master db1065 to db1132 - https://phabricator.wikimedia.org/T226952 (10Marostegui) Note: db2044 needs upgrading [09:53:55] 10DBA, 10Wikimedia-Site-requests: Need supervision with a global rename of BU_Rob13 → Renamed user mou89p43twvqcvm8ut9w3: supervision needed - https://phabricator.wikimedia.org/T226971 (10Oshwah) [09:54:40] 10DBA, 10Wikimedia-Site-requests: Global rename of BU_Rob13 → Renamed user mou89p43twvqcvm8ut9w3: supervision needed - https://phabricator.wikimedia.org/T226971 (10Oshwah) [09:55:01] 10DBA, 10Wikimedia-Site-requests: Global rename of BU_Rob13 → Renamed user mou89p43twvqcvm8ut9w3: supervision needed - https://phabricator.wikimedia.org/T226971 (10jbond) please go ahead [09:58:04] 10DBA, 10Wikimedia-Site-requests: Global rename of BU_Rob13 → Renamed user mou89p43twvqcvm8ut9w3: supervision needed - https://phabricator.wikimedia.org/T226971 (10Oshwah) Done. [09:58:29] 10DBA, 10Wikimedia-Site-requests: Global rename of BU_Rob13 → Renamed user mou89p43twvqcvm8ut9w3: supervision needed - https://phabricator.wikimedia.org/T226971 (10Oshwah) 05Open→03Resolved p:05Triage→03Low a:03Oshwah [09:59:27] 10DBA, 10Wikimedia-Site-requests: Global rename of [REDACTED] → Renamed user mou89p43twvqcvm8ut9w3: supervision needed - https://phabricator.wikimedia.org/T226971 (10Oshwah) [12:31:08] 10DBA, 10MediaWiki-extensions-OATHAuth, 10Schema-change: Drop old oathauth_users columns - https://phabricator.wikimedia.org/T226826 (10Reedy) >>! In T226826#5295046, @Marostegui wrote: > Also changed on an eqiad host (so we can check if there is something reading from those): > ` > > root@db1094.eqiad.wmne... [14:17:32] 10DBA, 10Analytics, 10Analytics-EventLogging, 10Operations, 10ops-eqiad: db1107 (eventlogging db master) possibly memory issues - https://phabricator.wikimedia.org/T222050 (10elukey) ping :) [15:01:57] 10DBA, 10Epic: Setup es4 and es5 replica sets for new read-write external store service - https://phabricator.wikimedia.org/T226704 (10Marostegui) Should we define new HW specs for these hosts? Currently es1015 specs: 128GB RAM 12x1.819 TB SATA HDD [15:28:42] 10DBA, 10MediaWiki-extensions-OATHAuth, 10Schema-change: Drop old oathauth_users columns - https://phabricator.wikimedia.org/T226826 (10Marostegui) Cool, I will give it some more 24h - so far nothing on logstash for db1094 [19:39:44] 10DBA, 10Data-Services: Compress and defragment tables on labsdb hosts - https://phabricator.wikimedia.org/T222978 (10bd808) [19:42:41] 10DBA, 10Data-Services: Compress and defragment tables on labsdb hosts - https://phabricator.wikimedia.org/T222978 (10bd808) [20:06:36] 10DBA, 10Analytics: hi.wikisource added to labs replicas? - https://phabricator.wikimedia.org/T227030 (10Nuria) [20:11:43] 10DBA, 10Analytics: hi.wikisource added to labs replicas? - https://phabricator.wikimedia.org/T227030 (10Nuria) Also seems that it is not replicated to analytics production replicas? [20:28:17] 10DBA, 10Operations, 10Performance-Team, 10Availability (MediaWiki-MultiDC): Apache <=> mariadb SSL/TLS for cross-datacenter writes - https://phabricator.wikimedia.org/T134809 (10Krinkle) 05Open→03Stalled Blocked on {T175672} or {T196378}. [20:33:03] 10DBA, 10Wikimedia-Site-requests: Global rename of [REDACTED] → Renamed user mou89p43twvqcvm8ut9w3: supervision needed - https://phabricator.wikimedia.org/T226971 (10DannyS712) @Oshwah if they user wants to keep this private, you may want to change the visibility of this task - the name is still visible in the... [22:19:18] 10DBA, 10Analytics: hi.wikisource added to labs replicas? - https://phabricator.wikimedia.org/T227030 (10Reedy) It hasn't actually been properly created... It's not public. ie you can't get on https://hi.wikisource.org See {T218155} and {T212881}