[00:53:54] PROBLEM - MariaDB sustained replica lag on db2132 is CRITICAL: 2.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2132&var-port=9104 [01:01:00] RECOVERY - MariaDB sustained replica lag on db2132 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2132&var-port=9104 [05:13:16] 10DBA, 10Patch-For-Review: Evaluate the impact of changing innodb_change_buffering to inserts - https://phabricator.wikimedia.org/T263443 (10Marostegui) I have enabled this on production, it will be picked up once the hosts start to restart. However, I am going to manually enable it on a few hosts per section... [05:43:49] 10DBA: New database request: image_matching - https://phabricator.wikimedia.org/T280042 (10Marostegui) a:05Marostegui→03None Thanks for the detailed submmit @gmodena - some questions: - Is MySQL the best place to store this materialized data? Have you considered hadoop perhaps? - How many wikis will this be... [05:54:36] 10DBA: Migrate codfw sanitarium hosts (db2094/db2095) to Buster and 10.4 - https://phabricator.wikimedia.org/T275112 (10Marostegui) 05Stalled→03Open [06:07:25] 10Blocked-on-schema-change, 10DBA: Schema change for renaming new_name_timestamp to rc_new_name_timestamp in recentchanges - https://phabricator.wikimedia.org/T276292 (10Marostegui) Altered also db1098:3316 [06:56:06] 10DBA, 10Patch-For-Review: Productionize db21[45-52] and db11[76-84] - https://phabricator.wikimedia.org/T275633 (10Marostegui) db1179 is now replicating, will give it a few days before starting to pool it. [07:01:08] 10DBA, 10Patch-For-Review: Relocate "old" s4 hosts - https://phabricator.wikimedia.org/T253217 (10Marostegui) [07:07:43] PROBLEM - MariaDB sustained replica lag on db2133 is CRITICAL: 4.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2133&var-port=9104 [07:09:27] RECOVERY - MariaDB sustained replica lag on db2133 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2133&var-port=9104 [07:35:58] 10DBA, 10Patch-For-Review: Productionize db21[45-52] and db11[76-84] - https://phabricator.wikimedia.org/T275633 (10Marostegui) Transfer on-going from db1146:s2 to db1182 [08:39:30] 10DBA, 10Patch-For-Review: Productionize db21[45-52] and db11[76-84] - https://phabricator.wikimedia.org/T275633 (10Marostegui) db1182 is now replicating, will give it a few days before starting to pool it. [09:08:02] PROBLEM - MariaDB sustained replica lag on db2133 is CRITICAL: 4.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2133&var-port=9104 [09:09:44] RECOVERY - MariaDB sustained replica lag on db2133 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2133&var-port=9104 [09:47:57] 10DBA, 10cloud-services-team (Kanban): Upgrade mysql on db1128 (m5 db master) - https://phabricator.wikimedia.org/T279657 (10aborrero) >>! In T279657#6998835, @Marostegui wrote: > Thanks @arturo - what about Monday 19th at 09:00 AM UTC? works for me, thanks! please ping me or @dcaro on IRC to make sure we are... [09:48:19] 10DBA, 10cloud-services-team (Kanban): Upgrade mysql on db1128 (m5 db master) - https://phabricator.wikimedia.org/T279657 (10Marostegui) Will do thanks! [10:07:37] PROBLEM - MariaDB sustained replica lag on db2133 is CRITICAL: 5 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2133&var-port=9104 [10:09:21] RECOVERY - MariaDB sustained replica lag on db2133 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2133&var-port=9104 [12:34:02] 10Blocked-on-schema-change, 10DBA: Schema change for renaming new_name_timestamp to rc_new_name_timestamp in recentchanges - https://phabricator.wikimedia.org/T276292 (10Marostegui) I am not seeing any errors coming from any index force or related, I will wait till Monday though before proceeding [12:35:59] 10DBA: Upgrade 10.4.13 hosts to a higher version - https://phabricator.wikimedia.org/T279281 (10Marostegui) es1021 is es4 master, requires write depooling from MW. I will do that next week [12:43:16] 10DBA, 10OTRS, 10Performance-Team, 10Recommendation-API, 10SRE-tools: Upgrade mysql on db1107 (m2 db master) - https://phabricator.wikimedia.org/T280251 (10Marostegui) [12:44:18] 10DBA, 10OTRS, 10Performance-Team, 10Recommendation-API, 10SRE-tools: Upgrade mysql on db1107 (m2 db master) - https://phabricator.wikimedia.org/T280251 (10Marostegui) p:05Triage→03Medium [13:02:51] 10DBA, 10OTRS, 10Performance-Team, 10Recommendation-API, 10SRE-tools: Upgrade mysql on db1107 (m2 db master) - https://phabricator.wikimedia.org/T280251 (10kostajh) Sounds good to me! [13:11:10] 10DBA, 10OTRS, 10Performance-Team, 10Recommendation-API, 10SRE-tools: Upgrade mysql on db1107 (m2 db master) - https://phabricator.wikimedia.org/T280251 (10akosiaris) Both OTRS and recommendationapi should not have an issue, fine by me. One thing that is not clear though. This is for 2021-04-16? [13:12:01] 10DBA, 10OTRS, 10Performance-Team, 10Recommendation-API, 10SRE-tools: Upgrade mysql on db1107 (m2 db master) - https://phabricator.wikimedia.org/T280251 (10Marostegui) Sorry @akosiaris I should've specified it, this is for next week (unless someone needs time to prepare things) [13:12:22] 10DBA, 10OTRS, 10Performance-Team, 10Recommendation-API, 10SRE-tools: Upgrade mysql on db1107 (m2 db master) - https://phabricator.wikimedia.org/T280251 (10Marostegui) [13:12:59] 10DBA, 10OTRS, 10Performance-Team, 10Recommendation-API, 10SRE-tools: Upgrade mysql on db1107 (m2 db master) - https://phabricator.wikimedia.org/T280251 (10akosiaris) Cool. Thanks! Date sounds fine to me. [13:26:17] 10DBA, 10OTRS, 10Performance-Team, 10Recommendation-API, 10SRE-tools: Upgrade mysql on db1107 (m2 db master) - https://phabricator.wikimedia.org/T280251 (10Volans) Anytime is ok for `debmonitor`. [13:46:22] PROBLEM - MariaDB sustained replica lag on pc2008 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [13:48:34] RECOVERY - MariaDB sustained replica lag on pc2008 is OK: (C)2 ge (W)1 ge 0.4 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [14:08:12] PROBLEM - MariaDB sustained replica lag on db2133 is CRITICAL: 5.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2133&var-port=9104 [14:09:56] RECOVERY - MariaDB sustained replica lag on db2133 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2133&var-port=9104 [14:59:50] 10DBA, 10OTRS, 10Performance-Team, 10Recommendation-API, 10SRE-tools: Upgrade mysql on db1107 (m2 db master) - https://phabricator.wikimedia.org/T280251 (10bd808) iegreview and scholarships should deal with a db interruption with no lasting issues. [15:36:01] PROBLEM - MariaDB sustained replica lag on pc2007 is CRITICAL: 2.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2007&var-port=9104 [15:41:07] RECOVERY - MariaDB sustained replica lag on pc2007 is OK: (C)2 ge (W)1 ge 0.4 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2007&var-port=9104 [16:38:04] 10DBA, 10OTRS, 10Performance-Team, 10Recommendation-API, 10SRE-tools: Upgrade mysql on db1107 (m2 db master) - https://phabricator.wikimedia.org/T280251 (10hnowlan) No issues with `sockpuppet` being out for a bit. [16:38:50] 10DBA, 10OTRS, 10Performance-Team, 10Recommendation-API, 10SRE-tools: Upgrade mysql on db1107 (m2 db master) - https://phabricator.wikimedia.org/T280251 (10dpifke) No objection from `xhgui` owners. [16:59:19] 10DBA, 10OTRS, 10Performance-Team, 10Recommendation-API, 10SRE-tools: Upgrade mysql on db1107 (m2 db master) - https://phabricator.wikimedia.org/T280251 (10Marostegui) Thank you all for the fast response! [20:28:39] 10DBA: New database request: image_matching - https://phabricator.wikimedia.org/T280042 (10gmodena) Hey @Marostegui, Thanks for detailed reply and constructive feedback. > - Is MySQL the best place to store this materialized data? Have you considered hadoop perhaps? I would very much appreciate your input her... [23:06:23] PROBLEM - MariaDB sustained replica lag on db2133 is CRITICAL: 8.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2133&var-port=9104 [23:08:51] RECOVERY - MariaDB sustained replica lag on db2133 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2133&var-port=9104