[04:59:13] 10Blocked-on-schema-change, 10DBA: Schema change to make rc_id unsigned and rc_timestamp BINARY - https://phabricator.wikimedia.org/T276150 (10Marostegui) s3 eqiad [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1004 [] db1175 [] db1171 [] db1166 [] db1157 [] db1154 [] db1124 [] db1123 [] db1112 [] clouddb... [04:59:16] 10Blocked-on-schema-change, 10DBA: Drop default of rc_timestamp - https://phabricator.wikimedia.org/T276156 (10Marostegui) s3 eqiad [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1004 [] db1175 [] db1171 [] db1166 [] db1157 [] db1154 [] db1124 [] db1123 [] db1112 [] clouddb1021 [] clouddb1017 [] clouddb1013 [05:12:19] 10DBA, 10Platform Engineering, 10SRE, 10Wikimedia-Incident: Appservers latency spike / parser cache growth 2021-03-28 - https://phabricator.wikimedia.org/T278655 (10Marostegui) I am not fully sure I am reading the disk space graph correctly as I don't see an increase there. There's surely an increase on th... [05:44:50] 10DBA, 10Platform Engineering, 10SRE, 10Wikimedia-Incident: Appservers latency spike / parser cache growth 2021-03-28 - https://phabricator.wikimedia.org/T278655 (10Marostegui) I have done some testing with pc000 in a testing host. Deleted everything under 20 days so simulating that we only keep 20 days in... [05:52:18] 10DBA, 10GrowthExperiments-MentorDashboard, 10Growth-Team (Current Sprint), 10Patch-For-Review, 10User-Urbanecm_WMF (Engineering): Create database table to cache data about mentees - https://phabricator.wikimedia.org/T279587 (10Marostegui) I was going to ask what @Tgr actually explained, so thanks for th... [05:52:54] 10DBA, 10GrowthExperiments-MentorDashboard, 10Growth-Team (Current Sprint), 10Patch-For-Review, 10User-Urbanecm_WMF (Engineering): Create database table to cache data about mentees - https://phabricator.wikimedia.org/T279587 (10Marostegui) p:05Triage→03Medium [07:04:06] 10DBA: Upgrade 10.4.13 hosts to a higher version - https://phabricator.wikimedia.org/T279281 (10Marostegui) [08:14:36] 10DBA, 10Phabricator: Upgrade mysql on db1132 (phabricator db master) - https://phabricator.wikimedia.org/T279625 (10Marostegui) [08:14:46] 10DBA, 10Phabricator: Upgrade mysql on db1132 (phabricator db master) - https://phabricator.wikimedia.org/T279625 (10Marostegui) p:05Triage→03Medium [08:22:54] 10DBA: Upgrade 10.4.13 hosts to a higher version - https://phabricator.wikimedia.org/T279281 (10Marostegui) [08:22:57] 10DBA, 10Patch-For-Review: Failover m1 master: db1080 -> db1159 - https://phabricator.wikimedia.org/T276448 (10Marostegui) [08:24:41] 10DBA, 10Patch-For-Review: Failover m1 master: db1080 -> db1159 - https://phabricator.wikimedia.org/T276448 (10Marostegui) @jcrespo I would like to do this Wednesday 14th April - is this a good day or will it mess up with the backups? I have no problems in scheduling it any other day [08:26:58] 10DBA, 10Patch-For-Review: Failover m1 master: db1080 -> db1159 - https://phabricator.wikimedia.org/T276448 (10Marostegui) [08:43:46] 10DBA, 10Patch-For-Review: Failover m1 master: db1080 -> db1159 - https://phabricator.wikimedia.org/T276448 (10Marostegui) [08:44:02] 10DBA, 10Patch-For-Review: Failover m1 master: db1080 -> db1159 - https://phabricator.wikimedia.org/T276448 (10Marostegui) >>! In T276448#6982841, @Marostegui wrote: > @jcrespo I would like to do this Wednesday 14th April - is this a good day or will it mess up with the backups? I have no problems in schedulin... [08:45:05] 10DBA, 10Patch-For-Review: Failover m1 master: db1080 -> db1159 - https://phabricator.wikimedia.org/T276448 (10Marostegui) [08:47:23] 10DBA, 10Patch-For-Review: Failover m1 master: db1080 -> db1159 - https://phabricator.wikimedia.org/T276448 (10Marostegui) [09:09:00] 10DBA, 10Phabricator: Upgrade mysql on db1132 (phabricator db master) - https://phabricator.wikimedia.org/T279625 (10Marostegui) [09:24:34] 10DBA, 10Patch-For-Review: Productionize db21[45-52] and db11[76-84] - https://phabricator.wikimedia.org/T275633 (10Marostegui) db1177 is now replicating in s8. Will check its tables and won't pool in before the long weekend. [09:28:15] 10DBA, 10decommission-hardware: decommission db1076.eqiad.wmnet - https://phabricator.wikimedia.org/T274752 (10Marostegui) I will fully decommission this host on Tuesday. No point in decommissioning it today before a long weekend, just in case. [09:48:42] 10DBA, 10Patch-For-Review: Productionize db21[45-52] and db11[76-84] - https://phabricator.wikimedia.org/T275633 (10Marostegui) Ran: `sudo cumin 'db11[76-84].eqiad.wmnet' 'sudo lvextend -L+1100G /dev/mapper/tank-data && sudo xfs_growfs /srv'` [10:41:29] if you see a bunch of red hosts in tendril it is because I am upgrading a bunch of kernels [10:47:15] 10DBA: Create a database CPU saturation dashboard for codfw - https://phabricator.wikimedia.org/T261868 (10Marostegui) I am not sure if we should keep this open anymore. We are not having CPU usage problems in s8 anymore (we have most of hosts running 10.4 and we have lots of hosts already). If we do want to kee... [12:26:44] 10DBA: Create a database CPU saturation dashboard for codfw - https://phabricator.wikimedia.org/T261868 (10LSobanski) p:05Medium→03Low Dropping priority for now. I'll poke around the dashboards, see if I have any ideas and close if I won't. [12:52:08] 10DBA, 10cloud-services-team (Kanban): Upgrade mysql on db1128 (m5 db master) - https://phabricator.wikimedia.org/T279657 (10Marostegui) [12:52:29] 10DBA, 10cloud-services-team (Kanban): Upgrade mysql on db1128 (m5 db master) - https://phabricator.wikimedia.org/T279657 (10Marostegui) p:05Triage→03Medium [14:07:36] 10DBA, 10GrowthExperiments-MentorDashboard, 10Growth-Team (Current Sprint), 10Patch-For-Review, 10User-Urbanecm_WMF (Engineering): Create database table to cache data about mentees - https://phabricator.wikimedia.org/T279587 (10Urbanecm_WMF) >>! In T279587#6982489, @Marostegui wrote: > I was going to ask... [14:20:08] 10DBA, 10GrowthExperiments-MentorDashboard, 10GrowthExperiments-Mentorship, 10Growth-Team (Current Sprint), and 2 others: Create growthexperiments_mentor_mentee database table on extension1 for wikis in growthexperiments.dblist - https://phabricator.wikimedia.org/T278573 (10Urbanecm_WMF) [14:21:09] 10DBA, 10GrowthExperiments-MentorDashboard, 10Growth-Team (Current Sprint), 10Patch-For-Review, 10User-Urbanecm_WMF (Engineering): Create database table to cache data about mentees - https://phabricator.wikimedia.org/T279587 (10Marostegui) Thanks for the answers. They look good to me. x1 should be fine a... [14:24:03] 10DBA, 10GrowthExperiments-MentorDashboard, 10GrowthExperiments-Mentorship, 10Growth-Team (Current Sprint), and 2 others: Create growthexperiments_mentor_mentee database table on extension1 for wikis in growthexperiments.dblist - https://phabricator.wikimedia.org/T278573 (10Urbanecm_WMF) >>! In T278573#697... [14:44:11] 10DBA, 10GrowthExperiments-MentorDashboard, 10GrowthExperiments-Mentorship, 10Growth-Team (Current Sprint), and 2 others: Create growthexperiments_mentor_mentee database table on extension1 for wikis in growthexperiments.dblist - https://phabricator.wikimedia.org/T278573 (10Marostegui) heh, it took me a wh... [14:55:52] 10DBA, 10GrowthExperiments-MentorDashboard, 10Growth-Team (Current Sprint), 10Patch-For-Review, 10User-Urbanecm_WMF (Engineering): Create database table to cache data about mentees - https://phabricator.wikimedia.org/T279587 (10Urbanecm_WMF) Sure, here you go: ` CREATE TABLE /*_*/growthexperiments_mente... [15:06:57] marostegui: thanks for your reviews. In https://phabricator.wikimedia.org/T279587#6984040 you said that "x1 this doesn't look like a table that will have much load". Does that mean that x1 should not be used for tables that will likely have a high traffic, like the one I requested in T278573? [15:06:57] T278573: Create growthexperiments_mentor_mentee database table on extension1 for wikis in growthexperiments.dblist - https://phabricator.wikimedia.org/T278573 [15:07:28] Urbanecm: meeting, sorry :) [15:07:40] np, can wait :) [16:24:46] 10DBA, 10Analytics-Clusters, 10Analytics-Kanban, 10Data-Services, and 2 others: Convert labsdb1012 from multi-source to multi-instance - https://phabricator.wikimedia.org/T269211 (10fdans) [16:24:58] 10DBA, 10Analytics-Clusters, 10Analytics-Kanban, 10Data-Services, and 2 others: Convert labsdb1012 from multi-source to multi-instance - https://phabricator.wikimedia.org/T269211 (10fdans) 05Open→03Resolved [16:29:29] 10DBA, 10Analytics-Clusters, 10Analytics-Kanban, 10Data-Services, and 2 others: Convert labsdb1012 from multi-source to multi-instance - https://phabricator.wikimedia.org/T269211 (10elukey) 05Resolved→03Open >>! In T269211#6974963, @elukey wrote: > Almost! There are a couple of things left: > > - clou... [16:38:29] 10DBA, 10Analytics-Clusters, 10Analytics-Kanban, 10Data-Services, and 2 others: Convert labsdb1012 from multi-source to multi-instance - https://phabricator.wikimedia.org/T269211 (10razzi) @elukey Good point. With regards to the memory warning, we can: - lower the memory allocated to the sections, but thi... [16:40:05] 10DBA, 10Analytics-Clusters, 10Analytics-Kanban, 10Data-Services, and 2 others: Convert labsdb1012 from multi-source to multi-instance - https://phabricator.wikimedia.org/T269211 (10elukey) We have the following in hiera: ` # clouddb1021 profile::base::notifications: disabled [..] ` That needs to be remov... [17:23:23] 10DBA, 10cloud-services-team (Kanban): Upgrade mysql on db1128 (m5 db master) - https://phabricator.wikimedia.org/T279657 (10Legoktm) > @Ladsgroup @Legoktm can you make sure no imports or other testing would be on-going once we've decided on a day/time? Yep! [17:33:03] marostegui: hi, I don't know if you're around now but enwiki's watchlist is now 5% smaller, so let's leave it like that for growth but wikidata's watchlist is already cut to half, I'll clean up a bit more and you can shrink it if you want to [17:45:28] 10DBA, 10SRE, 10Wikimedia-Mailing-lists: Create production databases for mailman3 - https://phabricator.wikimedia.org/T278614 (10Ladsgroup) With the wikitech-l imported my last offer is now: 34GB. [17:46:08] 10DBA, 10SRE, 10Wikimedia-Mailing-lists: Create production databases for mailman3 - https://phabricator.wikimedia.org/T278614 (10Ladsgroup) [18:55:23] PROBLEM - MariaDB sustained replica lag on pc2008 is CRITICAL: 2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [19:00:15] RECOVERY - MariaDB sustained replica lag on pc2008 is OK: (C)2 ge (W)1 ge 0.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [20:18:52] 10DBA, 10Add-Link, 10Growth-Team (Current Sprint): Grant ALTER privileges to adminlinkrecommendation user on m2 - https://phabricator.wikimedia.org/T279053 (10kostajh) [21:29:01] 10DBA, 10Platform Engineering, 10SRE, 10Wikimedia-Incident: Appservers latency spike / parser cache growth 2021-03-28 - https://phabricator.wikimedia.org/T278655 (10Krinkle) >>! In T278655#6982467, @Marostegui wrote: > I am not fully sure I am reading the disk space graph correctly as I don't see an increa... [21:29:13] 10DBA, 10Platform Engineering, 10SRE, 10Performance-Team (Radar), 10Sustainability (Incident Followup): Appservers latency spike / parser cache growth 2021-03-28 - https://phabricator.wikimedia.org/T278655 (10Krinkle)