[01:37:31] 10DBA, 10Performance-Team (Radar): Evaluate WMF's ParserCache database setup for Multi DC - https://phabricator.wikimedia.org/T277831 (10Krinkle) [01:39:29] 10DBA, 10Performance-Team (Radar): Evaluate WMF's ParserCache database setup for Multi DC - https://phabricator.wikimedia.org/T277831 (10Krinkle) [01:43:15] 10DBA, 10Performance-Team (Radar): Evaluate WMF's ParserCache database setup for Multi DC - https://phabricator.wikimedia.org/T277831 (10Krinkle) > The concerned raised by @Kormat is that the current set up causes lag spikes, […] Could you elaborate on this? What is the cause-effect we see, and how is it caus... [08:56:55] 10DBA, 10SRE-tools: db-compare doesn't accept host:section_identifier style host definitions, only host_port ones - https://phabricator.wikimedia.org/T277843 (10jcrespo) [09:01:08] ^please check this, probably it has an easy fix [09:03:13] 10DBA, 10SRE-tools: db-compare doesn't accept host:section_identifier style host definitions, only host_port ones - https://phabricator.wikimedia.org/T277843 (10jcrespo) p:05Triage→03Low [10:11:25] 10DBA, 10SRE-tools: db-compare doesn't accept host:section_identifier style host definitions, only host_port ones - https://phabricator.wikimedia.org/T277843 (10Kormat) 05Open→03Resolved a:03Kormat [11:08:04] marostegui: there is something we need knowledge of our DBAs, we are abstracting page table which page_random is "real" type, in mysql it's an alias to "double precision" is it fine to change it to "float" instead? https://gerrit.wikimedia.org/r/c/mediawiki/core/+/643435 for sake of standardization [11:09:46] float is single precission [11:17:29] page_random is between 0 and 1. Does it matter for that? I can't get my head around this [11:18:09] https://www.irccloud.com/pastebin/xFSuuuAB/ [11:18:26] that I cannot know [11:18:36] I am just stating it will alter the data itself [11:19:00] double precission not only changes the max and min, also the "resolution" (number of decimals) [11:19:18] in particular around 0 [11:21:53] I am really confused. I need to dig more to find a solution. [11:22:11] just to be clear- I am not saying altering the data is bad [11:22:35] but that it alters it- I don't know the impact it will have on mw [11:24:17] yeah, it'll affect only Special:Random [11:24:52] (and its API counter part) [11:25:03] however, now that I think it will be a difficult schema change [11:25:23] as the data will have lower resolution, not sure it will be compatible [11:25:51] definitely needs more discussion [11:26:08] e.g. if I replicate a double into a float, will replication break? [11:28:22] https://dev.mysql.com/doc/mysql-replication-excerpt/5.7/en/replication-features-different-data-types.html says not, but certainly some extra care may be needed [11:36:33] > As of MySQL 8.0.17, the nonstandard FLOAT(M,D) and DOUBLE(M,D) syntax is deprecated and you should expect support for it to be removed in a future version of MySQL. [11:36:38] https://dev.mysql.com/doc/refman/8.0/en/floating-point-types.html [11:36:39] ugh [11:37:50] 10DBA, 10Performance-Team (Radar): Evaluate WMF's ParserCache database setup for Multi DC - https://phabricator.wikimedia.org/T277831 (10jcrespo) This is a duplicate of T133523 [14:05:41] 10DBA, 10wikitech.wikimedia.org: Move database for wikitech (labswiki) to a main cluster section - https://phabricator.wikimedia.org/T167973 (10LSobanski) @Andrew I talked to Nicholas earlier this week and here's what the situation is: in Q4 we will be focusing on MariaDB upgrades that will finally be unblocke... [19:47:53] 10DBA, 10Performance-Team (Radar): Evaluate WMF's ParserCache database setup for Multi DC - https://phabricator.wikimedia.org/T277831 (10Krinkle) >>! In T277831#6928235, @jcrespo wrote: > This is a duplicate of T133523 That is quite a broad and open-ended ticket. But we can continue the discussion there, sure... [19:48:04] 10DBA, 10Performance-Team (Radar): Evaluate WMF's ParserCache database setup for Multi DC - https://phabricator.wikimedia.org/T277831 (10Krinkle) [19:48:07] 10DBA, 10SRE, 10Epic, 10Performance-Team (Radar), 10Sustainability (Incident Followup): Decide how to improve parsercache replication, sharding and HA - https://phabricator.wikimedia.org/T133523 (10Krinkle) [19:48:30] 10DBA, 10SRE, 10Epic, 10Performance-Team (Radar), 10Sustainability (Incident Followup): Decide how to improve parsercache replication, sharding and HA - https://phabricator.wikimedia.org/T133523 (10Krinkle) a:03Kormat [19:48:38] 10DBA, 10Performance-Team (Radar): Evaluate WMF's ParserCache database setup for Multi DC - https://phabricator.wikimedia.org/T277831 (10Krinkle) p:05Triage→03Medium [19:48:57] 10DBA, 10SRE, 10Epic, 10Performance-Team (Radar), 10Sustainability (Incident Followup): Decide how to improve parsercache replication, sharding and HA - https://phabricator.wikimedia.org/T133523 (10Krinkle) >>! In T277831#6927485, @Krinkle wrote: >> The concerned raised by @Kormat is that the current se... [20:19:20] 10DBA, 10SRE, 10Wikimedia-Mailing-lists: Create test databases for mailman3 - https://phabricator.wikimedia.org/T256538 (10Legoktm) >>! In T256538#6920846, @Marostegui wrote: > Databases are now created, once I get the IPs I will create the users :) 208.80.154.13 (https://netbox.wikimedia.org/virtualization... [20:24:36] PROBLEM - MariaDB sustained replica lag on pc2008 is CRITICAL: 3 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [20:28:12] RECOVERY - MariaDB sustained replica lag on pc2008 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [21:27:37] PROBLEM - MariaDB sustained replica lag on pc2008 is CRITICAL: 2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [21:28:45] RECOVERY - MariaDB sustained replica lag on pc2008 is OK: (C)2 ge (W)1 ge 0.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104