[00:45:57] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on db2132 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2132&var-port=9104
[00:50:02] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on pc2007 is CRITICAL: 12.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2007&var-port=9104
[00:52:52] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on db2132 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2132&var-port=9104
[00:55:34] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on pc2007 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2007&var-port=9104
[01:02:16] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on pc2009 is CRITICAL: 8.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2009&var-port=9104
[01:05:36] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on pc2009 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2009&var-port=9104
[01:31:58] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on pc2007 is CRITICAL: 6.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2007&var-port=9104
[01:45:07] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on pc2007 is OK: (C)2 ge (W)1 ge 0.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2007&var-port=9104
[03:28:57] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on pc2008 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104
[03:31:45] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on pc2008 is OK: (C)2 ge (W)1 ge 0.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104
[04:31:47] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on pc2008 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104
[04:36:19] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on pc2008 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104
[07:56:28] <wikibugs>	 10DBA, 10SRE-tools, 10IPv6: Some Data Persistence DB clusters apparently do not support IPv6 - https://phabricator.wikimedia.org/T271140 (10fgiunchedi) >>! In T271140#6961082, @crusnov wrote: > @fgiunchedi Is there any process we should follow to test/make sure everything is okay if we add ipv6 DNS for ms-be...
[08:06:27] <wikibugs>	 10DBA, 10SRE-tools, 10IPv6: Some Data Persistence DB clusters apparently do not support IPv6 - https://phabricator.wikimedia.org/T271140 (10Volans) AFAICT at least on `ms-be` hosts I don't see the swift processes listen on v6, shouldn't that be addressed too?
[10:57:50] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on pc2008 is CRITICAL: 2.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104
[11:02:22] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on pc2008 is OK: (C)2 ge (W)1 ge 0.4 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104
[11:24:44] <wikibugs>	 10DBA, 10Add-Link, 10Growth-Team: Grant ALTER privileges to adminlinkrecommendation user on m2 - https://phabricator.wikimedia.org/T279053 (10kostajh)
[11:29:19] <wikibugs>	 10DBA, 10Add-Link, 10Growth-Team: Grant ALTER privileges to adminlinkrecommendation user on m2 - https://phabricator.wikimedia.org/T279053 (10kostajh)
[13:31:04] <wikibugs>	 10DBA, 10Add-Link, 10Growth-Team: Grant ALTER privileges to adminlinkrecommendation user on m2 - https://phabricator.wikimedia.org/T279053 (10LSobanski) p:05Triage→03Medium
[14:12:14] <wikibugs>	 10DBA, 10GrowthExperiments-MentorDashboard, 10GrowthExperiments-Mentorship, 10Growth-Team (Current Sprint), and 2 others: Create growthexperiments_mentor_mentee database table on extension1 for wikis in growthexperiments.dblist - https://phabricator.wikimedia.org/T278573 (10Urbanecm_WMF)
[14:13:21] <wikibugs>	 10DBA, 10GrowthExperiments-MentorDashboard, 10GrowthExperiments-Mentorship, 10Growth-Team (Current Sprint), and 2 others: Create growthexperiments_mentor_mentee database table on extension1 for wikis in growthexperiments.dblist - https://phabricator.wikimedia.org/T278573 (10Urbanecm_WMF)
[14:17:30] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on pc2008 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104
[14:25:32] <wikibugs>	 10DBA, 10GrowthExperiments-MentorDashboard, 10GrowthExperiments-Mentorship, 10Growth-Team (Current Sprint), and 2 others: Create growthexperiments_mentor_mentee database table on extension1 for wikis in growthexperiments.dblist - https://phabricator.wikimedia.org/T278573 (10LSobanski)
[14:51:41] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on pc2008 is OK: (C)2 ge (W)1 ge 0.4 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104
[15:00:22] <wikibugs>	 10DBA, 10SRE-tools, 10IPv6: Some Data Persistence DB clusters apparently do not support IPv6 - https://phabricator.wikimedia.org/T271140 (10LSobanski) I think this one's done from the #DBA perspective. Let me know if you think it's not the case.
[15:59:54] <wikibugs>	 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Cmjohnson)
[19:25:34] <wikibugs>	 10DBA, 10MediaWiki-Debug-Logger, 10Wikimedia-Rdbms, 10Developer Productivity: Log the query that caused a lock timeout - https://phabricator.wikimedia.org/T198755 (10Krinkle)
[19:55:29] <wikibugs>	 10DBA, 10Add-Link, 10Growth-Team: Grant ALTER privileges to adminlinkrecommendation user on m2 - https://phabricator.wikimedia.org/T279053 (10kostajh)
[20:35:17] <wikibugs>	 10DBA, 10Add-Link, 10Growth-Team: Grant ALTER privileges to adminlinkrecommendation user on m2 - https://phabricator.wikimedia.org/T279053 (10kostajh)