[07:54:40] kormat: hey, for when you have time https://gerrit.wikimedia.org/r/675308 it's OKR stuff for migrating to puppet6 [07:57:44] please 🥺 [08:14:37] oh god, not the 🥺 [08:16:24] I have lots of weapons in my arsenal https://usercontent.irccloud-cdn.com/file/etH1N35P/image.png [08:17:10] 😭 [09:04:41] 10DBA, 10Platform Engineering, 10SRE, 10Wikimedia-Incident: Appservers latency spike / parser cache growth 2021-03-28 - https://phabricator.wikimedia.org/T278655 (10Kormat) Adding @Marostegui for visibility. [10:02:25] kormat, apologies for the ping, I went by the topic [10:02:55] jynus: no worries. note to self: when clinic duty is over next time, ensure i update the topic :) [10:03:09] indeed it is the first thing to do when leaving the "role" [10:03:15] to avoid this [10:03:18] :-P [10:03:36] or ensure someone else does :-) [10:06:26] I will remove subscribers from the new ticket, which was the whole point a creating a new one [15:20:44] 10DBA: Add a way to differentiate transcluding a redirect and transcluding a redirect and its target - https://phabricator.wikimedia.org/T278973 (10BrandonXLF) [15:21:52] 10DBA, 10Patch-For-Review: Add *_direct_link to imagelinks and templatelinks - https://phabricator.wikimedia.org/T278236 (10BrandonXLF) [15:21:54] 10DBA: Add a way to differentiate transcluding a redirect and transcluding a redirect and its target - https://phabricator.wikimedia.org/T278973 (10BrandonXLF) [15:24:44] 10DBA, 10SRE-tools, 10IPv6: Some Data Persistence DB clusters apparently do not support IPv6 - https://phabricator.wikimedia.org/T271140 (10crusnov) [15:25:16] 10DBA: Add a way to differentiate transcluding a redirect and transcluding a redirect and its target - https://phabricator.wikimedia.org/T278973 (10BrandonXLF) [15:25:34] 10DBA, 10SRE-tools, 10IPv6: Some Data Persistence DB clusters apparently do not support IPv6 - https://phabricator.wikimedia.org/T271140 (10crusnov) - Moved dbproxy to T271138 [15:26:04] 10DBA, 10SRE-tools, 10IPv6: Some Data Persistence DB clusters apparently do not support IPv6 - https://phabricator.wikimedia.org/T271140 (10crusnov) @fgiunchedi Is there any process we should follow to test/make sure everything is okay if we add ipv6 DNS for ms-be and ms-fe? [17:32:11] PROBLEM - MariaDB sustained replica lag on pc2008 is CRITICAL: 3.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [17:36:37] RECOVERY - MariaDB sustained replica lag on pc2008 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [19:16:23] PROBLEM - MariaDB sustained replica lag on pc2008 is CRITICAL: 3.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [19:23:23] RECOVERY - MariaDB sustained replica lag on pc2008 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [20:19:29] PROBLEM - MariaDB sustained replica lag on pc2008 is CRITICAL: 4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [20:26:25] RECOVERY - MariaDB sustained replica lag on pc2008 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [20:50:04] 10DBA, 10Phabricator, 10User-brennen: Phabricator intermittently slow; db connection failures to m3-master.eqiad.wmnet with "Temporary failure in name resolution" - https://phabricator.wikimedia.org/T279013 (10brennen) [20:53:43] 10DBA, 10Phabricator, 10serviceops, 10User-brennen: Phabricator intermittently slow; db connection failures to m3-master.eqiad.wmnet with "Temporary failure in name resolution" - https://phabricator.wikimedia.org/T279013 (10brennen) [20:56:51] 10DBA, 10Phabricator, 10serviceops, 10User-brennen: Phabricator intermittently slow; db connection failures to m3-master.eqiad.wmnet with "Temporary failure in name resolution" - https://phabricator.wikimedia.org/T279013 (10Reedy) Isn't this why in MW land we tend to use IP addresses rather than hostnames... [21:05:11] 10DBA, 10Phabricator, 10serviceops, 10User-brennen: Phabricator intermittently slow; db connection failures to m3-master.eqiad.wmnet with "Temporary failure in name resolution" - https://phabricator.wikimedia.org/T279013 (10Legoktm) For reference: ` $ host m3-master.eqiad.wmnet m3-master.eqiad.wmnet is an... [21:05:39] 10DBA, 10Phabricator, 10serviceops, 10User-brennen: Phabricator intermittently slow; db connection failures to m3-master.eqiad.wmnet with "Temporary failure in name resolution" - https://phabricator.wikimedia.org/T279013 (10mmodell) @reedy: perhaps? but we've had it configured that way forever. [21:07:46] 10DBA, 10Phabricator, 10serviceops, 10User-brennen: Phabricator intermittently slow; db connection failures to m3-master.eqiad.wmnet with "Temporary failure in name resolution" - https://phabricator.wikimedia.org/T279013 (10Reedy) Just because something has worked fine for a long time, doesn't mean it alwa... [21:09:22] 10DBA, 10Phabricator, 10serviceops, 10User-brennen: Phabricator intermittently slow; db connection failures to m3-master.eqiad.wmnet with "Temporary failure in name resolution" - https://phabricator.wikimedia.org/T279013 (10mmodell) I'm certainly not against doing it that way, I presume we could have puppe... [21:25:30] 10DBA, 10Phabricator, 10serviceops, 10Patch-For-Review, 10User-brennen: Phabricator intermittently slow; db connection failures to m3-master.eqiad.wmnet with "Temporary failure in name resolution" - https://phabricator.wikimedia.org/T279013 (10CDanis) I'm not sure if we tend to use IP addresses directly... [22:29:21] PROBLEM - MariaDB sustained replica lag on pc2008 is CRITICAL: 2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [22:33:51] RECOVERY - MariaDB sustained replica lag on pc2008 is OK: (C)2 ge (W)1 ge 0.4 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [23:06:17] PROBLEM - MariaDB sustained replica lag on pc2007 is CRITICAL: 2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2007&var-port=9104 [23:10:55] RECOVERY - MariaDB sustained replica lag on pc2007 is OK: (C)2 ge (W)1 ge 0.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2007&var-port=9104 [23:14:29] PROBLEM - MariaDB sustained replica lag on pc2008 is CRITICAL: 2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [23:16:51] RECOVERY - MariaDB sustained replica lag on pc2008 is OK: (C)2 ge (W)1 ge 0.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [23:52:17] PROBLEM - MariaDB sustained replica lag on pc2008 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [23:56:59] RECOVERY - MariaDB sustained replica lag on pc2008 is OK: (C)2 ge (W)1 ge 0.4 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104