[06:47:13] klausman: thanks for checking, but yes, it's OK to do so (which I see you did) [10:11:14] there's also a bit of an account/access question on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1164235 [10:18:59] Good news! 98.8% of commons thumbnail containers have consistent dbs. [10:19:11] Less good news! That means I've found 19 corrupt ones [10:39:44] sorry, that's a % of commons thumbnail containers that are consistent [10:39:55] sorry, that's a % of commons thumbnail container dbs that are consistent [10:39:59] ENOBRAIN [10:41:37] For the essential work tracker doc. I think something got messed up, there is a missing week (last week) and db updates of last week are in the current week [10:48:49] Emperor: based on history of the doc, I made changes, it might look like I deleted your updates but they are just moved to the top [11:19:34] I am soon finishing my day. I will have to return later to deploy https://gerrit.wikimedia.org/r/c/operations/puppet/+/1164302 as it has to be deployed before monday, but cannot be done until scheduled backups finish [11:34:31] Amir1: ack, ta [11:42:15] PROBLEM - MariaDB sustained replica lag on s7 on db2218 is CRITICAL: 21.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2218&var-port=9104 [11:43:51] a bunch of hosts are lagging a bit https://grafana.wikimedia.org/goto/rHkoKuPHR?orgId=1 [11:44:07] PROBLEM - MariaDB sustained replica lag on s7 on db2220 is CRITICAL: 14.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2220&var-port=9104 [11:44:47] db2218 overloaded with queries [11:44:57] updates [11:45:07] RECOVERY - MariaDB sustained replica lag on s7 on db2220 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2220&var-port=9104 [11:45:24] https://grafana.wikimedia.org/goto/jFgxKXEHR?orgId=1 [11:45:54] it's normalizing it seems [11:46:15] RECOVERY - MariaDB sustained replica lag on s7 on db2218 is OK: (C)10 ge (W)5 ge 4.4 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2218&var-port=9104 [11:46:18] https://grafana.wikimedia.org/goto/je0aKXENR?orgId=1 writes on the primary masters [11:46:21] now going down [11:48:27] UPDATE /* FRExtraCacheUpdate::invalidateIDs */ `page` SET page_touched = '20250627113557' WHERE page_id = * [11:48:33] huge invalidation [11:50:09] on arwiki [11:58:46] jynus: thanks for debugging, would you mind creating a ticket? [11:58:56] FlaggedRevs, the gift that keeps on giving [11:59:06] I'll fix it [11:59:32] sorry, I ended my day 15 minutes ago, I can do it next monday [11:59:52] nah, I'll do it then [12:00:01] enjoy your weekend [12:03:22] federico3: would you mind creating the ticket for this issue? I'm busy debugging https://phabricator.wikimedia.org/T397992 right now [12:03:44] ok [12:04:15] thanks [12:07:34] https://phabricator.wikimedia.org/T398033 [12:08:01] any specific next step I can do or put in the task? [12:26:24] that's good for now. Thanks! [14:49:02] I did my revert, have a nice week