[14:02:15] <Emperor>	 Amir1: team meeting?
[14:02:41] <Amir1>	 Emperor: can't make it today, sorry :(
[14:02:50] <Emperor>	 OK
[16:29:29] <Amir1>	 marostegui: look at this 🤯 https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&from=now-1h&to=now&timezone=utc&var-job=$__all&var-server=db2207&var-port=9104&refresh=1m&viewPanel=panel-3
[16:29:55] <Amir1>	 read_prev and read_rnd_next gone to zero basically
[16:31:17] <marostegui>	 just posted: https://phabricator.wikimedia.org/T416171#11597514
[16:51:50] <cdanis>	 wowoww
[16:57:32] <cdanis>	 is it normal to have so much locking go via mariadb?
[18:48:23] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s1 on db1154 is CRITICAL: 127 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1154&var-port=13311
[18:49:23] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s1 on db1154 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1154&var-port=13311
[21:42:28] <Amir1>	 cdanis: T366938 no not normal. As long as mediawiki treats db like a semaaphore as a service, we are going to have issues
[21:42:29] <stashbot>	 T366938: Reduce relying on database locks  - https://phabricator.wikimedia.org/T366938
[21:43:36] <cdanis>	 yeah
[21:43:40] <cdanis>	 we wrote poolcounter for a reason