[20:58:30] <mhbridgebot>	 [discord] <TechnoMage> So its cgroup OOM which means mariadb must have a misconfig some where @CosmicAlpha rather than something eating up all the ram
[20:58:52] <mhbridgebot>	 [discord] <TechnoMage> eg Maria is requesting more than its allowed in Cgroup config
[21:01:20] <mhbridgebot>	 [discord] <CosmicAlpha> Well yeah that makes sense, but they all should have the same amount on each of our db servers IIRC, so it didn't make sense why only one server (db121) seems to OOM often. Unless they don't actually have all the same resources.
[21:03:39] <mhbridgebot>	 [discord] <TechnoMage> I would run ```cat /proc/meminfo``` on them all to confirm they do have the same resources, and ill take a look at puppet to see what the config is 🙂
[21:07:02] <mhbridgebot>	 [discord] <CosmicAlpha> Thanks I'll try that in about 15 minutes. Stepped away from PC.
[21:08:07] <mhbridgebot>	 [discord] <TechnoMage> No rush from me, just throwing idea's/suggestions
[21:08:52] <mhbridgebot>	 [discord] <CosmicAlpha> Thanks a lot for that. This has been an issue I have been trying to figure out for months also, because it OOMs far to often (sometimes multiple times in just a few hours)
[21:10:33] <mhbridgebot>	 [discord] <paladox> we limited the buffer pool to 5g i think
[21:10:55] <mhbridgebot>	 [discord] <paladox> i have this change https://github.com/miraheze/puppet/pull/3013
[21:10:56] <MirahezeBot>	 [url] mariadb: stop using thread pool by paladox · Pull Request #3013 · miraheze/puppet · GitHub | github.com
[21:11:00] <mhbridgebot>	 [discord] <paladox> dunno if that would help
[21:11:12] <mhbridgebot>	 [discord] <paladox> also we have open tables cache at like 50k
[21:12:27] <mhbridgebot>	 [discord] <TechnoMage> So you have two things, the cgroup setting for the service, and then the config for maria... are the db servers dedicated nodes?
[21:13:12] <mhbridgebot>	 [discord] <paladox> well they are hosted as vms
[21:13:28] <mhbridgebot>	 [discord] <paladox> but they have around 20g of ram (apart from one of the smaller dbs that host our misc)
[21:13:48] <mhbridgebot>	 [discord] <paladox> it seems only db121 OOMs whereas the others doesn't
[21:13:58] <mhbridgebot>	 [discord] <paladox> https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=db121.miraheze.org&var-port=9100&viewPanel=78
[21:19:17] <mhbridgebot>	 [discord] <TechnoMage> 20G of ram, but a 5B buffer pool ? what else are you running on there
[21:19:39] <mhbridgebot>	 [discord] <paladox> Nothing else
[21:19:43] <mhbridgebot>	 [discord] <paladox> just mariadb 10.5
[21:34:49] <mhbridgebot>	 [discord] <CosmicAlpha> All the same
[21:42:58] <mhbridgebot>	 [discord] <paladox> we could lower the buffer pool to 1g, lower the open cache size and also stop using the thread mode in mariadb
[21:43:21] <mhbridgebot>	 [discord] <CosmicAlpha> Oh I might know why it happens only on db121
[21:43:26] <mhbridgebot>	 [discord] <paladox> oh?
[21:43:48] <mhbridgebot>	 [discord] <CosmicAlpha> Because that is our parsercache database, it is used more.
[21:43:56] <mhbridgebot>	 [discord] <paladox> oh
[21:47:37] <mhbridgebot>	 [discord] <paladox> > root@db121:/srv/mariadb/parsercache# du -sh ../parsercache
[21:47:38] <mhbridgebot>	 [discord] <paladox> > 63G    ../parsercache
[21:47:39] <mhbridgebot>	 [discord] <paladox> :wow:
[21:48:12] <mhbridgebot>	 [discord] <CosmicAlpha> I am looking at something if we can improve that...
[22:12:15] <mhbridgebot>	 [discord] <paladox> https://phabricator.wikimedia.org/T245489
[22:12:16] <MirahezeBot>	 [url] ⚓ T245489 Possibly disable optimizer flag: rowid_filter on 10.4 | phabricator.wikimedia.org
[22:12:22] <mhbridgebot>	 [discord] <paladox> wonder if we should disable that
[22:15:56] <mhbridgebot>	 [discord] <NotAracham | Away> This has been super fascinating to get caught up on.  Currently still in low availability, but love to get a better handle on service/logical server structures.
[22:16:28] <mhbridgebot>	 [discord] <CosmicAlpha> lol I was literally just about to say that (its why I opened this back up)
[22:16:40] <mhbridgebot>	 [discord] <CosmicAlpha> whoops. Wrong reply lol
[22:16:43] <mhbridgebot>	 [discord] <CosmicAlpha> @paladox
[22:17:33] <mhbridgebot>	 [discord] <paladox> heh
[22:17:46] <mhbridgebot>	 [discord] <paladox> well we could just use what the wmf set the optimiser is set to
[22:17:56] <mhbridgebot>	 [discord] <paladox> they use mrr which is https://mariadb.com/kb/en/multi-range-read-optimization/
[22:17:58] <MirahezeBot>	 [url] Multi Range Read Optimization - MariaDB Knowledge Base | mariadb.com
[22:18:05] <mhbridgebot>	 [discord] <paladox> was just reading which is festinating.
[22:21:01] <mhbridgebot>	 [discord] <paladox> optimiser can be done on the fly fyi
[22:21:03] <mhbridgebot>	 [discord] <paladox> it's dynamic
[22:23:33] <mhbridgebot>	 [discord] <paladox> ok... lets experiment. I'm going to set the optimiser_switch to 'mrr=on,mrr_cost_based=on,mrr_sort_keys=on,optimize_join_buffer_size=on,rowid_filter=off'
[22:23:36] <mhbridgebot>	 [discord] <paladox> @CosmicAlpha
[22:24:42] <mhbridgebot>	 [discord] <paladox> > | index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,ou
[22:24:46] <mhbridgebot>	 [discord] <paladox> is what it currently is
[22:28:03] <mhbridgebot>	 [discord] <paladox> ok, done, set.
[22:37:36] <mhbridgebot>	 [discord] <paladox> did you find anything
[22:43:23] <mhbridgebot>	 [discord] <paladox> @CosmicAlpha https://github.com/miraheze/mw-config/pull/4990 thoughts?
[22:43:24] <MirahezeBot>	 [url] Cache: shard parsercache table by paladox · Pull Request #4990 · miraheze/mw-config · GitHub | github.com
[22:43:52] <mhbridgebot>	 [discord] <CosmicAlpha> Not certain what that will do.
[22:44:39] <mhbridgebot>	 [discord] <paladox> ok we can give it a go i guess
[22:59:41] <mhbridgebot>	 [discord] <CosmicAlpha> @paladox https://github.com/miraheze/puppet/pull/3035 also
[22:59:42] <MirahezeBot>	 [url] mediawiki: add purge_parsercache cron by Universal-Omega · Pull Request #3035 · miraheze/puppet · GitHub | github.com
[22:59:53] <mhbridgebot>	 [discord] <paladox> nice
[23:00:29] <mhbridgebot>	 [discord] <paladox> shouldn't that equal 11 days not 21?
[23:00:39] <mhbridgebot>	 [discord] <paladox> as we have wgParserCacheExpireTime set to 10
[23:03:46] <mhbridgebot>	 [discord] <CosmicAlpha> Updated to 10 days. I will run it once to make sure it works right now also @paladox
[23:03:58] <mhbridgebot>	 [discord] <paladox> thanks!
[23:04:09] <mhbridgebot>	 [discord] <paladox> note that i've truncated the table
[23:06:24] <mhbridgebot>	 [discord] <CosmicAlpha> ```ps
[23:06:24] <mhbridgebot>	 [discord] <CosmicAlpha> Deleting objects expiring before Sat, 26 Nov 2022 23:05:57 GMT
[23:06:26] <mhbridgebot>	 [discord] <CosmicAlpha> ... 100.0% done (+1 iterations in 0.3s)
[23:06:27] <mhbridgebot>	 [discord] <CosmicAlpha> ```
[23:07:21] <mhbridgebot>	 [discord] <CosmicAlpha> @paladox okay to merge?
[23:07:30] <mhbridgebot>	 [discord] <paladox> yeh
[23:09:05] <mhbridgebot>	 [discord] <CosmicAlpha> done