[06:22:45] <elukey>	 joal: o/
[06:23:31] <elukey>	 you should be able (namely having perms) to restart hive on an-coord1001, don't be afraid of doing it in case it is needed and me/andrew are not around :)
[06:23:43] <elukey>	 if there are perms issues let me know and we'll sort them out
[06:26:04] <elukey>	 also it seems that an-coord1001 is too crowded
[06:26:05] <elukey>	 sigh
[07:09:56] <joal>	 hi elukey - I didn't even think about trying to restart hive - I asked in ops and got help
[07:10:10] <joal>	 elukey: I'll try to think about it next time :)
[07:11:11] <joal>	 elukey: also I don't think an-coord1001 is too crowded - it was me again not thinking enough and starting a job taking resources on it (namely, starting a manual rerun of mediawiki-history with local-driver, with a big driver)
[07:11:33] <joal>	 I'm sorry for too much not thinking enough - It seems everything is back in track
[07:18:54] <elukey>	 joal: https://grafana.wikimedia.org/d/000000377/host-overview?panelId=4&fullscreen&orgId=1&refresh=5m&var-server=an-coord1001&var-datasource=thanos&var-cluster=analytics&from=now-7d&to=now
[07:19:50] <elukey>	 the main issue is that a little more memory usage triggers the OOM
[07:21:18] <elukey>	 hive/presto have the xmx boundaries so we should be good, need to check mysql
[07:22:33] <elukey>	 also about the alarms - you definitely did the right thing asking for SRE's help, so please don't say sorry :) What I wanted to say is that if you want feel free to restart hive if you feel so, you should have the perms 
[07:22:49] <elukey>	 in case you don't feel strongly on it, it is super good to ask for SRE
[07:26:30] <elukey>	 the innodb innodb_buffer_pool_size is 4G, plus 8G for hive server, 4 for the metastore and 4 for presto
[07:27:00] <elukey>	 that is ~20G of fixed memory, plus the os consumption and a little page cache
[07:27:37] <elukey>	 we fluctuate between 21 and 25G, and the grand total of memory is 32G
[07:27:51] <elukey>	 ah wait, there is also oozie
[07:28:25] <elukey>	 but it is ~2g, so not a lot
[07:29:39] <elukey>	 I am saying too crowded since any variation of the workload (presto/hive/mysql/oozie) may force us to think about different memory settings, and we don't have a ton of space
[07:31:22] <elukey>	 I put some budget aside for expanding the hadoop master memory, I'll ask also for 64G of ram for an-coord 
[07:31:41] <elukey>	 (let me know if it makes sense or not)
[08:05:57] <wikibugs>	 10Analytics-Clusters, 10Project-Admins: Rename Analytics-Cluster to Analytics-Clusters (plural) and make it a subproject of Analytics - https://phabricator.wikimedia.org/T257059 (10Aklapper) 05Open→03Resolved I'm sorry for the repeated comments above; the Phab DB connection was quite flaky yesterday. Alrig...
[08:11:31] <wikibugs>	 10Analytics-Clusters, 10Project-Admins: Rename Analytics-Cluster to Analytics-Clusters (plural) and make it a subproject of Analytics - https://phabricator.wikimedia.org/T257059 (10elukey) thanks a lot @Aklapper!
[10:06:26] <joal>	 It makes a lot of sense elukey :)
[10:52:08] <wikibugs>	 (03PS1) 10Joal: Update geoeditors daily to not use map-join [analytics/refinery] - 10https://gerrit.wikimedia.org/r/609510
[10:52:38] <joal>	 !log Rerun mediawiki-geoeditors-monthly-wf-2020-06 after heisenbug (patch provided for long-term fix)
[10:52:40] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[13:01:17] <wikibugs>	 10Analytics, 10Analytics-Wikistats: Wikistats New Feature - https://phabricator.wikimedia.org/T257071 (10Aklapper) Hi @A455bcd9, thanks for taking the time to report this and welcome to Wikimedia Phabricator! For future reference, please always follow https://www.mediawiki.org/wiki/How_to_report_a_bug when cre...
[13:01:23] <wikibugs>	 10Analytics, 10Analytics-Wikistats: Recent "Edits per country" data not available - https://phabricator.wikimedia.org/T131596 (10Aklapper)
[13:01:25] <wikibugs>	 10Analytics, 10Analytics-Wikistats: Wikistats 2.0: Add statistics for the geographical origin of the contributors - https://phabricator.wikimedia.org/T188859 (10Aklapper)
[13:01:27] <wikibugs>	 10Analytics, 10Analytics-Wikistats: Wikistats New Feature - https://phabricator.wikimedia.org/T257071 (10Aklapper)
[19:31:56] <wikibugs>	 (03CR) 10Elukey: [C: 03+1] Update geoeditors daily to not use map-join [analytics/refinery] - 10https://gerrit.wikimedia.org/r/609510 (owner: 10Joal)
[19:37:34] <wikibugs>	 10Analytics, 10Analytics-Wikistats: Wikistats New Feature - https://phabricator.wikimedia.org/T257071 (10A455bcd9) Hi @Aklapper ,  Thanks. FYI on [[ https://stats.wikimedia.org/#/all-projects | Wikimedia Statistics ]], "New feature" directly points to Wikimedia Phabricator. It may be better to replace both the...
[19:40:58] <wikibugs>	 10Analytics, 10Analytics-Wikistats: Wikistats New Feature - https://phabricator.wikimedia.org/T257071 (10A455bcd9) @Aklapper , sorry I've just read T188859 and I don't see how they're related. T188859 is about "the geographical origin of the contributors for a given wiki" whereas this request is about the tota...