[00:49:00] PROBLEM - Check the last execution of monitor_refine_eventlogging_analytics_failure_flags on an-launcher1001 is CRITICAL: CRITICAL: Status of the systemd unit monitor_refine_eventlogging_analytics_failure_flags https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [05:39:31] RECOVERY - Check the last execution of monitor_refine_eventlogging_analytics_failure_flags on an-launcher1001 is OK: OK: Status of the systemd unit monitor_refine_eventlogging_analytics_failure_flags https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [05:45:37] !log reboot stat1007/8 for kernel upgrades [05:45:38] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [06:03:20] !log reboot an-conf100[1-3] for kernel upgrades [06:03:20] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [06:09:58] joal: bonjour! I solved the mistery of fsck (didn't forget about it) - on coord/launcher there is no truststore for the self-signed CA that we use, so it fails to validate.. if you execute the fsck from an-master1001 it works [06:17:47] Hi elukey - Ahhh! Fun :) Should we install a trust-store on an-coord, or do you prefer I take the usage of going to an-master for ops-style stuff? [06:18:19] elukey: I knew you had not forgotten, and I also knew you were having too many things to do and were having the right priorities :) [06:18:20] joal: let's use master for these use cases if possible [06:18:32] works for me elukey - thanks for trouble-shooting! [06:18:50] I should've thought about it sooner, I just realized it this morning [06:18:56] fresh mind + coffee :D [06:18:59] I know the feeling :) [06:19:08] COOOOoooooooOOOOOOOFFFEEEEEEE! [06:19:17] cheers [06:19:36] :) [06:19:55] I need to reboot the kerberos master host and the hadoop master hosts [06:19:58] fun morning [06:20:00] ack elukey [06:20:14] just done zookeeper, all good afaics [06:20:18] elukey: I can helps with eyes if you wish [06:20:54] (meaning I can look at stuff, restart failed jobs as needfed etc) [06:21:18] hopefully you'll not see anything :D [06:22:49] 10Analytics-Radar, 10Performance-Team: Invalid navigation timing events - https://phabricator.wikimedia.org/T254606 (10JAllemandou) >numeric JSON values outside of the normal long or double ranges are not valid JSON Hear! Hear! [06:24:25] !log reboot an-master100[1,2] for kernel upgrades [06:24:26] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [06:25:40] 10Analytics-Radar, 10Operations, 10Traffic: Spammy events coming our way for sites such us https://ru.wikipedia.kim - https://phabricator.wikimedia.org/T190843 (10JAllemandou) > I thought ru stand for Russia, this can just be a Russia version of wikipedia `ru.wikipedia.ORG` is official Russian wikipedia - `... [06:29:38] 10Analytics, 10Analytics-Kanban: Create intermediate dataset: pageview with actor information - https://phabricator.wikimedia.org/T255467 (10JAllemandou) [06:30:51] * joal has a grateful thought for elukey when pulling master and seeing a tons of `delete ... .jar` line [06:34:32] sloppy luca was sloppy sigh :( [06:34:57] nonono elukey - these deletio [06:35:01] are GREAT :) [06:37:22] speaking of space, https://grafana.wikimedia.org/d/000000585/hadoop?panelId=25&fullscreen&orgId=1&from=now-30d&to=now [06:38:12] ack elukey - will investigate - I have usual suspects in mind :) [06:40:11] !log reboot krb1001 for kernel upgrades [06:40:12] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [06:46:34] nothing exploded [06:46:45] doing the last master now [06:46:56] \o/ [06:50:04] interestngly elukey, my hdfs commands were stuck while you were doing the reboots (probably expected) [06:54:32] yeah I think that while doing reboots it may happen [06:54:49] no problem really - just info :) [06:56:35] 10Analytics-Radar, 10Performance-Team: Invalid navigation timing events - https://phabricator.wikimedia.org/T254606 (10Gilles) 05Resolved→03Open [07:02:03] all reboots done! [07:02:26] Awesome elukey [07:02:33] Thanks for that :) [07:08:11] elukey: No easy spotable culprit for data usage this time :( [07:10:55] 10Analytics-Radar, 10Performance-Team: Invalid navigation timing events - https://phabricator.wikimedia.org/T254606 (10Gilles) From our perspective, it would be ideal if these were validation errors that ended up in logstash. That would allow us to track per-UA trends without manual work. As for which limit t... [07:12:24] elukey: interestingly, it seems hdfs-rsync for dumps fail as well :( No data for 2020-06 :( [07:12:39] oh no not tru elukey - my bad sorry [07:14:03] I'll manually free some space by deleting old dumps, keeping 2 [07:14:13] elukey: ok with that? --^ [07:16:08] +1 [07:16:12] ack [07:16:59] (03PS3) 10Fdans: Add UDF that transforms Pagecounts-EZ projects into standard [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/597740 (https://phabricator.wikimedia.org/T252857) [07:17:05] !log Deleting mediawiki-history-text (avro) for 2020-01 and 2020-02 (we still have 2020-03 and 2020-04) - Expected free space: 160Tb [07:17:06] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:18:20] this is probably due to the rsyncs not working, I have to fix them this morning [07:18:39] elukey: nope - this is due to deletion-strategy suboptimal [07:19:09] elukey: I have deleted avro files (the ones generated parsing the XML and generating avro repr of text) [07:19:11] 10Analytics-Radar, 10Performance-Team: Invalid navigation timing events - https://phabricator.wikimedia.org/T254606 (10Gilles) @millimetric you can clear the data, thanks for extracting it for me. I was hoping that the UA string would tell us a more specific version of 78, but it does not. I guess we'll have t... [07:19:15] elukey: unrelated to rsync [07:20:51] joal: don't we use the --delete option for hdfs-rsync to drop old ones? [07:21:12] elukey: avro files are not generated by rsync :) [07:21:37] ok then I am confusing with dumps, nevermind [07:22:00] elukey: as for webrequest, there are 2 versions of dumps - One raw (copied via rsync), one extracted (avro, generated from raw with Spark) [07:22:40] The one taking most space is avro (using less efficient compression) [07:22:53] ah okok [07:23:15] elukey: I think we have a task about enforcing a smaller number of dumps being kept [08:04:41] * elukey brb [08:29:10] (03CR) 10Joal: "Ping @nuria on that one :)" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603590 (https://phabricator.wikimedia.org/T243241) (owner: 10Joal) [08:30:17] 10Analytics, 10Analytics-Kanban: Create intermediate dataset: pageview with actor information - https://phabricator.wikimedia.org/T255467 (10JAllemandou) [08:31:42] 10Analytics, 10Analytics-Kanban: Create intermediate dataset: pageview with actor information - https://phabricator.wikimedia.org/T255467 (10JAllemandou) I'm struggling with the name: The table will contain pageview AND redirect-to-pageview, so it's not not only pageview. Plus, it's a lot more similar to webre... [08:39:36] (03PS4) 10Fdans: Add special explode UDTF that turns EZ-style hourly strings into rows [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/596605 (https://phabricator.wikimedia.org/T252857) [08:39:59] (03PS4) 10Fdans: Add UDF that transforms Pagecounts-EZ projects into standard [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/597740 (https://phabricator.wikimedia.org/T252857) [08:45:58] (03PS1) 10Joal: [WIP] Add pageview_actor_hourly table and oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/606127 (https://phabricator.wikimedia.org/T255467) [09:00:58] joal: I am trying to fix the hdfs-rsync that handle the mediawiki/history data [09:01:12] is there any reason to keep the last two snapshots? [09:01:39] with only one the whole puppet code would be simpler, this is why I am asking [09:02:06] I also noticed that in data_purge.pp we have refinery-drop-mediawiki-history-dumps [09:02:45] ahhh ok so the --delete removes from the labstore nodes [09:02:46] okok [09:02:48] nevermind [09:13:31] (03PS5) 10Fdans: Add UDF that transforms Pagecounts-EZ projects into standard [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/597740 (https://phabricator.wikimedia.org/T252857) [09:19:48] sorry elukey I missed the ping - all good? [09:20:10] joal: yes, I was wondering if we really need two snapshots or only the last one [09:21:11] for the hdfs-rsync I mean [09:21:31] elukey: I think we do that in case some error happen on the new version, [09:22:11] okok [10:35:21] * elukey lunch! [10:38:33] (03PS2) 10Joal: [WIP] Add pageview_actor_hourly table and oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/606127 (https://phabricator.wikimedia.org/T255467) [11:15:36] 10Analytics, 10Analytics-Kanban: Make ActorSignatureGenerator class a non-singleton - https://phabricator.wikimedia.org/T255660 (10JAllemandou) [11:15:58] 10Analytics, 10Analytics-Kanban: Make ActorSignatureGenerator class a non-singleton - https://phabricator.wikimedia.org/T255660 (10JAllemandou) a:03JAllemandou [11:19:40] (03PS1) 10Joal: Make ActorSignatureGenerator non-singleton [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/606162 (https://phabricator.wikimedia.org/T255660) [12:46:46] wow - incredible bug :( [12:49:18] elukey: have you been doing restart or anything like that lately? [12:50:48] joal: lately means last hour, today? :D [12:50:58] elukey: like in the past hour [12:51:05] ah no I just got back [12:51:37] anything weird happening? [12:51:49] Got an heisenbug :() [12:52:20] Hive telling a class wasd not found, while the jar is there, the code is there and all other instances of the same workflow has been successful [12:52:46] anyway - restarted a second time and this time seems working [12:52:51] sorry for the noise elukey [12:54:59] nono it does ring a bell [12:55:30] but I can't recall exactly if it was a temp issue or something that I have investigated in the past [12:56:26] ack elukey - restarts have fixed the issue ... But needed more than one!P [12:58:06] very fun findings looking at fingerprinting today :) [12:59:47] taking a break - back for standup [13:00:28] (03CR) 10Joal: [V: 03+2] "tested on cluster - Marked WIP because of naming" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/606127 (https://phabricator.wikimedia.org/T255467) (owner: 10Joal) [13:03:02] ottomata: since the NewcomerTask error was with the schema, is there a way of pointing refine to the new schema? [13:04:23] milimetric: haven't checked emails yet but if the schema has a new fixed revision refine will noticei it if we re-refine it [13:04:27] refine picks the latest noe [13:04:28] one [13:05:09] oh my bad [13:05:24] ok, will re-refine, it should work ok now' [13:15:47] 10Analytics, 10Analytics-EventLogging, 10NewcomerTasks 1.2, 10Product-Analytics, and 2 others: NewcomerTask EventLogging schema has invalid array items type specification - https://phabricator.wikimedia.org/T255597 (10Milimetric) New events are being refined correctly with this schema, the hour that failed... [13:51:51] Only read a short explanation, but seems very cool and promising - https://jamesrwilcox.com/armada.pdf [14:11:08] hey mforns the data quality hourly bundle fails with "variable [user] cannot be resolved" [14:11:12] https://hue.wikimedia.org/oozie/list_oozie_workflow/0001293-200616151022463-oozie-oozi-W/?coordinator_job_id=0001292-200616151022463-oozie-oozi-C&bundle_job_id=0001290-200616151022463-oozie-oozi-B [14:16:13] milimetric: https://dumps.wikimedia.org/other/mediawiki_history/2020-05/ [14:16:16] :) [14:16:21] \o/ ! [14:17:22] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: mediawiki history dumps sync not working - https://phabricator.wikimedia.org/T255485 (10elukey) Done! https://dumps.wikimedia.org/other/mediawiki_history/2020-05/ [14:17:33] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: mediawiki history dumps sync not working - https://phabricator.wikimedia.org/T255485 (10elukey) a:03elukey [14:18:53] ah sweet elukey, I was looking for the CR this morning and didn't find it though [14:20:50] it wasn't there sorry [14:21:11] it took me a while, I wanted to refactor but at the end I used a shortcut [14:21:20] life is too short to battle with puppet and bash [14:31:01] milimetric: do you have a couple minutes to talk split/filter? [14:31:29] to the batcave! (fdans) [14:59:23] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Technical contributors emerging communities metric definition, thick data - https://phabricator.wikimedia.org/T250284 (10Nuria) Looking at data from the last 5 months: the bulk of edits is happening on the content namespace for both small and larger commu... [15:02:39] a-team finishing an interview be there in a bit [15:02:49] ack ottomata [15:02:54] fdans: ? [15:03:01] joal: doing interview! [15:03:08] ah sorry fdans [15:03:28] * joal would have liked to be interviewed by the ottomata/fdans pair :) [15:24:41] 10Analytics: Renaming "analytics-cluster" tag to "analytics-systems" and make into a subproject of analytics - https://phabricator.wikimedia.org/T255685 (10Nuria) [15:34:48] 10Analytics, 10Analytics-EventLogging, 10NewcomerTasks 1.2, 10Product-Analytics, and 2 others: NewcomerTask EventLogging schema has invalid array items type specification - https://phabricator.wikimedia.org/T255597 (10Ottomata) > This is actually valid syntax, although incorrect for the specific schema as... [15:35:06] 10Analytics, 10Analytics-EventLogging, 10NewcomerTasks 1.2, 10Product-Analytics, and 2 others: NewcomerTask EventLogging schema has invalid array items type specification - https://phabricator.wikimedia.org/T255597 (10Ottomata) Thanks BTW :) [15:36:48] * nshahquinn a-team: I have a whitelist patch (https://gerrit.wikimedia.org/r/c/analytics/refinery/+/604846) which Nuria has +2ed by which does not seem to be merged. Does anyone known what's going on there? [15:37:00] damn command + enter [15:37:14] (03CR) 10Fdans: [V: 03+2] Whitelist fields in the KaiOSAppFirstRun data stream [analytics/refinery] - 10https://gerrit.wikimedia.org/r/604846 (owner: 10Neil P. Quinn-WMF) [15:37:16] (03CR) 10Milimetric: [V: 03+2] Whitelist fields in the KaiOSAppFirstRun data stream [analytics/refinery] - 10https://gerrit.wikimedia.org/r/604846 (owner: 10Neil P. Quinn-WMF) [15:41:25] elukey: https://meet.google.com/puk-bnon-nvf [15:42:29] 10Analytics, 10Analytics-Kanban: Update skewed-join strategy in Mediawiki-history to prevent errors in case of task-retry - https://phabricator.wikimedia.org/T255548 (10Milimetric) +1 to the rev_id instead of the random number [15:45:06] weird, we fixed the mandatory user property here: https://github.com/wikimedia/analytics-refinery/commit/6a805b089dfbbeedf65942bc113f87eb393420aa [15:46:37] (03CR) 10Neil P. Quinn-WMF: "Does anyone know why this hasn't been merged even though Nuria has given a +2?" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/604846 (owner: 10Neil P. Quinn-WMF) [15:47:12] nshahquinn: that change is merged [15:47:19] (03CR) 10Neil P. Quinn-WMF: "> Patch Set 2:" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/604846 (owner: 10Neil P. Quinn-WMF) [15:47:40] 10Analytics-Radar, 10Product-Analytics (Kanban), 10User-Elukey: Learn how to make dashboard on top of data on hadoop/hive via presto - https://phabricator.wikimedia.org/T247329 (10cchen) created charts and filters via Presto from multiple tables in hive. (e.g. [[ https://superset.wikimedia.org/r/238 | new e... [15:47:41] * nshahquinn fdans: yup, I just noticed right after I replied. Sorry, and thanks! [15:47:47] damn command + enter [15:51:39] a-team: now that the white patch has been merged (thank you!), my main question: I think a separate deploy is needed for this change to take effect (because I remember a couple times over the years when it hasn't happened). Is there a schedule for that? How can I check whether it has happened? [15:51:49] woohoo I remembered not to hit command + enter [15:52:13] nshahquinn: we just deployed the weekly "train" so normally it would go out next week [15:52:39] but there might be random deploys in between now and then and if it's urgent we're happy to sync it up manually or do another deploy [15:53:15] mforns: ${user} is used here, but am I wrong that it should just use the user that launched the job? https://github.com/wikimedia/analytics-refinery/blob/master/oozie/data_quality_stats/workflow.xml#L105 [15:57:17] milimetric: ah, it goes out automatically on the train? That's good to know; I'll add it to the docs. [15:57:17] In this case, it would be very helpful if you could deploy it manually. It's critical that this data is preserved; I know there's a backup sanitization step after 45 days, but this way we can verify the preservation as soon as the production data starts arriving (which could be in the next couple of days). [15:57:51] ok nshahquinn I'll sync that file up manually soon [15:58:11] * nshahquinn thanks a bunch milimetric! [16:01:25] * elukey afk for a bit! [16:03:43] (03PS3) 10Joal: Add pageview_actor_hourly table and oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/606127 (https://phabricator.wikimedia.org/T255467) [16:04:13] (03CR) 10Joal: [C: 03+2] "Tested on cluster" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/606127 (https://phabricator.wikimedia.org/T255467) (owner: 10Joal) [16:28:11] milimetric: lookin [16:28:51] mforns: yeah I found it all over the place, same kind of use, so I still don't know what's wrong with this particular job [16:29:06] I've gotta do lunch but I can come brainbounce if you find something [16:29:20] milimetric: but other jobs have the user property defined in the properties file no? [16:29:26] oh milimetric no problem [16:30:06] nshahquinn: I synced up the whitelist, it should be getting used going forward unless I'm missing something. Let me know if you see problems [16:30:27] mforns: oh yeah, they have the user property defined... so are you saying we need to pass it for this one? [16:30:47] if so, let's add that to the command in the readme [16:30:51] milimetric: the property was there before, trying to remember why we removed it.. [16:57:34] !log produce searchsatisfaction events on group0 wikis via eventgate - T249261 [16:57:36] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:57:37] T249261: Vertical: Migrate SearchSatisfaction EventLogging event stream to Event Platform - https://phabricator.wikimedia.org/T249261 [16:57:46] milimetric: https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/591952/ PS4 mforns: "Should we remove this? It's not used anywhere." [16:58:01] Welllll, it turns out it is used somewhere :D [16:58:06] fixing [17:00:43] Gone for diner - back after [17:05:27] (03PS1) 10Mforns: Re-add user property to data quality bundle [analytics/refinery] - 10https://gerrit.wikimedia.org/r/606223 [17:05:35] milimetric: ^ [17:06:23] :) [17:07:51] (03PS2) 10Mforns: Re-add user property to data quality bundle [analytics/refinery] - 10https://gerrit.wikimedia.org/r/606223 [17:08:09] ok now it's correct [17:08:53] mforns: do you need it in the xmls? Doesn’t it just get passed regardless? [17:09:13] it does get passed regardless, yes [17:09:36] that's a decision that we took some time ago, to always pass all params explicitly [17:10:31] (03CR) 10Nuria: [C: 04-1] "mmmm... didn't we remove this and changed for it to be "database" instaed of user, right?" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/606223 (owner: 10Mforns) [17:10:32] I agree with you that having them all along the bundle-coord-wf chain is annoying, but there's the argument of fail-fast [17:11:22] nuria: database? [17:12:53] mforns: i need to look at that commit but the user there is used to persist data to a given database name right? [17:13:11] nuria: we removed the use of the ${user} to default fields like the database. But we also removed the actual user property (my fault as you can see above). That last part was wrong. This new patch only re-adds the user property, but does not make it the default for the database field. [17:13:41] mforns: ah i see [17:13:48] no no, the ${user} now only dictates temp folders [17:14:17] mforns: makes total sense, can we change commit message to express just that? [17:15:32] sure [17:16:48] (03PS3) 10Mforns: Re-add user property to data quality bundle [analytics/refinery] - 10https://gerrit.wikimedia.org/r/606223 [18:04:43] !log decommission matomo1001 [18:04:45] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:05:22] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Move Matomo to Debian Buster - https://phabricator.wikimedia.org/T252740 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by elukey@cumin1001 for hosts: `matomo1001.eqiad.wmnet` - matomo1001.eqiad.wmnet (**PASS**) -... [18:06:55] ok so matomo finally completely migrated to buster [18:07:05] old vm destroyed etc.. [18:09:02] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Move Matomo to Debian Buster - https://phabricator.wikimedia.org/T252740 (10elukey) [18:09:29] next in line is archiva :) [18:12:03] (03CR) 10Milimetric: [V: 03+2] "I manually synced this to the cluster today, so as to not have to wait for the next deploy." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/604846 (owner: 10Neil P. Quinn-WMF) [18:14:06] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Re-add user property to data quality bundle [analytics/refinery] - 10https://gerrit.wikimedia.org/r/606223 (owner: 10Mforns) [18:14:46] thx milimetric :] [18:14:56] gonna do a deploy so I can start the quality bundles back up [18:15:19] sorry for the spam etc. (thx you mforns for finding the problem) [18:15:42] no no, my bad for removing that property [18:15:52] btw mforns +1 to your backlog proposal, it's very clean [18:16:04] cool [18:21:17] (03PS1) 10Joal: Update unique-devices jobs to use pageview_actor [analytics/refinery] - 10https://gerrit.wikimedia.org/r/606233 (https://phabricator.wikimedia.org/T250744) [18:45:10] * elukey off! [19:00:53] Gone for tonight [19:11:52] 10Analytics-Radar, 10Product-Analytics (Kanban), 10User-Elukey: Learn how to make dashboard on top of data on hadoop/hive via presto - https://phabricator.wikimedia.org/T247329 (10cchen) 05Open→03Resolved [19:11:55] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review, 10User-Elukey: Kerberize Superset to allow Presto queries - https://phabricator.wikimedia.org/T239903 (10cchen) [19:27:36] 10Analytics, 10Growth-Team (Current Sprint), 10Product-Analytics (Kanban): Newcomer tasks: update schema whitelist for Guidance - https://phabricator.wikimedia.org/T255501 (10nettrom_WMF) [19:59:08] !log deployed quick fix for data stats job [19:59:09] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:20:15] (03CR) 10Nuria: [C: 04-1] Make ActorSignatureGenerator non-singleton (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/606162 (https://phabricator.wikimedia.org/T255660) (owner: 10Joal) [20:55:21] 10Analytics, 10Research: Can we have more RAM on stats machines - https://phabricator.wikimedia.org/T255716 (10Nuria) [21:17:45] 10Analytics, 10Research: Can we have more RAM on stats machines - https://phabricator.wikimedia.org/T255716 (10leila) For context, Nuria and I discussed this topic now and I'm going to make some changes in the title of the task and the description to expand it and make it also more specific. [21:21:04] 10Analytics, 10Research: Can we have more RAM and/or Storage on stat machines - https://phabricator.wikimedia.org/T255716 (10leila) [21:25:56] 10Analytics, 10Research: Can we have more RAM and/or Storage on stat machines - https://phabricator.wikimedia.org/T255716 (10Nuria) [21:28:29] 10Analytics, 10Research: Can we have more RAM and/or Storage on stat machines - https://phabricator.wikimedia.org/T255716 (10Nuria) These are teh machines in question: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Clients [21:47:04] 10Analytics, 10Research: Can we have more RAM on stat machines - https://phabricator.wikimedia.org/T255716 (10leila) [21:56:45] (03CR) 10Nuria: [C: 04-1] "Just 1 typo and some suggestions for commit message" (033 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/606233 (https://phabricator.wikimedia.org/T250744) (owner: 10Joal) [22:14:30] 10Analytics, 10Product-Analytics: Remove COUNT(*) from datasets when not useful in Superset & Turnilo - https://phabricator.wikimedia.org/T255725 (10kzimmerman)