[03:29:12] 10Analytics, 10Analytics-Kanban, 10Chinese-Sites: MediaWiki history dumps have some events in 2025 - https://phabricator.wikimedia.org/T235269 (10JAllemandou) Hi @Samwalton9, thanks @nshahquinn-wmf for reopening :) The revision data with wrong timestamp has been removed from the dumps we generate (see https:... [03:35:18] 10Analytics: Add "did edit" field to pageview_actor - https://phabricator.wikimedia.org/T277785 (10JAllemandou) If we were to add some info to the table, we would do it using a flag, similarly to how we flag `is_pageview` and `is_redirect_to_pageview`. I suggest we would use `is_edit` as field name. The 3 named... [03:46:50] (03CR) 10Joal: "I finally got to that - sorry for the long turnaround." (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/666209 (https://phabricator.wikimedia.org/T274690) (owner: 10Milimetric) [04:04:33] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10Patch-For-Review: Sanitize and ingest all event tables into the event_sanitized database - https://phabricator.wikimedia.org/T273789 (10JAllemandou) The hack is great, and it will enable us to move fast. I have concerns about two things: - It probably w... [06:32:03] good morning! [06:40:45] (03CR) 10Elukey: [WIP] Update mysql resolver to work with cloud replicas (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/666209 (https://phabricator.wikimedia.org/T274690) (owner: 10Milimetric) [07:33:23] Good morning :) [07:49:47] (03CR) 10Addshore: [V: 03+2 C: 03+2] Track editor numbers split by namespace [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/672823 (https://phabricator.wikimedia.org/T275999) (owner: 10Silvan Heintze) [07:50:47] (03Merged) 10jenkins-bot: Track editor numbers split by namespace [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/672823 (https://phabricator.wikimedia.org/T275999) (owner: 10Silvan Heintze) [08:14:21] joal: o/ [08:14:26] Hi elukey [08:14:56] yesterday I have rebuilt alluxio 2.4.1 on top of Bigtop 1.5 (since upstream ships a 1.6 version) [08:15:25] I'll try to play with it in hadoop test during the next days [08:15:46] ack elukey [08:16:22] elukey: let me know if you wish to test! [08:16:34] https://docs.alluxio.io/os/user/stable/en/deploy/Running-Alluxio-On-a-Cluster.html [08:16:40] this seems to be a good starting point [08:16:54] also elukey: shall we test the capacity scheduler on the test machine? [08:17:31] test cluster sorry [08:17:53] yes yes once we reach the final version for sure, I have already the patch ready for the test cluster [08:18:04] for alluxio, I bumped it to 2.x due to [08:18:05] https://docs.alluxio.io/os/user/stable/en/deploy/Running-Alluxio-On-a-HA-Cluster.html [08:18:30] it seems that to keep the shared journal raft can be used, without the need of zookeeper [08:19:33] but we could also use zookeeper since we have a cluster :) [08:21:50] elukey: you tell me what you prefer :) [08:24:37] 10Analytics: Test Alluxio as cache layer for Presto - https://phabricator.wikimedia.org/T266641 (10elukey) I rebuilt Alluxio with Docker using the bigtop1.5 repo as baseline, and applying https://github.com/apache/bigtop/pull/724 to get the 2.4.1 version. https://docs.alluxio.io/os/user/stable/en/deploy/Running... [08:31:03] elukey: about mysql for cloud - I wonder if we should make sure we have 2 different DNS sets, in case we need to evolve one and not the other [08:31:07] elukey: is it worth? [08:33:22] joal: the only set of DNS that I'd think to keep separate may be the CNAMEs, the SRV records are ok in my opinion [08:33:27] (since the ports are standard) [08:33:59] it takes really no time to create s1-analytics-clouddb.etc.. [08:34:05] if you want we can do it [08:35:54] elukey: ah - so the ports are globally standard? [08:37:24] elukey: my concern is diverging DNS records for analytics, making cloud connection break [08:38:12] elukey: if you think this is not probable, let's not add the DNS [08:39:07] also elukey: all HDFS nodes are centered in term od usage :) [08:42:07] joal: yes as far as I know we are using the same ports as the dbstores and the other clouddb nodes [08:42:13] there is a convention about it [08:42:18] (from data persistence) [08:42:26] ok great [08:42:33] it's about host(s) then :) [08:42:41] exactly yes [08:43:04] elukey: do we have multiple hosts in analytics cluster? [08:43:14] I know we have a single host in labs [08:45:33] joal: for dbstore we have 3, for cloud only the beefy ex labsdb1012, but we are still not sure if it will be enough.. I hope so, but Razzi and Dan are still testing sqoop [08:45:44] yup [08:45:44] (they were doing it yesterday, not sure how it went) [08:46:12] elukey: Shall we assume that hosts might diverge in the future for cloud? [08:46:43] elukey: that's my opinion (whether 2 hosts or 3, I think we're gonna end up using more) [08:47:06] joal: I have no idea :) [08:47:24] elukey: or, we'll get all data from analytics and sanitize ourselves [08:47:30] not sure ... [08:47:39] anyhow - Let's keep the hack :) [08:47:46] ok elukey --^ ? [08:47:51] sure :) [08:48:03] ok cool - adding a comment to Dan's CR [08:49:46] (03CR) 10Joal: [WIP] Update mysql resolver to work with cloud replicas (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/666209 (https://phabricator.wikimedia.org/T274690) (owner: 10Milimetric) [08:49:50] done elukey --^ [08:54:15] elukey: do you wish me to reset the HDFS bandwidth, or do we keep it as it is now? [08:55:39] joal: I'd keep it as it is, I don't see a lot of problems, the prev value was really low [08:57:08] works for me elukey thanks :) [09:04:08] * elukey coffee! [09:48:27] 10Analytics, 10Analytics-Kanban, 10Chinese-Sites: MediaWiki history dumps have some events in 2025 - https://phabricator.wikimedia.org/T235269 (10Samwalton9) 05Open→03Resolved Ah, that makes sense - I misread a comment above and thought this data shouldn't be in the table anymore. Thanks for the context :) [10:15:52] joal: ping me if you need me for the broken refinery build. https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/673290 should be enough to unbreak it. [10:17:09] Heya gehel - Will merge and recheck :) [10:17:20] scream if it is still broken! [10:17:21] many thanks gehel [10:17:52] and as specified in the comments, this will need to be reviewed when we move to Java 11, otherwise it is likely to break the build at that point. [10:19:01] makes sense gehel [10:20:04] gehel: no idea about java11 for us though (hadoop is slow moving) [10:20:20] we'll see when we get there ! [11:23:46] 10Analytics-EventLogging, 10Analytics-Radar, 10Front-end-Standards-Group, 10MediaWiki-extensions-WikimediaEvents, and 5 others: Provide a reusable getEditCountBucket function for analytics purposes - https://phabricator.wikimedia.org/T210106 (10awight) @Krinkle Here's a dirty idea: What if the producer cod... [11:25:20] (03CR) 10Joal: [C: 03+2] "Merging to fix other builds" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/673290 (owner: 10Gehel) [11:32:34] (03Merged) 10jenkins-bot: Fix failing sonar analysis due to JDK11 removing tools.jar [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/673290 (owner: 10Gehel) [11:33:39] 10Analytics-EventLogging, 10Analytics-Radar, 10Front-end-Standards-Group, 10MediaWiki-extensions-WikimediaEvents, and 5 others: Provide a reusable getEditCountBucket function for analytics purposes - https://phabricator.wikimedia.org/T210106 (10awight) >>! In T210106#6928191, @awight wrote: > @Krinkle Here... [11:37:15] * elukey lunch! [11:42:56] (03CR) 10Awight: "(did not merge yet, maybe it needs a rebase or manual submit?)" [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/668032 (https://phabricator.wikimedia.org/T276308) (owner: 10Awight) [12:09:58] (03PS3) 10Joal: Update WMF domain list with Cloud and toolforge [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/673202 (https://phabricator.wikimedia.org/T277536) [12:57:10] gehel: I confirm your fix fixes! [12:58:27] Cool ! [13:00:15] thanks again :) [13:55:41] 10Analytics: Add "did edit" field to pageview_actor - https://phabricator.wikimedia.org/T277785 (10Isaac) > If we were to add some info to the table, we would do it using a flag, similarly to how we flag is_pageview and is_redirect_to_pageview. I suggest we would use is_edit as field name. The 3 named flags woul... [14:15:40] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10Patch-For-Review: Sanitize and ingest all event tables into the event_sanitized database - https://phabricator.wikimedia.org/T273789 (10Ottomata) @JAllemandou yeah good point about the Spark 3...fortunately I just tried and it works in Spark 3 too! Agre... [14:39:45] a-team gamez? [14:39:57] we're all in the hangout you made yo [14:40:04] but we're working [14:40:08] wtf i changed it [14:40:12] milimetric: link? [14:40:18] https://meet.google.com/saw-bfdt-crd [14:41:17] 10Analytics: Test Alluxio as cache layer for Presto - https://phabricator.wikimedia.org/T266641 (10elukey) To ruin all the initial fun, https://docs.alluxio.io/ee/user/2.4/en/operation/Kerberos-Security-Setup.html lists quite a few things to keep in mind to test alluxio :) [14:47:49] wow elukey :S -^ [14:48:41] 10Analytics-Radar, 10Machine-Learning-Team, 10SRE: Kubeflow on stat machines - https://phabricator.wikimedia.org/T275551 (10akosiaris) Just a few clarifications and answers. > cloud vps is a kubernetes cluster It's toolforge that's half powered by a kubernetes cluster. The other half is powered by son of g... [14:51:58] joal: yes I skipped that part on purpose up to now, I knew kerberos needed to be involved :) [14:52:22] good for a friday :) [15:04:34] elukey: come join us!!! we're playing geoguessr [15:06:13] fdans: I'd prefer some quiet time hacking, this week was ETOOMANYMEETINGS for me [15:06:23] elukey: okeeeeei [15:44:04] !log rebalance kafka partitions for webrequest_text partition 1 [15:44:06] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:08:29] joal: Alluxio is a little more powerful/complicated than I expected - https://docs.alluxio.io/os/user/2.4/en/core-services/Unified-Namespace.html [16:15:05] 10Analytics, 10Patch-For-Review: Test Alluxio as cache layer for Presto - https://phabricator.wikimedia.org/T266641 (10elukey) Part of the complexity related to the security config is the fact that two users needs to be kerberized IIUC: * alluxio -> running master/worker daemons * alluxiohdfs -> for the UFS s... [16:40:41] 10Analytics, 10Product-Analytics (Kanban): Hive table neilpquinn.toledo_pageviews missing almost all data - https://phabricator.wikimedia.org/T277781 (10nshahquinn-wmf) a:03nshahquinn-wmf [17:01:00] hey a-team: when I create tables in Hive using default settings (in other words just "CREATE TABLE …"), querying it throws a lot of warning/info messages to the point where CLI querying becomes meaningless. Is this worthy of a phab task, or would you like a copy & paste of the warning? [17:01:18] Nettrom: +1 for the task [17:01:33] elukey: alright, I'll make that happen [17:07:38] 10Analytics, 10Product-Analytics: Default table creation settings results in warnings when querying - https://phabricator.wikimedia.org/T277822 (10nettrom_WMF) 05Invalid→03Open [17:16:58] 10Analytics-Radar, 10SRE, 10ops-eqiad: Try to move some new analytics worker nodes to different racks - https://phabricator.wikimedia.org/T276239 (10Cmjohnson) @elukey can I move the 2 servers anytime or does this need to be scheduled? Move an-worker1129 to A2 Move an-worker1139 to A7 [17:23:19] 10Analytics-Radar, 10SRE, 10ops-eqiad: Try to move some new analytics worker nodes to different racks - https://phabricator.wikimedia.org/T276239 (10elukey) @Cmjohnson anytime is fine! Thanks :) [17:31:01] (03PS1) 10Mforns: Comment out hourly traffic anomalies job in favor of the daily one [analytics/refinery] - 10https://gerrit.wikimedia.org/r/673547 [17:31:48] hmm, a-team did we ever have any fanciness for generating a good between where clause for hourly partitions? [17:32:15] Not that I know ottomata :( [17:32:19] ottomata: ah maybe [17:32:21] ottomata: I think there's some scala code somewhere that does it no? [17:32:33] I made a javascript bookmarklet for it like five years ago but I don't remember if I tested it well [17:32:46] milimetric: for between 2 dates or just equality? [17:33:49] I think it was for a range. It's pretty easy to whip up [17:34:11] ottomata: https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/383761/8/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/sessions/TimePartitions.scala [17:34:28] That's the best I can think of [17:34:29] joal!!!!!! [17:37:23] ottomata: HiveToDruid.scala has one too... https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/HiveToDruid.scala#L276 [17:40:57] mforns: !!!!! [17:41:10] these are amazing, we should put one of these into a useable place [17:41:44] sure, we can put both, they seem to cover different use cases: timestamps vs partition specs [17:42:00] but we'll use Iceberg soon no? [17:48:27] "soon" [17:48:28] :D [17:49:27] hehe [17:55:05] (03CR) 10Mforns: "So sorry... Forgot to click submit." [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/668032 (https://phabricator.wikimedia.org/T276308) (owner: 10Awight) [18:05:57] mforns: either interface is useful [18:05:59] i use DateTimes [18:06:05] not sure whihc implementaiton is better [18:08:19] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade to Superset 1.0 - https://phabricator.wikimedia.org/T272390 (10razzi) I built the latest superset wheel from source so it'd have the fix for https://github.com/apache/superset/issues/13229, but the wheel is too large to be uploaded to gerr... [18:11:00] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade to Superset 1.0 - https://phabricator.wikimedia.org/T272390 (10Ottomata) @EBernhardson somehow uses archiva.wikimedia.org. +1 for k8s. [18:18:27] razzi: how big is the superset wheel from pypi? It is strange that yours get to 100MB [18:18:37] also, how are you building it? [18:21:09] elukey: hmm good point, it's only 40mb from upstream [18:21:24] I ran python3 setup.py bdist_wheel to build mine [18:21:52] razzi: IIRC there should be a frozen-requirements file for the custom builds in the repo [18:22:19] in which you can specify a git path, I used it in the past and it seemed working [18:22:33] not sure if it will lead to a different wheel size [18:22:47] but it is definitely weird that there is a 60MB difference [18:24:17] the kubernetes move is surely something to consider, but superset is kerberized so a keytab needs to be available in the pods [18:24:37] and usually the principal is something/hostname@WIKIMEDIA [18:24:53] in case of a pod no idea what that should look like [18:26:02] (the ips of the pods are not fixed) [18:26:08] ottomata: do you have any idea? [18:26:28] if we solve this point (and it will be the first kerberos use case in k8s) then we should be good [18:26:53] elukey: ok cool, thanks for the tip on the frozen-requirements, trying that now [18:28:32] razzi: not sure if it changes anything though :( [18:31:11] 10Analytics, 10Patch-For-Review: Add better monitoring for Analytics UIs - https://phabricator.wikimedia.org/T277729 (10razzi) @elukey I made a patch for turnilo, should be straightforward as you said. For superset, if monitoring check come from the host itself, we should be able to use the user header and ca... [18:39:53] going afk folks, have a good weekend! [18:40:21] elukey: oo good point i had not thought of that complexity [18:40:31] laters elukey ! [18:48:58] 10Quarry, 10cloud-services-team (Kanban): Do some checks of how many Quarry queries will break in a multiinstance environment - https://phabricator.wikimedia.org/T267989 (10Bstorm) Did we ever get a tally of the queries that would break here? I'm not seeing it in the comments, but I thought you had numbers @dc... [19:02:45] milimetric: will lookinto that failing inukapageview shortly [19:04:06] ottomata: oh my bad I missed that it was passing title=null, indeed the config is there if you use https://meta.wikimedia.org/w/api.php?action=jsonschema&formatversion=2&format=json&title=InukaPageView [19:04:18] thanks for checking [19:05:12] milimetric: that schema has been migrrated...have to look to see how far along it got [19:10:56] 10Analytics: Odd behavior in unique device counts - https://phabricator.wikimedia.org/T276472 (10Milimetric) a:05Milimetric→03None Honestly I don't think I'll have time to look at this anytime soon, gotta finish up some quarter stuff and start two other big projects (Airflow and Data Governance). I'm unlick... [19:28:13] (03CR) 10Jdlrobson: [C: 03+1] "ottomata - any problems with me merging...?" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/668743 (https://phabricator.wikimedia.org/T275766) (owner: 10Phuedx) [19:46:48] (03CR) 10Ottomata: "I think this will work with our current Hive ingestion process, but I'm not so sure it will in the future." [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/668743 (https://phabricator.wikimedia.org/T275766) (owner: 10Phuedx) [19:48:09] mforns_brb: you have this old task! https://phabricator.wikimedia.org/T212451 [19:48:16] am adapting your code into HivePartition [19:48:59] 10Analytics, 10Analytics-Kanban: Create Spark code to compare DateTimes with partition columns - https://phabricator.wikimedia.org/T212451 (10Ottomata) a:03Ottomata [20:12:51] ottomata: uou, didn't remember [20:13:50] ottomata: are you using the one in the task or the one I pasted earlier? [20:18:05] (03PS1) 10Majavah: Switch to systemd [analytics/udplog] - 10https://gerrit.wikimedia.org/r/673596 (https://phabricator.wikimedia.org/T276623) [20:52:21] (03CR) 10Legoktm: [C: 04-1] "Yay! The dh_installinit in d/rules also needs to be swapped to dh_installsystemd." (033 comments) [analytics/udplog] - 10https://gerrit.wikimedia.org/r/673596 (https://phabricator.wikimedia.org/T276623) (owner: 10Majavah) [20:52:26] mforns: the one you pasted earlier [21:09:46] 10Analytics, 10Analytics-EventLogging, 10Better Use Of Data, 10Event-Platform, and 4 others: KaiOS / Inuka Event Platform client - https://phabricator.wikimedia.org/T273219 (10Ottomata) @sbisson I think there is a slight issue. Commented here: https://github.com/wikimedia/wikipedia-kaios/commit/0581c22b7... [21:58:06] (03PS1) 10Ottomata: [WIP] Add support for finding RefineTarget inputs from Hive [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/673604 (https://phabricator.wikimedia.org/T212451) [22:02:30] (03CR) 10Ottomata: "@Joal, this isn't quite ready for full review, but could you take a look at RefineTarget starting with around line 710, where I've redoing" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/673604 (https://phabricator.wikimedia.org/T212451) (owner: 10Ottomata) [22:06:23] (03PS2) 10Ottomata: [WIP] Add support for finding RefineTarget inputs from Hive [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/673604 (https://phabricator.wikimedia.org/T212451) [22:08:49] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Add support for finding RefineTarget inputs from Hive [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/673604 (https://phabricator.wikimedia.org/T212451) (owner: 10Ottomata) [22:16:00] 10Analytics, 10Patch-For-Review: Add better monitoring for Analytics UIs - https://phabricator.wikimedia.org/T277729 (10razzi) Here's a curl command that checks if superset is running: ` curl -L 'http://localhost:9080/login/' -H 'X-Remote-User: admin' -c cookiejar-$RANDOM ` For Nagios to implement this, we'l... [22:25:59] 10Analytics, 10Patch-For-Review: Add better monitoring for Analytics UIs - https://phabricator.wikimedia.org/T277729 (10razzi) Apparently check_http does not support following redirects with cookies: https://github.com/monitoring-plugins/monitoring-plugins/issues/1423 [22:42:16] (03CR) 10Sharvaniharan: "@MHolloway @ottomata one tiny clarification: The 'reasons' attribute on this schema is actually a list of integers in Java. In here I have" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/668244 (owner: 10Sharvaniharan) [23:03:15] 10Analytics-Radar, 10Growth-Scaling, 10Growth-Team (Current Sprint), 10Product-Analytics (Kanban): Growth: update welcome survey aggregation schedule - https://phabricator.wikimedia.org/T275172 (10nettrom_WMF) The first part of this work is now completed, and @Tgr can go ahead with T275171 and shorten the... [23:24:24] (03CR) 10Ottomata: "Arrays of integers are fine just fine :)" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/668244 (owner: 10Sharvaniharan) [23:38:47] (03CR) 10Sharvaniharan: "> Patch Set 15:" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/668244 (owner: 10Sharvaniharan) [23:40:52] (03PS16) 10Sharvaniharan: Image recommendations table for android [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/668244