[00:05:31] (03PS4) 10Nuria: Removing seasonality cycle as it is fixed once granularity is set [analytics/refinery] - 10https://gerrit.wikimedia.org/r/623456 (https://phabricator.wikimedia.org/T257691) [05:56:39] goood morning [06:18:04] Hello team [06:51:01] 10Analytics, 10Analytics-EventLogging, 10CSS: mw-indicators popups hidden under bodyContent content - https://phabricator.wikimedia.org/T262510 (10Yair_rand) [07:11:01] 10Analytics-Clusters, 10Discovery, 10Discovery-Search (Current work), 10Patch-For-Review: mjolnir-kafka-msearch-daemon dropping produced messages after move to search-loader[12]001 - https://phabricator.wikimedia.org/T260305 (10elukey) [07:35:34] (03CR) 10Joal: [C: 03+1] "One comment for discussion - Looks good." (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/623456 (https://phabricator.wikimedia.org/T257691) (owner: 10Nuria) [07:46:26] elukey: I guess we have not booked for apache-con online right? [07:46:32] elukey: shall we do it? [07:46:58] joal: yes I think we can do it anytime, but there shouldn't be any limit [07:47:19] I'd rather do it early and be sure to be able to attend :) [07:47:26] I'll ask nuria later today [07:47:39] coffee elukey? [07:48:08] joal: in 15? [07:48:17] sure elukey! ping when you wish [08:12:19] o/ I still didnt manage to make this python3 jupyter notebook run and connect to spark after moving from the notebook hosts to stat1007, any chance someone might be able to aid me? :) [08:13:38] *pyspark [08:14:32] Hi addshore - I'm no python expert but I can try to help [08:14:50] batcave? :D [08:15:05] * addshore will need a link [08:15:28] addshore: ehm [08:15:28] Sep 10 08:09:20 stat1007 bash[57042]: Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) [08:15:33] yessir - meet.google.com/rxb-bjxn-nip [08:15:34] oh dammit [08:15:46] I guess you know that one :) [08:16:14] addshore: before starting the kernel you need to kinit into a jupyter terminal [08:16:34] before starting the kernel? (that might actually have been my issue all along) [08:16:39] yep! [08:16:48] no batcave needed :) [08:16:53] *facepalm* [08:17:28] addshore: not easy to guess though [08:17:47] so I still have problems :P [08:18:09] ?? [08:18:43] other problems or notebook not starting again? [08:20:50] I see [08:20:50] Sep 10 08:17:15 stat1007 bash[57042]: ==== local-m2-cache: tried [08:20:53] Sep 10 08:17:15 stat1007 bash[57042]: file:/home/addshore/.m2/repository/org/slf4j/slf4j-api/1.7.7/slf4j-api-1.7.7.jar [08:21:01] and then [08:21:02] Sep 10 08:17:15 stat1007 bash[57042]: Exception in thread "main" java.lang.RuntimeException: [download failed: org.slf4j#slf4j-api;1.7.7!slf4j-api.jar] [08:24:41] elukey: this is weird :( --^ [08:25:10] I'm seeing Exception: Java gateway process exited before sending its port number [08:25:10] in the notebook [08:25:57] elukey: can addshore look at his logs or is it needed to be you? [08:29:45] nope he cannot, we'd need to dump those to a file probably, but I need to check how it is done [08:32:18] addshore: you can check /home/addshore/notebook.log [08:32:35] nce [08:32:45] *yes [08:32:47] addshore: the init worked for me on stat1008 [08:33:00] joal, interesting, perhaps it is a stat1007 thing then? [08:33:34] could be addshore - but I wou;d have expected the other way around: stat1008 has buster, and this can lead to some issues sometimes [08:34:18] addshore: can ou try restarting your kernel? [08:39:11] Yes, let me try that now [08:40:34] same error [08:41:14] :( [08:42:28] addshore: it looks like your kernel is trying to download stuff, but I have no idea why :( [08:42:28] https://github.com/jupyter/notebook/issues/743#issuecomment-165087698 ? [08:43:32] addshore: please try! [08:43:55] nope :( [08:44:09] is it possible that the .m2 cache/settings are messed up? [08:44:25] "One possible reason is JAVA_HOME is not set because java is not installed." [08:46:52] addshore: can you try stopping the kernel, deleting the venv, .m2 in your home dir and restart the kernel? [08:47:08] yes! [08:47:42] \o/! The HAMMER :) [08:49:39] 10Analytics-Clusters, 10Discovery, 10Discovery-Search (Current work), 10Patch-For-Review: mjolnir-kafka-msearch-daemon dropping produced messages after move to search-loader[12]001 - https://phabricator.wikimedia.org/T260305 (10elukey) I noticed that kafka-python==1.4.3 in the venv, can we try to upgrade i... [08:52:18] /bin/sh: 1: pip: not found [08:53:50] addshore: can you retry now? [08:53:58] the venv was not there, I restarted jupyterhub [08:54:27] https://usercontent.irccloud-cdn.com/file/scxfBHZ3/image.png [08:55:14] * addshore is free from meetings now and could batcave again [08:55:35] in da cave addshore [08:57:30] https://usercontent.irccloud-cdn.com/file/GTAzscq5/image.png [09:04:33] Okay, I got it back to "Exception: Java gateway process exited before sending its port number" now [09:10:11] https://www.irccloud.com/pastebin/lpQl7vmU/ [09:11:05] is there a way with mvn to pull down slf4j-api-1.7.7.jar ? [09:11:38] got it! [09:11:44] I also deleted `/home/addshore/.ivy2` [09:11:49] then it seems to be working [09:12:10] a nice! I see no more errors [09:12:14] so what did you do? [09:12:22] joal: suggested running the python in the shell by hand, and that gave me the full stacktrace and errors [09:12:26] then I spotted the thrid cache dir [09:12:53] so /home/addshore/.ivy2 was the problem? [09:12:55] weird [09:13:00] yeah [09:13:04] anyway, glad that it works :) [09:13:07] ty all :) [09:13:12] :) [09:21:19] 10Analytics, 10Analytics-Kanban: Check that mediawiki-events match mediawiki-history changes over a month - https://phabricator.wikimedia.org/T262261 (10JAllemandou) **CODE (For `simplewiki` only)** * Prepare snapshot data from hudi-saved dataset. ` import org.apache.spark.sql.SaveMode import org.apache.hudi.... [10:10:19] 10Analytics, 10Patch-For-Review: Fix TLS certificate location and expire for Hadoop/Presto/etc.. and add alarms on TLS cert expiry - https://phabricator.wikimedia.org/T253957 (10jbond) >>! In T253957#6351773, @elukey wrote: > I recently discovered that we have `base::expose_puppet_certs` in puppet. The class i... [10:13:01] 10Analytics, 10Patch-For-Review: Fix TLS certificate location and expire for Hadoop/Presto/etc.. and add alarms on TLS cert expiry - https://phabricator.wikimedia.org/T253957 (10jbond) Forgot to say role `analytics_test_cluster::coordinator` is currently [[ https://github.com/wikimedia/puppet/blob/production/h... [10:14:54] klausman hi! [10:15:37] klasuman: quick quesiton -- which tensorflow-rocm version should I install for tests? I am getting some errors with the prvious version [10:15:54] klausman: quick quesiton -- which tensorflow-rocm version should I install for tests? I am getting some errors with the prvious version [10:16:42] Hmm. We didn't change the rocm base version from before (3.3). What error do you get> [10:21:44] miriam_: ^^^ [10:27:15] * elukey found new Hue bugs.. sigh [10:27:24] :( [10:28:23] klausman it doesn't seem to be detecting the GPU [10:28:47] 10Analytics, 10Release-Engineering-Team, 10observability, 10serviceops, 10User-jijiki: Should we create a separate 'mwdebug' cluster? - https://phabricator.wikimedia.org/T262202 (10LarsWirzenius) @jijiki I haven't ever looked at the parts of Scap that would be affected by this, but I can't imagine it's t... [10:31:08] elukey: you did minimal tensorflow stuff yesterday, and it seemed to work, right? [10:32:28] miriam_: I presume the same job/command works fine on 1008? [10:33:00] klausman I am checking now [10:33:18] miriam_: you are using tensorflow-rocm==2.2.0 [10:33:32] "With the current version of ROCm, 3.3.0, only tensorflow-rocm 2.1.1 is supported." [10:33:35] :P [10:33:59] elukey: ok, then let me try with 2.1.1 [10:34:18] a-ha! :) [10:34:27] elukey: how could you tell? [10:34:57] (that miriam was using 2.2.0, that is) [10:35:39] klausman: I went in /home/myrris, activated the venv and used pip freeze | grep tensorflow [10:36:13] our dear rocm devs link only one version of the rocm drivers when they build/release tensorflow-rocm [10:36:31] so each time I have to update https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/AMD_GPU#Use_tensorflow [10:36:35] :( [10:37:07] Ah, right [10:41:41] ok now it works!! thanks elukey and klausman! [10:41:54] yessss \o/ [10:41:56] I also tried with 2 concurrent tasks [10:42:00] for image classification [10:42:12] (not training, just classification) [10:42:18] there is no conflict, they work in parallel [10:42:27] wow [10:42:55] I don't see the GPU usage raising on Graphana but both tasks seem to have detected the GPU [10:43:15] Found device 0 with properties: [10:43:15] pciBusID: 0000:06:00.0 name: Vega 10 XT [Radeon PRO WX 9100] ROCm AMD GPU ISA: gfx900 [10:43:38] this is a problem of the exporter I think, it runs every minute and if the tasks are too quick it will not detect them (I think) [10:43:47] Correct [10:44:04] The downside of the smi tool outputting gauges (instant values) rather than counters [10:44:54] and sorry elukey I should have checked the wiki!! [10:44:57] The Linux kernel uses millijoules (sic) for power consumption metrics, so it doesn't matter which frequency you read them, you get the right total amount (at whatever time-domain resolution you scrape at, of course) [10:46:07] miriam_: I think that we'll have a better time with the new settings, we have now more up to date drivers (rather then relying on the linux kernel ones, they are too far behind in our kernels) [10:47:44] also, https://rocmdocs.amd.com/en/latest/Current_Release_Notes/Current-Release-Notes.html shows 3.7 [10:49:50] klausman: if you want to also think about upgrading from 3.3 to 3.7 later on we can do it :) [10:51:16] elukey, oki, thanks!! Ia m now checking whether some keras funcionalities work now [10:53:27] oooh " 'Keras requires TensorFlow 2.2 or higher. '" [10:54:11] but it should be available with the new drivers I think [10:54:25] (I mean tf 2.2 support) [10:55:08] Oki, anyway for now I can use the normal tensorflow (not rocm) without GPU, I only need it for classification (not training) -- thanks elukey! [10:55:45] joal: https://issues.apache.org/jira/browse/BIGTOP-3403 [10:56:22] \o/ elukey [10:56:45] elukey: let me know if you want to me to do some tests/devs to help upstream [10:57:02] not even sure what is their state, we'll see [10:57:08] elukey: let's see if the new dkms driver works well first, then contemplate 3.7 :) [10:57:17] elukey,klausman, I tried another couple of classifiers (not using keras) and they seem to work. Thanks a lot, I'll update the task!! [10:57:22] klausman: yes yes you are right :) [10:57:36] miriam_: You're welcome :) [11:04:02] 10Analytics-Clusters, 10Patch-For-Review: AMD ROCm kernel drivers on stat1005/stat1008 don't support some features - https://phabricator.wikimedia.org/T260442 (10Miriam) I just did a few tests on stat1005 (image classification using 2 different classifiers) with the new settings. The GPU is detected and paral... [11:13:17] * elukey lunch1 [11:18:19] 10Analytics-Clusters, 10Patch-For-Review: AMD ROCm kernel drivers on stat1005/stat1008 don't support some features - https://phabricator.wikimedia.org/T260442 (10klausman) elukey and I discussed a bit how will proceed from here. Open things: - Update performance stuff so we get more insight in Grafana (T26242... [11:18:37] 10Analytics-Clusters: Add more metrics to prometheus-amd-rocm-stats Python script - https://phabricator.wikimedia.org/T262427 (10klausman) https://docs.google.com/document/d/1OX7JsearaFNO5b8cT4ScLrUppdvFJmz8gWE1nG6_smc/edit# has an overview of parameters and their outputs. Will discuss with elukey which options... [11:22:16] * klausman lunch as well [11:57:27] 10Analytics, 10Analytics-Kanban: Check that mediawiki-events match mediawiki-history changes over a month - https://phabricator.wikimedia.org/T262261 (10JAllemandou) Checks for discrenpencies (using previous comment setup): * page-delete ` // Try to match events using timestamps val jpdts = hfsnr.where("hudi_... [12:54:16] while working on hue I found https://docs.gethue.com/administrator/configuration/connectors/ [12:54:50] https://docs.gethue.com/administrator/configuration/connectors/#presto is nice [12:55:09] oh ho that is cool [12:55:16] it could be an alternative to the Superset SQLLab [12:55:17] maybe nicer than superset's sql lab? [12:55:17] yeah [12:55:28] i mean, itd' better to provide fewer duplicate tools [12:55:37] but hue does seem slightly nicer (less complicated?) in some ways [12:55:53] there's druid too! [12:56:49] also they have "notebooks" [12:57:34] but they seem very limited compared to the jupyter ones [12:57:56] we'll see [12:58:08] I am still fixing py3 issues, sigh [13:06:27] PHP is the worst [13:06:27] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/EventStreamConfig/+/626372/1/README.md [13:08:26] [obligatory link to the Wat? talk] [13:09:02] I do hope that is known material here? [13:10:01] i'm sure it is known by some. it may have been known by me 11 years ago when I was a regular PHP Wat-er [13:10:58] https://www.destroyallsoftware.com/talks/wat For those who don't. It's about JS and Ruby, but PHP is a fountain of similar surprises. [13:13:39] always great to watch [13:13:55] He has very, very good presentation/stage skills [13:14:06] hha i just checked the {} + [] vs [] + {} in my nodejs vefsion and it looks liek they at least fixed that and made it consisten t [13:14:29] {}+{} is my fave, tho [13:15:00] oh haha [13:15:00] wow [13:17:40] He also has a longer, but just as funny talk on the Birth and Death of JS: https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript [13:55:50] 10Analytics: pagecounts-ez of month 2020-08 is incomplete - https://phabricator.wikimedia.org/T262141 (10Danilo) I will make my script use the bz2 file when the uncompressed file is not complete. But maybe it is also a good idea someone decompress the complete bz2 file that is in the same folder to replace the... [14:05:56] 10Analytics-Radar, 10Operations, 10Patch-For-Review: Move Hue to a Buster VM - https://phabricator.wikimedia.org/T258768 (10elukey) My pull requests were merged, but I opened https://github.com/cloudera/hue/issues/1262 too. [14:06:26] o/ milimetric yt? [14:07:03] elukey: https://gerrit.wikimedia.org/r/c/operations/puppet/+/626386 when you have some time. [14:12:35] done :) [14:21:05] ottomata: oh sorry yea [14:21:08] why do I not get pings... [14:21:34] milimetric: am thinking about the event plaform tech talk I have to give and just wanted someone to brain bounce topics and outline :) [14:21:50] to the batcave! [14:21:54] ok! [14:27:34] (03CR) 10Nuria: Removing seasonality cycle as it is fixed once granularity is set (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/623456 (https://phabricator.wikimedia.org/T257691) (owner: 10Nuria) [14:31:34] (03CR) 10Mforns: [C: 03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/623456 (https://phabricator.wikimedia.org/T257691) (owner: 10Nuria) [14:33:34] 10Analytics, 10Event-Platform: Duplicated revision_create events - https://phabricator.wikimedia.org/T262203 (10Nuria) * i think* this duplication on events might mean that we are expiring cache more frequently than we should for those revisions, not sure but seems likely. This is an issue of data quality in t... [14:35:19] 10Analytics, 10Event-Platform: Need for new event-type - `user_create` and `user_rename` - https://phabricator.wikimedia.org/T262205 (10Nuria) @WDoranWMF please let us know if sending these events is something that core platform can do. [14:36:29] 10Analytics, 10Analytics-Kanban: Configure Oozie job for loading geoeditors data into Cassandra - https://phabricator.wikimedia.org/T248289 (10Nuria) 05Open→03Resolved [14:40:06] 10Analytics-Radar, 10Operations, 10Traffic, 10Patch-For-Review: Package varnish 6.0.x - https://phabricator.wikimedia.org/T261632 (10Vgutierrez) [14:41:22] (03CR) 10Nuria: [V: 03+2] Removing seasonality cycle as it is fixed once granularity is set [analytics/refinery] - 10https://gerrit.wikimedia.org/r/623456 (https://phabricator.wikimedia.org/T257691) (owner: 10Nuria) [14:56:44] 10Analytics-Clusters: install mwparserfromhell on spark for efficient usage of wikitext-dump in hive - https://phabricator.wikimedia.org/T262044 (10MGerlach) @Ottomata that would be great. do you have an indication when you are going to try this out. I am trying to anticipate whether I should wait or try workaro... [14:59:59] ping razzi mforns [15:13:42] 10Analytics, 10Event-Platform: Duplicated revision_create events - https://phabricator.wikimedia.org/T262203 (10Nuria) ping @hnowlan [15:21:36] 10Analytics, 10Event-Platform, 10Platform Engineering: Duplicated revision_create events - https://phabricator.wikimedia.org/T262203 (10WDoranWMF) [15:27:26] 10Analytics, 10Event-Platform: Need for new event-type - `user_create` and `user_rename` - https://phabricator.wikimedia.org/T262205 (10WDoranWMF) @Nuria We can check into it, do you have a timeline you need an answer by? [15:36:07] 10Analytics, 10Analytics-EventLogging, 10CSS: mw-indicators popups hidden under bodyContent content - https://phabricator.wikimedia.org/T262510 (10Milimetric) p:05Triage→03Low Low priority for us, people can feel free to raise priority or grab it (should be relatively easy) [15:36:54] 10Analytics, 10Event-Platform, 10Platform Engineering: Need for new event-type - `user_create` and `user_rename` - https://phabricator.wikimedia.org/T262205 (10Nuria) [15:37:25] 10Analytics-Radar, 10Event-Platform, 10Patch-For-Review, 10Platform Team Workboards (Clinic Duty Team), 10Wikimedia-production-error: PHP Notice: Array to string conversion - https://phabricator.wikimedia.org/T262462 (10Milimetric) [15:39:54] 10Analytics, 10Analytics-EventLogging, 10Event-Platform: eventlogging-processor should fail to produce schemas that have been migrated to Event Platform - https://phabricator.wikimedia.org/T262304 (10Milimetric) p:05Triage→03High [15:42:12] 10Analytics, 10Analytics-Kanban: Check that mediawiki-events match mediawiki-history changes over a month - https://phabricator.wikimedia.org/T262261 (10Milimetric) p:05Triage→03High [15:44:10] 10Analytics: Make hudi work with Hive - https://phabricator.wikimedia.org/T262260 (10Milimetric) p:05Triage→03High [15:44:47] 10Analytics, 10Analytics-Kanban: Test hudi as an incremental update system using 2 mediawiki-history snapshots - https://phabricator.wikimedia.org/T262256 (10Milimetric) p:05Triage→03High [15:44:51] 10Analytics, 10Event-Platform, 10Platform Engineering: Need for new event-type - `user_create` and `user_rename` - https://phabricator.wikimedia.org/T262205 (10Nuria) Within a quarter will be wonderful [15:49:13] 10Analytics-Radar, 10Event-Platform, 10Platform Engineering: Duplicated revision_create events - https://phabricator.wikimedia.org/T262203 (10Milimetric) [15:49:49] 10Analytics, 10Event-Platform, 10Platform Engineering: Need for new event-type - `user_create` and `user_rename` - https://phabricator.wikimedia.org/T262205 (10WDoranWMF) No worries, I'll make sure we review in our weekly tech planning this week or next. Thanks! [15:50:56] 10Analytics-Radar, 10Release-Engineering-Team, 10observability, 10serviceops, 10User-jijiki: Should we create a separate 'mwdebug' cluster? - https://phabricator.wikimedia.org/T262202 (10Milimetric) x-wikimedia-debug shouldn't affect anything we do [15:57:21] FYI, upgrading analytics stretch hosts to git 2.20 for https://phabricator.wikimedia.org/T262244 [15:57:49] 10Analytics-Radar, 10Operations, 10Traffic, 10Patch-For-Review: Package varnish 6.0.x - https://phabricator.wikimedia.org/T261632 (10Vgutierrez) @ema I've added to the task description the CRs required to get the packages of all the vmods and varnishkafka, I've seen that we have varnish-modules compiled on... [16:02:32] moritzm: ack thanks [16:06:11] 10Analytics: Gather all data-purge into a single job - https://phabricator.wikimedia.org/T262201 (10Milimetric) p:05Triage→03High [16:07:27] 10Analytics-Radar, 10Event-Platform, 10Platform Engineering: Duplicated revision_create events - https://phabricator.wikimedia.org/T262203 (10JAllemandou) More info on how frequently it happened: ` month | dup_rev_ids | not_dup_rev_ids -------+-------------+----------------- 1 | 0 |... [16:09:48] 10Analytics-Radar, 10Event-Platform, 10Platform Engineering: Duplicated revision_create events - https://phabricator.wikimedia.org/T262203 (10Nuria) so 0.006% of revision create events * might be* unnecessary purged [16:10:18] 10Analytics-Clusters, 10Analytics-Kanban: Create a cookbook to automate the bootstrap of new Hadoop workers - https://phabricator.wikimedia.org/T262189 (10Milimetric) p:05Triage→03High [16:11:22] 10Analytics-Radar, 10Event-Platform, 10Platform Engineering: Duplicated revision_create events - https://phabricator.wikimedia.org/T262203 (10JAllemandou) >>! In T262203#6451084, @Nuria wrote: > so 0.006% of revision create events * might be* unnecessary purged 1% in July - Can be interesting to understand... [16:12:01] 10Analytics: install mwparserfromhell on spark for efficient usage of wikitext-dump in hive - https://phabricator.wikimedia.org/T262044 (10Ottomata) [16:13:13] 10Analytics-Radar, 10Event-Platform, 10Platform Engineering: Duplicated revision_create events - https://phabricator.wikimedia.org/T262203 (10Pchelolo) In this case it's pretty clear the duplicates are a bug in MW code since request_id/id/dt are different and those AFAIK are set in MW code - this needs fixin... [16:14:29] 10Analytics, 10Analytics-Kanban: Sort editors-by-country by descending editor-ceil value in cassandra - https://phabricator.wikimedia.org/T262184 (10Milimetric) p:05Triage→03High [16:16:20] 10Analytics-Radar, 10Event-Platform, 10Platform Engineering: Duplicated revision_create events - https://phabricator.wikimedia.org/T262203 (10JAllemandou) > duplicates should be tolerated. They are :) The event-refine process de-duplicates events by meta.id. [16:17:54] 10Analytics-Radar, 10Release-Engineering-Team, 10observability, 10serviceops, 10User-jijiki: Should we create a separate 'mwdebug' cluster? - https://phabricator.wikimedia.org/T262202 (10jijiki) @Milimetric my question is, if I want to do a performance test and fire 700k requests towards our debug server... [16:19:51] 10Analytics-Radar, 10Event-Platform, 10Platform Engineering: Duplicated revision_create events - https://phabricator.wikimedia.org/T262203 (10Nuria) >However, Kafka as we use it has 'at least once' semantics, so analytics systems should not depend on any event being globally unique, duplicates should be tole... [16:21:21] 10Analytics-Radar, 10Event-Platform, 10Platform Engineering: Duplicated revision_create events - https://phabricator.wikimedia.org/T262203 (10Pchelolo) ` kafkacat -b localhost:9092 -p 0 -t 'codfw.mediawiki.revision-create' -o beginning | grep 977066159 | jq . ` {P12564} [16:22:35] 10Analytics-Clusters: Apply proper permissions to stat100x home directories - https://phabricator.wikimedia.org/T262183 (10Milimetric) p:05Triage→03High potential easier way: require belonging to analytics-privatedata to log into stat1xxx. [16:23:29] 10Analytics-Clusters: Upgrade to Superset 0.37.x - https://phabricator.wikimedia.org/T262162 (10Milimetric) p:05Triage→03Medium [16:24:34] 10Analytics-Radar, 10Event-Platform, 10Platform Engineering: Duplicated revision_create events - https://phabricator.wikimedia.org/T262203 (10JAllemandou) Wow that looks like you found it @Pchelolo! I wish I cou;d find bugs as fast as you do :) [16:26:03] 10Analytics-Radar, 10Event-Platform, 10Platform Engineering: Duplicated revision_create events - https://phabricator.wikimedia.org/T262203 (10Pchelolo) I found the example events. The reason why they are duplicated is still a mystery... [16:51:16] mgerlach: yt? [16:51:31] o/ [16:51:59] i want to test my prebuilt mwparserfromhell on stretch [16:52:26] oh was goign to ask you for a test...but i think the mwparserfromhell github readme has examples [16:52:47] yes, happy to test. [16:53:50] oh huh i think it works [16:53:52] ok great [16:53:58] luca said I could install everywhere, doing now :) [16:55:36] "luca said" implies some authority that I don't really have :D [16:57:29] * elukey afk! [16:57:30] o/ [16:57:36] elukey: FYI, i looked into building a stretch package and with my build process its hard! [16:57:41] since I do it all in prep steps! [16:57:57] and to use pbuilder dist env it needs to be done as part of e.g. debian/rules [16:58:00] which i don't use :/ [16:58:05] maybe I should, but i need access to the internet to build [16:58:11] which can be done...BUUUT NOT GOING TO DO IT NOW [16:58:25] i just tested pandas and pyarrow and mwparserfromhell [16:58:26] on stretch [16:58:32] fingers crossed the buster stull will all just work [18:02:17] ok mgerlach [18:02:27] do you have an idea of how to test with spark in yarn? [18:02:37] no [18:04:39] milimetric: finished first version of editors by country docs: https://wikitech.wikimedia.org/wiki/Analytics/AQS/Editors_by_country [18:04:50] please, feel free to change/add/remove! [18:05:20] and give feedback so I can improve! [18:05:33] k, I’ll look [18:07:39] well ok mgerlach i just installed anaconda-wmf everywhere :) [18:08:05] so, for use with pyspark, you'll just have to make sure pyspark is launched with the correct python [18:08:08] ottomata: cool. how do I use it with spark? [18:08:08] on the CLI [18:08:10] if you [18:08:14] source the base anaconda env [18:08:21] https://wikitech.wikimedia.org/wiki/Analytics/Systems/Anaconda#Anaconda_base_environment [18:08:25] and launch pyspark2 [18:08:27] it should just work [18:08:29] in Jupyter [18:08:37] you'll have to use the newpyter stuff on stat1008 [18:08:44] and launch your notebook server using the base anaconda-wmf enb [18:08:45] env [18:08:46] OR [18:08:59] create a custom spark kernel to launch using python out of that env [18:09:21] (03CR) 10Mforns: "@Paul Kernfeld, improving the code by removing 49 lines of code is amazing! Thanks." [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/623470 (https://phabricator.wikimedia.org/T173604) (owner: 10Paul Kernfeld) [18:09:35] looks great mforns, good note on non-additive [18:15:21] 10Analytics: Debianize Python's pid library to be able to use it from reportupdater - https://phabricator.wikimedia.org/T262574 (10mforns) [18:16:57] 10Analytics: Debianize Python's pid library to be able to use it from reportupdater - https://phabricator.wikimedia.org/T262574 (10mforns) [18:17:13] 10Analytics: Debianize Python's pid library to be able to use it from reportupdater - https://phabricator.wikimedia.org/T262574 (10mforns) [18:17:18] 10Analytics, 10Patch-For-Review, 10good first task: Reportupdater: do not write execution control files in source directories - https://phabricator.wikimedia.org/T173604 (10mforns) [18:19:17] 10Analytics: Debianize Python's pid library to be able to use it from reportupdater - https://phabricator.wikimedia.org/T262574 (10Ottomata) Hm, Perhaps @razzi can take this one? We have to get him permissions on the build server. It looks like we might need a new group for that, in lieu of making in an ops admin. [18:19:50] 10Analytics, 10Event-Platform, 10Technical-blog-posts: Story idea for Blog: Wikimedia's Event Platform - https://phabricator.wikimedia.org/T253649 (10srodlund) @Ottomata I have published this on the blog. Before I announce it, can you check and see if the title looks okay? I changed it slightly because the... [18:23:48] (03PS1) 10DannyS712: Review access change [analytics/quarry/web] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/626430 [18:24:19] (03PS2) 10DannyS712: Inherit from mediawiki instead of mediawiki/core [analytics/quarry/web] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/626430 (https://phabricator.wikimedia.org/T201435) [18:28:19] 10Analytics, 10Event-Platform, 10Technical-blog-posts: Story idea for Blog: Wikimedia's Event Platform - https://phabricator.wikimedia.org/T253649 (10Ottomata) @srodlund looks good to me thank you!!! [18:31:33] 10Quarry, 10Gerrit-Privilege-Requests, 10Patch-For-Review: Quarry repo access should be cleaned up - https://phabricator.wikimedia.org/T201435 (10DannyS712) Given the lack of responses objecting to the changes suggested, moving to ready for admin [18:34:31] 10Analytics, 10Event-Platform, 10Technical-blog-posts: Story idea for Blog: Wikimedia's Event Platform - https://phabricator.wikimedia.org/T253649 (10srodlund) Awesome! [18:36:55] (03CR) 10Thcipriani: [V: 03+2 C: 03+2] Inherit from mediawiki instead of mediawiki/core [analytics/quarry/web] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/626430 (https://phabricator.wikimedia.org/T201435) (owner: 10DannyS712) [18:38:31] (03CR) 10DannyS712: "This change is ready for review." [analytics/quarry/web] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/626431 (https://phabricator.wikimedia.org/T201435) (owner: 10DannyS712) [18:38:56] 10Quarry, 10Gerrit-Privilege-Requests, 10Patch-For-Review: Quarry repo access should be cleaned up - https://phabricator.wikimedia.org/T201435 (10thcipriani) [18:43:47] ottomata: thanks. got it. I will try out tomorrow and let you know how it goes [18:46:49] (03CR) 10Thcipriani: [V: 03+2 C: 03+2] "Most of the pywikibot folks seem to also be members of the quarry-devs group, so this mostly shouldn't affect folks ability to code review" [analytics/quarry/web] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/626431 (https://phabricator.wikimedia.org/T201435) (owner: 10DannyS712) [18:47:47] 10Quarry, 10Gerrit-Privilege-Requests, 10Patch-For-Review: Quarry repo access should be cleaned up - https://phabricator.wikimedia.org/T201435 (10DannyS712) [18:48:08] 10Quarry, 10Gerrit-Privilege-Requests: Quarry repo access should be cleaned up - https://phabricator.wikimedia.org/T201435 (10DannyS712) 05Open→03Resolved [18:48:21] 10Quarry, 10Gerrit-Privilege-Requests: Quarry repo access should be cleaned up - https://phabricator.wikimedia.org/T201435 (10thcipriani) > If all those people need merge access, they should be added to the owner group or to a bespoke participant group. The folks who seem to do code review in this repo are al... [18:51:34] gone for tonight team - see ou tomorrow [19:05:05] 10Analytics-Radar, 10Event-Platform, 10Platform Engineering: Duplicated revision_create events - https://phabricator.wikimedia.org/T262203 (10Pchelolo) @Ottomata I've dug into this quite a lot walking up and up the call stack, and couldn't find where could this be duplicated in MW. Honestly, the most reason... [19:35:13] ottomata: congrats on publication of https://techblog.wikimedia.org/2020/09/10/wikimedias-event-data-platform-or-json-is-ok-too/ !!! It's very good and I'm looking forward to sharing parts 2 and 3 too [19:40:04] 10Analytics-Radar, 10Release-Engineering-Team, 10observability, 10serviceops, 10User-jijiki: Should we create a separate 'mwdebug' cluster? - https://phabricator.wikimedia.org/T262202 (10Milimetric) >>! In T262202#6451136, @jijiki wrote: > @Milimetric my question is, if I want to do a performance test fr... [19:56:14] bearloga: thank you for your help with it! :) [19:56:55] :D [23:16:16] 10Analytics, 10Event-Platform: Remove http.client_ip from EventGate default schema (again) - https://phabricator.wikimedia.org/T262626 (10Krinkle) [23:18:01] 10Analytics, 10Event-Platform: Remove http.client_ip from EventGate default schema (again) - https://phabricator.wikimedia.org/T262626 (10Krinkle) [23:18:40] 10Analytics, 10Event-Platform: Remove http.client_ip from EventGate default schema (again) - https://phabricator.wikimedia.org/T262626 (10Krinkle) [23:19:03] 10Analytics, 10Event-Platform: Remove http.client_ip from EventGate default schema (again) - https://phabricator.wikimedia.org/T262626 (10Krinkle) [23:19:20] 10Analytics, 10Event-Platform: Remove http.client_ip from EventGate default schema (again) - https://phabricator.wikimedia.org/T262626 (10Krinkle) [23:19:35] 10Analytics, 10Event-Platform, 10Privacy Engineering: Remove http.client_ip from EventGate default schema (again) - https://phabricator.wikimedia.org/T262626 (10Krinkle)