[00:17:53] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4212728 (10Denny) @Rspeer If I link an article from the German Wikipedia to the English Wikipedia by adding an interwiki link on the German Wikipedia, and then an interwiki bot makes t... [05:34:44] 10Analytics, 10Research: Upload XML dumps to hdfs - https://phabricator.wikimedia.org/T186559#4212831 (10chelsyx) [06:03:44] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4212845 (10Rspeer) This reads like the transcript of a "sovereign citizen" arguing why they don't have to pay taxes because the flag in the courtroom doesn't have some feature they insi... [06:24:58] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4212862 (10Rspeer) My previous comment probably crossed a line. I'm sorry. But your convoluted argument has shown nothing and is irrelevant to Wikidata. Wikipedia is fine with a very... [06:38:23] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4212864 (10EgonWillighagen) >>! In T193728#4212862, @Rspeer wrote: > how to change Wikidata's copyright status. In which you assume it will chance license(/waiver)... If you seek **cer... [06:55:51] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4212870 (10Micru) I have a question regarding Wikipedia(s)->Wikidata imports. Since the license in the Wikipedia(s) is managed by the community, and the community has the power to chang... [07:22:22] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4212876 (10Nemo_bis) > Since the license in the Wikipedia(s) is managed by the community, Not really, the license is one of the non-negotiable aspects of Wikimedia projects. > and th... [07:36:15] (03CR) 10Nuria: "I do not get why the uas for test need fixing..." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/433633 (https://phabricator.wikimedia.org/T192463) (owner: 10Fdans) [07:37:04] (03CR) 10Nuria: "Ah, sorry , i see, you corrected the windows UAS." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/433633 (https://phabricator.wikimedia.org/T192463) (owner: 10Fdans) [07:47:38] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4212896 (10Cirdan) >>! In T193728#4212870, @Micru wrote: > would it be feasible to ask the several Wikipedia(s) communities to add a clause where it is stated that statements can be min... [07:57:24] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4212899 (10Cirdan) >>! In T193728#4212728, @Denny wrote: > And thus, since we never required to have the interwiki links attributed in the first place - as I just showed - we obviously... [07:59:00] (03PS2) 10Fdans: Update refinery user agent parser version [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/433633 (https://phabricator.wikimedia.org/T192463) [08:02:45] (03CR) 10jerkins-bot: [V: 04-1] Update refinery user agent parser version [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/433633 (https://phabricator.wikimedia.org/T192463) (owner: 10Fdans) [08:58:11] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4212948 (10Micru) > Not really, the license is one of the non-negotiable aspects of Wikimedia projects. With enough support, everything is negotiable. > Hardly anyone has! When we swi... [09:21:53] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Update version of ua-parser in refinery source - https://phabricator.wikimedia.org/T192463#4213010 (10Nuria) This looks fine to merge, let's make sure we coordinate deployments of java and python version [09:27:42] (03CR) 10Nuria: [C: 04-1] "Tests fail for me on this version:" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/433633 (https://phabricator.wikimedia.org/T192463) (owner: 10Fdans) [09:28:33] nuria_: yeah i’m correcting the refinery-hive tests [09:28:41] fdans: k [09:28:56] nuria_: didn’t realise both refinery core and give had checks on results [09:33:10] and hive* sorry [09:40:18] what is the kafka topic for mediawiki api actions? :O [10:49:32] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4213195 (10Cirdan) >>! In T193728#4212948, @Micru wrote: >> Not really, the license is one of the non-negotiable aspects of Wikimedia projects. > > With enough support, everything is n... [11:11:18] * fdans is cheating on Sublime Text with Visual Studio Code [11:12:13] (03PS3) 10Fdans: Update refinery user agent parser version [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/433633 (https://phabricator.wikimedia.org/T192463) [12:49:35] o/ [12:54:04] (03CR) 10Ottomata: [C: 032] "Ooook!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/433569 (https://phabricator.wikimedia.org/T193176) (owner: 10Mforns) [12:54:40] \o ottomata - Do we go for the PartitionedDataFrame patch? [13:00:50] joal: sure! [13:10:09] joal rebase conflict in gerrit :/ [13:10:54] joal: ottomata this is ready to be merged! [13:10:54] https://gerrit.wikimedia.org/r/#/c/433633/ [13:12:26] ok! [13:12:50] (03CR) 10Ottomata: [C: 032] Update refinery user agent parser version [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/433633 (https://phabricator.wikimedia.org/T192463) (owner: 10Fdans) [13:14:52] (03PS3) 10Joal: Add PartitionedDataFrame to spark Refine job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/427187 [13:16:52] thank youuuu ottomata [13:44:15] fdans: pull request done for ua-parser [13:44:39] nuria_: we should all go in and pressure them [13:45:06] fdans: you start [13:45:26] fdans: i will follow.. ahem ... close by [13:45:28] :/( [13:45:46] :'( * [13:47:33] addshore: actions such us add arevision and such? [13:48:13] addshore: like: https://grafana.wikimedia.org/dashboard/db/kafka-by-topic?from=1526564704082&to=1526651104082&refresh=5m&orgId=1&var-datasource=eqiad%20prometheus%2Fops&var-kafka_cluster=jumbo-eqiad&var-kafka_broker=All&var-topic=codfw.mediawiki.revision-create? [13:49:41] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4213766 (10Micru) > Aside from the fact that every single contributor would have to be asked to agree to the change of the license Not necessarily, a broad discussion with a majority a... [13:55:51] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4213776 (10Cirdan) >>! In T193728#4213766, @Micru wrote: >> Aside from the fact that every single contributor would have to be asked to agree to the change of the license > > Not neces... [14:08:58] (03CR) 10Milimetric: "Luca, just some style suggestions, I believe this code would work as-is but didn't have time to test." (034 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/433597 (https://phabricator.wikimedia.org/T194055) (owner: 10Elukey) [14:11:02] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4213806 (10Micru) > since I hold the rights to that text The concept of "rights" is quite flexible, as shows Wikipedia. The Wikipedias are based on texts that have copyrights but they... [14:48:23] nuria_: nope, use of the mediawiki action api [14:49:07] either https://grafana.wikimedia.org/dashboard/db/kafka-by-topic?from=now-30d&to=now&refresh=5m&orgId=1&var-datasource=eqiad%20prometheus%2Fops&var-kafka_cluster=jumbo-eqiad&var-kafka_broker=All&var-topic=apiaction [14:49:08] or https://grafana.wikimedia.org/dashboard/db/kafka-by-topic?from=now-30d&to=now&refresh=5m&orgId=1&var-datasource=eqiad%20prometheus%2Fops&var-kafka_cluster=jumbo-eqiad&var-kafka_broker=All&var-topic=mediawiki_ApiAction [14:49:13] but both seem to be empty [14:49:21] so maybe it doesnt go through kafka any more? [14:50:17] addshore: wrong cluster [14:50:23] oooh [14:50:26] we haven't been able to port over the mediawiki client to jumbo [14:50:32] it is still analytics-eqiad [14:50:34] or 'eqiad' [14:50:43] which actually [14:50:46] isn't event in that dashbaord [14:50:49] since it is graphite based [14:51:26] https://grafana-admin.wikimedia.org/dashboard/db/kafka-by-topic-graphite?refresh=5m&orgId=1&var-cluster=analytics-eqiad&var-kafka_brokers=All&var-topic=mediawiki_ApiAction [14:51:53] I see [14:52:21] https://phabricator.wikimedia.org/T188136 [14:54:00] ottomata: I was expecting something like this to work then [14:54:01] addshore@stat1005:~$ kafkacat -b kafka1001.eqiad.wmnet -t mediawiki_ApiAction [14:54:40] but, maybe I don't know what I am doing, or misusing this, or being silly [14:55:10] wrong broker [14:55:23] kafka1012.eqiad.wmnet:9092 [14:55:24] will work [14:55:28] but, that is going to get you a LOT of messages [14:55:58] ottomata: indeed, kafkacat has grep things though! [14:56:09] aaaah, wrong broker... I should have realized that sooner [14:56:17] you are also going to get avro [14:56:17] not json [14:56:36] So I still get " ERROR: Failed to query metadata for topic mediawiki_ApiAction: Local: Broker transport failure" [14:56:38] i think discovery folks have some example usage [14:56:58] addshore: -C [14:57:01] i think? [14:57:03] checking [14:57:29] hm no me too [14:57:32] so it said "% ERROR: Failed to query metadata for topic mediawiki_ApiA" [14:57:40] *"% Auto-selecting Consumer mode (use -P or -C to override)" [14:57:52] huh yeah me too [14:58:59] HM [14:59:00] oh [14:59:05] addshore: we updated librdkafka/kafkacat... [14:59:10] i think it might not work with the old version of kafka...... [14:59:12] addshore: qq [14:59:16] ooooooh [14:59:20] why do you want to consume this from kafka? [14:59:27] it is in hive table right? [14:59:29] do you need realtime? [14:59:49] and "regular" kafka servers are still running old kafka? [15:00:37] I was actually only intending on using it to grep for a request that I make, but then found out it was not working, so started down the rabbit hole of trying to figure out why [15:01:01] And the answer to that may now be version difference for kafkacat [15:01:05] so I'll just go query hive :) [15:01:07] addshore to get it to work [15:01:11] -X api.version.request=false -X broker.version.fallback=0.9.0.1 [15:01:14] add that to your kafkacat command [15:01:17] but [15:01:20] you are going to get binary data [15:01:23] so you can't just grep it [15:01:39] i'm looking for discovery's page about how to consume this data from kafka [15:01:43] you need the avro schema, etc. [15:01:49] aah indeed that does work!, okay good to know [15:01:58] ah! [15:01:59] https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/MediaWiki_Avro_Logging#Test_reading_an_Avro_record_from_a_Kafka_topic [15:02:58] and the avro for the action api topic is somewhere [15:03:00] *searches* [15:03:56] add on stat1005 it is cloned [15:04:02] /srv/event-schemas/avro/mediawiki/ApiAction/101453221640.avsc [15:04:54] awesome! [15:06:24] merged joal :) [15:06:30] Thanks a lot ottomata :) [15:06:55] (03CR) 10Ottomata: [C: 032] Add PartitionedDataFrame to spark Refine job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/427187 (owner: 10Joal) [15:31:37] 10Analytics, 10Android-app-Bugs, 10Wikipedia-Android-App-Backlog: Count link previews on the Android app - https://phabricator.wikimedia.org/T194961#4214065 (10Tbayer) [16:43:29] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4214437 (10MisterSynergy) >>! In T193728#4214033, @Denny wrote: > My current goal to shepherd this bug to a closure is to agree with people who have a different point of view on a quest... [16:47:13] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4214467 (10Rspeer) Once again, it's silly to talk about //this// issue going to court. Wikimedia contributors are not taking other Wikimedia contributors to court over internal disagree... [17:15:21] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4214578 (10Denny) @MisterSynergy yes, I agree, it would seriously weaken Wikidata. Nevertheless it is good to resolve legal uncertainties as far as reasonable. Regarding Gnom1 - well,... [17:23:40] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4214684 (10Denny) @Rspeer My previous suggestion to @Psychoslave was P) "Can you comment on the practise of extracting data from Wikipedia articles, which are published under CC-BY-SA... [18:59:45] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4215439 (10Mateusz_Konieczny) R2 sounds excellent. It covers main legal issue that absolutely needs resolving. Though I am also curious about OSM. My R-OSM-1 would be R-OSM-1: "What,...