[06:34:24] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4178076 (10EgonWillighagen) >>! In T193728#4189219, @Psychoslave wrote: > Let's recall that whether this transfer is done by automation or crowdsourcing doesn't matter, it's the quantit... [06:36:58] morning! [06:40:08] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4203592 (10Psychoslave) Hi @Denny, here are a small set questions that are hopefully simple and concrete: # should we allow transfer into Wikidata of any significant data sets which o... [06:41:03] I am doing a rolling restart of cassandra on aqs* for openjdk-8 upgrades [06:41:34] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4203596 (10Cirdan) >>! In T193728#4203583, @EgonWillighagen wrote: >>>! In T193728#4189219, @Psychoslave wrote: >> Let's recall that whether this transfer is done by automation or crowd... [06:44:50] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4203601 (10EgonWillighagen) Hi all, IANAL but have been professionally dealing with copyright for quite some time now (scholar, author, database creator, advisor, etc, etc). First, aut... [06:52:49] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4203607 (10Rspeer) If you really wish for your data to be under CC0, why would you have any preferences at all over what happens to it? CC0 is the license where your wishes don't matter... [06:59:14] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4203623 (10Psychoslave) >>! In T193728#4203583, @EgonWillighagen wrote: >>>! In T193728#4189219, @Psychoslave wrote: >> Let's recall that whether this transfer is done by automation or... [07:02:16] Morning elukey :) [07:02:28] o/ [07:02:52] elukey: I plan to test druid-kafka now Would that be good for you? [07:05:22] yep! [07:05:34] did you see my latest post in the task? [07:05:45] Nope [07:05:50] I've re-deployed the final package JUST-TO-BE-SURE [07:05:56] :) [07:06:09] and also turned off the "use_cdh" parameter in puppet [07:06:11] final meaning 0.11 without CDH [07:06:17] right [07:06:19] ok [07:06:30] I'm gonna re-test everything, just to be sure [07:06:33] ahhahaha [07:06:36] elukey: is it deployed :) [07:06:52] what do you mena? [07:06:54] *mean? [07:07:11] Oh sorry - didn't fully read- you've DEPLOYED, not updatedg [07:07:15] ok, will test this morning [07:07:45] Also elukey, I'm back to schedule where kids are at school [07:07:54] I'll be off mid-afternoon, then back in evening [07:11:31] ack! [07:12:23] joal: what plan do we want to follow with tranquillity? [07:12:49] elukey: if tests are successfull with kafka-druid, let's drop ranquility from the equation [07:13:53] ah ok so we want to test KIS and then deploy it if needed, okokk [07:14:06] we are not in a hurry, everything is ready for the migration if needed [07:14:28] I also need to swap all the hadoop labs hosts to stretch this week [07:14:48] (now all the workers are on stretch!) [07:15:29] yay - sounds good elukey [07:15:41] elukey: this is super not cool --> http://druid.io/docs/latest/tutorials/tutorial-kafka.html [07:15:59] elukey: the above example uses tranquility ... [07:17:54] elukey: also, I'd need a patch in druid to load the KIS extension - And while doing that, would it be feasible to also to add druid-avro-extensions and druid-parquet-extensions? [07:18:21] sure it should be a problem [07:18:45] hm - missong a NOT over there--^, or no? [07:18:56] yes :D [07:19:03] Ok :) [07:19:08] I am drinking my coffee sorry :P [07:19:22] no problem at all - Drinking mine as well ;) [07:20:30] I am wondering one thing Joseph.. Wouldn't it be cleaner if we separate Druid's 0.11 upgrade (bundled with your version of tranquillity) and KIS/avro/parquet ? [07:21:10] basically upgrading our current settings and then open a new task for the rest? [07:21:30] elukey: I'm ok to move without KIS, but if it's easy to make it work, I'd go for that instead of tranquility patch [07:22:08] elukey: tranquility patch is small, but the changes it involves are not that small (add jars, rebuild with those) [07:22:32] elukey: If KIS works simply, the amount of work might actually be smaller [07:22:41] okok! [07:22:54] elukey: And parquet addition is just for me to test :) [07:24:35] so under /usr/share/druid/extensions on druid1001 I can see druid-avro-extensions but not the parquet one, I think we'd need to add the jar and rebuild the packages [07:24:59] elukey: parquet is a community extension - I don't know what it involves [07:25:15] elukey: if it's easy to try, let's do it - if not, let's not :) [07:25:55] so as test, I think that we could simply wget the extension jar from the druid homepage and place it there [07:26:06] elukey: that'd be great :) [07:26:08] building the new pkg then should be super easy [07:27:46] it should be like the mysql extension that we use: download and place it under a dir, and the rebuild [07:43:17] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Google-Summer-of-Code (2018): GSoC Proposal 2018 : [Analytics] Improvements to Wikistats2 front-end - https://phabricator.wikimedia.org/T189964#4203688 (10sahil505) [07:45:51] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Google-Summer-of-Code (2018): GSoC Proposal 2018 : [Analytics] Improvements to Wikistats2 front-end - https://phabricator.wikimedia.org/T189964#4203699 (10sahil505) @mforns : Updated the timeline & deliverables based on the new tasks added. Have kep... [08:02:04] AQS cassandra restart completed! [08:02:38] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Restart Analytics hosts for Java 8 Security upgrades - https://phabricator.wikimedia.org/T194268#4203762 (10elukey) [08:11:45] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Reimage the Debian Jessie Analytics worker nodes to Stretch. - https://phabricator.wikimedia.org/T192557#4203773 (10elukey) Triple checking disk layout: ``` elukey@analytics1030:~$ df -h Filesystem Size Used Avai... [09:35:01] elukey: I confirm success for reindexation of druid with the latest package [09:35:12] elukey: how do we stand on KIS and other stuff? [09:37:59] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4204019 (10TomT0m) Just to check if there is an actual problem here: let’s imagine a theorical usecase. Alice, or the AliceAndBob group, contributed to Wikipedia and are hurt because s... [09:51:34] joal: here I am sorry! [09:51:38] \o/ [09:51:41] np :) [09:52:21] for KIS and other stuff I was waiting for your feedback, do you want me to try to set it up? [10:01:03] for KIS, I am wondering if this extension http://druid.io/docs/latest/development/extensions-core/kafka-ingestion.html is the right doc to check [10:01:15] rather than the one that we were checking earlier on [10:08:42] ah nice the druid-kafka-indexing-service extension is already in the 0.11 package [10:08:51] not sure why the parquet one is not [10:11:26] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4204063 (10Aschmidt) Am 14.05.18 um 11:37 Uhr schrieb TomT0m: > The more I personnaly dig into this questions, the more issues are > opened and the less clear it becomes that there is a... [10:17:02] ahhh [10:17:03] All of these community extensions can be downloaded using pull-deps with the coordinate io.druid.extensions.contrib:EXTENSION_NAME:LATEST_DRUID_STABLE_VERSION. [10:18:51] so they didn't bundle it in the 0.11 release, but it is only a matter of dropping a jar [10:19:49] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Restart Analytics hosts for Java 8 Security upgrades - https://phabricator.wikimedia.org/T194268#4204092 (10MoritzMuehlenhoff) [10:31:43] joal: all right going afk for ~2h, at my return I'll try to set up labs with parquet (and after that druid analytics if tests are fine!) [10:32:52] basically https://mvnrepository.com/artifact/io.druid.extensions.contrib/druid-parquet-extensions/0.11.0 [10:37:39] ah already managed to pull it down with the druid's pull-deps tool [10:37:43] seems super easy [10:40:04] joal: already deployed in labs, will read docs and enable it after lunch :) [10:40:25] if tests are ok I'll add it to another version of the druid's deb [10:40:27] :) [10:40:28] * elukey afk! [10:46:34] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4204170 (10TomT0m) > In such cases there is a simple rule which says that you should choose the safest way in order to comply with the law, as any doubt remaining would be baneful. Wel... [12:54:59] (03CR) 10Sahil505: "> Please do take a second look, the loadMorerRrows button does not" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/427774 (https://phabricator.wikimedia.org/T192407) (owner: 10Sahil505) [13:10:15] joal: druid avro/parquet extensions loaded in labs [13:10:27] you can play with indexation whenever you want :) [13:11:10] OOOOO [13:11:10] boy [13:11:20] o/ [13:12:23] o/ [13:21:59] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Reimage the Debian Jessie Analytics worker nodes to Stretch. - https://phabricator.wikimedia.org/T192557#4204536 (10Ottomata) Docs look good elukey thank you! [13:45:48] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4178076 (10Nemo_bis) >>! In T193728#4188820, @Denny wrote: > No, I was seriously not aware that we are uploading datasets with incompatible licenses. Perhaps because it's not so. CC-0... [13:51:56] 10Analytics, 10Analytics-Kanban: Update user_history and page_history column naming convention - https://phabricator.wikimedia.org/T188669#4204597 (10Milimetric) @TheDragonFire we keep tasks in the Done column until our manager can review them and close them, but I can see that this causes some friction when t... [13:57:41] one thing that I didn't like about the druid releases is https://github.com/druid-io/druid/issues/3924 [14:00:24] even if it takes a bit more, I'd be more inclined to use github via https to download the druid's git repo and then use maven to build [14:06:35] hm, yeah, maybe they'll take your comment seriously and just do that? [14:07:14] !log deploying refinery to enable dropping cu_changes data [14:07:15] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:07:19] not sure, the description is already very detailed but I haven't seen any movement in th right direction :( [14:09:22] yeah, how bad would it be to do the github -> maven build hing [14:09:24] *thing [14:17:40] already done this time to test some builds of druid (changing the hadoop-client etc..), it is boring and takes a bit more time but it works [14:27:49] I'm for it, can't be too careful these days [14:29:23] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Deploy Turnilo (possible pivot replacement) - https://phabricator.wikimedia.org/T194427#4204750 (10Milimetric) p:05Triage>03Normal [14:31:38] 10Analytics, 10Analytics-Kanban: Update anonymous grouping to use User Agent - https://phabricator.wikimedia.org/T193415#4204753 (10Milimetric) [14:32:32] 10Analytics, 10Analytics-Kanban: Rename new geowiki to geoeditors - https://phabricator.wikimedia.org/T193429#4204756 (10Milimetric) a:03Milimetric [14:32:42] 10Analytics, 10Analytics-Kanban: Archive old geowiki data (editors per country) and make it easily available at WMF - https://phabricator.wikimedia.org/T190856#4204760 (10Milimetric) a:03Milimetric [14:32:51] 10Analytics, 10Analytics-Kanban: Update anonymous grouping to use User Agent - https://phabricator.wikimedia.org/T193415#4168647 (10Milimetric) a:03Milimetric [14:33:54] 10Analytics, 10Easy: Productionize job for Global Innovation Index from Hadoop Geowiki data - https://phabricator.wikimedia.org/T190535#4204765 (10Milimetric) a:05Milimetric>03None [14:38:44] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4204771 (10Denny) @Psychoslave sorry to disagree on the questions, but are we in any disagreement on these three questions? We should not allow the (significant) import of data from da... [14:41:11] PROBLEM - turnilo on thorium is CRITICAL: connect to address 10.64.53.26 and port 9091: Connection refused [14:42:30] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4204779 (10Denny) @Nemo_bis thanks, I agree with your point a lot. But regarding your question - just because there is a database which happens to reproducible should not trigger any r... [14:46:52] (03CR) 10Fdans: [V: 031 C: 031] "Tested with regexes that after conducting the audit I know are parsed different between the two versions of regexes.yaml, so adding +1 on " (038 comments) [analytics/ua-parser/uap-java] (wmf) - 10https://gerrit.wikimedia.org/r/429527 (https://phabricator.wikimedia.org/T189230) (owner: 10Nuria) [14:47:00] so I came up with [14:47:02] A:hadoop or A:aqs or A:druid or A:eventlogging or A:notebook or A:kafka-analytics or A:kafka-jumbo or P{O:piwik or O:kafka::monitoring or O:analytics_cluster::webserver or O:statistics::web or O:statistics::cruncher or O:statistics::private} [14:47:21] those should be all the hosts that we manage and their puppet roles (cumin syntax) [14:47:25] that is 85! [14:48:23] excluding the ones that we partially own, like kafka main eqiad/codfw and db110[78] [14:49:26] !log deployment of refinery done [14:49:27] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:50:29] elukey: cool! [14:51:37] 10Analytics, 10Analytics-Kanban: Archive old geowiki data (editors per country) and make it easily available at WMF - https://phabricator.wikimedia.org/T190856#4204826 (10Milimetric) a:05Milimetric>03None [14:51:58] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Bar chart changes height when toggling splits - https://phabricator.wikimedia.org/T194431#4204843 (10Milimetric) a:03Milimetric [14:56:01] RECOVERY - turnilo on thorium is OK: TCP OK - 0.000 second response time on 10.64.53.26 port 9091 [14:56:06] (03PS1) 10Milimetric: Add package-lock [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/432992 [14:56:20] (03CR) 10Milimetric: [V: 032 C: 032] Add package-lock [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/432992 (owner: 10Milimetric) [14:58:19] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Google-Summer-of-Code (2018): GSoC Proposal 2018 : [Analytics] Improvements to Wikistats2 front-end - https://phabricator.wikimedia.org/T189964#4204874 (10mforns) @sahil505 The new timeline looks great. I wish you an awesome coding period :] [14:58:23] elukey: I have tested boh KIS and parquet - no success:( [14:58:41] elukey: what is the A: part of cumin syntax? [14:58:50] joal: KIS extension is not loaded yet, do you want me to? [14:59:01] elukey: Ah sorry ! [14:59:05] ottomata: it is a cumin alias (one of the existing ones) [14:59:07] yes please :) [14:59:10] elukey: --^ [14:59:11] joal: but the parquet one is! :( [14:59:14] does it fail? [14:59:21] elukey: as for parquet, I need to investigate :) [15:00:51] ping mforns ottomata milimetric [15:23:34] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4204925 (10Nemo_bis) >>! In T193728#4204779, @Denny wrote: > To give an example: it is easy to imagine a company that sells the list of all countries and their capitals as a dataset tha... [15:48:03] 10Analytics, 10Analytics-Kanban: Sesssion reconstruction - evaluate privacy threat - https://phabricator.wikimedia.org/T194058#4204992 (10fdans) p:05Triage>03Normal [15:49:38] 10Analytics, 10User-Elukey: Add maxmind ip info to webrequest dataset on druid - https://phabricator.wikimedia.org/T194055#4204995 (10fdans) p:05Triage>03Normal [15:52:54] 10Analytics, 10Operations, 10Patch-For-Review: Puppet admin module should support adding system users to managed groups - https://phabricator.wikimedia.org/T174465#4205006 (10fdans) p:05Normal>03High [16:02:19] hi mforns [16:03:04] mforns: can I make you a question about the pageviews API? [16:41:04] Hi! Are the Hive event.schemaname tables pre-filtered to eliminate known/declared spiders and robots, or is it just generally assumed that those don't send events? [16:41:37] (like you'd do with an agent_type = 'user' clause in a query of wmf.webrequest) [16:41:40] thx in advance!! [16:44:13] AndyRussG: you should be able to use the event.user_agent.is_bot field i think [16:44:19] where event.user_agent.is_bot = false [16:44:19] ? [16:44:20] or maybe [16:44:23] yeah [16:44:24] i think so... [16:44:26] somethign like that :) [16:46:29] ottomata: ah fantastic, thx!!! [16:46:33] didn't see that one :) [16:49:22] elukey: have you loaded KIS on labs (sorry, I can't recall he last status) [16:49:43] yep yep [16:49:56] Cool :) [16:50:16] brb! [16:52:54] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Intervals/buckets for data arround pageviews per country in wikistats maps - https://phabricator.wikimedia.org/T188928#4205176 (10Milimetric) For whomever works on this, we have two possible solutions, they can think of a third one and we generally tr... [17:03:00] 10Analytics, 10Analytics-Wikistats: Beta Release: Wikistats: support annotations in graphs - https://phabricator.wikimedia.org/T178015#3677997 (10mforns) As discussed in the tasking meeting, one idea could be having a set of pages on-wiki under a json namespace that store the annotations. Those would be divide... [17:03:06] milimetric, ^ [17:27:37] !log enabling main-eqiad job topics -> jumbo mirroring [17:27:38] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:48:36] thanks mforns [17:51:05] hey milimetric, if you send a new patch in https://gerrit.wikimedia.org/r/#/c/432104 with => aligned I can merge [17:51:45] doing [17:54:05] sorry I missed that, I assumed it was complaining that it couldn't merge [17:56:26] Jenkins always complains! :D [17:56:31] it is like oozie [18:01:28] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4205401 (10Denny) @Rspeer regarding the ontology: the ontology of Wikidata is genuinely unique and not copied from any Wikipedia project, or any other project. It has been created on Wi... [18:06:03] milimetric: qq - if we use 2>&1 at the end, don't we defeat the purpose of having the mailto option? [18:06:36] honestly I just copied from the other scripts [18:06:39] lemme see [18:07:16] yes yes it was more a generic question [18:09:01] I see, uh... I guess mailto is more elegant? [18:10:06] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4205410 (10Denny) @Nemo_bis : good point. I wouldn't know what a good example is, though, maybe someone else can come up with something. [18:10:21] k, removed the logging redirect elukey [18:12:28] Notice: /Stage[main]/Profile::Analytics::Refinery::Job::Data_purge/Cron[mediawiki-raw-cu-changes-drop-month]/ensure: created [18:12:31] Notice: /Stage[main]/Profile::Analytics::Refinery::Job::Data_purge/Cron[mediawiki-geoeditors-drop-month]/ensure: created [18:12:34] milimetric: --^ [18:12:37] :) [18:12:45] thanks elukey [18:21:50] 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Refresh zookeeper nodes in eqiad - https://phabricator.wikimedia.org/T182924#4205429 (10elukey) Highlight of the plan to swap conf1001 with conf1004: * merge https://gerrit.wikimedia.org/r/#/c/432615/ and run puppet on kafka[12]00[123] hosts (not relate... [18:31:30] joal: o/ do you need any help with Druid? [18:34:53] yo ottomata. [18:35:32] ottomata: will you allow me to find a new Analytics Systems Hangtime? [18:35:55] ottomata: dsaez and I have some things to talk about ;) but we can't make it tomorrow at the time scheduled. [18:36:19] ottomata: yes please! [18:36:22] oops [18:36:23] haha [18:36:25] leila: yes please! [18:36:59] * leila tries to find a new time. [18:42:41] ottomata: moved it. thanks. [18:42:46] dsaez: ^ [18:44:49] elukey: Thanks for offering, the druid-user-right issue is what we expected - now facing another interesting data-schema issue :) [18:45:01] nice [18:46:43] ack :) [18:51:16] elukey: I've hit https://groups.google.com/forum/#!topic/druid-user/Wdy3qxGwbek [18:51:24] elukey: the parquet stuff doesn't seem ready [18:56:38] avro issues? [19:00:06] elukey: versions differences between packages :( [19:00:12] uffffff [19:00:19] does parquet use avro under the hood? [19:00:24] yessir [19:00:44] ahhhh didn't know it! [19:03:22] /usr/share/druid/extensions/druid-avro-extensions/avro-1.7.7.jar [19:03:37] /usr/share/druid/extensions/druid-parquet-extensions/parquet-avro-1.8.2.jar [19:03:39] /usr/share/druid/extensions/druid-parquet-extensions/avro-1.8.0.jar [19:03:42] lol [19:03:42] reallyyyy [19:03:48] elukey: YAYYYYY :) [19:05:08] joal: we can try to download druid-parquet 0.10.0 and check? [19:05:23] I deployed 0.11 with the assumption that it was the right one [19:05:58] elukey: I would have expected the extension to follow the number of the main indeed - We can try with 0.10.0 :) [19:06:07] elukey: Other news - KIS works :) [19:06:45] elukey: We'll have to learn its config tricks and whereabouts but it works as is [19:07:27] 10Analytics, 10Analytics-Wikistats: Present a page view metric description to the user that they are likely to understand - https://phabricator.wikimedia.org/T182109#4205538 (10sahil505) Okay, would be working on 2 things for this task: 1. Improving Research:Page_view and other linked pages by summarizing its... [19:10:47] joal: nice!!!! [19:11:02] I just deployed druid-parquet 0.10 in labs and restarted overlord/middlemanager [19:11:10] now avro seems 1.7 on it [19:11:24] (I left the druid-avro extension to 0.11 though) [19:12:00] going to dinner and then I'll recheck later :) [19:24:59] elukey: I confirm it works :) [19:25:16] Great - This is finally all coming together :) [19:25:48] elukey: I'm gonna stop for tonight, thanks A LOT for all the testing with Druid :) [19:26:42] See you tomorrow a-team :) [19:27:40] joal: nice! [19:27:48] let's do a recap tomorrow then :) [19:27:56] great elukey :) [19:28:19] elukey: If you'll have some time tomorrow, I'll gladly show you the firs version of the prez I have for Thursday :) [19:29:08] joal: oh yes please! [19:29:15] I'd be happy to listen to it! [19:29:27] maybe I'll finally learn something :P [19:29:35] all right, going off as well! [19:29:54] elukey: you're far more epxert in those topics than many of the people I'll talk to ;) [19:29:58] Bye [19:32:28] 10Analytics, 10Analytics-Wikistats: Present a page view metric description to the user that they are likely to understand - https://phabricator.wikimedia.org/T182109#4205571 (10Nuria) Let’s please make sure the UI does not call mediawiki API. The friendly description will remain mostly not changed once created... [19:59:02] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4205679 (10Aschmidt) Am 14.05.18 um 17:23 Uhr schrieb Nemo_bis: > But I'd argue that nobody would see such a dataset as problematic, > especially because it's so small (few hundreds dat... [19:59:46] 10Analytics, 10Operations, 10SRE-Access-Requests: Access to usergroups for Marshall Miller - https://phabricator.wikimedia.org/T194550#4205682 (10Dzahn) a:03herron [20:22:27] 10Analytics-Cluster, 10Analytics-Kanban, 10Operations, 10Traffic, and 2 others: TLS security review of the Kafka stack - https://phabricator.wikimedia.org/T182993#4205765 (10Ottomata) @bblack did this end up being a Q4 goal for traffic team? [20:25:35] 10Analytics-Cluster, 10Analytics-Kanban, 10Operations, 10Traffic, and 2 others: TLS security review of the Kafka stack - https://phabricator.wikimedia.org/T182993#4205770 (10BBlack) I think this ended up being an Analytics Q4 goal? It's not on our goals list, but we agree to alot some time to it in this Q... [20:26:16] 10Analytics, 10Operations, 10Patch-For-Review: Puppet admin module should support adding system users to managed groups - https://phabricator.wikimedia.org/T174465#4205772 (10Ottomata) @akosiaris @MoritzMuehlenhoff, I need to resurrect this task. We also need this in order for the druid user to access `hdfs... [20:36:10] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4205798 (10Nemo_bis) >>! In T193728#4205679, @Aschmidt wrote: > Am 14.05.18 um 17:23 Uhr schrieb Nemo_bis: >> But I'd argue that nobody would see such a dataset as problematic, >> espec... [20:40:53] 10Analytics-Cluster, 10Analytics-Kanban, 10Operations, 10Traffic, and 2 others: TLS security review of the Kafka stack - https://phabricator.wikimedia.org/T182993#4205810 (10Ottomata) Ok, great! From our side, we're mostly looking on either more TODOs and/or approval to remove IPSec from jumbo + varnishka... [20:46:16] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Solve legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728#4205827 (10Aschmidt) Am 14.05.18 um 22:36 Uhr schrieb Nemo_bis: > But law is a matter of quality rather than size. > > But quantitative indicators can be a proxy for quality. A dat... [21:48:15] 10Quarry: Upgrade Quarry main server from Trusty - https://phabricator.wikimedia.org/T194691#4205967 (10zhuyifei1999) [21:48:57] 10Quarry, 10Patch-For-Review: Update dependencies - https://phabricator.wikimedia.org/T192731#4148133 (10zhuyifei1999) [21:49:01] 10Quarry: Upgrade Quarry main server from Trusty - https://phabricator.wikimedia.org/T194691#4205967 (10zhuyifei1999) [21:52:32] 10Quarry: Upgrade Quarry main server from Trusty - https://phabricator.wikimedia.org/T194691#4205998 (10zhuyifei1999) @Framawiki Do you think we should build two instances, one to serve the web and one for the persistence (database)? [22:21:02] 10Analytics: Use native timestamp types in Data Lake edit data (needs Hive 1.2) - https://phabricator.wikimedia.org/T161150#4206042 (10Neil_P._Quinn_WMF) 05Open>03Resolved a:03Neil_P._Quinn_WMF This is actually resolved! When I originally filed this, the timestamps were stored in Mediawiki's string format... [22:21:14] 10Analytics: Use native timestamp types in Data Lake edit data (needs Hive 1.2) - https://phabricator.wikimedia.org/T161150#4206045 (10Neil_P._Quinn_WMF) a:05Neil_P._Quinn_WMF>03None [22:30:20] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Fix Mirror Maker erratic behavior when replicating from main-eqiad to jumbo - https://phabricator.wikimedia.org/T189464#4206053 (10Ottomata) Hm, am seeing ``` [2018-05-14 22:18:20,458] 17217831 [mirrormaker-thread-6]... [22:35:32] PROBLEM - Kafka MirrorMaker main-eqiad_to_jumbo-eqiad max lag in last 10 minutes on einsteinium is CRITICAL: 4.533e+05 gt 1e+05 https://grafana.wikimedia.org/dashboard/db/kafka-mirrormaker?var-datasource=eqiad+prometheus/ops&var-lag_datasource=eqiad+prometheus/ops&var-mirror_name=main-eqiad_to_jumbo-eqiad [22:39:10] !log bouncing main-eqiad -> jumbo mirror maker after committing new offset for eqiad.mediawiki.job.RecordLintJob [22:39:10] [22:39:11] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:44:40] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Fix Mirror Maker erratic behavior when replicating from main-eqiad to jumbo - https://phabricator.wikimedia.org/T189464#4206061 (10Ottomata) I just recommitted the offset for eqiad.mediawiki.job.RecordLintJob, hopefu... [23:30:03] PROBLEM - Kafka MirrorMaker main-eqiad_to_jumbo-eqiad average message produce rate in last 30m on einsteinium is CRITICAL: 0 le 0 https://grafana.wikimedia.org/dashboard/db/kafka-mirrormaker?var-datasource=eqiad+prometheus/ops&var-lag_datasource=eqiad+prometheus/ops&var-mirror_name=main-eqiad_to_jumbo-eqiad [23:36:02] PROBLEM - Kafka MirrorMaker main-eqiad_to_jumbo-eqiad average message consume rate in last 30m on einsteinium is CRITICAL: 0 le 0 https://grafana.wikimedia.org/dashboard/db/kafka-mirrormaker?var-datasource=eqiad+prometheus/ops&var-lag_datasource=eqiad+prometheus/ops&var-mirror_name=main-eqiad_to_jumbo-eqiad [23:39:28] yar [23:39:30] looking into it [23:54:45] !log bouncing main -> jumbo MirrorMaker with larger max.request.size [23:54:46] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [23:57:43] RECOVERY - Kafka MirrorMaker main-eqiad_to_jumbo-eqiad average message consume rate in last 30m on einsteinium is OK: (C)0 le (W)100 le 7.994e+04 https://grafana.wikimedia.org/dashboard/db/kafka-mirrormaker?var-datasource=eqiad+prometheus/ops&var-lag_datasource=eqiad+prometheus/ops&var-mirror_name=main-eqiad_to_jumbo-eqiad [23:58:13] RECOVERY - Kafka MirrorMaker main-eqiad_to_jumbo-eqiad average message produce rate in last 30m on einsteinium is OK: (C)0 le (W)100 le 1.098e+05 https://grafana.wikimedia.org/dashboard/db/kafka-mirrormaker?var-datasource=eqiad+prometheus/ops&var-lag_datasource=eqiad+prometheus/ops&var-mirror_name=main-eqiad_to_jumbo-eqiad