[00:17:44] 10Analytics, 10Analytics-Wikistats, 10Operations, 10Traffic, 10Regression: [Regression] stats.wikipedia.org redirect no longer works ("Domain not served here") - https://phabricator.wikimedia.org/T126281 (10Krinkle) 05Resolved>03Open Broken again. is responding.. (good)... [00:44:48] 10Analytics, 10Analytics-Data-Quality, 10Contributors-Analysis, 10Product-Analytics: mediawiki_history missing page events - https://phabricator.wikimedia.org/T205594 (10Neil_P._Quinn_WMF) [06:44:13] !log rolling restart of Druid coordinators and historicals on the Druid public cluster to pick up new Hadoop masters (one at the time, very gently) [06:44:15] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [06:54:56] Hi elukey - I'm assuming you'rev already done with the druid ones? [06:55:05] morning :) [06:55:11] nope, doing very gently [06:55:21] :) [06:55:27] coordinators done (1 min of sleep between each one) [06:55:37] historicals in progress (5 mins of sleep) [06:58:17] elukey: morning-braindump: I think only historical nodes restart was necessary - Coordinator makes use of Mysql to manage segments (made available by indexation service) - Historical is the only folks I think actually interacting with HDFS on a non-per-task way [07:05:33] joal: I think so too, but a restart is not a bad thing JUST TO BE SURE [07:05:37] :) [07:05:53] no problem - better safe than sorry as would say a colleague of mine [07:06:01] :) [07:08:17] exactly :D [07:08:28] historicals restarted [07:09:45] \o/ [07:10:35] in ~20 mins db1108 (analytics-store) is going off for maintenance (max 15 mins) [07:10:42] err analytics-slave [07:10:56] mariadb upgrade + kernel upgrade [07:51:30] (03CR) 10Joal: [V: 032 C: 032] "Merging to move forward, post-merge tougher review expected from @ottomata." (031 comment) [analytics/wikihadoop] - 10https://gerrit.wikimedia.org/r/337004 (owner: 10Joal) [07:54:47] (03PS1) 10Joal: Correct scala version in pom.xml [analytics/wikihadoop] - 10https://gerrit.wikimedia.org/r/463210 [07:55:11] (03CR) 10Joal: [V: 032 C: 032] "Merging bug correction before release" [analytics/wikihadoop] - 10https://gerrit.wikimedia.org/r/463210 (owner: 10Joal) [08:25:03] joal: what is wikihadoop? [08:25:27] elukey: wikihadoop is a library providing hadoop InputFormat to read XML dumps [08:25:34] ahhhh [08:26:23] 10Analytics, 10Analytics-Kanban: Reboot Analytics hosts for kernel security upgrades - https://phabricator.wikimedia.org/T203165 (10elukey) [08:28:08] elukey: I broke archiva :( [08:28:40] elukey: I made a mistake uploading my jar -> I sent it to the snapshot repo instead of release one [08:28:55] And now, I can't even browse the artifact anymore ( [08:30:42] wait wait [08:31:43] with artifact you mean your jar right? [08:33:14] I am trying to get the impact :) [08:33:50] correct, even the artifact defininothing more than my jar I think :) [08:34:55] super ignorant but can we attempt to delete it and then re-upload? [08:34:58] elukey: I managed to delete my mistake, but archiva behaved in a weird way :S - Sorry to have asked for help [08:35:23] elukey: that's my plan, but I couldn't even manage to put my hands on it afterward (there an error from archiva) [08:36:08] elukey: I have found a way to reach the thing and deleted the artifact, no more errors from archiva, trying to re-upload [08:37:32] super :) [08:39:49] * joal really doesn't like archiva [09:06:25] something is still weird with druid [09:06:26] https://grafana.wikimedia.org/dashboard/db/druid?refresh=1m&panelId=46&fullscreen&orgId=1&var-cluster=druid_analytics&var-druid_datasource=All&from=now-2d&to=now [09:07:01] elukey: how? [09:08:27] elukey: seems flat now, no? [09:09:05] load queue is 130 [09:09:30] https://grafana.wikimedia.org/dashboard/db/druid?refresh=1m&panelId=49&fullscreen&orgId=1&var-cluster=druid_analytics&var-druid_datasource=All&from=now-2d&to=now [09:09:30] ohh ! didn't lok at tooltip, looked like 0 - my bad [09:10:26] druid coord says everything is fine [09:10:39] elukey: maybe need a bounce to get back on metrics? [09:10:42] bizarre though [09:10:42] it may be something related to metrics [09:16:54] elukey: could it be that when no segment is missing, druid doesn't actually output any metric (instead of 0), and that the metric-forwarder takes the last existing value? [09:19:28] sorry was checking another thing, going to verify on druid [09:21:43] so the metric is a sum of all the druid nodes, for druid1001 the count is 0, for druid1002 it is 130 [09:22:12] so yeah stale metric :( [09:22:41] now in theory if I restart the prometheus exporter on druid1002 it should be ok [09:28:33] yep :) [09:28:37] it worked [09:33:45] so in theory I could make a little change to the prometheus exporter that sets to zero the metric [09:33:53] the coordinators int metrics I mean [09:38:04] Makes sense elukey [09:51:59] the main problem with this approach is when to set the variable to zero [09:52:26] right - Does "no value" means 0 or problem?> [09:52:52] in theory, if I got it correctly, when a coordinator is leader it sends several metrics plus the jetty ones [09:53:03] when it is not, only the jetty ones [09:53:15] so this could be the "logic" to use [09:53:31] when you don't see metrics anymore related to druid coordinator, set all to zero [10:22:41] (03PS5) 10Fdans: Add top editors metric to Wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/461635 (https://phabricator.wikimedia.org/T189882) [10:28:06] (03PS1) 10Fdans: Add top pages by edits metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/463230 (https://phabricator.wikimedia.org/T204965) [10:45:34] (03CR) 10Fdans: [C: 031] "This looks good to me, just one garbage comment and that's it" (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/440971 (https://phabricator.wikimedia.org/T194705) (owner: 10Milimetric) [10:46:32] of course the prometheus druid exporter is very generic and it does not care about metric names [10:48:29] and also it works on separate data points (that are sent by druid daemons) [11:51:14] helloooo joal, I'm guessing that if a hive table has its timestamp split into partitions (so there's no actual ts column), it has to be transformed before indexing to druid? [12:09:26] hi fdans - There are chances what you say is true, but it really depends on dataset [12:17:56] 10Analytics: Update datasets to have explicit timestamp for druid indexation facilitation - https://phabricator.wikimedia.org/T205617 (10JAllemandou) [12:17:59] fdans: --^ [12:18:58] ooooo nice joal [12:29:36] joal: https://gerrit.wikimedia.org/r/#/c/operations/software/druid_exporter/+/463243/ [12:29:43] when you have time (README change) [12:35:10] reviewed elukey [12:36:31] thanks! amending and merging [12:36:38] +1P [12:36:46] !, not P [13:26:24] 10Analytics: Escape end-of-line special characters from pageview-dumps generation - https://phabricator.wikimedia.org/T205620 (10JAllemandou) [13:45:07] 10Analytics: Escape end-of-line or other special characters from pageview-dumps generation - https://phabricator.wikimedia.org/T205620 (10JAllemandou) [13:45:22] 10Analytics: Why do dumps and pageview api have slightly different counts? - https://phabricator.wikimedia.org/T205457 (10JAllemandou) I have investigated on this topic and have some answers. The main observed difference is due to a shift in hours between dumps and pageview-api. The pageview-dumps data was orig... [13:45:47] 10Analytics-Kanban: Why do dumps and pageview api have slightly different counts? - https://phabricator.wikimedia.org/T205457 (10JAllemandou) a:03JAllemandou [13:56:29] need to go to a doctor appointment people, ttl! [14:00:34] elukey: icinga-wm is back here at your service ;) [14:14:43] 10Analytics, 10DBA, 10User-Banyek: dbstore1002 /srv filling up - https://phabricator.wikimedia.org/T205544 (10Banyek) [14:20:04] 10Analytics, 10DBA, 10User-Banyek: dbstore1002 /srv filling up - https://phabricator.wikimedia.org/T205544 (10Banyek) I start the conversion with /srv/sqldata/srwiki/pagelinks.ibd it is 51G let's see what we get at the end [14:21:31] (03CR) 10Mforns: [V: 032 C: 032] "LGTM! In the future we should make all chart popups look similar to the one in the bar chart, no?" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/463102 (https://phabricator.wikimedia.org/T205461) (owner: 10Fdans) [14:22:18] (03PS13) 10Milimetric: Annotate wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/440971 (https://phabricator.wikimedia.org/T194705) [14:22:24] (03CR) 10jerkins-bot: [V: 04-1] Annotate wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/440971 (https://phabricator.wikimedia.org/T194705) (owner: 10Milimetric) [14:22:32] (03CR) 10Milimetric: Annotate wikistats (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/440971 (https://phabricator.wikimedia.org/T194705) (owner: 10Milimetric) [14:25:07] (03PS14) 10Milimetric: Annotate wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/440971 (https://phabricator.wikimedia.org/T194705) [14:26:20] (03CR) 10Fdans: "@Mforns I for one am pretty over the whole changing the chart tooltips every 3 months" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/463102 (https://phabricator.wikimedia.org/T205461) (owner: 10Fdans) [14:34:08] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Make hover info-box on bar charts consistent with line charts - https://phabricator.wikimedia.org/T205461 (10Milimetric) Apologies, I mis-wrote this task... it should've read the other way around. [14:34:17] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Make hover info-box on line charts consistent with bar charts - https://phabricator.wikimedia.org/T205461 (10Milimetric) [14:34:39] the info-box on line charts is now on top of the annotations, so I have to fix that, then we can merge and deploy [14:35:33] but fdans / mforns: it seems like we're not on the same page about these info boxes, lemme know if you want to talk. Maybe the right thing to do is to just remove the tooltip entirely and show the values in the simplelegend component [14:37:23] milimetric: the task just talked about the tooltip moving to where the data is [14:37:58] yeah, I know, but from marcel's merge message it looked like he didn't think it was yet the same as the bar, and I agree [14:41:55] (03PS15) 10Milimetric: Annotate wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/440971 (https://phabricator.wikimedia.org/T194705) [14:42:37] ok, that last patchset makes the hover hide when you hover on the annotations, and this change is pretty much set to go [14:42:44] let's merge/deploy if mforns / fdans are ok [14:43:00] lemme know when you do so I can add/update annotations pages [14:44:20] testing [14:46:49] milimetric: let's merge, can we merge pages to date too and deploy everything now? [14:47:54] fdans: yeah, good with me [14:49:09] (03CR) 10Fdans: [C: 032] Annotate wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/440971 (https://phabricator.wikimedia.org/T194705) (owner: 10Milimetric) [14:49:59] (03CR) 10Fdans: [V: 032 C: 032] Annotate wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/440971 (https://phabricator.wikimedia.org/T194705) (owner: 10Milimetric) [14:51:01] (03CR) 10Fdans: [V: 032 C: 032] "Agreed to merge on irc" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/458784 (https://phabricator.wikimedia.org/T203180) (owner: 10Fdans) [14:52:33] (03CR) 10jerkins-bot: [V: 04-1] Add pages to date metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/458784 (https://phabricator.wikimedia.org/T203180) (owner: 10Fdans) [14:53:33] (03PS11) 10Fdans: Add pages to date metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/458784 (https://phabricator.wikimedia.org/T203180) [14:53:52] (03CR) 10Fdans: [C: 032] Add pages to date metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/458784 (https://phabricator.wikimedia.org/T203180) (owner: 10Fdans) [14:56:28] (03Merged) 10jenkins-bot: Add pages to date metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/458784 (https://phabricator.wikimedia.org/T203180) (owner: 10Fdans) [15:01:51] (03PS1) 10Fdans: Release 2.4.0 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/463278 [15:03:03] (03CR) 10Fdans: [V: 032 C: 032] Release 2.4.0 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/463278 (owner: 10Fdans) [15:12:26] 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review: setup/install an-coord1001/wmf7621 - https://phabricator.wikimedia.org/T204970 (10Cmjohnson) [15:14:04] 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review: setup/install an-coord1001/wmf7621 - https://phabricator.wikimedia.org/T204970 (10Cmjohnson) a:05Cmjohnson>03None @elukey everything looks good on our end I was able to access the server root@an-coord1001.mgmt.eqiad.wmnet's password: /admin1-> [15:15:04] 10Analytics, 10Operations, 10Patch-For-Review, 10User-Elukey: rack/setup/install an-master100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T201939 (10Cmjohnson) [15:18:30] fdans: thx for deploy, I've stripped out the test annotations from meta and I'll set up empty annotation pages where they belong [15:18:53] I'm also working on cleaning up the dashiki extension messaging at the top so it doesn't tell you to build annotations as a layout with dashiki :) [15:21:58] milimetric: niiiiice [15:26:20] PROBLEM - Check the last execution of check_webrequest_partitions on analytics1003 is CRITICAL: CRITICAL: Status of the systemd unit check_webrequest_partitions [15:26:43] this is me testing --^ [15:26:45] no issue :) [15:26:55] but we have icinga notifications in analytics-alerts@ \o/ [15:28:57] 10Analytics, 10User-Banyek: dbstore1002 /srv filling up - https://phabricator.wikimedia.org/T205544 (10Banyek) [15:36:21] RECOVERY - Check the last execution of check_webrequest_partitions on analytics1003 is OK: OK: Status of the systemd unit check_webrequest_partitions [15:40:02] elukey: luca you are quite the troublemaker [15:40:33] I am [15:40:34] :D [15:45:23] pingyyy mforns you got a minute? [15:45:35] fdans, still in meeting, sooorrryyy! [15:46:07] oh nooooes! :D D: :D D: [15:46:21] that's ok, we can talk during/post standup [15:50:47] fdans, meeting ended, wanna bc? [15:50:54] On my way!! [15:51:18] goddamn safari, who asked you to convert o m w to that [15:51:38] heh [15:56:49] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Services (watching): Modern Event Platform: Event Schema Registry - https://phabricator.wikimedia.org/T201063 (10JMinor) Added per our meeting: As an **analyst or product manager** I want to able to search through existing schemas... [15:58:52] mmmm now I have a suspicion: were you guys already receiving emails from icinga about things failing? [15:59:01] I mean, did you get duplicates for the last alert? [15:59:22] elukey: i did receive e-mails from icianga yes [15:59:23] (one to your personal inbox, the other to analytics-alerts? [16:00:03] I'll add a post standup [16:29:39] 10Analytics, 10Analytics-Kanban: Add ability to bucketize integers as part of event ingestion - https://phabricator.wikimedia.org/T205641 (10Nuria) [16:33:15] 10Analytics, 10Analytics-Kanban: Add ability to bucketize integers as part of event ingestion - https://phabricator.wikimedia.org/T205641 (10Nuria) [16:33:17] 10Analytics, 10Analytics-Kanban: Ingest data into druid for readingDepth schema - https://phabricator.wikimedia.org/T205562 (10Nuria) [16:42:23] 10Analytics, 10Analytics-Kanban: Improve Dashiki extension messaging - https://phabricator.wikimedia.org/T205644 (10Milimetric) [16:42:45] 10Analytics, 10Analytics-Kanban: Improve Dashiki extension messaging - https://phabricator.wikimedia.org/T205644 (10Milimetric) p:05Triage>03High [16:51:47] joal: sooo I don't think I have access to aqs1004? [17:00:06] 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review: setup/install an-coord1001/wmf7621 - https://phabricator.wikimedia.org/T204970 (10Cmjohnson) [17:00:27] 10Analytics-Kanban, 10Beta-Cluster-Infrastructure, 10Operations, 10Patch-For-Review, and 2 others: Prometheus resources in deployment-prep to create grafana graphs of EventLogging - https://phabricator.wikimedia.org/T204088 (10Jdlrobson) We're seeing [[ https://grafana-labs-admin.wikimedia.org/dashboard/db... [17:00:41] 10Analytics, 10Analytics-Wikistats, 10Operations, 10Traffic, 10Regression: [Regression] stats.wikipedia.org redirect no longer works ("Domain not served here") - https://phabricator.wikimedia.org/T126281 (10Nuria) Ping @BBlack can we get the redirect from https://stats.wikipedia.org to https://stats.w... [17:00:44] 10Analytics, 10Analytics-Kanban: Ingest data into druid for readingDepth schema - https://phabricator.wikimedia.org/T205562 (10fdans) Hi @Tbayer, just wanted to let you know that at the moment we can't index the time values stored in this dataset due to their high cardinality, so I wanted to ask if the dataset... [17:02:14] fdans: that is possible, in which case you can work with joal to test your code from his end [17:03:30] nuria: yea, I'm making a change in ./dist that joal could just check out, which uses tunnel urls [17:12:04] 10Analytics, 10Analytics-Kanban: Ingest data into druid for readingDepth schema - https://phabricator.wikimedia.org/T205562 (10Nuria) The other option is that data time measurements are bucketized when sent from the client, in buckets like [0-50ms [50-100] etc , that will lower cardinality of integers and all... [17:26:15] (03PS1) 10Fdans: [DO NOT MERGE] Test new metrics using tunnel aqs url [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/463324 [17:26:46] joal: if you have time you can check this out on your local machine ^ [17:27:09] just do a SimpleHTTPServer on port 5000 on the repo and go to this url: [17:27:18] http://localhost:5000/dist/#/en.wikipedia.org/contributing/top-edited-pages/normal|table|2-Year|~total [17:27:28] http://localhost:5000/dist/#/en.wikipedia.org/contributing/top-editors/normal|table|2-Year|~total [17:27:47] (cc nuria) [17:28:46] (you obvs need a tunnel to aqs1004 to localhost:1004, I've set that up in the UI config) [17:42:02] Heya fdans [17:42:18] o/ [17:42:30] So you don't have access to aqs1004? [17:42:44] Then lets do from stat1004! [17:43:22] fdans: ssh -N stat1004.eqiad.wmnet -L 7232:aqs1004.eqiad.wmnet:7232 [17:43:34] Let's see if this works :) [17:43:45] joal: ohhh interesting [17:47:28] joal: I'm getting a time out on the tunnel [17:47:37] does that work for you? [17:48:15] ok - that means aqs1004 really doesn't want you :( [17:49:24] So - How may I test? [17:49:45] joal: if you clone the repo and check out the CR I passed you [17:49:46] (03PS8) 10Mforns: Add support for group param for chgrp-ing generated reports [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/462732 (https://phabricator.wikimedia.org/T205441) (owner: 10Bearloga) [17:50:06] joal: but one sec because i realized something [17:50:07] (03CR) 10Bearloga: "recheck" [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/462732 (https://phabricator.wikimedia.org/T205441) (owner: 10Bearloga) [17:52:15] (03PS2) 10Fdans: [DO NOT MERGE] Test new metrics using tunnel aqs url [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/463324 [17:52:22] (03CR) 10jerkins-bot: [V: 04-1] Add support for group param for chgrp-ing generated reports [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/462732 (https://phabricator.wikimedia.org/T205441) (owner: 10Bearloga) [17:52:32] joal: ok now check that out ^ [17:52:38] (my cr) [17:52:39] (03CR) 10jerkins-bot: [V: 04-1] Add support for group param for chgrp-ing generated reports [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/462732 (https://phabricator.wikimedia.org/T205441) (owner: 10Bearloga) [17:53:14] joal and do python -m SimpleHTTPServer 5000 on the repo dir [17:53:48] with that you can do http://localhost:5000/dist-dev/#/en.wikipedia.org/contributing/top-editors/normal|table|2-Year|~total [17:54:00] (assuming you have a ssh tunnel) [17:54:08] fdans: maybe I shall npm install, ruight? [17:54:39] nuria: sent an email with the goals that I added, will re-check later on if anything needs to be amended (going to dinner now) [17:54:46] joal: you change was deployed [17:55:10] \o/ Pchelolo ! Many thanks for that :) [17:55:20] Pchelolo: any rollback plans or good from side so far? [17:55:40] good so far, Parsoid was rescheduled for Monday [17:55:50] so on Monday there might be rollbacks [17:55:59] Ah ok - Mwarf - Sorry for the dpeloy on such a dumy patch :( [17:56:07] Pchelolo: --^ [17:56:50] joal: this means we can just test with prod? [17:57:18] Pchelolo: on monday there might be rollbacks sounds GREAT #not [17:57:49] nuria: that's the best we can deliver :( [17:57:59] fdans: indeed !!! [17:58:07] niiiice thanks Pchelolo [17:58:13] :D [17:58:18] fdans: The doc page is still a cached one with incorrect info, but the way to quer is good [17:58:20] Pchelolo: no worries, we will keep an eye on that [17:58:24] if the rollback is very problematic, we can do some git magic to reorder the commits, but I'd rather not [17:59:23] fdans: the /dist-dev url gives me a 404 :( [17:59:45] ah damn, i meant just /dist/ sorry [17:59:55] joal: but don't worry, I got it from here :) [17:59:59] :) [18:01:10] fdans: Interesting - When testing top metrics, I get a CORB on localhost :) [18:01:37] fdans: I must say that now I do have a version of wikistats2 codebase on my computer ;) [18:02:22] joal: the corb only appears when there's a 404 [18:02:30] I'm missing something [18:02:34] Ah - hm - not nice [18:02:40] this should be fine no? [18:02:45] https://wikimedia.org/api/rest_v1/metrics/edited-pages/top-by-edits/en.wikipedia.org/all-editor-types/all-page-types/2018/08/all-days [18:03:47] hm, it should [18:04:12] (03PS9) 10Bearloga: Add support for group param for chgrp-ing generated reports [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/462732 (https://phabricator.wikimedia.org/T205441) [18:04:16] joal: it's still asking me for the granularity [18:04:21] same for me [18:04:24] I don't get it [18:04:53] (03PS10) 10Bearloga: Add support for group param for chgrp-ing generated reports [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/462732 (https://phabricator.wikimedia.org/T205441) [18:06:41] (03CR) 10Mforns: [C: 032] "LGTM!" [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/462732 (https://phabricator.wikimedia.org/T205441) (owner: 10Bearloga) [18:07:23] fdans: I confirm aqs1004 answers correctly to the request [18:07:36] ah [18:07:41] Pchelolo: We have a weird behavior, as if the change had not been deployed - Can it be possible? [18:07:53] (03CR) 10jerkins-bot: [V: 04-1] Add support for group param for chgrp-ing generated reports [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/462732 (https://phabricator.wikimedia.org/T205441) (owner: 10Bearloga) [18:07:56] joal: moment [18:08:27] joal: does the ui look ok? [18:08:43] fdans: didn't continue testing - will get back to that :) [18:10:13] fdans: blcoked call because of CORS :( [18:10:13] joal: you are very right [18:10:18] https://gerrit.wikimedia.org/r/#/c/mediawiki/services/restbase/deploy/+/463305/ [18:10:48] Ah Pchelolo :( [18:10:54] joal: ohhhh the aqs host might not be configured for cors? [18:11:10] I think they don't [18:11:25] Well I mean, it's good pratice not allow CORS by default, no fdans ? [18:12:31] joal: you see the 'submit' button [18:12:50] this is an important lesson for me. never try to work when you're terribly sick [18:13:14] joal: should I fix this mistake now? [18:13:22] I can ask Marko to do it [18:13:37] joal: thought I'd get in touch about your and ottomata's comments on T204572 about moving from MySQL to Hive. I'm not against the idea, but what would it take? [18:14:09] the queries are run through ReportUpdater, would moving them mean switching to Python and query Hive through that? [18:14:27] T204572: Add index to mediawiki_page_create_3 - https://phabricator.wikimedia.org/T204572 [18:15:55] Pchelolo: Without even thinking of lessons or what, please rest if you're sick - Our stuff is not worth it :) [18:16:33] Pchelolo: If Marko has nothing else to do etc, why not, but I'm assuming he's busy and all - The deploy can happen next week [18:16:48] Hi Nettrom [18:16:56] joal: your stuff is totally worth it :) I'll ping Marko and ask if he's not going away yet [18:17:45] yeah Pchelolo we appreciate yalls work, please don't be here if you're sick :) [18:17:50] Pchelolo: My wording was definitely not well chosen - I meant no work-related item is more important than health - Please rest :) [18:18:29] Nettrom: ReportUpdater is hive-ready normally [18:18:37] thanks and really sorry about my mistake [18:19:17] No worries Pchelolo - Thanks for having made it despite sickness and all - We'll do that when you're back :) [18:20:30] Nettrom: I'm no report-updater expert, but I think the first step would be to manually ensure you have a hive query whose result match the SQL one [18:21:07] Nettrom: And then it's a matter of finding in the existing examples how to parameterize the query [18:21:29] mforns: Can you confirm I don't say too many mistakes here please --^ ? [18:22:00] joal, reading [18:22:01] fdans: Let's find a way to make that thing testable, shall we? [18:23:04] joal, Nettrom, yes, you can use RU with hive [18:23:11] joal: hmm, I'm thinking [18:23:49] Nettrom, currently you have to use a small hack to use it, but it works fine [18:24:39] joal: we could disable temporarily the same-origin policy in your browsers if you're feeling a lil crazy? [18:24:42] Nettrom, here's an example of hive usage: https://github.com/wikimedia/analytics-reportupdater-queries/blob/master/browser/all_sites_by_browser_family_and_major_percent [18:24:59] you have to use a wrapper script, but that's all [18:25:48] mforns: cool, thanks! [18:26:11] I can open a ticket to track this, and also make sure the documentation is updated as it currently doesn't seem to point to any Hive examples [18:26:54] not sure how quickly I'll get around to this, though [18:27:21] fdans: Could we have a running server on stat1004 querying aqs1004:7232 directly, and connect to stat1004:500 through tunnel? [18:29:53] joal: before we set up a big contraption, let me remind that there's no actual grave need to test this before the endpoint goes live [18:30:13] any changes on the UI will be minor tweaks that take literally seconds to deploy [18:31:39] (03PS2) 10Fdans: Add top pages by edits metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/463230 (https://phabricator.wikimedia.org/T204965) [18:40:26] 10Analytics, 10Community-Tech, 10Product-Analytics: Move page creation queries to Hive - https://phabricator.wikimedia.org/T205654 (10nettrom_WMF) [19:07:56] Nettrom, ping me anytime you need help with RU! [19:11:22] joal, fdans, can I help with AQS or Wikistats2? [19:11:34] (03CR) 10Bearloga: "ERROR - Report "basic_usage" could not be written because of error: Could not change group ownership of the output file (chown() takes no " [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/462732 (https://phabricator.wikimedia.org/T205441) (owner: 10Bearloga) [19:11:37] mforns: hopefully it'll be not be a lot, thanks for being available! :) [19:12:46] fdans: excuse me got an impromptu visit [19:13:16] Nettrom: feel free to ping me about reportupdater too :) [19:13:50] Nettrom: the entirety of search platform stats is generated with RU [19:14:13] bearloga: cool, good to know that you've messed with it too! [19:15:08] Nettrom: about to make my first patch to it too ^_^ [19:15:25] bearloga: nice! you're about to level up! [19:24:43] (03PS11) 10Bearloga: Add support for group param for chgrp-ing generated reports [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/462732 (https://phabricator.wikimedia.org/T205441) [19:26:48] mforns: removing the named arguments worked. tested with group in reports section and in defaults section of the yaml config. changing ownership of existing reports to whatever the current value of group in the config also works. gonna merge :D [19:27:02] er, gonna +2/+2 :D [19:27:30] (03CR) 10jerkins-bot: [V: 04-1] Add support for group param for chgrp-ing generated reports [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/462732 (https://phabricator.wikimedia.org/T205441) (owner: 10Bearloga) [19:29:04] (03CR) 10Bearloga: [V: 032 C: 032] "Tested it and it works. Whether the group is specified for individual report or in defaults section, RU will use that as the group when as" [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/462732 (https://phabricator.wikimedia.org/T205441) (owner: 10Bearloga) [19:29:34] great bearloga :] [19:30:20] mforns: phew, thanks for your help in this! now on to change the configs :P [19:31:37] (03CR) 10jerkins-bot: [V: 04-1] Add support for group param for chgrp-ing generated reports [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/462732 (https://phabricator.wikimedia.org/T205441) (owner: 10Bearloga) [19:32:55] mforns: hey flake8 seems to be preventing the auto-merge-by-CI [19:33:46] bearloga, I merged manually [19:34:00] mforns: oh cool, thanks! [19:34:13] np! [19:43:53] 10Analytics, 10Discovery-Analysis, 10Product-Analytics, 10Wikidata, and 2 others: Query stats dashboard not updating - https://phabricator.wikimedia.org/T204415 (10mpopov) [19:43:57] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: 'group' parameter in Reportupdater for automatic chgrp of generated reports - https://phabricator.wikimedia.org/T205441 (10mpopov) 05Open>03Resolved a:03mpopov Done, yay [20:20:53] 10Analytics, 10Community-Tech, 10Product-Analytics: Move page creation queries to Hive - https://phabricator.wikimedia.org/T205654 (10Neil_P._Quinn_WMF) p:05Triage>03Normal [20:21:31] 10Analytics, 10Community-Tech, 10Product-Analytics: Move page creation queries to Hive - https://phabricator.wikimedia.org/T205654 (10mpopov) For example usage of Hive with Reportupdater, see: https://github.com/wikimedia/wikimedia-discovery-golden/tree/master/modules/metrics/wdqs [20:35:20] Luca! Happy Birthday!!! [20:35:30] 10Analytics, 10Community-Tech, 10Product-Analytics: Move page creation queries to Hive - https://phabricator.wikimedia.org/T205654 (10Neil_P._Quinn_WMF) p:05Normal>03Low [20:36:07] By the way fdans - top-patch is not yet available, but, the per-project-family is :) [20:36:13] https://wikimedia.org/api/rest_v1/metrics/edits/aggregate/all-wikipedia-projects/all-editor-types/all-page-types/daily/20180801/20180901 [20:36:54] joal: this one we cannot display on UI yet cause ui is not ready but SOON [20:37:02] :) [20:37:30] (03PS1) 10Joal: Add MediawikiXMLDumpsConverter spark job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/463370 (https://phabricator.wikimedia.org/T202490) [20:39:01] 10Analytics, 10Discovery, 10Discovery-Analysis, 10Discovery-Search, 10Product-Analytics: UDF for language detection - https://phabricator.wikimedia.org/T182352 (10Neil_P._Quinn_WMF) p:05Normal>03Lowest [20:39:15] nuria joal we can put that on the ui tomorrow :) [20:39:29] fdans: we do not have pageview metrics per family [20:39:34] milimetric, mforns - Added you as reviewers of the patch above - Please feel free to ignore :) [20:39:48] fdans: nor unqiue devices yet [20:40:02] k [20:40:26] fdans: so we can put it on but we need to think how it would look a little bit [20:41:07] fdans: let's talk about it tomorrow. [20:42:57] 10Analytics-Kanban, 10Analytics-Wikistats: Per family metrics in wikistats UI (wikipedia, wikinews) - https://phabricator.wikimedia.org/T205663 (10Nuria) p:05Triage>03High [20:43:06] Done for me tonight - See y'all tomorrow :P) [20:44:16] 10Analytics, 10Analytics-Wikistats: Active Editors metric per project family - https://phabricator.wikimedia.org/T188265 (10Nuria) [20:45:27] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats 2.0 Remaining reports. - https://phabricator.wikimedia.org/T186121 (10Nuria) [20:45:31] 10Analytics-Kanban, 10Analytics-Wikistats: Per family metrics in wikistats UI (wikipedia, wikinews) - https://phabricator.wikimedia.org/T205663 (10Nuria) 05Open>03Invalid [20:46:42] 10Analytics-Kanban: Per Family Unique Devices Counts - https://phabricator.wikimedia.org/T143927 (10Nuria) [20:46:46] 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: Wikistats 2.0: allow to view stats for all language versions (a.k.a. Project families) - https://phabricator.wikimedia.org/T188550 (10Nuria) [20:47:38] 10Analytics, 10Operations, 10Traffic: Artificial spike in offset of unique devices from November to February 6th on wikidata - https://phabricator.wikimedia.org/T165560 (10Nuria) [20:47:42] 10Analytics-Kanban, 10Patch-For-Review: Count project-wide unique devices (like *.wikipedia.org) - https://phabricator.wikimedia.org/T143928 (10Nuria) [20:53:09] 10Analytics, 10Analytics-Wikistats: Make wikistats UI family aware: you should be able to select a family in drop down menu and request available metrics for it - https://phabricator.wikimedia.org/T205665 (10Nuria) p:05Triage>03High [20:53:19] 10Analytics, 10Analytics-Wikistats: Make wikistats UI family aware: you should be able to select a family in drop down menu and request available metrics for it - https://phabricator.wikimedia.org/T205665 (10Nuria) a:03fdans [20:54:24] 10Analytics, 10Analytics-Wikistats: How about historic data - https://phabricator.wikimedia.org/T186124 (10Nuria) [20:54:40] 10Analytics, 10Analytics-Wikistats: Wikistats Beta Localization on the UI - https://phabricator.wikimedia.org/T186122 (10Nuria) [20:54:43] 10Analytics, 10Analytics-Wikistats: How about historic data - https://phabricator.wikimedia.org/T186124 (10Nuria) [20:54:57] 10Analytics, 10Analytics-Wikistats: Wikistats. How about historic data? - https://phabricator.wikimedia.org/T186124 (10Nuria) [20:56:02] 10Analytics-Kanban, 10Product-Analytics, 10Reading-analysis: Final Vetting of Family Wide unique devices data - https://phabricator.wikimedia.org/T169550 (10Nuria) ping @tbayer, do you think you could get to this task in the next month? [23:04:36] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Make wikistats UI family aware: you should be able to select a family in drop down menu and request available metrics for it - https://phabricator.wikimedia.org/T205665 (10Nuria) [23:51:28] 10Analytics, 10Pageviews-API: Pageview API: Better filtering of bot traffic on top enpoints - https://phabricator.wikimedia.org/T123442 (10Nuria) [23:51:33] 10Analytics-Kanban, 10Patch-For-Review: Eventlogging's processors stopped working - https://phabricator.wikimedia.org/T200630 (10Nuria)