[06:20:47] Analytics-EventLogging, DBA, ImageMetrics: Drop EventLogging tables for ImageMetricsLoadingTime and ImageMetricsCorsSupport - https://phabricator.wikimedia.org/T141407#2627512 (Marostegui) Don't know if this throws more light into the issue, but the only table recreated is that one. I just checked a... [07:45:53] joal: o/ [07:50:38] Analytics-Kanban: Setup regular loading jobs new aqs cluster (per-article, top and unique devices) - https://phabricator.wikimedia.org/T145087#2627608 (JAllemandou) a:JAllemandou [07:52:58] Analytics-Kanban: Load top article data into new AQS cluster - https://phabricator.wikimedia.org/T145089#2627611 (JAllemandou) Full recomputation was actually fastest we could get. I started full backfilling job last Friday that will finish either late tonight or early tomorrow. [07:53:14] Analytics-Kanban: Load top article data into new AQS cluster - https://phabricator.wikimedia.org/T145089#2627612 (JAllemandou) a:Nuria>JAllemandou [08:11:34] Analytics-Kanban, Patch-For-Review: Continue New AQS Loading - https://phabricator.wikimedia.org/T140866#2627662 (JAllemandou) Data all loade for all endpoints except daily-top, currently finishing. [08:38:59] (CR) Joal: [C: 1] "One nit in commit message but code looks good." (1 comment) [analytics/aqs] (new-aqs-cluster) - https://gerrit.wikimedia.org/r/309602 (https://phabricator.wikimedia.org/T140866) (owner: Nuria) [08:44:01] (CR) Joal: [C: 1] "One nit in commit message, but code looks good." (1 comment) [analytics/aqs] (new-aqs-cluster) - https://gerrit.wikimedia.org/r/309604 (https://phabricator.wikimedia.org/T144521) (owner: Nuria) [08:48:09] adi elukey [08:56:38] elukey: What's up? [09:00:09] nothing, just wanted to say hello [09:00:10] :) [09:00:53] but since you are here.. is yarn.w.o working for you now? [09:00:57] or still seeing issue? [09:01:01] *issues [09:17:53] there still is one link column that is not translated on the scheduler page I think :) [09:18:08] elukey: --^ [09:18:38] elukey: Our new cluster is almost fully loaded (all endpoints) [09:19:03] already?? [09:19:04] \o [09:19:07] \o/ [09:19:27] joal: can you give me the link? [09:20:21] elukey: last column to the right in scheduler page (the Application Master link - uses analytics1001 instead of yarn° [09:20:38] elukey: as you say: \o/ [09:20:55] elukey: I wouldn't have expected top to go fast enough, and in fact it worked [09:21:43] elukey: compaction of the last month nuria_ loaded is almost finished as well, I think we'll be able to do some tests later this week :) [09:22:32] \o/ [09:22:39] * elukey hates the yarn UI [09:25:16] elukey: I know !! [09:25:18] joal: I got why we see the the application master link wrong only in there [09:25:24] it is inside a script [09:25:26] as string [09:25:30] Ahhhhhhh ! [09:25:33] and I filter only a hrefs [09:25:39] and not scripts [09:25:43] for performances [09:25:47] elukey: makes sense ! [09:48:51] !log restarted pivot on a tmux session on stat1002 since it died [09:48:53] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [10:01:07] hi team! [10:06:39] mforns: o/ [10:06:45] hello elukey :] [10:38:49] * elukey lunch! [11:38:47] joal: https://twitter.com/mfiguiere/status/773979084514729984 [11:38:55] "Cassandra keeps growing at Apple, now 115,000+ nodes in production" [11:42:03] Analytics, Dumps-Generation: Pageview dumps incorrectly formatted, looks like a result of possibly malicious activity - https://phabricator.wikimedia.org/T144100#2628133 (ArielGlenn) [12:14:07] Analytics-Tech-community-metrics, Developer-Relations (Jul-Sep-2016), Documentation: Create basic/high-level Kibana (dashboard) documentation - https://phabricator.wikimedia.org/T132323#2628225 (Aklapper) p:High>Normal [12:29:27] elukey: I had seen that yes :) [12:30:24] hey mforns ! [12:30:56] I'm sorry for not having been able to help on Friday :( [12:31:02] mforns: --^ [12:31:33] mforns: how is the denormalization code reading? I assume it's not easy and not fun :S [12:50:17] I know that I am always slow in finding things [12:50:36] but I just played a bit with pivot and pageview data understanding what I was doing [12:50:39] :D [13:01:00] elukey: it's great you're doing that, whatever timing ! [13:02:10] joal, hi! [13:02:14] was having lunch [13:02:21] np mforns :) [13:02:30] don't worry, health first [13:02:48] I'm looking at the code right now, because I found inconsistencies in the results [13:03:23] I executed the alg after your last changes, but still has the same problems [13:03:48] and I think the proble is in the joining of revisions with states [13:05:42] mforns: ok [13:06:09] mforns: today I'm travelling, so will miss standup and other meetings but will work later tonight [13:06:16] joal, ok [13:06:35] mforns: If it's ok with your timing; we can spend a couple hours on this this evening? [13:06:38] joal, if you want, ping me and we'll look at it in the batcave [13:06:48] mforns: sounds good :) [13:07:01] joal, sure, I'll finish at 21h today, though. is it ok for you? [13:07:04] Thanks for proofreading this [13:07:36] np [13:08:18] mforns: I'll probably be online around 20:30 if everythiong goes well [13:08:24] joal, ok [13:08:43] mforns: we can go on until 21; and then maybe tomorrow ? [13:08:51] joal, sure :] [13:09:08] awesome [13:09:55] a-team, Will move to the airport in 15 minutes, back online around 20:30 [13:10:15] joal, good travelling! [13:10:48] a-team, this weekend I backfilled new aqs almost completely (only top not yet done) [13:10:52] thanks mforns ! [13:11:12] backfilling: awesome! [13:15:04] joal: you rock :) have a good trip! [13:20:31] HlellOOoo joal wher you goin!? [13:20:36] camus today? [13:29:31] HELLO! [13:35:09] hlloo [13:44:45] (PS1) Addshore: Modify access rules [analytics/wmde] (refs/meta/config) - https://gerrit.wikimedia.org/r/309991 [13:51:39] (CR) Hashar: [C: 2 V: 2] Modify access rules [analytics/wmde] (refs/meta/config) - https://gerrit.wikimedia.org/r/309991 (owner: Addshore) [13:55:18] Analytics, GLAM-Tech, Pageviews-API: WMF pageview API (404 error) when requesting statitsics over around 1000 files on GLAMorgan - https://phabricator.wikimedia.org/T145197#2628567 (Sadads) @Musikanimal do you have a sense if there is a limit on the API or the computing for this, that would be prohib... [14:47:12] ottomata: I am thinking to try service-runner on stat1001 for pivot, wdyt? [14:47:26] elukey: y not? :) [14:47:35] can't hurt, might not be necessary, but it can only help so sure [14:47:46] i don't yet have a lot of experience with serivce runner [14:47:48] but mobrovac does :) [14:48:14] it seems easy enough from what I've read in puppet [14:48:21] and we could use only one worker [14:48:31] it is integrated with scap [14:48:37] so it looks really nice [14:48:54] but maybe a simple systemd unit could be enough for our use case? [14:51:12] elukey: service-runner might not be necessary, but if it makes things easier anyway, it is probably a good idea to use it [14:53:02] elukey: i can assist if you need me (w/ or w/o service-runner) [14:57:37] mobrovac: thanks! I am basically wrapping the nodejs service that powers the pivot.w.o UI (I already asked you some info and you pointed me to your OCG replacement patch) [14:57:54] wrapping == writing puppet code [14:58:19] so in your example a systemd unit was enough, but even service-runner seems really nice to use [14:59:05] service-runner and service::node are really nice abstractions so if you can, i encourage you to use it [15:01:04] hue question if anyone knows ... i accidently started a second copy of popularity_score-coord (under analytics-search user). I killed it, but now the orriginal popularity_score-coord isn't being listed on hue's coordinators page. I still have a direct link to the coordinator and it claims to be running (https://hue.wikimedia.org/oozie/list_oozie_coordinator/0000095-160420145651441-oozie-oozi-C/) but i worry because its not listed o [15:01:22] should i worry? or is it probably all just fine [15:01:34] i could probably kill and restart this as well to ensure everything still works righ [15:02:10] a-team: standdupppppp [15:02:31] EEK [15:21:31] elukey: \o [15:21:59] urandom: o/ [15:22:43] elukey: So, https://gerrit.wikimedia.org/r/#/c/282466 [15:23:19] elukey: at some point we decided to filter out StatusLogger messages from the stream that goes to logstash, because it was *really* chatty [15:23:35] it was consuming a ton of space in elasticsearch [15:23:41] and wasn't useful there [15:24:50] elukey: so added this bit: https://github.com/wikimedia/operations-puppet/blob/production/modules/cassandra/templates/logback.xml-2.2.erb#L93-L99 [15:25:06] which required some jars and stuff [15:25:34] ahhh and now you want to use additivity [15:25:42] TL;DR it filtered out StatusLogger [15:26:07] yes yes now it makes more sense [15:26:10] yeah, someone with greater logback-fu than me, realized we could do this without the filter (and corresponding dependencies) [15:26:26] and submitted this gerrit (it's not my gerrit) [15:26:49] and then shamelessly i let it languish for far too long before reviewing/testing [15:26:57] so that at this point it's become ancient history :) [15:27:39] but it should result in no actual change in behavior, it's just a better implementation of what is there [15:27:57] i tested it in our deployment-prep environment, and it seemed OK [15:29:27] okok it LGTM [15:29:52] maybe we could run it with puppet disable on restbase/maps/aqs for the first nodes [15:29:55] and then re-enable [15:37:33] Analytics, GLAM-Tech, Pageviews-API: WMF pageview API (404 error) when requesting statitsics over around 1000 files on GLAMorgan - https://phabricator.wikimedia.org/T145197#2623016 (Nuria) @Sadads: I think you need to work on this with @MusikAnimal regarding "number of requests" that the tool can... [15:45:00] Analytics, ChangeProp, EventBus, Wikimedia-Stream: Write node-rdkafka event.stats callback that reports stats to statsd - https://phabricator.wikimedia.org/T145099#2619979 (Nuria) This can benefit from code already existing on varnishkafka (for inspiration) that parses a flat json and sends it b... [15:55:22] Analytics, Dumps-Generation, Security: Pageview dumps incorrectly formatted, looks like a result of possibly malicious activity - https://phabricator.wikimedia.org/T144100#2629089 (Nuria) [15:57:08] a-team, sorry, both of the wifis here are not working :/ [15:57:21] Analytics, Analytics-Kanban, Pageviews-API: Special characters showing up as question marks in /pageviews/top endpoint - https://phabricator.wikimedia.org/T145043#2629103 (Nuria) [16:02:27] Analytics: Top Pageview stats for August 27th doesn't look right - https://phabricator.wikimedia.org/T144715#2608052 (Nuria) [16:02:29] Analytics, Pageviews-API: Pageview API: Better filtering of bot traffic on top enpoints - https://phabricator.wikimedia.org/T123442#2629120 (Nuria) [16:07:14] (PS4) Nuria: Update per-article compression scheme to default (LCS) [analytics/aqs] (new-aqs-cluster) - https://gerrit.wikimedia.org/r/309602 (https://phabricator.wikimedia.org/T140866) [16:07:45] (CR) Nuria: Update per-article compression scheme to default (LCS) (1 comment) [analytics/aqs] (new-aqs-cluster) - https://gerrit.wikimedia.org/r/309602 (https://phabricator.wikimedia.org/T140866) (owner: Nuria) [16:10:56] (PS3) Nuria: Map null count values to 0 in per-article output [analytics/aqs] (new-aqs-cluster) - https://gerrit.wikimedia.org/r/309604 (https://phabricator.wikimedia.org/T144521) [16:11:58] (Abandoned) Nuria: Map null count values to zeros in output [analytics/aqs] - https://gerrit.wikimedia.org/r/309386 (https://phabricator.wikimedia.org/T144521) (owner: Nuria) [16:12:08] (CR) Nuria: Map null count values to 0 in per-article output (1 comment) [analytics/aqs] (new-aqs-cluster) - https://gerrit.wikimedia.org/r/309604 (https://phabricator.wikimedia.org/T144521) (owner: Nuria) [16:13:29] Analytics, Pageviews-API: Pageview API: Better filtering of bot traffic on top enpoints - https://phabricator.wikimedia.org/T123442#2629178 (Nuria) We deprioritized this task earlier on on Q1, moving back to Q2 just in case we want to take a 2md look. [16:25:19] elukey: we could [16:25:35] elukey: we could disable everywhere but restbase staging if you like [16:27:52] sure.. tomorrow this time maybe? [16:28:35] elukey: works for me [16:29:04] ebernhardson: did you get your question answered? [16:33:55] nuria_: no, but since oozie seems to have all the right info, i'm going to guess hue is just incorrec [16:37:33] ebernhardson: right, that seems to be the case [16:46:02] mforns: can we look at reportupdater tests for a sec? [16:46:07] nuria_, sure [16:46:14] mforns: batcave? [16:46:15] joining the batcave [16:46:17] :] [17:02:03] * elukey afk! [17:02:06] o/ [17:26:09] Analytics-Tech-community-metrics: Missing time units for percentile values - https://phabricator.wikimedia.org/T145425#2629643 (Aklapper) [17:27:16] Analytics-Tech-community-metrics: Missing time units for percentile values - https://phabricator.wikimedia.org/T145425#2629643 (Aklapper) [17:27:49] Analytics-Tech-community-metrics: Missing time units for percentile values - https://phabricator.wikimedia.org/T145425#2629643 (Aklapper) p:Triage>Normal [18:22:59] hey ottomata ! [18:23:15] sorry didn't catch your ping before I left [18:23:23] ottomata: camus for sure ! [18:23:38] ottomata: after some talk I have planned to have with mforns ? [18:24:55] mforns: Hi ! [18:28:01] joal: sure thing whenever you are ready [18:29:24] ottomata: Given he doesn't answer as of now, we can go for it I guess [18:30:03] ottomata: I travelled to Paris, preparation for future classes I'll give :D [18:31:22] ottomata: Meeting tomorrow, diner with familly, then back home in two days [18:34:19] ottomata: IIRC what we need to do is: stop camus, wait for job to end if any, deploy/apply puppet patch, then restart camus and monitor, right? [18:38:03] joal, hi! [18:38:31] joal: ah sorry, yeah let's do it [18:38:39] oh mforns is back now too! [18:38:48] joal: awesome classes! cool [18:38:56] joal: i'll go ahead and stop camus [18:39:02] and then we can wait and merge puppet [18:39:12] i think as long as you are around to make sure the next run or two works thats all we need ja? [18:39:19] joal, ottomata whatever you prefer, you can meet first [18:39:35] we don't need to meet [18:39:36] :) [18:39:46] just want joal around for next camus run [18:39:48] ok [18:40:21] ottomata: great [18:40:27] mforns: batcave? [18:40:34] joal, sure! omw [18:41:53] !log disabled camus crons on analytics1027 [18:41:55] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [18:43:36] Analytics, Discovery-Analysis: [REQUEST] Extract search queries from HTTP_REFERER field for a Wikibook - https://phabricator.wikimedia.org/T144714#2629977 (Larsnooden) Google Search Console (webmaster tools) might be useful. Which Wikimedia group would that fall under? About the query part of the URI b... [18:49:19] joal: no camus jobs running [18:49:21] merging pppet [18:49:36] Analytics, Discovery-Analysis: [REQUEST] Extract search queries from HTTP_REFERER field for a Wikibook - https://phabricator.wikimedia.org/T144714#2608040 (Nuria) >Google Search Console (webmaster tools) might be useful. Which Wikimedia group would that fall under? Again, due to our privacy policy we do... [18:52:30] yargh a webrequest camus got launcehd before puppet could run [18:52:36] s'ok though, doesn't really matter until the next run [18:52:41] i guess we didin't really have to disable it for this [18:54:43] ok, watching logs, waiting for next run [18:54:55] !log reenabled camus with new version of camus checker jar [18:54:57] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [18:55:08] Analytics, Discovery-Analysis: [REQUEST] Extract search queries from HTTP_REFERER field for a Wikibook - https://phabricator.wikimedia.org/T144714#2630044 (Tbayer) >>! In T144714#2630030, @Nuria wrote: >>Google Search Console (webmaster tools) might be useful. Which Wikimedia group would that fall under?... [18:58:43] mforns: yt? [18:58:53] nuria_, yes, in meeting with joseph [18:59:08] what's up, do you want to join the batcave? [18:59:11] nuria_, ^ [18:59:13] nuria_: Hi, august data prep running for cassandra [18:59:39] joal: ah! [18:59:47] joal: what does it need to happen for dataprep? [18:59:56] nuria_: manual launch ! [19:00:00] mforns: irc is fine i can ask you once you are done talking to joal [19:00:04] backfill is supposedly one time [19:00:07] ok nuria_ [19:00:09] nuria_: --^ [19:00:14] joal: aham [19:01:21] joal: you can explain maybe later when you guys are done in batcave, can be tomorrow [19:01:38] nuria_: sure [19:01:47] joal, mforns : just let me know [19:01:52] ok nuria_ [19:02:46] Analytics, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown, Patch-For-Review: Implement Schema:ExternalLinksChange - https://phabricator.wikimedia.org/T115119#2630085 (Samwalton9) Any progress yet? We're so close, and would really like to move on to c... [19:19:25] hey ottomata, all good on camus? [19:41:28] Analytics, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown, Patch-For-Review: Implement Schema:ExternalLinksChange - https://phabricator.wikimedia.org/T115119#1808146 (Nuria) @Samwalton9: data is being collected. I think you need to follow up with CR... [19:45:07] joal: thanks for reminding me to check! [19:45:09] looks good from here [19:45:14] logs look normal [19:46:18] ottomata: for me as well, will wait for next flags to be created (15mins), and call it done :) [19:46:28] ottomata: Many thanks for the deploy :) [19:50:15] nuria_: hi [19:50:24] nuria_: ready [19:50:25] hello! [19:50:48] joal: My question was just what do we need to prepare to do the loading? [19:51:03] nuria_: batcave? [19:51:09] a-team, leaving for today, bye! [19:51:13] be mforns ! [19:51:25] mforns: Thanks for the very productive session ! [19:51:32] thank you too! [19:51:40] nuria_: or here, I don't mind :) [19:52:29] nuria_: data prep for the loading is generating data from hive [19:52:39] joal: batcave is fine, omw [20:04:22] ottomata: confirmed! Everything looks good :) [20:05:32] great! [20:05:33] thanks joal! [20:05:49] np ottomata, thanks as well [20:16:48] (PS3) Nuria: Add re-run script [analytics/reportupdater] - https://gerrit.wikimedia.org/r/308977 (https://phabricator.wikimedia.org/T117538) (owner: Mforns) [22:12:24] (PS1) Catrope: Update set of wikis that have the Flow beta feature [analytics/limn-flow-data] - https://gerrit.wikimedia.org/r/310151 (https://phabricator.wikimedia.org/T144515) [22:37:20] Quarry, Discovery, Labs-project-other, Wikidata, and 2 others: Setup sparqly service at https://sparqly.wmflabs.org/ (like Quarry) - https://phabricator.wikimedia.org/T104762#2631095 (Smalyshev) p:Triage>Low [22:38:32] (CR) Catrope: [C: 2] Update set of wikis that have the Flow beta feature [analytics/limn-flow-data] - https://gerrit.wikimedia.org/r/310151 (https://phabricator.wikimedia.org/T144515) (owner: Catrope) [22:39:10] (CR) Catrope: [V: 2] Update set of wikis that have the Flow beta feature [analytics/limn-flow-data] - https://gerrit.wikimedia.org/r/310151 (https://phabricator.wikimedia.org/T144515) (owner: Catrope)