[04:24:38] (03PS2) 10Mrcornacchio: Fix stylelint errors and TopicExplorer placeholder [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/426848 (https://phabricator.wikimedia.org/T185533) [04:38:11] (03PS3) 10Mrcornacchio: Fix stylelint errors and TopicExplorer placeholder [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/426848 (https://phabricator.wikimedia.org/T185533) [05:11:54] (03PS7) 10Nuria: SEO oriented changes for Wikistats2 Pages [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/427042 (https://phabricator.wikimedia.org/T182718) [05:12:13] (03CR) 10VolkerE: [C: 04-1] Fix stylelint errors and TopicExplorer placeholder (034 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/426848 (https://phabricator.wikimedia.org/T185533) (owner: 10Mrcornacchio) [05:28:09] (03PS4) 10Mrcornacchio: Fix stylelint errors and TopicExplorer placeholder [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/426848 (https://phabricator.wikimedia.org/T185533) [05:56:26] (03CR) 10VolkerE: [C: 032] "LGTM" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/426848 (https://phabricator.wikimedia.org/T185533) (owner: 10Mrcornacchio) [05:58:50] (03Merged) 10jenkins-bot: Fix stylelint errors and TopicExplorer placeholder [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/426848 (https://phabricator.wikimedia.org/T185533) (owner: 10Mrcornacchio) [07:45:51] 10Analytics, 10Patch-For-Review, 10User-Elukey: Update druid to latest release (0.11) - https://phabricator.wikimedia.org/T164008#4145200 (10elukey) >>! In T164008#4144424, @JAllemandou wrote: > First step of testing confirmed on labs with druid 0.9.2: > - Indexation from hadoop > - Realtime indexation wit... [10:08:26] 10Analytics, 10User-Elukey: Upgrade Druid nodes (1001->1006) to Debian Stretch - https://phabricator.wikimedia.org/T192636#4145484 (10elukey) [10:36:29] 10Analytics, 10User-Elukey: Upgrade Archiva (meitnerium) to Debian Stretch - https://phabricator.wikimedia.org/T192639#4145541 (10elukey) [10:44:12] * elukey lunch! [11:20:04] 10Analytics, 10User-Elukey: Reimage stat1004 with Debian Stretch - https://phabricator.wikimedia.org/T192640#4145579 (10elukey) [11:22:55] 10Analytics: Reimage thorium to Debian Stretch - https://phabricator.wikimedia.org/T192641#4145584 (10elukey) [11:23:21] 10Analytics: Upgrade Analytics infrastructure to Debian Stretch - https://phabricator.wikimedia.org/T192642#4145595 (10elukey) p:05Triage>03Normal [11:24:37] 10Analytics, 10User-Elukey: Upgrade Archiva (meitnerium) to Debian Stretch - https://phabricator.wikimedia.org/T192639#4145541 (10elukey) [11:24:39] 10Analytics, 10User-Elukey: Upgrade Druid nodes (1001->1006) to Debian Stretch - https://phabricator.wikimedia.org/T192636#4145484 (10elukey) [11:24:41] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Reimage the Debian Jessie Analytics worker nodes to Stretch. - https://phabricator.wikimedia.org/T192557#4142906 (10elukey) [11:24:43] 10Analytics: Upgrade Analytics infrastructure to Debian Stretch - https://phabricator.wikimedia.org/T192642#4145616 (10elukey) [11:24:45] 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Refresh zookeeper nodes in eqiad - https://phabricator.wikimedia.org/T182924#3838887 (10elukey) [11:24:53] oh yes, plenty of work for next quarter :D [11:32:37] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Reimage the Debian Jessie Analytics worker nodes to Stretch. - https://phabricator.wikimedia.org/T192557#4145647 (10elukey) [11:33:00] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Reimage the Debian Jessie Analytics worker nodes to Stretch. - https://phabricator.wikimedia.org/T192557#4142906 (10elukey) [11:33:19] !log reimage analytics1068 do Debian stretch [11:33:22] 41 to go.. [11:33:23] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:36:07] fdans: o/ - qq, did you manually upload the whitelist to hdfs this morning? [11:36:30] if so let's log it in https://tools.wmflabs.org/sal/analytics [11:36:47] (I saw this morning merge, so I suppose it didn't go out yesterday?) [11:37:36] oh yes sorry [11:37:51] !log manually uploaded refinery whitelist to hdfs [11:37:53] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:38:26] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Reimage the Debian Jessie Analytics worker nodes to Stretch. - https://phabricator.wikimedia.org/T192557#4145657 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts: ``` ['analytics1068.eqiad.wmnet'] ```... [11:38:30] fdans: :) [11:38:50] I am reimaing a worker node, so I might have caused some job failure [12:02:43] 10Analytics, 10CirrusSearch, 10Discovery, 10EventBus, and 5 others: Exception thrown while running DataSender::sendData in cluster codfw: Data should be a Document, a Script or an array containing Documents and/or Scripts - https://phabricator.wikimedia.org/T191024#4145711 (10dcausse) >>! In T191024#414568... [12:07:10] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Reimage the Debian Jessie Analytics worker nodes to Stretch. - https://phabricator.wikimedia.org/T192557#4145740 (10elukey) [12:07:52] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Reimage the Debian Jessie Analytics worker nodes to Stretch. - https://phabricator.wikimedia.org/T192557#4142906 (10elukey) [12:21:09] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Reimage the Debian Jessie Analytics worker nodes to Stretch. - https://phabricator.wikimedia.org/T192557#4145749 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['analytics1068.eqiad.wmnet'] ``` and were **ALL** successful. [12:23:13] an1068 back in production [12:59:18] brb [13:07:42] elukey: ok to install mysql security updates on bohrium/piwik? [13:18:09] moritzm: sure, best to disable piwik first [13:18:12] (doing it) [13:19:45] moritzm: you can proceed [13:22:17] uhm something is weird [13:22:24] apache2 seems not finding piwik? [13:24:21] so the webpage works, might be me not sending the POST correctly to verify maintenance [13:24:26] anyhow, should be ok now moritzm [13:24:31] k, upgrading [13:25:30] elukey: upgraded [13:26:42] super [13:57:36] ok so d-1.analytics updated to druid 0.10 [14:00:49] heyaaaaa [14:01:53] fdans: when you did the whitelist did you sync it up to hdfs or not? [14:02:13] I feel like there's a lot of confusion about this whitelist, everyone seems to do it differently and I'm not sure why [14:02:21] let's settle this once and for all after standup [14:02:26] early PS: pageview whitelist [14:02:58] milimetric: it was this morning when the alerts stopped [14:03:24] I think that he thought his patch was merged before yesterday's deploy [14:03:28] but it wasn't fully merged [14:04:01] hm, that's fine, but it doesn't matter, the procedure should always be the same regardless if there's a deploy: https://wikitech.wikimedia.org/wiki/Analytics/Team/Oncall#Find_and_fix_pageview_whitelist_exceptions [14:04:15] yep and I think everybody follows it [14:04:38] I don't think so, because I've personally synced that file every time we've gotten an alert in the last few months [14:04:43] to get the alerts to stop [14:05:27] which is wrong, I should've just told people, but I thought I did, and I thought everyone knew. Let's just make sure today [14:05:47] why is it wrong? [14:05:58] as long as it is logged in our sal, it is fine no? [14:06:09] then we wait for the next deploy to sync refinery [14:06:24] no I'm saying I was wrong to cover for people and do the sync myself [14:06:36] elukey: aha! :) Ok, so that's not what the procedure says [14:06:54] it says to sync it immediately after self-merging [14:08:28] no wait what I mean is [14:08:35] 1) manual sync of whitelist to hdfs [14:08:44] so alarms stop, and the !log in here [14:08:57] and/or reply to the email with the alert [14:09:27] 2) wait for the next deploy to see the change go in refinery as part of deploy etc.. [14:09:36] of course before 1) a code review would be nice :) [14:09:58] yeah, procedure doesn't say to log here, but maybe we should add that, but it's: [14:10:06] 1.) commit change to gerrit [14:10:08] 2.) merge [14:10:13] 3.) sync to hdfs [14:10:39] I'm saying that I've been doing 3.) for the last few months after people say they've fixed the whitelist and we keep getting emails [14:10:53] longer than last few months, for a long time [14:11:25] this is not a big deal, I don't mean to make it sound like it is [14:11:57] ahhhhh okok sorry [14:12:08] I didn't get the fact that you had to do this [14:12:26] no prob, that's what I was saying I was wrong to do that, I should've just had this chat earlier [14:12:33] but I always figured people were busy, etc. [14:12:39] and it's a minor thing [14:13:04] yeah but a reminder in the email saying "hello people, I just synced the whitelist, please whoever merges the CR next time do it" [14:13:20] would be good [14:13:39] it is not fair that you have to do it all the times [14:13:48] even if people are busy :) [14:17:29] !log d-[1,2,3] hosts in the analytics labs project upgraded to druid 0.10 [14:17:36] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:21:11] 10Analytics, 10Analytics-Wikistats: Improve scoping of CSS - https://phabricator.wikimedia.org/T190915#4145968 (10Milimetric) Sorry I missed this, I sort of wanted to tackle it myself but if @sahil505 finds it interesting, then great! Do add me to the review though, I'd like to take a look so I can understand... [14:22:37] 10Analytics, 10Analytics-Wikistats: Improve scoping of CSS - https://phabricator.wikimedia.org/T190915#4145969 (10Milimetric) That said, if you feel strongly in favor of a pre-processor, my favorite one is SASS, and I would support a refactor to move to that instead of this task - but that's entirely up to yo... [14:23:43] milimetric: I did, following the oncall document this morning [14:24:29] fdans: oh ok, so you merged it yesterday and thought it went out with the deploy, then saw the alarms and synced it, that makes sense [14:27:16] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Upgrading Wikistats 2.0 footer UI/design - https://phabricator.wikimedia.org/T191672#4145985 (10Milimetric) I'm not sure if we can change that copyright language, it was provided by legal and I'm sure there was a reason for how specific it is, "All data... [14:28:38] 10Analytics, 10Analytics-Wikistats: Routing code allows invalid routes - https://phabricator.wikimedia.org/T188792#4145999 (10Milimetric) (sorry for the priority, we thought it was bad but it's ok if it's not fixed right away) [14:30:50] 10Analytics, 10Analytics-Kanban, 10Wikipedia-iOS-App-Backlog: Decrease the request from iOS app to bohrium - https://phabricator.wikimedia.org/T190566#4146020 (10Milimetric) 05Open>03Resolved a:03Milimetric Since this is being worked on, I'm resolving it from our end, and we'll re-open if there's any p... [14:34:04] 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: Display of radio buttons in Wikistats 2 is somewhat confusing - https://phabricator.wikimedia.org/T183185#4146043 (10Milimetric) Thanks for the blue/green fix, I think the main problem is that these look like toggle buttons while people expect radio but... [14:40:12] strange... the virtualpageview aggregation oozie coordinator did only trigger until end of march, even if the source data is available for april... [14:42:50] mforns: that is super weird, it ran 3 hours out of April too: https://hue.wikimedia.org/oozie/list_oozie_coordinator/0026169-180330093100664-oozie-oozi-C/ [14:43:00] and it says the next materialized thing correctly [14:43:04] 04:00 [14:43:05] ? [14:43:20] like, everything looks ok but it's glitching out or something [14:43:21] oh! [14:43:24] joal: thoughts? [14:43:45] but it's still running then, because when I looked like 30 mins ago, it had not april hours [14:44:07] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Upgrading Wikistats 2.0 footer UI/design - https://phabricator.wikimedia.org/T191672#4146069 (10sahil505) @Milimetric : Thanks for the info. Will keep the language of the copyright same in the upcoming patch. [14:44:24] tha only issue is then, that it's not showing the future-to-run hours? [14:44:24] mforns: hm, maybe cluster's busy and it wasn't able to get jobs or something [14:44:38] if it's busy, they wouldn't show up [14:44:45] really? [14:44:46] ok [14:45:07] yeah, they'd have to get queued as jobs, and if it can't get resources to even queue them, then they wouldn't show I think, but it's definitely weird [14:45:11] yea, it's writing data [14:45:20] ok, makes sense [14:46:01] but yarn says the cluster is quite idle... [14:46:08] I'm not basing this on any deep knowledge, just from seeing it sometimes not show up for a few moments, and then first it's orange and says what's missing, then turns orange with a - in the requrements, then runs [14:46:30] aha [14:47:06] maybe it rate-limited itself for some reason? maybe look at the file timestamps for the April hours to see if they're exactly 10 minutes or 1 hour apart or something [14:48:03] hm [14:48:53] Hey folks [14:48:59] back from travel! [14:49:07] hey joal :] [14:50:04] milimetric, the only special thing is that the hours are processed in pairs, so 2 hour directories have the same timestamp, I guess that's because I set parallelism=2 [14:50:27] just saw various things - d-[1,2,3] upgraded - I'll start testing once we're done with the VirtualPageview thing [14:50:41] elukey: --^ Many thanks for the upgrade [14:50:41] woo! new druids [14:50:47] in labs only ;) [14:50:59] thank you so much sensei Luca [14:51:34] milimetric, mforns - I need some time to grasp what's going on with VPv [14:51:45] milimetric, actually, yes, it seems it's executing the virtualpageview aggregation every 5 minutes exactly [14:51:58] interesting! [14:52:16] joal, the only weird thing is that non-materialized hours do not show up in hue [14:52:22] only when they are done [14:53:27] joal: I also added the prometheus monitoring, since I realized that the prometheus agent is not tested for 0.10! /o\ [14:53:40] milimetric, 5 minutes is probably oozie's clock [14:53:57] joal: I have also a good idea about how to upgrade druid analytics without stopping real time monitoring [14:54:03] buuut we'll see if it works [14:55:43] elukey: I'm not even sure my job works without upgrqding tranquility :) [14:56:54] 10Analytics, 10Patch-For-Review, 10User-Elukey: Update druid to latest release (0.11) - https://phabricator.wikimedia.org/T164008#4146093 (10elukey) Upgraded d[1-3] in labs to druid 0.10, adding manual hiera config as replacement for https://gerrit.wikimedia.org/r/#/c/355471. [14:57:56] joal: ah and the hadoop-workers have now 50G datanode partitions :) [14:58:08] err 60G sorrry [14:58:37] awesome elukey :) [14:59:41] mforns, milimetric: Materialization of jobs to run is regulated by throttle [14:59:54] oh cool, didn't know that [15:00:16] joal, aaaaand... what is that? xD [15:00:18] Throttle is the number of materialized jobs accepted concurrently [15:00:49] throttle is a parameter in coordinator.xml [15:00:55] aaah [15:01:25] 2 parameters define the speed of the flow of jobs: concurrentcy (speaks for itself, number of paral execution jobs) [15:01:30] so throttle=2 and concurrency=2 means non-materialized jobs will not show! [15:01:40] And throttle, number of jobs waiting for execution [15:02:15] ooooK [15:02:19] cool, makes sense [15:02:29] nope, throttle=2 and concurrency = 2 means you can have, at the same time, 2 jobs executing and 2 jobs waiting for execution [15:03:05] joal, but... then why can't we see those 2 jobs waiting for execution in hue? [15:03:17] I don't even see the ones that are being run [15:03:32] only the succeeded ones... [15:03:46] Now, if job execution is so fast that any waiting job is executed as it comes in, well, job moves straight from created to running, not waiting for an execution slot to be ready [15:04:28] hmm [15:06:22] 10Analytics, 10Analytics-Wikistats: Improve scoping of CSS - https://phabricator.wikimedia.org/T190915#4146109 (10sahil505) @Milimetric : Thanks for your views on this. I would be very interested to work on this task as a part of my GSoC project. @mforns has already added this as one of the subtasks for the GS... [15:07:08] mforns: I actually glanced an moment of execution stages (2 jobs as exepected), before they quickly moved to success [15:07:22] joal, ok! [15:07:31] mforns: And you were right about the 5 minutes polling [15:07:46] it's weird that they are so fast, checking right now that the data looks good [15:08:13] throttle / concurrency should actually be set differently on regular forward jobs than backfilling ones [15:08:32] Backfilling could use higher throttle for fast jobs, to speed up the backfilling process [15:09:07] 10Analytics, 10Reading List Service, 10Reading-Infrastructure-Team-Backlog, 10Wikipedia-Android-App-Backlog, and 2 others: Enable Reading List Syncing usage stats - https://phabricator.wikimedia.org/T191859#4146131 (10Nuria) [15:09:17] joal, aaaaaaaah [15:10:24] joal, should we restart the coordinator with higher throttle until it has catched up? at the speed it goes, it will take 1 hour per day = 20 hours [15:10:50] mforns: for long jobs (like webrequest load, or things like that), there is no point of having higher throttle (jobs wait for slots in any case from concurrency settings) [15:11:44] For very fast jobs as this one, high throttle means, every 5 minutes, a lot more jobs are instanciated, therefore possibly fillling up concurrency slots for the 5 minutes to the next materialization [15:12:14] mforns: As you with - I don't mind waiting but I can also understand you're eager for data :) [15:12:44] joal, no no, I'm good, I have already 3 weeks for testing druid loading :] [15:13:46] BTW joal, I created 3 druid loading jobs, one for hourly segments, one for daily segments, and one for monthly segments [15:13:53] but all of them are hourly granularity [15:14:10] because virtualpageviews (for now) are supposed to be 6000 smaller than pageview [15:14:38] *6000 times [15:15:02] does that make sense? [15:23:31] mforns: It does :) [15:23:41] k, thanks! [15:23:47] mforns: if data is small enough, do we need daily ? [15:24:11] Like, even if many segments, would monthly agregation do the thing? [15:24:27] ? [15:24:52] Go directly from hourly to monthly [15:25:14] oh, you mean skip daily segments [15:25:15] ? [15:25:26] yes [15:25:30] but with hourly query granularity all the time no? [15:25:46] correct, if it is what you think is best [15:26:21] It seems that in term of data size, we should be good in keeping hourly query granularity [15:26:24] fff, dunno [15:26:29] So, it's a matter of need [15:26:58] I don't think the daily job will hurt [15:27:24] one day virtualpageviews has up to 15M records [15:27:33] About segment granularity, if you already have hourly, daily and monthly, let's keep them - Even if more jobs, less pressure on druid coordinator [15:28:03] yea, already in gerrit, I'm in the process of testing them now [15:28:09] super [15:28:12] k [15:28:14] mforns: Another question [15:28:32] mforns: Do you reuse the already indexed data, or do you recompute from hive [15:28:35] ? [15:29:09] hmmmmmm, I forgot about that... I wonder if I copy-pasted from pageview hourly correctly or... [15:29:12] will check [15:29:50] mforns: I can't recall if pageviews is done this way [15:29:57] oh... [15:30:08] then probably it's wrong [15:30:17] will work on that now [15:34:42] mforns: let's discuss in stadup [15:34:50] ok [15:35:18] mforns: The only case I have found where we use reindexing in druid is for banner [15:35:29] ok, I remember [15:35:43] but that is better anyway no? [15:35:44] The reason for that is peventing scanning all webrequest for only a fraction of it [15:36:48] Now, from a perf perspective, if we reindex, we only use a single hafoop job (druid indexation one), instead of 2 (data prep + indexation) [15:37:09] But at least for pageviews or others, we do use the whole datasets [15:41:30] 10Analytics, 10EventBus, 10Wikimedia-Stream: Hits from private AbuseFilters aren't in the stream - https://phabricator.wikimedia.org/T175438#4146205 (10Milimetric) 05Open>03Invalid Sounds like the system is working as designed. [15:50:46] 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: Display of radio buttons in Wikistats 2 is somewhat confusing - https://phabricator.wikimedia.org/T183185#4146218 (10Amitjoki) @Millimetric what do you think of the Semantic's own Radio styling? I guess it would go well with the site's design. This is t... [16:04:08] 10Analytics-Kanban, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review: Record and aggregate page previews - https://phabricator.wikimedia.org/T186728#4146248 (10Nuria) [16:31:22] joal: openjdk-7 dropped from hadoop and druid nodes in labs [16:32:06] awesome [16:32:08] let me know in here if you want to follow up more, I am going to stay another half an hour for sure :) [16:32:25] elukey: error seems classpath related [16:33:18] elukey: Actually not true - related to available memory for peons [16:33:40] d-1:/var/log/druid/middlemanger.log [16:33:50] At about 15:57 today [16:33:55] elukey: --^ [16:34:46] ah yes there might be some things to tune, some daemons weren't starting due to high mem requirements and I tuned them down [16:35:54] ah not a good news, d-1 has 2G of ram (I added other two similar ones yesterday, so we don't have a lot) [16:39:12] elukey: I think maybe providing less direct memory for the various processes; given how much we have for real, could make sense [16:39:26] yeah, I am looking into it [16:39:41] I always forget how MaxDirectMemorySize differs from the heap size [16:40:06] so we don't set it by default [16:40:11] and I think it is ~64M [16:40:39] druid.processing.buffer.sizeBytes[64,000,000] * (druid.processing.numMergeBuffers[2] + druid.processing.numThreads[1] + 1) [16:40:40] hm - 64M seems so small [16:40:51] I'll try to lower down sizeBytes a bit [16:41:27] k [16:46:34] done joal, decreased to 30M [16:46:39] let's see if it works [16:49:33] yes! [16:49:36] checking now [16:51:36] (03CR) 10Sturmkrahe: Fix accessibility/markup issues (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/425925 (https://phabricator.wikimedia.org/T185533) (owner: 10Sturmkrahe) [16:52:20] elukey: waiting for indexation to finish - This looks like a success [16:53:45] elukey: I confirm - Success! [16:53:52] elukey: now testing realtime [16:54:08] 10Analytics, 10Analytics-Wikistats: Improve scoping of CSS - https://phabricator.wikimedia.org/T190915#4146448 (10Milimetric) Ok, we talked about SASS and we think it's probably better to keep it simple for now. So CSS variables and PostCSS for now is our preference. And by the way, nice to meet you and do p... [16:57:32] elukey: confirmation again :) [16:57:52] elukey: I +1 a rollout of druid upgrade early next week ! [17:05:35] joal: wooooooooooowwww \o/ [17:05:42] indeed elukey :) [17:05:42] 10Analytics, 10Patch-For-Review, 10User-Elukey: Update druid to latest release (0.11) - https://phabricator.wikimedia.org/T164008#4146466 (10JAllemandou) After memory tricks from @elukey , both hadoop indexation and realtime indexation went fine (without any change - Incredible). Let's plan on an update next... [17:05:50] elukey: --^ Just commented [17:05:57] joal: so on Monday let's quickly check monitoring + pivot [17:06:05] and then if you have time we can do it [17:06:21] works for me :) [17:06:26] super [17:06:29] thanks a lot!!! [17:06:44] going off now, talk with you on Monday :) [17:06:50] have a good weekend! [17:06:51] * elukey off! [17:06:56] Bye elukey - Thank YOU 1 [17:15:44] 10Analytics, 10Analytics-Kanban: Update user_history and page_history column naming convention - https://phabricator.wikimedia.org/T188669#4146486 (10JAllemandou) a:03JAllemandou [17:31:39] 10Analytics, 10Analytics-Kanban: Update user_history and page_history column naming convention - https://phabricator.wikimedia.org/T188669#4146521 (10JAllemandou) Due to https://gerrit.wikimedia.org/r/#/c/388265/ not having been reflected on `wmf.mediawiki_user_history` and `wmf.mediawiki_page_history`, we ex... [17:31:46] milimetric: Hi ! [17:32:06] milimetric: if you have a minute, do you mind having a look at --^ ? [17:32:24] If you and nuria_ +1 the solution, I do it now [17:36:41] 10Analytics, 10Analytics-Kanban: Update user_history and page_history column naming convention - https://phabricator.wikimedia.org/T188669#4146527 (10Milimetric) @JAllemandou thanks for following up on this, the solution looks good to me. The only thing is then we won't have any private partitions, but people... [17:38:21] Thanks for the quick answer milimetric [17:43:43] nuria_: ping as well on https://phabricator.wikimedia.org/T188669#4146521 [17:45:28] joal: looking [17:50:36] joal: ok, i think i get it but aren't we missing a changeset in hive that either has the alter tables to change column names or rather a new create table with teh right column names? [17:50:48] joal: what hql are we recreating tables from? [17:53:22] nuria_: I didn't link the patch in comment;, but hive creation table script had been updated: https://gerrit.wikimedia.org/r/#/c/388266/ [17:56:25] joal: ok, i see. You would need to repair tables per partition right? [17:56:44] correct [17:56:57] not per partition, repair does it by itslef [17:57:14] joal: ah i see [17:57:57] joal: ok, plan looks good, if you are going to do it now will love to shadow on batcave [17:59:30] nuria_: on my way :) [18:00:42] joal: your internet kaput? [18:23:31] !log Drop/recreate wmf.mediawiki_user_history andwmf.mediawiki_page_history for T188669 [18:23:34] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:23:34] T188669: Update user_history and page_history column naming convention - https://phabricator.wikimedia.org/T188669 [18:23:48] 10Analytics, 10Analytics-Kanban: Update user_history and page_history column naming convention - https://phabricator.wikimedia.org/T188669#4146707 (10JAllemandou) Done. [18:56:44] 10Analytics-Kanban, 10Analytics-Wikistats: Make mediawiki-history-reduced table permanent (snapshot partitioning) - https://phabricator.wikimedia.org/T192482#4140406 (10JAllemandou) [18:57:07] (03PS1) 10Joal: Make mediawiki-history-reduced data permanent [analytics/refinery] - 10https://gerrit.wikimedia.org/r/427948 (https://phabricator.wikimedia.org/T192482) [18:57:45] (03PS2) 10Joal: Make mediawiki-history-reduced data permanent [analytics/refinery] - 10https://gerrit.wikimedia.org/r/427948 (https://phabricator.wikimedia.org/T192482) [19:07:21] milimetric: w/o a computed and w/o watching state what is what i am watching to change title? [19:07:29] milimetric: i need to watch state some way right? [19:08:15] nuria_: watching is ok, I'll sketch it up, one sec [19:08:38] milimetric: in what way is different from this then? [19:08:39] https://gerrit.wikimedia.org/r/#/c/427042/1..4/src/mixins/title-mixin.js [19:09:13] ^ milimetric [19:09:29] https://www.irccloud.com/pastebin/g8toIJVy/ [19:10:22] nuria_: the main difference is mapping the $store into a computed and using that instead of the store directly, which is how vuex guidelines say to do it, and also no need for the method [19:10:28] does my sketch make sense? [19:10:29] milimetric ah i see, the computed are still there [19:10:39] the mapped ones, yes [19:10:48] but I think in this case you don't need any other computed [19:10:58] milimetric: yaya, ok. will redo and submit [19:11:29] nuria_: and one other reason to do it this way is that you may in the future need to watch something more complicated than a plain title, it may need to mix other parts of the state, like the bookmarks [19:11:53] in which case you can have a computedTitle made up of the sitematrixTitle + $store.state.bookmarks or something like that [19:12:06] and this sets you up to change that data flow in an easy and consistent way [19:12:37] and then instead of watching title to update document.title, you'd just watch computedTitle [19:15:51] (03PS1) 10Framawiki: Implement browser notifications [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/427952 (https://phabricator.wikimedia.org/T124625) [19:18:21] (03PS2) 10Framawiki: Implement browser notifications [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/427952 (https://phabricator.wikimedia.org/T124625) [19:29:10] (03CR) 10Zhuyifei1999: Implement browser notifications (036 comments) [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/427952 (https://phabricator.wikimedia.org/T124625) (owner: 10Framawiki) [19:31:48] 10Quarry, 10Patch-For-Review: Show desktop notification when a query is done - https://phabricator.wikimedia.org/T124625#4146891 (10Framawiki) a:03Framawiki [19:36:46] (03PS4) 10Zhuyifei1999: Update username in database when a renamed user logins [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/398711 (https://phabricator.wikimedia.org/T73064) [19:45:44] 10Analytics, 10Analytics-EventLogging, 10Performance: Spin out a tiny EventLogging RL module for lightweight logging - https://phabricator.wikimedia.org/T187207#4146907 (10AndyRussG) Just to note, another use case is [[ https://gerrit.wikimedia.org/r/#/c/423952/ | this recent addition ]] to the FundraiserLan... [19:53:39] 10Analytics, 10Analytics-EventLogging, 10Performance: Spin out a tiny EventLogging RL module for lightweight logging - https://phabricator.wikimedia.org/T187207#4146915 (10Krinkle) The "mini module" is essentially [ext.eventLogging.subscriber](https://github.com/wikimedia/mediawiki-extensions-EventLogging/bl... [19:54:01] 10Analytics, 10Analytics-EventLogging, 10Performance-Team: Spin out a tiny EventLogging RL module for lightweight logging - https://phabricator.wikimedia.org/T187207#4146916 (10Krinkle) [20:01:23] 10Analytics, 10Analytics-EventLogging, 10Performance-Team: Spin out a tiny EventLogging RL module for lightweight logging - https://phabricator.wikimedia.org/T187207#4146930 (10Nuria) >The "mini module" is essentially ext.eventLogging.subscriber. Currently, it has no dependencies besides mediawiki.user from... [20:20:10] (03PS8) 10Nuria: SEO oriented changes for Wikistats2 Pages [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/427042 (https://phabricator.wikimedia.org/T182718) [20:21:06] milimetric: do check and let me know if this is what you were thinking: https://gerrit.wikimedia.org/r/#/c/427042/ [20:46:26] 10Analytics, 10Reading List Service, 10Reading-Infrastructure-Team-Backlog, 10Wikipedia-Android-App-Backlog, and 2 others: Enable Reading List Syncing usage stats - https://phabricator.wikimedia.org/T191859#4118961 (10Nuria) Note retention of this data is subjected to 90 days unless granted otherwise by le... [21:15:50] (03CR) 10Framawiki: "Can we add some kind of log to be able to verify that this patch works as excepted?" [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/398711 (https://phabricator.wikimedia.org/T73064) (owner: 10Zhuyifei1999) [21:28:54] 10Quarry: Add current git version somewhere in the interface - https://phabricator.wikimedia.org/T192506#4147078 (10Framawiki) Effectively, implementing this display would be difficult. `quarry-main-01` would have to get the versions of the two workers to be able to display it. Otherwise, probably silly question... [21:39:13] (03CR) 10Framawiki: Add export to HTML (032 comments) [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/427020 (https://phabricator.wikimedia.org/T117644) (owner: 10Framawiki) [21:52:05] (03CR) 10Framawiki: "recheck" [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/176506 (https://phabricator.wikimedia.org/T76084) (owner: 10Rtnpro) [21:52:20] (03CR) 10jerkins-bot: [V: 04-1] Remember recent queries filter last used by a user. [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/176506 (https://phabricator.wikimedia.org/T76084) (owner: 10Rtnpro) [22:11:54] 10Analytics, 10Reading List Service, 10Reading-Infrastructure-Team-Backlog, 10Wikipedia-Android-App-Backlog, and 2 others: Enable Reading List Syncing usage stats - https://phabricator.wikimedia.org/T191859#4147129 (10Nuria) [22:14:34] bearloga: please be so kind to look at my proposal of how to gather data for Android Reading lists here: https://phabricator.wikimedia.org/T191859 [22:37:02] 10Analytics, 10Reading List Service, 10Reading-Infrastructure-Team-Backlog, 10Wikipedia-Android-App-Backlog, and 2 others: Enable Reading List Syncing usage stats - https://phabricator.wikimedia.org/T191859#4147159 (10Nuria) [23:09:47] bearloga: i will add MORE WORDS cause it wi [23:09:56] it is a bit meager [23:24:37] 10Analytics, 10Reading List Service, 10Reading-Infrastructure-Team-Backlog, 10Wikipedia-Android-App-Backlog, and 2 others: Enable Reading List Syncing usage stats - https://phabricator.wikimedia.org/T191859#4147202 (10Nuria) [23:46:30] 10Quarry: Add current git version somewhere in the interface - https://phabricator.wikimedia.org/T192506#4147236 (10zhuyifei1999) >>! In T192506#4147078, @Framawiki wrote: > -Is it necessary to have different versions between the three machines? No, it's usually not necessary except for testing purposes. > Has t... [23:49:41] (03CR) 10Zhuyifei1999: "> add some kind of log" [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/398711 (https://phabricator.wikimedia.org/T73064) (owner: 10Zhuyifei1999) [23:54:31] (03CR) 10Zhuyifei1999: Add export to HTML (033 comments) [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/427020 (https://phabricator.wikimedia.org/T117644) (owner: 10Framawiki)