[00:16:59] 10Analytics, 10EventBus, 10Core Platform Team Kanban (Current), 10Services (doing): Revision visibility change event sets a wrong performer - https://phabricator.wikimedia.org/T206277 (10Pchelolo) [01:08:59] 10Analytics, 10Analytics-Cluster, 10Contributors-Analysis, 10Product-Analytics: Hive join fails when using a HiveServer2 client - https://phabricator.wikimedia.org/T206279 (10Neil_P._Quinn_WMF) [01:18:41] nuria: is there an easy/safe-for-presentation way I can visualize event stream in a slide? I want to leave it in the background as I speak so people see how quickly events come in. [01:19:11] nuria: I meant recent changes stream. sorry. [01:47:48] leila: https://wikitech.wikimedia.org/wiki/EventStreams [01:47:55] one of the codepen.io examples there? [02:24:44] ottomata:There is also: https://wikimediablog.files.wordpress.com/2017/03/eventstreamsglobe.gif?w=480&h=515&zoom=2 [03:07:14] 10Analytics, 10Product-Analytics: Metrics request on portal namespace usage - https://phabricator.wikimedia.org/T205681 (10JKatzWMF) @Afrothundr3007730 Hi! I don't think we've crossed paths yet, but you are clearly very active on both the wikis and phab. I'm a director of product at the foundation and the in... [07:35:11] 10Analytics, 10Analytics-Cluster, 10Contributors-Analysis, 10Product-Analytics: Hive join fails when using a HiveServer2 client - https://phabricator.wikimedia.org/T206279 (10JAllemandou) Some more info on the issue as I understand it: Given the data size to work and the fact that `hive.auto.convert.join`... [08:07:26] joal: o/ [08:07:36] morning elukey :) [08:07:43] did you see my email?? [08:07:52] * elukey now cheers for Big Top [08:08:19] I have, yes [08:08:42] elukey: I would have cheered for bigtop even if CDH6 had been debian-ready ;) [08:10:10] 10Analytics-Legal, 10WMF-Legal, 10Wikidata: Address concerns about perceived legal uncertainty of Wikidata - https://phabricator.wikimedia.org/T193728 (10Nemo_bis) [08:11:22] I am still following the 1.3 release process [08:11:36] in the emails I got reference of Debian 9 (Stretch) support [08:12:05] so Hive could be a good first test [08:12:30] in theory all our puppet code base for cdh is not that specific, and probably can be adapted for big top easily [08:12:47] I have some concerns for oozie, that doesn't seem to be in there [08:13:48] even if I can see https://github.com/apache/bigtop/tree/master/bigtop-packages/src/deb/oozie [08:13:51] mmmm [08:15:10] elukey: I think I read oozie was out of bigtop somewhere - Give me 1 sec [08:16:03] https://issues.apache.org/jira/browse/BIGTOP-2986 [08:16:20] this one looks promising [08:16:28] I think that they only dropped pig [08:16:32] that had an issue with Hive 2 [08:16:35] but not oozie [08:16:35] k [08:17:01] Arf - While it's good, it;s also bad - It would have pushed us to move to airflow :D [08:17:25] well we can decide to research airflow in Q3 [08:17:29] anyway :) [08:26:56] (03PS9) 10Joal: Add python script importing xml dumps onto hdfs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/456654 (https://phabricator.wikimedia.org/T202489) [08:27:26] git up [08:27:32] oops [08:28:01] (03PS10) 10Joal: Add python script importing xml dumps onto hdfs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/456654 (https://phabricator.wikimedia.org/T202489) [08:37:45] joal: git: 'up' is not a git command. See 'git --help'. [08:38:06] Hi ema :) [08:38:38] hey there :) [08:39:17] ema: There a few aliases I like, and I must say 'git up' makes me feel reaggae :) [08:39:39] hahaha [08:39:49] git up stind up [08:40:20] git up stin dup - I like that [10:17:25] !log set piwik/matomo in maintenance mode on matomo1001 [10:17:27] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:18:20] (03PS3) 10Fdans: Allow several attempts to get latest data in top metrics [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/464375 (https://phabricator.wikimedia.org/T205915) [10:19:56] (03CR) 10Fdans: [C: 032] "Comment removed, merging once CI is done" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/464375 (https://phabricator.wikimedia.org/T205915) (owner: 10Fdans) [10:24:32] (03Merged) 10jenkins-bot: Allow several attempts to get latest data in top metrics [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/464375 (https://phabricator.wikimedia.org/T205915) (owner: 10Fdans) [10:32:22] !log piwik/matomo out of maintenance [10:32:24] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:36:00] * elukey lunch! [10:39:29] (03PS5) 10Fdans: Allow breakdown filtering in top metrics [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/463964 (https://phabricator.wikimedia.org/T205725) [10:40:21] (03CR) 10Fdans: Allow breakdown filtering in top metrics (032 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/463964 (https://phabricator.wikimedia.org/T205725) (owner: 10Fdans) [10:57:14] 10Analytics-Kanban: Make area metrics collapsible - https://phabricator.wikimedia.org/T206311 (10fdans) [10:57:28] 10Analytics-Kanban, 10Analytics-Wikistats: Make area metrics collapsible - https://phabricator.wikimedia.org/T206311 (10fdans) [10:57:44] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Make area metrics collapsible - https://phabricator.wikimedia.org/T206311 (10fdans) [10:58:20] (03PS2) 10Fdans: Adds logic and configuration for project families [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/464583 (https://phabricator.wikimedia.org/T188550) [10:58:26] (03CR) 10jerkins-bot: [V: 04-1] Adds logic and configuration for project families [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/464583 (https://phabricator.wikimedia.org/T188550) (owner: 10Fdans) [11:33:04] (03PS1) 10Fdans: Allow whole metric areas to be collapsed [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/464800 (https://phabricator.wikimedia.org/T206311) [11:37:27] 10Analytics, 10Analytics-Kanban, 10Operations: Decommission bohrium - https://phabricator.wikimedia.org/T206315 (10elukey) p:05Triage>03Normal [11:49:39] (03CR) 10Fdans: [C: 032] Remove adjustedGraphData and update children [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/464599 (https://phabricator.wikimedia.org/T206171) (owner: 10Milimetric) [11:49:54] 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Decommission bohrium - https://phabricator.wikimedia.org/T206315 (10ops-monitoring-bot) wmf-decommission-host was executed by elukey for bohrium.eqiad.wmnet and performed the following actions: - Revoked Puppet certificate - Removed from Pu... [11:50:05] (03CR) 10Fdans: [C: 031] "Works fine, lgtm!" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/464599 (https://phabricator.wikimedia.org/T206171) (owner: 10Milimetric) [12:00:38] 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Decommission bohrium - https://phabricator.wikimedia.org/T206315 (10elukey) ``` elukey@ganeti1001:~$ sudo gnt-instance remove bohrium.eqiad.wmnet This will remove the volumes of the instance bohrium.eqiad.wmnet (including mirrors), thus rem... [12:00:51] bohrium is gone! [12:00:57] 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Decommission bohrium - https://phabricator.wikimedia.org/T206315 (10elukey) [12:02:04] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Upgrade bohrium (piwik/matomo) to Debian Stretch - https://phabricator.wikimedia.org/T202962 (10elukey) [12:02:08] 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Decommission bohrium - https://phabricator.wikimedia.org/T206315 (10elukey) 05Open>03Resolved [12:02:38] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Upgrade bohrium (piwik/matomo) to Debian Stretch - https://phabricator.wikimedia.org/T202962 (10elukey) [12:02:50] * elukey dances [12:27:59] also created all the accounts on mysql on an-coord1001 [12:28:06] I think that we are ready to migrate [12:50:02] elukey: when would you want us to migrate? [12:53:17] whenever you prefer, I guess it is only a matter of me and you scheduling 1 hour [12:54:41] elukey: I do hope it'll only be 1 hour :) Next week? [12:55:15] Also elukey - Kudos for bohrium :) [12:58:56] joal: oh yes not now, something like Wed morning 10 CEST? [12:59:03] or Tue [12:59:35] elukey: Tuesday a lot better (Wed for kids) [12:59:41] ack! [13:00:06] Tuesday 9th, 10CEST is perfect :) [13:07:32] (03CR) 10Fdans: [C: 031] "Looks great! :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/463548 (https://phabricator.wikimedia.org/T202490) (owner: 10Joal) [13:19:16] so joal I am thinking about a two steps procedure for the swap [13:19:24] on monday we move druid to an-coord1001 [13:19:37] and we verify that everything works as expected [13:19:48] then on Tuesday we move oozie/hive [13:20:43] divide et impera :D [13:21:34] 10Analytics-Kanban, 10User-Elukey: Upgrade Analytics infrastructure to Debian Stretch - https://phabricator.wikimedia.org/T192642 (10elukey) [13:42:56] so in theory we could do something like the following: [13:43:13] 1) stop all the indexation recurrent jobs just to be sure [13:43:25] hi joal, decided I'm working, wanna do quality for a bit today? [13:43:30] 2) dump the druid databases and quickly storing to an-coord1001 [13:43:36] and then merge the puppet change [14:00:25] * elukey afk for a bit! [14:15:01] fdans: I can merge your change now, but were you going to try and have all the breakdown values expanded by default? [14:15:58] (the filter one) [14:26:46] ottomata: o/ [14:27:46] milimetric: right now that’s what it does right? I changed it so that when you activate the RB no one is selected and all values appear [14:29:09] fdans: oh maybe I misunderstood what mforns meant. Right now it's as I was arguing, but I thought mforns suggested further just expanding all the breakdown values of _all_ breakdowns. That way basically you could directly go to filter by a specific breakdown value with one click instead of two? [14:30:37] milimetric, fdans, yea, that was my annoying idea :] [14:31:12] cool. So I was thinking the only snag would be selecting the parent breakdown automatically when they select the breakdown value. [14:31:27] mforns milimetric 👀 [14:31:29] but maybe you can add a click handler to do that on the container of the breakdown section? [14:31:39] eyes!! [14:31:45] I don't know what that means [14:31:46] :) [14:39:57] o/ [14:40:03] milimetric: I'm going to try a nuria technique [14:40:09] oo power dying, running back home! [14:40:52] milimetric: how about we merge this, then yall make some wireframes of how you want it, and then we push another patch? [14:41:11] ok, works for me [14:41:16] one minute [14:46:54] milimetric: this one allows us to hide areas with no metrics working, like in the case of all-wikipedias or whatevert [14:46:54] https://gerrit.wikimedia.org/r/#/c/analytics/wikistats2/+/464800/ [14:48:28] back [14:49:53] 10Analytics, 10Analytics-Wikistats: Git push and pull don't complete - https://phabricator.wikimedia.org/T206331 (10ezachte) [14:50:03] milimetric: if you got a few, wanna brain bounce about mep schema reg stuff real quick? [14:53:03] ok ottomata, going to cave [14:54:25] fdans: I saw the hiding thing, not sure about the click-to-hide idea, but it's cool that we can hide it in config [14:55:25] milimetric: yeah that's what I was hinting in the task, maybe we only want to click-to-toggle when it's hidden by configuration [14:56:28] hm... that sounds interesting... I should play with it a bit [14:56:52] nuria's gonna kill me that we don't have designer budget :( my bad [14:59:38] 10Analytics, 10Analytics-Wikistats: Git push and pull don't complete - https://phabricator.wikimedia.org/T206331 (10elukey) Hi Erik, can you give me an example of command that you give that hangs? git should use by default http_proxy configs on stat1005 (system property), but it might not work with your settings. [15:00:57] 10Analytics, 10Analytics-Wikistats: Git push and pull don't complete - https://phabricator.wikimedia.org/T206331 (10elukey) ``` Last login: Wed Oct 3 10:23:32 2018 from 91.198.174.113 elukey@stat1005:~$ cat /etc/gitconfig # vim: set ts=4 sw=4 et: # This file is managed by Puppet! # puppet:://modules/git/gitco... [15:01:23] milimetric: oh sorry missed ya [15:01:25] going to cave [15:05:37] 10Analytics, 10Operations, 10ops-eqiad: analytics1068 doesn't boot - https://phabricator.wikimedia.org/T203244 (10elukey) ping :) [15:05:41] milimetric: i think for now we can just allow toggling if all metrics are disabled for that area [15:12:51] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Replace the Analytics Hadoop coordinator - Hive/Oozie/etc... (hardware refresh) - https://phabricator.wikimedia.org/T205509 (10elukey) [15:12:56] PROBLEM - Number of segments reported as unavailable by the Druid Coordinators of the Analytics cluster on einsteinium is CRITICAL: 1152 gt 200 https://grafana.wikimedia.org/dashboard/db/druid?refresh=1m&panelId=46&fullscreen&orgId=1&var-cluster=druid_analytics&var-druid_datasource=All [15:15:34] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Replace the Analytics Hadoop coordinator - Hive/Oozie/etc... (hardware refresh) - https://phabricator.wikimedia.org/T205509 (10elukey) The host has been set up with basic functionalities, and all daemons and mariadb seem working fine. I al... [15:16:06] This is marcel loading NavTiming segments [15:16:14] yes! [15:16:18] thsi alert is annoying [15:17:05] * elukey sings "Marcellona!" asking fdans to do the same [15:17:33] excuse me elukey there's only one L in Marcelona [15:17:43] ahahahahahah [15:17:46] yes you are right [15:18:01] I ruined all the magic [15:21:10] mforns: Marcelo I am trying to reduce the noise of the Druid alarm, can you ping me the next time that you load so we can see if it is good or not? [15:35:43] (03CR) 10Milimetric: [C: 032] Allow breakdown filtering in top metrics [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/463964 (https://phabricator.wikimedia.org/T205725) (owner: 10Fdans) [15:36:00] (03PS2) 10Milimetric: Remove adjustedGraphData and update children [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/464599 (https://phabricator.wikimedia.org/T206171) [15:39:02] RECOVERY - Number of segments reported as unavailable by the Druid Coordinators of the Analytics cluster on einsteinium is OK: (C)200 gt (W)180 gt 0 https://grafana.wikimedia.org/dashboard/db/druid?refresh=1m&panelId=46&fullscreen&orgId=1&var-cluster=druid_analytics&var-druid_datasource=All [15:42:34] 10Analytics, 10Analytics-Kanban: [Wikistats2] Bug in Top Viewed Articles since bookmark routing changes - https://phabricator.wikimedia.org/T198258 (10Milimetric) a:03Milimetric [15:50:08] milimetric: sorry was gone - some quality work later on? [15:51:24] joal: it's ok, but my brain is complicated now, I'll let you know when it resolves :) [15:51:33] np milimetric :) [15:51:33] I'm lazy evaluating a lot of things [15:52:01] With lazy-evaluation, call for action can trigger a lot of cascading stuff :) [15:53:17] yeah, tell me about it :) [15:54:00] fdans: what's going on with this: https://gerrit.wikimedia.org/r/#/c/analytics/wikistats2/+/462688/? [15:54:12] 10Analytics, 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Allow Analytics team members to restart Turnilo and Superset - https://phabricator.wikimedia.org/T206217 (10herron) Is https://gerrit.wikimedia.org/r/464831 what you had in mind in terms of sudo privs? [15:54:36] (03CR) 10Milimetric: [C: 032] Remove adjustedGraphData and update children [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/464599 (https://phabricator.wikimedia.org/T206171) (owner: 10Milimetric) [15:55:03] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Discovery-Search (Current work), 10Services (designing): Huge messages in eqiad.mediawiki.job.cirrusSearchElasticaWrite (and other?) topics - https://phabricator.wikimedia.org/T196032 (10debt) 05Open>03Resolved a:03debt [15:55:05] milimetric: it can be merged, but right now no metric uses it [15:55:28] oh ok, we can just wait until it's needed [15:55:30] I did it a moment during the offsite [15:55:45] that way it'll have a phab task associated [16:00:30] ping milimetric ottomata [16:04:11] (03PS1) 10Milimetric: Cache annotations for 30 minutes [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/464835 (https://phabricator.wikimedia.org/T205933) [16:04:24] (03CR) 10Milimetric: [V: 032 C: 032] Cache annotations for 30 minutes [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/464835 (https://phabricator.wikimedia.org/T205933) (owner: 10Milimetric) [16:10:02] oh elukey did you see that we reverted the refinery scap target change for an-coord01? [16:10:57] ottomata: ah snap the refinery profile is not there, I thought you guys were deploying next week [16:11:00] sorryyyy [16:11:01] :( [16:11:29] np! [16:17:58] 10Analytics, 10Analytics-Kanban: Parametize eventlogging to druid ingestion with a whitelist instead of a blacklist - https://phabricator.wikimedia.org/T206342 (10Nuria) [16:19:46] (03PS11) 10Joal: Add python script importing xml dumps onto hdfs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/456654 (https://phabricator.wikimedia.org/T202489) [16:41:16] mforns: talking to perf guys about turnilo in perf channel, can you check my answer? [16:44:21] nuria, reading [16:45:17] nuria, what is the name of the channel? I can not find it in the irc logs [16:45:30] mforns: wikimedia-perf [16:45:34] going to lunch [16:45:57] nuria, I think it is not publicly logged, and when I log, I don't get a scrollback... [16:46:16] mforns: just ping Krinkle [16:46:20] (timo) [16:49:53] ottomata: got a minute to talk about refinement? [16:51:30] ya [16:51:33] bc joal? [16:51:36] sure [17:31:19] joal: when you are back: very bad pseudo code: https://gist.github.com/ottomata/212befe63e7ea993645d3d675303df84 [17:44:51] * elukey off! [18:13:02] 10Analytics, 10Analytics-Kanban, 10Wikimedia-Site-requests, 10MobileFrontend (MobileFrontend.js), and 2 others: Turn on MinervaErrorLogSamplingRate (Schema:WebClientError) - https://phabricator.wikimedia.org/T203814 (10Jdlrobson) 05Open>03Resolved a:03Jdlrobson All done here! [19:55:57] Hey ottomata - indeed, this should work - It however loops over the data 'tags.length + 1' time (distinct count) [19:58:22] ottomata: the partitionBy function should only loop once [19:59:45] aye yeah [20:01:16] ottomata: I think I'm gonna try bioth solutions: a simple spark job (which we would to updating upon webrequest schema change), and a patch of DataFrameToHive, so see how far I can go that way [20:03:30] joal: i was with you for the patch all the way to the needing to run repair table instead of add partitions conditionally [20:03:36] although, it does sound like a nice feature [20:03:37] ... [20:03:38] i dunno [20:03:51] ottomata: I'll write it, and we'll see :) [20:03:53] kinda nice if you use DF to Hive too [20:03:57] because then it is webrequest agnostic! [20:04:08] give source table, distinct tags, etc. do the thing! [20:04:10] indeed, this is my will as qwell [20:04:21] ok, i guess let's see, if you are up for trying it [20:04:27] see if you can do partitionBy by default too [20:04:36] and get rid of the direct path write [20:04:45] ottomata: I would actually use s TransformationFunction to link to tags [20:04:52] yup [20:04:53] oh [20:04:56] ? [20:05:21] Yeah, use a webrequest DF, and in the transformation function, link it to tags [20:05:45] tags is already there? OH i see you mean link the PartitionDataframe to tags? [20:05:47] So we have a DataframeToHive from webrequest though transform function :) [20:05:53] yes ! [20:06:00] aye i think i get it [20:06:32] ottomata: I'll try to have a patch for early next week :) [20:34:40] (03PS12) 10Joal: Add python script importing xml dumps onto hdfs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/456654 (https://phabricator.wikimedia.org/T202489) [20:38:29] 10Analytics, 10Analytics-Kanban: [Wikistats2] Bug in Top Viewed Articles since bookmark routing changes - https://phabricator.wikimedia.org/T198258 (10Milimetric) I can't reproduce this or any other bugs on the top viewed articles, and I tried a bunch of things. @mforns let me know if it still happens to you... [20:41:25] (03CR) 10Milimetric: "this could/should be done as part of https://phabricator.wikimedia.org/T204717 I think. I'll leave it up to you but I'm happy to take it " [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/464800 (https://phabricator.wikimedia.org/T206311) (owner: 10Fdans) [20:42:48] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Routing code allows invalid routes - https://phabricator.wikimedia.org/T188792 (10Milimetric) a:03Milimetric [20:43:41] (03CR) 10Fdans: "@Milimetric the motivation of this change is just to move on with https://gerrit.wikimedia.org/r/#/c/analytics/wikistats2/+/464583/ so we " [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/464800 (https://phabricator.wikimedia.org/T206311) (owner: 10Fdans) [20:47:18] (03CR) 10Milimetric: Allow whole metric areas to be collapsed (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/464800 (https://phabricator.wikimedia.org/T206311) (owner: 10Fdans) [21:07:29] (03PS1) 10Milimetric: Defer to the config to specify the area [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/464906 (https://phabricator.wikimedia.org/T188792) [21:16:42] (03CR) 10Nuria: "Let's talk about whether we need this work, I am of the opinion that we should load the data we have for families and unique devices and" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/464800 (https://phabricator.wikimedia.org/T206311) (owner: 10Fdans) [21:23:11] (03CR) 10Nuria: [C: 04-1] Allow whole metric areas to be collapsed [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/464800 (https://phabricator.wikimedia.org/T206311) (owner: 10Fdans) [21:48:49] (03CR) 10Nuria: [C: 04-1] "Do not see many changes between patch 1 and 2, maybe there are files that were not committed?" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/464583 (https://phabricator.wikimedia.org/T188550) (owner: 10Fdans)