[03:03:03] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Reimage thorium to Debian Stretch - https://phabricator.wikimedia.org/T192641 (10Dzahn) ``` Cron /usr/local/bin/hardsync -t /srv /srv/published-datasets-rsynced/* /srv/analytics.wikimedia.org/datasets 2>&1 > /dev/null Inbox x Cron Daemon ro... [03:05:21] 10Analytics, 10Readers-Web-Backlog, 10Wikimedia-Site-requests, 10MobileFrontend (MobileFrontend.js), 10Patch-For-Review: Turn on MinervaErrorLogSamplingRate (Schema:WebClientError) - https://phabricator.wikimedia.org/T203814 (10Nuria) After talking with @Ottomata on irc about the reasons why eventlogging... [04:02:54] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Google-Summer-of-Code (2018): GSoC Proposal 2018 : [Analytics] Improvements to Wikistats2 front-end - https://phabricator.wikimedia.org/T189964 (10srishakatux) 05Open>03Resolved [06:21:19] !log re-run webrequest-load-wf-text-2018-9-12-4, failed due to sql exceptions/timeouts to the database [06:21:21] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [06:21:24] this is very new [06:28:46] (03PS1) 10Amire80: Add cx2 translators count [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/459904 (https://phabricator.wikimedia.org/T196435) [06:52:03] the new job is refining [06:53:30] the error was [06:53:31] org.apache.oozie.command.CommandException: E0603: SQL error in operation, org.apache.openjpa.persistence.PersistenceException: The last packet successfully received from the server was 115,156,272 milliseconds ago. [06:53:56] now if we translate 115,156,272 milliseconds in days ago it is ~1.3 if I am not mistaken [06:54:02] that is when I rebooted the host [06:54:33] so oozie might have been got into a confusing state after the reboot? [06:57:27] http://mail-archives.apache.org/mod_mbox/oozie-user/201205.mbox/%3CCBE141EC.13716%25kernst@mobileposse.com%3E [07:07:20] there seems to be workaround for this issue, like adding auto_reconnect, wait_timeout, etc.. but I am inclined not to mess with them unless we see a regression in oozie's failures [07:12:52] good morning, I'm installing PHP security updates on bohrium, there might be a small glitch in the force [07:13:26] done [07:20:02] thanks! [07:20:26] as FYI we have a task to migrate bohrium to stretch + php7 [07:21:20] good :-) [07:39:01] 10Analytics, 10Analytics-Cluster: Upgrade Hive to ≥ 2.0 - https://phabricator.wikimedia.org/T203498 (10elukey) LLAP seems to me (I am very ignorant on this front so don't take this as authoritative source :) very close to what [[https://prestodb.io/| Presto]] does to achieve its query speed, so it might be an... [07:52:52] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Test role::analytics_cluster::coordinator on Debian Stretch - https://phabricator.wikimedia.org/T204060 (10elukey) 05Open>03stalled Created a subtask to track the current issue with (managed) Mariadb on Stretch, waiting for its completion before re-tes... [07:59:39] 10Analytics, 10Operations, 10hardware-requests: Decommission Ganeti vm meitnerium.wikimedia.org (old Archiva host) - https://phabricator.wikimedia.org/T203087 (10elukey) >>! In T203087#4574561, @Krenair wrote: > No switch port disabling step for VMs either Yep this makes sense since the underlying ganeti ho... [08:00:03] 10Analytics, 10Operations, 10hardware-requests: Decommission Ganeti vm meitnerium.wikimedia.org (old Archiva host) - https://phabricator.wikimedia.org/T203087 (10elukey) [08:17:47] morning all [08:18:28] * addshore wants a number :D [08:18:48] mwhahahahahha [08:19:57] How many unique properties are referred to on the entity that refers to the most properties? (covering all references to properties, so main snaks, references, qualifiers) [08:25:23] addshore: I didn't get what you are asking so I suppose you had a past conversation with Joseph :D He will join the chan later on in the afternoon, so you might not get a quick response if you need something now [08:25:27] :) [08:32:05] 10Analytics, 10Analytics-Cluster: Update to CDH 6 or other up-to-date Hadoop distribution - https://phabricator.wikimedia.org/T203693 (10elukey) Added a question about Debian and CDH6 to http://community.cloudera.com/t5/CDH-Manual-Installation/CDH-6-supported-OS/m-p/79686#M1766 [08:32:55] 10Analytics, 10Analytics-Cluster: Update to CDH 6 or other up-to-date Hadoop distribution - https://phabricator.wikimedia.org/T203693 (10elukey) [08:39:39] elukey: I figure joal Might have a good idea of how to get the answer :D [09:08:44] 10Analytics, 10Patch-For-Review: Remove webrequest misc analytics related jobs and code after cache misc -> text merge is complete - https://phabricator.wikimedia.org/T200822 (10ema) >>! In T200822#4574800, @elukey wrote: > @ema just to be sure, can you confirm that cache misc is gone and that we can get rid o... [10:03:08] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Remove webrequest misc analytics related jobs and code after cache misc -> text merge is complete - https://phabricator.wikimedia.org/T200822 (10elukey) [10:36:40] 10Analytics, 10DBA: mariadb::service and managed services don't play well on Stretch - https://phabricator.wikimedia.org/T204074 (10jcrespo) I think removing the require would suffice, this was a relic from when we setup the systemd unit with puppet, which we don't do anymore. If for some reason the systemd un... [11:25:12] 10Analytics, 10Operations, 10hardware-requests: Decommission Ganeti vm meitnerium.wikimedia.org (old Archiva host) - https://phabricator.wikimedia.org/T203087 (10akosiaris) [11:31:06] 10Analytics, 10Operations, 10hardware-requests: Decommission Ganeti vm meitnerium.wikimedia.org (old Archiva host) - https://phabricator.wikimedia.org/T203087 (10akosiaris) The steps listed in the description look correct and sufficient to me. There is one thing to add and it would be the removal from DebMon... [11:59:15] 10Analytics, 10Operations, 10hardware-requests: Decommission Ganeti vm meitnerium.wikimedia.org (old Archiva host) - https://phabricator.wikimedia.org/T203087 (10elukey) ``` elukey@puppetmaster1001:~$ sudo -i puppet node clean meitnerium.wikimedia.org Notice: Revoked certificate with serial 1595 Notice: Remo... [12:00:12] 10Analytics, 10Operations, 10hardware-requests: Decommission Ganeti vm meitnerium.wikimedia.org (old Archiva host) - https://phabricator.wikimedia.org/T203087 (10elukey) ``` elukey@neodymium:~$ sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/meitnerium.wikimedia.org --cert /etc/debmonitor/ssl/ce... [12:03:38] 10Analytics, 10Operations, 10hardware-requests: Decommission Ganeti vm meitnerium.wikimedia.org (old Archiva host) - https://phabricator.wikimedia.org/T203087 (10elukey) ``` elukey@ganeti1001:~$ sudo gnt-instance remove meitnerium.wikimedia.org This will remove the volumes of the instance meitnerium.wikimedi... [12:06:25] 10Analytics, 10Operations, 10hardware-requests: Decommission Ganeti vm meitnerium.wikimedia.org (old Archiva host) - https://phabricator.wikimedia.org/T203087 (10elukey) [12:07:05] bye bye meitnerium [12:07:07] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Upgrade Archiva (meitnerium) to Debian Stretch - https://phabricator.wikimedia.org/T192639 (10elukey) [12:07:12] 10Analytics, 10Operations, 10hardware-requests: Decommission Ganeti vm meitnerium.wikimedia.org (old Archiva host) - https://phabricator.wikimedia.org/T203087 (10elukey) 05Open>03Resolved [12:59:22] ottomata: o/ [13:00:19] yoohooo [13:07:45] joal: got a sec for joal got a sec for ? [13:09:03] (03PS2) 10Ottomata: Throw usable exception when missing required field in ConfigHelper configure [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/459845 (https://phabricator.wikimedia.org/T203804) [13:11:14] !log otto@deploy1001 Started deploy [eventlogging/analytics@5c6fab6]: Support loading plugins in eventlogging-processor - T203596 [13:11:22] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:11:23] T203596: Flip blacklist for MySQL eventlogging consumer to be a whilelist of allowed schemas - https://phabricator.wikimedia.org/T203596 [13:16:50] elukey: any objections to merging https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/459807/ ? [13:17:23] its currently cherry picked in beta and works [13:22:18] ottomata: +2 [13:28:21] cool, looks good so far [13:40:35] (03CR) 10Milimetric: [C: 032] Add cx2 translators count [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/459904 (https://phabricator.wikimedia.org/T196435) (owner: 10Amire80) [13:41:15] (03CR) 10Milimetric: [C: 032] "This will run the next time puppet pulls the latest code (every 30 min - 1 hr), and the data will be available when it's done, probably an" [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/459904 (https://phabricator.wikimedia.org/T196435) (owner: 10Amire80) [13:46:42] (03CR) 10Ottomata: [V: 032 C: 032] "Want to keep moving on this, joal feel free to add post-merge comments and I will respond to them." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/459845 (https://phabricator.wikimedia.org/T203804) (owner: 10Ottomata) [13:46:55] (03PS8) 10Ottomata: Use ConfigHelper for RefineMonitor instead of scopt [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/458862 (https://phabricator.wikimedia.org/T203804) [13:47:08] (03CR) 10Ottomata: [V: 032 C: 032] Use ConfigHelper for RefineMonitor instead of scopt [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/458862 (https://phabricator.wikimedia.org/T203804) (owner: 10Ottomata) [14:13:50] heya teaaaam :] [14:29:33] o/ [14:31:40] hiaaa [14:33:41] ottomata: when you have time can you check my code review for cache misc deprecation in refinery? [14:36:20] O!H! sorry i checked it yesterday and didn't +1 [14:36:29] elukey: the only q i had was about wdqs_extractor [14:36:34] we should find out if that is actually used [14:36:39] seeing as its been running on misc for a while nonw [14:36:40] now [14:36:42] it hasn't had any data [14:36:49] maybe we can just turn it off, instead of also processing all of text for it [14:37:17] ottomata: yes I thought the same, today I found that the original task was filed from leila, so she might be out poc ? [14:37:31] poc [14:37:32] ? [14:37:40] point of contact sorry :) [14:37:44] oh [14:37:44] ya [14:37:45] ok [14:37:47] let's ask her [14:38:06] (03CR) 10Ottomata: [C: 031] "+1! Let's find out if we can turn of wdqs_extract." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/459827 (https://phabricator.wikimedia.org/T200822) (owner: 10Elukey) [14:38:46] all right going to ping her in the task [14:43:47] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Remove webrequest misc analytics related jobs and code after cache misc -> text merge is complete - https://phabricator.wikimedia.org/T200822 (10elukey) Hi @leila! Our dear cache misc (the Varnish hosts that were hosting misc websites like phabricator, ya... [14:44:28] there you go --^ [14:46:41] mforns: o/ [14:46:45] do you have a min for T128623 ? [14:46:46] T128623: Purge all Schema:Echo data after 90 days - https://phabricator.wikimedia.org/T128623 [14:49:36] elukey, reading [14:50:12] mforns: IIUC the schema is not used anymore, and from the mysql point of view we have to just drop all the Echo_* tables? [14:50:27] elukey, yes, from mysql and from hive no? [14:50:42] want to do that now? [14:51:26] nope the sre team is switching to codfw and they could kill me if I touch anything now :D [14:51:30] let's do it tomorrow [14:51:43] ok, do you think we need to stop the purging script? [14:53:05] in theory no, but it takes so little to do it that we can just take the safe way :) [14:53:14] I'd do it with you on hangouts whenever you have time [14:53:30] so we can both check that I am not dropping random things :D [15:00:31] elukey, sure! just ping me and we pair [15:00:42] ack :) [15:01:13] elukey, tomorrow around this time I'll be here and free [15:01:21] super [15:02:12] elukey: when you have time https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/460023/ [15:18:50] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Q1 2018/19 Analytics procurement - https://phabricator.wikimedia.org/T198694 (10Cmjohnson) [15:20:15] 10Analytics, 10Operations, 10ops-eqiad: rack/setup/install stat1007.eqiad.wmnet (stat1005 user replacement) - https://phabricator.wikimedia.org/T203852 (10Cmjohnson) [15:45:44] 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install stat1007.eqiad.wmnet (stat1005 user replacement) - https://phabricator.wikimedia.org/T203852 (10Cmjohnson) [15:46:14] 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install stat1007.eqiad.wmnet (stat1005 user replacement) - https://phabricator.wikimedia.org/T203852 (10Cmjohnson) a:05Cmjohnson>03RobH @robh all yours to finish installations [15:58:45] Hallo [15:59:00] Is it possible to add a new data cube to Turnilo? [16:03:40] joal, standup? :] [16:17:46] joal: qq - is it ok then to swap the master nodes (and probably analytics1003) on Sept 22 10 AM CEST? [16:18:18] need to have you on my side while doing dangerous things :D [16:18:40] elukey: Sept 22 is a Saturday - On purpose I imagine? [16:18:59] sorry 25th [16:19:01] Tue [16:19:16] we'll be in NYC or flying on the 22nd :) [16:19:30] Yeah, was computing that in my head as well :) [16:19:43] 25th no issue - Available all day :) [16:20:00] super [16:20:04] I'll send the notification [16:20:15] ok ottomata, done [16:20:19] Awesome [16:22:51] mmmm, I'm gonna go pick up a sammich, ottomata call me on the phone if you wanna talk meanwhile, be back in 20 min or so [16:24:47] milimetric: i need to eat too, let's chat after lunch [16:41:41] * elukey off! [16:42:26] Bye elukey :) [16:43:51] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Flip blacklist for MySQL eventlogging consumer to be a whitelist of allowed schemas - https://phabricator.wikimedia.org/T203596 (10Tbayer) [16:46:35] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Flip blacklist for MySQL eventlogging consumer to be a whitelist of allowed schemas - https://phabricator.wikimedia.org/T203596 (10Tbayer) BTW, ReadingDepth should be blacklisted/de-whitelisted from MySQL too, as we are planning to increase its event rate... [16:49:54] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Flip blacklist for MySQL eventlogging consumer to be a whitelist of allowed schemas - https://phabricator.wikimedia.org/T203596 (10Tbayer) >>! In T203596#4577520, @Tbayer wrote: > BTW, ReadingDepth should be blacklisted/de-whitelisted from MySQL too, as we... [16:50:37] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Flip blacklist for MySQL eventlogging consumer to be a whitelist of allowed schemas - https://phabricator.wikimedia.org/T203596 (10Ottomata) Naw, i'll just do it as part of this one, thanks. [16:56:04] Another attempt: Is it possible to add a new data cube to Turnilo? [16:57:31] (03PS3) 10Joal: Update python/refinery/utils/HdfsUtils [analytics/refinery] - 10https://gerrit.wikimedia.org/r/459780 (https://phabricator.wikimedia.org/T202489) [16:58:12] aharoni: Hi - Adding data to turnillo is actually adding data to Druid (turnillo backend) [16:58:35] aharoni: This can be done, but also depends on size, updates, etc [16:59:04] OK, is there a guide for that? It's a very small update every day. [16:59:05] (03CR) 10Joal: Update python/refinery/utils/HdfsUtils (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/459780 (https://phabricator.wikimedia.org/T202489) (owner: 10Joal) [17:00:45] aharoni: very small update every day is actually kinda counter-intuitive for Druid - We don't have docs per say about adding datasources to druid, but we have oozie examples (https://github.com/wikimedia/analytics-refinery/tree/master/oozie/unique_devices/per_domain/druid for instance) [17:01:11] joal: you can see an example here: https://phabricator.wikimedia.org/T189475#4043166 . It's around 10 lines per day. I discussed it with Neil PQ and he thinks that Turnilo is a good platform to display it, and I agree. [17:02:22] I understand that Turnilo/Druid is for large data sets, but Neil and I thought that it's a kind of data that will be convenient to display there. [17:02:30] aharoni: I also think turnilo is good for exploring this - I however think Druid is not really the good backend for this use-case - The concern now is that turnilo only works for druid ... [17:03:00] Hmm... so what can you suggest? [17:06:32] aharoni: for this data-size, MySQL would do - Then superset can query it and show charts [17:07:35] milimetric: ready whenever you ar [17:07:35] e [17:07:40] aharoni: I'm actually thinking analytics-store [17:08:09] OK... and is there a guide for that? I have a script for querying this from EventLogging. I just need a convenient way to do two things: 1. run it automatically every day. 2. show the results conveniently on a dashboard. [17:08:24] ottomata: in cave [17:11:45] aharoni: I think the easiest for 1. is to use ReportUpdater (https://wikitech.wikimedia.org/wiki/Analytics/Systems/Reportupdater) - As for 2. there is no doc, but once data is in analytics-store, showing it in superset will be easy [17:16:50] Gone for diner team - back after [17:18:28] joal: OK... I have a little bit of experience with using report updater for relatively simple things like https://analytics.wikimedia.org/datasets/periodic/reports/metrics/published_cx2_translations/ , with two columns: date and value. This is a bit different: every day there are several values associated with a number. Can report updater do it? [17:21:45] (03PS19) 10Zhuyifei1999: Update dependencies [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/428140 (https://phabricator.wikimedia.org/T192731) (owner: 10Framawiki) [17:22:01] (03PS16) 10Zhuyifei1999: Port to Python3 [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/440007 (https://phabricator.wikimedia.org/T192698) (owner: 10Framawiki) [17:22:27] (03PS17) 10Zhuyifei1999: Port to Python3 [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/440007 (https://phabricator.wikimedia.org/T192698) (owner: 10Framawiki) [17:34:13] !log deploying new version of refinery-source, and then refinery for properties based RefineMonitor job - https://phabricator.wikimedia.org/T203804 [17:34:15] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 2 others: RFC: Modern Event Platform: Scalable Event Intake Service - https://phabricator.wikimedia.org/T201963 (10Ottomata) [17:34:17] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:02:35] 10Quarry, 10Patch-For-Review: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 (10Framawiki) [18:17:34] (03PS5) 10Joal: Add python script importing xml dumps onto hdfs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/456654 (https://phabricator.wikimedia.org/T202489) [18:23:44] mforns: Dear ReportUpdater master - I have a question for you :) [18:23:56] O.o .. [18:23:57] yea [18:24:01] ? [18:25:07] mforns: I suggested ReportUpdater as a tool to automate running a query against a DB to populate another DB - Am I in an esoteric reportupdater world, or is it reasonable to do it this way? [18:25:37] hmmmm [18:26:01] the stardard way of using it, is just generating reports [18:26:31] however, you could execute a script that queries and populates [18:26:39] instead of giving just a query... [18:27:07] but you'd need an outputed report in any case, because that's the way RU knows which data is missing [18:27:59] makes sense - So one solution could be to write the query script as 2 queries: one to populate the new DB, on to populate the report (from the new DB) [18:27:59] tl;dr: I believe it's possible (even if it wasn't thought for that), but never done before... [18:28:28] yea I guess! [18:29:20] or just have a script that calls the dbclient first to query and pipes results to another call to dbclient that populates second db [18:30:11] mforns: Aouch - You reached my non-understanding point - I was feeling good about faking knowing RU, now I'm discovered ! [18:30:12] and then have the whole script return a "fake" report, that will serve merely for RU to determine next execution [18:30:22] xD [18:30:27] no, do you want to bc? [18:30:41] omw [18:30:43] ok [18:31:00] aharoni: see the discussion with mforns above --^ [18:32:37] thanks joal mforns . If you don't mind, I'll add you to a task to try to make some kind of a plan. [18:33:42] aharoni: while thinking of it - Would a report as the one you built for content-translation do the job? [18:37:49] joal: which one? The one I did at https://language-reportcard.wmflabs.org/cx2/#translations (with help from milimetric and fdans - thanks!) is very simple: just date and number. For this new thing I need something more complicated. [18:40:43] aharoni - So that mforns follows us - We are talking about https://phabricator.wikimedia.org/T189475 [18:40:49] aharoni, if you want we can discuss whether RU will be enough here or Hangouts or Phab, can you... [18:40:57] oh! you just did, thanks [18:41:05] hangouts is good any time [18:41:08] even right now [18:41:20] ok, let's talk then! [18:42:12] mforns, aharoni, where are you going? Let me follow you ! [18:42:18] oh ok! [18:43:19] joal, aharoni https://hangouts.google.com/hangouts/_/wikimedia.org/a-batcave [18:44:49] mforns: https://phabricator.wikimedia.org/T189475#4043166 [19:04:12] ottomata: if you're around, I think I have witnessed a failure of superset loading a new user :) [19:05:43] mforns: the change in user-agent in the https://analytics.wikimedia.org/dashboards/browsers/#all-sites-by-os chart on May 20th is just drastic :D [19:06:11] (03CR) 10Zhuyifei1999: [C: 032] Update dependencies [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/428140 (https://phabricator.wikimedia.org/T192731) (owner: 10Framawiki) [19:06:17] joal, oh yea! [19:06:34] (03Merged) 10jenkins-bot: Update dependencies [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/428140 (https://phabricator.wikimedia.org/T192731) (owner: 10Framawiki) [19:06:35] that's because of the ua-parser right? [19:06:42] correct mforns :) [19:06:43] I think it's much better like this. [19:06:50] (03CR) 10Zhuyifei1999: [C: 032] Port to Python3 [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/440007 (https://phabricator.wikimedia.org/T192698) (owner: 10Framawiki) [19:07:08] mforns: agreed, but I think I had prefered not to see windows owning 30% :D [19:07:08] (03Merged) 10jenkins-bot: Port to Python3 [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/440007 (https://phabricator.wikimedia.org/T192698) (owner: 10Framawiki) [19:07:15] joal, heh [19:08:06] joal, I would prefer not to see that huuuuuuuge Other category... We have to fix that.. [19:08:59] mforns: is it fixable? Some say: if you can't fix it, it's not broken :) [19:09:05] hehehe [19:09:15] yes, we can reduce it by half I think [19:09:23] Could be nice [19:09:40] I think there's already a ticket for that [19:10:11] joal, https://phabricator.wikimedia.org/T131127 [19:13:37] 10Quarry, 10Patch-For-Review: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 (10Framawiki) [19:15:05] joal: yt there for some scala help? [19:15:10] :D [19:16:06] ottomata: OMW :) [19:16:13] 10Quarry, 10Patch-For-Review: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 (10Framawiki) For the record New instances: quarry-db-01.quarry.eqiad.wmflabs quarry-main-01.quarry.eqiad.wmflabs quarry-runner-01.quarry.eqiad.wmflabs quarry-runner-02.quarry.eqiad.wmflabs Old ones: quarry-... [19:18:06] hm chrome is crashing [19:36:55] 10Analytics, 10Contributors-Analysis, 10Product-Analytics: Streamline Superset signup and authentication - https://phabricator.wikimedia.org/T203132 (10Ottomata) Ok, this kinda works, but then it doesn't. We need to find a way to get the user's email address from ldap when they first sign up. I think this... [19:37:33] (03PS3) 10Joal: Cleanup scala input format making it generic [analytics/wikihadoop] - 10https://gerrit.wikimedia.org/r/337004 [19:37:38] ottomata: --^ [19:50:04] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 2 others: RFC: Modern Event Platform: Schema Registry - https://phabricator.wikimedia.org/T201643 (10daniel) @Milimetric what'S yoour take on this? Can this do to last call, or should it see more discussion? [20:03:29] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 2 others: RFC: Modern Event Platform: Schema Registry - https://phabricator.wikimedia.org/T201643 (10Milimetric) I acknowledge my bias here, as a member of the team that's going to implement this feature. We have meetings to make s... [20:04:36] 10Quarry, 10Patch-For-Review: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 (10zhuyifei1999) [20:14:23] 10Quarry: Add statistics to the homepage - https://phabricator.wikimedia.org/T204157 (10Framawiki) p:05Triage>03Low [20:21:29] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 2 others: RFC: Modern Event Platform: Schema Registry - https://phabricator.wikimedia.org/T201643 (10Ottomata) What happens with metadata will probably interest TechComm (e.g. allowing remote clients to configure their sampling rate... [20:21:50] 10Quarry: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 (10Framawiki) [20:33:20] 10Quarry: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 (10zhuyifei1999) [20:38:05] ottomata: thoughts on the schema registry are that Timo's adding a couple of questions about, for example, how a developer would test by pointing to their own schema git repository, and once those get resolved that RFC can go to last call [20:38:19] it looks like intake won't make it next week [20:38:59] 10Quarry: "Recent queries" page badly shows time - https://phabricator.wikimedia.org/T204161 (10Framawiki) [20:43:10] ok [20:44:30] 10Quarry: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 (10Framawiki) {nav icon=file, name=Mentioned in SAL (#wikimedia-cloud), href=https://tools.wmflabs.org/sal/log/AWXPhBGDoDEJc1hAyJp-} [2018-09-12T20:42:34Z] switched quarry.wmflabs.org proxy to new quarry-web-01.quarry.e... [20:47:58] 10Quarry: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 (10Framawiki) [20:52:06] 10Analytics, 10Community-consensus-needed: Decide whether enable per-editor edits stats (community decision) - https://phabricator.wikimedia.org/T203826 (10Milimetric) @Cirdan, The "sharing your information" you're referring to from the Privacy Policy has a very different meaning from the kind of statistics we... [20:53:36] (03PS1) 10Zhuyifei1999: timesince: use floor-division [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460103 (https://phabricator.wikimedia.org/T204161) [20:56:33] (03PS1) 10Framawiki: worker.py: fix indexing of sql errors [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460104 (https://phabricator.wikimedia.org/T192698) [20:58:03] (03PS2) 10Framawiki: worker.py: fix indexing of sql errors [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460104 (https://phabricator.wikimedia.org/T192698) [20:58:21] (03CR) 10Zhuyifei1999: [V: 032 C: 032] worker.py: fix indexing of sql errors [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460104 (https://phabricator.wikimedia.org/T192698) (owner: 10Framawiki) [20:59:51] 10Analytics: Better redirect handling for pageview API - https://phabricator.wikimedia.org/T121912 (10Milimetric) Hm, @bd808 do you think it would actually be useful to address issues like that one by one or just wait for a better way to group pages with all their redirects? (as in, parsing wikitext historically... [21:01:44] (03CR) 10Framawiki: [V: 032 C: 032] timesince: use floor-division [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460103 (https://phabricator.wikimedia.org/T204161) (owner: 10Zhuyifei1999) [21:02:08] (03Merged) 10jenkins-bot: timesince: use floor-division [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460103 (https://phabricator.wikimedia.org/T204161) (owner: 10Zhuyifei1999) [21:05:14] 10Quarry, 10Patch-For-Review: "Recent queries" page badly shows time - https://phabricator.wikimedia.org/T204161 (10zhuyifei1999) 05Open>03Resolved a:03zhuyifei1999 [21:06:22] (03PS1) 10Framawiki: Add config.yaml to .gitignore [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460106 [21:10:01] (03CR) 10Zhuyifei1999: [C: 032] Add config.yaml to .gitignore [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460106 (owner: 10Framawiki) [21:10:34] (03Merged) 10jenkins-bot: Add config.yaml to .gitignore [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460106 (owner: 10Framawiki) [21:11:57] (03PS1) 10Ottomata: Use ConfigHelper for Refine job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/460108 (https://phabricator.wikimedia.org/T203804) [21:15:27] (03PS2) 10Ottomata: Use ConfigHelper for Refine job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/460108 (https://phabricator.wikimedia.org/T203804) [21:42:58] (03PS1) 10Zhuyifei1999: connections.py: Explicitly enable MULTI_STATEMENTS for replica [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460111 (https://phabricator.wikimedia.org/T202588) [21:44:22] (03CR) 10Framawiki: [V: 032 C: 032] connections.py: Explicitly enable MULTI_STATEMENTS for replica [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460111 (https://phabricator.wikimedia.org/T202588) (owner: 10Zhuyifei1999) [21:44:40] (03Merged) 10jenkins-bot: connections.py: Explicitly enable MULTI_STATEMENTS for replica [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460111 (https://phabricator.wikimedia.org/T202588) (owner: 10Zhuyifei1999) [22:14:49] (03Abandoned) 10Aklapper: Insert the description of the change. [analytics/aggregator/projectview/data] - 10https://gerrit.wikimedia.org/r/337228 (owner: 10Testing0922) [22:22:33] 10Quarry, 10Patch-For-Review: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 (10Framawiki) [22:43:16] 10Analytics, 10Community-consensus-needed: Decide whether enable per-editor edits stats (community decision) - https://phabricator.wikimedia.org/T203826 (10Nuria) @Cirdan: the harhsness of your responses is really uncalled for. Our team mission is to make data available for the community and WMF within the v... [22:44:37] 10Analytics, 10Contributors-Analysis, 10Product-Analytics: Streamline Superset signup and authentication - https://phabricator.wikimedia.org/T203132 (10Nuria) @Ottomata can you explain what is missing? [23:00:50] 10Quarry, 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): Let quarry use the mariadb module - https://phabricator.wikimedia.org/T181205 (10Dzahn) This might be resolved now since the merge above. Much appreciated, thank you @zhuyifei1999 and @Bstorm Did that already actively switch it? [23:17:19] 10Quarry, 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): Let quarry use the mariadb module - https://phabricator.wikimedia.org/T181205 (10zhuyifei1999) The new instance quarry-db-01 is now on mariadb, but the old instance quarry-main-01 (with puppet disabled) is still on mysql. If everyt... [23:17:31] 10Quarry: Quarry results serialization (web) should have its memory usage upper-bounded and streamed - https://phabricator.wikimedia.org/T192696 (10zhuyifei1999) [23:17:35] 10Quarry, 10Patch-For-Review: Upgrade Quarry to python 3 - https://phabricator.wikimedia.org/T192698 (10zhuyifei1999) 05Open>03Resolved a:03zhuyifei1999 [23:17:49] 10Quarry, 10Patch-For-Review: Upgrade Quarry to python 3 - https://phabricator.wikimedia.org/T192698 (10zhuyifei1999) a:05zhuyifei1999>03Framawiki [23:18:08] 10Quarry, 10Patch-For-Review: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 (10zhuyifei1999) [23:18:12] 10Quarry: Upgrade Quarry main server from Trusty - https://phabricator.wikimedia.org/T194691 (10zhuyifei1999) 05Open>03Resolved a:03zhuyifei1999 [23:40:59] 10Analytics, 10Operations, 10hardware-requests, 10User-Elukey: eqiad | (14 + 6) hadoop hardware refresh and expansion - https://phabricator.wikimedia.org/T199673 (10RobH)