[00:30:11] 10Analytics: Better redirect handling for pageview API - https://phabricator.wikimedia.org/T121912 (10bd808) >>! In T121912#4578455, @Milimetric wrote: > Hm, @bd808 do you think it would actually be useful to address issues like that one by one or just wait for a better way to group pages with all their redirect... [02:10:50] 10Analytics, 10EventBus, 10WMF-JobQueue, 10Wikimedia-production-error: EventBus error "Unable to deliver all events: (curl error: 28) Timeout was reached" - https://phabricator.wikimedia.org/T204183 (10Krinkle) [05:48:33] 10Analytics, 10EventBus, 10WMF-JobQueue, 10Wikimedia-production-error: EventBus error "Unable to deliver all events: (curl error: 28) Timeout was reached" - https://phabricator.wikimedia.org/T204183 (10Pchelolo) [05:48:37] 10Analytics, 10EventBus, 10Patch-For-Review, 10Services (next): Timeouts on event delivery to EventBus - https://phabricator.wikimedia.org/T180017 (10Pchelolo) [05:49:26] 10Analytics, 10EventBus, 10Patch-For-Review, 10Services (next): Timeouts on event delivery to EventBus - https://phabricator.wikimedia.org/T180017 (10Pchelolo) p:05Triage>03High Given that it reappeared again in T204183 I'm inclined to raise the priority. [06:22:37] 10Quarry, 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): Let quarry use the mariadb module - https://phabricator.wikimedia.org/T181205 (10jcrespo) why not modify mariadb::packages and use that instead? Was there a blocker for that? Packages is supposed to be mostly for non mw core db ins... [06:58:47] 10Analytics, 10Analytics-Kanban: Reboot Analytics hosts for kernel security upgrades - https://phabricator.wikimedia.org/T203165 (10elukey) [06:58:53] morning! [06:59:11] As FYI I just rebooted stat* and notebook* as announced to various email lists [07:34:27] 10Analytics, 10Analytics-Kanban: Reboot Analytics hosts for kernel security upgrades - https://phabricator.wikimedia.org/T203165 (10elukey) [08:31:15] and now I am rebooting kafka10[12-23] [08:31:41] I wish we could finally move monolog to jumbo and decom this cluster [09:05:31] 10Analytics: Upgrade bohrium (piwik/matomo) to Debian Stretch - https://phabricator.wikimedia.org/T202962 (10elukey) There seems to be a new upstream version - 3.6.0 - that contains some security fixes but it is not available via debian.matomo.org's mirror. https://github.com/matomo-org/matomo-package/issues/81... [10:08:12] * elukey early lunch + errand! [12:02:18] 10Analytics, 10DBA, 10Patch-For-Review: mariadb::service and managed services don't play well on Stretch - https://phabricator.wikimedia.org/T204074 (10elukey) First issue solved, now I can see the following one: ``` elukey@hadoop-coordinator-2:~$ sudo puppet agent -tv Info: Using configured environment 'pr... [12:21:47] 10Quarry: Quarry html output has wrong encoding - https://phabricator.wikimedia.org/T204226 (10IKhitron) [12:34:31] 10Quarry: Quarry html output has wrong encoding - https://phabricator.wikimedia.org/T204226 (10Aklapper) Please always include steps to reproduce in bug reports. [12:40:00] 10Quarry: Quarry html output has wrong encoding - https://phabricator.wikimedia.org/T204226 (10IKhitron) I do not see what to write here, but here you are: # Run 27365. # Check html output. [13:22:19] 10Analytics, 10User-Elukey: Return to real time banner impressions in Druid - https://phabricator.wikimedia.org/T203669 (10elukey) @AndyRussG Hi! I'd like to ask you a couple of questions before starting to work on this task.. are `eventlogging_CentralNoticeImpression` and `eventlogging_LandingPageImpression`... [13:32:44] 10Analytics, 10Contributors-Analysis, 10Product-Analytics: Streamline Superset signup and authentication - https://phabricator.wikimedia.org/T203132 (10Ottomata) When superset (and Flask) auto create the user account in superset on first LDAP login, the only information they have from ldap is the username.... [13:38:04] 10Quarry: Quarry html output has wrong encoding - https://phabricator.wikimedia.org/T204226 (10Aklapper) Clear steps with links, so people don't have to find out themselves (I guess that's https://quarry.wmflabs.org/query/27365 now that I've taken a look at Quarry), browser information and browser version inform... [13:38:39] 10Analytics, 10EventBus, 10WMF-JobQueue, 10Wikimedia-production-error: EventBus error "Unable to deliver all events: (curl error: 28) Timeout was reached" - https://phabricator.wikimedia.org/T204183 (10Ottomata) Luca did some restarts of the main Kafka clusters recently, I wonder if these correlate to thos... [13:40:44] 10Quarry: Quarry html output has wrong encoding - https://phabricator.wikimedia.org/T204226 (10IKhitron) I understand your problem. Usually it's OK. But now the bug report goes to one particular person, that know all that. Here is your link: [[https://quarry.wmflabs.org/run/290198/output/0/html?download=true]]. [13:48:18] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Test role::analytics_cluster::coordinator on Debian Stretch - https://phabricator.wikimedia.org/T204060 (10elukey) [13:48:22] 10Analytics, 10DBA, 10Patch-For-Review: mariadb::service and managed services don't play well on Stretch - https://phabricator.wikimedia.org/T204074 (10elukey) 05Open>03Resolved a:03elukey Of course if I don't grep the latest operations/puppet code I will not find what I am looking for: ``` git grep m... [13:48:47] oh a-team, i'm going to go to this 5 minute tech fun demo meeting [13:48:52] and show spark structured streaming [13:48:55] just for fun! [13:49:30] nice! [13:50:54] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Test role::analytics_cluster::coordinator on Debian Stretch - https://phabricator.wikimedia.org/T204060 (10elukey) The mariadb issues seems resolved, but now we have this: ``` elukey@hadoop-coordinator-2:~$ sudo puppet agent -tv Info: Using configured env... [13:50:55] ottomata: --^ :( [13:51:27] maybe I can add a hack that defaults to eqiad for labs? [13:51:41] 10Quarry: Quarry html output has wrong encoding - https://phabricator.wikimedia.org/T204226 (10IKhitron) N3: About you asked: Last Quantum on Windows 7. [13:52:18] elukey: i saw your thing there, i'm looking for where that could be. OH its the camus stuff? [13:52:22] sorry i was looking in mariadb classes [13:52:23] ok [13:52:37] yes, it should just be skipped in labs [13:52:39] elukey: +1 [13:53:04] elukey: actually the monitor check shoudl just be disabled in labs [13:53:12] sorry i cwant to prep for this meeting... [13:53:16] will help more after [13:53:43] sure sure! [13:53:58] I got what you are saying, I'll make a patch now [14:08:22] 10Quarry: Quarry html output has wrong encoding - https://phabricator.wikimedia.org/T204226 (10zhuyifei1999) View > Text Encoding > Unicode. The file is UTF-8 encoded. [14:10:40] 10Quarry: Quarry html output has wrong encoding - https://phabricator.wikimedia.org/T204226 (10IKhitron) Hi, @zhuyifei1999, and thank you. Could you please add an explanation somewhere in the output? [14:15:47] 10Quarry: Quarry html output has wrong encoding - https://phabricator.wikimedia.org/T204226 (10Aklapper) >>! In T204226#4580670, @IKhitron wrote: > I understand your problem. Usually it's OK. But now the bug report goes to one particular person, that know all that. @IKhitron: No, nobody else knows which browser... [14:17:03] 10Quarry: Quarry html output has wrong encoding - https://phabricator.wikimedia.org/T204226 (10IKhitron) Very well. [14:17:07] 10Quarry: Quarry html output has wrong encoding - https://phabricator.wikimedia.org/T204226 (10Aklapper) >>! In T204226#4580744, @IKhitron wrote: > N3: About you asked: Last Quantum on Windows 7. For future reference, please provide exact version information. "Last" means nothing (ESR? stable? nightly? aurora?)... [14:18:14] 10Quarry: Quarry html output has wrong encoding - https://phabricator.wikimedia.org/T204226 (10IKhitron) Very well. Yes. [14:23:16] 10Quarry: Quarry html output has wrong encoding - https://phabricator.wikimedia.org/T204226 (10zhuyifei1999) >>! In T204226#4580805, @IKhitron wrote: > Could you please add an explanation somewhere in the output? Well, why? The HTML file itself is okay. If you open it with a random Unicode text editor it'll s... [14:24:38] 10Quarry: Quarry html output has wrong encoding - https://phabricator.wikimedia.org/T204226 (10Aklapper) Proposing to decline this task as this seems to be either a local Firefox setting to overwrite the charset, or a mistake in Firefox code (which I cannot reproduce). [14:25:19] 10Quarry: Quarry html output has wrong encoding - https://phabricator.wikimedia.org/T204226 (10IKhitron) > Honestly, I would prefer just get rid if this ?download=true and leave the url as https://quarry.wmflabs.org/run/290198/output/0/html, which works fine despite having the exact same binary contents. Great i... [14:26:55] ottomata: really interesting! [14:27:10] ottomata: does it create extra topics to support the whole thing in kafka? [14:31:30] elukey: no! [14:31:38] that makes it nicer than kafka streams eh? [14:31:51] kafka streams uses kafka for all intermediate stuff [14:32:05] for all distributed intermediate stuff* [14:32:10] spark has its own framework for that kinda thing [14:32:12] so it odesn't have to [14:37:37] niceeee [14:40:41] 10Analytics, 10Community-consensus-needed: Decide whether enable per-editor edits stats (community decision) - https://phabricator.wikimedia.org/T203826 (10Cirdan) >>! In T203826#4578764, @Nuria wrote: > @Cirdan: the harhsness of your responses is really uncalled for. I did not intend for my responses to be h... [14:55:27] 10Analytics: Upgrade bohrium (piwik/matomo) to Debian Stretch - https://phabricator.wikimedia.org/T202962 (10akosiaris) [14:55:32] 10Analytics, 10Operations, 10vm-requests, 10Patch-For-Review: eqiad (1) - VM request for Piwik/Matomo - https://phabricator.wikimedia.org/T202963 (10akosiaris) 05Open>03Resolved @elukey VM is up and running. No role assigned in puppet so you probably want to handle that. Resolving this. [14:58:26] 10Quarry: Quarry html output has wrong encoding - https://phabricator.wikimedia.org/T204226 (10zhuyifei1999) 05Open>03declined (Thinking about it again, it's 'Download data' not 'Format data') [14:59:36] 10Quarry: Quarry html output has wrong encoding - https://phabricator.wikimedia.org/T204226 (10IKhitron) Still, you can add one more option. [15:03:46] 10Quarry: Quarry html output has wrong encoding - https://phabricator.wikimedia.org/T204226 (10zhuyifei1999) One more option of what? https://quarry.wmflabs.org/query/27365 already renders the results into html. [15:05:41] 10Quarry: Quarry html output has wrong encoding - https://phabricator.wikimedia.org/T204226 (10IKhitron) As you suggested, without the parameter. It is not urgent know, because you solved the problem with encoding, but maybe you can thing about it. And also about the warning I mentioned. Thank you. [15:22:26] q about EventLogging: can I set it up so it only logs to a file, not the daemon on a private wiki? [15:24:20] hexmode: sure! [15:24:33] oh [15:24:36] via the extension itself? [15:24:51] not sure about that. the EventLogging server can be configured to write just to a file [15:25:08] The extension I don't think so, not without modifying the code to do so [15:25:44] ottomata: I was afraid of that, but then I said, hmm, that may be just the thing to do [15:26:10] I saw that it used to have a file but doesn't do anything with it now... [15:26:27] hexmode: how are you running the server? [15:26:32] do you know? [15:28:20] ottomata: I'm trying to avoid running yet-another-daemon just to handle logging. Worse, I just want some debugging for a small bit of code for a production server I have no direct control of. [15:30:08] 10Analytics, 10User-Elukey: Return to real time banner impressions in Druid - https://phabricator.wikimedia.org/T203669 (10AndyRussG) >>! In T203669#4580601, @elukey wrote: > @AndyRussG Hi! I'd like to ask you a couple of questions before starting to work on this task.. are `eventlogging_CentralNoticeImpressio... [15:31:57] 10Analytics, 10User-Elukey: Return to real time banner impressions in Druid - https://phabricator.wikimedia.org/T203669 (10elukey) All right so IIUC for this moment I should just use webrequest :) [15:43:38] 10Analytics, 10Analytics-Wikimetrics, 10Security-Reviews: security review of Wikimetrics {dove} - https://phabricator.wikimedia.org/T76782 (10chasemp) >>! In T76782#4045746, @Bawolff wrote: > Umm, so this was filed in 2014. What is wikimetrics? Is it something that (still) needs a security review? @charlott... [15:47:51] hexmode: you want remote logging then? if you don't have control of the server i'm assuming you can't log in? [15:47:56] but, you can deploy eventlogging code to it? [15:51:16] (03PS3) 10Ottomata: Use ConfigHelper for Refine job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/460108 (https://phabricator.wikimedia.org/T203804) [15:51:54] (03CR) 10Mforns: "Looks good to me overall! I think the code is well distributed, and easy to understand. It's not trivial, but I believe that's because ann" (033 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/440971 (https://phabricator.wikimedia.org/T194705) (owner: 10Milimetric) [15:54:52] (03PS2) 10Fdans: Add total article count metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/458784 (https://phabricator.wikimedia.org/T203180) [16:00:34] (03CR) 10jerkins-bot: [V: 04-1] Add total article count metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/458784 (https://phabricator.wikimedia.org/T203180) (owner: 10Fdans) [16:02:38] 10Quarry, 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): Let quarry use the mariadb module - https://phabricator.wikimedia.org/T181205 (10zhuyifei1999) >>! In T181205#4579461, @jcrespo wrote: > why not modify mariadb::packages and use that instead? Was there a blocker for that? Packages... [16:05:35] ottomata: standup? [16:13:40] ACK [16:13:41] oh man [16:13:42] so late [16:14:18] 10Quarry, 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): Let quarry use the mariadb module - https://phabricator.wikimedia.org/T181205 (10jcrespo) > profile 'profile::quarry::database' includes non-profile class mariadb::packages That is probably pedantic, to use class { 'mariadb::packa... [16:15:13] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Remove webrequest misc analytics related jobs and code after cache misc -> text merge is complete - https://phabricator.wikimedia.org/T200822 (10leila) @elukey I can't make the call for deleting it until I have the confirmation from @mkroetzsch . I've alre... [16:21:46] Wooow -- We're not into DL yet but this very interesting: https://engineering.linkedin.com/blog/2018/09/open-sourcing-tony--native-support-of-tensorflow-on-hadoop [16:22:53] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Remove webrequest misc analytics related jobs and code after cache misc -> text merge is complete - https://phabricator.wikimedia.org/T200822 (10Ottomata) It's already been broken for a few weeks. We don't need to delete the data at all, but the change Lu... [16:22:57] And another one for my yamler (so true) https://twitter.com/Caged/status/1039937162769096704 [16:27:25] 10Analytics, 10DBA, 10Growth-Team, 10Notifications: Purge all Schema:Echo data after 90 days - https://phabricator.wikimedia.org/T128623 (10Milimetric) p:05Triage>03High [16:27:38] 10Analytics, 10Analytics-Kanban, 10DBA, 10Growth-Team, 10Notifications: Purge all Schema:Echo data after 90 days - https://phabricator.wikimedia.org/T128623 (10Milimetric) a:03elukey [16:27:47] 10Analytics, 10Analytics-Kanban, 10DBA, 10Growth-Team, 10Notifications: Purge all Schema:Echo data after 90 days - https://phabricator.wikimedia.org/T128623 (10Milimetric) p:05High>03Normal [16:30:33] 10Analytics, 10Community-consensus-needed: Decide whether enable per-editor edits stats (community decision) - https://phabricator.wikimedia.org/T203826 (10Milimetric) p:05Triage>03Low [16:31:23] 10Analytics: Fix download-project-namespace-map script to send alert if it fails - https://phabricator.wikimedia.org/T203824 (10Milimetric) p:05Triage>03High [16:32:48] 10Analytics: Sqoop e-mail is emailing errors in try1 for actions that suceeed in try 3 - https://phabricator.wikimedia.org/T203811 (10Milimetric) p:05Triage>03Normal [16:35:05] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Remove webrequest misc analytics related jobs and code after cache misc -> text merge is complete - https://phabricator.wikimedia.org/T200822 (10leila) >>! In T200822#4581325, @Ottomata wrote: > Luca, I suggest removing the job and if we hear back otherwi... [16:37:24] wow nice joal [17:34:18] (03PS2) 10Elukey: Remove cache misc from Refinery [analytics/refinery] - 10https://gerrit.wikimedia.org/r/459827 (https://phabricator.wikimedia.org/T200822) [17:34:33] ottomata: --^ removed wdqs stuff [17:37:20] joal: do you think that we should merge the cache misc removal post offsite? [17:44:43] (03PS4) 10Ottomata: Use ConfigHelper for Refine and EventLoggingSanitization [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/460108 (https://phabricator.wikimedia.org/T203804) [17:44:53] cool [17:44:59] elukey: wdqs is finished as well? [17:45:04] nice !! [17:45:12] elukey: i think removing cache misc jobs is unlikely to break anytyhing [17:45:16] since there isn't data there anymore [17:45:25] buuuut, famous last words right? [17:45:33] i guess its not hurting us at all to leave it for now [17:46:10] elukey, ottomata: as you prefer - We can do it tonight or tomorrow if you want, or post-offsite [17:46:25] (03PS5) 10Ottomata: Use ConfigHelper for Refine and EventLoggingSanitization [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/460108 (https://phabricator.wikimedia.org/T203804) [17:47:44] (03PS6) 10Ottomata: Use ConfigHelper for Refine and EventLoggingSanitization [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/460108 (https://phabricator.wikimedia.org/T203804) [17:47:50] i'm not working tomorrow yall! [17:47:57] so i'll see you on Sunday (AND LUCA ON SATURDAY!) [17:49:09] mforns: if you have a min: https://gerrit.wikimedia.org/r/#/c/analytics/refinery/source/+/460108/ [17:49:43] (03PS7) 10Ottomata: Use ConfigHelper for Refine and EventLoggingSanitization [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/460108 (https://phabricator.wikimedia.org/T203804) [17:50:02] let's do it post-offsite :) [17:50:23] or joal it could be something that we can pair in NYC as hacking time if you have patience [17:50:29] so it would be a good refresh for me [17:51:41] all right going offline folks! Talk with the EU team tomorrow, see the others in NYC! :) [17:51:55] joal: OR if marcel is gone and U have a minute: https://gerrit.wikimedia.org/r/#/c/analytics/refinery/source/+/460108/ :p [17:52:06] hey ottomata I'm here [17:53:38] didn't get your previous ping, looking [17:54:55] ottomata: if you have time later on tell me what you think about https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/460399/ :) [17:55:00] going off o/ o/ [18:01:12] byee luca see you soon! [18:20:30] (03PS8) 10Ottomata: Use ConfigHelper for Refine and EventLoggingSanitization [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/460108 (https://phabricator.wikimedia.org/T203804) [19:10:08] (03CR) 10Mforns: [C: 032] "LGTM! Just left a typo comment on patch 7. I didn't test it though (not verified)!" (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/460108 (https://phabricator.wikimedia.org/T203804) (owner: 10Ottomata) [19:16:27] 10Analytics, 10Community-consensus-needed: Decide whether enable per-editor edits stats (community decision) - https://phabricator.wikimedia.org/T203826 (10Nuria) @Cirdan OK understood, we will wait for input of others , pinging @Nemo_bis for his opinion on this regard [19:16:55] (03Merged) 10jenkins-bot: Use ConfigHelper for Refine and EventLoggingSanitization [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/460108 (https://phabricator.wikimedia.org/T203804) (owner: 10Ottomata) [19:20:13] 10Quarry, 10Patch-For-Review: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 (10Framawiki) 05Open>03Resolved a:03Framawiki The migration is complete ! \o/ [19:25:17] 10Analytics, 10Operations, 10ops-eqiad: analytics1068 doesn't boot - https://phabricator.wikimedia.org/T203244 (10Cmjohnson) attempted to update bios but the update does not run and the server remains stuck in the "Loading BIOS Drivers" during post [19:26:01] OH mforns it got merged :p :) [19:26:02] but ok! [19:26:14] i'll make your fix when I update the changelog too [19:26:15] ottomata, oh..... [19:26:18] thank you ! [19:26:35] no need to change that alone, I can do it next time I modify that code [19:27:15] i need to make a changelog.md update anyway [19:34:49] 10Quarry: Quarry html output has wrong encoding - https://phabricator.wikimedia.org/T204226 (10Framawiki) 05declined>03Open If I understand correctly with the ?download=true firefox downloads the html file and then open it, but without the good encoding. Content Type is not kept during the file save. Without... [19:35:03] 10Quarry: Get ride of this ?download=true parameter - https://phabricator.wikimedia.org/T204226 (10Framawiki) [19:44:12] (03PS1) 10Ottomata: Update changelog for 0.0.75 ConfigHelper changes [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/460418 (https://phabricator.wikimedia.org/T203804) [19:58:19] (03CR) 10Ottomata: [C: 032] Update changelog for 0.0.75 ConfigHelper changes [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/460418 (https://phabricator.wikimedia.org/T203804) (owner: 10Ottomata) [20:08:34] 10Quarry: A fork that has never been executed otself is stuck on Checking query status - https://phabricator.wikimedia.org/T204277 (10Framawiki) [20:15:50] 10Quarry: A fork that has never been executed itself is stuck on Checking query status - https://phabricator.wikimedia.org/T204277 (10Framawiki) [20:21:47] (03PS4) 10Joal: Cleanup scala input format [analytics/wikihadoop] - 10https://gerrit.wikimedia.org/r/337004 [20:22:19] ottomata: --^ Dead code is gone, doc have been added, and code somehow revamped - Hopefully more understandable (even if still complex) [20:22:53] ooooboy [20:23:00] ottomata: I now think that, instead of adding the spark job to wikihadoop, I'll add wikihadoop to refinery and add the job in refinery-job [20:23:10] ottomata: thoughts? --^ [20:26:23] hm, sure! if you prefer [20:26:31] why tho? [20:26:47] ottomata: prevent wikihadoop to be dependent of spark [20:26:51] but can be [20:28:34] joal: the wikihadoop now is just the hadoop input format stuff, right? [20:28:39] correct [20:28:45] could you keep that separate and use it? [20:28:53] or would you abandon the separate wikihadoop altogether? [20:29:08] 10Analytics, 10Community-consensus-needed: Decide whether enable per-editor edits stats (community decision) - https://phabricator.wikimedia.org/T203826 (10mforns) Hi all :] I agree with @Cirdan that, generally speaking, monthly and daily editing statistics per individual user, "should remain as hard to come... [20:29:23] nono, keep wikihadoop as is, add the dependency to refinery-job, and add the spark job in refinery-job [20:29:29] ottomata: --^ [20:29:29] ahhh i see [20:29:31] that makes sense [20:29:35] the spark job will be the part that is converting to parquet? [20:29:36] great [20:29:39] yes [20:29:41] from the input key/vals? [20:30:16] joal! what is this notation? [20:30:16] Hey anybody got an old spare 26" polo tire (or 2?) lying around? My old bike / sometimes loaner is running on threads. It'd be nice to have to useable for guests and/or maybe @ [20:30:17] oops [20:30:18] hahahaha [20:30:19] wrong paste [20:30:19] I actually want to try better implementations (for now, parse XML into a Map, convert to Json, then to parquet [20:30:25] MwObjectsFactory#MwRev [20:30:25] ? [20:30:28] # ??? [20:31:04] This is the notation to reference a inner type (the MwRev type of the MwObjectsFactory type) [20:34:12] huh [20:34:15] Joal why not . [20:34:16] ? [20:37:14] I don't know !! [20:39:17] ottomata: my understanding is that . is for objects or classes (instances), and # for types definition [20:39:41] we use . in refinery stuff [20:39:49] ottomata: for types? [20:39:52] type TransormFunction = DataFrameToHive.TransformFunction [20:40:37] hmmm, mabye that's because DataFrameToHive is an object? [20:40:40] instead of a class? [20:40:57] if you want to access an inner type defined in a class, (without an instance) i guess you need # ? [20:41:20] ottomata: That's my understanding [20:41:37] ottomata: Even more than in a class - In a trait (no instanciation !) [20:41:43] (03PS1) 10Framawiki: view.js: explain that this query has never yet been executed [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460433 (https://phabricator.wikimedia.org/T204277) [20:42:30] aye [20:42:32] ah ha! [20:42:34] https://www.scala-lang.org/old/node/115 [20:42:36] we are right [20:42:57] if you have instance you can use . if you want to access inner type in a type(class, trait, whatever), use # [20:43:09] and an object is a singleton class insgtance so yaa makes sense [20:43:15] yop [20:44:11] joal i'd still like a nice readme outlining the steps and purposes of each thing. it only now became clear to me (i think) that you have these different factory types for vary different purposes (right?) [20:44:14] object, map, case class [20:44:20] i don't fully understand why you need each one and when [20:44:41] (i guess for transforming from json to map to case class? ) [20:45:22] ottomata: this is where I wonder about over-enginerring - as of now, I only use the Map one, but I think it would be good to use the class one for optimization [20:45:57] Now, does it mean having this level of genericity, probably not (over-eng) - But now that it's there ... [20:45:58] you don't use it yet? because you don't convert to json? [20:46:10] ottomata: I use Map to convert to Json [20:46:25] ok, and the case classes would be used for? [20:46:44] ottomata: And I'd like to make the case classes Writable to prevent having to convert to json [20:47:08] Like that, you can parse into a case class, and do whatever you like after [20:47:25] But having the Map was the easier and first solution I put in place [20:47:52] (03PS2) 10Framawiki: view.js: explain that query has never been executed [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460433 (https://phabricator.wikimedia.org/T204277) [20:49:39] joal: i see. what do you get the data into from the input format to begin with? the objects? [20:50:12] hm - not sure I understand [20:51:07] So, The InputFormat instanciate a RecordReader per split. Each RecordReader uses a MwParser - Which is parameterized by the MwObjectsFactory [20:51:18] So basically, it MwObjectsFactory from the bottom up [20:51:36] Whichever implementation of it you choose [20:52:19] AH i see [20:53:19] OHHH i see [20:53:22] sorry [20:53:27] MediawikiObjectsFactory is a trait [20:53:51] so the parser could bring the xml into the maps or into the case classes [20:53:56] cool. [20:54:04] joal: don't see why not have it like this [20:54:04] but [20:54:06] Correct - I actually have a test for that :) [20:54:07] GIVE ME A README! [20:54:11] :D [20:54:12] so that is clear ^^^^ :p [20:54:30] I'll give you a READ..YOU ? [20:54:35] ok - I'll do that [20:54:38] hahaha [20:54:39] ok cool [20:54:54] i gotta run pretty sooooon, i'm not working tomorrow so i'll see you SUNDAY! [20:55:07] Thanks for the quick notes :) Will do the README tomorrow :) [21:04:27] (03CR) 10Zhuyifei1999: view.js: explain that query has never been executed (031 comment) [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460433 (https://phabricator.wikimedia.org/T204277) (owner: 10Framawiki) [21:11:38] 10Quarry, 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): Let quarry use the mariadb module - https://phabricator.wikimedia.org/T181205 (10zhuyifei1999) 05Open>03Resolved a:03zhuyifei1999 [21:16:40] (03PS3) 10Framawiki: view.js: explain that query has never been executed [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460433 (https://phabricator.wikimedia.org/T204277) [21:17:21] (03CR) 10Framawiki: view.js: explain that query has never been executed (031 comment) [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460433 (https://phabricator.wikimedia.org/T204277) (owner: 10Framawiki) [21:19:43] 10Analytics, 10Discovery-Search (Current work), 10Patch-For-Review: Use kafka for communication from analytics cluster to elasticsearch - https://phabricator.wikimedia.org/T198490 (10debt) [21:19:45] 10Analytics, 10Discovery-Search (Current work), 10Patch-For-Review: Deploy mjolnir msearch daemon to the elasticsearch clusters - https://phabricator.wikimedia.org/T200740 (10debt) 05Open>03Resolved [21:22:39] 10Analytics, 10Discovery-Search (Current work), 10Patch-For-Review: Use kafka for communication from analytics cluster to elasticsearch - https://phabricator.wikimedia.org/T198490 (10debt) 05Open>03Resolved [21:37:48] (03CR) 10Zhuyifei1999: [C: 032] view.js: explain that query has never been executed [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460433 (https://phabricator.wikimedia.org/T204277) (owner: 10Framawiki) [21:38:06] (03Merged) 10jenkins-bot: view.js: explain that query has never been executed [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460433 (https://phabricator.wikimedia.org/T204277) (owner: 10Framawiki) [21:38:08] (03PS1) 10Zhuyifei1999: results: Fixes for memory consumption and bytes support [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460443 [21:41:02] 10Quarry, 10Patch-For-Review: A fork that has never been executed itself is stuck on Checking query status - https://phabricator.wikimedia.org/T204277 (10zhuyifei1999) 05Open>03Resolved [21:46:13] 10Quarry, 10Patch-For-Review: Update dependencies - https://phabricator.wikimedia.org/T192731 (10zhuyifei1999) 05Open>03Resolved a:03Framawiki [21:46:42] (03PS2) 10Zhuyifei1999: results: Fixes for memory consumption and bytes support [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460443 (https://phabricator.wikimedia.org/T192696) [22:03:05] 10Quarry: puppet disabled on quarry-main-01 - https://phabricator.wikimedia.org/T136315 (10zhuyifei1999) 05Open>03Invalid Old instance deleted as part of T194691. For the record: I never saw puppet disabled on this instance before T202588. [22:04:43] 10Quarry, 10Patch-For-Review: Remove dependency on labs_debrepo - https://phabricator.wikimedia.org/T153615 (10Dzahn) [22:04:47] 10Quarry, 10Patch-For-Review: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 (10Dzahn) [22:10:52] 10Quarry, 10Patch-For-Review: Remove dependency on labs_debrepo - https://phabricator.wikimedia.org/T153615 (10zhuyifei1999) 05Open>03Resolved a:03zhuyifei1999 [23:03:26] (03PS3) 10Zhuyifei1999: results: Fixes for memory consumption and bytes support [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460443 (https://phabricator.wikimedia.org/T192696) [23:09:20] (03PS4) 10Zhuyifei1999: results: Fixes for memory consumption and bytes support [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460443 (https://phabricator.wikimedia.org/T192696) [23:13:45] (03CR) 10Zhuyifei1999: "I did a test on how much performance penalty flushing on every write has:" [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460443 (https://phabricator.wikimedia.org/T192696) (owner: 10Zhuyifei1999) [23:28:41] (03PS5) 10Zhuyifei1999: results: Fixes for memory consumption and bytes support [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460443 (https://phabricator.wikimedia.org/T192696) [23:32:11] (03CR) 10Zhuyifei1999: "Testing with:" [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460443 (https://phabricator.wikimedia.org/T192696) (owner: 10Zhuyifei1999)