[00:30:33] (03PS1) 10MaxSem: Add another schema version [analytics/discovery-stats] - 10https://gerrit.wikimedia.org/r/325719 (https://phabricator.wikimedia.org/T152513) [11:45:06] 06Analytics-Kanban: Puppetize clickhouse - https://phabricator.wikimedia.org/T150343#2853757 (10elukey) I managed to build version `54010` but not more recent "stable" ones, due to various build errors (see https://github.com/yandex/ClickHouse/issues/228 for more info). Lintian returns some concerns: ``` eluke... [12:00:33] 06Analytics-Kanban: Puppetize clickhouse - https://phabricator.wikimedia.org/T150343#2853772 (10elukey) Summary: All the Clickhouse dependencies are built statically in one binary that depends only on libc at runtime (verified with ldd). Available things: 1) A Trusty Debian package that can theoretically work... [12:19:03] 10Analytics, 10EventBus, 06Services (watching): Allow multiple schemas in a single EventBus topic. - https://phabricator.wikimedia.org/T152557#2853789 (10mobrovac) I'm generally +1 on the idea, but this might complicate the consumer side of the story. If a topic may have multiple schemas, then the consumer n... [12:37:34] * elukey lunch! [13:22:29] joal: o/ [13:22:44] whenever you have time I'd need to know some details about clickhouse [13:22:52] like where things are stored, etc.. [13:22:59] I am reading the config.xml [13:23:06] BUT I could use some help :) [13:23:32] the debian packaging has some issues, so while waiting I am going to write the puppet code [13:24:23] Heya elukey [13:25:13] I think I'm not the best to answer: while I can explain to you the difference between the MergeTree, log and distributed engines, I don't really know uch about the internal [13:25:37] elukey: Probably best to ask ottomata how he confirgured the original instances [13:26:53] mforns: Hi ! [13:27:48] mforns: the correction we found yesterday reduced massivly the issue, but there is some (58 instead of ~26k over 3g events) [13:27:49] * elukey blames ottomata [13:27:51] :D [13:27:52] :P [13:28:03] * elukey blames joal as well [13:28:06] :D [13:28:17] elukey: But if you want I can tell you more about how to make the thing work ;) [13:28:40] joal, awesome [13:28:52] :] [13:29:08] joal: nono I am reading nice xmls, I am super happy :) [13:29:13] mforns: I'm planning to spend some time this evening investigating why there still are some diffs [13:29:20] hehehe elukey [13:29:31] I am wondering how it is possible that ports are opened between the druid hosts [13:29:42] but there must be some ferm things that I don't get [13:29:48] anyhowwww [13:29:55] joal, OK, but 58 is acceptable no? [13:30:25] mforns: completely acceptable, it's more about us understanding why [13:30:42] for the moment, I don't know why, and THAT is not acceptable :) [13:31:12] I'd rather have a log line that says - 58 removed because incorrect - instead of an unexplained diff in values that should be the same [13:31:17] mforns: --^ [13:31:41] joal, I see [13:31:49] mforns: makes sense? [13:32:20] Like, in pages and users history, we count parsing errors and unmatch events and report them -- That's cool [13:32:21] joal, makes sense! [13:32:30] cool :) [13:32:49] I guess though that there will always be corner cases that we do not understand... [13:32:53] mforns: I'm gently warming up for when ottomata arrives ;) [13:32:59] heheheheh [13:35:17] mforns: I'm writing comments in denormalization code for that ;) [13:35:29] ok [13:49:45] joal: qq - how much data is currently stored in clickhouse? [13:50:13] elukey: hm, I'm not sure [13:50:48] I am exploring the config and it seems all in Andrew's home dir, but can't find much [13:50:59] I am a bit ignorant about how/where/what data clickhouse stores [13:51:15] just to plan how to config it [13:51:21] and what partitions to use [13:51:31] elukey: in /var/lib/druid/clickhouse-otto-test/opt/clickhouse/data [13:51:39] ~500G in joal db [13:53:29] I didn't know this dir.. sigh [13:55:11] and now I am wondering where it is configure [13:55:14] *configured [13:56:16] elukey: I really don't know :( [13:58:47] elukey: just had a quick look: in /home/otto/clickhouse/base, opt -> /var/lib/druid/clickhouse-otto-test/opt [13:59:18] elukey: so, configured in /home/otto/clickhouse/base/etc/clickhouse-server/config.xml [14:00:15] joal: aahhh thanks it is a symlink! [14:00:19] now it makes sense [14:00:22] :) [14:00:27] I am a bit slow this afternoon [14:00:29] thanks :) [14:01:52] elukey: you're welcome ;) [14:14:30] gooood morning joal! checking email, then wanna do walkthrough? [14:14:49] ottomata: I've been warming up for you, let me know when you're READY ! ;) [14:14:54] k! :) [14:17:33] ottomata: o/ [14:17:38] whenever you have time https://phabricator.wikimedia.org/T150343#2853772 [14:17:44] I'd need your opinion [14:20:55] hmmm elukey, i lean toward option 1, but curious, why does it need Sid? [14:20:59] and not jessie? [14:21:23] gcc6/g++6 mostly [14:21:30] (or at least gcc5) [14:21:41] IIRC jessie is still on gcc 4.x [14:22:15] is thirdparty the right component for this use case? [14:22:24] I didn't find any indication when to use it in the wiki [14:22:24] that i don't know [14:22:27] yeah [14:22:32] i think maybe they wanted to phase out third party [14:22:34] def check with paravoid [14:22:41] sure sure :) [14:23:04] atm I am writing the puppet part [14:23:08] hahahah [14:23:09] https://wiki.debian.org/qa.debian.org/jsonevil [14:23:10] oh man [14:23:21] I know, weird :) [14:23:36] weirdest license ever [14:23:57] and it sjust build depends on that? [14:23:59] hm, [14:24:06] could we just put gcc6 into our apt? [14:24:09] for jessie? [14:24:12] and then build for jessie? [14:24:13] i would say: [14:24:19] if we have to build for a different distro than we use anyway [14:24:24] I think that we could probably use jessie + backports, but I haven't investigate.. [14:24:26] we might as well just use the already made trusty one [14:24:32] exactly [14:24:35] I thought the same.. [14:25:08] if it works 100% with the libc we ahve on jessie [14:25:13] then yeah, that would be easiest [14:25:19] well we are using it now :) [14:25:25] ja :) [14:25:55] it is more or less what Go does with builds [14:26:00] 06Analytics-Kanban: Puppetize clickhouse - https://phabricator.wikimedia.org/T150343#2854033 (10faidon) The better the Debian package is, the better for future maintainability, but I wouldn't say that good Debian packages are a requirement for us, especially for something with an as limited deployment as this on... [14:26:10] prometheus exporters are all built using sid [14:26:24] because jessie does not have the build dep yet [14:30:46] hm, well ok then [14:31:03] i don't have an opinion about the json evil thing [14:31:51] i think getting using either trusty, or building for sid, or including gcc6 in our jessie apt and building for jessie, are all fine solutions :) [14:31:54] haha [14:31:58] sorry, guess i'm not helping much [14:32:06] i wouldn't bother improving the deb packaging much at this time [14:32:18] from what I could tell, the actual installed files were decently organized [14:32:25] (its not shoving everyting in /opt or antyhing) [14:32:42] so, mostly this will be about upgrades, not installation. [14:33:03] so, if we want to get it puppetized sooner rather than later, and we can use the packaging as is (either the trusty .deb, or rebuilding), then we should do it [14:34:18] joal: i'm ready! [14:34:23] me too [14:34:41] yeah, I'll dig a bit more into this and then will find a solution [14:35:01] I'll proceed with the puppet code in the meantime [14:35:23] cool [14:35:39] ottomata: Yay ! [14:35:50] ottomata, milimetric, mforns, to the batcave ! [14:40:37] omw joal [15:53:34] Thanks ottomata for having followed that complex discutssion :) [15:53:45] That's cool to have to explain this thing :) [16:39:17] Hey milimetric, can I post the link to your file on Wikistats usages on SoS page? [16:50:38] 10Analytics, 10EventBus, 06Services (watching): Allow multiple schemas in a single EventBus topic. - https://phabricator.wikimedia.org/T152557#2854391 (10Ottomata) I mentioned this to Petr in IRC yesterday, but let's discuss here. I'm not opposed to implementing this, but if change-prop doesn't use the even... [16:51:44] 10Analytics, 10EventBus, 06Services (watching): Allow multiple schemas in a single EventBus topic. - https://phabricator.wikimedia.org/T152557#2854394 (10Ottomata) > add the field meta.schema which can be either populated by the proxy service or the producer `meta.schema_uri` is set by eventbus service, but... [17:25:28] 10Analytics, 10EventBus, 06Services (watching): Allow multiple schemas in a single EventBus topic. - https://phabricator.wikimedia.org/T152557#2854532 (10Pchelolo) > This also makes me wonder if we should even include the change-prop related topics in the topic -> schema mapping config. Since we have autom... [17:28:20] 10Analytics, 10EventBus, 06Services (watching): Allow multiple schemas in a single EventBus topic. - https://phabricator.wikimedia.org/T152557#2854560 (10Ottomata) The service doesn't automatically create topics, and we recently removed the `ensure-kafka-topics-exist` script from eventlogging. Didn't you te... [17:29:08] 10Analytics, 10EventBus, 06Services (watching): Allow multiple schemas in a single EventBus topic. - https://phabricator.wikimedia.org/T152557#2854567 (10Ottomata) > The service doesn't automatically create topics Let me rephrase: The service doesn't do anything special to create topics, it just produces t... [17:30:05] 10Analytics, 10Dumps-Generation, 05Security: Pageview dumps incorrectly formatted, looks like a result of possibly malicious activity - https://phabricator.wikimedia.org/T144100#2854569 (10dpatrick) p:05Triage>03Normal [17:30:40] 10Analytics, 10Dumps-Generation, 05Security: Pageview dumps incorrectly formatted, looks like a result of possibly malicious activity - https://phabricator.wikimedia.org/T144100#2854571 (10Bawolff) btw, sometimes people use the curid parameter to get to a page with an invalid title (e.g. if a bug causes a pa... [17:31:30] milimetric: plop? [17:31:39] hi joal [17:31:46] cave? [17:31:50] sure [17:31:52] 10Analytics, 10EventBus, 06Services (watching): Allow multiple schemas in a single EventBus topic. - https://phabricator.wikimedia.org/T152557#2854584 (10Pchelolo) @Ottomata Oh, right, right... I actually did check it but completely forgot about that :) [17:34:35] * milimetric going to grab lunch [17:49:20] 10Analytics, 10EventBus, 06Services (watching): Allow multiple schemas in a single EventBus topic. - https://phabricator.wikimedia.org/T152557#2854709 (10Ottomata) :) Should we decline this ticket make a patch to remove the change prop topic configs? [17:50:55] 10Analytics, 10EventBus, 06Services (watching): Allow multiple schemas in a single EventBus topic. - https://phabricator.wikimedia.org/T152557#2854717 (10Pchelolo) @Ottomata Let's see what @mobrovac has to say here, but overall I would agree. I will then put the CP schemas to CP repo under some `doc` folder.... [17:59:49] 10Analytics, 10EventBus, 06Services (watching): Allow multiple schemas in a single EventBus topic. - https://phabricator.wikimedia.org/T152557#2854753 (10Ottomata) I'm totally fine with keeping the schemas in the event-schemas repo. I think its nice to have them all in one place, and to go through the same... [18:24:56] 10Analytics, 10EventBus, 06Services (watching): Allow multiple schemas in a single EventBus topic. - https://phabricator.wikimedia.org/T152557#2854828 (10mobrovac) >>! In T152557#2854753, @Ottomata wrote: > I'm totally fine with keeping the schemas in the event-schemas repo. I think its nice to have them al... [18:36:24] 10Analytics-EventLogging, 06Analytics-Kanban, 10EventBus, 13Patch-For-Review: Ensure no dropped messages in eventlogging producers when stopping broker - https://phabricator.wikimedia.org/T142430#2854869 (10Ottomata) ! Ah! Increasing `sync_timeout` to 10.0 worked! The default of 2.0 was too little. I... [18:39:59] * elukey afk! [19:12:58] 10Analytics-EventLogging, 06Analytics-Kanban, 10EventBus, 13Patch-For-Review: Ensure no dropped messages in eventlogging producers when stopping broker - https://phabricator.wikimedia.org/T142430#2534697 (10Nuria) Nice! [19:34:06] 10Analytics, 06Operations, 07Puppet: Refactor eventlogging.pp role into multiple files (and maybe get rid of inheritance) - https://phabricator.wikimedia.org/T152621#2855084 (10Ottomata) [19:39:52] !log restarting analytics eventlogging to test out confluent kafka producer for processors [19:39:53] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:58:51] 10Analytics, 10Dumps-Generation, 05Security: Pageview dumps incorrectly formatted, looks like a result of possibly malicious activity - https://phabricator.wikimedia.org/T144100#2588550 (10Tgr) These are pageview requests which result in the successful display of an existing, valid page. HTTP 200 is the only... [20:03:12] 10Analytics-EventLogging, 06Analytics-Kanban, 10EventBus, 13Patch-For-Review: Ensure no dropped messages in eventlogging producers when stopping broker - https://phabricator.wikimedia.org/T142430#2855194 (10Ottomata) I've just been trying out the confluent kafka producer for sync analytics purposes. Looki... [20:18:03] 10Analytics, 10Dumps-Generation, 05Security: Pageview dumps incorrectly formatted, looks like a result of possibly malicious activity - https://phabricator.wikimedia.org/T144100#2588550 (10Anomie) >>! In T144100#2855187, @Tgr wrote: > There is no HTTP response code for "this content has a different canonical... [20:35:25] 10Analytics, 10Dumps-Generation, 05Security: Pageview dumps incorrectly formatted, looks like a result of possibly malicious activity - https://phabricator.wikimedia.org/T144100#2855268 (10Nuria) >These are pageview requests which result in the successful display of an existing, valid page. HTTP 200 is the o... [20:38:40] 06Analytics-Kanban, 06Operations, 07Puppet: Refactor eventlogging.pp role into multiple files (and maybe get rid of inheritance) - https://phabricator.wikimedia.org/T152621#2855278 (10Ottomata) [20:56:42] @milimetric @mforns @nuria model to review tomorrow is here: https://www.dropbox.com/s/xwj1z40hj7qnfcy/Wikistats%20Nav%20Model.png?dl=0 [20:56:54] thanks ashgrigas :] [20:56:57] you can comment right in dropbox if needed otherwise we'll chat tomorrow [20:57:05] ashgrigas: k [20:57:15] 9:30am PT [20:57:18] sure [20:58:08] 10Analytics, 10Dumps-Generation, 05Security: Pageview dumps incorrectly formatted, looks like a result of possibly malicious activity - https://phabricator.wikimedia.org/T144100#2855352 (10Tgr) Per RFC 2616, * (301) //The requested resource has been assigned a new permanent URI and any future references to t... [21:08:19] ottomata: aware of anything having problem on vk recently ? [21:08:45] ottomata: We got data loss warning emails on every webrequest_source [21:11:03] 10Analytics, 10Dumps-Generation, 05Security: Pageview dumps incorrectly formatted, looks like a result of possibly malicious activity - https://phabricator.wikimedia.org/T144100#2855427 (10Tgr) Filed {T152628}. As long as the special handling is limited to requests where the apparent page name is invalid, I... [21:14:53] hm joal i did restart a kafka broker a few times [21:15:04] but that shouldn't cause data loss [21:15:04] ottomata: weird [21:15:09] nope [21:15:46] hm [21:20:24] milimetric, mforns; i think this explains why some of our tests do not run: https://github.com/jasmine/jasmine/issues/941 [21:20:38] looking [21:20:42] ottomata: seems related to the new alerts mforns has devised [21:20:51] I assume too many instances of vk got restarted [21:20:52] hm [21:20:56] nuria: I'm seeing tests failing oto [21:20:57] *too [21:21:18] mforns, milimetric right they fail due to closure variables not being cleaned up [21:22:15] milimetric, mforns: I think is easy to fix. as far as i can see problem is just in 1 file [21:22:30] hm ottomata -- Actually, seems to have some real data loss :( [21:22:31] the api tests? [21:22:45] mmm [21:23:38] milimetric: yes [21:23:51] milimetric. mforns : teh scoping doesn't work like you might think [21:24:25] milimetric, mforns; somewhat explained here: https://jasmine.github.io/edge/introduction.html#section-The_this_keyword but not super clear [21:24:42] nuria: this change makes sense, some minor nits, but shouldn't we just refactor to use the sitematrix instead? [21:25:23] yeah, basically the "describe" scope is a fake global scope [21:25:25] ottomata: We should triple check camus with kafka restarts -- I think there have been some dataloss when you've been restarting brokers [21:25:39] milimetric: I think chewing up sitematrix into the custom format for the menu will just delay 1st paint plus will require custom code as ( i think) not all can be inferred [21:26:08] milimetric: so i prefer to leave it as is so we have files taylored for an speedy UI [21:26:09] joal: hm, yeah, 2.72551016% loss in text in hour 18 [21:26:22] hmmm, you think its camus? [21:26:23] hmmm [21:26:28] nuria: yeah, we have to port the reverse lookup building from wikimetrics, but the way it is means we don't get automatic updates, we have to go add wikis manually... [21:26:33] ottomata: biggy loss in every webrequest source at hour 18 [21:26:36] (as it is, it's already out of date by a few wikis [21:26:39] milimetric: I take that [21:26:58] milimetric: over having to rebuild it upon every request [21:26:58] ottomata: Triple checked also the new metric from mforns (counting '-') --> only text has some, and it's very small [21:27:05] but would it really be that slow? [21:27:11] milimetric, nuria, we now have several systems that have that same problem, need updating of wiki list [21:27:16] I mean, we still parse it even as-is [21:27:19] So those loss are not due to timestamp not being present and having fake order [21:27:24] ottomata: --^ [21:27:45] milimetric: ya, let's do it on another change, we now parse json and turn that into html [21:27:50] aye [21:27:52] the current implementation still traverses what comes back because the availableProjects wasn't tailored for dashiki, it was tailored for wikimetrics [21:27:54] ottomata: The fact that every source has loss with the almost same amouhnt doesn't smell good [21:27:57] joal: yeah, camus logs look pretty noremal though [21:27:58] yeah [21:27:58] taylored for Ui, that is pretty fast [21:28:05] looking [21:28:10] I can help if need [21:28:23] uh... I'm getting lost - wanna talk in cave nuria? [21:28:31] milimetric: sorry i cannot [21:28:39] ok, I'll comment on patch [21:29:06] lol, gerrit is down [21:29:09] milimetric: but anyways i do not think we should process the sitematrix to turn it into a menu every time, seems unnecessary [21:29:10] milimetric, nuria, I'll try to help in oozie alarms [21:29:26] k [21:29:31] milimetric: and likely to add overhead [21:29:40] milimetric: but probably that should be proven with numbers [21:30:11] yeah, if we were talking about parsing vs. pure fetch, I'd agree maybe. But we're talking parsing vs. parsing [21:30:15] milimetric: so let's get rid of dependency on labs and after do refactor [21:30:30] sure, that's ok [21:30:35] milimetric: well, no the sitematrix is fetched too , with a ttl of 1 hour [21:30:35] but let's fix the tests first [21:30:42] ottomata, joal: the error file : wmf/data/raw/webrequests_data_loss/maps/2016/12/7/18/WARNING/000000_0 says: 0 requests (0.0% of total) have incomplete records. 30210 requests (2.321% of valid ones) were lost. [21:30:45] so gerrit is down but basically my comment would be: -1 to fix the tests [21:30:57] milimetric: tests are broken in master [21:31:02] milimetric: not in thsi patch [21:31:04] *this [21:31:05] nuria: no, I mean what we're doing now with available-projects is actually parsing and re-shaping [21:31:18] mforns: Yes, but I looked at text first, that a few incomplete, misleading me :) [21:31:20] this would happen with holes in sequence numbers [21:31:29] yeah, looking at sequence stats [21:31:30] see that too [21:31:35] yup [21:31:38] looking at vk logs for one of the hosts that lost stuff [21:31:40] you're right, nuria, I'll file a bug [21:31:43] issue seems a real one [21:31:43] so then +2 [21:31:44] i see lots of connection refused to the host that i restarted [21:31:45] wait maybe other hours [21:31:46] but that's expected [21:31:50] milimetric: i got it, I am fixing those [21:31:53] i don't see logs about failed produces [21:31:57] k [21:31:58] milimetric: no worries i will do it [21:31:59] which, i think it what we've seen when vk drops messages [21:32:04] mforns: I checked ;) [21:32:10] but, joal these correlate with when i restarted kafka [21:32:19] yes, makes sense [21:32:38] ok, so potential loss comes from vk not being able to reconnect? [21:32:56] (03CR) 10Milimetric: [V: 032 C: 032] "TODO in the near future: refactor available-projects to just fetch sitematrix and reshape that." [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/325182 (https://phabricator.wikimedia.org/T136120) (owner: 10Nuria) [21:33:06] joal: i'm not totally sure. it seems like they must have been produce errors [21:33:10] or else we would see something wrong with camus [21:33:21] camus seems to have consumed everything in kafka fine, at least at a cursory glance [21:33:24] ottomata: I think you're right [21:33:31] k [21:33:50] hmm - maybe vk is not as resilient as it was with new kafka version ? [21:33:58] new kafka version? [21:34:06] don't think we've changed anythign in a while [21:34:08] well the one you deployed months ago [21:34:11] oh [21:34:12] ya [21:34:13] maybe. [21:34:14] dunno [21:34:34] We've not gone through restarts regularly since the last update [21:34:41] it looks like it spit those errors out for about 1.5 minutes straight on this host, but i don't see any produce errors [21:34:42] hmmm [21:34:46] let's look at the vk dash! [21:34:46] :) [21:34:49] we've through a lot of vk changes, but no real kafka downtime [21:35:00] joal: don't think that's true, luca and moritz do restarts all the time [21:35:11] ottomata: True [21:35:13] for jvm security update stuff [21:35:26] yeah you're right ottomata [21:35:28] hm [21:36:12] joal: dunno [21:36:13] https://grafana.wikimedia.org/dashboard/db/varnishkafka?from=1481131209843&to=1481139147010&panelId=1&fullscreen&var-instance=webrequest [21:36:16] looks pretty good... [21:36:41] 06Analytics-Kanban: Dashiki tests broken on master - https://phabricator.wikimedia.org/T152631#2855509 (10Nuria) [21:37:54] nuria, milimetric, since when are dashiki tests broken? [21:38:11] apparently for a while [21:38:13] mforns: ahem ... must be a while but [21:38:24] mforns: didn't you say some did not execute? [21:38:29] they were working after the refactor and we didn't do anything since then really [21:38:31] mforns: i bet that is why we did not see it before [21:38:44] no, all of them passed after the refactor [21:38:49] nuria, yes, after the refactor, they were fine AFAIR [21:38:51] that's why I was happy but I guess this bug was hiding [21:39:09] milimetric, mforns: nah, they *seem* to pass sometimes [21:39:18] milimetric, mforns: but that is another bug [21:39:21] ottomata: looked again at camus logs - clean [21:39:23] ahem .. * think* [21:39:31] yeah [21:39:32] nuria, yes, before the refactor, some tests were not executing, but we fixed that [21:39:38] milimetric, mforns : i just saw that happening [21:39:49] hehehe, I believe you :] [21:40:02] yeah, ok, then there's this bug and it means we have to be more careful with this fake "describe" scope [21:40:08] or to switch testing frameworks [21:40:18] I never really liked how we use both jasmine and sinon since they both do the same thing [21:40:25] best part is when i was looking for info on mocha cc mforns milimetric [21:40:27] on this [21:40:35] it took me a while to figure out [21:40:44] that we do not use mocha -> facepalm [21:41:09] milimetric: sinonmocks but jasmine doesn't [21:41:14] milimetric: let me see [21:41:31] aha [21:41:42] yes, no mocha that I can recall [21:41:43] milimetric: ah no it can create spys of objects you define [21:41:52] yeah, it has decent mocking [21:42:09] milimetric: but not of jquery objects [21:42:12] this doesn't make sense ottomata -- all charts looks good -- but data loss -- at the moment of kafka restart [21:42:23] milimetric: as in you need to specify base for your mock [21:42:52] I don't see why spies wouldn't work for jquery: http://stackoverflow.com/questions/5337481/spying-on-jquery-selectors-in-jasmine [21:43:33] and for .get: http://stackoverflow.com/a/6199132/180664 [21:44:07] but anyway, point being: I don't love having two frameworks, makes it confusing [21:44:53] milimetric: right that seems like it might work, we will need to try it, i might do it as part of refactoring these tests [21:45:10] up to you, I didn't mean it was urgent [21:45:25] just something that always bothered me. And if Jasmine spying sucks, we can look at mocha or the million other frameworks [21:47:44] yeah joal i dunno [21:47:49] i mean, it must be produce errors [21:47:59] or, possibly 0s? you checked 0s right? [21:48:04] nwa can't be [21:48:24] ottomata: hourly stats doesn't take 0s into account [21:48:30] ottomata: will still triple check [21:49:21] ottomata: no 0s [21:50:07] 10Analytics, 10Dumps-Generation, 05Security: Pageview dumps incorrectly formatted, looks like a result of possibly malicious activity - https://phabricator.wikimedia.org/T144100#2855621 (10Nuria) >(404) The server has not found anything matching the Request-URI. >so using either of those while actually retur... [21:59:11] !log restarting eventlogging again to pick up puppet changes to use kafka-confluent writer [21:59:12] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:59:33] hey team, I'm logging off for today [21:59:48] see you on friday! [22:04:38] 10Analytics-EventLogging, 06Analytics-Kanban, 10EventBus, 13Patch-For-Review: Ensure no dropped messages in eventlogging producers when stopping broker - https://phabricator.wikimedia.org/T142430#2855653 (10Ottomata) Great. I feel good about this! I think the async kafka-confluent is gonna do us better f... [22:18:22] 10Analytics, 10Dumps-Generation, 05Security: Pageview dumps incorrectly formatted, looks like a result of possibly malicious activity - https://phabricator.wikimedia.org/T144100#2855716 (10Tgr) >>! In T144100#2855621, @Nuria wrote: > mmm ...mediawiki has its own stand when it comes to urls pointing to non ex... [22:25:03] ottomata: must be produce errors - we might elukey to take a look tomorrow morning [22:28:00] (03PS4) 10Joal: Add mediawiki history spark jobs to refinery-job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/325312 (https://phabricator.wikimedia.org/T141548) [22:34:07] milimetric: still around? [22:34:11] hi joal [22:41:26] disconnecting a-team, see you tomorrow ! [23:02:57] ye ya [23:02:59] l;aters all! [23:09:03] nite everyone [23:28:36] 06Analytics-Kanban, 10EventBus, 10Wikimedia-Stream, 13Patch-For-Review, and 2 others: RecentChanges in Kafka - https://phabricator.wikimedia.org/T152030#2855890 (10GWicke) >>! In T152030#2848542, @Ottomata wrote: >> The returned information is not yet integrated into the RC feed though. > Why would you int...