[01:53:10] Quarry: Mutiple columns with the same name will cause the result to not be shown - https://phabricator.wikimedia.org/T141233#2491058 (Huji) [07:54:34] joal: o/ [07:54:38] goooooood morning! [07:54:42] How's Lino?? [07:55:28] I just checked the AQS compaction graphs and the second month's compaction time looks awesome (I am guessing that the loading time looks good too) [07:55:49] I wanted to restart the cluster but I saw that you have started the third month :P [07:56:02] even sstable space occupied looks good [07:56:09] seems a good start of the week :) [08:45:33] elukey: o/ [08:45:47] elukey: Lino is good! He was teething the other day [08:45:56] elukey: Indeed cassandra looks ok so far [08:46:14] Third month loading is done (finished yesterday night), currently compacting [08:48:52] elukey: How was your weekend? [08:52:16] (PS1) Addshore: +x user_groups script [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/300828 [08:53:03] (PS1) Addshore: +x user_groups script [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/300829 [08:53:27] (CR) Addshore: [C: 2 V: 2] +x user_groups script [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/300829 (owner: Addshore) [08:53:36] (Merged) jenkins-bot: +x user_groups script [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/300829 (owner: Addshore) [08:53:39] (CR) Addshore: [C: 2] +x user_groups script [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/300828 (owner: Addshore) [08:53:47] (Merged) jenkins-bot: +x user_groups script [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/300828 (owner: Addshore) [08:57:11] (PS1) Addshore: betafeature count script - Group by feature [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/300831 [08:57:28] (PS1) Addshore: betafeature count script - Group by feature [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/300832 [08:57:35] (CR) Addshore: [C: 2 V: 2] betafeature count script - Group by feature [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/300832 (owner: Addshore) [08:57:40] (CR) Addshore: [C: 2] betafeature count script - Group by feature [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/300831 (owner: Addshore) [08:57:43] (Merged) jenkins-bot: betafeature count script - Group by feature [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/300832 (owner: Addshore) [08:57:48] (Merged) jenkins-bot: betafeature count script - Group by feature [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/300831 (owner: Addshore) [08:57:52] lalalalaaa spam spam spam [08:59:03] :D [08:59:32] joal: good! [09:00:15] I added some stats to https://wikitech.wikimedia.org/wiki/User:Elukey/Ops/AQS_Settings#Traffic_analytics from your spreadsheet [09:00:26] and also updated the data about data requirement per instance [09:00:34] elukey: If cassandra compaction is stable (would be awesome), restart can happen tomorrow morning [09:00:39] that now should be ~3.5T for three years of data more or less [09:00:48] (per instance) [09:00:55] elukey: right [09:01:10] elukey: how much do we have per instance? [09:01:16] 6TB :D [09:01:24] but raid0 [09:01:35] I hoped that we could have gone for RAID10 [09:01:39] right, I recall that [09:01:40] but it doesn't fit [09:01:53] we also need to account space for compactions etc.. [09:02:02] say 4TB but could be more [09:02:09] elukey: yup, looks like we have enough for 3 years, but no more ;) [09:02:57] Thanks for the awesome doc elukey [09:03:39] it keeps my mind clear :D [09:03:58] joal: I think that we could retain 4 years of data minimum with 6TB no? [09:04:14] 5 might be too much [09:04:19] elukey: I don't think so, as you said compaction space [09:04:30] more than 3, probably, but probably not up to 4 [09:04:57] yeah we don't know how bad it will be with 3TB loaded [09:05:15] elukey: Exactly, we need to monitor how the thing behaves when loaded [09:06:05] elukey: Look at that: https://grafana.wikimedia.org/dashboard/db/aqs-cassandra-system?panelId=12&fullscreen [09:06:20] elukey: over 7 days, on one of the new instances [09:06:33] 1 month load really double used space [09:07:18] ahhh Cassandra [09:07:53] you are so nice and beutiful but sometimes you are a bitch [09:08:09] elukey: godesses :) [09:08:35] elukey: But I don't think we could have found so good RAS [09:12:39] :) [13:25:54] mooornin! [13:27:08] helloooo [13:31:10] ottomata: Hi :) [13:31:18] ottomata: ping for schemas when you want :) [13:31:57] oh jaaa oh man i'm at that cafe again...maybe my phone internet will be good [13:32:10] joal in 30 mins? [13:32:22] ottomata: no rush though, if you prefer we can go for it after standup for instance [13:32:30] ottomata: sure ! [13:32:38] naw let's do it before [13:32:40] ops meeting today too [13:32:43] do we need milimetric? [13:32:45] okey [13:32:50] nope, I think we are ok [13:32:54] k [13:32:58] milimetric knows everything :) [13:35:36] ottomata: https://puppet-compiler.wmflabs.org/3455/analytics1028.eqiad.wmnet/ - what do you think? [13:35:40] looks good? [13:35:45] I didn't find a good way to test it [13:35:58] pcc looks good [13:36:30] the only thing that looks weird is , "typeNames": ["name"] [13:37:34] i think typeNames: name is correct, or perhaps even arbitrary [13:37:36] not totally sure about that [13:37:47] elukey: i think that'll work [13:46:15] Analytics-Kanban, EventBus, Patch-For-Review: Upgrade kafka main clusters - https://phabricator.wikimedia.org/T138265#2491900 (Ottomata) a:Ottomata [13:47:09] Analytics-Kanban, EventBus, Patch-For-Review: Upgrade kafka main clusters to 0.9 - https://phabricator.wikimedia.org/T138265#2395136 (Ottomata) [13:47:18] Analytics-Kanban, EventBus, Patch-For-Review: Upgrade kafka main clusters to 0.9 - https://phabricator.wikimedia.org/T138265#2395136 (Ottomata) [13:47:22] Analytics-Kanban, EventBus, Patch-For-Review: Upgrade kafka main clusters to 0.9 - https://phabricator.wikimedia.org/T138265#2395136 (Ottomata) I'd like to do main-codfw this week. Will coordinate with services on this. [13:51:42] ottomata: https://graphite.wikimedia.org/S/Bj :) [13:52:02] I applied the patch there manually [13:52:08] nice! [13:52:09] final metric is kafka.cluster.main-codfw.jvm_memory.kafka2001_codfw_wmnet_9999.sun_management_GarbageCollectorImpl.G1OldGeneration.CollectionCount.count [13:52:20] hmm, [13:52:24] sun_management eh? [13:52:26] is that expected? [13:52:29] i guess so [13:52:30] ? [13:52:39] name of the class maybe? [13:52:43] aye guess so [14:02:28] ottomata / joal: were you guys gonna talk schemas now? [14:02:48] milimetric: waiting for ottomata to ping, but yes [14:03:34] I don't need to be there, but I think we should think about how the new schemas map to our denormalized schema [14:04:01] if there's a difference, where would we get the rest of the data [14:04:12] ayyye k [14:04:24] joal: am ready, let's try this cafe internet [14:04:28] if its bad i'll switch to phone [14:04:33] milimetric: I think we're good on that: we spent a long time the other with you and mforns about that IIRC [14:04:44] ottomata: OMW [14:14:50] Analytics, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown, Patch-For-Review: Implement Schema:ExternalLinkChange - https://phabricator.wikimedia.org/T115119#2492017 (Milimetric) yaay! We have 91 rows of data in table ExternalLinksChange_15716074, fr... [14:21:08] Analytics-Kanban, EventBus, Patch-For-Review: Upgrade kafka main clusters to 0.9 - https://phabricator.wikimedia.org/T138265#2492048 (Ottomata) @eevans let's sync up on IRC today about this. [14:22:42] Analytics-Wikimetrics, Continuous-Integration-Config: tox runs all tests (including manual ones) - https://phabricator.wikimedia.org/T71183#2492051 (Milimetric) @hashar, yes wikimetrics is maintained but it lost most of its stakeholders in reorganizations. So it's not very active. It has value for a fe... [14:30:47] Analytics, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown, Patch-For-Review: Implement Schema:ExternalLinkChange - https://phabricator.wikimedia.org/T115119#2492070 (Samwalton9) Hooray! Can we access the details of those events? Diffs/dates should he... [14:39:06] ottomata: I think that the code review is good but two things must be decided: [14:39:38] 1) the hadoop GC metrics will have a new name [14:40:15] 2) we have metric names with stuff like sun_management_GarbageCollectorImpl [14:40:37] I don't think that those are problems but I want to have your opinion on them [14:40:58] I am looking for a way to remove sun_management_GarbageCollectorImpl but I don't think there is a simple one [14:44:06] elukey: i think its ok [14:44:11] a-team, I'm away until standup see you there [14:44:17] jmx metrics are going to be pretty messy i think [14:44:26] we can do our best to make them clean [14:44:30] but for the most we will just have to deal with them [14:44:47] i think its better to reduce the configs like we've done rather then special casing all of them [14:44:50] Analytics-Wikimetrics, Continuous-Integration-Config: tox runs all tests (including manual ones) - https://phabricator.wikimedia.org/T71183#2492094 (hashar) p:Triage>Low Gave a try again and tox/nosetests fail on my local machine: ``` IOError: [Errno 2] No such file or directory: '/srv/wikimetric... [14:44:57] ottomata: sure.. I was checking https://grafana.wikimedia.org/dashboard/db/aqs-cassandra-gc [14:45:06] but I can't find how restbase is doing it [14:45:11] probably not with jmxtrans [14:52:58] nope we don't.. I thought it was diamond but I am a bit confused [14:53:55] oh hm [14:54:19] hm [14:54:22] dunno how they do that elukey [14:57:10] milimetric: let us know when you are around [14:57:49] I'm here [14:57:55] ottomata: [14:58:05] /usr/bin/java org.wikimedia.cassandra.metrics.service.Service [14:58:06] ahhh [14:58:09] yall in batcave? [14:59:04] milimetric: can join, joseph is out til standup, milimetric let's sync [14:59:13] k [15:19:01] You? [15:22:02] yo we good joal, dan and I are going to meet up in person this week and hammer some stuff out and then present [15:23:24] ok ottomata sounds [15:23:26] good [15:25:39] Analytics-Wikimetrics, Continuous-Integration-Config: tox runs all tests (including manual ones) - https://phabricator.wikimedia.org/T71183#2492178 (Nuria) @harshar: tests cannot be run from depo alone as they require a wikimetrics instance running, there are a few unit tests but mostly they are integrat... [15:31:01] ottomata: damage is done, new metrics coming in for Kafka, Hadop, Zookeeper :) [15:31:39] coool! [15:31:54] https://www.youtube.com/watch?v=VLrd7tbiQQE [15:32:20] ahhaaah [15:32:28] I am going to play it until icinga complains [15:32:30] :P [15:33:00] now youtube wants me to listen "Cry me a river" [15:33:05] I blame you ottomata [15:33:21] haha [15:41:18] haha, ops playlists, I like it [15:49:24] hey opsy people quick question [15:49:37] we have a research group that has signed an NDA [15:49:57] they'll request access to login to stat1002/1004 soon. [15:50:03] (I guess just 1002) [15:50:13] and they sent their public keys by email [15:50:33] is that ok for verification that they are associated to their public keys or is there another process or...? [15:50:45] I know for us we upload on wikitech and then magic happens behind the scenes [15:52:03] milimetric: i'm not sure [15:52:21] the best you can do is have them follow this [15:52:21] https://wikitech.wikimedia.org/wiki/Production_shell_access#Requesting_access [15:52:27] k [15:53:04] Analytics-Kanban: Browser dashboard blogpost - https://phabricator.wikimedia.org/T141267#2492325 (Nuria) [15:54:53] elukey: whenever you get a chance, woudl you review this? https://gerrit.wikimedia.org/r/#/c/300879/ [15:54:58] not a hurry [15:56:18] sure! [16:23:38] Analytics, Operations, Performance-Team, Traffic: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2492507 (BBlack) Noting from last meeting about this: We've **tentatively** said we'll try to make this (implementing a robust A/B test infrastructure at the Varnish level) an... [16:23:57] ottomata: bottom graphs :) https://grafana.wikimedia.org/dashboard/db/kafka [16:25:07] Analytics, Operations, Performance-Team, Traffic: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2492511 (Nuria) Second @BBlack. We will make this a shared goal among traffic and analytics team [16:25:25] awesooome! [16:29:57] one thing that I've noted across our G1 JVM deployments is that Young gen collections are frequent and Old ones seems not be non existent.. Not an expert but it would make sense (Garbage first) but there might be some gotchas to make G1 more perfomant that we could explore [16:30:11] also G1 tests would be nice on Hadoop [16:30:13] Analytics, Operations, Performance-Team, Traffic: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2310174 (ori) What's the rationale for prioritizing it? [16:31:37] elukey: +1, I am also not an expert at all [16:31:45] I know very little about jvm gc [16:34:06] Analytics, Operations, Performance-Team, Traffic: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2492540 (BBlack) It's a seasonal issue that's come up every few months for the past couple of years. Every time we need to run an A/B test, we go back through the same conver... [16:34:45] Analytics, Analytics-Dashiki: Searching for nl... does not bring nlwikipedia only nlwikidata - https://phabricator.wikimedia.org/T133718#2492544 (Nuria) Open>Resolved a:Nuria Not an issue, nlwikimedia is the true project name (there are no pageviews but that is an issue with "knowledge pagevi... [16:36:20] hm, elukey, i just noticed that main-codfw doesn't show up as a possible kafka cluster... [16:36:23] in that grafana dash [16:36:27] strange... [16:37:15] ahhhh, weird, i think the template values dont' reload automatically [16:37:32] Analytics: Capacity projections of pageview API document on wikitech - https://phabricator.wikimedia.org/T138318#2492553 (Nuria) Assigning to @elukey as he has already taken care of this: https://wikitech.wikimedia.org/wiki/User:Elukey/Ops/AQS_Settings#Traffic_analytics [16:37:34] i had to go to the template var and tell it to update possible values from graphite manually [16:37:36] Analytics: Capacity projections of pageview API document on wikitech - https://phabricator.wikimedia.org/T138318#2492558 (Nuria) The page just needs to be moved to /analytics/pageviewAPI/capacityProjections [16:38:01] Analytics-Kanban: Capacity projections of pageview API document on wikitech - https://phabricator.wikimedia.org/T138318#2492559 (Nuria) a:elukey [16:44:11] Analytics: Migrate the simplest limn dashboards to dashiki tabular {frog} - https://phabricator.wikimedia.org/T126358#2012091 (Nuria) This should be on operational excellence if possible q1 , if not for sure q2. [16:45:51] Analytics-Kanban: Compile a request data set for caching research and tuning - https://phabricator.wikimedia.org/T128132#2492617 (Danielsberger) Starting with a 1G dataset is a great idea. I don't know about the max file size (on datasets.wikimedia.org), but the largest files I've seen there are about 300-50... [16:45:53] Analytics: Clean up datasets.wikimedia.org - https://phabricator.wikimedia.org/T125854#2492618 (Nuria) p:Normal>Low [16:46:33] Analytics: Move datasets.wikimedia.org to analytics.wikimedia.org/datasets - https://phabricator.wikimedia.org/T132594#2492622 (Nuria) p:Triage>Low [16:47:30] (CR) Amire80: Add a script for checking number of pages published despite failures (2 comments) [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/282312 (https://phabricator.wikimedia.org/T127283) (owner: Amire80) [16:50:16] Analytics: Pageview API alarms - https://phabricator.wikimedia.org/T132456#2492633 (Nuria) This alarm is useful in the case restbase might not be able to connect to cassandra (and thus we are sending 500 and we do not have an alarm from storage) which is unlikely. Throttling and aqs alarms provide adequate... [16:50:24] Analytics: Pageview API alarms - https://phabricator.wikimedia.org/T132456#2492634 (Nuria) Open>Resolved [16:51:28] Analytics-Cluster, Analytics-Kanban: Deploy hive-site.xml to HDFS separately from refinery - https://phabricator.wikimedia.org/T133208#2492637 (Nuria) [16:52:29] Analytics, Analytics-Dashiki: Timeseries on browser reports broken when going back 18 months - https://phabricator.wikimedia.org/T141166#2492652 (Nuria) [16:53:37] Analytics, Analytics-Dashiki: Timeseries on browser reports broken when going back 18 months - https://phabricator.wikimedia.org/T141166#2489099 (Nuria) Time selector should not let you go back before may 2015 as we have no data for that period. [16:57:22] ottomata: added also G1 metrics to EB https://grafana.wikimedia.org/dashboard/db/eventbus [16:57:27] nice [16:57:34] hmm [16:57:36] elukey: [16:57:41] those shoudl probably just stay in kafka, no? [16:57:47] i don't think we need them on eb dash [16:57:54] Analytics-Dashiki, Analytics-Kanban: Default date selection to currently applied date for browser reports - https://phabricator.wikimedia.org/T141165#2492710 (Nuria) [16:58:43] Analytics, Analytics-Dashiki: Default date selection to currently applied date for browser reports - https://phabricator.wikimedia.org/T141165#2489086 (Nuria) [16:59:16] Analytics: Check if we can deprecate legacy TSVs production (same time as pagecounts?) - https://phabricator.wikimedia.org/T130729#2492715 (Nuria) a:madhuvishy>None [16:59:51] ottomata: ah right, but it might be good maybe renamed as "Kafka bla bla bla" ? [17:00:01] ? [17:00:28] I meant leave the GC graphs in there but prefixed with "Kafka" [17:00:37] hm, i dunno, i think its too specific [17:00:51] not needed there, too cluttery. i also think the changeprop graphs should go elsewhere [17:00:58] but am not in a hurry to move them :) [17:01:34] the evenbus dash is more about eventbus stuff and events, kafka is just the transport, and more kafka specific operations stuff should just be in the kafka dash [17:02:37] mmmm maybe with a separate section only for event bus? [17:03:01] because it might become a mess if we mix say main-analytics and main-eqiad metrics [17:03:07] it is already crowded :D [17:03:17] * milimetric lunching [17:03:17] ? [17:03:22] elukey: its alredy there [17:03:24] just change the cluster [17:03:25] var [17:03:53] all the graphs switch automatically [17:04:05] ahhahahaha I never noticed that var! [17:04:06] sorry! [17:04:08] hahahaah [17:04:12] not sure why your new GC ones don't show any data though [17:04:14] for main-eqiad [17:04:22] I need to put the var in the metrics [17:04:25] let me fix them [17:04:30] :P [17:04:44] yeah now your point makes complete sense [17:04:45] okok [17:04:45] HMm why is kafka1001 taking in most messages? [17:04:58] leaders are not unbalacned [17:06:13] fixed the graphs, now amending eventbus [17:06:13] hm actually they are unbalanced, but in favor a bit of 1002 [17:06:14] not 1001 [17:06:17] cool danke [17:07:19] done [17:09:42] (talking in services) kafka1001 has so many more messages because most messages are in resource_change topic, which has only 1 partition [17:16:00] ahhh makes sense [17:16:05] in the meantime, I changed https://grafana.wikimedia.org/dashboard/db/zookeeper [17:18:13] Analytics-Kanban: Compile a request data set for caching research and tuning - https://phabricator.wikimedia.org/T128132#2492836 (Nuria) Excellent suggestion with hashing, hopefully we can get started on thisi this week. [17:18:45] going offline people! [17:18:52] talk with you tomorrow :) [17:20:01] ottomata: o/ [17:25:36] (PS1) Addshore: Improve output of betafeatures script [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/300907 [17:25:53] (PS1) Addshore: Improve output of betafeatures script [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/300908 [17:25:53] laterrrs! [17:25:56] (CR) Addshore: [C: 2] Improve output of betafeatures script [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/300907 (owner: Addshore) [17:25:59] (CR) Addshore: [C: 2] Improve output of betafeatures script [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/300908 (owner: Addshore) [17:26:04] (Merged) jenkins-bot: Improve output of betafeatures script [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/300907 (owner: Addshore) [17:26:07] (Merged) jenkins-bot: Improve output of betafeatures script [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/300908 (owner: Addshore) [17:32:30] Analytics, New-Readers: Split opera mini in proxy or turbo mode - https://phabricator.wikimedia.org/T138505#2492892 (atgo) [17:49:30] (PS1) Addshore: Only send global betafeature metric if whitelisted [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/300912 [17:49:44] (PS1) Addshore: Only send global betafeature metric if whitelisted [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/300913 [17:49:51] (CR) Addshore: [C: 2] Only send global betafeature metric if whitelisted [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/300913 (owner: Addshore) [17:49:54] (CR) Addshore: [C: 2] Only send global betafeature metric if whitelisted [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/300912 (owner: Addshore) [17:49:59] (Merged) jenkins-bot: Only send global betafeature metric if whitelisted [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/300913 (owner: Addshore) [17:50:02] (Merged) jenkins-bot: Only send global betafeature metric if whitelisted [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/300912 (owner: Addshore) [17:50:06] spam spam spam [18:07:11] Analytics, Pageviews-API: Count pageviews for all wikis/systems behind varnish - https://phabricator.wikimedia.org/T130249#2493058 (Nuria) Per our conversation with research (cc @DarTar and @Erik_Zachte) we are going to add "not knowledge wikis" to our pageview pipeline. For two reasons: 1) magnitude of... [18:15:09] Analytics, Analytics-EventLogging, EventBus: Upgrade eventlogging kafka client used for producing - https://phabricator.wikimedia.org/T141285#2493093 (Ottomata) [18:16:04] Analytics, Analytics-EventLogging, EventBus: Upgrade eventlogging kafka client used for producing - https://phabricator.wikimedia.org/T141285#2493121 (Ottomata) See also: - https://phabricator.wikimedia.org/T133779 - https://gerrit.wikimedia.org/r/#/c/292976/ [18:31:04] !log upgrading kafka to 0.9 in main-codfw, first kafka2001 then 2002 [18:31:06] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [18:31:35] ottomata: if you "undo" eventbus role and try to add it again on vagrant , does it work? I am running into errors due to install of kafka packages when doing vagrant upgrade , mostly ususal sudo linux stuff so no big deal but vagrant upgrade errors [18:32:06] nuria_: did you have kafka role installed before? [18:32:23] ottomata: mmm.. i do not think so but let me try to install kafka role [18:33:10] nuria_: eventbus will install that [18:33:23] but, if you had it installed before, mw vagrant puppet has changed it to confluent version [18:33:25] which will conflict [18:33:30] ottomata: i see [18:33:31] so you'd might need a fresh mw vagrant [18:45:07] ottomata: i just removed all kafka via apt-get and will try again to install [18:45:35] ok, do it via vagrant provision [18:50:54] ottomata: the code for eventbus gets downloaded to its usual place /mediawiki/extensions/eventlogging/service or is there another place? [18:51:21] nuria_: do you mean eventlogging or the eventbus mw extension? [18:51:39] ottomata: eventbus service [18:52:00] /srv/eventlogging [18:52:10] welll [18:52:15] srv/eventlogging in your host [18:52:19] /vagrant/srv/eventlogging on the vm [18:53:20] ottomata: ah, that's right! will document here: https://wikitech.wikimedia.org/wiki/EventBus#Development_Environment [18:54:05] ottomata: but the vagrant provision doesn't start teh service right? [18:54:13] it should ja [18:54:20] nuria_: if you enable the eventbus role [18:54:23] it hsould do everything [18:54:34] install kafka, add the extension, set up eventlogging and eventbus service [18:55:05] ottomata: kafka as vagrant role or as unix pkg [18:55:08] ? [18:56:43] nuria_: as vagrant role [18:56:48] well, as puppet class [18:56:49] ottomata: k [18:56:50] but same same [18:56:55] ottomata: right right [19:10:27] ottomata: ok, kafka and zookeeper seem to be starting ok but not the eventbus.. where would i look for logs? var/log doesn't seem to have anything [19:10:45] srv/logs [19:39:04] Analytics-Kanban, Patch-For-Review: Event Logging doesn't handle kafka nodes restart cleanly - https://phabricator.wikimedia.org/T133779#2493536 (Ottomata) I'm reverting my confluent-kafka-python consumer change for now. It is not working in beta, and we want to do a prod deploy for eventbus soon. http... [19:48:02] Analytics: Add global last-access cookie for top domain (*.wikipedia.org) - https://phabricator.wikimedia.org/T138027#2493564 (Tbayer) Confirming that there is a need for such a metric from the perspective of the Reading team, and also from the perspective of the Communications team (based on our conversatio... [19:48:55] ottomata: ok, all working well now, testing patch [19:51:00] great [20:02:28] nuria_: got another quick EL review for you [20:02:29] https://gerrit.wikimedia.org/r/#/c/300944/ [20:02:46] ottomata: this new Ui for gerrit is killing me [20:02:56] haha [20:03:00] Analytics, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown, Patch-For-Review: Implement Schema:ExternalLinkChange - https://phabricator.wikimedia.org/T115119#2493611 (Samwalton9) Oh, though is this test wiki? In which case that might be reasonable. [20:03:00] i'm getting used to it too.. [20:10:37] Analytics-Kanban, EventBus, Patch-For-Review: Upgrade kafka main clusters to 0.9 - https://phabricator.wikimedia.org/T138265#2493664 (Ottomata) codfw has been upgraded to 0.9. We found a bug in the version of kafka-python we are using for eventbus. To work around this for this deploy, before we upg... [20:18:39] running home, back shortly [21:32:51] (PS1) Addshore: Track global user enables * disables of beta features [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/301002 (https://phabricator.wikimedia.org/T140226) [21:35:16] (PS2) Addshore: Track global user enables * disables of beta features [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/301002 (https://phabricator.wikimedia.org/T140226) [21:36:31] (PS3) Addshore: Track global user enables * disables of beta features [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/301002 (https://phabricator.wikimedia.org/T140226) [21:42:14] (PS4) Addshore: Track global user enables & disables of beta features [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/301002 (https://phabricator.wikimedia.org/T140226) [22:19:35] (PS5) Addshore: Track global user enables & disables of beta features [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/301002 (https://phabricator.wikimedia.org/T140226) [22:48:00] (PS1) Addshore: Track global user enables & disables of beta features [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/301018 (https://phabricator.wikimedia.org/T140226) [22:48:11] (CR) Addshore: [C: 2 V: 1] Track global user enables & disables of beta features [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/301002 (https://phabricator.wikimedia.org/T140226) (owner: Addshore) [22:48:15] (CR) Addshore: [C: 2 V: 1] Track global user enables & disables of beta features [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/301018 (https://phabricator.wikimedia.org/T140226) (owner: Addshore) [22:59:20] (Merged) jenkins-bot: Track global user enables & disables of beta features [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/301002 (https://phabricator.wikimedia.org/T140226) (owner: Addshore) [22:59:23] (Merged) jenkins-bot: Track global user enables & disables of beta features [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/301018 (https://phabricator.wikimedia.org/T140226) (owner: Addshore) [23:08:21] Analytics, Operations, Performance-Team, Traffic: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2494260 (Nuria) @BBlack: i volunteer to write a design doc with user cases /high level design ideas and issues by the end of this quarter so we can use it to scope the work we... [23:09:57] Analytics, Revision-Slider, TCB-Team, WMDE-Analytics-Engineering, and 5 others: Data need: Explore range of article revision comparisons - https://phabricator.wikimedia.org/T134861#2494262 (Addshore) Open>Resolved Changed merged and the logs are now accessible on fluorine