[03:54:24] 10Analytics-Kanban, 10Analytics-Wikistats: Alpha Release: Breakdowns don't work in Firefox - https://phabricator.wikimedia.org/T180556#3761595 (10Milimetric) [03:56:25] (03PS1) 10Milimetric: [VERY VERY WIP] Work so far on simplifying and fixing breakdowns [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/391490 (https://phabricator.wikimedia.org/T180556) [06:48:41] (03CR) 10jerkins-bot: [V: 04-1] [VERY VERY WIP] Work so far on simplifying and fixing breakdowns [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/391490 (https://phabricator.wikimedia.org/T180556) (owner: 10Milimetric) [08:52:01] hello people [08:52:08] I am about to reboot thorium for kernel updates [09:02:23] all up and running [09:07:28] now I am going to migrate db1046 to db1107 with manuel [09:07:41] so all eventlogging activities will be stopped [09:30:47] elukey: all consumers you mean?n [09:31:46] joal: yes, I used eventloggingctl stop [09:32:06] ok elukey [09:32:13] I mean, we can stop only the mysql one I know, but you know that I am paranoid [09:32:20] :) [09:32:48] data is being copied from db1046 to db1107 now [09:33:07] joal: do you have 5 mins to do some sanity checks with me? [09:33:25] elukey: I think we're going to need mforns for the mediawiki-history cleaner alerts [09:33:51] elukey: I can do sanity checks with you, but I'm not sure sanity is the area where I can most help currently :) [09:35:08] joal: are you suggesting that we are insane? :D [09:35:20] I'd agree with you! :) [09:35:32] elukey: about me it;s not a suggestion - I let you decide for yourself :) [09:35:59] so I checked the mysql consumer config and it uses m4-master.eqiad.wmnet/log in its config, so I am pretty sure that db1046 is not hardcoded anywhere [09:36:11] joal: you know that I'll always follow you :D [09:36:38] elukey: IIRC the only way we configure EL is through consumers/processors config files [09:36:55] our super dbas are going to switch the m4-master domain to db1107 so everything should magically work [09:37:27] (it points to a proxy that then points to our dbs) [09:37:29] And my memory has no instance of any hard-name of dbs in EL config (but it makes long time I've been looking at it) [09:37:47] * joal loves dba's mgic [09:37:53] I am pretty sure that Andrew wouldn't have allowed it but better safe than sorry [09:38:03] elukey: ? [09:38:09] --verbose pleaser [09:38:39] joal: I meant having db1046 hardcoded somewhere [09:38:45] Ah, yes [09:39:15] and now [09:39:16] ;; ANSWER SECTION: [09:39:16] m4-master.eqiad.wmnet. 300 IN CNAME dbproxy1004.eqiad.wmnet. [09:39:17] dbproxy1004.eqiad.wmnet. 3600 IN A 10.64.16.148 [09:39:33] so once dbproxy1004 is updated we are fine [09:40:04] all data is copied already? [09:40:59] nono still in flight, it will probably take a couple of hours (hopefully) [09:41:18] I would have guessed so [10:13:50] 10Analytics, 10DBA, 10Operations, 10Patch-For-Review, 10User-Elukey: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#2987633 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts: ``` ['dbproxy1009.eqiad... [10:46:47] 10Analytics, 10DBA, 10Operations, 10Patch-For-Review, 10User-Elukey: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3762299 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['dbproxy1009.eqiad.wmnet'] ``` and were **ALL** successful. [10:53:44] helloooo [10:54:34] hi mforns ! [10:54:59] mforns: we need YOU!! We have plenty alarms about mediawiki history cleaner: ( [10:55:08] joal, O.O [10:55:17] ok checking... [10:55:35] oh man... [10:55:38] partitions again [10:55:39] mforns: I prefered to wait for you, since you built the thing [10:55:54] mforns: I can help if you need /want [10:56:10] but... this should not be executing every minute now... after the fix... [10:56:12] hmmm [10:56:33] mforns: o/ - I stopped all the eventlogging consumers since we are moving the master db [10:56:40] I preferred the paranoid approach [10:56:52] elukey, cool np [11:03:11] joal, I think what happened is: we fixed the puppet code that ensured presence of the right cron job, but I forgot to remove the old cron job that was already there. makes sense? cc: elukey [11:04:00] ah I noticed an error in my cronspam this morning, didn't have time to follow up [11:04:30] Arf mforns [11:07:29] mforns: removed the root crontab from an1003 [11:08:12] re-run puppet and it doesn't get installed again (since it is in the hdfs user's crontab now) [11:08:41] elukey, do I have access to an1003? [11:08:46] thanks a lot! [11:09:24] I was looking for it in stat1005 for some reason [11:10:12] yes, I have access [11:10:15] PROBLEM - Check status of defined EventLogging jobs on eventlog1001 is CRITICAL: CRITICAL: Stopped EventLogging jobs: consumer/mysql-m4-master-00 consumer/mysql-eventbus consumer/client-side-events-log consumer/all-events-log processor/client-side-11 processor/client-side-10 processor/client-side-09 processor/client-side-08 processor/client-side-07 processor/client-side-06 processor/client-side-05 processor/client-side-04 proces [11:10:16] processor/client-side-02 processor/client-side-01 processor/client-side-00 forwarder/legacy-zmq [11:10:16] :] [11:11:14] adding downtime sorryyyy [11:11:17] it expired :( [11:12:25] elukey, I can see the correct cron job in an1003 hdfs crontab -l ! Thanks a lot, this should solve the alarms. [11:12:28] cc joal [11:12:37] oh, wait... [11:13:04] the old job might have corrupted the partitions... [11:13:14] will reparit them [11:13:20] *repair [11:14:15] mforns: Question for you - Shouldn't the job have deleted data? [11:15:16] joal, so 2 cron jobs were scheduled for today, the old wrong one, and the new correct one (I forgot to remove the old one when fixing) [11:15:44] it seems that the old one executed first and deleted some partitions [11:15:55] but could not delete the corresponding files [11:16:23] Ahhhh, then new job starts, and doesn't delete because partitions are not present anymore [11:16:48] yes, doesn't delete because partitions and data do not match, and exits [11:16:55] right right [11:17:09] and more, old wrong job executes every minute! [11:17:12] and hence the alarms [11:17:29] I will repair the tables and run manually [11:19:53] Thanks mforns :) [11:25:14] mforns: do you have 5 mins ? [11:25:21] db1107 is ready \o/ [11:25:33] if you could do some sanity checks with me it would be awesome [11:26:04] elukey, sure, but... I'm in a library that has no room where you can speak... [11:26:38] mforns: it would be super fine just to log in on db1107, check data [11:26:45] etc.. [11:26:51] it seems all good from my side [11:26:59] but I'd like you to double check to be sure [11:27:01] ok, will do! [11:27:17] if we are ok then the dbas will point the m4-master domain to db1107 [11:27:26] and then we'll be free to re-enable eventlogging [11:27:26] I'm finishing the mediawiki-snapshot thing, and right after I'll look into that [11:27:31] super [11:27:33] aha [11:28:57] hm, I guess it's urgent, will look at it first. [11:31:56] elukey, I can not connect to db1107's mysql server... [11:32:08] with db1108 works, but not 1107 [11:32:11] good start :D [11:32:13] xD [11:32:18] were you able to connect to db1046? [11:32:24] hmmm [11:32:41] maybe not... [11:33:17] although I remember having connected to the master somehow... maybe it was someone else sharing screen [11:34:09] lemme check, you guys should definitely be able to connect [11:34:17] mforns: can you ssh to db1046 ? [11:34:45] no, asks for pasword [11:37:54] mforns: ah you definitely don't have a user over there [11:37:55] sigh [11:38:10] can you tell me what you'd have checked ? [11:38:20] I'll take care of the perms later on then [11:38:21] elukey, but shouldn't I be able to connect via creds file though stat1006?? [11:38:41] maybe not like replica... [11:38:51] I don't see the research user though [11:38:57] aha [11:39:35] elukey, which data have you copied already? [11:40:39] all data since you stopped EL I guess right? [11:41:16] manuel copied all the databases over [11:41:25] I verified the log database, 383 tables listed [11:41:42] dunno, I would pick a couple tables and look for key fields like max(timestamp) should match the time we stopped EL [11:42:20] and min(timestamp) should match old master's min timestamp [11:42:42] and check encodings maybe? [11:44:29] good point [11:44:29] so [11:44:30] log | CREATE DATABASE `log` /*!40100 DEFAULT CHARACTER SET binary */ | [11:45:03] I would grab Edit_ and check that all fields are there, without weird conversions? [11:46:15] what is the ? [11:47:44] the biggest number? [11:47:48] elukey, I think for Edit it's 13457736 [11:47:50] yes [11:48:32] MariaDB [log]> select max(timestamp) from Edit_13457736; [11:48:32] +----------------+ [11:48:32] | max(timestamp) | [11:48:32] +----------------+ [11:48:32] | 20171115090914 | [11:48:34] +----------------+ [11:48:44] that looks sane [11:49:12] Maybe over-checking, but I maybe would do a histogram like: SELECT LEFT(timestamp, 7) as ts, count(*) FROM Edit_13457736 group by ts [11:49:17] in both old and new [11:49:28] but this is probably paranoia [11:50:13] nice query [11:50:14] maybe just a couple of select * limit 10, and look at the data [11:50:22] I just did them, all look good [11:50:54] mforns: I asked to the dbas to keep mysql on db1046 to see straight away if anything is wrong (like conn failures, etc..) [11:50:56] elukey, how about the uuids and ids [11:51:17] will 1107 pick up from last id? [11:51:32] in the case of uuids not ordered, but ids? [11:52:16] I am a bit confused, I thought that uuids were generated by EL [11:52:47] elukey, yes you're right... [11:53:03] and ids? [11:53:44] joal, mediawiki snapshots are clean now, only last 6 versions [11:54:00] let's see if next month we got any more problems [11:54:14] Sounds good mforns :) [11:54:43] 6 versions seems big to me, and I don't recall when we discussed it, but since we made it this way, I'm prety sure we discussed i :) [11:55:27] yes, we discussed it several times, but not in a dedicated meeting, rather in post-standup [11:55:56] several=2 or 3 times? [11:56:03] I remember the 6 version thing too, I think we pinged people on the list or somewhere, I remember Neil or someone chiming in [11:56:12] mforns: so data has been copied over as it was, I think that ids should just restart from the last one (otherwise there will surely be insertion errors and we'll notice them) [11:56:22] elukey, ok [11:56:23] makes sense? [11:56:37] yes, I think it's good :] [11:56:51] super [11:57:14] \o/ [11:57:27] joal, milimetric, any of you has experience with scalamock? [11:57:33] hey mforns I see that you saw the duplicate jobs cron problem, but I saw an access denied message in the spam last night, it looked like maybe root was trying to delete something owned by hdfs or something like that? [11:57:38] mforns: nope [11:57:55] milimetric, yes, it was the old cron job that I forgot to remove... [11:57:55] mforns: what are you willing to mock? [11:58:07] joal, DefaultHttpClient [11:58:07] no scalamock experience, sorry, but mforns you figured out MagicMock in like two minutes on wikimetrics [11:58:26] :] [11:59:19] scalamock seems super simple, but it says it can not mock DefaultHttpClient and I can not find anyhting in the internets... :( [12:00:12] I'm considering implementing a DefaultHttpClientMock class that extends DefaultHttpClient myself, if that is possible... [12:00:51] mforns: what is DefaultHttpClient? [12:00:57] anymore info on hat class? [12:01:37] joal, It's the class DataFrameToDruid uses to do a POST request to trigger Druid ingestion [12:02:03] mforns: right - where does it come from? Which package? [12:02:04] https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/client/DefaultHttpClient.html [12:02:07] thanks [12:02:50] mforns: from http://scalamock.org/quick-start/ (last paragraph): val httpClient = mock[HttpClient] [12:03:17] mforns: Idea here is to use a HttpClient (https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/client/HttpClient.html), and instnaciate it with the default one [12:03:56] makes sense mforns ? [12:05:21] joal, so the abstract HttpClient [12:06:00] mforns: Same as you'd do a List, and implement it using an ArrayList for instance [12:06:31] joal, you're saying that when I do mock[HttpClient] scalamock returns a DefaultHttpClient? [12:06:39] mforns: no [12:06:41] (mocked) [12:06:58] mforns: I'm saying, in your code, you should use a HttpClient (abstract) [12:07:04] oh! [12:07:06] I see [12:07:20] ahaaaa... light in the end of the tunnel [12:07:22] And only at creation time, you instantia te the DefaultHttpClient [12:07:29] will try :] [12:07:47] mforns: Prefere to use interfaces than implementation classes when feasible :) [12:08:08] you mean when defining parameters? [12:08:24] I mean when using objects [12:08:40] val httpClient: HttpClient = new DefaultHttpClient() [12:08:44] mforns: --^ [12:08:58] aha [12:08:59] ok [12:09:09] httpClient is a generic HttpClient, can be implemented by any of the clients [12:09:15] And therefore, mocked :) [12:10:44] ahaaa [12:10:48] trying :] [12:17:34] milimetric: any useful trick to check page_ids on wikipedia site? [12:17:42] by the wa hi milimetric :) [12:17:53] milimetric: if you have a minute, I'd show you some findings [12:18:20] joal: sorry gotta do some work around the house. I'll ping when I'm done [12:18:42] no prob milimetric - probably later tonight then, will take a break in the afternoon [12:20:12] joal, it vvvvvorks! [12:20:20] mforns: Yay ! [12:21:13] mforns: Inheritance programming rule: Always use the most generic definition that solves your issue (most generic interface) [12:22:10] 10Analytics, 10EventBus, 10Services (next): Clean up retry-retry Kafka topics - https://phabricator.wikimedia.org/T179958#3762631 (10Pchelolo) Here's the list of topics that should be deleted: {P6319} [12:22:13] mforns: Like that, for instance, if at some point we want to use https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/client/AutoRetryHttpClient.html, can be done almost without changes [12:22:48] mforns: I am seeing a ton of cronspam like [12:22:50] 2017-11-15T11:05:31 ERROR Selected partitions extracted from table specs (set([])) does not match selected partitions extracted from data paths (set(['snapshot=2017-04'])). [12:23:07] up to one hour ago [12:23:12] is it fixed right? [12:26:30] joal, yes makes sense, thanks a lot! [12:26:43] elukey, yes, should be fixed after you dropped old crontab [12:26:50] thanks for that :] [12:26:50] mforns: <3 [12:27:04] np mforns :) Thanks for doing good testing :) [12:27:53] hehe, so much wikilove :] [12:33:03] mforns: hehe :) [12:34:45] ok, taking a break ! later a-tem [12:36:45] me too.. [12:36:58] mforns: would you stick with me for some minutes more? [12:37:04] I am about to re-enable eventlogging [12:40:02] RECOVERY - Check status of defined EventLogging jobs on eventlog1001 is OK: OK: All defined EventLogging jobs are runnning. [12:42:09] so everything looks fine from the mysql consumer perspective [12:42:21] some warnings for duplicate entries in ix_MobileWikiAppPageScroll_14591606_uuid [12:42:27] but this happened also before the switch [12:42:31] not sure why [12:42:39] 10Analytics, 10Services (later): Update node-rdkafka to v2 - https://phabricator.wikimedia.org/T180591#3762674 (10Pchelolo) [12:43:14] 10Analytics, 10Services (later): Update node-rdkafka to v2 - https://phabricator.wikimedia.org/T180591#3762688 (10Pchelolo) [12:43:18] mmm ix_MobileWikiAppShareAFact_12588711_uuid too [12:43:21] weird [12:44:13] it has been happening for a while, do we know the root cause? [12:53:05] * elukey lunch! [13:27:49] back! nothing exploded so far [13:31:47] re-enabled eventlogging_sync everywhere, it is now pulling data correctly [13:31:56] soooooo db1046 migrated! \o/ [14:22:46] !log addshore@stat1005:/srv/analytics-wmde$ sudo -u analytics-wmde rm -rf /srv/analytics-wmde/installRlib [14:22:52] !log addshore@stat1005:/srv/analytics-wmde$ sudo -u analytics-wmde rm -rf /srv/analytics-wmde/r-library [14:22:53] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:22:57] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:34:55] 10Analytics, 10EventBus, 10MW-1.31-release-notes (WMF-deploy-2017-11-28 (1.31.0-wmf.10)), 10Patch-For-Review, 10Services (next): Timeouts on event delivery to EventBus - https://phabricator.wikimedia.org/T180017#3763080 (10Pchelolo) I've increased request timeout in #eventbus extension to 10 seconds to m... [14:47:46] Congrats elukey :) [15:06:11] 10Analytics, 10EventBus, 10MW-1.31-release-notes (WMF-deploy-2017-11-14 (1.31.0-wmf.8)), 10Patch-For-Review, 10Services (next): Timeouts on event delivery to EventBus - https://phabricator.wikimedia.org/T180017#3763155 (10mobrovac) https://groups.google.com/forum/#!msg/python-tornado/qqF5JQdP6XU/ZB8bI4AI... [15:17:11] 10Analytics, 10Services (later): Update node-rdkafka to v2 - https://phabricator.wikimedia.org/T180591#3763168 (10mobrovac) [15:17:17] 10Analytics, 10ChangeProp, 10EventBus, 10Reading-Infrastructure-Team-Backlog, and 2 others: Update node-rdkafka version to 2.0 - https://phabricator.wikimedia.org/T176126#3763170 (10mobrovac) [15:18:27] 10Analytics, 10ChangeProp, 10EventBus, 10Reading-Infrastructure-Team-Backlog, and 2 others: Update node-rdkafka version to v2.x - https://phabricator.wikimedia.org/T176126#3614140 (10mobrovac) p:05Triage>03High [15:37:13] joal: if you wanna talk before standup, or after, I'm all yours [15:38:02] heya milimetric - in ops meeting with Luca and Andre [15:51:09] milimetric: 10 minutes now if you want, or after :) [15:56:18] joal: omw cave [16:01:42] (03CR) 10Mforns: [C: 031] "Looks really good!" (033 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/391003 (https://phabricator.wikimedia.org/T178084) (owner: 10Fdans) [16:23:08] milimetric: cave for finishing that talk? [17:22:01] 10Analytics, 10EventBus, 10MW-1.31-release-notes (WMF-deploy-2017-11-14 (1.31.0-wmf.8)), 10Patch-For-Review, 10Services (next): Timeouts on event delivery to EventBus - https://phabricator.wikimedia.org/T180017#3763565 (10Ottomata) > https://groups.google.com/forum/#!msg/python-tornado/qqF5JQdP6XU/ZB8bI4... [17:28:01] 10Analytics, 10Phabricator: Create phabricator space for tickets with legal restrictions - https://phabricator.wikimedia.org/T174675#3763578 (10Aklapper) This is blocked on > * Add one or two Analytics folks (who?) Someone please answer. [17:29:17] 10Analytics, 10Phabricator: Create phabricator space for tickets with legal restrictions - https://phabricator.wikimedia.org/T174675#3763584 (10Aklapper) >>! In T174675#3609362, @ZhouZ wrote: > Thanks also Legal might also want to create a separate Legal space in addition to the one with Analytics folks [[ ht... [17:29:34] 10Analytics, 10Phabricator: Create phabricator space for tickets with legal restrictions - https://phabricator.wikimedia.org/T174675#3763586 (10Aklapper) 05Open>03stalled [17:37:02] joal: I'm free now if you are [17:42:24] (Joseph is gone for dinner - copying from research channel) [17:54:56] * elukey off! [17:56:10] 10Analytics, 10DBA, 10Operations, 10Patch-For-Review, 10User-Elukey: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3763672 (10elukey) [17:56:13] 10Analytics-Kanban, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: rack and setup db1107 and db1108 - https://phabricator.wikimedia.org/T177405#3763670 (10elukey) 05Open>03Resolved All the work has been completed, closing the task! [18:57:45] 10Analytics, 10EventBus, 10MW-1.31-release-notes (WMF-deploy-2017-11-14 (1.31.0-wmf.8)), 10Patch-For-Review, 10Services (next): Timeouts on event delivery to EventBus - https://phabricator.wikimedia.org/T180017#3763857 (10Ottomata) > Event production to Kafka times out I'm pretty sure this will be logge... [19:33:34] Thanks milimetric for following me in me lost-in-channels [20:13:48] 10Analytics-Kanban, 10Patch-For-Review: Remove AppInstallIId from EventLogging purging white-list - https://phabricator.wikimedia.org/T178174#3764486 (10Tbayer) (To record some more information here while other conversations are ongoing:) >>! In T178174#3684426, @Tbayer wrote: > Folks, we spent quite a bit of... [20:19:36] 10Analytics, 10EventBus, 10MW-1.31-release-notes (WMF-deploy-2017-11-14 (1.31.0-wmf.8)), 10Patch-For-Review, 10Services (next): Timeouts on event delivery to EventBus - https://phabricator.wikimedia.org/T180017#3764512 (10Ottomata) Ok, I've merged https://gerrit.wikimedia.org/r/#/c/302372/ and deployed i... [20:23:16] 10Analytics, 10EventBus, 10MW-1.31-release-notes (WMF-deploy-2017-11-14 (1.31.0-wmf.8)), 10Patch-For-Review, 10Services (next): Timeouts on event delivery to EventBus - https://phabricator.wikimedia.org/T180017#3764555 (10Ottomata) Ergh, I was hoping we could use this to see if the timeouts would disappe... [20:23:58] 10Analytics, 10EventBus, 10MW-1.31-release-notes (WMF-deploy-2017-11-14 (1.31.0-wmf.8)), 10Patch-For-Review, 10Services (next): Timeouts on event delivery to EventBus - https://phabricator.wikimedia.org/T180017#3764560 (10Ottomata) FYI @elukey ^^ [20:31:51] hey joal :] you still around? [20:32:03] Hi mforns [20:32:07] whassup? [20:32:21] hi, qq: can scala-spark tests write files in hdfs? [20:32:38] no permits right? [20:33:04] mforns: it would mean tests are only runnable from a place having access to an HDFS system - nop [20:33:35] hmmm, any suggestion what to do if the code writes temp files? [20:34:10] mforns: temp files as files you generate, then read, as in checkpoint files? [20:34:32] joal, yes, the ingestion json files [20:34:38] and finally delete [20:35:42] mforns: I suggest testing json generation (without actually writing), then posting [20:35:53] 2 separate tests for 2 separate tests [20:35:54] aha [20:38:07] fiu, but then, I won't be able to test the rest of the functionality that needs that temp file... [20:38:44] hm, I don't understand why it would be needed [20:38:57] hmm, can you write a temp file outside of hdfs during tests? [20:39:05] since the tests will all run local [20:39:21] not sure what your temp file uri looks like, but you could start it with file:///tmp/... [20:39:27] instead of hdfs:///tmp/... [20:42:35] ottomata, that would make sense! the uri does not have any prefix, it's just /tmp/... [20:42:53] but the method I use is inputDfWithCount.write.json(tempFilePath) [20:43:12] mforns, I guess that assumes hdfs [20:43:20] mforns? ottomata xD [20:44:26] ya, mforns it probably does, you can try it with file:/// and see [20:44:36] ottomata, k! [20:54:41] I also think making sure the thing is testable without temporary files is prefereable from a code design perspective - jsut saying mforns ;) [20:55:02] joal, yea... will think of it [20:55:44] though in this case, the way druid is loaded demands a file [21:01:07] 10Analytics-Data-Quality, 10Reading-analysis: Number of nlwiki (biography) articles getting consistently ~70 hits per day for the past months - https://phabricator.wikimedia.org/T180621#3764715 (10Tbayer) [21:23:50] 10Analytics, 10DBA, 10Data-Services, 10Research, 10cloud-services-team (Kanban): Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3764830 (10zhuyifei1999) [22:02:15] (03PS1) 10Ottomata: Support consumption from multiple topics [analytics/statsv] - 10https://gerrit.wikimedia.org/r/391703 (https://phabricator.wikimedia.org/T179093) [22:07:15] 10Analytics, 10Analytics-Cluster, 10Patch-For-Review: Support multi DC statsv - https://phabricator.wikimedia.org/T179093#3764929 (10Ottomata) @Krinkle, I may have forgotten the exact implementation we talked about. Was this it? Should statsv instances in each DC consume all statsv messages from both DCs?... [22:07:31] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Support multi DC statsv - https://phabricator.wikimedia.org/T179093#3764931 (10Ottomata) [22:47:37] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Support multi DC statsv - https://phabricator.wikimedia.org/T179093#3765016 (10Krinkle) >>! In T179093#3764929, @Ottomata wrote: > @Krinkle, I may have forgotten the exact implementation we talked about. Was this it? Should statsv instances in eac...