[05:43:44] 10Analytics, 10Operations, 10SRE-Access-Requests: Nuria's volunteer account - https://phabricator.wikimedia.org/T266086 (10Marostegui) I can confirm https://phabricator.wikimedia.org/L2 has been signed by @Nuria We'd still need a C-Level approval for this. I will seek Grant's approval for this [05:45:47] 10Analytics, 10Operations, 10SRE-Access-Requests: Nuria's volunteer account - https://phabricator.wikimedia.org/T266086 (10Marostegui) >>! In T266086#6575066, @Dzahn wrote: > The offboarding script has an option for "stay volunteer". This is very useful! Thanks [07:29:18] bonjour [07:46:47] 10Analytics-Clusters, 10Operations: Rename an-scheduler1001 to an-coord1002 - https://phabricator.wikimedia.org/T265620 (10elukey) 05Open→03Resolved [07:48:05] 10Analytics-Clusters: Review recurrent Hadoop worker disk saturation events - https://phabricator.wikimedia.org/T265487 (10elukey) I added some Datanode metrics to the Hadoop grafana dashboard, and started 3 iotop sessions (dumping to a file) on an-worker108[1-3] to get some idea about what processes are hammeri... [07:56:08] Good morning [07:58:00] 10Analytics-Clusters: Review recurrent Hadoop worker disk saturation events - https://phabricator.wikimedia.org/T265487 (10elukey) [08:02:17] 10Analytics, 10Operations, 10SRE-Access-Requests: Nuria's volunteer account - https://phabricator.wikimedia.org/T266086 (10MoritzMuehlenhoff) >>! In T266086#6575708, @Dzahn wrote: >>>! In T266086#6575705, @Stashbot wrote: >> {nav icon=file, name=Mentioned in SAL (#wikimedia-operations), href=https://sal.t... [08:29:00] 10Analytics, 10Product-Analytics: Analyze differences between checksum-based and revert-tag based reverts in mediawiki_history - https://phabricator.wikimedia.org/T266374 (10JAllemandou) The expected difference is to have more (possibly many more) tag-based reverts than checksum-based reverts. [08:31:58] 10Analytics-Clusters: Review recurrent Hadoop worker disk saturation events - https://phabricator.wikimedia.org/T265487 (10elukey) There was an error on https://grafana.wikimedia.org/d/VSyI1AWMk/cluster-overview-thanos and the other one (user cluster overview), namely read/write metrics were supposed to be -/+ b... [08:36:13] 10Analytics-Clusters: Review recurrent Hadoop worker disk saturation events - https://phabricator.wikimedia.org/T265487 (10elukey) [08:37:18] there was an viz error in one of the cluster overview dashboards, the disk saturation events seems to happen when read load traffic happens [08:37:21] not write [08:37:54] elukey: interesting - The usual workload of workers is indeed a heavy-read (more than heavy write) [08:40:28] it might be something that we can't really avoid, or it might be something that we can improve via settings [08:41:53] elukey: I'd be interested to see if small-files read would lead to more saturation than big-files read (big sequential read is supposedly better) [08:43:21] maybe we could use http://www.brendangregg.com/ebpf.html#bcc (will be rolled out soon to stretch nodes) [08:43:47] but stuff like iotop tells you only what processes are hitting the disks [08:44:17] I added some datanode metrics to grafana as well [08:44:32] elukey: With iotop manual exploration we probably can manage to get some info, but it'll be complicated [08:44:48] the grafana metrics are great :) [08:46:57] but as always more questions than answers :D [08:47:58] ahahahah just seen the token [08:48:16] :) [08:48:21] joal: the other iiiiinteresting one is https://phabricator.wikimedia.org/T266322 (maybe more pressing) [08:48:47] not sure if you seen it [08:51:09] interesrting https://github.com/maxmind/GeoIP2-java/issues/129 [08:51:33] elukey: I have seen that, it's in my priority ticket :) [08:52:09] thanks a lot :) [08:52:26] I am not sure why I didn't get this before [08:52:27] elukey: I have an idea, will start working once my emails and com chans are empty [08:52:39] ah yes even tomorrow joal [08:54:45] luckily the new test cluster unveiled some things to test [08:54:58] for example, I didn't even think about the consequences of having all stat100x on buster [08:55:04] since bigtop packages are only for stretch [08:55:05] :D [08:55:38] elukey: not only problems is great :) [08:55:40] Razzi created an-test-client1001, and the problem arose.. for the moment I added the stretch packages to buster and it seems to work [08:56:02] elukey: I'm also a big fan of the CNAME/LVs change for an-coords - I think this will be great [08:56:13] (we are using CDH jessie packages on buster so it shouldn't be an issue, but better testing..) [08:56:24] joal: ah yes I wanted to know your opinion! Good :) [09:01:01] elukey: another thing - I have experienced problems with pagination of oozie-jobs in hue - Do we have that on our radar? [09:02:20] not that I know, if you could report it in https://phabricator.wikimedia.org/T264896 it would be great [09:02:33] (also how to repro, so I can collect logs etc..) [09:04:04] Addin a comment elukey - thanks for the ticket [09:10:55] 10Analytics, 10Analytics-Kanban: Fix the remaining bugs open on for Hue next - https://phabricator.wikimedia.org/T264896 (10JAllemandou) I have experienced problems with jobs pagination as well: - Go to a webrequest-load coordinator page (or any cooridnator with many historical jobs) - The page shows the last... [09:11:03] done elukey --^ [09:11:51] super thanks [09:26:00] (03CR) 10Joal: "See comments inline. Also this means that schemas having objects to be defined as struct (vs map) must have all their properties defined a" (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/629406 (https://phabricator.wikimedia.org/T263466) (owner: 10Ottomata) [09:27:19] elukey: Starting to investigate the GeoIP problem in test cluster [09:27:41] elukey: the test cluster being bigtop with new versions and all, have you updated the refinery-source dependencies? [09:29:17] joal: I am using the same refinery source as production, in the past we didn't have to do anything to make webrequest_load working [09:29:34] but I haven't kept up with the refinery deployments, so something might have changed [09:30:17] elukey: I don't think anything changed - it feels bizarre that jobs worked as-is with new hive versions and all - but ok :) [09:30:39] well I can confirm it since I kept checking for errors etc. [09:31:03] the idea was to modify deps, if possible, only after the move to bigtop [09:32:58] elukey: the jump between refinery v83 and v137 feels big - will investage what happens [09:34:02] yes it was :( [09:44:46] elukey: is the wikitech page for tst cluster up-to-date for hosts? [09:51:07] joal: which one are you checking? [09:51:26] elukey: I'm on https://wikitech-static.wikimedia.org/wiki/User:Elukey/Analytics/Hadoop_testing_cluster [09:51:31] I think it's old [09:52:10] it is, we probably need a more canonical page [09:52:22] all hosts have the an-test- prefix in front [09:52:43] an-test-master100[1,2], an-test-coord1001, an-test-worker100[1-3] [09:52:55] and an-test-client1001 or an-tool1006 (buster vs stretch) [09:53:08] Ah right - I forgot - sorry :S [09:53:51] elukey: I assume we expect to launch the webrequest job from an-test-client - right? [09:55:34] in theory yes, I have a script on an-tool1006 in my home [09:55:43] /home/elukey/launch_webrequest_bundle.sh [09:56:04] I have upoloaded my oozie modified dir to /user/elukey [09:56:28] So the test leading to the error was launched from an-t00l1006 with that script - ok thanks for the details :) [09:56:37] will try to repro [09:58:32] yes correct [09:58:43] ah there is no hue yet [09:59:02] so I checked logs via oozie cli [10:00:09] ack [10:08:10] 10Analytics-Clusters, 10Analytics-Kanban: Add more metrics to prometheus-amd-rocm-stats Python script - https://phabricator.wikimedia.org/T262427 (10elukey) 05Open→03Resolved [10:39:01] Morning [10:45:19] good morning :) [10:48:08] (03CR) 10Joal: Add oozie webrequest test bundle (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/491791 (https://phabricator.wikimedia.org/T212259) (owner: 10Elukey) [10:50:49] (03CR) 10Elukey: Add oozie webrequest test bundle (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/491791 (https://phabricator.wikimedia.org/T212259) (owner: 10Elukey) [10:51:24] (03PS7) 10Elukey: Add oozie webrequest test bundle [analytics/refinery] - 10https://gerrit.wikimedia.org/r/491791 (https://phabricator.wikimedia.org/T212259) [11:04:10] elukey: I'm using your version of refinery-test - You might receive a bunch of alert emails - Let me know if it is a problem, I'll change the addresses [11:07:25] nono please go forward [11:07:41] I have a dir in my inbox for spam from an-test-coord :) [11:07:55] I knew you'd have somthing ;) [11:23:10] What was that Oozie-Spam on Friday night/the weekend, btw? [11:23:25] klausman: I think 2 things happened [11:23:54] klausman: some test jobs were started with prod alert email (see emails about that) [11:24:45] klausman: and some changes were made on camus with errors, leading to data not being imported on HDFS, and jobs sending SLA errors because they were late [11:25:05] I might be missing something, but I think that's most of it [11:25:16] yep [11:29:52] morning team [11:29:56] looking at alarms [11:38:49] Fascinating :) [11:46:55] elukey: would you have a moment to supervise the deletion of leila's old hive database? [11:47:55] fdans: good morning! Is it ok if we do it after EU lunch time? [11:48:20] elukey: sure thing! [11:49:17] ack thanks! [12:11:38] * elukey lunch! [12:43:25] 10Analytics: Check home/HDFS leftovers of rodolfovalentim - https://phabricator.wikimedia.org/T266467 (10MoritzMuehlenhoff) [13:52:52] 10Analytics, 10Product-Analytics: Add timestamps of important revision events to mediawiki_history - https://phabricator.wikimedia.org/T266375 (10Isaac) Thanks @nettrom_WMF for creating this ticket. I think I'm going to leave it just as Morten requested (which I agree would be useful) because I was misremember... [14:09:32] (03CR) 10Ottomata: Make explicit that _SUCCESS flag is written (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/636051 (owner: 10Milimetric) [14:13:01] ottomata: hellooo I'm working on the task to ban Fuzz Faster U Fool [14:14:41] ottomata: I don't think we have a user agent blacklist that we use to discard events right? I was thinking adding the check to the place that we check for user agent length [14:17:33] (03CR) 10Ottomata: [C: 04-1] "Sorry didn't mean to make you review, I just added the [WIP] tag to the commit message so we could let this sit. There's much more to tal" (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/629406 (https://phabricator.wikimedia.org/T263466) (owner: 10Ottomata) [14:26:07] ottomata: hmmm I see, we would probably do it at the processor level, because it would never reach the user agent parsing step [14:26:43] like, the solution we talked about last week is for this user agent, skipping the issuance of EventErrors [14:31:13] elukey: good time now? to do it before standup :) [14:40:06] ottomata: yeah, it writes that flag (though I’m not 100% clear on what code exactly writes it, something between the parquet write function and whatever interaction that has with Hive) [14:40:40] fdans: hiyaaa [14:40:42] sorry just saw ping [14:41:32] fdans: yeah this is tricky because the event just totally fails parsing beacuse it is too long, the UA given to us separately from the event though (in the log line data from varnishkafka) [14:41:47] so you'd need some custom stuff to examine the raw log line before it is parsed [14:42:08] or pass it thorugh a custom 'parse' function that only extracts the UA (via %u ?) [14:42:12] and then check it? [14:42:35] yeah milimetric i tried googling around too, it seems like it is the parquet writer somehow [14:42:42] which i think ends up using some hadoop parquet code that does it [14:42:44] very strange [14:42:52] in any case milimetric i think that is the wrong flag to use for data jobs [14:43:10] we don't right it explicitly, and if we used a different format than parquet, it wouldn't exist (IIUC) [14:45:29] fdans: sorry in meetings :( [14:45:31] agreed, we should use _REFINED. Either way, we can't make that job use canaries, we need some other solution for cross-datacenter data consolidation [14:45:51] ok that's where we wanted to discuss! [14:45:57] milimetric: early BC for that now? [14:46:01] yea [14:47:29] fdans: I have meetings also after ours, a mess today, but we can do it later on [14:47:52] should be quick, maybe in the meantime check how/what to drop the db so we just pull the trigger when we meet [14:48:02] elukey: nono I don't want to extend your day, let's do it tomorrow [14:48:03] (there should be some documentation about nuking a db) [14:48:09] I already have the hive command [14:48:14] nice [15:11:38] (03PS1) 10Milimetric: Use _REFINED flag instead of side-effect _SUCCESS [analytics/refinery] - 10https://gerrit.wikimedia.org/r/636442 [15:31:53] (03CR) 10Joal: [C: 03+1] "Yes!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/636442 (owner: 10Milimetric) [15:32:48] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Use _REFINED flag instead of side-effect _SUCCESS [analytics/refinery] - 10https://gerrit.wikimedia.org/r/636442 (owner: 10Milimetric) [15:41:45] ping razzi klausman [15:47:57] 10Analytics, 10Product-Infrastructure-Data, 10Wikimedia-Logstash, 10observability: Create a separate logstash ElasticSearch index for schemaed events - https://phabricator.wikimedia.org/T265938 (10lmata) hey @Ottomata! we've been working on a consolidated logging schema that might prove to be very helpful... [15:48:50] 10Analytics, 10Analytics-Kanban: Possible issue between Maxmind and Hive 2.x libs in Refinery source - https://phabricator.wikimedia.org/T266322 (10JAllemandou) a:03JAllemandou [15:49:14] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats active editors metric reporting unrealistic numbers - https://phabricator.wikimedia.org/T265322 (10Milimetric) [15:51:22] 10Analytics-Radar, 10Operations, 10SRE-Access-Requests: Nuria's volunteer account - https://phabricator.wikimedia.org/T266086 (10fdans) [15:54:54] 10Analytics, 10Event-Platform, 10Platform Engineering Roadmap Decision Making, 10Platform Team Workboards (S&F Workboard): Need for new event-type - `user_create` and `user_rename` - https://phabricator.wikimedia.org/T262205 (10Helga_sf) [15:56:58] 10Analytics: Retain nonsensitive mediawiki_api_request logging data - https://phabricator.wikimedia.org/T265952 (10fdans) p:05Triage→03High [15:57:50] 10Analytics-Radar, 10Discovery-Search, 10MediaWiki-General: Proposal: drop avro dependency from mediawiki - https://phabricator.wikimedia.org/T265967 (10Ottomata) +1 [15:57:54] 10Analytics-Radar, 10Discovery-Search, 10MediaWiki-General: Proposal: drop avro dependency from mediawiki - https://phabricator.wikimedia.org/T265967 (10fdans) [15:58:53] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10Patch-For-Review: Make node-rdkafka an optional dependency of EventGate - https://phabricator.wikimedia.org/T266058 (10fdans) p:05Triage→03High [16:01:20] 10Analytics: Retain nonsensitive mediawiki_api_request logging data - https://phabricator.wikimedia.org/T265952 (10Milimetric) Just following up, the data is being collected on an ongoing basis, and we always have the last 90 days of data. (I initially made a mistake by looking only at eqiad but right now the d... [16:03:21] 10Analytics, 10Browser-Support-Apple-Safari: On analytics.wikimedia.org, Safari on iOS ignores collapsed tabs interaction - https://phabricator.wikimedia.org/T266122 (10fdans) [16:03:23] 10Analytics, 10Analytics-Kanban: analytics.wikimedia.org TLC - https://phabricator.wikimedia.org/T253393 (10fdans) [16:03:36] 10Analytics: Stats dashboards on analytics.wikimedia.org are poorly navigable on mobile devices - https://phabricator.wikimedia.org/T266142 (10fdans) [16:03:38] 10Analytics, 10Analytics-Kanban: analytics.wikimedia.org TLC - https://phabricator.wikimedia.org/T253393 (10fdans) [16:04:07] 10Analytics, 10Mobile: analytics.wikimedia.org is less readable on smartphones - https://phabricator.wikimedia.org/T266132 (10fdans) [16:04:09] 10Analytics, 10Analytics-Kanban: analytics.wikimedia.org TLC - https://phabricator.wikimedia.org/T253393 (10fdans) [16:05:41] 10Analytics, 10Browser-Support-Apple-Safari: On analytics.wikimedia.org, Safari on iOS ignores collapsed tabs interaction - https://phabricator.wikimedia.org/T266122 (10fdans) p:05Triage→03Low [16:06:07] 10Analytics, 10Mobile: analytics.wikimedia.org is less readable on smartphones - https://phabricator.wikimedia.org/T266132 (10fdans) p:05Triage→03Low [16:06:09] 10Analytics: Stats dashboards on analytics.wikimedia.org are poorly navigable on mobile devices - https://phabricator.wikimedia.org/T266142 (10fdans) p:05Triage→03Low [16:09:03] 10Analytics, 10Analytics-Kanban: Possible issue between Maxmind and Hive 2.x libs in Refinery source - https://phabricator.wikimedia.org/T266322 (10fdans) p:05Triage→03High [16:18:45] 10Analytics, 10Product-Analytics: Add timestamps of important revision events to mediawiki_history - https://phabricator.wikimedia.org/T266375 (10fdans) p:05Triage→03Medium [16:28:48] 10Analytics-Clusters: Review recurrent Hadoop worker disk saturation events - https://phabricator.wikimedia.org/T265487 (10elukey) As far as I can see from iotop's logs, a lot of the following are present when disks are saturated: ` 10:19:19 29017 be/4 hdfs 4597.34 K/s 0.00 K/s 0.00 % 1.09 % java -Dpro... [16:30:21] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Add data quality alarm for mobile-app data - https://phabricator.wikimedia.org/T257692 (10fdans) a:05Nuria→03mforns [16:32:50] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Stats for newer projects not available - https://phabricator.wikimedia.org/T258033 (10JAllemandou) [16:32:54] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats - Add avk.wikipedia.or to scoop list - https://phabricator.wikimedia.org/T264660 (10JAllemandou) [16:37:04] 10Analytics: Fix the remaining bugs open on for Hue next - https://phabricator.wikimedia.org/T264896 (10fdans) p:05High→03Medium [16:39:05] 10Analytics, 10Analytics-Kanban: Test hudi and Iceberg as an incremental update system using 2 mediawiki-history snapshots - https://phabricator.wikimedia.org/T262256 (10JAllemandou) [16:39:44] 10Analytics, 10Analytics-Kanban: Test hudi and Iceberg as an incremental update system using 2 mediawiki-history snapshots - https://phabricator.wikimedia.org/T262256 (10JAllemandou) Next up: test Apache Iceberg when data mutation feature (Copy on Write) is released. [16:44:32] 10Analytics-Radar, 10Discovery-Search, 10MediaWiki-General: Proposal: drop avro dependency from mediawiki - https://phabricator.wikimedia.org/T265967 (10dcausse) +1 [16:44:56] 10Analytics, 10Analytics-Wikistats: Define reduce calculations needed to compute active editors per project family - https://phabricator.wikimedia.org/T249751 (10fdans) [16:45:05] 10Analytics, 10Patch-For-Review: HiveExtensions.convertToSchema does not properly convert arrays of structs - https://phabricator.wikimedia.org/T259924 (10fdans) [16:45:16] 10Analytics, 10Event-Platform, 10Patch-For-Review: Refine drops $schema field values - https://phabricator.wikimedia.org/T255818 (10fdans) [16:47:22] 10Analytics, 10Analytics-Kanban: Archive /home/ezachte data on stat1007 - https://phabricator.wikimedia.org/T238243 (10fdans) a:03fdans [16:47:43] (03CR) 10Bstorm: multiinstance: Attempt to make quarry work with multiinstance replicas (034 comments) [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/632804 (https://phabricator.wikimedia.org/T264254) (owner: 10Bstorm) [16:53:34] this is very nice https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.5/bk_hdfs-administration/content/configuring_short_circuit_local_reads_hdfs.html [16:54:13] it seems that there is a way to speed up client to datanode reads, bypassing the datanode's heavy lift [16:54:27] but it would probably make our problem of disk saturation worse :D [17:07:45] elukey: I assume so :) [17:12:50] (03PS2) 10Joal: Update MediawikiXMLDumpsConverter repartitioning [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/629659 (https://phabricator.wikimedia.org/T263736) [17:22:04] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats active editors metric reporting unrealistic numbers - https://phabricator.wikimedia.org/T265322 (10fdans) a:05Milimetric→03fdans [17:22:11] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Add dimensions to editors_daily dataset - https://phabricator.wikimedia.org/T256050 (10JAllemandou) Ping @cchen on the code review - The CR moves slowly, is that ok for you @cchen or should we try to find a better coordination? [17:58:10] * elukey afk! [18:17:20] (03CR) 10Ottomata: Make explicit that _SUCCESS flag is written (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/636051 (owner: 10Milimetric) [18:35:26] 10Analytics, 10Product-Infrastructure-Data, 10Wikimedia-Logstash, 10observability: Create a separate logstash ElasticSearch index for schemaed events - https://phabricator.wikimedia.org/T265938 (10Ottomata) Hiya! Interesting indeed. Maybe share some details and we can discuss? Happy to have a meeting too. [18:39:14] 10Analytics-Radar, 10Operations, 10SRE-Access-Requests: Nuria's volunteer account - https://phabricator.wikimedia.org/T266086 (10gsingers) Having never dealt with this before here, can someone clarify for me what exactly keeping access entails and how it compares to our policies? In other words, what am I be... [18:46:12] (03CR) 10Mforns: [V: 03+2] "@Arzhel, @Faidon: One question inline. Thanks!" (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/634328 (https://phabricator.wikimedia.org/T254332) (owner: 10Mforns) [19:44:12] hey all, question for you all. Can a machine on codw access the analytics cluster? Or can only boxes in eqiad access it? [19:48:34] Hi chrisalbon - I have only accessed the cluster from eqiad machines (and more precisely machines in the analytics vlan) - However I know some kafka hosts from codfw send data to eqiad-analytics kafka cluster [19:48:49] ottomata: any insight about chrisalbon question? --^ [19:49:39] chrisalbon: generally: yes [19:49:55] there are no network rules in place to restrict that (but there are to restrict analytics VLAN to other networks) [19:50:06] there are local iptables/ferm firewall rules though [19:50:08] ah! thanks ottomata :) [19:50:34] ah got it, thanks [20:02:43] 10Analytics, 10Patch-For-Review, 10User-Elukey: Move https termination from nginx to envoy (if possible) - https://phabricator.wikimedia.org/T240439 (10razzi) [20:17:42] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Filter out EventLogging data with bunk user-agents - https://phabricator.wikimedia.org/T266130 (10Ottomata) [20:33:45] (03PS14) 10Bstorm: multiinstance: Attempt to make quarry work with multiinstance replicas [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/632804 (https://phabricator.wikimedia.org/T264254) [22:07:00] 10Analytics, 10MediaWiki-Page-editing, 10Platform Engineering, 10User-DannyS712: EditPage save hooks pass an entire `EditPage` object - https://phabricator.wikimedia.org/T251588 (10Pchelolo) [22:49:41] 10Analytics, 10MediaWiki-Page-editing, 10Platform Engineering, 10User-DannyS712: EditPage save hooks pass an entire `EditPage` object - https://phabricator.wikimedia.org/T251588 (10DannyS712) For now, the constraint system is completely internal, and until the backend is fully factored out of EditPage it s... [22:58:45] (03PS15) 10Bstorm: multiinstance: Attempt to make quarry work with multiinstance replicas [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/632804 (https://phabricator.wikimedia.org/T264254) [23:04:36] (03CR) 10Bstorm: multiinstance: Attempt to make quarry work with multiinstance replicas (034 comments) [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/632804 (https://phabricator.wikimedia.org/T264254) (owner: 10Bstorm)