[02:36:48] (03PS1) 10Fdans: Expand EZ project conversion to adapt to raw format [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/632597 [06:27:29] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10Patch-For-Review: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10ayounsi) >>! In T254332#6523115, @mforns wrote: > This, I assume, needs to be added to the pmacct producer? That's correct,... [06:52:09] good morning [07:34:47] 10Analytics, 10User-Elukey: Move https termination from nginx to envoy (if possible) - https://phabricator.wikimedia.org/T240439 (10elukey) @razzi a little bit of context on this, if not clear please ask any questions in the task and we'll try to answerer asap (or we could chat over meet, no problem). **Curre... [08:45:46] !log decom analytics1042 from hadoop [08:45:47] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:47:23] as expected under replicated blocks for hdfs spiked, let's see how much time it will take [08:51:08] PROBLEM - Hadoop NodeManager on analytics1042 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [08:54:46] yarn nodemanager gracefully shutsdown, all good [08:56:12] ~1.8M blocks to replicate, it will take 2/3 hours I guess [09:04:16] * elukey bbiab [09:22:06] 10Analytics-Clusters, 10Operations, 10Traffic: varnishkafka 1.1.0 CPU usage increase - https://phabricator.wikimedia.org/T264074 (10Vgutierrez) yeah.. I'll handle the backport :) [09:33:01] 10Analytics-Clusters, 10Operations, 10Traffic: varnishkafka 1.1.0 CPU usage increase - https://phabricator.wikimedia.org/T264074 (10Vgutierrez) @elukey double checking https://gerrit.wikimedia.org/r/plugins/gitiles/operations/debs/varnish4/+/refs/heads/debian-wmf/lib/libvarnishapi/vut.c#424 it looks like ed1... [09:33:33] 10Analytics: Categorylinks dump might have some problem with the encoding - https://phabricator.wikimedia.org/T264850 (10marcmiquel) [09:49:08] 10Analytics-Clusters, 10Operations, 10Traffic: varnishkafka 1.1.0 CPU usage increase - https://phabricator.wikimedia.org/T264074 (10elukey) ah snap I didn't check, will do the next time using the gerrit repo. It is not clear to me why we have all those fstat calls though.. [10:05:02] I was too optimistic, the node drain might take 5/6 hours :( [10:22:10] Ouch. [10:22:42] In lighter news, I now have a tool that will crawl a tree and dump out all the usernames/group names and paths [10:23:11] I plan to extend it to be able to re-enact said un/gn on a tree as needed. [10:31:22] klausman,elukey sorry to always bring GPU bad news :) [10:32:00] uh-oh [10:32:23] what happens? [10:32:52] so I launched yesterday a GAN training on stat1008 - after elukey's symlink fix everything seemed to work fine until now. It now seems stuck, but I don't see any excessive memory usage when I do randeontop [10:33:07] maybe it's me, I did something wrong [10:33:20] which machine? [10:33:27] stat1008 [10:33:43] klausman: the symlink fix is the following, I did it manually and forgot to tell you [10:33:46] elukey@stat1008:~$ ls -l /opt/rocm [10:33:49] lrwxrwxrwx 1 root root 10 Oct 5 18:12 /opt/rocm -> rocm-3.3.0 [10:34:01] otherwise tf wasn't able to find /opt/rocm/lib [10:34:25] *nod* [10:34:34] I am now trying to launch something, and see whether it works, I just wanted to double check that there is nothing else I should be aware of on stat1008 :) I'll keep you posted! [10:35:10] it looks like it's saturating the VRAM as soon as I launch something, but maybe what I am doing is too heavy [10:35:37] OHH wait - it's back to normal :) [10:35:48] sorry sorry, false alarm [10:36:04] :D [10:36:33] miriam: Tobias just used the force with the GPU, I am not able to do it :D [10:36:36] Still, I noticed we're not getting GPU RAM stats in Grafana anymore, so at least it's something I need to investigate [10:36:58] I have been known to fix hardware by staring daggers at it [10:37:16] I was about to say it, interestingly an-worker1096 (the only one that we rebooted) publishes metrics [10:37:18] (and by browsing webshops for replacement hw) [10:38:23] https://phabricator.wikimedia.org/P12947 [10:38:32] weird. [10:39:12] changed the labels selected in the grafana dropdown for "instance", now it polls "amd_rocm_gpu_fan_usage_percent" [10:39:21] and it now shows 1096/97 [10:40:51] for stat1005 it stopped on the 5th [10:40:52] https://grafana.wikimedia.org/d/ZAX3zaIWz/amd-rocm-gpu?orgId=1&from=now-7d&to=now&var-source=eqiad%20prometheus%2Fops&var-instance=stat1005:9100 [10:41:00] that is monday, when I rebooted them [10:41:22] same for 1008 [10:41:38] I am wondering if for some reason the kernel drivers takes precedence over the rock-dkms ones, is it possible? [10:41:55] Though there is a short blip of data on the 5th, at 1630, for 1008 [10:42:30] that was when Chris worked on it, since it wasn't booting for the power issue [10:42:35] (ram + gpu) [10:43:02] elukey :D btw, I think the problem was another user was using the GPU concurrently, maybe without adding the tensorflow memory growth bit - so I am reaching out to them now to make sure they do [10:44:09] miriam: super thanks, I think that we may have a bigger problem with GPU drivers :( [10:44:31] We should give 3.8 a shot soonish [10:44:44] miriam: also, wasn't that fixed with the last round of driver deployment that we did? [10:45:07] 3.8 may not solve these problems, but I doubt we'll get any support from upstream if we're running 3.3. [10:45:14] +100 [10:48:41] elukey: not sure if it was fixed, I always tried concurrent tasks using the memory growth catch [10:51:09] thanks and let me know if I can help! [10:53:21] ahh ok I thought we tested it [10:59:57] going to lunch! [11:18:37] elukey: I know why the memstats are gone. For some reason, 1008 loaded the old (non DKMS) amdgpu module on boot. [11:18:50] which might also explain the issues miriam saw [11:25:20] A-ha! [11:25:35] stat1008 has a dewer kernel, for the dkms was not build [12:08:19] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10Patch-For-Review: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10mforns) Thanks @ayounsi > `"comms": "14907:0_14907:2_14907:3"` > To mean that the flow has the 3 communities 14907:0 14907... [12:17:04] klausman: good finding! And again I think that root cause is me, since Moritz told me that a new kernel was installed but I didn't do dpkg-reconfigure rock-dkms and install new headers [12:17:17] didn't even think about it [12:17:49] in theory this could be done now, but there is the risk of a host-level stall [12:18:10] so I guess that we should send an email for maintenance? [12:20:31] *normally* a new kernel being installed should trigger kernel module rebuilds. That's like half the purpose of DKMS in the first place [12:21:28] /usr/lib/dkms/dkms_autoinstaller start $(uname -r) is the command to trigger DKMS rebuild for the running kernel [12:21:52] As for when: yes, we should send a maint' mail, and maybe do it tomorrow morning? [12:27:42] hellooo [12:28:00] klausman: I got the impression from doing dpkg-reconfigure that the module rebuild is skipped if the kernel's headers are not installed [12:28:05] elukey: we can merge that patch if you're ok with that [12:28:10] fdans: sure! [12:28:41] klausman: so maybe on GPU nodes we'll have to remember to install kernel + headers before rebooting? [12:28:49] (let me know if you think that I am crazy) [12:30:53] 10Analytics: Establish what data must be backed up before the HDFS upgrade - https://phabricator.wikimedia.org/T260409 (10elukey) Some notes about the amount of space that we could get with the backup cluster. We have the following hardware to use: * 16 nodes to decommission (analytics1042 -> analytics1057). To... [12:32:12] joal: --^ [12:32:55] elukey: ah yes, now I remember: I had to install the kernel headers manually for rock-dkms [12:33:17] I should have documented that somewhere [12:33:21] me too :D [12:34:34] Would installing linux-headers-amd64 help? [12:34:52] So when a new kernel image arrives, the headers are also updated? [12:35:41] moritzm: hi, when you have a moment we'd have some questions for you :) [12:36:12] fdans: do you want me to run manually the timer to see if it works etc..? [12:37:16] elukey: that would great, thank you luca :) [12:41:37] Oct 07 12:41:16 labstore1007 kerberos-run-command[31921]: Error: Argument parsing error: [12:41:40] Oct 07 12:41:16 labstore1007 kerberos-run-command[31921]: Error validating src list: [12:41:43] Oct 07 12:41:16 labstore1007 kerberos-run-command[31921]: hdfs:///wmf/data/archive/archive/pageview/complete/ does not exist [12:41:46] fdans: --^ [12:41:49] :) [12:42:04] do you want to create it? Otherwise I'll do it (check perms also, the other dirs should be a good reference) [12:42:28] elukey: wait the path is wrong [12:42:33] that's why it doesn't exist [12:43:10] * klausman bbiab, need to run an errand [12:43:18] elukey: it has an extra "archive" [12:44:02] elukey: although the destination directory has to be created, and I don't think I have permissions to do that [12:44:11] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10Patch-For-Review: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10ayounsi) > Awesome. Yes, as you said, Druid allows for multi-value dimensions. Either the Refine job or a subsequent job can... [12:44:24] fdans: ahhhh snap yes you are right [12:45:19] elukey: would it be possible for you to create the destination dir? [12:46:16] elukey: heya - do you have am inute? [12:46:52] fdans: I think we need to fix the timer, sending a code review (to avoid the extra /archive/) [12:46:55] joal: sure [12:47:04] fdans: https://gerrit.wikimedia.org/r/632724 - lemme know your thoughts [12:47:18] elukey: ohhh you were quicker! :) [12:47:34] fdans: elukey is always faster [12:47:40] ahahah yes sure [12:47:41] elukey: batcave for a sec? [12:47:51] joal: gimme a sec so I can merge the patch [12:47:59] sure elukey [12:50:38] fdans: ok it seems running, will keep an eye on it [12:50:43] joal: joining bc [12:51:14] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10Patch-For-Review: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10mforns) OK, after a very interesting chat with Joseph, here's our conclusions: * It would be cool to have the core of the r... [12:52:15] thanks a lot elukey [12:53:45] np! [13:19:08] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10Patch-For-Review: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10mforns) @ayounsi > Ok to merge anytime or should I sync up with you? I believe it's OK to merge, and that Refine should id... [13:21:05] 10Analytics, 10Analytics-Kanban: Set up automatic deletion/snitization for netflow data set in Hive - https://phabricator.wikimedia.org/T231339 (10mforns) A comment on sanitization: I was looking at T254332 and one option is to move the netflow data and table to the event database. This way, we could use the a... [13:52:20] (03CR) 10MNeisler: Add DesktopWebUIActionsTracking fields to eventlogging allowlist (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/631988 (https://phabricator.wikimedia.org/T263143) (owner: 10MNeisler) [14:02:52] 10Analytics: Establish what data must be backed up before the HDFS upgrade - https://phabricator.wikimedia.org/T260409 (10elukey) Me and Joseph had a chat about possible solutions/scenarios and we came up with this: * we start decomming the analytics1042 -> analytics1057, it should take a couple of weeks (one n... [14:09:36] 10Analytics-Clusters, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-09-15) upgrade/replace memory in stat100[58] - https://phabricator.wikimedia.org/T260448 (10Cmjohnson) a:05Cmjohnson→03wiki_willy assigning this to @wiki_willy to figure out whether we want to upgrade the power supplies [14:45:18] 10Analytics, 10Analytics-Kanban: Analytics Ops Technical Debt - https://phabricator.wikimedia.org/T240437 (10elukey) [14:45:20] 10Analytics-Clusters: Create a small VM to host jobs/system timers currently running on stat1007 - https://phabricator.wikimedia.org/T255265 (10elukey) 05Open→03Declined Declined for the moment, not super useful plus we don't have a lot of time for it. I'll re-open in case it is needed. [14:54:43] 10Analytics: Fix the remaining bugs open on for Hue next - https://phabricator.wikimedia.org/T264896 (10elukey) [14:55:55] 10Analytics-Kanban: Deprecate Python 2 software from the Analytics infrastructure - https://phabricator.wikimedia.org/T204734 (10elukey) [14:56:01] 10Analytics-Clusters, 10Analytics-Kanban, 10User-Elukey: Test if Hue can run with Python3 - https://phabricator.wikimedia.org/T233073 (10elukey) [14:56:03] 10Analytics-Clusters, 10Analytics-Kanban, 10User-Elukey: Test if Hue can run with Python3 - https://phabricator.wikimedia.org/T233073 (10elukey) 05Open→03Resolved Hue has been running on python3 (hue-next) for a while, I opened T264896 to fix the remaining bugs. Closing this for the moment. [14:56:17] 10Analytics: Fix the remaining bugs open on for Hue next - https://phabricator.wikimedia.org/T264896 (10elukey) [14:56:19] 10Analytics-Kanban: Deprecate Python 2 software from the Analytics infrastructure - https://phabricator.wikimedia.org/T204734 (10elukey) [14:58:35] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10Patch-For-Review: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10Nuria) >Maybe leaning towards using a transform function, because code would be shorter and less moving pieces? I think havi... [14:58:37] 10Analytics-Clusters: Create the new Hadoop test cluster - https://phabricator.wikimedia.org/T255139 (10Cmjohnson) [14:58:39] 10Analytics: Enable Security (stronger authentication and data encryption) for the Analytics Hadoop cluster and its dependent services - https://phabricator.wikimedia.org/T211836 (10Cmjohnson) [15:03:58] reminder: Ben's talk is happening NOW [15:04:02] a-team: friendly remainder that confluent call is going on: https://wikimedia.zoom.us/j/94665533658?pwd=SXVqclJqMGIzQ3JsYmV3V0I0ZHg4QT09#success [15:06:13] cc razzi and klausman in case they do not have an invite to zzom [15:06:19] * zoom meeting [15:06:29] It's in the calendar entry :) [15:59:50] a-team: let's listen to giuseppe's question (cc klaumasn, razzi) [16:00:04] nuria: there is the team-cto meeting now [16:00:11] ah, ok [16:00:22] but the cto is in the meeting too no? [16:00:40] a-team: let's move standup to after that meeting then? [16:00:43] I Grant will wait for the confluent meeting finishes [16:00:46] sure nuria [16:01:22] moved now [16:01:58] 10Analytics-Radar, 10Better Use Of Data, 10Product-Analytics, 10Epic, and 4 others: Session Length Metric. Web implementation - https://phabricator.wikimedia.org/T248987 (10jlinehan) [16:02:26] a-team: moved standup to be after cto meeting [16:02:31] ok [16:02:33] nuria: I'll be interested to talk about the confluent talk after meetings if you've been there :) [16:02:44] joal: ou yeah [16:05:58] 10Analytics: Establish what data must be backed up before the HDFS upgrade - https://phabricator.wikimedia.org/T260409 (10Nuria) >After talking with the, we chose to backup all data except for logs, raw data (unprocessed webrequest, events, and dumps), 2 month of webrequest, and processed wikitext (heavy). You m... [16:07:24] 10Analytics-Radar, 10Better Use Of Data, 10Product-Analytics, 10Product-Infrastructure-Data, and 4 others: Session Length Metric. Web implementation - https://phabricator.wikimedia.org/T248987 (10jlinehan) [16:38:28] nuria: we're still in cto meeting [16:43:23] 10Analytics-Clusters, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-09-15) upgrade/replace memory in stat100[58] - https://phabricator.wikimedia.org/T260448 (10wiki_willy) Weird, I've never seen PSU's being affected like that, after a memory before, so I'll take it as an action item to reach out to t... [16:43:26] mforns: k [16:43:32] mforns: taking to lexnasser here [16:52:55] razzi: standup! [17:10:14] 10Analytics: Establish what data must be backed up before the HDFS upgrade - https://phabricator.wikimedia.org/T260409 (10JAllemandou) > You mean "unprocessed events" ? cause we need to backup the sanitized and unsanitized versions "unprocessed" as in raw-from-camus (the `/wmf/data/raw` doesn't get backed-up). `... [17:19:01] razzi, klausman - do we want to do the ops sync, or not? [17:19:06] maybe even a quick one [17:19:28] also please check the meeting invite for friday :) [17:19:59] From my pov, most of what I'd bring to ops sync already was in the chat just now [17:20:31] But sure, if there's something syncworthy [17:20:55] ack then we can skip [17:21:04] Alrighty [17:21:31] SGTM [17:22:38] razzi: I left some comments today in the envoy task, if you have time can you tell me if it is clear or not? [17:23:05] (in case I can add more ideaS) [17:23:45] elukey: Yeah, I'll take a look [17:24:07] * razzi taking a walk for 20 minutes [17:31:34] Gone for diner, back after [17:33:02] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10Patch-For-Review: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10mforns) @ayounsi Confirmed that you can merge the changes that add BGP communities to pmacct! We'll be monitoring the kafka... [17:45:31] 10Analytics, 10Analytics-Wikistats, 10Inuka-Team, 10Language-strategy, and 2 others: Add more popular articles per country data to AQS - https://phabricator.wikimedia.org/T263697 (10Nuria) @lexnasser As a remainder here is the design document for the prior AQS endpoint: https://drive.google.com/drive/u/0/... [17:46:38] fdans: in the calendar the "grooming" meeting on thu does not change so the additional one you set I do not think is needed. [17:47:46] nuria: makes sense [17:52:21] * elukey afk! [17:54:27] 10Analytics, 10Analytics-Wikistats, 10Inuka-Team, 10Language-strategy, and 2 others: Have a way to show the most popular pages per country - https://phabricator.wikimedia.org/T207171 (10Nuria) @AMuigai @Amire80 @lexnasser is going to start working on this project this quarter. I propose we use a top 100 of... [18:00:56] 10Analytics, 10Analytics-Wikistats, 10Inuka-Team, 10Language-strategy, and 2 others: Have a way to show the most popular pages per country - https://phabricator.wikimedia.org/T207171 (10Milimetric) >>! In T207171#6526385, @Nuria wrote: > ... I propose we use a top 100 of articles per country where these ar... [18:01:11] elukey: can you re-enable my stat1007 user/data when you have a chance? [18:02:10] elukey: do we need to re-open https://phabricator.wikimedia.org/T235688 for lexnasser to get access? [18:11:51] lexnasser: i also asked moritzm via e-mail cause i am not sure what is the process when you "come back" it might be we need new keys [18:14:53] 10Analytics: Establish what data must be backed up before the HDFS upgrade - https://phabricator.wikimedia.org/T260409 (10JAllemandou) Ok after a quick chat with Dan it appears my way of presenting might have been confusing. Here are some clarification (hopefully?) We wish to backup everything **except** - raw d... [18:15:00] milimetric: --^ please :) [18:16:12] joal: ok, it's clearer now, but then something doesn't make sense. Why did you include /wmf/data/raw in your size estimate? Without it, you save about 140T (*2 -> 280T) [18:17:08] milimetric: the table is actually split (I should have made two tables) - the first row is everything, then rows to remove, finally total [18:17:17] nuria: thanks for reaching out to him, I'll get started on the AQS design doc [18:17:49] ah, I get it now joal [18:17:53] k, all good [18:18:33] milimetric: I just updated the comment, splitting tables [18:42:44] 10Analytics, 10Analytics-Data-Quality: page_id is null where it shouldn't be in mediawiki history - https://phabricator.wikimedia.org/T259823 (10Milimetric) @Meghajain171192 do ping me if you want to talk over this, I'm @milimetric on IRC [18:48:24] 10Analytics, 10Analytics-Kanban: Purge raw webrequest_stats and webrequest_stats_hourly - https://phabricator.wikimedia.org/T262826 (10JAllemandou) [18:48:37] 10Analytics, 10Analytics-Kanban: Purge raw webrequest_stats and webrequest_stats_hourly - https://phabricator.wikimedia.org/T262826 (10JAllemandou) a:03JAllemandou [18:49:01] 10Analytics-Data-Quality, 10Analytics-EventLogging, 10Analytics-Radar, 10Product-Analytics, and 3 others: WikiEditor records all edits as platform = desktop in EventLogging - https://phabricator.wikimedia.org/T249944 (10Milimetric) >>! In T249944#6482316, @DLynch wrote: > ...and make sure it didn't wildly... [18:56:39] 10Analytics, 10Analytics-Wikistats, 10Inuka-Team, 10Language-strategy, and 2 others: Have a way to show the most popular pages per country - https://phabricator.wikimedia.org/T207171 (10Amire80) >>! In T207171#6526431, @Milimetric wrote: >>>! In T207171#6526385, @Nuria wrote: >> ... I propose we use a top... [19:03:20] 10Analytics-Radar, 10Product-Analytics, 10Anti-Harassment (The Letter Song): Capture special mute events in Prefupdate table [4 hour spike] - https://phabricator.wikimedia.org/T261461 (10dbarratt) a:03dbarratt [20:00:02] 10Analytics-Clusters, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-09-15) upgrade/replace memory in stat100[58] - https://phabricator.wikimedia.org/T260448 (10wiki_willy) Spoke to our technical Dell rep today, and followed up with an email. Hopefully there's an easy way to get it working. If not,... [20:19:20] 10Analytics, 10Analytics-Wikistats, 10Inuka-Team, 10Language-strategy, and 2 others: Have a way to show the most popular pages per country - https://phabricator.wikimedia.org/T207171 (10Nuria) >Top 100 in each language. To be clear, this is mostly not possible due to privacy reasons as we can only releas... [20:21:27] 10Analytics: Update Wikidata usage metric - https://phabricator.wikimedia.org/T264945 (10Isaac) [20:23:02] 10Analytics: Update Wikidata usage metric - https://phabricator.wikimedia.org/T264945 (10Nuria) @JAllemandou this is a good example to be able to keep as many mediawiki history dumps as possible, correcting this metric can only go as far back as the snapshots we keep [20:23:19] 10Analytics: Update Wikidata usage metric - https://phabricator.wikimedia.org/T264945 (10Nuria) cc @Milimetric [23:02:15] (03PS1) 10Bstorm: multiinstance: Attempt to make quarry work with multiinstance replicas [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/632804 (https://phabricator.wikimedia.org/T264254) [23:02:45] (03CR) 10jerkins-bot: [V: 04-1] multiinstance: Attempt to make quarry work with multiinstance replicas [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/632804 (https://phabricator.wikimedia.org/T264254) (owner: 10Bstorm) [23:08:37] (03PS2) 10Bstorm: multiinstance: Attempt to make quarry work with multiinstance replicas [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/632804 (https://phabricator.wikimedia.org/T264254) [23:13:55] 10Quarry, 10Patch-For-Review, 10cloud-services-team (Kanban): Prepare Quarry for multiinstance wiki replicas - https://phabricator.wikimedia.org/T264254 (10Bstorm) Minimum working patch up. Hopefully review will improve it 😝 [23:49:36] (03CR) 10BryanDavis: multiinstance: Attempt to make quarry work with multiinstance replicas (033 comments) [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/632804 (https://phabricator.wikimedia.org/T264254) (owner: 10Bstorm)