[08:35:41] (03PS1) 10Addshore: Add advancedsearch to betafeature sount script [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/392593 (https://phabricator.wikimedia.org/T180570) [08:36:06] (03PS2) 10Addshore: Add advancedsearch to betafeature count script [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/392593 (https://phabricator.wikimedia.org/T180570) [08:36:10] (03CR) 10Addshore: [C: 032] Add advancedsearch to betafeature count script [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/392593 (https://phabricator.wikimedia.org/T180570) (owner: 10Addshore) [08:36:18] (03Merged) 10jenkins-bot: Add advancedsearch to betafeature count script [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/392593 (https://phabricator.wikimedia.org/T180570) (owner: 10Addshore) [08:38:28] (03CR) 10Addshore: "It looks like this hasn't been deployed yet (only things on the production branch get deployed) :(" [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/376849 (owner: 10Catrope) [08:38:33] (03PS1) 10Addshore: Add codemirror-syntax-highlight beta feature [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/392595 [08:38:42] (03CR) 10Addshore: [C: 032] Add codemirror-syntax-highlight beta feature [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/392595 (owner: 10Addshore) [08:38:51] (03Merged) 10jenkins-bot: Add codemirror-syntax-highlight beta feature [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/392595 (owner: 10Addshore) [08:38:54] (03PS1) 10Addshore: Add advancedsearch to betafeature count script [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/392596 (https://phabricator.wikimedia.org/T180570) [08:38:59] (03CR) 10Addshore: [C: 032] Add advancedsearch to betafeature count script [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/392596 (https://phabricator.wikimedia.org/T180570) (owner: 10Addshore) [08:39:08] (03Merged) 10jenkins-bot: Add advancedsearch to betafeature count script [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/392596 (https://phabricator.wikimedia.org/T180570) (owner: 10Addshore) [08:40:37] (03PS1) 10Addshore: Add .editorconfig [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/392597 [08:40:49] (03CR) 10Addshore: [C: 032] Add .editorconfig [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/392597 (owner: 10Addshore) [08:40:55] (03Merged) 10jenkins-bot: Add .editorconfig [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/392597 (owner: 10Addshore) [08:41:01] (03PS1) 10Addshore: Add .editorconfig [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/392598 [08:42:15] (03PS1) 10Addshore: Note about deployment in README [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/392599 [08:42:31] (03CR) 10Addshore: [C: 032] Note about deployment in README [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/392599 (owner: 10Addshore) [08:42:35] (03PS1) 10Addshore: Note about deployment in README [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/392600 [08:42:37] (03Merged) 10jenkins-bot: Note about deployment in README [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/392599 (owner: 10Addshore) [08:42:39] (03CR) 10Addshore: [C: 032] Note about deployment in README [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/392600 (owner: 10Addshore) [08:42:44] (03Merged) 10jenkins-bot: Note about deployment in README [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/392600 (owner: 10Addshore) [09:08:26] 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Improve purging for analytics-slave data on Eventlogging - https://phabricator.wikimedia.org/T156933#3777064 (10elukey) >>! In T156933#3774716, @jcrespo wrote: > Probably I am stating the obvious, but some thottling could be disabled if only affects the... [09:32:47] Morning elukey - Is today cassandra and spark-streaming time? [09:35:07] joal: Bonjour! If you gimme 20mins I'll be available, I am deploying the druid_exporter 0.4 [09:35:18] For sure elukey - ping me when ready :) [09:36:30] joal: https://phabricator.wikimedia.org/P6357 [09:37:21] * joal is eager to look at those metrics when the system will be announced :) [09:37:51] me too! [09:45:20] 10Analytics-Kanban: Add action api counts to graphite-restbase job - https://phabricator.wikimedia.org/T176785#3636597 (10JAllemandou) a:03JAllemandou [09:53:15] (03CR) 10Addshore: [C: 032] Add .editorconfig [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/392598 (owner: 10Addshore) [09:53:21] (03Merged) 10jenkins-bot: Add .editorconfig [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/392598 (owner: 10Addshore) [09:57:45] 10Analytics-Kanban: Add action api counts to graphite-restbase job - https://phabricator.wikimedia.org/T176785#3777182 (10JAllemandou) Check in which webrequext_source partition data lives (and in the mean time check for api/rest too): ``` SELECT webrequest_source, (uri_path LIKE '/api/rest_v1/%') AS rest,... [10:12:04] joal: I have time now if you have! [10:13:54] I do :) [10:14:51] elukey: batcave? [10:14:55] ack! [10:18:07] 10Analytics-Kanban: Add action api counts to graphite-restbase job - https://phabricator.wikimedia.org/T176785#3777250 (10JAllemandou) Ping @mobrovac, @Pchelolo, @Gilles , @Anomie, @Krinkle for naming confirmation: The existing metric is named `restbase.requests.varnish_requests`, and we plan to go with `MediaWi... [10:52:07] 10Analytics-Kanban, 10Services (watching): Add action api counts to graphite-restbase job - https://phabricator.wikimedia.org/T176785#3777332 (10mobrovac) >>! In T176785#3777250, @JAllemandou wrote: > The existing metric is named `restbase.requests.varnish_requests`, and we plan to go with `MediaWiki.api.varni... [11:01:49] mmm jsonrefine is failing for Popups [11:04:26] (03PS1) 10Joal: Update MR cassandra loader to local quorum write [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/392623 [11:04:38] yes elukey - I have seen that [11:04:50] elukey: I suggest we wait for mforns, he knows bes [11:05:14] it might be related to the time capsule refactoring, but no idea :( [11:05:17] 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Improve purging for analytics-slave data on Eventlogging - https://phabricator.wikimedia.org/T156933#3777375 (10jcrespo) With throttling I meant making either replication or sanitization faster- I think it created load problems in the past according to... [11:05:40] elukey: java CR sent :) [11:05:54] \o/ [11:06:51] (03CR) 10Elukey: [C: 031] Update MR cassandra loader to local quorum write [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/392623 (owner: 10Joal) [11:06:58] (03PS1) 10Joal: Update cassandra load jobs to local quorum write [analytics/refinery] - 10https://gerrit.wikimedia.org/r/392624 [11:07:04] elukey: --^ Now the oozie one [11:07:57] ok elukey - taking my break now, back at 14:00 [11:08:13] super! [11:22:31] (03PS4) 10Fdans: Add central notice component and detect adblock [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/383798 (https://phabricator.wikimedia.org/T177491) [11:22:57] (03CR) 10Fdans: Add central notice component and detect adblock (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/383798 (https://phabricator.wikimedia.org/T177491) (owner: 10Fdans) [11:23:41] 10Analytics, 10Operations, 10monitoring, 10netops, 10User-Elukey: Pull netflow data in realtime from Kafka via Tranquillity/Spark - https://phabricator.wikimedia.org/T181036#3777449 (10elukey) [12:00:31] * elukey lunch + errand! [12:59:05] 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Improve purging for analytics-slave data on Eventlogging - https://phabricator.wikimedia.org/T156933#3777709 (10elukey) >>! In T156933#3777375, @jcrespo wrote: > With throttling I meant making either replication or sanitization faster- I think it create... [14:02:50] joal: yoohoo [14:18:37] (03CR) 10Fdans: [C: 032] Add central notice component and detect adblock [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/383798 (https://phabricator.wikimedia.org/T177491) (owner: 10Fdans) [14:20:00] 10Analytics-Cluster, 10Analytics-Kanban, 10Analytics-Wikistats: Design map visualization on UI - https://phabricator.wikimedia.org/T175422#3777959 (10fdans) [14:20:20] 10Analytics-Cluster, 10Analytics-Kanban, 10Analytics-Wikistats: Design map visualization on UI - https://phabricator.wikimedia.org/T175422#3592991 (10fdans) a:03fdans [15:29:22] Hey analytics team! I’ve seen the emails about the reshuffling of databases and the log DB moving off dbstore1002. Looks like that’ll affect our page creation dashboard so we’ll have to update the configuration. Should I open a ticket for that? Should it be related to T156844? [15:29:23] T156844: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844 [15:31:08] elukey: what's your ticket for EL db upgrades? [15:32:16] ottomata: T156844 :) [15:32:32] Nettrom: hi! You can use that ticket yes! [15:33:02] oh hahah, i didn't read comment above :) [15:33:03] thanks [15:33:23] :D :D [15:33:23] elukey: i'm going to rename that ticket and drop the 'prep to' part [15:33:23] since it has all [15:33:41] 10Analytics, 10DBA, 10Operations, 10Patch-For-Review, 10User-Elukey: Decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3778201 (10Ottomata) [15:40:10] 10Analytics, 10DBA, 10Operations, 10Patch-For-Review, 10User-Elukey: Decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3108353 (10Nettrom) The data behind [[ https://page-creation.wmflabs.org/#projects=nlwiki,eswiki,plwiki,itwiki,enwiki,jawiki,dewiki,svwiki,ruwik... [15:41:58] 10Analytics, 10DBA, 10Operations, 10Patch-For-Review, 10User-Elukey: Decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3778229 (10elukey) >>! In T156844#3778225, @Nettrom wrote: > The data behind [[ https://page-creation.wmflabs.org/#projects=nlwiki,eswiki,plwiki... [15:43:01] elukey: thanks for helping me figure that out, I’ll get the patch submitted later today [15:43:37] Nettrom: sorry for the trouble! [15:44:17] Nettrom: now that I think about it, mforns went through all the ReportUpdater code and fixed those references, you might already be using analytics-slave [15:45:40] Nettrom, are you using reportupdater jobs to generate data for your dashboard? [15:46:07] mforns: yes, checking the configuration now [15:46:16] ah, the configuration is up to date [15:46:19] thanks mforns! :) [15:46:34] should’ve checked that first, sorry [15:46:43] Nettrom, ok, no problem at all :] [15:46:47] (03PS1) 10Fdans: [wip] Map component and Pageviews by Country metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/392661 [15:47:51] 10Analytics, 10DBA, 10Operations, 10Patch-For-Review, 10User-Elukey: Decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3778241 (10Nettrom) Nevermind, turns out @mforns has already updated that configuration, should've checked that first. Thanks again for taking c... [15:48:12] updated the ticket just so there’s a paper trial of it :) [15:48:40] Nettrom: mforns rocks :D [15:48:57] O.o [15:49:06] as far as I can tell, you’re all rockstars! :) [15:49:29] (03CR) 10jerkins-bot: [V: 04-1] [wip] Map component and Pageviews by Country metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/392661 (owner: 10Fdans) [16:09:18] (03PS16) 10Mforns: [WIP] Add scala-spark core class and job to import data sets to Druid [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/386882 (https://phabricator.wikimedia.org/T166414) [16:13:36] 10Analytics-EventLogging, 10Analytics-Kanban: Purge refined JSON data after 90 days - https://phabricator.wikimedia.org/T181064#3778325 (10Ottomata) [16:16:34] (03PS17) 10Mforns: [WIP] Add scala-spark core class and job to import data sets to Druid [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/386882 (https://phabricator.wikimedia.org/T166414) [16:19:32] * elukey afk for a bit! [16:29:53] 10Analytics-EventLogging, 10Analytics-Kanban: Purge refined JSON data after 90 days - https://phabricator.wikimedia.org/T181064#3778325 (10Tbayer) From https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/Data_retention_and_auto-purging#Work_in_progress I understand that this will implement the... [16:31:07] 10Analytics-EventLogging, 10Analytics-Kanban: Purge refined JSON data after 90 days - https://phabricator.wikimedia.org/T181064#3778433 (10Tbayer) [16:34:44] hello! I am downloading around 3M Commons images on the Stat1005 machines, I will need around 50GB, I hope that is OK :) [17:46:18] 10Analytics-EventLogging, 10Analytics-Kanban: Purge refined JSON data after 90 days - https://phabricator.wikimedia.org/T181064#3778626 (10Ottomata) This task is just to purge 90 days. Implementing the intelligent whitelist based refining will be a different task. [17:46:31] 10Analytics-EventLogging, 10Analytics-Kanban: Purge refined JSON data after 90 days - https://phabricator.wikimedia.org/T181064#3778627 (10Ottomata) [17:50:24] ottomata, I'm having problems specifying a druid host, because the host that is supposed to accept ingestion tasks changes from druid1001 to druid1003 or to druid1002. Is there a static pointer to the correct one? [17:51:42] mforns: do you mean the overlord master? [17:51:54] elukey, yep lord of the druids [17:52:09] it should redirect you to the correct one if you hit the ones that it is not the master iirc [17:52:41] yes, the browser does redirect me, but not the library I'm using... [17:52:53] ok, so I should find a way to follow the 307 right? [17:53:02] makes sense [17:53:12] ah snap [17:53:29] yeah in theory this is the only way that I know :( [17:53:33] it's ok, will find something [17:56:55] mforns: yeah that is sorta what https://phabricator.wikimedia.org/T180971 is about [17:57:46] ottomata, I see, thanks. no big deal however. I hope will find some option in http scala lib to follow redirect [18:14:34] 10Analytics, 10Proton, 10Readers-Web-Backlog, 10MW-1.31-release-notes (WMF-deploy-2017-10-31 (1.31.0-wmf.6)), 10Patch-For-Review: Implement Schema:Print purging strategy - https://phabricator.wikimedia.org/T175395#3778694 (10ovasileva) [18:36:01] * elukey off [18:36:04] o/ [19:20:41] (03CR) 10Joal: "Tested on an small hourly job. Should be tested with a big daily-per-articles job, to make sure it doesn't derail the system." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/392624 (owner: 10Joal) [19:39:44] (03CR) 10Ottomata: "Couple of nitty Qs, but looking COOOOOL to me!" (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/386882 (https://phabricator.wikimedia.org/T166414) (owner: 10Mforns) [19:42:29] thx otto! [19:51:12] (03PS1) 10Joal: Grow RestbaseMetrics spark job to count MW API [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/392700 (https://phabricator.wikimedia.org/T176785) [20:00:46] (03CR) 10Joal: "Tested on cluster." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/392700 (https://phabricator.wikimedia.org/T176785) (owner: 10Joal) [20:11:30] (03PS1) 10Joal: Change restbase job to also count MW-API requests [analytics/refinery] - 10https://gerrit.wikimedia.org/r/392703 (https://phabricator.wikimedia.org/T176785) [20:24:33] (03PS2) 10Joal: Change restbase job to also count MW-API requests [analytics/refinery] - 10https://gerrit.wikimedia.org/r/392703 (https://phabricator.wikimedia.org/T176785) [20:25:03] (03CR) 10Joal: "Tested on cluster with new jar." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/392703 (https://phabricator.wikimedia.org/T176785) (owner: 10Joal) [20:27:01] Ok a-team, gone for tonight :) [20:53:41] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Please install R on stat1004 - https://phabricator.wikimedia.org/T181094#3779313 (10GoranSMilovanovic) [21:05:37] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Please install R on stat1004 - https://phabricator.wikimedia.org/T181094#3779342 (10GoranSMilovanovic) [21:09:09] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Please install R on stat1004 - https://phabricator.wikimedia.org/T181094#3779346 (10GoranSMilovanovic) BTW can we do access MariaDB replicas from **stat1004**? Because if we can't, then this ticket does not make sense, and the conclusion i... [21:12:03] Q: on of our old statboxes (1002, 1003) was able to do all of the following: R, beeline, sqoop, and mySQL. Does anyone remembers which one? [21:14:22] 10Analytics-EventLogging, 10Analytics-Kanban: Purge refined JSON data after 90 days - https://phabricator.wikimedia.org/T181064#3779359 (10Tbayer) I see, but that will be a problem in case of the Popups schema (and possibly others too which are no longer stored in MySQL), as the [[https://wikitech.wikimedia.or... [21:33:27] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Please install R on stat1004 - https://phabricator.wikimedia.org/T181094#3779471 (10GoranSMilovanovic) [21:40:09] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: A statbox to update the WDCM system - https://phabricator.wikimedia.org/T181094#3779502 (10GoranSMilovanovic) [21:44:14] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: A statbox to update the WDCM system - https://phabricator.wikimedia.org/T181094#3779313 (10GoranSMilovanovic) p:05Triage>03High [22:27:03] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: A statbox to update the WDCM system - https://phabricator.wikimedia.org/T181094#3779615 (10Ottomata) > Also, if we could only have R on stat1004 in the same way as if we have it on stat1005 I think this isn't going to be possible atm; stat... [22:29:16] 10Analytics-EventLogging, 10Analytics-Kanban: Purge refined JSON data after 90 days - https://phabricator.wikimedia.org/T181064#3779622 (10Ottomata) > Is there already a task for that BTW? Not yet! I just made this one today :) The default is delete after 90 days so we need at least that. Smarter purging is... [22:30:21] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: A statbox to update the WDCM system - https://phabricator.wikimedia.org/T181094#3779623 (10GoranSMilovanovic) @Ottomata > We can install R and stuff on stat1004, but I doubt you'll get the exact same versions. I don't care about the ver... [22:36:00] (03PS1) 10Ottomata: Add refinery-drop-hive-partitions [analytics/refinery] - 10https://gerrit.wikimedia.org/r/392733 (https://phabricator.wikimedia.org/T181064) [22:44:22] 10Analytics-EventLogging, 10Analytics-Kanban: Purge refined JSON data after 90 days - https://phabricator.wikimedia.org/T181064#3779716 (10Tbayer) >>! In T181064#3779622, @Ottomata wrote: ... > >> that will be a problem in case of the Popups schema (and possibly others too which are no longer stored in MySQL... [22:56:09] 10Analytics, 10Analytics-EventLogging: design a system to assist software developers and researchers to perform automated data unit testing before pushing to production - https://phabricator.wikimedia.org/T85032#3779820 (10ggellerman) [22:56:11] 10Analytics, 10Analytics-Cluster, 10Spike: SPIKE: experimenting with importing Revision history from XML dumps into an easier to use format, Avro - https://phabricator.wikimedia.org/T78405#3779825 (10ggellerman) [22:56:13] 10Analytics, 10Analytics-Wikimetrics, 10Epic: EPIC: Longterm fix of data warehouse replicating to Labs - https://phabricator.wikimedia.org/T78009#3779826 (10ggellerman) [22:56:15] 10Analytics, 10Analytics-Wikimetrics: create new group for devs to sudo run wikimetrics admin script {ops} {dove} - https://phabricator.wikimedia.org/T76792#3779831 (10ggellerman) [22:56:17] 10Analytics, 10Analytics-Wikimetrics: change puppetization for scheduler/queue {ops} {dove} - https://phabricator.wikimedia.org/T76790#3779830 (10ggellerman) [22:56:19] 10Analytics, 10Analytics-Wikimetrics: Debianize wikimetrics dependencies not in trusty {ops} {dove} - https://phabricator.wikimedia.org/T76789#3779833 (10ggellerman) [22:56:22] 10Analytics, 10Analytics-Wikimetrics: change wikimetrics in labs to use debian packages [13 pts] {dove} - https://phabricator.wikimedia.org/T76770#3779838 (10ggellerman) [22:56:24] 10Analytics, 10Analytics-Wikimetrics: Change wikimetrics admin script - no symlinking [8 pts] {dove} - https://phabricator.wikimedia.org/T76780#3779837 (10ggellerman) [22:56:26] 10Analytics, 10Analytics-Wikimetrics: upgrade wikimetrics to trusty [8 pts] {dove} - https://phabricator.wikimedia.org/T76769#3779839 (10ggellerman) [22:56:29] 10Analytics, 10Analytics-EventLogging: Talk to research about Mobile EL pipeline use case - https://phabricator.wikimedia.org/T75913#3779846 (10ggellerman) [22:56:30] 10Analytics, 10Analytics-Wikimetrics, 10Epic: EPIC: Productionizing Wikimetrics {dove} - https://phabricator.wikimedia.org/T76726#3779842 (10ggellerman) [22:56:34] 10Analytics, 10Analytics-Dashiki: weekly/monthly granularity for pageviews in Dashiki - https://phabricator.wikimedia.org/T76092#3779844 (10ggellerman) [22:56:36] 10Analytics, 10Analytics-EventLogging: Consider idea of 1 sprint for EL operational love sprint - https://phabricator.wikimedia.org/T75831#3779848 (10ggellerman) [22:56:38] 10Analytics, 10Analytics-EventLogging: Dashboard of status of our systems (one stop shopping) - https://phabricator.wikimedia.org/T75830#3779850 (10ggellerman) [22:56:40] 10Analytics, 10Analytics-EventLogging: Create monitoring checks to make sure events end up in DB - https://phabricator.wikimedia.org/T75828#3779852 (10ggellerman) [22:56:42] 10Analytics, 10Analytics-EventLogging: Create single place to look at graphs indicating outages - https://phabricator.wikimedia.org/T75827#3779855 (10ggellerman) [22:56:44] 10Analytics, 10Analytics-EventLogging: Establish support model for EL? (e.g. more ops involvement) - https://phabricator.wikimedia.org/T75824#3779856 (10ggellerman) [22:58:20] 10Analytics-EventLogging, 10Analytics-Kanban, 10Readers-Web-Backlog (Tracking): Schema:Popups suddenly stopped logging events in MariaDB, but they are still being sent according to Grafana - https://phabricator.wikimedia.org/T174815#3779860 (10Krinkle)