[00:03:42] 10Analytics, 10Discovery-Search (Current work): Deploy mjolnir msearch daemon to the elasticsearch clusters - https://phabricator.wikimedia.org/T200740 (10EBernhardson) [05:07:12] 10Analytics, 10Analytics-Kanban, 10wikimediafoundation.org: Measure traffic for new wikimedia foundation site - https://phabricator.wikimedia.org/T188419 (10Varnent) [06:15:36] 10Analytics-Kanban: Set a timeout for regex parsing in the Eventlogging processors - https://phabricator.wikimedia.org/T200760 (10elukey) p:05Triage>03High [06:50:38] Didn't we use to have a link to the public logs in this channel's topic? https://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-analytics/ [06:50:58] would be nice to put it back [06:54:16] Something must be broken from the logging side, we didn't disable it for sure [07:09:55] 10Analytics-Kanban: Set a timeout for regex parsing in the Eventlogging processors - https://phabricator.wikimedia.org/T200760 (10Volans) Using a contextmanager decorator and the built-in `TimeoutError` exception (if Py3 only) the code could be much simpler, something like: ```lang=python import signal from co... [08:53:32] 10Analytics-Kanban: Simplify and document how to increase log verbosity/level for Eventlogging - https://phabricator.wikimedia.org/T200765 (10elukey) p:05Triage>03High [08:56:36] elukey: wm-bot2 is still logging fine (see today's log at https://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-analytics/20180731.txt ). It's just about the link to that log in the channel topic... [08:57:19] ...someone with the necesary rights (channel op) can simply edit the topic to re-add the link [09:08:24] HaeB: ahhh sorry I didn't check it, it was in my list today (I was working on eventlogging) [09:08:43] I don't think that I can become op in here, but I can try [09:10:47] Will ak andrew to do it [09:17:04] cool thanks [09:28:28] 10Analytics: Upgrade librdkafka on eventlog1002 - https://phabricator.wikimedia.org/T200769 (10elukey) p:05Triage>03Normal [11:55:18] * elukey afk for a couple of hours! [12:42:26] 10Quarry: Add a possibility to delete a draft - https://phabricator.wikimedia.org/T135908 (10Dvorapa) >>! In T135908#4461580, @Halfak wrote: > Right. I think it makes a lot of sense to *archive* old queries so that they don't clutter up useful, but not-worth-publishing queries. Given that queries don't take up... [12:53:51] o/ o/ o/ [12:54:20] holy moly its like jetlag but for email [12:54:55] who am i? [12:59:16] Hey Hey [12:59:50] PLEASE, anyone who can take care of this: I have some simple and clean data sets waiting for review before they go public from stat1005: https://phabricator.wikimedia.org/T199851 [13:00:35] README.txt provided. It really should not take too much time to review them, and they need to go public in order to support a Wiktionary dashboard that we have developed in WMDE. Thanks a lot! [13:04:16] GoranSM: i'll bring it up in our standup today if no one looks at it before [13:04:58] ottomata: Thanks a lot! It's really, really simple, no private data, all aggregate statistics on Wiktionary entries. Thanks! [13:05:40] GoranSM: cool oh i see it doesn't have the 'Analytics' tag on it [13:05:54] we groom tasks that are tagged with Analytics, so that might be why it hasn't had any motion [13:06:18] 10Analytics, 10Analytics-Data-Quality, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Data set review for the Wiktionary Cognate Dashboard - https://phabricator.wikimedia.org/T199851 (10Ottomata) [13:47:02] ottomata: hellooooooo [13:47:09] how are things?? [13:50:05] HELLO! [13:50:07] things are great! [13:50:15] wow typing what a thing! [13:50:16] computers! [13:50:18] desks [13:50:18] wow [13:50:21] ahahahha [13:50:22] emails [13:50:29] how was the trip?? [13:50:35] i don' think i've ever gone 3 weeks without typing on a computer before [13:50:37] since...when? [13:50:40] never? [13:50:47] no i must have [13:50:52] maybe 10 years ago [13:50:54] (03CR) 10Sahil505: [C: 04-1] "Something interesting : Try consoling barBB.width inside on mouseover function after line 164. The width of each bar column is not constan" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/449029 (https://phabricator.wikimedia.org/T192416) (owner: 10Sahil505) [13:51:00] the trip was amazing [13:51:07] like, amazing amazing [13:51:39] are you able to fluently read newspapers now? [13:51:49] one of the most beautiful and special things i've ever done [13:51:51] newspapers... [13:51:56] newspapers [13:51:58] ? [13:52:11] Ukraine ones I mean :D [13:52:26] oh [13:52:43] hm, no but i can mostly pronounce cyrillic now! [13:52:50] wow [13:52:52] well, kind of like a 4 year old reads [13:56:27] kudos anyway :D [14:01:09] ottomata: if you want to have a chat about things exploding while you were away (Kafka missed you and hated me) we can chat whenever you want [14:01:13] (03CR) 10Milimetric: Add saltrotate, a script that manages cryptographic salts (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/449249 (https://phabricator.wikimedia.org/T199899) (owner: 10Mforns) [14:01:36] ahha ok elukey lemme get through emails first [14:01:44] saw a some of that stuff will check up and then let's chat [14:01:55] might not be til later today or tomorrow though! (there are a lot of emails!) [14:01:59] yep yep [14:25:37] elukey: very nice incident report btw [14:25:39] wow [14:27:14] 10Analytics, 10Beta-Cluster-Infrastructure, 10Services (watching): deployment-tin:/srv/mediawiki-staging/wmf-config/event-schemas is out of sync - https://phabricator.wikimedia.org/T199270 (10Pchelolo) In production, the event-schemas are deployed automatically as soon as a patch is merged in gerrit in this... [14:27:55] ottomata: thanks! [14:28:02] let me know if anything looks weird or missing [14:28:08] I tried to add as much info as possible [14:28:17] yeah the eventstreams thing is tricky [14:28:21] you can't really fail it over... [14:28:36] unless we default to using timestamps in the Last-Event-Id, but then that would be imprecise [14:32:08] (03CR) 10Ottomata: [C: 031] "Deferring to other's reviews, but +1 from me for the idea." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/445395 (https://phabricator.wikimedia.org/T198600) (owner: 10Fdans) [14:33:08] ottomata: thanks for the review andrew! [14:38:50] 10Analytics, 10Beta-Cluster-Infrastructure, 10Services (watching): deployment-tin:/srv/mediawiki-staging/wmf-config/event-schemas is out of sync - https://phabricator.wikimedia.org/T199270 (10Krenair) I'm assuming whoever set up the submodule in mediawiki-config forgot to add a `git submodule update` or some... [14:49:02] 10Analytics, 10Beta-Cluster-Infrastructure, 10Services (watching): deployment-tin:/srv/mediawiki-staging/wmf-config/event-schemas is out of sync - https://phabricator.wikimedia.org/T199270 (10Ottomata) I'm not sure I knew this repo was even in wmf-config. Not sure why it is there...I'm pretty sure the Event... [15:01:56] 10Analytics, 10Beta-Cluster-Infrastructure, 10Services (watching): deployment-tin:/srv/mediawiki-staging/wmf-config/event-schemas is out of sync - https://phabricator.wikimedia.org/T199270 (10dcausse) If this submodule is not referenced by anything else other than mediawiki then there should be no reason to... [15:11:47] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Drop mediawiki history old snapshots from druid public cluster - https://phabricator.wikimedia.org/T197889 (10Milimetric) p:05Triage>03Normal [15:32:10] (03CR) 10Ottomata: [C: 031] "didn't review the hashing details, but +1 for idea and code" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/446592 (https://phabricator.wikimedia.org/T198426) (owner: 10Mforns) [15:37:35] 10Analytics, 10Discovery-Search (Current work): Deploy mjolnir msearch daemon to the elasticsearch clusters - https://phabricator.wikimedia.org/T200740 (10EBernhardson) [16:07:08] hey, forgot to join before meetings... [16:59:43] 10Analytics: Remove webrequest misc analytics related jobs and code after cache misc -> text merge is complete - https://phabricator.wikimedia.org/T200822 (10Ottomata) p:05Triage>03Normal [17:01:35] ottomata: o/ - going offline now, the network maintenance seemed fine [17:01:46] if you see anything weird ping Arzhel [17:01:57] but afaics there is only one issue with the puppetmaster, unrelated to our hosts [17:02:02] (so some failures are expected) [17:06:11] ok great, latterrrs [17:12:45] 10Analytics, 10Operations, 10Discovery-Search (Current work), 10Patch-For-Review, 10Services (watching): Create kafka topic for mjolinr bulk daemon and decide on cluster - https://phabricator.wikimedia.org/T200215 (10EBernhardson) If my reading of puppet is right, what we need will be: on kafkamon1001.e... [17:15:15] 10Analytics, 10MinervaNeue, 10Readers-Web-Backlog, 10Design, 10Readers-Web-Kanbanana-Board: [Spike 8hrs] Sticky header instrumentation - https://phabricator.wikimedia.org/T199157 (10ovasileva) Discussed in standup and noted that the next way forward might be to create a draft of the schema in parallel to... [17:17:09] 10Analytics, 10MinervaNeue, 10Readers-Web-Backlog, 10Design, 10Readers-Web-Kanbanana-Board: [Spike 8hrs] Sticky header instrumentation - https://phabricator.wikimedia.org/T199157 (10phuedx) ^ If creating a //draft// schema helps the investigator to think about the implementation, then they should feel fr... [17:40:11] 10Analytics, 10Operations, 10Discovery-Search (Current work), 10Patch-For-Review, 10Services (watching): Create kafka topic for mjolinr bulk daemon and decide on cluster - https://phabricator.wikimedia.org/T200215 (10Ottomata) Do you need the data mirrored between the two different Kafka clusters? If so... [17:44:03] 10Analytics, 10Operations, 10Discovery-Search (Current work), 10Patch-For-Review, 10Services (watching): Create kafka topic for mjolinr bulk daemon and decide on cluster - https://phabricator.wikimedia.org/T200215 (10EBernhardson) Yes it will need to be mirrored, to be consumed in each datacenter separat... [17:44:29] 10Analytics, 10Operations, 10Discovery-Search (Current work), 10Patch-For-Review, 10Services (watching): Create kafka topic for mjolinr bulk daemon and decide on cluster - https://phabricator.wikimedia.org/T200215 (10EBernhardson) [17:53:02] (03CR) 10Ottomata: "Cool!" (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/449249 (https://phabricator.wikimedia.org/T199899) (owner: 10Mforns) [17:53:34] (03CR) 10Ottomata: "See also: http://snakebite.readthedocs.io/en/latest/client.html#snakebite.client.Client.stat" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/449249 (https://phabricator.wikimedia.org/T199899) (owner: 10Mforns) [18:18:12] 10Analytics, 10Analytics-Cluster: Upgrade spark 2.3.0 -> 2.3.1 on analytics cluster - https://phabricator.wikimedia.org/T200732 (10Ottomata) Upgrading the version here will likely be pretty easy... we'll triage this task soon. [18:21:22] (03CR) 10Mforns: "> BTW, I wonder if we should start using snakebite instead of" (033 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/449249 (https://phabricator.wikimedia.org/T199899) (owner: 10Mforns) [18:42:13] 10Analytics, 10EventBus, 10Wikimedia-Stream, 10Services (watching), 10Wikimedia-Incident: Redesign EventStreams for better multi-dc support - https://phabricator.wikimedia.org/T199433 (10Ottomata) > Perhaps we could deprecate by-offset reconnection and only support by-timestamp reconnection and lower the... [18:46:07] 10Analytics, 10EventBus, 10Wikimedia-Stream, 10Services (watching), 10Wikimedia-Incident: Redesign EventStreams for better multi-dc support - https://phabricator.wikimedia.org/T199433 (10Pchelolo) What's exactly the benefit of providing the exact offset? The timestamp has millisecond resolution and in pr... [18:46:45] 10Analytics, 10ChangeProp, 10Operations, 10Services (designing), 10Wikimedia-Incident: Separate dev Change-Prop from production Kafka cluster - https://phabricator.wikimedia.org/T199427 (10Ottomata) Should be doable! [18:49:50] 10Analytics, 10EventBus, 10Services (watching): Remove `kafka-mirror` unit from main kafka cluster - https://phabricator.wikimedia.org/T199443 (10Ottomata) I created this when we were running multiple MirrorMaker instance on the same box. This service allows you to stop and start all of them at once. As we... [18:52:50] 10Analytics, 10EventBus, 10Services (watching): Remove `kafka-mirror` unit from main kafka cluster - https://phabricator.wikimedia.org/T199443 (10Pchelolo) Ye, I just didn't really know what exactly that is and in the heat of the outage I was a bit confused. [18:53:30] (03PS3) 10Sahil505: Made Bar-chart popup dynamic [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/449029 (https://phabricator.wikimedia.org/T192416) [18:56:54] (03CR) 10Sahil505: "Working perfectly for non breakdown charts. Will continue for breakdown charts tomorrow." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/449029 (https://phabricator.wikimedia.org/T192416) (owner: 10Sahil505) [18:57:10] (03CR) 10Sahil505: [C: 04-1] "WIP" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/449029 (https://phabricator.wikimedia.org/T192416) (owner: 10Sahil505) [19:08:56] (03CR) 10Ottomata: "Ya another patch for snakebite sounds fine." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/449249 (https://phabricator.wikimedia.org/T199899) (owner: 10Mforns) [19:21:50] WOW I DID IT EmAIL! [19:21:51] wow [20:03:22] 10Analytics, 10EventBus, 10Wikimedia-Stream, 10Services (watching), 10Wikimedia-Incident: Redesign EventStreams for better multi-dc support - https://phabricator.wikimedia.org/T199433 (10Ottomata) H, you might be right. Especially for EventStreams, it is unlikely that consumers will care about resuming... [20:10:08] 10Analytics, 10EventBus, 10Wikimedia-Stream, 10Services (watching), 10Wikimedia-Incident: Redesign EventStreams for better multi-dc support - https://phabricator.wikimedia.org/T199433 (10Pchelolo) > If a message arrives late, there are scenarios where a consumer will get a lot of extra messages Arrives... [20:13:01] 10Analytics, 10EventBus, 10Wikimedia-Stream, 10Services (watching), 10Wikimedia-Incident: Redesign EventStreams for better multi-dc support - https://phabricator.wikimedia.org/T199433 (10Ottomata) The message timestamp should be set by the producer (is EventBus doing this? it should), and it should corr... [20:22:19] 10Analytics, 10EventBus, 10Wikimedia-Stream, 10Services (watching), 10Wikimedia-Incident: Redesign EventStreams for better multi-dc support - https://phabricator.wikimedia.org/T199433 (10Pchelolo) > The message timestamp should be set by the producer (is EventBus doing this? it should), and it should cor... [20:25:49] 10Analytics, 10EventBus, 10Wikimedia-Stream, 10Services (watching), 10Wikimedia-Incident: Redesign EventStreams for better multi-dc support - https://phabricator.wikimedia.org/T199433 (10Ottomata) I'm going to add a config option to KafkaSSE: `useTimestampForId`. This will instruct KafkaSSE to set eithe...