[07:21:03] <elukey>	 good morning
[07:29:33] <lexnasser>	 elukey: happy monday!
[07:29:52] <lexnasser>	 just a heads-up to the team - I won't be at standup today
[07:31:31] <elukey>	 lexnasser: good night! :D
[07:31:39] <elukey>	 ack noted
[09:32:32] <wikibugs>	 (03CR) 10Elukey: "recheck" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/661968 (owner: 10Elukey)
[09:33:57] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Release 2.8.5 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/661968 (owner: 10Elukey)
[09:55:03] <wikibugs>	 (03CR) 10Elukey: "This is due to a big change in package-lock.json, bringing in git+ssh dependencies.. no idea why this happens, I followed the guide on wik" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/661968 (owner: 10Elukey)
[09:55:15] <wikibugs>	 (03Abandoned) 10Elukey: Release 2.8.5 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/661968 (owner: 10Elukey)
[11:40:45] <zpapierski>	 o/ 
[11:41:04] <zpapierski>	 a question - I want to add jars to my pyspark in jupyter
[11:41:26] <zpapierski>	 I'd normally do that with launching pyspark with --jars - how (if possible) can I do this with jupyter?
[11:45:00] <wikibugs>	 10Analytics: Automate deploy Wikistats 2 to Production - https://phabricator.wikimedia.org/T274126 (10elukey)
[11:45:22] <wikibugs>	 10Analytics: Automate the deployment procedure of Wikistats 2 to Production - https://phabricator.wikimedia.org/T274126 (10elukey)
[11:47:41] <elukey>	 zpapierski: o/ never tried, but I guess that something would need to change the pyspark submit args before creating the yarn app
[11:49:49] <zpapierski>	 from what I look online, apparently there is a way from within pyspark already running, I'm gonna try that
[11:51:16] <elukey>	 ack let us know how you do it so we can document
[11:51:25] <zpapierski>	 sure
[11:51:29] <elukey>	 <#
[11:51:30] <elukey>	 <3
[11:58:55] <wikibugs>	 10Quarry: Quarry queries forever stuck in queue - https://phabricator.wikimedia.org/T274071 (10Framawiki) From the logs regarding the last stuck query (https://quarry.wmflabs.org/query/52267):  ` Feb 08 07:00:50 quarry-worker-01 celery-quarry-worker[17475]: [2021-02-08 07:00:50,687: ERROR/ForkPoolWorker-143] Tas...
[12:47:14] <elukey>	  /me lunch!
[13:04:09] <zpapierski>	 ok, so apparently there's an option to add jars in runtime in jupyter
[13:04:17] <zpapierski>	 but it has to be done before first sparksubmit
[13:04:35] <zpapierski>	 I tried stopping and restarting spark context, but parameters aren't passed on
[13:04:47] <zpapierski>	 I need to think about it a bit more
[13:06:52] <zpapierski>	 elukey: is it possible to have a new kernel, with proper jars needed for my task? I need to be able to test out writing to elasticsearch
[13:07:11] <zpapierski>	 in the end, it's not how I will launch it, but it would greatly help my dev time
[13:28:57] <fdans>	 good morning y'all
[13:29:14] <fdans>	 elukey: thank you for creating the wikistats automation ticket, I wanted to look at that as well
[13:46:12] <zpapierski>	 ok, irccloud threw me out  - elukey, if you answered my question, I'll need to hear the answer once again :)
[14:05:33] <wikibugs>	 10Analytics, 10Patch-For-Review: Presto should warn or prevent users from querying without Hive partition predicates - https://phabricator.wikimedia.org/T273004 (10Ottomata) @elukey we tried to applied this to the test cluster and did some restarts there but had some trouble getting presto queries to run, also...
[14:17:17] <mforns>	 hello teammm
[14:23:06] <ottomata>	 o/
[14:25:31] <wikibugs>	 (03CR) 10Awight: Added editor type preferences (032 comments) [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/657362 (https://phabricator.wikimedia.org/T271902) (owner: 10Svantje Lilienthal)
[14:29:31] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 4 others: KaiOS / Inuka Event Platform client - https://phabricator.wikimedia.org/T273219 (10Ottomata) @SBisson any tips for getting this work scheduled? :)
[14:35:37] <cdanis>	 ottomata: re: access requests in general, hopefully Soon™ we find time to do some work on the process/onboarding overall
[14:36:19] <ottomata>	 yeah soon™ indeed
[14:41:11] <wikibugs>	 (03CR) 10Mforns: Added editor type preferences (033 comments) [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/657362 (https://phabricator.wikimedia.org/T271902) (owner: 10Svantje Lilienthal)
[14:44:27] <wikibugs>	 (03CR) 10Awight: Added editor type preferences (031 comment) [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/657362 (https://phabricator.wikimedia.org/T271902) (owner: 10Svantje Lilienthal)
[14:51:26] <wikibugs>	 (03CR) 10Awight: Added editor type preferences (031 comment) [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/657362 (https://phabricator.wikimedia.org/T271902) (owner: 10Svantje Lilienthal)
[14:53:50] <wikibugs>	 (03PS9) 10Awight: Added editor type preferences [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/657362 (https://phabricator.wikimedia.org/T271902) (owner: 10Svantje Lilienthal)
[14:53:53] <wikibugs>	 10Analytics, 10Patch-For-Review: Presto should warn or prevent users from querying without Hive partition predicates - https://phabricator.wikimedia.org/T273004 (10elukey) I am testing 0.246 in there, sadly it is broken see https://phabricator.wikimedia.org/T266640
[14:59:37] <wikibugs>	 (03CR) 10Awight: "PS 9: add some inline documentation" [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/657362 (https://phabricator.wikimedia.org/T271902) (owner: 10Svantje Lilienthal)
[15:03:32] <wikibugs>	 (03CR) 10Mforns: Added editor type preferences (031 comment) [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/657362 (https://phabricator.wikimedia.org/T271902) (owner: 10Svantje Lilienthal)
[15:06:25] <wikibugs>	 (03CR) 10Awight: Added editor type preferences (033 comments) [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/657362 (https://phabricator.wikimedia.org/T271902) (owner: 10Svantje Lilienthal)
[15:11:33] <wikibugs>	 10Analytics, 10Patch-For-Review: Upgrade the Analytics Hadoop cluster to Apache Bigtop - https://phabricator.wikimedia.org/T273711 (10elukey) There is one last thing that we haven't discussed, I forgot to follow up at the time. The avro-libs and parquet packages (that were shipped by CDH) are not present in Bi...
[15:11:53] <ottomata>	 !log set kafka topic retention to 31 days for (eqiad|codfw.rdf-streaming-updater.mutation) in kafka main-eqiad and main-codfw - T269619
[15:12:05] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[15:12:05] <stashbot>	 T269619: Create pipelines for late/spurious/failed events - https://phabricator.wikimedia.org/T269619
[15:13:34] <fkaelin>	 zpapierski did you resolve the jar issue? the existing pyspark kernels create the spark sessions at startup of the kernel through the kernel.json (eg /usr/local/share/jupyter/kernels/spark_yarn_pyspark/kernel.json). While it seems stopping that session (spark.stop()) and restarting a custom one in your notebook should work, I would recommend starting with a plain python 3 kernel and create the initial session manually. 
[15:14:01] <zpapierski>	 no I didn't, I think I'll follow your advice
[15:14:11] <fkaelin>	 You can also create a custom kernel config and add the jar config to the kernel.json, but in view of the move to newpyter that doesn't seem like the way to go. In fact, the newpyter based notebooks are probably already the best option if you want to customize your environment ( https://wikitech.wikimedia.org/wiki/Analytics/Systems/Jupyter#Newpyter), but note that the existing pyspark kernels don't work well with newpyter.
[15:14:24] <ottomata>	 +1 
[15:14:28] <zpapierski>	 I mean - for now I just went with a plan old cli pyspark2, I can control it's parameters
[15:14:47] <ottomata>	 btw, draft updated jupyter docs are here
[15:14:48] * ottomata https://wikitech.wikimedia.org/wiki/User:Ottomata/Jupyter#PySpark_and_wmfdata
[15:14:54] <zpapierski>	 hmm, haven't tried this newpyter
[15:14:58] <ottomata>	 i'm just waiting for a release of wmfdata before i can make them official
[15:15:19] <ottomata>	 zpapierski:  we are going to get rid of the old venv based notebook soon(ish) this quarter hhopefully
[15:15:42] <ottomata>	 i'd also like to remove the custom python kernels
[15:15:52] <ottomata>	 they aren't really that useful; it isn't that hard to instantiate your own sparksession
[15:15:57] <ottomata>	 in a regular python notebook
[15:17:00] <zpapierski>	 yeah, it makes sense to expose spark configuration options
[15:18:14] <wikibugs>	 10Analytics, 10Patch-For-Review: Upgrade the Analytics Hadoop cluster to Apache Bigtop - https://phabricator.wikimedia.org/T273711 (10Ottomata) I don't have any knowledge of those specific packages being needed.
[15:18:44] <ottomata>	 elukey:  o/ helllOoooO
[15:18:55] <ottomata>	 anything i can do to help prepare for tomomrrow?
[15:19:03] <wikibugs>	 (03CR) 10Mforns: refinery-drop-hive-partitions: Ensure verbose logging goes somewhere (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/661799 (owner: 10Ebernhardson)
[15:19:06] <wikibugs>	 10Analytics: Add analytics-presto.eqiad.wmnet CNAME for Presto coordinator failover - https://phabricator.wikimedia.org/T273642 (10elukey) Remember https://phabricator.wikimedia.org/T158757 when doing it, the new cert needs to have a SAN :)
[15:20:25] <elukey>	 ottomata: o/ heellooooo - the answer to the task was a big help, I am finishing https://etherpad.wikimedia.org/p/analytics-bigtop-upgrade with commands etc.. but so far nothing else standing
[15:20:36] <ottomata>	 awesome
[15:22:44] <elukey>	 the fact that etherpad doesn't work is not really great, I wanted to copy the plan to phab for this reason :D
[15:25:16] <elukey>	 ottomata: does etherpad works for you?
[15:27:32] <wikibugs>	 10Analytics, 10Analytics-Kanban: Generalize the current Airflow puppet/scap code to deploy a dedicated Analytics instance - https://phabricator.wikimedia.org/T272973 (10Ottomata) Add 1 thing to the 'Downsides' of multi-instance approach:  - when doing maintenance, the Data Engineering team will have to rangle...
[15:28:34] <ottomata>	 elukey:  etherpad works for me
[15:28:44] <ottomata>	 elukey:  its nice to be able to live edit more easily; you could use google docs
[15:28:51] <elukey>	 ah yes now it works
[15:29:11] <elukey>	 I'll save a copy in the phab task JUST IN CASE
[15:34:44] <ottomata>	 mforns:  hello
[15:34:51] <mforns>	 ottomata: hi!
[15:34:53] <ottomata>	 quick sync about EL?
[15:34:57] <mforns>	 k, bc?
[15:35:00] <ottomata>	 ya
[15:35:02] <mforns>	 omw
[15:40:31] <mforns>	 ottomata: I think it's fixed now, bc?
[15:41:25] <ottomata>	 ya
[15:44:46] <wikibugs>	 (03CR) 10Awight: [C: 03+2] "Would be great if this were merged, it blocks other work." [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/656901 (https://phabricator.wikimedia.org/T269986) (owner: 10WMDE-Fisch)
[15:50:13] <wikibugs>	 (03CR) 10Ebernhardson: refinery-drop-hive-partitions: Ensure verbose logging goes somewhere (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/661799 (owner: 10Ebernhardson)
[15:54:26] <wikibugs>	 (03CR) 10Ottomata: Update schema with core bucket labels (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/656901 (https://phabricator.wikimedia.org/T269986) (owner: 10WMDE-Fisch)
[16:24:49] <wikibugs>	 10Analytics-Clusters: Balance Kafka topic partitions on Kafka Jumbo to take advantage of the new brokers - https://phabricator.wikimedia.org/T255973 (10razzi)
[16:38:35] <wikibugs>	 10Analytics-Radar, 10Better Use Of Data, 10Product-Analytics, 10Product-Data-Infrastructure, and 4 others: Develop a new schema for MediaSearch analytics or adapt an existing one - https://phabricator.wikimedia.org/T263875 (10jlinehan)
[16:38:45] <wikibugs>	 10Analytics-Radar, 10Better Use Of Data, 10Product-Analytics, 10Product-Data-Infrastructure: prefUpdate schema contains multiple identical events for the same preference update - https://phabricator.wikimedia.org/T218835 (10jlinehan)
[16:39:12] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, and 5 others: Remove http.client_ip from EventGate default schema (again) - https://phabricator.wikimedia.org/T262626 (10jlinehan)
[16:39:22] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Better Use Of Data, and 5 others: KaiOS / Inuka Event Platform client - https://phabricator.wikimedia.org/T273219 (10jlinehan)
[16:39:52] <wikibugs>	 10Analytics, 10Analytics-Wikistats: Wikistats Bug: Bulgarian Language... - https://phabricator.wikimedia.org/T273677 (10fdans) 05Open→03Invalid Hi, Bulgarian is only translated 19% of the way. If you want Bulgarian to be available as a selectable language, you can contribute to the translation work in Tran...
[16:39:57] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Better Use Of Data, 10Product-Analytics, and 2 others: Document how ad blockers / tracking blockers interact with EventLogging - https://phabricator.wikimedia.org/T263503 (10jlinehan)
[16:40:09] <wikibugs>	 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Data-Infrastructure, 10Platform Team Initiatives (Modern Event Platform (TEC2)): Allow disabling/enabling configured streams via wgEventStreams config - https://phabricator.wikimedia.org/T259712 (10jlinehan)
[16:40:25] <wikibugs>	 10Analytics-Clusters, 10Analytics-Kanban: Re-create deployment-aqs cluster - https://phabricator.wikimedia.org/T272722 (10Ottomata)
[16:40:37] <wikibugs>	 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Data-Infrastructure: Automate EventGate validation error reporting - https://phabricator.wikimedia.org/T268027 (10jlinehan)
[16:40:45] <wikibugs>	 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Data-Infrastructure: Client-side error logging should use Elastic Common Schema (ECS) fields when possible - https://phabricator.wikimedia.org/T267602 (10jlinehan)
[16:40:57] <wikibugs>	 10Analytics-EventLogging, 10Analytics-Radar, 10Better Use Of Data, 10Event-Platform, and 3 others: OperationError: The operation failed for an operation-specific reason  in generateRandomSessionId - https://phabricator.wikimedia.org/T263041 (10jlinehan)
[16:41:02] <wikibugs>	 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Data-Infrastructure, 10Performance-Team (Radar): Research and consider network connections made due to Event Platform - https://phabricator.wikimedia.org/T263049 (10jlinehan)
[16:41:10] <wikibugs>	 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Data-Infrastructure, 10Patch-For-Review: EventGate idea: use presence of schema properties in http.(request|response)_headers to automatically set header values in event data - https://phabricator.wikimedia.org/T263466 (10jlinehan)
[16:41:57] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Better Use Of Data, and 5 others: Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163 (10jlinehan)
[16:44:00] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Growth-Team, 10Product-Analytics: Revisions missing from mediawiki_revision_create - https://phabricator.wikimedia.org/T215001 (10Zbyszko) Since we're close to deploying WDQS streaming updater that uses, among others, revision-create - are there any updates on this?
[16:45:20] <wikibugs>	 10Analytics: Turnilo "Display Druid query" gives "general error" - https://phabricator.wikimedia.org/T273685 (10Milimetric) Weird, it works on https://w.wiki/yPx for example.  It seems that something's different about the netflow dataset, we'll try and brainstorm what that might be.
[16:46:48] <wikibugs>	 10Analytics: Groom Incoming for Security Tasks - https://phabricator.wikimedia.org/T273329 (10fdans) 05Open→03Resolved a:03fdans
[16:48:10] <wikibugs>	 10Analytics: Druid datasource drop triggers segment reshuffling by the coordinator - https://phabricator.wikimedia.org/T270173 (10Ottomata) p:05High→03Medium
[16:48:31] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: Uncaught TypeError: navigator.sendBeacon is not a function - https://phabricator.wikimedia.org/T273374 (10fdans) p:05Triage→03High
[16:49:23] <wikibugs>	 10Analytics, 10Analytics-Kanban: Check data currently stored on thorium and drop what it is not needed anymore - https://phabricator.wikimedia.org/T265971 (10Ottomata)
[16:54:06] <wikibugs>	 10Analytics, 10Event-Platform: Sanitize and ingest event tables defined in the event_sanitized database - https://phabricator.wikimedia.org/T273789 (10mforns) @Ottomata For new streams, shouldn't the stream owners work on this?
[16:54:43] <wikibugs>	 10Analytics: Upgrade to Superset 1.0 - https://phabricator.wikimedia.org/T272390 (10Ottomata) a:03razzi
[16:55:11] <wikibugs>	 10Analytics: Upgrade to Superset 1.0 - https://phabricator.wikimedia.org/T272390 (10Ottomata) We should talk as a team on how to manage and own Superset upgrades going forward. Razzi will bring this up in a meeting.
[16:55:52] <wikibugs>	 10Analytics, 10Growth-Scaling, 10Growth-Team, 10Product-Analytics: Growth: delete data older than 90 days - https://phabricator.wikimedia.org/T273821 (10fdans) Hi @mpopov is there anything needed from us for this task and the parent?
[16:57:54] <wikibugs>	 10Analytics-Radar, 10Product-Analytics, 10Growth-Team (Current Sprint): remove all Growth schemas from the schema whitelist - https://phabricator.wikimedia.org/T273826 (10fdans)
[17:05:37] <elukey>	 fdans, milimetric - forgot to ask, can we help with https://phabricator.wikimedia.org/T273741 ?
[17:06:19] <milimetric>	 elukey: wanna pop into the batcave for a sec actually, we need your thoughts/help on something else
[17:06:20] <elukey>	 I can check webrequest, but I am wondering if media_request or similar can help with 90+ data
[17:06:29] <elukey>	 milimetric: sre meeting :(
[17:06:33] <milimetric>	 oh right
[17:06:34] <milimetric>	 sorry
[17:06:36] <milimetric>	 later
[17:07:16] <milimetric>	 (we'll need to talk about a process to make HDFS users for people who aren't able to access data via superset
[17:07:17] <milimetric>	 )
[17:07:54] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Growth-Team, 10Product-Analytics: Revisions missing from mediawiki_revision_create - https://phabricator.wikimedia.org/T215001 (10Ottomata) We might be able to mitigate this if we find the reason it is happening now, but I don't think we can rely on 100% revision-create c...
[17:10:56] <wikibugs>	 (03CR) 10Awight: [C: 03+1] "Ideally, this is merged after 1.36.0-wmf.29 is deployed." [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/659227 (https://phabricator.wikimedia.org/T272569) (owner: 10Andrew-WMDE)
[17:11:15] <ottomata>	 milimetric:  we have that process, no?  https://wikitech.wikimedia.org/wiki/Analytics/Data_access#What_access_should_I_request?
[17:12:02] <milimetric>	 ottomata: it seems like some folks got left without access after the HDFS permissions changes, I guess when they have an LDAP account but no matching Hadoop account?
[17:12:51] <elukey>	 milimetric: they are probably not in analytics-private-data
[17:13:04] <elukey>	 so they cannot check presto dashboards, only druid ones
[17:13:08] <milimetric>	 as I understood Kate is having trouble, we're asking who else
[17:13:26] <elukey>	 milimetric: yeah kate was not in privatedata, Marcel took care of it IIRC
[17:13:56] <wikibugs>	 10Analytics-Radar, 10Product-Analytics: Provide a list of 100 most popular articles of Russian and English Wikipedias in terms of page views from Ukraine - https://phabricator.wikimedia.org/T273924 (10fdans)
[17:14:13] <milimetric>	 (he made a task, but apparently it's not done yet)
[17:14:16] <elukey>	 milimetric: there was also a problem with Erin, tracked in another task, but it was related to old superset accounts (already fixed)
[17:14:32] <ottomata>	 milimetric:  https://phabricator.wikimedia.org/T272982
[17:14:50] <milimetric>	 gotcha, so Kate's left for now and we'll see who else, but you're all saying it's not a problem with the process, other folks should be fine going forward?  If so, great
[17:15:37] <milimetric>	 ottomata: ok, so that's all approved, who does the work?
[17:16:25] <elukey>	 milimetric: the SRE doing clinic duty should pick it up asap this week I think
[17:16:43] <wikibugs>	 10Analytics, 10Event-Platform: Sanitize and ingest event tables defined in the event_sanitized database - https://phabricator.wikimedia.org/T273789 (10Ottomata) @mforns ?  Is that like asking Eric Gardner who worked on {T258183} to create a sanitization refine job?
[17:16:49] <elukey>	 it is Valentin, I can ping him, otherwise we can just send the code review
[17:18:18] <wikibugs>	 10Analytics: Build a process to check permissions when changing datasets from non-PII to PII - https://phabricator.wikimedia.org/T273818 (10fdans) p:05Triage→03Medium
[17:34:18] <wikibugs>	 (03CR) 10Awight: [C: 03+1] "This can be deployed any time.  We have some migration to do once it's deployed (see T273450), the old metrics should be purged, start_dat" [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/661108 (https://phabricator.wikimedia.org/T273454) (owner: 10Awight)
[17:51:20] <klausman>	 ottomata: if you have questions about Prom queries, you can always poke me. I've used it failry extensively in past jobs (and for private stuff)
[18:00:51] <ottomata>	 oh ok cool!
[18:00:52] <ottomata>	 thank you
[18:11:50] <klausman>	 Alos, Prometheus metrics and the query language are modeled on (aka fixing the mistakes of) Borgmon, the old internal monitoring system at the Goo.
[18:13:16] <wikibugs>	 10Analytics, 10Patch-For-Review: Upgrade the Analytics Hadoop cluster to Apache Bigtop - https://phabricator.wikimedia.org/T273711 (10elukey)
[18:15:05] <elukey>	 mforns: didn't know about the mediaviwer in toolforge! Thanks!
[18:18:24] <wikibugs>	 10Analytics, 10Patch-For-Review: Upgrade the Analytics Hadoop cluster to Apache Bigtop - https://phabricator.wikimedia.org/T273711 (10elukey)
[18:31:16] <elukey>	 ottomata: forgot to ask - did you see https://phabricator.wikimedia.org/T269832#6805972 ?
[18:35:54] <ottomata>	 oh yes elukey !  had seen it! :) kinda liike the druid one.  do you think we should make it easier to access like yarn?
[18:38:05] <elukey>	 ottomata: it is a little tricky, since we'd need to allow http traffic basically (so bypassing kerberos and https) - I thought to use it with a ssh tunnel for the moment, but we can expand it in the future in theory
[18:44:21] <razzi>	 !log rebalance kafka partitions for eventlogging_VirtualPageView
[18:44:23] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[18:53:46] <ottomata>	 elukey:  sounds good.  does it expose PII?  i guess some queries could be PII?
[18:53:52] <ottomata>	 i was thinking jsut CAS would be enough
[18:54:05] <ottomata>	 jsut like we do for yarn.wm.org
[18:54:11] <ottomata>	 gooo VirtualPageView! :)
[18:57:50] <elukey>	 ottomata: it doesn't no but I think that if the port is exposed then you can issue queries to it
[18:57:53] <elukey>	 via a presto client
[18:57:55] <elukey>	 bypassing auth
[19:01:33] <elukey>	 I am personally more and more convinced that we should try Trino, I'll attempt a package after Hadoop's upgrade
[19:01:56] <elukey>	 the prestodb docs are really not great, and the community support is basically zero
[19:11:13] <sukhe>	 mforns: good points on the task but I went through some of the popular apps already (asked my family!) and still couldn't figure it out
[19:11:28] * razzi afk for lunch
[19:11:29] <mforns>	 sukhe: Interesting!
[19:11:47] <sukhe>	 I am going to try to figure out if the requests are coming from a particular geographic region to try to narrow it down
[19:13:10] <mforns>	 sukhe, would be cool (if it hasn't been done already) to know how many distinct IPs there are, no?
[19:15:28] <sukhe>	 yeah, I have not but yep... 
[19:15:40] <wikibugs>	 10Analytics, 10Product-Data-Infrastructure, 10Wikimedia-Logstash, 10observability: Create a separate logstash ElasticSearch index for schemaed events - https://phabricator.wikimedia.org/T265938 (10Krinkle) 05Resolved→03Open >>! In T265938#6614593, @CDanis wrote: > […] For Mediawiki client error logging...
[19:16:05] <sukhe>	 I first thought it was a popular flower delivery app but it wasn't :) 
[19:16:33] <wikibugs>	 10Analytics-Radar, 10Better Use Of Data, 10MediaWiki-API, 10Platform Engineering, and 2 others: Load API request count and latency data from Hadoop to a dashboard - https://phabricator.wikimedia.org/T108414 (10kzimmerman) @sdkim this is in Tracking for BUOD; is this still relevant for your team?
[19:17:17] <sukhe>	 I am guessing it's a popular app but not in regions where we have asked. that is certainly possible
[19:17:33] <sukhe>	 also given the population, the request numbers don't seem to stand out much, at least relatively!
[19:19:06] <wikibugs>	 10Analytics-Radar, 10Better Use Of Data, 10MediaWiki-API, 10Platform Engineering, and 2 others: Load API request count and latency data from Hadoop to a dashboard - https://phabricator.wikimedia.org/T108414 (10mpopov) Also, with the addition of SQL Lab & Presto to Superset (https://superset.wikimedia.org/s...
[19:20:32] <mforns>	 sukhe: but the numbers are huge no? 90M/day, if it's an app, people look at it on a daily basis
[19:30:07] <wikibugs>	 10Analytics-Kanban, 10Better Use Of Data, 10Product-Analytics, 10Product-Data-Infrastructure: Roll-up raw sessionTick data into distribution - https://phabricator.wikimedia.org/T271455 (10kzimmerman) p:05Triage→03Medium
[19:36:29] <wikibugs>	 10Analytics, 10Better Use Of Data, 10Event-Platform, 10MW-1.35-notes (1.35.0-wmf.37; 2020-06-16), and 2 others: Clients need to generate an ISO 8601 formatted timestamp - https://phabricator.wikimedia.org/T240460 (10jlinehan) 05Open→03Resolved
[19:37:39] <sukhe>	 mforns: true...
[19:39:44] <wikibugs>	 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Patch-For-Review, 10Performance-Team (Radar): Convert WikimediaEvents to use ResourceLoader packageFiles - https://phabricator.wikimedia.org/T253634 (10LGoto) a:03Mholloway
[19:40:17] <sukhe>	 mforns: I have been away for a long time but I was looking at a popular app, Snapdeal (shopping): https://play.google.com/store/apps/details?id=com.snapdeal.main&hl=en_CA&gl=US 100 million downloads
[19:40:44] <sukhe>	 but still, 70 is a lot. the surprising part though is why we haven't identified it
[19:40:50] <sukhe>	 I haven't given up!
[19:41:29] <mforns>	 heheh sukhe, could it be a TV streaming service?
[19:42:41] <sukhe>	 hm possible. checking!
[19:43:26] <iflorez>	 Hello A team, I'm not able to access Jupyter Labs this morning. Maybe related to HDFS permission changes work?
[19:46:08] <mforns>	 sukhe: made some queries: for 2021-01-07T22 there are 368448 mediarequests to the image; 294826 distinct IPs requested them
[19:47:15] * elukey afk!
[20:12:00] <ottomata>	 mforns:  oof FYI, i made a mistake when I rolled back SpecialMuteSubmit
[20:12:02] <ottomata>	 https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/662766
[20:12:07] <ottomata>	 almost a month ago
[20:12:17] <mforns>	 oh, looking
[20:12:23] <ottomata>	 all of those events have thus been invalid since then :(
[20:12:30] <ottomata>	 i can backfil them from validation errors
[20:12:32] <ottomata>	 luckily
[20:12:50] <mforns>	 ok
[20:29:55] <milimetric>	 pinging a-team on behalf of iflorez above, I'm not sure why she can't access it either.  She's in analytics-privatedata-users and has a hadoop account with a home folder and everything.
[20:30:36] <ottomata>	 iflorez: what's the issue?
[20:30:38] <milimetric>	 (it == JupyterLab interface, it says "Bad local forwarding specification" after kinit, ssh tunnel)
[20:30:45] <milimetric>	 ottomata: ^
[20:31:48] <milimetric>	 iflorez: I was able to get it to work with "ssh -N stat1006.eqiad.wmnet -L 8000:127.0.0.1:8000", I'm assuming that's the command you're using too?
[20:31:57] <wikibugs>	 10Analytics-Kanban, 10Better Use Of Data, 10Product-Analytics, 10Product-Data-Infrastructure: Roll-up raw sessionTick data into distribution - https://phabricator.wikimedia.org/T271455 (10Mayakp.wiki)
[20:32:28] <wikibugs>	 10Analytics-Kanban, 10Better Use Of Data, 10Product-Analytics, 10Product-Data-Infrastructure: Roll-up raw sessionTick data into distribution - https://phabricator.wikimedia.org/T271455 (10Mayakp.wiki) Status Update: Currently performing data-QA (added checkbox to description). Will update status this week.
[20:33:23] <iflorez>	 yup, no luck with  "ssh -N stat1006.eqiad.wmnet -L 8000:127.0.0.1:8000"
[20:35:45] <wikibugs>	 (03CR) 10Milimetric: [C: 03+1] "Looks good.  I don't love that repeated hard-coded encoding in HQL (the big day case statement), but I'll live without refactoring it.  We" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/658348 (https://phabricator.wikimedia.org/T265732) (owner: 10Fdans)
[20:37:49] <ottomata>	 iflorez:  can you try newpyter instead?
[20:37:50] <ottomata>	 https://wikitech.wikimedia.org/wiki/Analytics/Systems/Jupyter#Newpyter
[20:37:53] <wikibugs>	 (03CR) 10Milimetric: WIP: Add daily referrers Hive table and Oozie job (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/655804 (https://phabricator.wikimedia.org/T270140) (owner: 10Bmansurov)
[20:37:54] <ottomata>	 port 8880
[20:38:01] <iflorez>	 also failed
[20:38:06] <ottomata>	 same?
[20:38:13] <iflorez>	 yes
[20:40:34] <ottomata>	 iflorez:  i killed your jupyter notebook server
[20:40:43] <ottomata>	 can you try again now?  i'm watching loggs on stat1006
[20:44:24] <iflorez>	 tried a few times, can you see the attempts?
[20:45:57] <ottomata>	 hm no
[20:46:31] <ottomata>	 iflorez: newpyter, right?
[20:50:55] <iflorez>	 yes; the error message i'm getting: 
[20:50:55] <iflorez>	 ssh: connect to host bast4002.wikimedia.org port 22: Operation timed out
[20:50:55] <iflorez>	 ssh_exchange_identification: Connection closed by remote host
[20:54:17] <ottomata>	 ah
[20:54:18] <ottomata>	 iflorez: 
[20:54:20] <ottomata>	 bastions have changed
[20:54:21] <ottomata>	 https://wikitech.wikimedia.org/wiki/Bastion
[20:54:37] <ottomata>	 edit your .ssh/config and change bast4002 to bast4003 (and any otherr bastion corrections you might need)
[20:54:44] <iflorez>	 excellent, ty! 
[21:09:50] <wikibugs>	 10Analytics, 10Analytics-Kanban: Make stats.wikimedia.org point to wikistats2 by default - https://phabricator.wikimedia.org/T237752 (10fdans)
[21:09:56] <wikibugs>	 10Analytics: Archive docs for old Wikistats and update links to Wikistats 2 - https://phabricator.wikimedia.org/T238001 (10fdans) 05Open→03Resolved This is a pretty old task. At this point there's probably a few documents with ambiguous references to Wikistats, but we can correct them as they come. Closing.
[21:24:37] <milimetric>	 argh, forgot about the bastions, sorry.  I've been using bast1002 for a while and haven't had any trouble when it changes
[21:57:55] <icinga-wm>	 PROBLEM - Hadoop DataNode on an-worker1118 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.hdfs.server.datanode.DataNode https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration
[22:06:04] <elukey>	 this is me --^
[22:06:43] <icinga-wm>	 RECOVERY - Hadoop DataNode on an-worker1118 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.hdfs.server.datanode.DataNode https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration
[22:29:00] <elukey>	 !log hdfs master failover an-worker1118 -> an-worker1124, created dedicated partition for /var/lib/hadoop/name (root partition filled up), restarted namenode on 1118 (now recovering edit logs)
[22:29:02] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[22:29:17] <elukey>	 !log the previous entry was related to the Hadoop backup cluster
[22:29:19] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[22:44:18] <wikibugs>	 10Analytics, 10Product-Data-Infrastructure, 10Wikimedia-Logstash, 10observability: Create a separate logstash ElasticSearch index for schemaed events - https://phabricator.wikimedia.org/T265938 (10Tgr) >>! In T265938#6812467, @Krinkle wrote: > I don't expect JS and PHP errors to ever be displayed on the sa...
[22:47:34] <wikibugs>	 10Analytics, 10Product-Data-Infrastructure, 10Wikimedia-Logstash, 10observability: Create a separate logstash ElasticSearch index for schemaed events - https://phabricator.wikimedia.org/T265938 (10Tgr) In any case I don't see how ECS would prevent conflicts. It's basically a type system, with types being a...
[23:30:27] <sukhe>	 fdans: yes, definitely a good catch!
[23:30:36] <sukhe>	 I tried to find alternatives to apps that became popular after that
[23:30:56] <sukhe>	 I ma going with the theory that someone copied the code but the image is actually not displayed
[23:34:22] <fdans>	 sukhe: I wonder if one of those apps, when accessed from india, uses the image as error/"your country is blocking us" screen
[23:34:40] <fdans>	 but if that were the case you'd think the requests would gradually slow down with time
[23:39:31] <wikibugs>	 10Analytics, 10Product-Data-Infrastructure, 10Wikimedia-Logstash, 10observability: Create a separate logstash ElasticSearch index for schemaed events - https://phabricator.wikimedia.org/T265938 (10Krinkle) >>! In T265938#6813541, @Tgr wrote: >>>! In T265938#6812467, @Krinkle wrote: >> I don't expect JS and...