[06:40:55] 10Analytics, 10EventBus, 10serviceops: helmfile apply with values.yaml file change did not deploy new k8s pods - https://phabricator.wikimedia.org/T228700 (10fsero) the main issue is in notifying changes to the deployment object department, not in helmfile. helmfile is AFAICT working as intended. In the eve... [08:01:18] 10Analytics, 10Analytics-Kanban, 10Operations, 10Traffic, and 2 others: TLS certificates for Analytics origin servers - https://phabricator.wikimedia.org/T227860 (10elukey) 05Open→03Stalled Stalled due to https://phabricator.wikimedia.org/T228730 [08:49:02] 10Analytics, 10Wikimedia-Portals, 10cloud-services-team: https://dumps.wikimedia.org/other/pageviews/ lacks hourly pageviews since 20190722-16:00 - https://phabricator.wikimedia.org/T228731 (10hashar) [09:01:16] 10Analytics, 10Wikimedia-Portals, 10cloud-services-team: https://dumps.wikimedia.org/other/pageviews/ lacks hourly pageviews since 20190722-16:00 - https://phabricator.wikimedia.org/T228731 (10hashar) The DNS change is f4c51db24a4953eb4b24b3480b555ad5cf61b219 `Mon Jul 22 16:31:01 2019 +0000` ` name=templates... [09:02:37] 10Analytics, 10Wikimedia-Portals, 10cloud-services-team: https://dumps.wikimedia.org/other/pageviews/ lacks hourly pageviews since 20190722-17:00 - https://phabricator.wikimedia.org/T228731 (10hashar) [09:11:03] 10Analytics, 10Wikimedia-Portals, 10cloud-services-team: https://dumps.wikimedia.org/other/pageviews/ lacks hourly pageviews since 20190722-17:00 - https://phabricator.wikimedia.org/T228731 (10elukey) @hashar thanks a lot for the ping. Yesterday we restarted the job that produces the files, but it has been f... [09:23:53] !log restart projectview-hourly-coordinator with correct config - T228731 [09:23:56] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:23:57] T228731: https://dumps.wikimedia.org/other/pageviews/ lacks hourly pageviews since 20190722-17:00 - https://phabricator.wikimedia.org/T228731 [09:31:06] 10Analytics, 10Wikimedia-Portals, 10cloud-services-team: https://dumps.wikimedia.org/other/pageviews/ lacks hourly pageviews since 20190722-17:00 - https://phabricator.wikimedia.org/T228731 (10elukey) Ah no ok there is an explanation for this trouble. @nuria deployed yesterday refinery as mentioned, but she... [09:37:06] 10Analytics, 10Analytics-Kanban, 10Wikimedia-Portals, 10cloud-services-team: https://dumps.wikimedia.org/other/pageviews/ lacks hourly pageviews since 20190722-17:00 - https://phabricator.wikimedia.org/T228731 (10elukey) p:05Triage→03Normal a:03elukey [09:37:20] 10Analytics, 10Analytics-Kanban: Frequent/Long GC old gen collections for HDFS namenodes on an-master100[1,2] - https://phabricator.wikimedia.org/T228620 (10elukey) [09:56:31] 10Analytics, 10Analytics-Kanban, 10Wikimedia-Portals, 10cloud-services-team: https://dumps.wikimedia.org/other/pageviews/ lacks hourly pageviews since 20190722-17:00 - https://phabricator.wikimedia.org/T228731 (10elukey) I can see data at the bottom of https://dumps.wikimedia.org/other/pageviews/2019/2019-... [10:37:13] * elukey lunch! [10:37:15] 10Analytics, 10WMDE-Analytics-Engineering: Public Data Review Needed - https://phabricator.wikimedia.org/T227905 (10GoranSMilovanovic) @Nuria As stated in the README.txt: ` ### --- Description The WD_pageviewsPerType.Rds file is an .Rds serialization of the data set meant to support the WMDE WD_PageviewsPer... [11:02:24] (03CR) 10Fdans: [C: 03+2] Add file extension and media type classification to media files UDF [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/517641 (https://phabricator.wikimedia.org/T225911) (owner: 10Fdans) [11:41:14] (03PS6) 10Fdans: Add UDF to get wiki project from referer string [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/523903 (https://phabricator.wikimedia.org/T228151) [11:55:38] (03CR) 10Fdans: "my bad, I just realized that I never sent my responses to the previous CR, so they're included in these comments. I hate gerrit with pass" (038 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/523903 (https://phabricator.wikimedia.org/T228151) (owner: 10Fdans) [12:15:17] 10Analytics, 10EventBus, 10serviceops, 10Patch-For-Review: helmfile apply with values.yaml file change did not deploy new k8s pods - https://phabricator.wikimedia.org/T228700 (10akosiaris) I think the issue is on the stream-config.yaml file, not the config.yaml template. Using `.Files.Get` means the file i... [12:15:27] 10Analytics, 10EventBus, 10serviceops, 10Patch-For-Review: helmfile apply with values.yaml file change did not deploy new k8s pods - https://phabricator.wikimedia.org/T228700 (10akosiaris) p:05Triage→03Normal [12:49:04] 10Analytics, 10EventBus, 10serviceops, 10Patch-For-Review: helmfile apply with values.yaml file change did not deploy new k8s pods - https://phabricator.wikimedia.org/T228700 (10Ottomata) Thanks so much you two! I'll take some of what Fabian wrote and add it to my EventGate docs too. [12:53:01] 10Analytics, 10WMDE-Analytics-Engineering, 10Wikidata, 10Wikidata-Campsite: Track WDQS updater UA in wikidata-special-entitydata grafana dashboard - https://phabricator.wikimedia.org/T218998 (10alaa_wmde) [12:53:28] 10Analytics, 10WMDE-Analytics-Engineering, 10Wikidata, 10Wikidata-Campsite: Track WDQS updater UA in wikidata-special-entitydata grafana dashboard - https://phabricator.wikimedia.org/T218998 (10alaa_wmde) [12:54:19] 10Analytics, 10WMDE-Analytics-Engineering, 10Wikidata, 10Wikidata-Campsite: Track WDQS updater UA in wikidata-special-entitydata grafana dashboard - https://phabricator.wikimedia.org/T218998 (10alaa_wmde) [12:56:54] 10Analytics, 10Analytics-Kanban, 10Wikimedia-Portals, 10cloud-services-team: https://dumps.wikimedia.org/other/pageviews/ lacks hourly pageviews since 20190722-17:00 - https://phabricator.wikimedia.org/T228731 (10JHedden) >>! In T228731#5356497, @hashar wrote: > Maybe related are some hiera values at: > `... [12:58:17] 10Analytics, 10Analytics-Kanban, 10Wikimedia-Portals, 10cloud-services-team: projectview-hourly-coordinator needs to alarm when in error - https://phabricator.wikimedia.org/T228747 (10Nuria) [13:50:03] 10Analytics, 10Operations, 10Core Platform Team Backlog (Watching / External), 10Patch-For-Review, and 2 others: Replace and expand codfw kafka main hosts (kafka200[123]) with kafka-main200[12345] - https://phabricator.wikimedia.org/T225005 (10Ottomata) [13:51:15] 10Analytics, 10EventBus, 10Core Platform Team Backlog (Watching / External), 10Patch-For-Review, 10Services (later): Modern Event Platform: Stream Intake Service: Migrate change-prop events to new (EventGate) style schemas - https://phabricator.wikimedia.org/T226522 (10Ottomata) [13:51:20] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), and 2 others: Use new event schema format for change-prop events - https://phabricator.wikimedia.org/T228318 (10Ottomata) [14:12:56] PROBLEM - superset.wikimedia.org HTTPS on analytics-tool1004 is CRITICAL: connect to address 10.64.36.116 and port 443: Connection refused https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster [14:15:06] yessss [14:15:09] this is expected [14:15:14] it is the new version of the alarm [14:28:17] 10Analytics, 10LDAP-Access-Requests, 10Operations, 10wikimediafoundation.org: Access to WikimediaFoundation.org analytics for Deb - https://phabricator.wikimedia.org/T227496 (10herron) 05Open→03Resolved I wasn't able to find an ldap account with shell username `Deb_Zierten`, but I do see shell username... [14:44:43] hey teammmmmm :D [14:44:58] holaaaaaaaaa [14:45:02] o/ o/ o/ [14:45:26] mforns: HOLA [14:45:41] :] [14:46:09] mforns: weren't you back next week? [14:46:20] no... today! [14:46:24] milimetric: ah RIGHT [14:46:29] cause today is teh 22! [14:46:37] yea :] [14:46:45] it is the 23rd to be honest :P [14:46:54] heh, yes [14:46:57] :) [14:47:05] welcome back! [14:47:13] thaananks! [14:47:14] elukey: jajajaja [14:47:30] elukey: i am on TOP of it [14:47:35] elukey: as you can tell [14:49:04] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Add new mrediatypes to media classification refinery code - https://phabricator.wikimedia.org/T225911 (10Nuria) [14:49:18] 10Analytics, 10Discovery, 10Operations, 10Research-Backlog, 10Patch-For-Review: Workflow to be able to move data files computed in jobs from analytics cluster to production - https://phabricator.wikimedia.org/T213976 (10Ottomata) [14:49:59] (03PS1) 10Ottomata: Create container with read access during swift upload [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525106 (https://phabricator.wikimedia.org/T219544) [14:50:30] fdans: Hola! let's talk in standup where do we stand with reportupdater, did we verify all jobs are running from the new depot? [14:52:25] (03CR) 10Ottomata: [V: 03+2 C: 03+2] Create container with read access during swift upload [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525106 (https://phabricator.wikimedia.org/T219544) (owner: 10Ottomata) [14:59:23] nuria: hmmm I checked sooome of them, they were running fine, let me check the rest [15:00:06] 10Analytics, 10WMDE-Analytics-Engineering: Public Data Review Needed - https://phabricator.wikimedia.org/T227905 (10Nuria) Looks good, closing. [15:00:12] 10Analytics, 10WMDE-Analytics-Engineering: Public Data Review Needed - https://phabricator.wikimedia.org/T227905 (10Nuria) 05Open→03Resolved [15:05:59] (03CR) 10Nuria: "Couple nits and one outstanding question about data that comes from the apps." (033 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/523903 (https://phabricator.wikimedia.org/T228151) (owner: 10Fdans) [15:09:41] nuria: why transforming the host to lower case? [15:11:23] fdans: cause that way you are sure of case before doing comparations that rely on case? see method below case conversion before comparations [15:11:28] fdans: makes sense? [15:13:06] nuria: that makes sense, but should that conversion be in the udf? I wanted to avoid changing the webrequest/pageview def functions as much as possible [15:13:50] 10Analytics, 10Analytics-EventLogging, 10Operations: Decommission m4 proxies (dbproxy1004 and dbproxy1008) - https://phabricator.wikimedia.org/T228768 (10Marostegui) [15:14:34] fdans: the function that executes the comparation should do that so if affects all callers. [15:19:28] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), and 4 others: Modern Event Platform: Stream Intake Service: Migrate eventlogging-service-eventbus events to eventgate-main - https://phabricator.wikimedia.org/T211248 (10Pchelolo) [15:20:21] 10Analytics, 10Analytics-Kanban, 10Wikimedia-Portals, 10cloud-services-team: https://dumps.wikimedia.org/other/pageviews/ lacks hourly pageviews since 20190722-17:00 - https://phabricator.wikimedia.org/T228731 (10elukey) [15:23:41] ottomata: i'm about to start writing the thing to produce kafka messages about swift uploads. I'm thinking same upload workflow, but separate action calling a python script? It could be done with shell but python is nicer :) [15:25:37] ebernhardson: +1 same workflow and different action [15:28:43] minor problem is doesn't look like hadoop executor nodes have the kafka python library, but they don't have kafkacat either so either way have to install something (or i can port over how we build python venv's with deps, but it seems for your own stuff you install to cluster) [15:29:41] hm [15:29:49] ebernhardson: iw as thinking eventgate [15:29:53] then it can just be http post [15:29:59] ottomata: ok [15:30:11] ebernhardson: i was planning on working on this today :0 [15:30:15] ottomata: :) [15:30:50] let's figure out the event schema first [15:31:10] i'm not sure about one event stream to rule them all for uploads, different things need different metadata [15:31:18] or we have to hax it in place by having one of the files contain metadata to read [15:31:58] different streams is good [15:32:01] but one schema? [15:32:20] the schema's have to vary to allow the different metadata, or they have to be so generic as to not validate much [15:32:34] how so? [15:32:40] what different metadata? [15:32:53] so for example, our ML uploads need to have information about the feature set it needs to be attached to, the wiki it was built for, the features that it needs calculated, and perhaps some other stuff [15:33:47] the files themselves are, for our purposes, a black box text file that gets passed out of the training and into the inferrence side [15:34:23] ebernhardson: would a string: string map work for that purpose? [15:34:32] ottomata: yes, but then the schema isn't validating much :P [15:35:06] it is validating types and fields :) [15:35:37] i guess what i'd like is for the producer code to not need to know too much about how to construct the event. [15:35:42] if there only one event schema [15:35:48] hmm [15:36:00] it can do it the same way for every upload type, just with different parameters [15:39:22] nuria: just took a look at referer strings from webrequests that come from the mobile app, and there doesn't seem to be any difference in the referer strings themselves, so I'm not sure how to test that [15:39:45] i suppose my concern is failing as early as possible when things are wrong, the consumer has to be resilient anyways but will basically have to define it's own schema to read since map won't be enough validation to continue processing [15:39:58] fdans: do referrers from mobile app have mobile site as referrer? [15:40:08] nope [15:40:29] fdans: ay, then , i do not understand what you mean [15:40:37] so where the producer has map, the consumer would need a more specific object/properties definition and just throw away/log invalid messages. but it would be nice to log/fail those at the source instead of the destination [15:41:29] i suppose if it's python there really isn't a difference between map and and object with all string properties, the script i suppose could take a parameter that tells it which schema? [15:41:53] nuria: hmmm you're asking for a test that imitates referer strings as they come from the wikipedia app right? [15:42:25] but those referer strings just look like https://en.wikipedia.org/, https://pt.wikipedia.org/, '-'... [15:42:46] so there's no special case to test here [15:49:28] 10Analytics, 10Analytics-Kanban, 10Research-Backlog: Release edit data lake data as a public json dump /mysql dump, other? - https://phabricator.wikimedia.org/T208612 (10Milimetric) @leila / @nettrom_WMF: fyi I'm working on this now. I've started a draft page where I'm thinking out loud about how to publis... [15:57:25] fdans: that is the way they would look if the app is displaying a webview of the mobile site, for other requests (non webview ones) they should look different , right? [15:59:16] nuria: if it was displaying a webview of the mobile site, wouldn't it show es.m.wikipedia.org? [16:00:23] nuria: most requests from mobile-app actually have '-' as referer string [16:00:27] fdans: ah I see, those urls are desktop [16:00:36] fdans: ah ok, that is what i would expect [16:00:43] fdans: more often thna not [16:00:45] *that [16:01:12] ping ottomata milimetric standdduppp [16:02:30] ping ottomata [16:04:18] be there asap phpne... [16:11:58] (sorry ebernhardson phone call, meetings will get back to you... ) [16:12:30] no worries :) i'm here all day [16:22:59] 10Analytics-Kanban, 10Product-Analytics: Make aggregate data on editors per country per wiki publicly available - https://phabricator.wikimedia.org/T131280 (10Nuria) a:05Milimetric→03mforns [17:17:23] nuria: is it okay if I move our meeting to later in the day PST? [17:22:09] oh nuria, forgot you had a question [17:22:14] leila: I can do tomorrow or later, like 4pm [17:22:23] mforns: ya, let's talk in the bc? [17:22:41] nuria: I'll move it to later (4pm) today. Tomorrow I have to be in WikiLead and the schedule may be too packed to plan for meetings. [17:22:44] nuria: thanks! [17:24:52] mforns: batcave? [17:26:59] nuria, oh sorry, yes! [17:27:02] omw [17:37:31] 10Analytics, 10Analytics-Kanban, 10Discovery-Search (Current work): Spike. Load search data into turnilo to test whether exploratory data can do away with some of the dashboards - https://phabricator.wikimedia.org/T216058 (10debt) @EBernhardson will work on this along with the DYM work, probably before Q2 :) [17:41:01] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10ORES, and 4 others: Fix "Must provide the 'topic' parameter" in ORES /precache endpoint - https://phabricator.wikimedia.org/T228689 (10Halfak) Is there documentation for meta.stream? Should we expect the same data in a different field? [17:41:07] 10Analytics, 10EventBus, 10Core Platform Team Backlog (Watching / External), 10Patch-For-Review, 10Services (later): Modern Event Platform: Stream Intake Service: Migrate change-prop events to new (EventGate) style schemas - https://phabricator.wikimedia.org/T226522 (10Pchelolo) 05Open→03Resolved All... [17:41:13] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Core Platform Team Backlog (Watching / External), 10Services (watching): Modern Event Platform: Stream Intake Service - https://phabricator.wikimedia.org/T201068 (10Pchelolo) [17:42:22] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10ORES, and 4 others: Fix "Must provide the 'topic' parameter" in ORES /precache endpoint - https://phabricator.wikimedia.org/T228689 (10Pchelolo) I'm not sure about the doc, @Ottomata might know more, but yes, the only change is `.meta.topic` rena... [17:43:37] 10Analytics, 10Analytics-Kanban, 10Discovery-Search (Current work): Spike. Load search data into turnilo to test whether exploratory data can do away with some of the dashboards - https://phabricator.wikimedia.org/T216058 (10EBernhardson) reference druid schema for mediawiki history: https://docs.google.com/... [17:44:02] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10ORES, and 4 others: Fix "Must provide the 'topic' parameter" in ORES /precache endpoint - https://phabricator.wikimedia.org/T228689 (10Ottomata) https://github.com/wikimedia/eventgate#streams And we set `stream_field` to `meta.stream` in https:/... [17:48:58] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10ORES, and 4 others: Fix "Must provide the 'topic' parameter" in ORES /precache endpoint - https://phabricator.wikimedia.org/T228689 (10Halfak) It looks like we are using "meta.topic" here: https://github.com/wikimedia/ores/blob/master/ores/wsgi/u... [17:49:35] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10ORES, and 4 others: Fix "Must provide the 'topic' parameter" in ORES /precache endpoint - https://phabricator.wikimedia.org/T228689 (10Halfak) [17:52:50] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10ORES, and 4 others: Fix "Must provide the 'topic' parameter" in ORES /precache endpoint - https://phabricator.wikimedia.org/T228689 (10Ottomata) Great! I left a comment there. It is 'meta.stream' not meta. schema. The [[ https://github.com/wik... [17:59:23] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10ORES, and 4 others: Fix "Must provide the 'topic' parameter" in ORES /precache endpoint - https://phabricator.wikimedia.org/T228689 (10Halfak) Thanks! Was working too fast. I think I've got your changes. Please take another look :) [18:00:24] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10ORES, and 4 others: Fix "Must provide the 'topic' parameter" in ORES /precache endpoint - https://phabricator.wikimedia.org/T228689 (10Ottomata) That should do it! [18:02:17] bbl [18:08:43] * elukey off! [19:22:58] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10ORES, and 4 others: Fix "Must provide the 'topic' parameter" in ORES /precache endpoint - https://phabricator.wikimedia.org/T228689 (10Halfak) a:03Halfak [20:32:06] 10Analytics, 10Product-Analytics: Hash all pageTokens or temporary identifiers from the EL Sanitization white-list for Android - https://phabricator.wikimedia.org/T226852 (10mpopov) p:05Triage→03Normal [20:38:28] 10Analytics, 10Product-Analytics: Hash all pageTokens or temporary identifiers from the EL Sanitization white-list for Android - https://phabricator.wikimedia.org/T226852 (10Nuria) Also, please remember to remove userId, revisionId pairs, or userId, pageId [20:43:18] 10Analytics, 10Pageviews-API, 10Tool-Pageviews: 429 Too Many Requests hit despite throttling to 100 req/sec - https://phabricator.wikimedia.org/T219857 (10kaldari) I've been hitting this problem consistently with the Massviews tool (only using 1 tab). I wonder if slightly tweaking the 10ms pause would fix it... [21:04:18] 10Analytics, 10Pageviews-API, 10Tool-Pageviews: 429 Too Many Requests hit despite throttling to 100 req/sec - https://phabricator.wikimedia.org/T219857 (10Nuria) Given that this tool for links like the one above (see couple comments up) does 5000 requests on 1 tab (see network panel for chrome) it is unlikel... [21:07:09] mforns: tomorrow we can talk talk about the geoeditors data and what milimetric and myself were discussing [21:10:50] 10Analytics, 10Pageviews-API, 10Tool-Pageviews: 429 Too Many Requests hit despite throttling to 100 req/sec - https://phabricator.wikimedia.org/T219857 (10MusikAnimal) > you can send N request (the browser will multiplex) and when the first one gets a 429 you stop , message user on UI and continue some time... [21:13:05] nuria, ok [21:20:15] 10Analytics, 10Pageviews-API, 10Tool-Pageviews: 429 Too Many Requests hit despite throttling to 100 req/sec - https://phabricator.wikimedia.org/T219857 (10Nuria) >because again we are abiding by the advertised rate limit but are still getting 429s. Ok, maybe we need to look a this a bit more but in any case... [22:53:01] 10Analytics, 10Analytics-Kanban, 10Discovery, 10Operations, 10Research-Backlog: Make oozie swift upload emit event to Kafka about swift object upload complete - https://phabricator.wikimedia.org/T227896 (10Ottomata) Ok, from discussions with Erik today, we are going with an event like: `lang=json { "$... [23:16:57] 10Analytics, 10Analytics-Kanban, 10Discovery, 10Operations, 10Research-Backlog: Make oozie swift upload emit event to Kafka about swift object upload complete - https://phabricator.wikimedia.org/T227896 (10Nuria) Let me catch up here, seems that urls should have versions and not only be defined by a loca... [23:20:45] 10Analytics, 10Analytics-Kanban, 10Discovery, 10Operations, 10Research-Backlog: Make oozie swift upload emit event to Kafka about swift object upload complete - https://phabricator.wikimedia.org/T227896 (10Ottomata) The object URLs are totally up to the user, the script just uploads whatever is in the hd...