[00:53:29] 10Analytics, 10Pageviews-API, 10Tool-Pageviews: 429 Too Many Requests hit despite throttling to 100 req/sec - https://phabricator.wikimedia.org/T219857 (10Milimetric) So, looked into code history more carefully. There's literally [[ https://github.com/wikimedia/analytics-aqs/commit/ccd65c11bac2630969df45df1... [01:06:22] (03CR) 10Milimetric: [C: 03+2] Improving examples arround how to start jobs (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525652 (owner: 10Nuria) [01:06:25] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Improving examples arround how to start jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525652 (owner: 10Nuria) [01:08:15] (03CR) 10Milimetric: [C: 03+2] "I didn't think of that, it makes sense, keep it." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525507 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [01:11:43] (03CR) 10Nuria: [V: 03+2] cassandra: move oozie bundle to hive2 actions [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525507 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [01:52:49] 10Analytics: Request for a large request data set for caching research and tuning - https://phabricator.wikimedia.org/T225538 (10Nuria) @Danielsberger we are in Q1, as in the fiscal year has just started as it starts in July 1st. This quarter our team has couple less people due to family leave so it will be hard... [04:10:28] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Core Platform Team Legacy (Watching / External), 10Services (watching): Factor out eventgate-wikimedia factory into its own gerrit repo and use it for deployment pipeline - https://phabricator.wikimedia.org/T226668 (10Jdforrester-WMF) Needs a task to kill of... [06:28:40] !log restart aqs coordinator with hive2 actions [06:28:42] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [06:33:09] (03PS1) 10Elukey: aqs: re-add job_tracker parameter to oozie coordinator [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525725 [06:37:09] (03CR) 10Elukey: [V: 03+2 C: 03+2] aqs: re-add job_tracker parameter to oozie coordinator [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525725 (owner: 10Elukey) [06:37:28] 10Analytics: Old job_tracker setting in oozie properties - https://phabricator.wikimedia.org/T216519 (10elukey) 05Open→03Declined It is needed by the oozie XML validator, I just got an error while restarting the aqs hourly coordinator (I removed the job_tracker parameter as test in https://gerrit.wikimedia.o... [07:18:48] !log deploy last version of refinery to HDFS [07:18:49] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:27:55] !log restart aqs coordinator to pick up hive2 settings [07:27:57] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:31:37] !log restart banner_impressions daily coordinator to pick up hive2 settings [07:31:38] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:33:06] !log restart browser-general daily coordinator to pick up hive2 settings [07:33:08] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:42:58] !log restart mediacounts-load hourly coordinator after refinery deployment to hdfs [07:43:00] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:46:18] 10Analytics, 10Analytics-Kanban, 10Wikimedia-Portals, 10cloud-services-team: projectview-hourly-coordinator needs to alarm when in error - https://phabricator.wikimedia.org/T228747 (10elukey) I think that we could expand a little bit this task and audit all the coordinators that we care, since today I kill... [07:49:20] 10Analytics, 10Research: Check home leftovers of ISI researchers - https://phabricator.wikimedia.org/T215775 (10elukey) @Isaac what are the next steps? :) [07:52:02] 10Analytics, 10Android-app-Bugs, 10Wikipedia-Android-App-Backlog: App requests classified as pageviews that probably should not be so - https://phabricator.wikimedia.org/T229068 (10Charlotte) Thanks for asking @Nuria. @Dbrant is out on holiday, so @Sharvaniharan or @cooltey might be able to take a look for you. [10:37:55] * elukey lunch! [10:42:42] (03CR) 10Fdans: Add access type to mediacounts hourly dataset (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/517426 (https://phabricator.wikimedia.org/T225910) (owner: 10Fdans) [12:10:11] heya teamm [12:43:47] 10Analytics, 10Research: Check home leftovers of ISI researchers - https://phabricator.wikimedia.org/T215775 (10Isaac) > what are the next steps? :) @leila -- what needs to happen before these four directories are permanently removed (minus any datasets approved to be released)? [13:20:56] (03PS6) 10Fdans: [wip]Add file extension and media classification to mediacounts job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/522390 (https://phabricator.wikimedia.org/T225911) [13:36:00] ah ebernhardson sorry! Interesting about the analytics user require ment for druid load [13:36:26] a-team let's answer ebernhardson question, I don't know why it is like that [13:37:16] ottomata: o/ [13:37:26] o/! [13:37:51] IIRC we did it to avoid accidental druid loads/indexations from "regular" users (due to testing, etc..) [13:38:30] do we want to allow e.g. analytics-search user? [13:38:46] +1 from my side, I think it is fine [13:38:49] ok [13:39:01] ebernhardson: then ya go ahead, you can submit patch to make it work for analytics-* [13:41:38] 10Analytics, 10EventBus, 10Release-Engineering-Team: Delete eventgate-ci repository from gerrit - https://phabricator.wikimedia.org/T229111 (10Ottomata) [13:41:57] 10Analytics, 10EventBus, 10Gerrit, 10Release-Engineering-Team: Delete eventgate-ci repository from gerrit - https://phabricator.wikimedia.org/T229111 (10Ottomata) [13:43:54] my noob self just learned about doing + <~> + <.> to quit a frozen ssh session [13:44:24] i need more ninja skills like this [13:45:15] I am n00ber than you since I didn't know/remember the trick :) [13:45:21] thanks! [13:46:21] (03PS1) 10Elukey: data_quality: move the oozie coordinator to hive2 actions [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525809 (https://phabricator.wikimedia.org/T227257) [13:48:36] (03PS1) 10Elukey: edit-hourly: move oozie coordinator to hive2 actions [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525810 (https://phabricator.wikimedia.org/T227257) [13:52:00] (03PS1) 10Elukey: interlanguage: move oozie coordinator to hive2 actions [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525811 (https://phabricator.wikimedia.org/T227257) [13:53:02] i didn't know that either! [13:58:39] (03PS1) 10Elukey: mediacounts: move archive and load oozie coord to hive2 actions [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525813 (https://phabricator.wikimedia.org/T227257) [14:12:53] ottomata elukey yeah my connection here is a total turd so my ssh sessions are always freezing. Until now I was just closing the terminal window and getting angry but this works perfect and you don't have to get rid of the console history [14:13:34] i always did that too! [14:25:41] ottomata: do you have time to work on https://phabricator.wikimedia.org/T228291 during the next days? [14:25:59] Ah sure elukey i maybe can try today, or if not then next week. [14:26:15] yep yep no rush whenever you want! Next week would be perfect so we can test it [14:26:15] 10Analytics, 10Analytics-Kanban: Refine should accept principal name for hive2 jdbc connection for DDL - https://phabricator.wikimedia.org/T228291 (10Ottomata) [14:46:43] ottomata: I am setting up camus_eventlogging for the testing cluster, I am wondering if you have any preference about what topics to import.. I am thinking about 1 or 2 topics maximum [14:46:58] to apply refine on [14:47:03] there are 2 main different jobs for refine [14:47:04] eventlogging [14:47:06] and mediawiki_events [14:47:07] so [14:47:09] maybe one of each? [14:47:15] ah you want both? [14:47:16] perhaps [14:47:18] ah [14:47:20] no i dunno [14:47:25] if one works with kerberos [14:47:27] the other will too [14:47:38] let's do EL since it has to talk to meta.wm.org just for fun [14:47:39] so [14:47:41] NavigationTiming [14:47:43] is always good :) [14:47:48] perfect, I'll do it :) [14:47:50] (03CR) 10Nuria: [C: 04-1] "This explodes the current data potentially two orders of magnitude, i think we might need to think of abetter schema" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/522390 (https://phabricator.wikimedia.org/T225911) (owner: 10Fdans) [14:48:37] (03CR) 10Nuria: Add access type to mediacounts hourly dataset (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/517426 (https://phabricator.wikimedia.org/T225910) (owner: 10Fdans) [14:51:46] fdans: let's talk about mediacounts today or friday, we probably need a new table schema [14:53:41] 10Analytics, 10Analytics-Kanban, 10Wikimedia-Portals, 10cloud-services-team: projectview-hourly-coordinator needs to alarm when in error - https://phabricator.wikimedia.org/T228747 (10Nuria) @elukey sounds good, let's add those coordinators [14:53:46] nuria: yea we could have a mediarequests_hourly table in hive, with no transcoding info [14:55:05] especially since transcodings are not going to be exposed in the api and it is the dimension that potentially would make the dataset bigger than pageview_hourly [14:55:42] 10Analytics, 10EventBus, 10Gerrit, 10Release-Engineering-Team, 10Repository-Admins: Delete eventgate-ci repository from gerrit - https://phabricator.wikimedia.org/T229111 (10Jdforrester-WMF) [14:56:37] fdans: did users provided any feedback whether transcoding would be useful in mediacounts api (gergo, musikanimal , kaldari ...at all) [14:56:40] *et all [14:56:58] nope [15:18:25] 10Analytics, 10EventBus, 10Gerrit, 10Release-Engineering-Team, 10Repository-Admins: Delete eventgate-ci repository from gerrit - https://phabricator.wikimedia.org/T229111 (10MarcoAurelio) @Ottomata Can this be done now or are we waiting for the other tasks to be done first? Thanks. [15:29:08] 10Analytics, 10EventBus, 10Gerrit, 10Release-Engineering-Team, 10Repository-Admins: Delete eventgate-ci repository from gerrit - https://phabricator.wikimedia.org/T229111 (10Ottomata) Can be done now! Thank you! [15:29:22] 10Analytics, 10EventBus, 10Gerrit, 10Release-Engineering-Team, 10Repository-Admins: Delete eventgate-ci repository from gerrit - https://phabricator.wikimedia.org/T229111 (10Ottomata) a:05Ottomata→03None [15:29:37] 10Analytics, 10EventBus, 10Gerrit, 10Release-Engineering-Team, 10Repository-Admins: Delete eventgate-ci repository from gerrit - https://phabricator.wikimedia.org/T229111 (10Ottomata) Me being assigned was because Phab defaulted to that as a sub task. I don't have privs to delete afaik. [15:36:05] 10Analytics: Review Bacula home backups set for stat100[56] - https://phabricator.wikimedia.org/T201165 (10elukey) 05Stalled→03Resolved a:03elukey [15:50:30] 10Analytics, 10Analytics-Kanban, 10Wikimedia-Portals: projectview-hourly-coordinator needs to alarm when in error - https://phabricator.wikimedia.org/T228747 (10bd808) [15:59:24] (03CR) 10Nuria: [C: 03+2] "Looks good, let's make sure yo add this job to the ones that need to be restarted" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525810 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [16:02:10] ping : milimetric standdduppp [16:02:39] off today nuria, couldn’t get backup baby care [16:17:38] (03CR) 10Nuria: [C: 03+2] "Added this job to train ether pad so it can be re-started next week" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525813 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [16:31:28] me off! [16:47:58] 10Analytics, 10EventBus, 10Gerrit, 10Release-Engineering-Team-TODO, and 2 others: Delete eventgate-ci repository from gerrit - https://phabricator.wikimedia.org/T229111 (10greg) p:05Triage→03Normal [16:48:31] 10Analytics, 10EventBus, 10Gerrit, 10Release-Engineering-Team-TODO, and 2 others: Delete eventgate-ci repository from gerrit - https://phabricator.wikimedia.org/T229111 (10greg) #repository-admins usually take care of these. [17:39:48] 10Analytics, 10Core Platform Team (Modern Event Platform (TEC2)): Ingest api data (for posts) into druid - https://phabricator.wikimedia.org/T218348 (10CCicalese_WMF) [17:40:03] 10Analytics, 10EventBus, 10Operations, 10Core Platform Team (Modern Event Platform (TEC2)), and 2 others: Possibly expand Kafka main-{eqiad,codfw} clusters in Q4 2019. - https://phabricator.wikimedia.org/T217359 (10CCicalese_WMF) [17:40:15] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), and 2 others: Modern Event Platform: Stream Connectors - https://phabricator.wikimedia.org/T214430 (10CCicalese_WMF) [17:41:06] 10Analytics, 10Analytics-EventLogging, 10Better Use Of Data, 10EventBus, and 5 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10CCicalese_WMF) [17:41:29] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), 10Services (watching): CI Support for Schema Registry - https://phabricator.wikimedia.org/T206814 (10CCicalese_WMF) [17:41:32] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), 10Services (watching): Modern Event Platform: Schema Registry: Implementation - https://phabricator.wikimedia.org/T206789 (10CCicalese_WMF) [17:41:39] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), and 2 others: Modern Event Platform: Stream Configuration - https://phabricator.wikimedia.org/T205319 (10CCicalese_WMF) [17:42:03] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), and 2 others: Modern Event Platform (TEC2) - https://phabricator.wikimedia.org/T185233 (10CCicalese_WMF) [17:46:48] (03CR) 10Nuria: [C: 03+2] "Added to train etherpad" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525811 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [17:46:50] (03CR) 10Nuria: [V: 03+2 C: 03+2] interlanguage: move oozie coordinator to hive2 actions [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525811 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [17:47:00] (03CR) 10Nuria: [V: 03+2 C: 03+2] mediacounts: move archive and load oozie coord to hive2 actions [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525813 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [17:48:36] (03CR) 10Nuria: [C: 03+2] data_quality: move the oozie coordinator to hive2 actions [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525809 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [17:48:38] (03CR) 10Nuria: [V: 03+2 C: 03+2] data_quality: move the oozie coordinator to hive2 actions [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525809 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [17:49:15] (03CR) 10Nuria: [V: 03+2 C: 03+2] edit-hourly: move oozie coordinator to hive2 actions [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525810 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [18:30:57] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10MediaWiki-JobQueue, and 2 others: Migrate JobQueue to eventgate - https://phabricator.wikimedia.org/T228705 (10WDoranWMF) [18:31:01] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10ORES, and 4 others: Fix "Must provide the 'topic' parameter" in ORES /precache endpoint - https://phabricator.wikimedia.org/T228689 (10WDoranWMF) [18:31:06] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Services (watching): Fix revision-score event production in change-prop after migration of revision-create to eventgate-main - https://phabricator.wikimedia.org/T228688 (10WDoranWMF) [18:31:21] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), and 4 others: Modern Event Platform: Stream Intake Service: Migrate eventlogging-service-eventbus events to eventgate-main - https://phabricator.wikimedia.org/T211248 (10WDoranWMF) [18:31:24] 10Analytics, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Wikimedia-production-error: EventBus rejecting events because of malformed characters in the comment - https://phabricator.wikimedia.org/T184698 (10WDoranWMF) [18:31:34] 10Analytics, 10Core Platform Team, 10MediaWiki-API, 10CPT Initiatives (Modern Event Platform (TEC2)), and 2 others: Run ETL for wmf_raw.ActionApi into wmf.action_* aggregate tables - https://phabricator.wikimedia.org/T137321 (10WDoranWMF) [18:31:35] 10Analytics, 10EventBus, 10Product-Analytics, 10CPT Initiatives (Modern Event Platform (TEC2)): Eventbus revisions are duplicated in event.mediawiki_revision_tags_change - https://phabricator.wikimedia.org/T218246 (10WDoranWMF) [18:31:40] 10Analytics, 10MediaWiki-extensions-ORES, 10Scoring-platform-team, 10CPT Initiatives (Modern Event Platform (TEC2)), and 3 others: ORES hook integration with EventBus - https://phabricator.wikimedia.org/T201869 (10WDoranWMF) [18:31:44] 10Analytics, 10CPT Initiatives (Modern Event Platform (TEC2)): Ingest api data (for posts) into druid - https://phabricator.wikimedia.org/T218348 (10WDoranWMF) [18:31:50] 10Analytics, 10EventBus, 10Operations, 10CPT Initiatives (Modern Event Platform (TEC2)), and 2 others: Possibly expand Kafka main-{eqiad,codfw} clusters in Q4 2019. - https://phabricator.wikimedia.org/T217359 (10WDoranWMF) [18:31:54] 10Analytics, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Services (next): EventBusRCFeedFormatter should clean up events from nulls - https://phabricator.wikimedia.org/T216567 (10WDoranWMF) [18:32:00] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), and 2 others: Modern Event Platform: Stream Connectors - https://phabricator.wikimedia.org/T214430 (10WDoranWMF) [18:32:13] 10Analytics, 10Analytics-EventLogging, 10Better Use Of Data, 10EventBus, and 5 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10WDoranWMF) [18:32:16] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Core Platform Team Workboards (Clinic Duty Team): Develop a library for JSON schema backwards incompatibility detection - https://phabricator.wikimedia.org/T206889 (10WDoranWMF) [18:32:22] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 2 others: Make it possible to use $ref in JSONSchemas - https://phabricator.wikimedia.org/T206824 (10WDoranWMF) [18:32:26] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Services (watching): CI Support for Schema Registry - https://phabricator.wikimedia.org/T206814 (10WDoranWMF) [18:32:32] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Services (watching): Modern Event Platform: Schema Registry: Implementation - https://phabricator.wikimedia.org/T206789 (10WDoranWMF) [18:32:40] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), and 2 others: Modern Event Platform: Stream Configuration - https://phabricator.wikimedia.org/T205319 (10WDoranWMF) [18:32:47] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), and 2 others: Modern Event Platform (TEC2) - https://phabricator.wikimedia.org/T185233 (10WDoranWMF) [18:32:52] 10Analytics, 10EventBus, 10Multi-Content-Revisions, 10CPT Initiatives (Modern Event Platform (TEC2)): Redesign revision-related event schemas for MCR - https://phabricator.wikimedia.org/T186371 (10WDoranWMF) [18:33:02] 10Analytics, 10MediaWiki-API, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Patch-For-Review, 10User-Addshore: Run ETL for wmf_raw.ActionApi into wmf.action_* aggregate tables - https://phabricator.wikimedia.org/T137321 (10WDoranWMF) [18:35:37] 10Analytics, 10Android-app-Bugs, 10Wikipedia-Android-App-Backlog: App requests classified as pageviews that probably should not be so - https://phabricator.wikimedia.org/T229068 (10cooltey) Noticed that the logs were from a very old version `2.7.225-r-2018-02-06` I am guessing that they were because when th... [19:04:01] 10Analytics, 10Android-app-Bugs, 10Wikipedia-Android-App-Backlog: App requests classified as pageviews that probably should not be so - https://phabricator.wikimedia.org/T229068 (10Nuria) @cooltey let me make sure i understand your request, the app version might be old but these are recent logs from july 1st. [19:13:40] 10Analytics, 10Android-app-Bugs, 10Wikipedia-Android-App-Backlog: App requests classified as pageviews that probably should not be so - https://phabricator.wikimedia.org/T229068 (10Nuria) Also, there are a lot of requests that request the same content over and over which might indicate a problem, see three... [19:32:25] ebernhardson: yt? [19:34:55] 10Analytics, 10Multimedia, 10Tool-Pageviews: Statistics for views of individual Wikimedia images - https://phabricator.wikimedia.org/T210313 (10Abit) This looks very promising! I'd like to go over it with @SandraF_WMF when she gets back first week of August to consider the GLAM perspective. [19:38:03] 10Analytics, 10Multimedia, 10Tool-Pageviews: Statistics for views of individual Wikimedia images - https://phabricator.wikimedia.org/T210313 (10Ramsey-WMF) In addition to what Amanda mentions above, in regards to this little bit: > referer wiki, if available, otherwise "external", "internal" or "unknown" I... [19:54:10] nuria: [19:54:16] reading https://phabricator.wikimedia.org/T227896#5333961 again [19:54:23] it sounds like he does want to different types of events [19:54:36] he has a use for events per file [19:54:41] and a use case for one event per upload [19:56:27] ottomata:mmm, i think both are same , maybe he is thinking you can send events as "small" files are moved to swift but i do not think we should do that , maybe a meeting to clarify? [19:57:08] "The messages in this case should be a single kafka message with a swift prefix that returns the full directory upload." [19:57:36] nuria: maybe but i'm sure he'll get back to me about it [20:00:22] ottomata: it seems to me you can always have a directory on the event , and an optional file inside that directory, if file is present consumer can download file and if *ONLY* dir is present consumer downloads dir? does this sound incorrect? [20:00:40] ? [20:01:10] the upload always is from a directory. the directory has N files. [20:01:15] which create N objects in swift [20:01:16] ottomata: so for large uploads you would sent at the conclusion of the upload and event that contains the directory as well as every file that is to be downloaded independently [20:01:27] his uses cases are: [20:02:08] - 1 consumer downloads all files from a given swift upload job [20:02:08] - up to N consumers download all files from a given up swift upload job [20:02:19] in either case the events will be sent at conclusion of upload [20:02:49] ottomata: yes, case 1 can be marked by an event that includes all files or a directory [20:02:56] ottomata: case two would be N events [20:03:09] ottomata: each of which contains a directory and a file , right? [20:03:32] in either case the event has: [20:03:55] swift_object_prefix: /version/dir-prefix [20:03:55] swift_object_uris: [...] [20:04:22] but yes, i think that's what he and I are saying? [20:04:30] ottomata: i need events per file and events per upload. I don't need separate event types [20:04:31] i'm sayingg that it is different than waht we talked about in standup. [20:04:39] ottomata: they are the same event, one just happens to have a single url [20:04:58] ebernhardson: o/ [20:04:58] :) [20:05:03] ya, we were trying to make the event the same in both cases. [20:05:05] that is [20:05:08] ebernhardson: let's see [20:05:11] always emit an event per file/object [20:05:13] if we did that [20:05:15] always [20:05:17] would that work for you? [20:05:25] then i can't do promotion for multi-file [20:05:51] because you'd need to be sure to consume all events per object before you promote? [20:06:02] when importing multiple files with a promotion procedure i need a point in time when i can guarantee the import is done and ready to promote [20:06:14] 10Analytics, 10EventBus, 10phan: Result of EventFactory in EventBus extension is passed to undeclared arrays - https://phabricator.wikimedia.org/T224352 (10Umherirrender) 05Open→03Resolved a:03Umherirrender Fixed as part of https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/EventBus/+/513664/3/inc... [20:06:17] that is case 2 and 3 in your comment, ya? [20:06:27] in my case it happens that the things with promotion procedures are fast, and the thing thats slow has no promotion [20:06:31] whereas case 1 it doesn't matter, you can import a single object at any time? [20:06:52] ottomata, ebernhardson you want to talk about this in batcave? [20:06:58] ya sure [20:06:58] with 2 and 3 they need promotion, yes. With 1 it doesn't matter, it writes directly to a live index [20:07:06] sure, lemme grab headphones [20:08:36] have a bat-cave url? [20:09:08] https://meet.google.com/rxb-bjxn-nip [20:38:15] ebernhardson: got an estimate for max # of events we mgiht be emitting at once? [20:38:20] 1000? [20:38:38] for an upload that wants an event per file/object? [20:46:42] ottomata: 500 or so [20:46:59] call it 1k, close enoug [20:52:39] ok cool that's fine i think. [21:26:01] (03PS4) 10Ottomata: [WIP] swift-upload.py to handle upload and event emitting [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525435 (https://phabricator.wikimedia.org/T227896) [21:51:37] 10Analytics, 10Product-Analytics, 10Readers-Web-Backlog: Reading_depth remove eventlogging instrumentation? - https://phabricator.wikimedia.org/T229042 (10Groceryheist) I do not think we should be in a rush to remove this instrumentation. Rather I think this data is useful in general for learning about Wik... [21:53:54] 10Analytics, 10Operations, 10SRE-Access-Requests: Access to HUE for Mayakpwiki - https://phabricator.wikimedia.org/T229143 (10Peachey88) [21:54:03] 10Analytics, 10Android-app-Bugs, 10Wikipedia-Android-App-Backlog: App requests classified as pageviews that probably should not be so - https://phabricator.wikimedia.org/T229068 (10cooltey) Thanks @Nuria for providing the requests logs from a newer version of the release. (`2.7.50282-r-2019-05-24`) I believ... [21:54:49] 10Analytics, 10Operations, 10SRE-Access-Requests: Access to HUE for Mayakpwiki - https://phabricator.wikimedia.org/T229143 (10Peachey88) [22:09:51] hi, is there a hue admin around? [22:10:25] mayakpwiki would need one to give her access (https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Access#Admin_Instructions_to_sync_a_Hue_account) [22:10:55] the request ticket is https://phabricator.wikimedia.org/T229143 [22:11:30] i was first wondering which LDAP group this is and thought i could deal with it ..but then there is that sync button and i dont have hue access myself [22:19:28] mutante: given that this task was filed today it can wait until next Monday [22:20:12] mutante: all these require shell access to prod [22:20:35] mutante: after there are two ldap groups wmf (for employees) and nda; other-nda holders [22:21:02] nuria: i was under the impression the whole nda group thing was already done and just needed the click. nevermind then [22:21:23] mayakpwiki: it will be handled on Monday [22:21:57] nuria: ack, it's always a bit unclear in which of the groups a contractor falls. i will let the clinic duty person handle that [22:22:02] mutante: if she does not have access to hue, she was not added to ldap group [22:22:54] nuria: based on "also didn't work for me" and my assumptions because most services are "wmf/ops/nda" [22:23:07] gotcha now [22:23:49] 10Analytics, 10Operations, 10SRE-Access-Requests: Access to HUE for Mayakpwiki - https://phabricator.wikimedia.org/T229143 (10Nuria) @Mayakp.wiki the nda group will give you access to hue, best place to do your work is probably jupyter notebooks as they are intended as a repository of queries and work to sh... [22:26:35] Thanks Nuria and Mutante. I have been added to NDA group as per my earlier request and have access to Jupyter notebook and can ssh to the required databases. Ok to wait for this until Monday. Thanks! [22:27:18] mayakpwiki: hue has no more functionality than jupyter to do queries, so you are aware