[00:01:43] RECOVERY - eventbus grafana alert on icinga1001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [00:06:49] PROBLEM - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [00:09:21] RECOVERY - eventbus grafana alert on icinga1001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [00:15:04] (03PS1) 10GoranSMilovanovic: Wikitech [analytics/wmde/WD/WD_identifierLandscape] - 10https://gerrit.wikimedia.org/r/502363 [00:15:20] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] Wikitech [analytics/wmde/WD/WD_identifierLandscape] - 10https://gerrit.wikimedia.org/r/502363 (owner: 10GoranSMilovanovic) [00:17:03] PROBLEM - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [00:18:21] RECOVERY - eventbus grafana alert on icinga1001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [00:29:51] PROBLEM - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [00:32:09] 10Analytics, 10ChangeProp, 10Community-Tech, 10EventBus, and 6 others: Provide the ability to have time-delayed or time-offset jobs in the job queue - https://phabricator.wikimedia.org/T218812 (10Krinkle) [00:32:23] RECOVERY - eventbus grafana alert on icinga1001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [00:36:02] 10Analytics, 10ChangeProp, 10Community-Tech, 10EventBus, and 6 others: Provide the ability to have time-delayed or time-offset jobs in the job queue - https://phabricator.wikimedia.org/T218812 (10Krinkle) Tracking on the RFC board. As Daniel mentioned, it's not yet in the stage where it's seeking input or... [00:38:40] 10Analytics, 10ExternalGuidance, 10Release-Engineering-Team, 10MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), and 3 others: Unknown dependency: mw.externalguidance.init - https://phabricator.wikimedia.org/T219841 (10Jdlrobson) cc analytics and RelEng as there's potential for a lot of traffic to EventLogging (I... [01:12:11] PROBLEM - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [01:28:55] RECOVERY - eventbus grafana alert on icinga1001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [01:43:01] PROBLEM - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [02:02:29] RECOVERY - eventbus grafana alert on icinga1001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [02:14:05] PROBLEM - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [02:28:13] RECOVERY - eventbus grafana alert on icinga1001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [02:33:19] PROBLEM - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [02:41:50] 10Analytics, 10ExternalGuidance, 10Release-Engineering-Team, 10MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), and 3 others: Unknown dependency: mw.externalguidance.init - https://phabricator.wikimedia.org/T219841 (10Krinkle) p:05Unbreak!→03High [02:57:49] RECOVERY - eventbus grafana alert on icinga1001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [03:00:04] 10Analytics, 10Language-strategy, 10Tool-Pageviews: Have a way to show the most popular pages per country - https://phabricator.wikimedia.org/T207171 (10Tbayer) >>! In T207171#5056374, @Nuria wrote: >>Is this also an issue for the topviews that are shown per language? > Yes, it is an issue with any top list.... [03:02:57] PROBLEM - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [03:19:37] RECOVERY - eventbus grafana alert on icinga1001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [03:28:37] PROBLEM - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [03:36:13] RECOVERY - eventbus grafana alert on icinga1001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [03:41:19] PROBLEM - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [04:10:59] 10Analytics, 10ExternalGuidance, 10Release-Engineering-Team, 10MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), and 3 others: Unknown dependency: mw.externalguidance.init - https://phabricator.wikimedia.org/T219841 (10Jdlrobson) Thank you @krinkle for taking care of this. [04:39:51] 10Analytics, 10ExternalGuidance, 10Release-Engineering-Team, 10MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), and 3 others: Unknown dependency: mw.externalguidance.init - https://phabricator.wikimedia.org/T219841 (10santhosh) Thanks @Jdlrobson and @Krinkle. [04:44:21] RECOVERY - eventbus grafana alert on icinga1001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [04:49:31] PROBLEM - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [05:43:39] RECOVERY - eventbus grafana alert on icinga1001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [05:54:33] morning! [05:55:21] PROBLEM - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [06:00:31] RECOVERY - eventbus grafana alert on icinga1001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [06:12:05] PROBLEM - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [06:42:29] Failed processing event: Failed validating at path rev_id: . {} is not of type 'integer' [06:42:35] this is kinda new [06:42:49] it is probably known to Services [06:42:50] mmmm [07:00:54] ok alerted sre and services [07:01:15] I can see occurrences on April 03/04 so probably not super pressing [07:01:29] but all these alarms have happened after .wmf24 got deployed afaics [07:01:56] anyway, probably this alert needs to be fully transitioned away from analytics? [07:08:39] RECOVERY - eventbus grafana alert on icinga1001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [07:14:59] PROBLEM - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [07:28:27] brb [07:40:38] Morning elukey [07:41:08] elukey: all the above eventbus alerts are related to wrong schema? [07:42:19] joal: bonjour [07:42:22] yep afaics [07:44:51] ok [08:34:33] RECOVERY - eventbus grafana alert on icinga1001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [08:39:39] PROBLEM - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [08:42:13] RECOVERY - eventbus grafana alert on icinga1001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [08:47:21] PROBLEM - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [08:57:44] 10Analytics, 10EventBus, 10Operations: Eventbus errors: Failed processing event: Failed validating at path rev_id - https://phabricator.wikimedia.org/T220477 (10elukey) p:05Triage→03High [08:57:53] opened --^ [08:58:31] 10Analytics, 10EventBus, 10Operations, 10Services: Eventbus errors: Failed processing event: Failed validating at path rev_id - https://phabricator.wikimedia.org/T220477 (10elukey) [08:59:40] ACKNOWLEDGEMENT - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. Elukey https://phabricator.wikimedia.org/T220477 https://wikitech.wikimedia.org/wiki/EventBus [09:29:54] (03PS3) 10Joal: Update mediawiki-history per-page and per-editor [analytics/aqs] - 10https://gerrit.wikimedia.org/r/502198 (https://phabricator.wikimedia.org/T219910) [09:30:31] (03CR) 10Joal: Update mediawiki-history per-page and per-editor (031 comment) [analytics/aqs] - 10https://gerrit.wikimedia.org/r/502198 (https://phabricator.wikimedia.org/T219910) (owner: 10Joal) [09:30:49] (03CR) 10jerkins-bot: [V: 04-1] Update mediawiki-history per-page and per-editor [analytics/aqs] - 10https://gerrit.wikimedia.org/r/502198 (https://phabricator.wikimedia.org/T219910) (owner: 10Joal) [09:30:58] WAT? [09:31:25] Oh crap ... [09:31:30] sorry fixing [09:33:32] (03PS4) 10Joal: Update mediawiki-history per-page and per-editor [analytics/aqs] - 10https://gerrit.wikimedia.org/r/502198 (https://phabricator.wikimedia.org/T219910) [09:33:38] Hi everyone, is Wikistats 2.0 actively maintained? [09:33:58] Hi Urbanecm - It is [09:34:29] Urbanecm: you could even say 'under active development' :) [09:34:38] that's great joal :) [09:34:55] WMCZ might bring few ideas about things that would be good (for WMCZ) to include [09:36:26] Urbanecm: Please file tickets with projects 'Analytics' and 'Analytics-Wikistats' :) [09:36:51] Will do, thanks! [09:38:14] Just a quick question, it looks it isn't possible to view Top edited pages from January 2018, for instance. Or am I just bind? [09:39:29] Urbanecm: you can see 'top' (edited pages, contributors...) month by month [09:40:09] Urbanecm: there also is a limitation in the UI currently, allowing you to vizualize only last month (not previous ones), but data is available in the backend API [09:41:06] joal, aha, so the data is available, but not displayed in the UI. Do we have a task about it? [09:41:37] Urbanecm: There is an ongoing task of changing the time-selector, and make it usable in top views yes :) [09:41:49] joal, great. Do you have a link at hand? [09:42:06] nope Urbanecm - will look for it [09:42:14] thanks a lot joal [09:43:19] Urbanecm: First, the data - Sorry, it's not user-friendly readable, but at least you'll have it [09:43:22] Urbanecm: https://wikimedia.org/api/rest_v1/metrics/edited-pages/top-by-edits/cs.wikipedia.org/all-editor-types/all-page-types/2019/01/all-days [09:43:57] Urbanecm: and the task - https://phabricator.wikimedia.org/T219112 [09:44:03] thanks a lot joal [09:44:33] Urbanecm: You can add a comment on the task to confirm it will make time selection available in top pages (I'm not so good at UI, more on the backend side :) [09:44:48] same here :D [09:56:02] RECOVERY - eventbus grafana alert on icinga1001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [09:56:40] 10Analytics, 10Analytics-Wikistats: Add "Top large articles" - https://phabricator.wikimedia.org/T220482 (10Urbanecm) [09:57:01] 10Analytics, 10Analytics-Wikistats: Add "Top large articles" metric¨ - https://phabricator.wikimedia.org/T220482 (10Urbanecm) [09:57:06] 10Analytics, 10Operations, 10Research-management, 10Patch-For-Review, 10User-Elukey: Remove computational bottlenecks in stats machine via adding a GPU that can be used to train ML models - https://phabricator.wikimedia.org/T148843 (10MoritzMuehlenhoff) >>! In T148843#5090494, @elukey wrote: > The https:... [09:57:08] 10Analytics, 10Analytics-Wikistats: Add "Top large articles" metric - https://phabricator.wikimedia.org/T220482 (10Urbanecm) [09:59:02] 10Analytics, 10Analytics-Wikistats: Add "Number of stub articles" metric - https://phabricator.wikimedia.org/T220483 (10Urbanecm) [10:00:57] 10Analytics, 10Analytics-Wikistats: Add "Top linked article" metric - https://phabricator.wikimedia.org/T220484 (10Urbanecm) [10:01:44] PROBLEM - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [10:02:16] so the ack doesn't really work as I imagined [10:02:24] maybe I can simply disable it [10:05:09] 10Analytics, 10Analytics-Wikistats: Add "Top used photos" metric - https://phabricator.wikimedia.org/T220485 (10Urbanecm) [10:36:58] fdans: o/ if the cr is ready I can merge it (puppet) [10:53:18] RECOVERY - eventbus grafana alert on icinga1001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [10:58:20] PROBLEM - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [11:23:34] (03PS1) 10Joal: Refactor python util.py into smaller files [analytics/refinery] - 10https://gerrit.wikimedia.org/r/502469 (https://phabricator.wikimedia.org/T220111) [11:27:05] (03PS2) 10Joal: Refactor python util.py into smaller files [analytics/refinery] - 10https://gerrit.wikimedia.org/r/502469 (https://phabricator.wikimedia.org/T220111) [11:29:09] elukey: it’s ready! [11:55:05] fdans: all right merging.. do you have time after that to verify that everything is right? [12:07:11] elukey: just back from lunch, yes! [12:25:36] fdans: done! [12:25:40] you can check on an-coord1001 [12:26:05] elukey: thank youuuuu [12:43:38] 10Analytics, 10Operations, 10Research-management, 10Patch-For-Review, 10User-Elukey: Remove computational bottlenecks in stats machine via adding a GPU that can be used to train ML models - https://phabricator.wikimedia.org/T148843 (10elukey) After some tests with Moritz we did the following: * install... [13:03:05] RECOVERY - eventbus grafana alert on icinga1001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [13:05:04] Hey all! [13:05:19] Do we install piwik on any of our wiki's? [13:06:14] 10Analytics, 10Operations, 10Research-management, 10Patch-For-Review, 10User-Elukey: Remove computational bottlenecks in stats machine via adding a GPU that can be used to train ML models - https://phabricator.wikimedia.org/T148843 (10elukey) Ok I found a simple and hacky way to test the removal of : `... [13:06:27] or matomo and its known these days [13:06:34] as* [13:07:55] PROBLEM - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [13:09:09] RECOVERY - eventbus grafana alert on icinga1001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [13:16:11] hey team :] [13:16:49] PROBLEM - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [13:17:21] mforns: o/ [13:17:34] forgot to send an email but I created https://phabricator.wikimedia.org/T220477 for all the eventbus alerts [13:17:45] going afk for ~1h :) [13:17:59] ok [13:26:27] RECOVERY - eventbus grafana alert on icinga1001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [13:30:57] PROBLEM - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [13:33:07] (03CR) 10Ottomata: "For it! Maybe we can just drop the 'utils' part of these file/module names? E.g. python/refinery/hdfs.py seems fine to me." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/502469 (https://phabricator.wikimedia.org/T220111) (owner: 10Joal) [13:33:16] (03CR) 10Ottomata: [C: 03+1] "Other than that nit +1" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/502469 (https://phabricator.wikimedia.org/T220111) (owner: 10Joal) [13:40:35] RECOVERY - eventbus grafana alert on icinga1001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [13:42:30] (03PS7) 10Mforns: Add edit_hourly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/501197 (https://phabricator.wikimedia.org/T220092) [13:42:43] (03PS8) 10Mforns: Add edit_hourly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/501197 (https://phabricator.wikimedia.org/T220092) [13:43:13] (03PS4) 10Mforns: Add edit_hourly to list of tables to be purged of old snapshots [analytics/refinery] - 10https://gerrit.wikimedia.org/r/501328 (https://phabricator.wikimedia.org/T220092) [13:43:29] (03PS4) 10Mforns: Add oozie job to load edit_hourly to druid [analytics/refinery] - 10https://gerrit.wikimedia.org/r/501607 (https://phabricator.wikimedia.org/T211173) [13:45:43] PROBLEM - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [13:48:43] joal: do you feel like we had agreement yesterday on start_timestamp for first events [13:48:44] ? [13:48:56] or shall I open a discussion on phab? [13:49:54] milimetric: I felt there was a wooly consensus - Did you feel similarly?\ [13:50:51] IIRC we agreed on MIN(create, firstEdit) for users, and create for pages, but I didn't feel there was a strong enthusiasm :) [13:51:12] haha, joal no, I didn't think so, because I thought it'd be easier to just see the fields laid out so the choice could be clearer [13:51:20] Maybe strong enthusiasm is too much of an expectation given the topic :) [13:51:35] :) ok, I'll open a discussion with that on phab [13:53:13] !log restarted turnilo to clear deleted datasource [13:53:15] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:56:47] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Services (watching): EventGate should extract event time from events and produce to kafka with timestamp - https://phabricator.wikimedia.org/T219513 (10Ottomata) [14:03:54] (03PS3) 10Joal: Refactor python util.py into smaller files [analytics/refinery] - 10https://gerrit.wikimedia.org/r/502469 (https://phabricator.wikimedia.org/T220111) [14:05:02] (03CR) 10Joal: "I actually wanted to go for the short name, but did not in order to keep smaller difference. Since @ottomata suggests it, smaliffying name" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/502469 (https://phabricator.wikimedia.org/T220111) (owner: 10Joal) [14:07:40] 10Analytics: Decide: start_timestamp for mediawiki history - https://phabricator.wikimedia.org/T220507 (10Milimetric) [14:13:36] joal: so wait for pages you want to leave page_creation_timestamp null for all pages that don't have a create event? [14:14:35] milimetric: tha'ts how I recall Tilman's point: to make a difference for pages, since their timeline is not in coherence with the one of hte edits it holds [14:15:07] joal: ok, and populate page_first_edit_timestamp (not there on the schema yet) [14:15:35] correct milimetric - firstEditTimestamp is in the page-refactor (not yet ready) [14:19:06] 10Analytics: Decide: start_timestamp for mediawiki history - https://phabricator.wikimedia.org/T220507 (10Milimetric) [14:19:26] ok joal feel free to edit ^, I feel like I didn't explain very well this time [14:21:22] milimetric: I think users shouldn't be created after they make an edit - it is an error-case IMO - should I update? [14:21:51] joal: are there other examples of when create happens after edit? I was just trying to explain [14:22:00] yeah, of course edit, feel free [14:24:28] (03CR) 10Milimetric: "comment nit, looks good" (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/502469 (https://phabricator.wikimedia.org/T220111) (owner: 10Joal) [14:27:23] 10Analytics, 10Analytics-Kanban, 10Better Use Of Data, 10Product-Analytics, 10Patch-For-Review: "Edit" equivalent of pageviews daily available to use in Turnilo and Superset - https://phabricator.wikimedia.org/T211173 (10mforns) This is corrected now. See: https://turnilo.wikimedia.org/#edits_hourly [14:28:47] (03PS4) 10Joal: Refactor python util.py into smaller files [analytics/refinery] - 10https://gerrit.wikimedia.org/r/502469 (https://phabricator.wikimedia.org/T220111) [14:28:57] (03CR) 10Joal: Refactor python util.py into smaller files (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/502469 (https://phabricator.wikimedia.org/T220111) (owner: 10Joal) [14:36:47] 10Analytics: Decide: start_timestamp for mediawiki history - https://phabricator.wikimedia.org/T220507 (10JAllemandou) [14:37:02] milimetric: updated the text - hopefully for good? [14:37:56] (03CR) 10Milimetric: [C: 03+2] Update mediawiki-history per-page and per-editor [analytics/aqs] - 10https://gerrit.wikimedia.org/r/502198 (https://phabricator.wikimedia.org/T219910) (owner: 10Joal) [14:41:54] (03CR) 10Milimetric: "ok, tested, looks great, just the comments from before" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/498002 (owner: 10Fdans) [14:43:18] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Services (watching): EventGate should extract event time from events and produce to kafka with timestamp - https://phabricator.wikimedia.org/T219513 (10Ottomata) @Pchelolo q for ya. Once we start using dt for Kafka message timestamp, the test/event examples f... [15:00:04] !log backfilling data between previous backfill end and start of puppetized job for PrefUpdate [15:00:05] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:02:04] 10Analytics, 10EventBus, 10Growth-Team, 10Internet-Archive, and 3 others: Edits to Flow pages result in a page-links-change event with no performer - https://phabricator.wikimedia.org/T216726 (10bmansurov) a:05bmansurov→03None No bandwidth to work on this task. [15:03:11] 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey: Archival of home directories on servers with very large homes - https://phabricator.wikimedia.org/T215171 (10Milimetric) a:03elukey [15:03:53] 10Analytics: Set a timeout for regex parsing in the Eventlogging processors - https://phabricator.wikimedia.org/T200760 (10Milimetric) [15:12:17] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Ingest data from PrefUpdate EventLogging schema into Druid - https://phabricator.wikimedia.org/T218964 (10fdans) Confirmed that all events available for this schema have been loaded into druid and the job is loading the data periodic... [15:13:51] RECOVERY - eventbus grafana alert on icinga1001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [15:18:07] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Event counts from Mysql and Hive don't match. Refine is persisting data from crawlers. - https://phabricator.wikimedia.org/T210006 (10Milimetric) @Nuria & @Tbayer, I looked into this a bit. The is_bot flag is added to the parsed user agent, so all EL eve... [15:18:10] (03Abandoned) 10Joal: [WIP] update script dropping old druid datasources [analytics/refinery] - 10https://gerrit.wikimedia.org/r/501551 (owner: 10Joal) [15:19:00] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Event counts from Mysql and Hive don't match. Refine is persisting data from crawlers. - https://phabricator.wikimedia.org/T210006 (10Milimetric) a:03Milimetric [15:19:45] (03PS5) 10Joal: Refactor python util.py into smaller files [analytics/refinery] - 10https://gerrit.wikimedia.org/r/502469 (https://phabricator.wikimedia.org/T220111) [15:19:50] a-team: reminder that retro today comes before standup, so in 10 minutes [15:20:01] aye danke! [15:20:03] a-team: to the etherpad! [15:20:06] ottomata: hiiiiii [15:20:21] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Event counts from Mysql and Hive don't match. Refine is persisting data from crawlers. - https://phabricator.wikimedia.org/T210006 (10Nuria) >we made the mysql filter before EventLogging refine -> Hadoop was operational, so we forgot to apply the filter t... [15:20:28] I have a couple of code reviews for you [15:20:31] whenever you have time [15:21:07] https://gerrit.wikimedia.org/r/#/c/501578/ [15:21:19] https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/502233/ [15:21:24] 10Analytics, 10Analytics-Kanban: Metric should not rotate if they are not available for the selected wiki - https://phabricator.wikimedia.org/T220083 (10Milimetric) a:03Milimetric [15:21:29] PROBLEM - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [15:21:32] https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/500453/ [15:23:46] (03PS6) 10Joal: Refactor python util.py into smaller files [analytics/refinery] - 10https://gerrit.wikimedia.org/r/502469 (https://phabricator.wikimedia.org/T220111) [15:27:37] ottomata: Let's talk on standup about those eventbus alarms if we can [15:27:44] a-team: nobody's filling out retro etherpad :) [15:27:53] we got 3 minutes! [15:28:00] oh [15:28:12] retro-filling is not the way? [15:28:17] better :) [15:30:29] RECOVERY - eventbus grafana alert on icinga1001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [15:31:27] ping milimetric, ottomata RERTOOOOO [15:32:49] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Services (watching): EventGate should extract event time from events and produce to kafka with timestamp - https://phabricator.wikimedia.org/T219513 (10Ottomata) Hm, I went ahead and modified scripts/post-events to support a `--dt-field` option. If given, the... [15:35:37] PROBLEM - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [15:56:13] RECOVERY - eventbus grafana alert on icinga1001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [16:03:53] PROBLEM - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [16:12:51] RECOVERY - eventbus grafana alert on icinga1001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. https://wikitech.wikimedia.org/wiki/EventBus [16:15:14] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Change permissions for daily traffic anomaly reports on stat1007 - https://phabricator.wikimedia.org/T219546 (10ssingh) @elukey: Sorry for the late reply. Yes, that's fine with me as I do plan on maintaining this and having control over all the scripts will ma... [16:19:15] PROBLEM - eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. https://wikitech.wikimedia.org/wiki/EventBus [16:27:07] just removed ourselves from --^ [16:27:18] when puppet runs on icinga we'll stop to receive these alerts [16:43:05] 10Analytics, 10EventBus, 10Operations, 10Patch-For-Review, 10Services (doing): Eventbus errors: Failed processing event: Failed validating at path rev_id - https://phabricator.wikimedia.org/T220477 (10Pchelolo) Sorry about that. Fixed by above patch. [16:43:35] sorry about the alert.. My fault not swatting a fix sooner [16:47:31] Pchelolo: removed analytics from the alert, we can't really do much anymore :( [16:47:38] me and andrew will check icinga and help in case needed [16:49:02] mforns: did waht I say about rows and aggregations make sense? [16:53:57] ottomata: Can i bring this patch to your attention? https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/500076/ [16:54:16] ottomata: it willremove couple schemas from being persisted to mysql [16:55:05] k! [16:55:24] oh nuria its ready, bearloga is ready? [16:55:51] ottomata: i think so chelsyx just merged https://gerrit.wikimedia.org/r/#/c/wikimedia/discovery/golden/+/499938/ [16:56:19] cc bearloga so he knows this is happening [16:58:32] ottomata: ah wait chelsyx says there are other couple patches [16:58:41] ottomata: we can merge once those two are merged [16:59:37] k [17:14:27] 10Analytics, 10EventBus, 10Operations, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), and 2 others: Eventbus errors: Failed processing event: Failed validating at path rev_id - https://phabricator.wikimedia.org/T220477 (10mobrovac) 05Open→03Resolved Patch merged, will g... [17:21:56] joal, yes, but I imagine that to do that, you need to know the possible values for each field, to be able to change it if necessary, no? [17:22:06] 10Analytics, 10Language-strategy, 10Tool-Pageviews: Have a way to show the most popular pages per country - https://phabricator.wikimedia.org/T207171 (10MusikAnimal) >>>Is this also an issue for the topviews that are shown per language? >> Yes, it is an issue with any top list. Now, topviews has a "spam" lis... [17:22:53] 10Analytics, 10Operations, 10Research-management, 10Patch-For-Review, 10User-Elukey: Remove computational bottlenecks in stats machine via adding a GPU that can be used to train ML models - https://phabricator.wikimedia.org/T148843 (10elukey) >>! In T148843#5095613, @dr0ptp4kt wrote: > Hi, I'm requesting... [17:35:51] gilles: o/ - if you have time, can you try to deploy your repository to stat1007 ? [17:36:05] sure, I'll try now [17:37:29] elukey: failed [17:37:38] 17:37:16 ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'performance/asoranking', '-g', 'default', 'fetch', '--refresh-config'] on stat1007.eqiad.wmnet returned [255]: sign_and_send_pubkey: signing failed: agent refused operation [17:37:38] Received disconnect from 2620:0:861:105:10:64:21:118 port 22:2: Too many authentication failures [17:40:02] gilles: I might need to restart the keyholder on deploy1001 then [17:42:24] gilles: mind to retry? [17:42:39] elukey: still the same [17:44:31] ah ok so signing failed: agent refused operation [17:44:43] this probably means that I need to restart the keyholder agent [17:44:56] but that means re-arming it with all the identities [17:57:33] ottomata: o/ I'm trying to create a directory for some reports on notebook1004 and getting the following: [17:57:41] bearloga@notebook1004:~$ mkdir /srv/published-datasets/wikipedia-android-app-reports [17:57:41] mkdir: cannot create directory ‘/srv/published-datasets/wikipedia-android-app-reports’: Read-only file system [17:57:49] whoa [17:57:50] hm [17:57:51] looking [17:57:54] gilles: doesn't work even after restart of the keyholder, sigh [17:58:16] indeed [17:59:12] bearloga: i cannot reproduce [17:59:55] ottomata: oh it just worked [18:00:16] ¯\_(ツ)_/¯ [18:01:22] ottomata: thanks for checking! I wonder if it's because the first time I tried it through Terminal tab created from Jupyter Lab, and now it worked because I tried it through SSH. [18:02:00] ¯\_(ツ)_/¯ indeed [18:05:17] gilles: this might be needed https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/502562 [18:05:20] :D [18:10:52] bearloga: hm yeah maybe terminal tab in jupyter won't let you create dirs outside of your homedir [18:10:58] i thinik that is correct [18:11:13] gilles: now it should work :) [18:11:31] elukey: it does, thanks [18:11:36] ottomata: makes sense! [18:11:53] gooooood! [18:12:16] nuria: array items? [18:15:11] * elukey off! [18:16:03] milimetric: do you know, do we need the ability to push publish from e.g. stat1007 to thorium for stats.wikimedia.aorg wikistats1 anymore? [18:16:16] will wikistats 1 be updated like it used to by ezachte? [18:16:30] there's a writeable rsync mount i'd like to remove [18:16:39] ottomata: no, the jobs are stopped, but there is one job that we need to keep running a little while longer [18:16:44] his pagecounts-ez job [18:16:53] hm, but that doesn't push to stats.wikimedia.org, does it? [18:16:58] luca re-enabled that after we realized we accidentally disabled it with the rest [18:17:02] that is for dumps.wm.org? [18:17:08] yeah, it's just for dumps [18:17:10] ok [18:17:24] i just want to remove the rsync config that allow sus to push to stats.wm.org from stat1007 [18:17:52] yeah, I think that should be safe, but we should make sure the pagecounts-ez job keeps working [18:17:57] k [18:18:19] (maybe it tries to push just pageview stuff to stats.wikimedia.org and if it fails it stops or something weird, I haven't looked at the code) [18:19:14] milimetric: jobs are on 1007? [18:19:27] yes [18:20:02] i'm pretty sure he pushed manually [18:20:03] joseph and luca re-enabled them last week, actually I'm not sure if they moved them, but they were running as erik's user cron [18:20:15] yeah, most likley [18:28:13] (03PS1) 10Ladsgroup: Remove Google+ [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/502570 [18:28:36] (03CR) 10jerkins-bot: [V: 04-1] Remove Google+ [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/502570 (owner: 10Ladsgroup) [18:32:24] ottomata: have meetings for next hour, let's touch base in 1.5 hrs or so? [18:34:21] sure [18:37:36] Hello a-team, is there an equivalent of `wmf_raw.mediawiki_private_cu_changes`in eventbus? Or other daily updated tables that contain useragent info of edits? [18:38:35] chelsyx: lemme take a look at mediawiki events, but I don't think so [18:40:10] !log chowning files in analytics.wm.org/datasets/archive/public-datasets/ as stats:wikidev [18:40:11] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:41:46] chelsyx: no, only the original in mediawiki, the cu_changes or recentchanges tables, how do you need to use it? [18:43:30] milimetric: I want to get the device type (iphone or ipad) of edits and aggregate them, but wmf_raw.mediawiki_private_cu_changes only update once a month [18:46:10] chelsyx: if you need just a few specific wikis, and just the aggregate of device type, you could run the query on the production replicas [18:47:25] chelsyx: is this a regular report or a one-time thing? [18:48:45] milimetric: I want to make it regular and for ios app edits on all wikis [18:50:39] chelsyx: ok, if you need it daily, then we'd have to sqoop that table daily. You can file a request, but there's nothing in place right now [18:50:41] elukey: still around? [18:50:46] oh no [18:50:49] see your 'off' above [18:50:50] nm! [18:51:34] (03PS2) 10Ladsgroup: Fix PHP CodeSniffer [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/501329 (owner: 10Hoo man) [18:51:53] (03CR) 10Ladsgroup: [C: 03+2] Fix PHP CodeSniffer [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/501329 (owner: 10Hoo man) [18:52:22] milimetric: ok. I will file a request, but it's not super important. Thanks! [18:52:46] (03Merged) 10jenkins-bot: Fix PHP CodeSniffer [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/501329 (owner: 10Hoo man) [18:53:10] 10Analytics, 10EventBus, 10Operations, 10vm-requests, and 3 others: Create schema[12]00[12] (schema.svc.{eqiad,codfw}.wmnet) - https://phabricator.wikimedia.org/T219556 (10Ottomata) @akosiaris, I'd love to do this sooner rather than later. It'd make some configuration/deployment stuff in the Hive/Hadoop w... [18:53:14] (03CR) 10Ladsgroup: "recheck" [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/502570 (owner: 10Ladsgroup) [18:53:50] (03CR) 10Ladsgroup: [C: 03+2] Remove Google+ [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/502570 (owner: 10Ladsgroup) [18:54:16] (03Merged) 10jenkins-bot: Remove Google+ [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/502570 (owner: 10Ladsgroup) [18:54:39] chelsyx: have you consider for apps to send events when edits happen cc milimetric [18:54:48] chelsyx: that seems it woudl be more effective [18:54:55] chelsyx: some edits are instrumented as part of EditAttemptStep, right? [18:55:01] oh, but not iOS ones, right [18:56:10] yeah, cu_changes would catch everything but be kind of imprecise, depending on parsing useragents, etc. Instrumentation could be much more targeted [18:58:51] nuria: iOS app trigger eventlogging only when users agree to share their usage info with us. Although we can configure the edit event to not restricted by this, we probably need more discussion. And it's not super important [19:07:52] milimetric: presumably one can use the 'ios app edit' change tag to identify the CU table entries coming from the app [19:09:28] HaeB: ah, yeah, that could work. It seems like this is not important but let's discuss if you have more use cases for this data [19:11:19] chelsyx: In this case events are a much better way to get the data you are interested. We want to move towards a system where we do not need to parse the entire dataset to get what you are interested on but rather to have targeted streams of events (cc ottomata ) [19:11:30] cc milimetric [19:11:44] darn tootin [19:15:25] hm, this one's got some nuance nuria [19:15:52] because it's between platform support and user privacy [19:16:34] event generation is limited by the "share data" agreement [19:17:20] so if the iOS team needs general metrics across the entire population, I'm not sure what the right answer is [19:17:31] *user population [19:17:35] milimetric: ya, i think we probably need to start by messaging ios users that although they did not want to share their data the edit data is public nonetheless [19:17:37] cc chelsyx [19:18:08] milimetric: i bet that is somewhat surprising for users [19:18:16] well, it's not public, it's available to checkusers I believe [19:18:32] the ip/useragent are pretty guarded [19:18:34] milimetric: no, the edits: user, page, time are public [19:18:59] nuria: right, but that's available on mediawiki.revision_create, and not what chelsy needs here [19:19:00] milimetric: regardless of the option you clicked at the start of the app [19:19:41] milimetric: right, i was thinking from a user standpoint probably users do not expect that info to be public at all [19:20:17] not sure, but in any case, they would be right to expect their IP and useragent to not be available if they didn't opt-in [19:20:44] milimetric: right [19:22:56] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Event counts from Mysql and Hive don't match. Refine is persisting data from crawlers. - https://phabricator.wikimedia.org/T210006 (10Milimetric) Hm, I thought I was onto something with the `is_bot` flag, because the mysql consumer filters out events when... [19:28:37] (03CR) 10Milimetric: "ping @fdans, thought you said you were going to push another patch here" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/498002 (owner: 10Fdans) [19:35:27] 10Analytics, 10Product-Analytics: Update R from 3.3.3 to 3.5.3 on stat and notebook machines - https://phabricator.wikimedia.org/T220542 (10mpopov) [19:35:44] 10Analytics, 10Product-Analytics: Update R from 3.3.3 to 3.5.3 on stat and notebook machines - https://phabricator.wikimedia.org/T220542 (10mpopov) [19:39:50] 10Analytics, 10ExternalGuidance, 10Release-Engineering-Team, 10MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), and 3 others: Unknown dependency: mw.externalguidance.init - https://phabricator.wikimedia.org/T219841 (10Jdlrobson) 05Open→03Resolved a:03Jdlrobson Errors are back to their normal level. Thanks... [19:40:29] hey ottomata, /usr/local/bin/published-datasets-sync -q is cronspamming from stat1004, 6 and 7, just started this evening... "rsync: chgrp "/stat1004/README" (in publshed-datasets-destination) failed: Operation not permitted (1)" in case you have any idea [19:40:44] uh oh [19:40:51] thanks, i was trying to test that and it seemed fine [19:41:01] i did just modify that job [19:41:02] looking... [19:42:07] it also whined from notebook1003 and 4 in case that's the same issue [19:42:17] ya same [19:46:03] ok, i think its because one of the files it is rsyncing is owned by root on the source, and i change the rsync module to run as stats user. i chowned the file to stats on the dest, but shouldn't have done it for tat one file. [19:46:20] the rsync completed this time on stat1004. I think it should do the same on all others now too [19:46:26] apergos let me know if you still see it [19:46:45] ok, will do! [20:03:17] stat1006 new error: rsync: chgrp "/stat1006/archive" (in publshed-datasets-destination) failed: Operation not permitted (1) rsync error: some files/attrs were not transferred [20:03:23] ottomata: ^^ [20:04:09] hm ok... [20:04:35] sorry... [20:06:23] oh [20:06:24] hm [20:07:02] ok better fix... [20:12:36] (03PS1) 10Milimetric: Filter out unavailable metrics on desktop [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/502599 (https://phabricator.wikimedia.org/T220083) [20:18:54] 10Analytics, 10Operations, 10Research-management, 10Patch-For-Review, 10User-Elukey: Remove computational bottlenecks in stats machine via adding a GPU that can be used to train ML models - https://phabricator.wikimedia.org/T148843 (10dr0ptp4kt) Thanks @elukey [20:19:10] rsync: chgrp "/stat1006/archive" (in publshed-datasets-destination) failed: Operation not permitted (1) a minute ago ottomata [20:21:19] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Discovery, and 4 others: Rewrite Avro schemas (ApiAction, CirrusSearchRequestSet) as JSONSchema and produce to EventGate - https://phabricator.wikimedia.org/T214080 (10Ottomata) Ok, I think https://gerrit.wikimedia.org/r/#/c/mediawiki/event-schem... [20:22:00] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: All Metrics link doesn't show up on mobile - https://phabricator.wikimedia.org/T219581 (10Milimetric) a:03fdans [20:23:43] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: First access to the detail page causes glitchy loading - https://phabricator.wikimedia.org/T220088 (10Milimetric) a:03Milimetric [20:56:07] milimetric: i'm not working with ios app edit data currently (chelsyx is), but in case you were asking whether i have use cases for cu_changes in connection with change tags, yes, i might have some soon - i understand https://phabricator.wikimedia.org/T161149 is on its way already though [20:57:23] HaeB: yes, but that'll still be monthly [20:57:46] nuria / milimetric: all editors (on the apps or elsewhere) are already asked to agree to have their IP+UA stored for a limited time, as per the privacy policy [20:58:04] HaeB: for the special case of cu_changes, we may be able to get data imported more regularly since it's an append-only table and there's a clear way to import incrementally [20:58:24] (speaking as a former community elected checkuser myself ;) [20:58:30] HaeB: good, then we're probably fine on any privacy concerns [20:58:41] monthly: right, i know [21:00:52] good news no more spam after that one! \o/ [21:04:54] greata! [21:31:28] (03PS1) 10Milimetric: [WIP] Need review/opinion on what Loading language should look like, for now I worked out the mechanism to display it when loading the detail page, but I could make it generic. I'm thinking plain text should be fine since we're talking about low bandwidth, but want to check with everyone else. [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/502623 (https://phabricator.wikimedia.org/T220088) [23:00:14] HaeB: does the app prompt you about that specifically? [23:10:18] HaeB: when you edit? [23:11:01] nuria: not sure. the android app points to the TOU before saving an edit [23:11:30] i would hope it was all reviewed by the legal team at some point [23:12:41] HaeB: ya, I think if there is language that users see that leads them to understand that their IP +UA is saved when they edit, despite of them having opted out of data collection chelsyx use case is better served by sending edit data from ios [23:23:00] BTW, wanted to say goodbye to everyone on the a-team as i'm leaving wmf today. truly appreciated your support over the past few years and all that you have achieved for the movement! my gmail address is haebwiki in case you want to reach out about anything [23:39:39] Is anyone about who understands the processes running on the notebook systems? [23:58:32] robh: not really, but I’m around. In general if something’s misbehaving badly we kill it [23:58:54] yeah i had to kill a user job for puppet but thats cool [23:59:03] when i saw the box is ALL user jobs [23:59:07] seems like something that just happens [23:59:40] ie: when there wasnt an answer after 5 minutes i just starting killing the jobs i needed to kill and all is good now