[00:19:23] (03CR) 10Chelsyx: Hash tokens from the EL Sanitization white-list for iOS app (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/520134 (https://phabricator.wikimedia.org/T226849) (owner: 10Chelsyx) [00:36:01] nuria: I missed your ping, the problem in T228187 was the .pid files I cleaned up so the reports could run. The dashboard didn't show up because the default is to only show the past month of data, and all of that was null due to reports not running [00:36:01] T228187: Pie charts not showing on "User Agent Breakdowns" dashboard - https://phabricator.wikimedia.org/T228187 [04:18:49] PROBLEM - Check the last execution of monitor_refine_sanitize_eventlogging_analytics_immediate on an-coord1001 is CRITICAL: CRITICAL: Status of the systemd unit monitor_refine_sanitize_eventlogging_analytics_immediate https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [04:25:01] PROBLEM - Check the last execution of monitor_refine_sanitize_eventlogging_analytics_delayed on an-coord1001 is CRITICAL: CRITICAL: Status of the systemd unit monitor_refine_sanitize_eventlogging_analytics_delayed https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [06:41:15] is https://phabricator.wikimedia.org/T226219 related to https://phabricator.wikimedia.org/T226219 ? [06:41:58] mmm not sure [07:27:23] it is still not clear to me what the monitor refine is complaining about [07:29:54] in theory the alert starts from https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/refine/RefineMonitor.scala#L222 [07:30:15] that is the result of https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/refine/RefineTarget.scala#L533 [07:31:46] I've re-run the mnitor refine sanitize, and now it complains about [07:31:47] `event_sanitized`.`MobileWikiAppDailyStats` (year=2019,month=7,day=18,hour=2) [07:32:57] elukey@an-coord1001:/mnt/hdfs/wmf/data$ ls -l event/MobileWikiAppDailyStats/year\=2019/month\=7/day\=18/hour\=2 [07:33:00] total 3239 [07:33:02] -rw-r--r-- 1 analytics analytics 1107178 Jul 19 02:58 part-00000-a32f8b00-b3b8-4f0d-b8ad-c19576d15585-c000.snappy.parquet [07:33:05] -rw-r--r-- 1 analytics analytics 1104953 Jul 19 02:58 part-00001-a32f8b00-b3b8-4f0d-b8ad-c19576d15585-c000.snappy.parquet [07:33:08] -rw-r--r-- 1 analytics analytics 1104713 Jul 19 02:58 part-00002-a32f8b00-b3b8-4f0d-b8ad-c19576d15585-c000.snappy.parquet [07:33:11] -rw-r--r-- 1 analytics analytics 26 Jul 19 02:58 _REFINED [07:33:14] -rw-r--r-- 1 analytics analytics 0 Jul 19 02:58 _SUCCESS [07:33:17] elukey@an-coord1001:/mnt/hdfs/wmf/data$ ls -l event_sanitized/MobileWikiAppDailyStats/year\=2019/month\=7/day\=18/hour\=2 [07:33:19] total 2575 [07:33:22] -rw-r--r-- 1 analytics analytics 878680 Jul 18 06:03 part-00000-d351559b-ca9b-4a78-8a88-f39087948763-c000.snappy.parquet [07:33:25] -rw-r--r-- 1 analytics analytics 878866 Jul 18 06:03 part-00001-d351559b-ca9b-4a78-8a88-f39087948763-c000.snappy.parquet [07:33:28] -rw-r--r-- 1 analytics analytics 8 [07:33:30] err the last one was truncated [07:33:33] elukey@an-coord1001:/mnt/hdfs/wmf/data$ ls -l event_sanitized/MobileWikiAppDailyStats/year\=2019/month\=7/day\=18/hour\=2 [07:33:36] total 2575 [07:33:38] -rw-r--r-- 1 analytics analytics 878680 Jul 18 06:03 part-00000-d351559b-ca9b-4a78-8a88-f39087948763-c000.snappy.parquet [07:33:41] -rw-r--r-- 1 analytics analytics 878866 Jul 18 06:03 part-00001-d351559b-ca9b-4a78-8a88-f39087948763-c000.snappy.parquet [07:33:45] -rw-r--r-- 1 analytics analytics 879115 Jul 18 06:03 part-00002-d351559b-ca9b-4a78-8a88-f39087948763-c000.snappy.parquet [07:33:47] -rw-r--r-- 1 analytics analytics 26 Jul 18 06:03 _REFINED [07:33:50] -rw-r--r-- 1 analytics analytics 0 Jul 18 06:03 _SUCCESS [07:36:15] just re-ran the job again, it doesn't complain anymore [07:36:57] RECOVERY - Check the last execution of monitor_refine_sanitize_eventlogging_analytics_delayed on an-coord1001 is OK: OK: Status of the systemd unit monitor_refine_sanitize_eventlogging_analytics_delayed https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [07:41:21] RECOVERY - Check the last execution of monitor_refine_sanitize_eventlogging_analytics_immediate on an-coord1001 is OK: OK: Status of the systemd unit monitor_refine_sanitize_eventlogging_analytics_immediate https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [10:34:20] * elukey lunch + errand! [12:55:13] elukey: helloooo I'm having permissions issues when using journalctl [12:55:17] https://www.irccloud.com/pastebin/QBkMoMev/ [12:56:34] fdans: o/ [12:56:38] try to use only sudo [12:57:31] elu thank youuuu al good [12:58:18] super :) [13:26:34] (03Abandoned) 10Awight: Schema for ORES scores [analytics/refinery] - 10https://gerrit.wikimedia.org/r/481025 (https://phabricator.wikimedia.org/T209732) (owner: 10Awight) [13:26:40] (03Abandoned) 10Awight: Oozie jobs to produce ORES data [analytics/refinery] - 10https://gerrit.wikimedia.org/r/482753 (https://phabricator.wikimedia.org/T209732) (owner: 10Awight) [14:26:54] (03CR) 10Nuria: "Chelsey to send another patch with user id removed and nuria to CR" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/520134 (https://phabricator.wikimedia.org/T226849) (owner: 10Chelsyx) [14:47:06] ottomata: hola,question if i may [14:47:28] ottomata: when you did run refine yesterday, did you also run refine monitor? [15:01:13] ottomata: sorry, sanitize? [15:03:31] (03PS1) 10Nuria: Making error message on refine monitor more precise [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/524536 (https://phabricator.wikimedia.org/T228522) [15:28:59] pondering but seems like the wrong tool...would superset reasonably analyze AB tests if you do all the pre-processing? I have a transformation that turns some event logging into a row per-session or per-search with various boolean flags. I can visualize this with a jupyter notebook, but was pondering if there would be a more automated way to setup the metrics reporting [15:51:47] metrics so far: https://people.wikimedia.org/~ebernhardson/dym_metrics.pdf [15:52:02] graphs are probability of mean value [15:52:24] wrong room ... :) [16:00:05] ebernhardson: or maybe RIGHT room! want to send jupyter notebook along? [16:01:22] nuria: notebook1004.eqiad.wmnet:~ebernhardson/DYM_metrics.ipynb [16:28:05] (03CR) 10Elukey: "Thanks a lot!" (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/524536 (https://phabricator.wikimedia.org/T228522) (owner: 10Nuria) [16:30:28] (03PS2) 10Nuria: Making error message on refine monitor more precise [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/524536 (https://phabricator.wikimedia.org/T228522) [16:31:17] (03CR) 10Elukey: [C: 03+1] Making error message on refine monitor more precise [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/524536 (https://phabricator.wikimedia.org/T228522) (owner: 10Nuria) [16:33:22] (03CR) 10Nuria: [C: 03+2] "Merging." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/524536 (https://phabricator.wikimedia.org/T228522) (owner: 10Nuria) [16:41:36] (03Merged) 10jenkins-bot: Making error message on refine monitor more precise [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/524536 (https://phabricator.wikimedia.org/T228522) (owner: 10Nuria) [16:42:27] * elukey off! [16:48:13] (03PS2) 10Chelsyx: Hash tokens from the EL Sanitization white-list for iOS app [analytics/refinery] - 10https://gerrit.wikimedia.org/r/520134 (https://phabricator.wikimedia.org/T226849) [17:09:24] (03CR) 10Nuria: Hash tokens from the EL Sanitization white-list for iOS app (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/520134 (https://phabricator.wikimedia.org/T226849) (owner: 10Chelsyx) [17:12:06] (03PS3) 10Chelsyx: Hash tokens from the EL Sanitization white-list for iOS app [analytics/refinery] - 10https://gerrit.wikimedia.org/r/520134 (https://phabricator.wikimedia.org/T226849) [17:59:10] (03CR) 10Nuria: Hash tokens from the EL Sanitization white-list for iOS app (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/520134 (https://phabricator.wikimedia.org/T226849) (owner: 10Chelsyx) [17:59:27] (03CR) 10Nuria: "Sorry, I did not make my comment about revision id before." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/520134 (https://phabricator.wikimedia.org/T226849) (owner: 10Chelsyx) [18:02:45] (03PS4) 10Chelsyx: Hash tokens from the EL Sanitization white-list for iOS app [analytics/refinery] - 10https://gerrit.wikimedia.org/r/520134 (https://phabricator.wikimedia.org/T226849) [18:06:46] (03CR) 10Nuria: [C: 03+2] Hash tokens from the EL Sanitization white-list for iOS app [analytics/refinery] - 10https://gerrit.wikimedia.org/r/520134 (https://phabricator.wikimedia.org/T226849) (owner: 10Chelsyx) [18:08:54] submitted WIP patches for Queue, still working on adding tests [18:09:06] have to run some errands and may be working offline from the house if I can't get reception [18:16:50] milimetric: NICE [18:33:43] (03CR) 10Nuria: "Some of pageview definition code is use from scala (see transform functions) so that would also need refactoring. And subsequent bump of " (036 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/523903 (https://phabricator.wikimedia.org/T228151) (owner: 10Fdans) [18:35:31] nuria: I thought I changed the scala code tho? [18:36:23] fdans: ah YES you did! [18:36:39] fdans: how did i missed that? was i looking at the 1st patch all along? [18:37:29] nuria: probably you were looking at ps2 [18:37:48] i changed scala code once refinery job tests exploded [18:37:59] fdans: k, ya that makes sense cause i though i had submitted review yesterday morning, sorry [20:38:05] any idea when we'll have buster on the cluster? [20:39:44] groceryheist: probably next quarter! [20:41:27] when does that start? [20:41:40] October? [20:47:28] (03CR) 10Ottomata: Making error message on refine monitor more precise (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/524536 (https://phabricator.wikimedia.org/T228522) (owner: 10Nuria) [20:57:30] ya [21:00:44] thanks