[04:38:38] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Upgrade turnilo to latest upstream - https://phabricator.wikimedia.org/T253294 (10JKatzWMF) @elukey Ahh, bummer! Thanks for looking into it. [06:44:21] good morning :) [07:02:20] Hello! [07:09:15] o- [07:09:17] o- [07:09:19] argh çD [07:09:25] ok o/ :D [07:09:34] wow - elukey --more-coffee [07:09:37] (italian keyboard layout fail) [07:10:01] \o [07:10:07] nono I discovered a way to keep multiple layouts for my keyboard and I was writing in italian, then #fail :D [07:10:12] bonjour [07:20:53] (03PS60) 10Fdans: Add pageview historical dumps backfilling job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/597541 (https://phabricator.wikimedia.org/T252857) [07:26:05] elukey, cdanis: I confirm the change on netflw has teh expected impact on data flowing to druid - No more late data (therefore no more issue indexing) - This is great :) [07:29:07] elukey: I had a question for you - But first some context - Every month I currently need to rerun dumps-to-avro jobs because of the bzip2 issue in the apache-compress lib [07:29:27] (03PS61) 10Fdans: Add pageview historical dumps backfilling job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/597541 (https://phabricator.wikimedia.org/T252857) [07:29:37] elukey: Now that the patch has been release, would it be feasible to manually bump the jar in the correct lib folder? [07:30:12] elukey: I understand if you tell me it's not really feasible - But if it is, it would reduce manual rerun for me :) [07:30:32] 10Analytics, 10Core Platform Team, 10Dumps-Generation: HTML Dumps - June/2020 - https://phabricator.wikimedia.org/T254275 (10Kelson) I can only emphasis that a ticket which does not transparently explain the problem which is tried to be solved is going to be successfuly only by chance. Therefore, this is pro... [07:31:20] joal: great for netflow! [07:31:29] joal: can you give me more info about the jar_ [07:31:32] ? [07:31:38] is it a hadoop jar? [07:31:54] elukey: sure! the original ticket is that one: https://phabricator.wikimedia.org/T243241 [07:33:10] so, the JAR IIUC is shipped by one of the cloudera packages, and we need an override [07:33:20] elukey: the jar is common-compress [07:34:12] correct elukey - I did: find /usr/lib -name *commons-compress* on stat1004 and there are some instances (sqoop, hadoop, spark, parquet, mahout, hive etc) [07:34:31] elukey: I guess this is not easy :( [07:34:41] joal: yes I was about to say "run dpkg -S commons-compress" on a worker node [07:34:51] that will tell you the packages that ship the jar [07:35:07] now, a clean solution would be to find the package and rebuild it with the jar [07:35:20] something that is currently a bit cumbersome, but it will be easier with BigTop [07:35:28] right [07:36:59] joal: in this case, do you know where the error comes from? [07:37:08] is it a spark job, another thing, etc.? [07:37:26] for example: spark2: /usr/lib/spark2/jars/commons-compress-1.8.1.jar [07:37:32] this one can be easily bumped [07:37:43] since we build the package [07:37:59] elukey: It is a spark job using a hadoop-reader - So I think it uses the hadoop-client one (but I can't be sure :(, difficult to test ) [07:38:11] elukey: could we do some testing on the test cluster? [07:39:18] 10Analytics, 10Core Platform Team, 10Dumps-Generation: HTML Dumps - June/2020 - https://phabricator.wikimedia.org/T254275 (10Dzahn) >>! In T254275#6200624, @Kelson wrote: > a ticket which does not transparently explain the problem +1, could a problem statement be added? > having read the document on offi... [07:42:15] joal: sure, but I am wondering if spark allows to just add a custom jar to spark-submit to override a default one [07:42:32] elukey: I tried that without success :( [07:42:33] like --jar some/local/path in spark-submit [07:42:55] what parameters did you pass? [07:42:58] (to understand) [07:43:06] elukey: --jars new-commons-compress jar [07:43:22] because I am reading https://hadoopsters.com/2019/05/08/how-to-override-a-spark-dependency-in-client-or-cluster-mode/ [07:43:29] that talks also about extraClassPath [07:43:32] for driver/executor [07:43:54] right - Will try that anew [07:44:17] super [07:44:26] that would be ideal if it worked [07:44:37] of course [08:16:30] (03PS62) 10Fdans: Add pageview historical dumps backfilling job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/597541 (https://phabricator.wikimedia.org/T252857) [08:37:33] matomo on buster seems to run fine! [08:37:39] \o/ [08:37:48] doing the last tests, then I'll ask people to test as well [08:37:53] and then we'll switch VM [08:38:01] same thing will need to happen for archiva [08:58:02] elukey: very fun finding :) [08:58:34] elukey: refinery-job jar references a scala version that prevents it to be added to spark with the spark.driver.extraClassPath option :) [09:00:53] joal: do you know why it prevents it? Do you get an error or another behavior? [09:01:08] elukey: java.lang.NumberFormatException: Not a version: 9 [09:01:34] elukey: this happens when there is a version mismatch in java (it thinks it's using 9 - Don't know why9 [09:02:20] joal: where did you get the jar? Was is built using java 9? [09:02:31] elukey: nope - built on stat1004 [09:03:54] joal: IIUC there is something that was built using java 9 and scala doesn't like it no? [09:04:00] correct elukey [09:04:28] now the thing is: which one!!! [09:04:43] the funny bit is that scala doesn't complain when passed using --jars [09:05:30] is there a stacktrace or something? [09:06:53] there is elukey, but it only talks about scala [09:07:30] elukey: https://gist.github.com/jobar/1d479046b0518a5fad2d15217ce68d30 [09:08:07] elukey: this feels weird (working with --jars and failing with class-path) [09:12:02] joal: what is the path of the jar? curious about something [09:12:18] elukey: /home/joal/code/refinery-source/refinery-job/target/refinery-job-0.0.126-SNAPSHOT.jar [09:12:23] on stat1004 elukey [09:21:51] joal: I am super ignorant about maven, but IIUC the refinery-job jar gets the commons-compress jar from archiva and then includes it into it.. Could it be possible that for some reason the commons-compress jar is built using java 9 when published to central? [09:21:54] elukey: I managed to test compress using a standalone jar (spark-local, using --jars and extraClassPath) - still failing [09:21:55] (total speculation) [09:22:31] elukey: unrelated to commons-compress - same issue without it (it's actually commented on the code) [09:22:55] elukey: don't bother too much, I was just sharing my fight :) [09:23:11] elukey: and as just said - no luck with spark options :( [09:23:46] joal: then I am not getting where the problem comes from, I thought you were testing common-compress override via spark [09:25:06] elukey: I'm testing with spark options, and it fails - In order for the test to work, I needed to create a dedicated jar containing commons-compress only, as refinery-job fails [09:27:40] joal: ok so this means that we usually don't pass refinery-job anywhere right? (I am not super familiar with all our jars this is why I need more details :) [09:28:06] and you discovered that adding it via spark extra jar causes the java 9 error [09:28:14] elukey: nope - it means we pass refinery-job using the --jars property, not using the classpath one [09:28:15] probably because there is a jar that we include built via java 9 [09:29:25] That's where I am as well elukey [09:30:01] That's where I am as well elukey : spark2-shell --master local --jars /home/joal/code/refinery-source/refinery-job/target/refinery-job-0.0.126-SNAPSHOT.jar --conf spark.driver.extraClassPath=refinery-job-0.0.126-SNAPSHOT.jar [09:30:08] oops sorry [09:30:34] elukey: I found a way --^ This doesn't solve the compress issue but it does solve the java9 issue [09:31:27] and the diff is that you add both options? [09:33:05] nope elukey - I added only the jar-name, not the full path in the classPath option [09:33:24] And it works as the jar is made available through --jars [09:33:32] bizarre nonetheless [09:33:32] (03PS63) 10Fdans: Add pageview historical dumps backfilling job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/597541 (https://phabricator.wikimedia.org/T252857) [09:34:13] joal: yep no idea, there must be a weird explanation for this, but can't think of any [09:34:25] it seems totally random [09:34:28] no prob elukey - Thanks for listening to my rants ) [09:35:03] joal: I am always willing to listen and help if I can but I'd need some details otherwise I keep asking questions :) [09:35:14] hehe :) [09:35:30] ok so package override in spark doesn't seem to work for compress :( [09:48:51] (03PS64) 10Fdans: Add pageview historical dumps backfilling job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/597541 (https://phabricator.wikimedia.org/T252857) [10:36:19] (03CR) 10Fdans: "Last comments by joal applied" (035 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/597541 (https://phabricator.wikimedia.org/T252857) (owner: 10Fdans) [10:39:06] * elukey lunch! [11:06:49] the pagecounts_ez re-loading project: https://usercontent.irccloud-cdn.com/file/W2WrulvP/EMeuuxFXUAElPlF.jpg [11:07:19] :) [11:15:22] (03CR) 10Joal: "I think we're close to the end with that - Thanks Fran for the dedicated commitment" (039 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/597541 (https://phabricator.wikimedia.org/T252857) (owner: 10Fdans) [11:16:27] Gone for dentist appointment - Meh :( [11:24:57] joal: we do need the last step of moving back the files to the original folder [11:25:13] otherwise they'll stay in their day ones [11:26:27] (03PS65) 10Fdans: Add pageview historical dumps backfilling job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/597541 (https://phabricator.wikimedia.org/T252857) [11:50:06] had to check the author ;) https://towardsdatascience.com/stop-using-pandas-and-start-using-spark-with-scala-f7364077c2e0 [12:49:52] 10Analytics, 10Analytics-EventLogging, 10Beta-Cluster-Infrastructure, 10Product-Analytics, 10Wikipedia-Android-App-Backlog: MobileWikiAppProtectedEditAttempt: 'protectionStatus' is a required property - https://phabricator.wikimedia.org/T254567 (10Charlotte) @Dbrant - If we are not using these events for... [12:50:31] 10Analytics, 10Analytics-EventLogging, 10Beta-Cluster-Infrastructure, 10Product-Analytics, 10Wikipedia-Android-App-Backlog: Remove MobileWikiAppProtectedEditAttempt schema from Android app - https://phabricator.wikimedia.org/T254567 (10Charlotte) p:05Triage→03Medium [12:50:48] 10Analytics, 10Analytics-EventLogging, 10Beta-Cluster-Infrastructure, 10Product-Analytics, 10Wikipedia-Android-App-Backlog (Android-app-release-v2.7.31x-P-Pryanik): Remove MobileWikiAppProtectedEditAttempt schema from Android app - https://phabricator.wikimedia.org/T254567 (10Charlotte) [13:26:23] !log stop timers on an-launcher to drain jobs and restart hive/oozie for openjdk upgrades [13:26:24] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:27:42] 10Analytics, 10Core Platform Team, 10Dumps-Generation: HTML Dumps - June/2020 - https://phabricator.wikimedia.org/T254275 (10Isaac) > Having it in HDFS first would allow it to be more easily used by internal WMF researchers and analysts. Speaking personally but from the Research team, I also +1 this many man... [13:36:55] 10Analytics, 10Analytics-Kanban, 10EventStreams, 10Operations, and 2 others: EventStreams drops the connection after 15 minutes, which makes it unreliable - https://phabricator.wikimedia.org/T242767 (10ema) 05Open→03Declined >>! In T242767#6199410, @MrJaroslavik wrote: > Hey, can be fixed this problem?... [13:41:03] (03CR) 10Fdans: "applied all except the comment to remove the move/delete fs part" (039 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/597541 (https://phabricator.wikimedia.org/T252857) (owner: 10Fdans) [13:42:36] !log roll restart kafka jumbo brokers for openjdk upgrades [13:42:40] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:48:59] brb [13:52:07] joal: o/ [13:52:11] still looking for review on this one [13:52:11] https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/601749 [13:55:14] (03PS1) 10Ottomata: Fix RefineTarget.hoursInBetween to do what its method doc says it should [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603474 [13:58:47] 10Analytics, 10Analytics-Kanban, 10EventStreams, 10Operations, and 2 others: EventStreams drops the connection after 15 minutes, which makes it unreliable - https://phabricator.wikimedia.org/T242767 (10Ottomata) Hm, EventStreams uses the Server Sent Events for this very reason. I don't think anyone is exp... [14:01:44] !log restart hive/oozie on an-coord1001 for openjdk upgrades [14:01:58] !log re-enable timers on an-coord1001 [14:02:09] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:02:10] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:03:54] all good :) [14:04:03] ottomata: o/ FYI I am roll restarting jumbo [14:04:40] ok! [14:07:06] !log move matomo cron archiver to systemd timer archiver (with nagios alarming) [14:07:07] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:09:37] (03CR) 10Mforns: [V: 03+2] Make anomaly detection correctly handle holes in time-series (038 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/595189 (https://phabricator.wikimedia.org/T251542) (owner: 10Mforns) [14:33:13] (03PS3) 10Mforns: Make anomaly detection correctly handle holes in time-series [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/595189 (https://phabricator.wikimedia.org/T251542) [14:35:26] uh... joal, Bob did our work for us: https://zenodo.org/record/3605388#.Xt1pPfJOnOQA [14:36:09] thanks for the link, isaacj! I had no idea, and ... we can probably just load this up and serve it. What's the best way to reach Bob and team? [14:36:11] (03CR) 10jerkins-bot: [V: 04-1] Make anomaly detection correctly handle holes in time-series [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/595189 (https://phabricator.wikimedia.org/T251542) (owner: 10Mforns) [14:36:15] Heya team [14:36:27] ottomata: I'm on it now [14:36:45] milimetric: yup, this was a one off IIRC [14:37:10] ty [14:37:14] joal: yeah but it's the whole history, rendered with old versions of MediaWiki so as correct as we're ever going to have it [14:37:21] joal: another simple one too https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/603474 [14:37:22] ty [14:37:25] we can import it into HDFS, then you can do what you said on Friday and start appending to it [14:37:42] that's a good idea milimetric! [14:38:03] milimetric: we should also discuss with those folks about replicating their work for other projects [14:38:11] yes [14:38:24] if they can do enwiki though I'm very optimistic [14:40:22] (03PS4) 10Mforns: Make anomaly detection correctly handle holes in time-series [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/595189 (https://phabricator.wikimedia.org/T251542) [14:40:29] (03CR) 10Joal: [C: 03+2] "LGTM - Merging - Nice catch" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603474 (owner: 10Ottomata) [14:44:54] (03Merged) 10jenkins-bot: Fix RefineTarget.hoursInBetween to do what its method doc says it should [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603474 (owner: 10Ottomata) [14:51:12] 10Analytics, 10Analytics-Kanban: Move Archiva to Debian Buster - https://phabricator.wikimedia.org/T252767 (10elukey) Some info about the move from meitnerium to archiva1001 for Stretch: https://phabricator.wikimedia.org/T192639 [14:53:54] is there a number of refinery releases that we want to keep in archiva? [14:54:50] it would be great to clean them up in case and establish a policy, we are using 85/90G of the 100G partition [14:55:25] if we want to have more I'll ask for more disk space to SRE (for the new VM) but in theory it seems that we are keeping too much [14:55:46] we have refinery-job for 0.0.11 :D [14:56:31] (03CR) 10Mforns: [V: 03+2] "Tested this with real data and looks good!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/595189 (https://phabricator.wikimedia.org/T251542) (owner: 10Mforns) [14:59:59] 10Analytics, 10Core Platform Team, 10Dumps-Generation: HTML Dumps - June/2020 - https://phabricator.wikimedia.org/T254275 (10Nuria) [15:00:26] ping ottomata fdans [15:26:52] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Product-Analytics (Kanban): SQL definition for wikidata metrics for tunning session - https://phabricator.wikimedia.org/T247099 (10Nuria) data is been posted here; https://analytics.wikimedia.org/published/datasets/periodic/reports/metrics/structured-d... [15:41:08] milimetric: I can send an email with you cc'ed. should i include anyone else? [15:42:55] !log remove access to notebook100[3,4] - T249752 [15:42:57] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:42:58] T249752: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 [15:54:31] the notebooks are officially not accessible for analytics users [16:07:04] 10Analytics: reset of burrow metrics for consumer group - https://phabricator.wikimedia.org/T254498 (10Milimetric) p:05Triage→03High a:03elukey [16:10:21] 10Analytics, 10Operations, 10Traffic: Compare logs produced by atskfafka with those produced by varnishkafka - https://phabricator.wikimedia.org/T254317 (10Milimetric) p:05Medium→03High [16:12:26] 10Analytics, 10Analytics-Kanban: Upgrade Druid to Debian Buster - https://phabricator.wikimedia.org/T253980 (10Milimetric) we got a buster cluster! [16:14:03] 10Analytics, 10Product-Analytics: /srv/published should be structured similarly, have identical README across stat hosts describing said structure - https://phabricator.wikimedia.org/T254189 (10Milimetric) we fully support the beautification of the folders there, let us know if we can help [16:16:12] 10Analytics, 10Inuka-Team, 10Language-strategy, 10Privacy Engineering, 10Tool-Pageviews: Have a way to show the most popular pages per country - https://phabricator.wikimedia.org/T207171 (10Milimetric) p:05Low→03High From Nuria on priority: we need to finish the API work (Lex's work in T238365) befor... [16:17:13] 10Analytics, 10Analytics-General-or-Unknown, 10AbuseFilter: Provide regular cross-wiki reports on abuse filters actions - https://phabricator.wikimedia.org/T44359 (10Milimetric) 05Open→03Declined [16:17:49] 10Analytics, 10Analytics-General-or-Unknown, 10AbuseFilter: Provide regular cross-wiki reports on abuse filters actions - https://phabricator.wikimedia.org/T44359 (10Milimetric) 05Declined→03Open sorry, just meant to untag, carry on [16:37:04] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Create intermediate table that holds public data for geoeditors dataset so it can be used to load cassandra - https://phabricator.wikimedia.org/T244597 (10Milimetric) a:05lexnasser→03None [16:37:15] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Configure Oozie job for loading geoeditors data into Cassandra - https://phabricator.wikimedia.org/T248289 (10Milimetric) a:05lexnasser→03None [16:37:20] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Configure Oozie job for loading geoeditors data into Cassandra - https://phabricator.wikimedia.org/T248289 (10Milimetric) a:03Milimetric [16:37:29] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Create intermediate table that holds public data for geoeditors dataset so it can be used to load cassandra - https://phabricator.wikimedia.org/T244597 (10Milimetric) a:03Milimetric [16:39:06] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Migrate pagecounts-ez generation to hadoop - https://phabricator.wikimedia.org/T192474 (10Milimetric) a:05Milimetric→03fdans [16:39:25] 10Analytics, 10Analytics-Kanban: Archive /home/ezachte data on stat1007 - https://phabricator.wikimedia.org/T238243 (10Milimetric) a:05Milimetric→03None [16:42:51] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Analytics: Augment Hive event data with normalized host info from meta.domain - https://phabricator.wikimedia.org/T251320 (10Milimetric) [16:44:16] 10Analytics, 10Analytics-Kanban, 10EventStreams, 10Operations, and 2 others: EventStreams drops the connection after 15 minutes, which makes it unreliable - https://phabricator.wikimedia.org/T242767 (10stjn) This is a very strange conclusion to this task. There was never an assumption that you do not need... [16:45:12] 10Analytics, 10Analytics-Kanban, 10Analytics-SWAP, 10Product-Analytics, 10User-Elukey: pip not accessible in new SWAP virtual environments - https://phabricator.wikimedia.org/T247752 (10Milimetric) p:05High→03Medium [16:46:45] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Vertical: Migrate SearchSatisfaction EventLogging event stream to Event Platform - https://phabricator.wikimedia.org/T249261 (10Milimetric) p:05Triage→03High [16:55:24] 10Analytics, 10Analytics-Kanban: Language selector is not pressable in mobile site - https://phabricator.wikimedia.org/T246971 (10Milimetric) Reverting this change - it accidentally got deployed and broke the language selector on all interfaces. [16:55:42] (03PS1) 10Milimetric: Revert "Fix language dropdown for ios devices" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/603541 [16:55:53] (03CR) 10jerkins-bot: [V: 04-1] Revert "Fix language dropdown for ios devices" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/603541 (owner: 10Milimetric) [16:56:07] (03PS66) 10Milimetric: Add pageview historical dumps backfilling job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/597541 (https://phabricator.wikimedia.org/T252857) (owner: 10Fdans) [16:57:13] 10Analytics, 10Analytics-Kanban: Language selector is not working anywhere now - https://phabricator.wikimedia.org/T246971 (10Milimetric) [17:00:07] 10Analytics, 10Analytics-Kanban, 10EventStreams, 10Operations, and 2 others: EventStreams drops the connection after 15 minutes, which makes it unreliable - https://phabricator.wikimedia.org/T242767 (10BBlack) >>! In T242767#6201754, @Ottomata wrote: [reordering a little] > What happens right now if someon... [17:05:50] 10Analytics, 10Analytics-EventLogging, 10Beta-Cluster-Infrastructure, 10Product-Analytics, 10Wikipedia-Android-App-Backlog (Android-app-release-v2.7.31x-P-Pryanik): Remove MobileWikiAppProtectedEditAttempt schema from Android app - https://phabricator.wikimedia.org/T254567 (10Charlotte) [17:16:37] ok so both kafka and mirror maker have been restarted on Jumbo [17:16:39] all good [17:16:47] 10Analytics, 10Analytics-EventLogging, 10Beta-Cluster-Infrastructure, 10Product-Analytics, 10Wikipedia-Android-App-Backlog (Android-app-release-v2.7.31x-P-Pryanik): Remove MobileWikiAppProtectedEditAttempt schema from Android app - https://phabricator.wikimedia.org/T254567 (10Dbrant) a:03Dbrant [17:23:30] 10Analytics, 10Analytics-Kanban, 10EventStreams, 10Operations, and 2 others: EventStreams drops the connection after 15 minutes, which makes it unreliable - https://phabricator.wikimedia.org/T242767 (10Ottomata) (Thanks for the response bblack!) > 2. Does the typical client handle the disconnect gracefull... [17:23:50] nice! [17:40:19] 10Analytics, 10Fundraising-Backlog: Dashboard for CentralNotice impression rates using Druid, centralnotice_analytics and CN events - https://phabricator.wikimedia.org/T254792 (10AndyRussG) [17:49:43] 10Analytics: reset of burrow metrics for consumer group - https://phabricator.wikimedia.org/T254498 (10elukey) ` elukey@kafkamon1001:~$ curl -X DELETE localhost:8100/v3/kafka/main-eqiad/consumer/cpjobqueue-low_traffic_jobs {"error":false,"message":"consumer group removed","request":{"url":"/v3/kafka/main-eqiad/c... [17:54:43] (03PS1) 10Lucas Werkmeister (WMDE): Fix and reenable terms_by_language script [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/603557 [17:54:52] (03PS2) 10Lucas Werkmeister (WMDE): Fix and reenable terms_by_language script [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/603557 (https://phabricator.wikimedia.org/T154601) [17:55:43] 10Analytics: reset of burrow metrics for consumer group - https://phabricator.wikimedia.org/T254498 (10elukey) 05Open→03Resolved Seems good! https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?orgId=1&from=1591638119505&to=1591638865281&var-datasource=eqiad%20prometheus%2Fops&var-cluster=main-eqiad&... [17:59:09] 10Analytics, 10Patch-For-Review: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10elukey) Access to the hosts removed for all analytics users, I'll wait a week before repurposing the hosts to see if anybody still need to migrate data to stat100x :) [18:15:05] * elukey off! [18:27:25] (03CR) 10Joal: "Minimal comments - looks good globally" (038 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/601749 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [18:41:54] (03CR) 10Ottomata: Refactor JsonSchemaLoader into JsonLoader to allow for easy loading of remote JSON blobs (036 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/601749 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [18:42:28] (03CR) 10Ottomata: Refactor JsonSchemaLoader into JsonLoader to allow for easy loading of remote JSON blobs (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/601749 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [18:43:10] (03PS5) 10Ottomata: Refactor JsonSchemaLoader into JsonLoader to allow for easy loading of remote JSON blobs [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/601749 (https://phabricator.wikimedia.org/T251609) [18:44:01] (03PS4) 10Ottomata: Add EvolveHiveTable CLI tool to manually evolve Hive tables from JSONSchemas [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/602475 (https://phabricator.wikimedia.org/T238230) [18:45:16] (03CR) 10Joal: Refactor JsonSchemaLoader into JsonLoader to allow for easy loading of remote JSON blobs (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/601749 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [18:45:24] last one ottomata --^ [18:45:27] Then ready I think [18:55:21] (03CR) 10Ottomata: Refactor JsonSchemaLoader into JsonLoader to allow for easy loading of remote JSON blobs (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/601749 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [18:56:12] (03PS6) 10Ottomata: Refactor JsonSchemaLoader into JsonLoader to allow for easy loading of remote JSON blobs [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/601749 (https://phabricator.wikimedia.org/T251609) [18:57:31] (03PS5) 10Ottomata: Add EvolveHiveTable CLI tool to manually evolve Hive tables from JSONSchemas [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/602475 (https://phabricator.wikimedia.org/T238230) [18:57:43] thanks joal ! [18:57:52] np ottomata :) [19:09:18] 10Analytics, 10Event-Platform, 10Operations, 10Wikidata, and 7 others: WDQS Updater ran into issue and stopped working - https://phabricator.wikimedia.org/T207817 (10Gehel) [19:22:54] (03PS7) 10Ottomata: Refactor JsonSchemaLoader into JsonLoader to allow for easy loading of remote JSON blobs [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/601749 (https://phabricator.wikimedia.org/T251609) [19:23:02] (03PS1) 10Ottomata: Add EventStreamConfig class [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603582 (https://phabricator.wikimedia.org/T251609) [19:23:30] (03PS2) 10Ottomata: Add EventStreamConfig class [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603582 (https://phabricator.wikimedia.org/T251609) [19:29:43] grrrr something rebased my recent jsonloader patch [19:29:44] grrr [19:34:11] 10Analytics, 10Analytics-Kanban: Corrupted parquet statistics when querying webrequest data via Superset/Presto - https://phabricator.wikimedia.org/T251231 (10Nuria) Closing as this is reported as a bug on presto ordering and accented chars: https://github.com/prestodb/presto/issues/12338 [19:36:32] 10Analytics, 10Analytics-Kanban: Corrupted parquet statistics when querying webrequest data via Superset/Presto - https://phabricator.wikimedia.org/T251231 (10Nuria) And to be clear, presto does not stop scanning those rows, it does. [19:36:43] 10Analytics, 10Analytics-Kanban: Corrupted parquet statistics when querying webrequest data via Superset/Presto - https://phabricator.wikimedia.org/T251231 (10Nuria) 05Open→03Resolved [19:39:17] 10Analytics, 10Analytics-Kanban: Upgrade Druid to Debian Buster - https://phabricator.wikimedia.org/T253980 (10Nuria) 05Open→03Resolved [19:39:21] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Move the Analytics infrastructure to Debian Buster - https://phabricator.wikimedia.org/T234629 (10Nuria) [19:39:35] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Upgrade turnilo to latest upstream - https://phabricator.wikimedia.org/T253294 (10Nuria) 05Open→03Resolved [19:48:49] 10Analytics, 10Event-Platform, 10Product-Analytics (Kanban): Mikhail's review of Event Platform Instrumentation How-To - https://phabricator.wikimedia.org/T254810 (10mpopov) [19:48:52] 10Analytics, 10Event-Platform, 10Product-Analytics (Kanban): Morten's review of Event Platform Instrumentation How-To - https://phabricator.wikimedia.org/T254811 (10mpopov) [19:48:54] 10Analytics, 10Event-Platform, 10Product-Analytics (Kanban): Maya's review of Event Platform Instrumentation How-To - https://phabricator.wikimedia.org/T254812 (10mpopov) [19:49:43] 10Analytics, 10Event-Platform, 10Product-Analytics (Kanban): Product Analytics to review & provide feedback for Event Platform Instrumentation How-To - https://phabricator.wikimedia.org/T253269 (10mpopov) [19:50:46] 10Analytics, 10Event-Platform, 10Product-Analytics (Kanban): Jason's review of Event Platform Instrumentation How-To - https://phabricator.wikimedia.org/T254813 (10mpopov) [19:50:59] 10Analytics, 10Event-Platform, 10Product-Analytics (Kanban): Product Analytics to review & provide feedback for Event Platform Instrumentation How-To - https://phabricator.wikimedia.org/T253269 (10mpopov) [19:57:45] (03PS1) 10Joal: Add a corrected bzip2 codec for spark [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603590 (https://phabricator.wikimedia.org/T243241) [20:00:52] joal: i have no idea why i can't revert the jsonloader change to patchset 6. it just won't let me [20:01:04] hm [20:01:06] working with gerrit gets weird when there are multiple in progress patchsets [20:01:15] i'm goign to abandon that gerrit change and push a new clean one [20:01:22] works for me ottomata [20:01:40] (03PS1) 10Ottomata: Refactor JsonSchemaLoader into JsonLoader to allow for easy loading of remote JSON blobs [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603591 (https://phabricator.wikimedia.org/T251609) [20:02:01] (03Abandoned) 10Ottomata: Refactor JsonSchemaLoader into JsonLoader to allow for easy loading of remote JSON blobs [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/601749 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [20:03:32] 10Analytics, 10Product-Analytics: /srv/published should be structured similarly, have identical README across stat hosts describing said structure - https://phabricator.wikimedia.org/T254189 (10mpopov) I don't think this should be done manually by hand because we'd need to remember to set up the folders if a n... [20:06:23] 10Analytics, 10Fundraising-Backlog, 10fundraising-tech-ops: Bring Banner History data into Fundraising infrastructure - https://phabricator.wikimedia.org/T253050 (10DStrine) [20:10:17] 10Analytics, 10Analytics-Kanban, 10Operations: Increase memory available for an-launcher1001 - https://phabricator.wikimedia.org/T254125 (10Nuria) 05Open→03Resolved [20:10:35] 10Analytics, 10Analytics-Kanban: Test superset running on gunicorn + gevent - https://phabricator.wikimedia.org/T253545 (10Nuria) 05Open→03Resolved [20:10:37] 10Analytics-Kanban, 10Better Use Of Data, 10Product-Analytics: Superset Updates - https://phabricator.wikimedia.org/T211706 (10Nuria) [20:11:38] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: EventGate validation errors should be visible in logstash - https://phabricator.wikimedia.org/T116719 (10Nuria) 05Open→03Resolved [20:12:00] 10Analytics, 10Analytics-Kanban: Wikistats time selector shifts backwards when selecting custom ranges - https://phabricator.wikimedia.org/T253861 (10Nuria) 05Open→03Resolved [20:12:59] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Product-Analytics (Kanban): SQL definition for wikidata metrics for tunning session - https://phabricator.wikimedia.org/T247099 (10Nuria) 05Open→03Resolved [20:13:00] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Epic: Tech Tunning Session metrics - https://phabricator.wikimedia.org/T247100 (10Nuria) [20:19:56] Gone for tonight - See you tomorrow team [20:22:25] 10Analytics, 10Performance-Team: Invalid navigation timing events - https://phabricator.wikimedia.org/T254606 (10Gilles) p:05Triage→03Medium a:03Gilles [20:49:27] 10Analytics, 10Fundraising-Backlog: Dashboard for CentralNotice impression rates using Druid, centralnotice_analytics and CN events - https://phabricator.wikimedia.org/T254792 (10AndyRussG) [21:26:03] 10Analytics, 10Analytics-Kanban, 10EventStreams, 10Operations, and 2 others: EventStreams drops the connection after 15 minutes, which makes it unreliable - https://phabricator.wikimedia.org/T242767 (10stjn) >>! In T242767#6202740, @Ottomata wrote: > I guess I'd like to hear from the EventStreams users on... [21:27:45] (03CR) 10Ottomata: [C: 04-1] "Still WIP" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603582 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata)