[00:16:27] PROBLEM - Hue CherryPy python server on analytics-tool1001 is CRITICAL: connect to address 10.64.36.110 port 5666: Connection refused https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hue/Administration [00:17:11] PROBLEM - yarn.wikimedia.org HTTPS on analytics-tool1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster [00:39:31] RECOVERY - yarn.wikimedia.org HTTPS on analytics-tool1001 is OK: HTTP OK: HTTP/1.1 200 OK - 247 bytes in 0.011 second response time https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster [01:04:13] RECOVERY - Hue CherryPy python server on analytics-tool1001 is OK: PROCS OK: 1 process with command name python2.7, args /usr/lib/hue/build/env/bin/hue runcherrypyserver https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hue/Administration [07:01:27] fdans: o/ [07:02:11] I noticed some cron spam from stat1007 to root@ (it is a sre email list): [07:02:27] /bin/sh: 1: /user/fdans/backfilling-mediarequests/mediarequests-per-file-backfilling.sh: not found [07:02:53] so I changed to [07:03:01] 0 5 * * * /home/fdans/backfilling-mediarequests/mediarequests-per-file-backfilling.sh [07:03:10] and added the MAILTO to analytics alerts [07:04:18] didn't re-execute since I thought to wait for you :) [08:16:21] 10Analytics, 10ChangeProp, 10Community-Tech, 10Event-Platform, and 5 others: Provide the ability to have time-delayed or time-offset jobs in the job queue - https://phabricator.wikimedia.org/T218812 (10ArielGlenn) [10:03:49] 10Analytics, 10Better Use Of Data, 10Product-Infrastructure-Team-Backlog, 10Wikimedia-Logstash, and 3 others: Client side error logging production launch - https://phabricator.wikimedia.org/T226986 (10fgiunchedi) >>! In T226986#5645588, @Ottomata wrote: > FYI: https://blog.sentry.io/2019/11/06/relicensing-... [11:09:23] (03CR) 10Elukey: [C: 03+1] "Looks good! Left two nits but no blockers. The code looks good, I also tested it on stat1004 and checked some next run dates, didn't spot " (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/549861 (https://phabricator.wikimedia.org/T237271) (owner: 10Joal) [11:28:06] elukey: sorry luca I'm just now in [11:28:32] ughhhh all this time thinking in hdfs terms corrupt my mind [11:29:33] elukey: yeah thank you for not executing, I have to change the backfilling start date in the script now [11:30:00] :) [11:34:29] * elukey lunch! [12:00:00] (03PS5) 10Ladsgroup: Add query to track WDQS updater hitting Special:EntityData [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/549859 (https://phabricator.wikimedia.org/T218998) [12:00:11] (03CR) 10Ladsgroup: Add query to track WDQS updater hitting Special:EntityData (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/549859 (https://phabricator.wikimedia.org/T218998) (owner: 10Ladsgroup) [14:38:50] (03PS3) 10Fdans: Add mediarequests per referer metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/542999 (https://phabricator.wikimedia.org/T234589) [14:44:44] 10Analytics, 10Better Use Of Data, 10Product-Infrastructure-Team-Backlog, 10Wikimedia-Logstash, and 3 others: Client side error logging production launch - https://phabricator.wikimedia.org/T226986 (10jlinehan) >>! In T226986#5645588, @Ottomata wrote: > FYI: https://blog.sentry.io/2019/11/06/relicensing-se... [15:03:49] (03CR) 10Fdans: [C: 03+2] Add mediarequests per referer metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/542999 (https://phabricator.wikimedia.org/T234589) (owner: 10Fdans) [15:06:32] hellooo [15:09:14] o/ [15:21:00] o/ [15:26:57] 10Analytics, 10Operations, 10Traffic: Create replacement for Varnishkafka - https://phabricator.wikimedia.org/T237993 (10elukey) [15:28:23] ottomata: o/ [15:28:28] :) [15:28:30] ok if I roll restart jumbo? [15:28:33] sure! [15:28:36] ack :) [15:33:55] (03CR) 10Mforns: [C: 03+1] "LGTM!" (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/549861 (https://phabricator.wikimedia.org/T237271) (owner: 10Joal) [15:53:04] 10Analytics, 10Research: Taxonomy of new user reading patterns - https://phabricator.wikimedia.org/T234188 (10MGerlach) == Updates 2019-11-11 *Replicate analysis for 6 different wikis : en, de, fr, ar, cs, ko [[ https://meta.wikimedia.org/wiki/Research:New_user_reading_patterns#Other_languages | metawiki ]]... [15:54:39] 10Analytics: Request for a large request data set for caching research and tuning - https://phabricator.wikimedia.org/T225538 (10lexnasser) Hi @Danielsberger, > Are we narrowing the query to a single server, e.g., via WHERE x_cache like '%cp3033%' ? Yes. I’m using WHERE x_cache like '%cp5006%' . > Which server... [16:01:30] a-team standup!! [16:01:49] ok! [16:15:13] 10Analytics, 10Analytics-Kanban, 10Tool-Pageviews: Topviews Analysis of the Hungarian Wikipedia is flooded with spam - https://phabricator.wikimedia.org/T237282 (10Bencemac) @Nuria Thanks for the details! Is there anything further we can do? [16:16:38] 10Analytics, 10Analytics-Kanban, 10Tool-Pageviews: Topviews Analysis of the Hungarian Wikipedia is flooded with spam - https://phabricator.wikimedia.org/T237282 (10Nuria) @Bencemac not for known, @JAllemandou and myself are thinking this quarter how to best deploy our bot spike detection algorithms, when we... [16:21:31] 10Analytics: Address refinery security vulnerabilities with jackson and netty - https://phabricator.wikimedia.org/T237774 (10Ottomata) p:05Triage→03High [16:21:59] 10Analytics, 10Analytics-Kanban: Make stats.wikimedia.org point to wikistats2 by default - https://phabricator.wikimedia.org/T237752 (10Ottomata) p:05Triage→03Normal a:03fdans [16:24:29] 10Analytics, 10Analytics-Kanban, 10Inuka-Team (Kanban): Update ua parser on analytics stack - https://phabricator.wikimedia.org/T237743 (10Ottomata) p:05Triage→03Normal a:03Ottomata [16:24:40] 10Analytics, 10Analytics-Kanban: Add notice to Wikistats 1 about the move to Wikistats 2 - https://phabricator.wikimedia.org/T237999 (10fdans) [16:25:53] 10Analytics, 10Analytics-Kanban: Add notice to Wikistats 1 about the move to Wikistats 2 - https://phabricator.wikimedia.org/T237999 (10Ottomata) p:05Triage→03Normal [16:27:29] 10Analytics, 10Analytics-Kanban: Archive docs for old Wikistats and update links to Wikistats 2 - https://phabricator.wikimedia.org/T238001 (10fdans) [16:30:37] 10Analytics, 10Analytics-Kanban: Archive docs for old Wikistats and update links to Wikistats 2 - https://phabricator.wikimedia.org/T238001 (10Ottomata) p:05Triage→03Normal [16:31:18] 10Analytics, 10Analytics-Kanban: Create kerberos principals for users - https://phabricator.wikimedia.org/T237605 (10Ottomata) p:05Triage→03High [16:31:54] 10Analytics, 10incubator.wikimedia.org: Create dashiki dashboard / small tool to track statistics about incubated wikis - https://phabricator.wikimedia.org/T237389 (10Ottomata) a:03fdans [16:32:09] 10Analytics, 10incubator.wikimedia.org: Create dashiki dashboard / small tool to track statistics about incubated wikis - https://phabricator.wikimedia.org/T237389 (10Ottomata) p:05Triage→03Low [16:33:51] 10Analytics, 10Analytics-Kanban: Add Sakizaya Wikipedia to analytics setup - https://phabricator.wikimedia.org/T237378 (10Ottomata) p:05Triage→03Lowest [16:34:01] 10Analytics, 10Analytics-Kanban: Add Sakizaya Wikipedia to analytics setup - https://phabricator.wikimedia.org/T237378 (10Ottomata) p:05Lowest→03Low [16:35:34] 10Analytics, 10Analytics-Kanban, 10Tool-Pageviews: Topviews Analysis of the Hungarian Wikipedia is flooded with spam - https://phabricator.wikimedia.org/T237282 (10Ottomata) a:03Nuria [16:35:53] 10Analytics, 10Analytics-Kanban, 10Tool-Pageviews: Topviews Analysis of the Hungarian Wikipedia is flooded with spam - https://phabricator.wikimedia.org/T237282 (10Ottomata) p:05Triage→03High [16:38:01] 10Analytics, 10Growth-Team, 10Product-Analytics: Growth: implement wider data purge window - https://phabricator.wikimedia.org/T237124 (10Ottomata) 05Open→03Declined We can't do this now. If you need to retain data longer, you'll have to do it yourselves manually by copying the data away from the normal... [16:46:34] mforns: the sanitization script is still working from last week on db1107 (the log db master) [16:46:45] meanwhile it is already finished yesterday for db1108 (the replica_ [16:46:58] so there is definitely some data difference [16:47:11] I would backup both databases before send db1107 to the oblivion [16:49:15] Cc: ottomata --^ [16:49:29] ok! [16:49:49] tomorrow I should be able to clean up [16:49:56] and then we'll do the dump [17:09:31] ok oh boy [17:10:16] (03PS1) 10Fdans: Add notice at the top of Wikistats about deprecation [analytics/wikistats] - 10https://gerrit.wikimedia.org/r/550338 (https://phabricator.wikimedia.org/T237999) [17:19:01] (03PS1) 10Fdans: Release 2.6.9 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/550340 [17:19:06] (03PS1) 10Fdans: Release 2.6.9 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/550341 [17:19:17] ? [17:19:43] (03Abandoned) 10Fdans: Release 2.6.9 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/550340 (owner: 10Fdans) [17:20:53] (03CR) 10Fdans: [V: 03+2 C: 03+2] Release 2.6.9 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/550341 (owner: 10Fdans) [17:22:58] (03Abandoned) 10Fdans: Release 2.6.9 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/550341 (owner: 10Fdans) [17:26:01] (03PS1) 10Fdans: Release 2.6.9 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/550343 [17:26:26] (03CR) 10Fdans: [V: 03+2 C: 03+2] Release 2.6.9 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/550343 (owner: 10Fdans) [18:53:21] * elukey off! [19:29:40] 10Analytics, 10Analytics-EventLogging, 10Better Use Of Data, 10Event-Platform, and 6 others: Modern Event Platform: Stream Configuration: Implementation - https://phabricator.wikimedia.org/T233634 (10Ottomata) eventgate-wikimedia dynamic stream config usage patch ready for review: https://gerrit.wikimedia.... [20:07:26] 10Analytics, 10Event-Platform, 10WMF-JobQueue, 10CPT Initiatives (Modern Event Platform (TEC2)), and 2 others: EventBus extension must not send batches that are too large - https://phabricator.wikimedia.org/T232392 (10daniel) p:05Triage→03Normal [20:58:14] (03CR) 10Nuria: [C: 03+2] Add notice at the top of Wikistats about deprecation [analytics/wikistats] - 10https://gerrit.wikimedia.org/r/550338 (https://phabricator.wikimedia.org/T237999) (owner: 10Fdans) [20:58:29] (03Merged) 10jenkins-bot: Add notice at the top of Wikistats about deprecation [analytics/wikistats] - 10https://gerrit.wikimedia.org/r/550338 (https://phabricator.wikimedia.org/T237999) (owner: 10Fdans) [20:58:31] (03CR) 10Nuria: [C: 03+2] "nice job with docs" [analytics/wikistats] - 10https://gerrit.wikimedia.org/r/550338 (https://phabricator.wikimedia.org/T237999) (owner: 10Fdans) [21:08:01] 10Analytics: Request for a large request data set for caching research and tuning - https://phabricator.wikimedia.org/T225538 (10Nuria) Let's first narrow down the upload dataset. from your request the text dataset is quite a different one. > hash(en.wikipedia.org/w/index.php?title=Draft:IM_Entertainment) Th... [21:15:23] 10Analytics, 10Analytics-EventLogging, 10Better Use Of Data, 10Event-Platform, and 6 others: Modern Event Platform: Stream Configuration: Implementation - https://phabricator.wikimedia.org/T233634 (10Nuria) >@jlinehan we should start reviewing and testing this stuff together in MW vagrant with your client... [21:15:56] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Infrastructure-Team-Backlog, 10Epic: Event Platform Client Libraries - https://phabricator.wikimedia.org/T228175 (10Ottomata) In prep for our meeting tomorrow to see if we can be backwards compatible with EventLogging events, I started brains... [21:17:05] 10Analytics, 10Analytics-EventLogging, 10Better Use Of Data, 10Event-Platform, and 6 others: Modern Event Platform: Stream Configuration: Implementation - https://phabricator.wikimedia.org/T233634 (10Ottomata) > The client code on prototype needs quite a few changes to be added to EL. It will be a patch fo... [21:20:43] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Create test Kerberos identities/accounts for some selected users in hadoop test cluster - https://phabricator.wikimedia.org/T212258 (10Isaac) > Not sure if we can do it in Jupyterhub, but probably we'll be able to add something to the MOTD of the stat/notebook... [22:37:21] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Infrastructure-Team-Backlog, 10Epic: Event Platform Client Libraries - https://phabricator.wikimedia.org/T228175 (10Nuria) Nice, this is a good start. >I'm not sure how to solve the apps problem though. They make GET requests to produce Eve...