[04:14:05] (03PS2) 10Nuria: For Android and iOS we only count pageviews with x-Analytics marker [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/616629 (https://phabricator.wikimedia.org/T257860) [04:15:08] (03CR) 10jerkins-bot: [V: 04-1] For Android and iOS we only count pageviews with x-Analytics marker [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/616629 (https://phabricator.wikimedia.org/T257860) (owner: 10Nuria) [05:56:35] 10Analytics, 10Product-Analytics: Set up environment for Product Analytics system user - https://phabricator.wikimedia.org/T258970 (10elukey) @mpopov I can add the repository to puppet and make it deployed on all stat nodes, should be relatively easy. As FYI, our super @Ottomata is working on T251006, to add t... [06:33:20] 10Analytics, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: TBD) rack/setup/install an-worker11[02-17] - https://phabricator.wikimedia.org/T259071 (10elukey) I think that we should coordinate first about how to proceed, given what discussed in T243521#6005828. There are two things to keep in mind: * rack... [06:35:55] 10Analytics, 10Analytics-Kanban: Hadoop Hardware Orders FY2019-2020 - https://phabricator.wikimedia.org/T243521 (10elukey) [07:08:18] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10jcrespo) Defaulting to ipv6 is becoming a high complex problem because while mysql has no issue with it, our mysql accounting is based... [07:13:43] 10Analytics, 10Product-Analytics: Investigate accessing superset via internal VPN or google oauth - https://phabricator.wikimedia.org/T258962 (10elukey) The Analytics team had a quick chat about the current issues in authentication for users of Analytics UIs, the high level summary is the following (IIUC): * w... [07:16:28] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10elukey) I added the AAAA record since all the Analytics hosts have one, so replication from db1108 would fail/timeout multiple times b... [07:29:23] * elukey afk for a couple of hours [09:41:56] o/ how can i urlencode a value in hive? [09:42:17] myabe reflect("java.net.URLEncoder", "encode","") ? [12:47:34] helloooo [12:47:40] o/ [12:47:53] thanks for the druid tests mforns ! [12:48:20] elukey: still not finished... I know it took a long time, but I'd like to make 2 more, hopefully shorter! [12:49:08] (03PS1) 10Urbanecm: Add avkwiki to analytics whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/617111 (https://phabricator.wikimedia.org/T257943) [12:49:21] about schema evolutiona and datasource sanitization [12:51:26] oh yes take your time, no rush [12:51:43] I was saying thank you for helping in testing! [13:04:16] elukey: :D [13:13:15] elukey: is it possible there's a problem with test Druid in analytics1041? [13:13:40] I can not connect, and looking at the node, I see lots of prometheus JVMs [13:14:52] wait, no they are not prometheus processes, they are Druid processes [13:14:56] but there are a lot of them! [13:16:45] java.io.IOException: No space left on device - this is a problem :D [13:17:07] the machine looks good, CPU and mem wise [13:17:29] zookeeper is not up, nor the historical [13:17:34] lemme try to fix [13:19:42] I see now, the historical is loaded failed failed [13:22:11] yes it is also due to a prev change that I made I think [13:22:33] it says Cannot construct instance of `org.apache.druid.server.metrics.MonitorsConfig` [13:23:23] this is surely a pebcak [13:23:38] it was working yesterday! [13:24:22] yes but I think that the disk filled up recently [13:24:29] so daemons down, and then errors [13:24:38] but it is weird [13:26:11] so org.apache.server.metrics.QueryCountStatsMonitor wrong [13:26:16] org.apache.druid.server.metrics.QueryCountStatsMonitor ok [13:26:26] can you spot why I am stupid? [13:27:49] ok works! [13:27:59] lemme stop the netflow indexation to clean up some space [13:28:25] sl [13:28:39] so nice, we can kill kafka supervisors from the ui [13:29:19] mforns: all yours! [13:30:33] thanks elukey :] navigationtiming indexation is also activated through puppet [13:30:48] mforns: but on analytics1030 [13:31:07] do you need to change it? [13:31:27] ? [13:31:43] mforns: you wrote "navigationtiming indexation is also activated through puppet" [13:32:05] yes, for test Druid no? [13:32:16] yes yes [13:32:20] ah you meant for the space consumption [13:32:26] yes yes [13:32:31] okok sorry I didn't get it [13:32:54] but it's super small now that I think [13:35:15] yep yep [13:45:33] 10Analytics: Flink Spike - https://phabricator.wikimedia.org/T241185 (10eprodromou) [13:50:12] 10Analytics: MW REST API Historical Data Endpoint Needs - https://phabricator.wikimedia.org/T240387 (10eprodromou) [14:00:18] elukey o/ [14:00:20] +1 on https://gerrit.wikimedia.org/r/c/operations/debs/anaconda-wmf/+/610880 ? [14:03:28] yep! [14:10:39] ty! [14:10:48] i'll merge that later today and test install in some places [14:11:59] elukey@an-tool1009:~$ dpkg -L hue | grep python2.7 | wc -l [14:12:00] 13138 [14:12:11] this is the hue provided by CDH 6.3 [14:12:15] I am so sad [14:23:08] 10Analytics-Clusters: Move the stat1004-6-7 hosts to Debian Buster - https://phabricator.wikimedia.org/T255028 (10mpopov) @elukey: I'm checking with my team right now for which days/weeks would be bad and will let you know soon, but I did want to highlight @Iflorez's for now: > Downtime in August on stat6 would... [14:23:55] 10Analytics-Clusters, 10Patch-For-Review, 10User-Elukey: Upgrade Druid to its latest upstream version (currently 0.18.1) - https://phabricator.wikimedia.org/T244482 (10mforns) Sanitization by reloading data (or re-compacting) leaving out some fields, works as well. The test I did was using the same ingestion... [14:24:13] oof [14:24:27] \o/ (for druid) [14:24:30] elukey: I think we're good with tests, I did one more about sanitization, and it worked well. Those plus your three tests, I think it's OK. Thoughts? [14:24:49] mforns: I think we are, shall we upgrade tomorrow? ottomata ? [14:24:55] (the public cluster) [14:25:16] sure! [14:25:21] i'll be around [14:30:15] 10Analytics, 10Product-Analytics: Set up environment for Product Analytics system user - https://phabricator.wikimedia.org/T258970 (10mpopov) > make it deployed on all stat nodes Are there any benefits to having it on more than one stat node? I have no idea if Oozie is able to automatically the least busy nod... [14:33:21] (03PS1) 10Mforns: Add avk.wikipedia to pageview whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/617151 [14:34:53] (03CR) 10Mforns: [V: 03+2 C: 03+2] "Self-merging to stop alarms." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/617151 (owner: 10Mforns) [14:35:30] 10Analytics, 10Product-Analytics: Set up environment for Product Analytics system user - https://phabricator.wikimedia.org/T258970 (10elukey) >>! In T258970#6344944, @mpopov wrote: >> make it deployed on all stat nodes > > Are there any benefits to having it on more than one stat node? I have no idea if Oozie... [14:37:52] !log quick deployment of pageview white-list [14:37:54] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:58:40] 10Analytics-Clusters: Move the stat1004-6-7 hosts to Debian Buster - https://phabricator.wikimedia.org/T255028 (10elukey) Yes definitely, let's define together a schedule that is a good compromise for all of you! [15:00:39] ping ottomata mforns fdans [15:02:03] AH [15:05:23] 10Analytics-Radar, 10Product-Analytics, 10Release-Engineering-Team, 10Repository-Admins: Create a repository and user for Product Analytics Oozie jobs - https://phabricator.wikimedia.org/T230743 (10mpopov) [15:06:09] 10Analytics-Radar, 10Product-Analytics, 10Release-Engineering-Team, 10Repository-Admins: Create a repository and user for Product Analytics Oozie jobs - https://phabricator.wikimedia.org/T230743 (10mpopov) 05Open→03Resolved Thanks, Luca! Just ran `sudo -u analytics-product kerberos-run-command analytic... [15:06:10] 10Analytics, 10Product-Analytics: Set up environment for Product Analytics system user - https://phabricator.wikimedia.org/T258970 (10mpopov) [15:14:01] 10Analytics, 10Product-Analytics: Set up environment for Product Analytics system user - https://phabricator.wikimedia.org/T258970 (10mpopov) [15:14:21] 10Analytics, 10Product-Analytics: Set up environment for Product Analytics system user - https://phabricator.wikimedia.org/T258970 (10mpopov) Thank you for the clarification! I updated the task description accordingly. [15:17:38] 10Analytics, 10Product-Analytics: Set up environment for Product Analytics system user - https://phabricator.wikimedia.org/T258970 (10mpopov) [16:39:04] (03CR) 10Tsevener: [C: 03+1] "Code looks good to me too - happy to help out with testing on the iOS side. We are sending the same as Android, that is header key of X-An" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/616629 (https://phabricator.wikimedia.org/T257860) (owner: 10Nuria) [16:41:19] 10Analytics, 10Better Use Of Data, 10Event-Platform: Obtain evidence-based guidance on capacity for event streams - https://phabricator.wikimedia.org/T259155 (10jlinehan) [16:41:36] (03CR) 10Nuria: "> Patch Set 1: Code-Review+1" (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/616629 (https://phabricator.wikimedia.org/T257860) (owner: 10Nuria) [16:42:09] 10Analytics, 10Better Use Of Data, 10Event-Platform: Obtain evidence-based guidance on capacity for event streams - https://phabricator.wikimedia.org/T259155 (10jlinehan) [16:45:13] 10Analytics, 10Better Use Of Data, 10Event-Platform: Obtain evidence-based guidance on capacity for event streams - https://phabricator.wikimedia.org/T259155 (10jlinehan) According to @Ottomata , a good starting point is if > 1000 events / s. There's no real lmit, event gate and kafka are horizontally scalab... [16:51:14] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Update PageviewDefinition to only include /api/rest_v1/page/mobile-html requests with X-Analytics: pageview=1 in pageviews - https://phabricator.wikimedia.org/T257860 (10Nuria) Run 1 hour with new code: OLD code: ` _c0 access_method... [17:01:41] 10Analytics, 10Better Use Of Data, 10Event-Platform: Goal: New schema and instruments must use the MEP system - https://phabricator.wikimedia.org/T259157 (10jlinehan) [17:02:08] 10Analytics, 10Better Use Of Data, 10Event-Platform: Goal: New schema and instruments must use the MEP system - https://phabricator.wikimedia.org/T259157 (10jlinehan) [17:02:10] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Infrastructure-Team-Backlog: Develop test environment solution for MEP analytics events - https://phabricator.wikimedia.org/T238837 (10jlinehan) [17:15:14] 10Analytics-Clusters, 10Operations, 10ops-eqiad: (Need By: TBD) upgrade ram in an-master100[12] - https://phabricator.wikimedia.org/T259162 (10RobH) p:05Triage→03Medium [17:15:18] 10Analytics-Clusters, 10Operations, 10ops-eqiad: (Need By: TBD) upgrade ram in an-master100[12] - https://phabricator.wikimedia.org/T259162 (10RobH) [17:15:19] 10Analytics, 10Better Use Of Data, 10Event-Platform: Goal: New schema and instruments must use the MEP system - https://phabricator.wikimedia.org/T259157 (10jlinehan) [17:15:22] 10Analytics-Clusters, 10Operations, 10ops-eqiad: (Need By: TBD) upgrade ram in an-master100[12] - https://phabricator.wikimedia.org/T259162 (10RobH) Please note that while I just created this task, the actual memory has NOT yet been placed to order. It was escalated for approvals and placement today. [17:15:31] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform: Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163 (10Ottomata) [17:20:48] 10Analytics-Clusters, 10Operations, 10ops-eqiad: (Need By: TBD) upgrade ram in an-master100[12] - https://phabricator.wikimedia.org/T259162 (10RobH) > 10:15 < robh> : So we have a number (at least 3) tasks for upgrading memory in existing hosts > 10:15 < robh> : ive just been pushing the actual upgrade t... [17:21:40] 10Analytics-Clusters, 10Operations, 10ops-eqiad: (Need By: TBD) upgrade ram in an-master100[12] - https://phabricator.wikimedia.org/T259162 (10RobH) a:03Cmjohnson [17:21:51] 10Analytics, 10Better Use Of Data, 10Event-Platform: Goal: New schema and instruments must use the MEP system - https://phabricator.wikimedia.org/T259157 (10jlinehan) [17:21:57] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Decommission EventLogging backend components by migrating to MEP - https://phabricator.wikimedia.org/T238230 (10jlinehan) [17:25:19] 10Analytics-Clusters, 10Operations, 10ops-eqiad: (Need By: TBD) upgrade ram in an-master100[12] - https://phabricator.wikimedia.org/T259162 (10elukey) [17:33:46] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Product-Analytics (Kanban): SQL definition for wikidata metrics for tunning session - https://phabricator.wikimedia.org/T247099 (10Nuria) Ping @isaac on whether there has been more research on wikidata's usage (per your comment above) and whether you... [17:39:27] * elukey afk! [17:41:19] (03PS3) 10Nuria: For Android and iOS we only count pageviews with x-Analytics marker [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/616629 (https://phabricator.wikimedia.org/T257860) [17:41:34] (03CR) 10jerkins-bot: [V: 04-1] For Android and iOS we only count pageviews with x-Analytics marker [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/616629 (https://phabricator.wikimedia.org/T257860) (owner: 10Nuria) [17:42:27] (03PS4) 10Nuria: For Android and iOS we only count pageviews with x-Analytics marker [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/616629 (https://phabricator.wikimedia.org/T257860) [17:47:41] 10Analytics, 10Better Use Of Data, 10Event-Platform: Goal: New schema and instruments must use the MEP system - https://phabricator.wikimedia.org/T259157 (10jlinehan) [18:01:43] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform: Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163 (10Ottomata) [18:06:29] milimetric: can I ask where did you find the user_preference table? I'm looking at dbstore1005.eqiad.wmnet (x1-analytics), the DB configured in the RU job, and there's no such table... [18:40:00] hey I get this error: [18:40:03] ladsgroup@stat1006:~$ mysql -h s8-analytics-replica.eqiad.wmnet -P 3328 [18:40:04] ERROR 2003 (HY000): Can't connect to MySQL server on 's8-analytics-replica.eqiad.wmnet' (110 "Connection timed out") [18:41:16] what am I doing wrong? https://wikitech.wikimedia.org/wiki/Stat1006 says it has MariaDB and https://wikitech.wikimedia.org/wiki/Analytics/Systems/MariaDB says this command should work [18:47:13] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Product-Analytics (Kanban): SQL definition for wikidata metrics for tunning session - https://phabricator.wikimedia.org/T247099 (10Isaac) @nuria thanks for the ping -- I finally have been making progress on this and am hoping to have some early statist... [18:59:49] Amir1: looks worthy of aticket, not sure what is going on [19:00:23] Sure. Thanks! [19:03:15] Done [19:03:27] 10Analytics: Can't connect to s8 from stat1006 - https://phabricator.wikimedia.org/T259185 (10Ladsgroup) [19:06:28] Amir1: did you pass the credentials file? [19:07:44] milimetric, mforns I think this is reday to merge https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/616629 [19:08:08] milimetric, mforns : see ticket for effect in pageviews [19:08:32] mforns: https://wikitech.wikimedia.org/wiki/Analytics/Systems/MariaDB#cite_note-1 ? [19:08:52] it should not be needed [19:09:28] 10Analytics-Kanban, 10Analytics-Radar, 10Privacy Engineering, 10Privacy, and 3 others: Identify pending analyses needing access to data older than 90 days - https://phabricator.wikimedia.org/T250857 (10nettrom_WMF) 05Open→03Resolved Closing this as resolved, as analyses are either done or we have made... [19:09:30] 10Analytics, 10Analytics-Kanban, 10Privacy Engineering, 10Product-Analytics, and 3 others: Drop data from Prefupdate schema that is older than 90 days - https://phabricator.wikimedia.org/T250049 (10nettrom_WMF) [19:41:52] Amir1: o/ - s8-replica is hosted on dbstore1005, and I don't see any 3328 port listening [19:42:04] what db are you trying to connect to? [19:43:34] Amir1: I think you may need mysql -h s8-analytics-replica.eqiad.wmnet -P 3318 (see different port) [19:43:52] the rule is mysql -h sX-analytics-replica.eqiad.wmnet -P 331X [19:44:03] (on 3328 there is wikidatawiki) [19:44:24] or you can use analytics-mysql wikidatawiki [19:44:28] that does the same [19:44:34] * elukey writes in the task [19:46:35] 10Analytics: Can't connect to s8 from stat1006 - https://phabricator.wikimedia.org/T259185 (10elukey) Hey Amir, I don't see any port bound on dbstore1005 (CNAME of s8-analytics-replica.eqiad.wmnet) for 3328, but you may need 3318? (it gets to the instance with wikidatawiki). The rule in the docs is: `mysql -h... [19:51:09] mforns: argh I'm sorry I missed your ping [19:51:23] hey milimetric no problemo [19:51:33] user_properties is on all the wikis (enwiki, etwiki, eswiki, etc) [19:51:43] nuria: will CR! [19:51:47] I don't remember what shard those are on, I just use analytics-mysql [19:52:13] wait is it using "user_preferences"? Because then that's the problem :) That's not a table [19:52:25] milimetric: the job uses x1 and none of those databases contains the user_properties [19:52:36] ? [19:52:44] yeah, x1 is for shared databases [19:52:46] like flow [19:53:09] lemme look at this job more carefully [19:53:21] milimetric: sorry, user_properties [19:54:15] milimetric: don't worry! it's my ops week :] [19:54:37] maybe the problem is the job is configured to query the wrong shard [19:55:11] mforns: yeah those dbs aren't on x1 [19:55:17] like look at https://github.com/wikimedia/analytics-reportupdater-queries/blob/master/ee-beta-features/crosswiki_dbs.txt [19:55:26] it's listing stuff like enwiki, wikidatawiki, etc [19:55:36] those aren't on x1, I'm trying to find one that is... [19:55:50] cawiki [19:55:55] for the other job all of them are definitely not on x1: https://github.com/wikimedia/analytics-reportupdater-queries/blob/master/ee-beta-features/rcfilters_dbs.txt [19:56:05] aha [19:56:17] nope, cawiki isn't on x1 [19:56:24] 10Analytics, 10Better Use Of Data, 10Event-Platform: Obtain evidence-based guidance on capacity for event streams - https://phabricator.wikimedia.org/T259155 (10Ottomata) The quickest way to see per stream share usage of the EventGate instance for external clients (like EventLogging) is to check the [[ https... [19:57:25] I'm pretty sure all dbs on both files are wikis, which means they're on the wiki shards [19:57:31] x1 is like a shared common place [19:58:02] eg. analytics-mysql flowdb --use-x1 [19:58:10] but analytics-mysql enwiki [20:02:15] elukey: oh thanks! [20:02:24] milimetric: wait... what I did to determine the shard that the job is hitting was look at the config and replicate RU code, which uses the refinery/python/utils.py [20:02:38] I executed that code with shard='x1' [20:02:48] and the resulting host and port were [20:02:55] mforns: sure but analytics-mysql is using the same python logic [20:03:03] -h dbstore1005.eqiad.wmnet -P 3320 [20:03:06] --use-x1 is the same as shard=x1 [20:03:23] yes, and if I connect to that host and port, I can see all wikis [20:03:28] cawiki and enwiki etc [20:03:30] that makes no sense then :) [20:04:12] but those wikis only have 5 tables each [20:04:14] something's broken somewhere, could be my understanding of what we mean by x1 on the latest incarnation of the replicas [20:04:23] +-------------------+ [20:04:23] | Tables_in_dewiki | [20:04:23] +-------------------+ [20:04:23] | aft_feedback | [20:04:23] | echo_email_batch | [20:04:23] uh... [20:04:24] | echo_event | [20:04:24] | echo_notification | [20:04:24] lol [20:04:25] | echo_target_page | [20:04:25] +-------------------+ [20:04:38] ok, so maybe then that's how some of the "shared" dbs are done [20:04:46] aha [20:08:42] 10Analytics, 10Better Use Of Data, 10Event-Platform: Obtain evidence-based guidance on capacity for event streams - https://phabricator.wikimedia.org/T259155 (10jlinehan) [20:08:44] 10Analytics, 10Better Use Of Data, 10Event-Platform: Goal: New schema and instruments must use the MEP system - https://phabricator.wikimedia.org/T259157 (10jlinehan) [20:16:14] logging off, see you tomorrow! [20:26:14] milimetric: can you do 1 last CR of the pageview code? [20:26:53] looking [20:28:10] nuria: looks all set, I'll merge after techcom [20:28:17] milimetric: k [20:34:50] heya milimetric and mforns [20:34:53] could you look at [20:34:53] https://gerrit.wikimedia.org/r/c/schemas/event/primary/+/615415 [20:35:06] this is adding the revert info to revision-create [20:35:09] (marcel's off for the night, but I can take a look) [20:35:18] ok, maybe he can look tomorrow [20:35:23] it'd be nice if joseph could look but that's pok [20:35:40] want to make sure those fields as added will work for revert info in mw history stuff [20:39:22] yeah, this is kind of a stressful side effect of how we decide to evolve schemas [23:13:13] 10Analytics, 10Analytics-EventLogging, 10Event-Platform: Prevent schema creation in meta for eventlogging schemas - https://phabricator.wikimedia.org/T259201 (10Nuria) [23:16:55] 10Analytics-Radar, 10Better Use Of Data, 10Wikimedia-Logstash, 10Documentation, and 3 others: Documentation of client side error logging capabilities on mediawiki - https://phabricator.wikimedia.org/T248884 (10Nuria) Any ETA when this will get done? (cc @dcipoletti ) Error logging should be quite important... [23:17:07] 10Analytics-Radar, 10Dumps-Generation, 10Platform Engineering: HTML Dumps - June/2020 - https://phabricator.wikimedia.org/T254275 (10emboldened) would dedup be feasible at all if block size was 512b for reducing storage size with .html files for archival purposes for differently dated dumps on the same very... [23:19:37] 10Analytics, 10Event-Platform: MEP development environment - https://phabricator.wikimedia.org/T259202 (10Nuria)