[00:22:47] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Review and improve Oozie authorization permissions - https://phabricator.wikimedia.org/T262660 (10razzi) Alright, looks like this is working. I created a "hello world" job, restarted oozie: ` sudo service oozie restart ` I ran the following comma... [01:27:25] 10Analytics, 10Tool-Pageviews: Statistics for views of individual Wikimedia images - https://phabricator.wikimedia.org/T210313 (10Nuria) 05Open→03Resolved [01:37:21] 10Analytics-Radar, 10Data-release, 10Privacy Engineering, 10Privacy: An expert panel to produce recommendations on open data sharing for public good - https://phabricator.wikimedia.org/T189339 (10Nuria) for #3. see release: https://techblog.wikimedia.org/2020/10/01/mediawiki-history-the-best-dataset-on-wik... [01:45:25] (03PS2) 10Milimetric: Bump jar version for mediawiki/wikitext jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/636606 (https://phabricator.wikimedia.org/T263736) (owner: 10Joal) [01:45:31] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Bump jar version for mediawiki/wikitext jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/636606 (https://phabricator.wikimedia.org/T263736) (owner: 10Joal) [01:45:41] (03PS2) 10Milimetric: Bump hive-jar of webrequest-load job to v0.0.138 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/636721 (https://phabricator.wikimedia.org/T266322) (owner: 10Joal) [01:45:46] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Bump hive-jar of webrequest-load job to v0.0.138 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/636721 (https://phabricator.wikimedia.org/T266322) (owner: 10Joal) [02:04:45] 10Analytics-Radar, 10Data-release, 10Privacy Engineering, 10Research, 10Privacy: Evaluate a differentially private solution to release wikipedia's project-title-country data - https://phabricator.wikimedia.org/T267283 (10Nuria) [02:05:32] !log deployed refinery pointing to refinery-source v0.0.138 [02:05:34] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [02:24:04] 10Analytics, 10Analytics-Kanban: Check whether mediawiki production event data is equivalent to mediawiki-history data over a month - https://phabricator.wikimedia.org/T262261 (10Milimetric) Re-read and re-thought. All sounds good, and I especially like the insight from T262261#6478831 that some failed joins... [06:19:23] goood morning [06:23:41] root@an-worker1113:/var/lib/hadoop/data/e/yarn/logs# du -hs * [06:23:41] 20K application_1601916545561_126766 [06:23:42] 591G application_1601916545561_129457 [06:23:58] this is on an-worker1113, the worker that sent the disk alert issue [06:24:32] interesting, https://yarn.wikimedia.org/cluster/app/application_1601916545561_129457, flink [06:25:26] a lot of repetitions like [06:25:26] java.lang.IllegalArgumentException: A metric named an-worker1113-eqiad-wmnet.taskmanager.container_e25_1601916545561_129457_01_000003.WDQS Streaming Updater POC.Source: Custom Source.0.numRecordsOutPerSecond already exists [06:25:33] I am going to ping the discovery team [06:30:24] !log truncate application_1601916545561_129457's taskmanager.log (~600G) on an-worker1113 due to partition 'e' full [06:30:26] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [06:33:04] RECOVERY - Disk space on Hadoop worker on an-worker1113 is OK: DISK OK https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [06:33:59] nice [07:37:53] Good morning [07:38:01] elukey: Hi! [07:38:51] elukey: flink apps are bound to be long living, so I guess we're gonna see some of related examples once every now and then as flink knowledge ramps up [07:40:27] joal: could be yes, but it seemed also misconfigured, too many errors logged in a short timeframe [07:40:40] I hear that elukey [09:55:51] so I think that hue in test tries to open a thrift connection to the hive server using the wrong principal, and it fails [09:55:57] but no idea why [09:57:00] meh :( [09:57:21] elukey: could it be something similar to our previous weird java/kerb experience? [10:00:44] elukey: would you have a minute for me to show the change we have thought about with Marcel and Dan yesterday for webrequest? [10:02:54] joal: sure [10:03:01] batcave? [10:03:38] yep [10:31:37] ah! Hue now works [10:32:17] \o/ [10:32:21] what was it? [10:32:42] I am still a bit confused about the whole thing, but https://web.mit.edu/kerberos/krb5-devel/doc/admin/princ_dns.html helped a lot [10:33:04] so we have krb 1.15 on stretch, and 1.17 on buster [10:33:11] and hue runs on buster [10:33:37] now if I set dns_canonicalize_hostname = false in /etc/krb conf on the hue host, hue doesn't complain [10:34:23] but this is the only use case that needs it [10:34:40] an-test-client1001 runs on buster, hive/beeline/spark/etc.. all work fine without the dns setting [10:35:18] from 1.19 onward this is the behavior [10:35:19] If dns_canonicalize_hostname is set to fallback (the default value in release 1.19 and later), the hostname is initially treated according to the rules for dns_canonicalize_hostname=false. If a ticket request fails because the service principal is unknown, it the hostname will be canonicalized according to the rules for dns_canonicalize_hostname=true and the request will be retried. [10:36:27] hm [10:36:57] so the thrift client in hue might try to do something different from the rest [10:37:53] but I'll investigate more, I am not 100% fine with what happens [10:38:28] elukey: it feels weird that the behavior is expected to change in 1.19, we have lower versions, and behavior is not consistent :( [10:39:13] joal: I think that at some point they realized that dns_canonicalize_hostname was too strict [10:40:07] one thing that we have also to take into account is if java-based clients consider dns_canonicalize_hostname when doing sasl auth, vs say python based ones that may use c-based sasl/gss-api libs in the background [10:40:32] in theory, to prove this, I could try pyhive on an-test-client1001 (buster) and see if it fails [10:40:35] right - this is tricky [10:40:55] okok now it is starting to make more sense [10:41:01] doing some tests [11:32:48] yeah I can repro with pyhive [11:33:02] ok it is java vs c-based libs [11:44:30] 10Analytics-Clusters, 10Patch-For-Review: Review an-coord1001's usage and failover plans - https://phabricator.wikimedia.org/T257412 (10elukey) Very interesting issue happened with hue and hive while testing the `analytics-test-hive.eqiad.wmnet` CNAME in Hadoop test. Hue emits the following error when loading... [11:46:28] joal: summary in --^ [11:50:09] * elukey lunch! [12:31:40] morning team! [12:44:46] Hi fdans [13:32:49] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Review and improve Oozie authorization permissions - https://phabricator.wikimedia.org/T262660 (10Ottomata) Nice! [13:46:08] RECOVERY - Check the last execution of produce_canary_events on an-launcher1002 is OK: OK: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [14:11:55] 10Analytics: Fix the remaining bugs open on for Hue next - https://phabricator.wikimedia.org/T264896 (10elukey) Created https://github.com/cloudera/hue/pull/1367 for the kt_renewer (kerberos error handling, not user facing). [14:14:26] ottomata: about alerts, do you look into NewcomerTask only or do you also look at ContentTranslationAbuseFilter? [14:14:33] ottomata: and, hello :) [14:15:14] both joal, i think they are the same [14:15:19] this is why [14:15:19] https://gerrit.wikimedia.org/r/c/schemas/event/secondary/+/639531 [14:15:22] ack - thanks ottomata! [14:16:19] joal there was only one of those, right? one from yesterday that I responded to? [14:17:06] ottomata: there is one this morning at 5:19AM my time [14:17:25] ottomata: for ContentTranslationAbuseFilter - there were 4 a [14:17:41] about NewcomerTask [14:18:03] i see it now thank you [14:18:03] looking [14:19:15] OH that is almost certainly some weird old client sending bad data; i haven't excluded it from eventlogging-processor yet [14:19:17] ok [14:20:00] joal: any objection to https://phabricator.wikimedia.org/T267073 ? [14:20:12] i think i need to do that sooner rrather than later [14:25:30] heya teamm [14:25:34] hiyaa [14:25:51] ottomata: no objection! [14:25:55] seeing activity on newcomertask [14:26:14] ya see email mforns [14:26:24] yes, read it [14:26:27] k [14:26:48] re: refine errors; my mgration script missed an important field for the migration [14:26:53] the legacy `schema` field [14:27:02] which allowed the old refine job to find the schema on metawiki! [14:27:13] our examples didn't have that [14:27:19] so the canary events that started being produced didn't have that [14:27:30] so there is canary data that is failing to look up its schema and faililng refinement :) [14:28:00] I see [14:36:52] 10Analytics, 10Analytics-EventLogging, 10Event-Platform: Exclude canary events from refined event Hive tables - https://phabricator.wikimedia.org/T267073 (10Ottomata) 05Open→03Declined OH! we are already currently doing this for EventLogging data via the `filter_allowed_domains` transform function. Not... [14:36:54] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 3 others: Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events - https://phabricator.wikimedia.org/T251609 (10Ottomata) [14:37:46] 10Analytics, 10Analytics-Wikistats: stats.wikimedia.org not available in Finnish - https://phabricator.wikimedia.org/T266974 (10Jnovikov) @fdans [14:39:21] 10Analytics, 10Analytics-Wikistats: stats.wikimedia.org not available in Finnish - https://phabricator.wikimedia.org/T266974 (10fdans) @Jnovikov I admire and appreciate your commitment! We'll be adding Finnish on the next Wikistats deployment, next week at the latest. Thank you so much for your work. [14:39:34] 10Analytics, 10Analytics-EventLogging, 10Event-Platform: Exclude canary events from refined event Hive tables - https://phabricator.wikimedia.org/T267073 (10Ottomata) Actually, we are not using `filter_allowed_domains` for non-EventLogging data, and we had intended to not do so. Instead, we wanted to add a... [14:46:17] 10Analytics, 10Better Use Of Data, 10Product-Analytics: Revamp analytics.wikimedia.org data portal & landing page - https://phabricator.wikimedia.org/T266834 (10mforns) Thanks for merging @mpopov! [14:51:25] 10Analytics, 10Analytics-Wikistats: stats.wikimedia.org not available in Finnish - https://phabricator.wikimedia.org/T266974 (10Jnovikov) Good! 😄 [14:55:24] !log shutdown kafka-jumbo1001 to swap NICs (1g -> 10g) [14:55:26] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:55:42] (chris has time now for a couple of nodes) [14:56:23] ok I was able to recover analytics1044, one of the two broken ones [14:58:03] ottomata: have you created a task for all the growth schemas, or should I create it? [14:59:58] ottomata: could not find it, so will create one! [15:01:29] mforns: thank you! [15:01:38] np :] [15:01:38] i'm working on fixing those failed refine hours [15:05:29] 10Analytics: Migrate Growth EventLogging schemas to Event Platform - https://phabricator.wikimedia.org/T267333 (10mforns) [15:05:43] 10Analytics, 10Analytics-Kanban: Migrate Growth EventLogging schemas to Event Platform - https://phabricator.wikimedia.org/T267333 (10mforns) [15:15:58] mforns: there was a note in the etherpad about how you were the best person to restart the data quality job, so I deployed and just left that for you? [15:16:44] milimetric: great! :] [15:16:45] thanks [15:17:14] ottomata: arf, sent the other message in the wrong channel [15:17:46] was this: already added this to the morning backport window https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/639539 [15:18:15] have a meeting now, but later will try to add the other Growth ones if I can [15:19:23] milimetric: cool, just to be safe, maybe we should wait to hear from Morten before we proceed with that [15:20:26] (assuming that was for Marcel) [15:21:14] ottomata: then, should I remove that from backport window? [15:26:49] mforns: i think so [15:26:53] k [15:43:03] 10Analytics-Clusters, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: TBD) rack/setup/install an-worker10[18-41] - https://phabricator.wikimedia.org/T260445 (10Cmjohnson) @elukey There are 2 480GB SSDs and 12 4TB disks in each of the servers. They are all unpacked and I can rack some but not all of them. [15:45:14] mforns: in the meantime, we can start to work on another schema [15:45:28] make a task (i guess we should do that), prep patches, reach out to schema owner [15:46:44] ottomata: you mean a schema that is not growth [15:46:59] ya [15:47:14] i'm writing some docs now so we can point teams to info about what is changing [15:47:19] ok, I have a meeting in short, but after I will start [15:47:22] they should have received this info in an email from seve a month or two ago [15:47:30] aha [15:47:31] but it'll be good to point them to specific docs and notify again [15:47:36] ok [15:57:08] fdans (cc isaacj ) wrote a ticket about a problem we have been grappling with for a while: https://phabricator.wikimedia.org/T267283 [15:57:38] I met some people at a privacy conference that are DP specialists that I think will be able to add a lot of value [15:57:57] hopefully they comment on ticket [15:58:43] 10Analytics-Clusters, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: TBD) rack/setup/install an-worker10[18-41] - https://phabricator.wikimedia.org/T260445 (10Cmjohnson) @elukey to answer some of the earlier questions. @wiki_willy and I identified all the 1G servers in 10G racks that we could potentially... [16:01:50] 10Analytics, 10Analytics-Kanban: Data Quality Alarms - https://phabricator.wikimedia.org/T198986 (10mforns) [16:01:55] 10Analytics: [data quality alarms] add traffic metrics to test whether they help - https://phabricator.wikimedia.org/T246683 (10mforns) 05Open→03Declined Declining this, as we're reorganizing tasks for this project. [16:02:06] 10Analytics, 10Analytics-Kanban: Data Quality Alarms - https://phabricator.wikimedia.org/T198986 (10mforns) [16:02:11] 10Analytics: Add data quality metric: distribution of page view article titles - https://phabricator.wikimedia.org/T235483 (10mforns) 05Open→03Declined Declining this, as we're reorganizing tasks for this project. [16:02:33] 10Analytics: [data quality alarms] try hourly granularity for traffic entropy metrics - https://phabricator.wikimedia.org/T246680 (10mforns) 05Open→03Resolved a:03mforns This is already done! [16:02:35] 10Analytics, 10Analytics-Kanban: Data Quality Alarms - https://phabricator.wikimedia.org/T198986 (10mforns) [16:13:24] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163 (10Ottomata) [16:24:16] thanks nuria -- i'll take a look! looking forward to hearing outside perspectives too [16:25:40] !log shutting down kafka-jumbo1002 to allow dcops to upgrade NIC [16:25:43] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:27:53] (03PS1) 10Milimetric: Restore url format from filter/split review [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/639580 [16:28:34] (03CR) 10Milimetric: [C: 03+2] "Testable at https://wikistats-canary.wmflabs.org/filtersplit/" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/639580 (owner: 10Milimetric) [16:29:39] (03PS1) 10Milimetric: Release 2.8.1 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/639582 [16:29:54] (03CR) 10Milimetric: [C: 03+2] Release 2.8.1 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/639582 (owner: 10Milimetric) [16:30:06] (03Merged) 10jenkins-bot: Restore url format from filter/split review [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/639580 (owner: 10Milimetric) [16:31:43] (03Merged) 10jenkins-bot: Release 2.8.1 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/639582 (owner: 10Milimetric) [16:41:10] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Migrate Growth EventLogging schemas to Event Platform - https://phabricator.wikimedia.org/T267333 (10Ottomata) [16:41:12] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163 (10Ottomata) [16:42:36] 10Analytics, 10Event-Platform: AutoblockIpBlock to Event Platform Migration - https://phabricator.wikimedia.org/T267340 (10Ottomata) [16:42:58] 10Analytics, 10Anti-Harassment, 10Event-Platform: AutoblockIpBlock to Event Platform Migration - https://phabricator.wikimedia.org/T267340 (10Ottomata) [16:43:12] 10Analytics, 10Anti-Harassment, 10Event-Platform: CookieBlock to Event Platform Migration - https://phabricator.wikimedia.org/T267341 (10Ottomata) [16:43:47] 10Analytics, 10Event-Platform: DesktopWebUIActionsTracking to Event Platform Migration - https://phabricator.wikimedia.org/T267342 (10Ottomata) [16:44:08] 10Analytics, 10Editing-team, 10Event-Platform: EditAttemptStep to Event Platform Migration - https://phabricator.wikimedia.org/T267343 (10Ottomata) [16:44:40] mforns: should EditorJourney also be grouped with the other Growth Schemas? [16:45:10] ottomata: no idea, in the email they did not mention this one [16:45:19] 10Analytics, 10Event-Platform, 10Inuka-Team: InukaPageView to Event Platform Migration - https://phabricator.wikimedia.org/T267344 (10Ottomata) [16:45:24] it is listed as a growth schema in the audit doc [16:45:27] i'll ask on ticket [16:45:55] ottomata: ServerSideAccountCreation is also growth [16:46:01] oh yeah [16:46:03] will ask too [16:46:06] k [16:46:22] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Migrate Growth EventLogging schemas to Event Platform - https://phabricator.wikimedia.org/T267333 (10Ottomata) @nettrom_WMF Should EditorJourney and ServerSideAccountCreation also be grouped in with these? [16:46:46] 10Analytics, 10Event-Platform, 10Inuka-Team: KaiOSAppFeedback to Event Platform Migration - https://phabricator.wikimedia.org/T267345 (10Ottomata) [16:47:00] 10Analytics, 10Event-Platform, 10Inuka-Team: KaiOSAppFirstRun to Event Platform Migration - https://phabricator.wikimedia.org/T267346 (10Ottomata) [16:47:17] 10Analytics, 10Event-Platform: MobileWebUIActionsTracking to Event Platform Migration - https://phabricator.wikimedia.org/T267347 (10Ottomata) [16:48:00] 10Analytics, 10Event-Platform, 10Product-Infrastructure-Data: PrefUpdate to Event Platform Migration - https://phabricator.wikimedia.org/T267348 (10Ottomata) [16:48:24] 10Analytics, 10Event-Platform, 10Product-Infrastructure-Data: PrefUpdate Event Platform Migration - https://phabricator.wikimedia.org/T267348 (10Ottomata) [16:48:31] 10Analytics, 10Event-Platform: MobileWebUIActionsTracking Event Platform Migration - https://phabricator.wikimedia.org/T267347 (10Ottomata) [16:48:38] 10Analytics, 10Event-Platform, 10Inuka-Team: KaiOSAppFirstRun Event Platform Migration - https://phabricator.wikimedia.org/T267346 (10Ottomata) [16:48:45] 10Analytics, 10Event-Platform, 10Inuka-Team: KaiOSAppFeedback Event Platform Migration - https://phabricator.wikimedia.org/T267345 (10Ottomata) [16:48:53] 10Analytics, 10Event-Platform, 10Inuka-Team: InukaPageView Event Platform Migration - https://phabricator.wikimedia.org/T267344 (10Ottomata) [16:49:03] 10Analytics, 10Editing-team, 10Event-Platform: EditAttemptStep Event Platform Migration - https://phabricator.wikimedia.org/T267343 (10Ottomata) [16:49:13] 10Analytics, 10Event-Platform: DesktopWebUIActionsTracking Event Platform Migration - https://phabricator.wikimedia.org/T267342 (10Ottomata) [16:49:19] 10Analytics, 10Anti-Harassment, 10Event-Platform: CookieBlock Event Platform Migration - https://phabricator.wikimedia.org/T267341 (10Ottomata) [16:49:26] 10Analytics, 10Anti-Harassment, 10Event-Platform: AutoblockIpBlock Event Platform Migration - https://phabricator.wikimedia.org/T267340 (10Ottomata) [16:50:05] 10Analytics, 10Anti-Harassment, 10Event-Platform: SpecialInvestigate Event Platform Migration - https://phabricator.wikimedia.org/T267349 (10Ottomata) [16:50:19] 10Analytics, 10Anti-Harassment, 10Event-Platform: SpecialMuteSubmit Event Platform Migration - https://phabricator.wikimedia.org/T267350 (10Ottomata) [16:50:41] 10Analytics, 10Event-Platform, 10Structured-Data-Backlog: SuggestedTagsAction to Event Platform Migration - https://phabricator.wikimedia.org/T267351 (10Ottomata) [16:51:03] 10Analytics, 10Event-Platform, 10Language-analytics: UniversalLanguageSelector to Event Platform Migration - https://phabricator.wikimedia.org/T267352 (10Ottomata) [16:51:05] razzi: nic swapped then! Goooood :) [16:51:17] so last thing to do is to check if the partitions get eventually rebalanced [16:51:20] if not we'll have to force it [16:51:22] 10Analytics, 10Editing-team, 10Event-Platform: VisualEditorFeatureUse to Event Platform Migration - https://phabricator.wikimedia.org/T267353 (10Ottomata) [16:51:29] 10Analytics, 10Editing-team, 10Event-Platform: VisualEditorFeatureUse Event Platform Migration - https://phabricator.wikimedia.org/T267353 (10Ottomata) [16:51:38] razzi: another thing to check if the NIC is behaving correctly is [16:51:38] sudo ethtool eno1 [16:51:40] 10Analytics, 10Event-Platform, 10Language-analytics: UniversalLanguageSelector Event Platform Migration - https://phabricator.wikimedia.org/T267352 (10Ottomata) [16:51:52] among the other things, the important bit is Speed: 10000Mb/s [16:51:52] 10Analytics, 10Event-Platform, 10Structured-Data-Backlog: SuggestedTagsAction Event Platform Migration - https://phabricator.wikimedia.org/T267351 (10Ottomata) [16:52:02] so we know that it is running a 10g link [16:52:27] yup! [16:52:39] Should I downtime 1003? [16:52:42] and also Advertised link modes: 10000baseT/Full [16:52:59] not sure if chris has time, I think we'll have to do the rest in another day [16:53:04] let's ask to chris in -dcops [16:53:11] Alright [16:57:46] !log shutting down kafka-jumbo1003 to allow dcops to upgrade NIC [16:57:48] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:58:11] elukey: partitions have re-replicated, proceeding with 1003 unless you say otherwise [16:58:20] 10Analytics, 10Anti-Harassment, 10Event-Platform: AutoblockIpBlock Event Platform Migration - https://phabricator.wikimedia.org/T267340 (10Ottomata) @Niharika Let us know if this schema needs client IP and/or geocoded data? If not, it will be removed as part of this migration [16:58:34] 10Analytics, 10Anti-Harassment, 10Event-Platform: CookieBlock Event Platform Migration - https://phabricator.wikimedia.org/T267341 (10Ottomata) @Niharika Let us know if this schema needs client IP and/or geocoded data? If not, it will be removed as part of this migration. [16:58:38] razzi: mmm not yet, I am checking graphs [16:58:45] 10Analytics, 10Event-Platform: DesktopWebUIActionsTracking Event Platform Migration - https://phabricator.wikimedia.org/T267342 (10Ottomata) @MNeisler Let us know if this schema needs client IP and/or geocoded data? If not, it will be removed as part of this migration. [16:59:00] 10Analytics, 10Editing-team, 10Event-Platform: EditAttemptStep Event Platform Migration - https://phabricator.wikimedia.org/T267343 (10Ottomata) @MNeisler Let us know if this schema needs client IP and/or geocoded data? If not, it will be removed as part of this migration. [16:59:04] razzi: see the traffic and the unbalance partitions? [16:59:08] 10Analytics, 10Event-Platform, 10Inuka-Team: InukaPageView Event Platform Migration - https://phabricator.wikimedia.org/T267344 (10Ottomata) @nshahquinn-wmf Let us know if this schema needs client IP and/or geocoded data? If not, it will be removed as part of this migration. [16:59:17] 10Analytics, 10Event-Platform, 10Inuka-Team: KaiOSAppFeedback Event Platform Migration - https://phabricator.wikimedia.org/T267345 (10Ottomata) @nshahquinn-wmf Let us know if this schema needs client IP and/or geocoded data? If not, it will be removed as part of this migration. [16:59:20] https://grafana.wikimedia.org/d/000000027/kafka?viewPanel=54&orgId=1&from=now-3h&to=now [16:59:23] 10Analytics, 10Event-Platform, 10Inuka-Team: KaiOSAppFirstRun Event Platform Migration - https://phabricator.wikimedia.org/T267346 (10Ottomata) @nshahquinn-wmf Let us know if this schema needs client IP and/or geocoded data? If not, it will be removed as part of this migration. [16:59:36] razzi: to solve this, we'd need to run 'kafka preferred-replica-election' on any of the jumbo nodes [16:59:42] Gotcha [16:59:42] 10Analytics, 10Event-Platform: MobileWebUIActionsTracking Event Platform Migration - https://phabricator.wikimedia.org/T267347 (10Ottomata) @MNeisler Let us know if this schema needs client IP and/or geocoded data? If not, it will be removed as part of this migration. [16:59:43] wait a bit for full recover, thant 1003 [16:59:46] *tan [16:59:49] aaaaa [16:59:51] 10Analytics, 10Event-Platform, 10Product-Infrastructure-Data: PrefUpdate Event Platform Migration - https://phabricator.wikimedia.org/T267348 (10Ottomata) @jlinehan Let us know if this schema needs client IP and/or geocoded data? If not, it will be removed as part of this migration. [16:59:51] *then :D [17:00:04] 10Analytics, 10Anti-Harassment, 10Event-Platform: SpecialInvestigate Event Platform Migration - https://phabricator.wikimedia.org/T267349 (10Ottomata) @Niharika Let us know if this schema needs client IP and/or geocoded data? If not, it will be removed as part of this migration. [17:00:15] 10Analytics, 10Anti-Harassment, 10Event-Platform: SpecialMuteSubmit Event Platform Migration - https://phabricator.wikimedia.org/T267350 (10Ottomata) @Niharika Let us know if this schema needs client IP and/or geocoded data? If not, it will be removed as part of this migration. [17:00:21] 10Analytics, 10Event-Platform, 10Structured-Data-Backlog: SuggestedTagsAction Event Platform Migration - https://phabricator.wikimedia.org/T267351 (10Ottomata) @Ramsey-WMF Let us know if this schema needs client IP and/or geocoded data? If not, it will be removed as part of this migration. [17:00:54] 10Analytics, 10Event-Platform, 10Language-analytics: UniversalLanguageSelector Event Platform Migration - https://phabricator.wikimedia.org/T267352 (10Ottomata) @nshahquinn-wmf Let us know if this schema needs client IP and/or geocoded data? If not, it will be removed as part of this migration. [17:01:11] 10Analytics, 10Editing-team, 10Event-Platform: VisualEditorFeatureUse Event Platform Migration - https://phabricator.wikimedia.org/T267353 (10Ottomata) @MNeisler Let us know if this schema needs client IP and/or geocoded data? If not, it will be removed as part of this migration. [17:02:34] elukey: looks like things have balanced out, good to continue? [17:03:13] 10Analytics, 10Analytics-Kanban: Traffic anomaly alarms - https://phabricator.wikimedia.org/T267355 (10mforns) [17:03:19] razzi: yes better now :) [17:03:25] 10Analytics, 10Analytics-Kanban: Traffic anomaly alarms - https://phabricator.wikimedia.org/T267355 (10mforns) [17:05:05] 10Analytics, 10Analytics-Kanban: Traffic anomaly alarms - https://phabricator.wikimedia.org/T267355 (10mforns) @ssingh @diego I created this parent task for traffic anomalies as per our conversation in our follow up meeting. Please add projects, subscribers, etc. and/or update task description if needed! Cheers [17:05:41] 10Analytics: [Data quality stats] Add dsaez to receive traffic anomaly alarms - https://phabricator.wikimedia.org/T267356 (10mforns) [17:06:03] 10Analytics: [Data quality stats] Add dsaez to receive traffic anomaly alarms - https://phabricator.wikimedia.org/T267356 (10mforns) [17:06:04] 10Analytics, 10Analytics-Kanban: Traffic anomaly alarms - https://phabricator.wikimedia.org/T267355 (10mforns) [17:06:35] 10Analytics: Separate RSVD anomaly detection into a systemd timer for better alarming with Icinga - https://phabricator.wikimedia.org/T263030 (10mforns) [17:06:37] 10Analytics, 10Analytics-Kanban: Traffic anomaly alarms - https://phabricator.wikimedia.org/T267355 (10mforns) [17:07:54] 10Analytics: [data quality alarms] Reduce the K to generate more reports - https://phabricator.wikimedia.org/T246682 (10mforns) [17:07:56] 10Analytics, 10Analytics-Kanban: Traffic anomaly alarms - https://phabricator.wikimedia.org/T267355 (10mforns) [17:07:58] 10Analytics, 10Analytics-Kanban: Data Quality Alarms - https://phabricator.wikimedia.org/T198986 (10mforns) [17:08:18] 10Analytics: [data quality alarms] Reduce the K to generate more reports - https://phabricator.wikimedia.org/T246682 (10mforns) @ssingh Is this something we still want to do? [17:09:17] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: stats.wikimedia.org not available in Finnish - https://phabricator.wikimedia.org/T266974 (10Milimetric) [17:09:40] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: stats.wikimedia.org not available in Finnish - https://phabricator.wikimedia.org/T266974 (10Milimetric) This is now deployed, and looking great. Thanks so much, @Jnovikov. [17:14:10] 10Analytics, 10Patch-For-Review: Add urlshortener button to Turnilo - https://phabricator.wikimedia.org/T233336 (10Milimetric) FYI that latest fix, that should enable us to finally turn on this feature, is available in the 1.28 branch. I'm not sure exactly when this will be released, but we could always set t... [17:16:21] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 3 others: Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163 (10Ottomata) [17:17:19] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 3 others: Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163 (10Ottomata) [17:19:25] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Migrate Growth EventLogging schemas to Event Platform - https://phabricator.wikimedia.org/T267333 (10Ottomata) [17:23:12] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 4 others: Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163 (10nettrom_WMF) I've added Product Analytics so the team's aware of this, we have our board refinement coming up today. I... [17:24:36] 10Analytics, 10Analytics-Kanban, 10Growth-Team, 10Product-Analytics, 10Patch-For-Review: Migrate Growth EventLogging schemas to Event Platform - https://phabricator.wikimedia.org/T267333 (10nettrom_WMF) [17:24:56] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 4 others: Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163 (10Ottomata) [17:33:45] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: stats.wikimedia.org not available in Finnish - https://phabricator.wikimedia.org/T266974 (10Jnovikov) Thanks a lot! (If someone still have enough strength to this case, I can tell that I have noticed some translations that are translated but aren't se... [17:36:12] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: stats.wikimedia.org not available in Finnish - https://phabricator.wikimedia.org/T266974 (10fdans) 05Open→03Resolved @Jnovikov bear in mind that TranslateWiki pushes new translations to our repository twice a week, so the most recent translations ma... [17:37:40] 10Analytics, 10Analytics-Kanban, 10Growth-Team, 10Product-Analytics, 10Patch-For-Review: Migrate Growth EventLogging schemas to Event Platform - https://phabricator.wikimedia.org/T267333 (10nettrom_WMF) @Ottomata : It would be helpful to have ServerSideAccountCreation grouped with these, I've updated the... [17:38:20] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: stats.wikimedia.org not available in Finnish - https://phabricator.wikimedia.org/T266974 (10Jnovikov) As more specifically viewed I think that all of these translations which are seen to me existed already when I started to translate Wikistats 2.0. 😕 [17:39:09] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: stats.wikimedia.org not available in Finnish - https://phabricator.wikimedia.org/T266974 (10Jnovikov) Oh, fdans wrote at a same time as me :D [17:42:42] 10Analytics, 10Analytics-Kanban, 10Growth-Team, 10Product-Analytics, 10Patch-For-Review: Migrate Growth EventLogging schemas to Event Platform - https://phabricator.wikimedia.org/T267333 (10Ottomata) > Maybe @Ottomata and @kostajh can discuss the engineering cost/benefits for the options of keeping it le... [17:47:57] !log shutting down kafka-jumbo1004 to allow dcops to upgrade NIC [17:47:59] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:04:15] 10Analytics-Clusters, 10Patch-For-Review: Review an-coord1001's usage and failover plans - https://phabricator.wikimedia.org/T257412 (10JAllemandou) Awesome analysis :) [18:04:46] 10Analytics-Radar, 10Data-release, 10Privacy Engineering, 10Research, 10Privacy: Evaluate a differentially private solution to release wikipedia's project-title-country data - https://phabricator.wikimedia.org/T267283 (10TedTed) Hi all, I'm one of the folks behind https://github.com/google/differential-... [18:05:25] 10Analytics, 10Analytics-Kanban, 10Growth-Team, 10Product-Analytics, 10Patch-For-Review: Migrate Growth EventLogging schemas to Event Platform - https://phabricator.wikimedia.org/T267333 (10mforns) Also, @nettrom_WMF, can you confirm whether we should migrate all those at the exact same time, or just mig... [18:30:26] 10Analytics, 10Analytics-Kanban: Set up automatic deletion/snitization for netflow data set in Hive - https://phabricator.wikimedia.org/T231339 (10mforns) After discussing it with @Ottomata, it seems that moving netflow to the event database is feasible. This means we can apply a per-field sanitization to netf... [18:32:00] !log shutting down kafka-jumbo1005 to allow dcops to upgrade NIC [18:32:02] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:43:30] 10Analytics, 10Product-Analytics: Content for analytics.wikimedia.org - https://phabricator.wikimedia.org/T267254 (10kzimmerman) p:05Triage→03Medium a:03mpopov [18:44:48] hey ottomata I saw you created all tasks, and pinged owners. I think I will start preparing schemas for all of them in schemas/event/secondary, while we wait for owner's responses. I will assign myself to the ones I start. Let me know if that's OK with you. [18:48:08] 10Analytics, 10Analytics-Wikistats, 10Inuka-Team, 10Language-strategy, and 2 others: Have a way to show the most popular pages per country - https://phabricator.wikimedia.org/T207171 (10lexnasser) Hey everyone, I just created a table `lex.pageview_ranks_with_unique`, available on Hive and Superset, that ho... [18:49:22] 10Analytics, 10Event-Platform, 10Structured-Data-Backlog: SuggestedTagsAction Event Platform Migration - https://phabricator.wikimedia.org/T267351 (10Ramsey-WMF) @Ottomata we don't need IP/geocoded data. thanks! [18:51:07] mforns: that's great, thank you! [18:51:17] k :] [18:52:54] * elukey afk! [19:37:25] 10Analytics-Clusters, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: TBD) rack/setup/install an-worker10[18-41] - https://phabricator.wikimedia.org/T260445 (10wiki_willy) a:05Jclark-ctr→03Cmjohnson [20:06:50] 10Analytics, 10Analytics-Kanban, 10Growth-Team, 10Product-Analytics, 10Patch-For-Review: Migrate Growth EventLogging schemas to Event Platform - https://phabricator.wikimedia.org/T267333 (10mforns) [20:21:15] 10Analytics-Radar, 10Operations, 10ops-eqiad, 10User-Elukey: Upgrade kafka-jumbo100[1-6] to 10G NICs (if possible) - https://phabricator.wikimedia.org/T220700 (10Cmjohnson) [20:21:57] 10Analytics-Clusters, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: replace onboard NIC in kafka-jumbo100[1-6] - https://phabricator.wikimedia.org/T236327 (10Cmjohnson) 05Open→03Resolved this has been completed [20:37:05] 10Analytics-Clusters, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-09-15) upgrade/replace memory in stat100[58] - https://phabricator.wikimedia.org/T260448 (10RobH) >>! In T260448#6604137, @wiki_willy wrote: > Tracking #935433832396 Entered ticket 1-202596400888 to have this received and put in our... [20:53:59] 10Analytics, 10Data-release, 10Privacy Engineering, 10Research, 10Privacy: Evaluate a differentially private solution to release wikipedia's project-title-country data - https://phabricator.wikimedia.org/T267283 (10Nuria) [20:57:20] 10Analytics, 10Data-release, 10Privacy Engineering, 10Research, 10Privacy: Evaluate a differentially private solution to release wikipedia's project-title-country data - https://phabricator.wikimedia.org/T267283 (10Nuria) @TedTed Super thanks for chiming in >what is the schema of the original data you... [21:38:52] 10Analytics, 10Data-release, 10Privacy Engineering, 10Research, 10Privacy: Evaluate a differentially private solution to release wikipedia's project-title-country data - https://phabricator.wikimedia.org/T267283 (10TedTed) > https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Pageview_hourl... [22:49:18] 10Analytics, 10Data-release, 10Privacy Engineering, 10Research, 10Privacy: Evaluate a differentially private solution to release wikipedia's project-title-country data - https://phabricator.wikimedia.org/T267283 (10Nuria) >Say that field has value 5, does it means that the page had 5 different views, po... [22:59:14] 10Analytics, 10Product-Analytics: Consider recalculating revert rate - https://phabricator.wikimedia.org/T267053 (10kzimmerman) When we tackle this, we should loop in Analytics Engineering and consider whether we need new calculations in mediawikihistory. We should also consider implications from T266374 -- d... [23:50:25] 10Analytics, 10Analytics-Kanban, 10Growth-Team, 10Product-Analytics, 10Patch-For-Review: Migrate Growth EventLogging schemas to Event Platform - https://phabricator.wikimedia.org/T267333 (10nettrom_WMF) >>! In T267333#6606982, @mforns wrote: > Also, @nettrom_WMF, can you confirm whether we should migrate...