[03:18:24] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade furud/flerovium to Debian Buster - https://phabricator.wikimedia.org/T278421 (10razzi) Upgrade procedure: 1. Rename `data` volume group to `srv` so it matches other hosts and the partman recipe `lvrename flerovium-vg data srv` 2. Merge... [04:04:06] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade furud/flerovium to Debian Buster - https://phabricator.wikimedia.org/T278421 (10razzi) @elukey I saw in your comment on https://gerrit.wikimedia.org/r/c/operations/puppet/+/679607 you'd expect something like ` /dev/md0|1 lvmpv ignore non... [05:57:05] good morning [06:49:43] 10Analytics, 10Patch-For-Review: Fix the remaining bugs open on for Hue - https://phabricator.wikimedia.org/T264896 (10elukey) Finally found the problem with Hue 4.9, sent a pull request: https://github.com/cloudera/hue/pull/2022 [06:50:05] found the problem with the Hue 4.9 release --^ [07:21:00] Nice catch [07:22:07] :) [07:43:45] Good morning [07:44:17] bonjour [07:49:53] 10Analytics, 10WMCZ-Stats: Review request: New datasets for WMCZ published under analytics.wikimedia.org - https://phabricator.wikimedia.org/T279567 (10Urbanecm) >>! In T279567#7014978, @JAllemandou wrote: > Follow up questions after having talked to the team: > * How frequent does the job need to be run, and... [08:43:57] 10Analytics-Radar, 10Better Use Of Data, 10Instrument-ClientError, 10Wikimedia-Logstash, and 3 others: Documentation of client side error logging capabilities on mediawiki - https://phabricator.wikimedia.org/T248884 (10fgiunchedi) [09:45:10] hnowlan: Good morning :) How are we doing on firewall conf for cassandra loading? [09:48:49] joal: it's coming along, just waiting on a clarification on the review [09:49:08] ack hnowlan - thanks for the heads up [09:53:33] hnowlan: we can ask to Arzhel if we can deploy [09:54:14] Yay: electrician called to tell me when he'll show up. Boo: Not before tomorrow afternoon. [09:54:15] have you ever done it? I can give you an intro about the firewall filters on the routers if you want [09:54:21] klausman: :( [09:54:31] elukey: nah, I haven't - that'd be much appreciated [09:55:55] hnowlan: ack! So the firewall rules are applied to all the analytics vlans (one for each eqiad row basically, even if we talk about a single "Analytics vlan") [09:56:33] the juniper filters are applied to the inbound traffic of a port on the router, so basically traffic from the analytics host to the router's port [09:56:59] in this way we filter traffic towards production, for historical reasons SRE prefers us to be filtered [09:57:15] (we have enough firepower to ddos production basically :D) [09:58:07] :D [09:58:10] heh [09:58:38] the switches are aware of the VLANs but they don't know anything about the IP subnets, so the filters are deployed on cr1/cr2 eqiad routers [09:58:45] with homer it is sufficient to run something like [09:59:32] homer 'cr*eqiad*' commit "Update analytics-in4 filter term aqs with new host IPs" (or something similar) [09:59:38] (from cumin1001) [09:59:48] this will show a diff + confirm for every router [10:00:16] to avoid any manual change etc.. (when I started I had to modify routers manually and it was very scary) [10:00:30] yikes :E [10:00:33] then it is also good policy to !log this in #operations of course [10:00:36] that all sounds good though [10:00:47] but in general, there are two main "firewalls" in analytics [10:00:58] 1) ferm - traffic from production to analytics hosts [10:01:09] 2) router filters - traffic from analytics hosts to production [10:01:22] 1) in pupept and 2) in homer [10:01:37] *puppet [10:02:07] Arzhel is busy right now, we can either wait for his +1 or proceed [10:04:12] hmm... I'm obviously a bit nervous about a first change like this, heh. But at the same time it's a fairly straight-forward change and we're not revoking any access so I'm tempted to go ahead with it. What do you think? [10:05:23] hnowlan: I think it is fine, homer makes it very straightforward, I can assist if you want :) [10:05:43] the IPs looks good, worst case scenario Arzhel might come back with suggestions about how to improve the list [10:06:35] ack - the netbox solution volans mentions in-ticket seems like the right approach in future [10:06:48] definitely yes [10:09:38] hnowlan: you also have to run puppet on cumin1001 [10:09:48] and !log before proceeding :) [10:10:10] for sure [10:13:19] diff looks okay, committing [10:14:45] +! [10:14:47] +1 [10:15:25] all done, either nothing broke or things broke so comprehensively that nothing can alert [10:16:09] joal: could you try running that job again whenever you have a chance please? [10:17:56] hnowlan: sure! trying now [10:18:15] <3 [10:18:32] elukey: thanks for the pointers btw! [10:20:27] hnowlan: np! did we try with telnet/nc if the ips are reachable? [10:21:24] they seem so [10:22:31] hnowlan: tried all -a/-b hostnames, they should be reachable now! [10:23:05] nice! these jobs will ultimately be running on an-worker* hosts right? [10:24:26] hnowlan: different error this time :) [10:24:44] hnowlan: com.datastax.driver.core.exceptions.InvalidQueryException: unconfigured table schema_keyspaces [10:25:26] agh [10:25:40] hnowlan: the keyspace the job loads is `local_group_default_T_pageviews_per_project_v2`, table `data`n [10:25:54] I think that means we need to update the libraries [10:26:13] hnowlan: it would make sense :) [10:26:31] (to have to upgrade the libs, not that the error is explicit about that :D) [10:26:36] joal: a bunch of the system tables got renamed between 2.x and 3.x [10:27:00] right - this definitely could explain that [10:27:29] hnowlan: updating libraries is the mess that we didn't want to do? [10:27:40] ok - Will try to bump up versions hnowlan - Would ou mind creating a task and assign it to me please? [10:27:55] (the thing that we were discussing with Eric about aqs being more than deprecated/unsupported) ? [10:29:41] joal: will do! [10:29:55] elukey: this is java libraries for the actual import which is hopefully* a bit less painful [10:30:35] It is at least a handful of library updates to be able to talk to cassandra rather than bumps that might involve schema migraitons and other scary stuff [10:31:19] ah okok fiuuu [10:31:21] good :) [10:31:48] * elukey lunch! [10:38:08] 10Analytics-Kanban: Update refinery-cassandra dependencies to have support for Cassandra 3 - https://phabricator.wikimedia.org/T280649 (10hnowlan) [10:48:24] 10Analytics: Improve Sonar job for analytics-refinery-source - https://phabricator.wikimedia.org/T279841 (10awight) >>! In T279841#7014974, @fdans wrote: > @awight do you feel like taking this one? I'd be honored. It's probably easiest to add me as a GitHub project member (user `adamwight`) or whatever else co... [11:04:20] 10Analytics-Clusters: Migrate eventlog1002 to buster - https://phabricator.wikimedia.org/T278137 (10hnowlan) Just to clarify - should eventlog1002 be upgraded to buster and //then// decommissioned as part of this task or is decommissioning work part of another task? Should the new eventlog VM (eventlog1003 I gue... [11:09:41] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Data-Infrastructure, 10Readers-Web-Backlog: VirtualPageView should use EventLogging api to send virtual page view events - https://phabricator.wikimedia.org/T279382 (10MSantos) [11:12:07] (03PS3) 10Awight: New UDF to normalize edit count bucket [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/678293 (https://phabricator.wikimedia.org/T210106) [11:12:40] (03CR) 10Awight: New UDF to normalize edit count bucket (033 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/678293 (https://phabricator.wikimedia.org/T210106) (owner: 10Awight) [11:30:45] (03CR) 10Joal: New UDF to normalize edit count bucket (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/678293 (https://phabricator.wikimedia.org/T210106) (owner: 10Awight) [11:32:11] 10Analytics-EventLogging, 10Analytics-Radar, 10Front-end-Standards-Group, 10MediaWiki-extensions-WikimediaEvents, and 4 others: Provide a reusable getEditCountBucket function for analytics purposes - https://phabricator.wikimedia.org/T210106 (10phuedx) I propose deploying the above as [[ https://logstash.w... [11:38:38] 10Analytics-Clusters: Migrate eventlog1002 to buster - https://phabricator.wikimedia.org/T278137 (10elukey) @hnowlan in theory this could be the perfect scenario: 1) We create eventlog1003 on Ganeti (sizing the VM appropriately) using Buster and Python 3.7 (shipped with it), and we run it in parallel with event... [11:50:05] (03CR) 10Awight: New UDF to normalize edit count bucket (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/678293 (https://phabricator.wikimedia.org/T210106) (owner: 10Awight) [11:56:23] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade furud/flerovium to Debian Buster - https://phabricator.wikimedia.org/T278421 (10elukey) >>! In T278421#7017404, @razzi wrote: > Upgrade procedure: > > 1. Rename `data` volume group to `srv` so it matches other hosts and the partman recipe... [12:37:01] (03CR) 10Awight: New UDF to normalize edit count bucket (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/678293 (https://phabricator.wikimedia.org/T210106) (owner: 10Awight) [12:40:11] !log PURGE BINARY LOGS BEFORE '2021-04-12 00:00:00'; on an-coord1001 - T280367 [12:40:14] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:40:15] T280367: Mysql partition on an-coord1001 sudden change in growth rate since Apr 14th - https://phabricator.wikimedia.org/T280367 [12:41:11] 10Analytics: Mysql partition on an-coord1001 sudden change in growth rate since Apr 14th - https://phabricator.wikimedia.org/T280367 (10elukey) 05Open→03Resolved a:03elukey Executed another purge just to free some space, the new binary logs are back to normal number/size. @razzi please review this task an... [12:41:39] (03CR) 10Awight: New UDF to normalize edit count bucket (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/678293 (https://phabricator.wikimedia.org/T210106) (owner: 10Awight) [12:41:50] * elukey bbiab [12:55:46] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade furud/flerovium to Debian Buster - https://phabricator.wikimedia.org/T278421 (10Ottomata) > were those disks attached to the host only temporarily? Yes, IIRC the disks were unattached and put into a closet somewhere. I don't think there... [13:00:07] o/ a-team, i'm going to do the deployment train today, hopefully before standup [13:00:12] lemme know if you want to get anythign else in there [13:00:14] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade furud/flerovium to Debian Buster - https://phabricator.wikimedia.org/T278421 (10elukey) @Ottomata perfect! Can you review the content of /srv to see if anything needs to be preserved of if we can nuke the partitions and standardize them? [13:01:31] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade furud/flerovium to Debian Buster - https://phabricator.wikimedia.org/T278421 (10Ottomata) Just in case, i'll tarball them into hdfs archive. [13:02:06] Moritz is already reimaging the first host to bullseye :D [13:03:11] good morning ottomata [13:07:19] ottomata: if you are ok I'd merge https://gerrit.wikimedia.org/r/c/operations/dns/+/681359 [13:07:32] so we can test an-coord1002 on buster (just triple checking that all works) [13:07:47] before doing the an-coord1001 reimage (I am planning to do it tomorrow) [13:09:29] ]OOO ggo for it elukey [13:15:33] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade the Hadoop coordinators to Debian Buster - https://phabricator.wikimedia.org/T278424 (10elukey) Procedure: * drain the cluster from applications * stop druid load timers on an-launcher1002 * disable puppet on an-coord1001 * disable replic... [13:16:49] !log failover analytics-hive to an-coord1002 to test the host (running on buster) [13:16:51] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:20:55] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade furud/flerovium to Debian Buster - https://phabricator.wikimedia.org/T278421 (10Ottomata) Done, archived and stored in hdfs in ` -rw-r----- 3 hdfs analytics-admins 53282186 2021-04-20 13:19 /wmf/data/archive/backup/wmf-vs-nsa/flerovi... [13:21:10] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade furud/flerovium to Debian Buster - https://phabricator.wikimedia.org/T278421 (10Ottomata) I've deleted the files from furud and flerovium, re-image at will! ;) [13:22:44] milimetric: yt? [13:22:44] \if i deployed https://gerrit.wikimedia.org/r/c/analytics/aqs/+/679398 could you test it? [13:22:53] present ottomata [13:22:57] and yes [13:23:30] k [13:25:22] an-coord1002 seems working fine [13:25:29] :) [13:27:05] (03PS1) 10Ottomata: changelog entry for 0.1.6 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/681387 [13:28:14] (03CR) 10Ottomata: [V: 03+2 C: 03+2] changelog entry for 0.1.6 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/681387 (owner: 10Ottomata) [13:29:05] (03PS1) 10Ottomata: Update aqs to ef27738 [analytics/aqs/deploy] - 10https://gerrit.wikimedia.org/r/681388 [13:29:31] nice elukey that is so cool [13:29:57] (03CR) 10Ottomata: [V: 03+2 C: 03+2] Update aqs to ef27738 [analytics/aqs/deploy] - 10https://gerrit.wikimedia.org/r/681388 (owner: 10Ottomata) [13:30:16] ottomata: I added a plan for an-coord1001 (with a little downtime for hive etc..), I am prepping for tomorrow morning (unify partitions under /srv + reimage to buster) [13:30:32] (03PS1) 10Awight: Use base class methods to check arguments [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/681389 [13:31:09] awesome [13:31:18] (03PS1) 10Awight: Remove some lines tagged as unused by the linter [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/681390 [13:31:45] milimetric: depoloyed to canary aqs1004, can you test there before I proceed [13:32:08] testing [13:33:36] ottomata: looks fine [13:34:11] ok proceedingg [13:34:55] (03CR) 10jerkins-bot: [V: 04-1] Use base class methods to check arguments [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/681389 (owner: 10Awight) [13:35:26] (03PS2) 10Ottomata: event_sanitized_main_allowlist - mediawiki_page_delete: keep_all [analytics/refinery] - 10https://gerrit.wikimedia.org/r/681099 (https://phabricator.wikimedia.org/T273789) [13:36:30] mforns: look ok? [13:36:30] https://gerrit.wikimedia.org/r/c/analytics/refinery/+/681099 [13:36:33] i'd like to deploy that [13:36:51] milimetric: done deploying [13:37:10] !log deployed aqs [13:37:14] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:39:04] thx! looks good (side note: pink is crushing fuschia: https://pageviews.toolforge.org/?project=en.wikipedia.org&platform=all-access&agent=user&redirects=0&range=latest-20&pages=Pink|Fuschia) [13:46:20] hi teamm!! [13:49:52] ottomata: the allow-list looks good to me! [13:50:03] great ty [13:50:07] (03CR) 10Mforns: [C: 03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/681099 (https://phabricator.wikimedia.org/T273789) (owner: 10Ottomata) [13:50:16] (03CR) 10Ottomata: [V: 03+2] event_sanitized_main_allowlist - mediawiki_page_delete: keep_all [analytics/refinery] - 10https://gerrit.wikimedia.org/r/681099 (https://phabricator.wikimedia.org/T273789) (owner: 10Ottomata) [13:56:16] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.1.6 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/681395 [13:58:08] (03CR) 10Awight: [C: 04-1] "Some copypasta to fix." (037 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/681389 (owner: 10Awight) [13:58:19] (03CR) 10Ottomata: [V: 03+2 C: 03+2] Add refinery-source jars for v0.1.6 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/681395 (owner: 10Maven-release-user) [13:59:31] !log deploying refinery and refinery source 0.1.6 for weekly train [13:59:36] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:05:24] 10Analytics-Clusters: Migrate eventlog1002 to buster - https://phabricator.wikimedia.org/T278137 (10hnowlan) Sounds good to me! I don't think there's much point in exploring Kubernetes as opposed to using a VM if our medium-term plan is to get rid of the system altogether. Based on the graphs it looks like we... [14:07:01] 10Analytics-Clusters: Migrate eventlog1002 to buster - https://phabricator.wikimedia.org/T278137 (10elukey) >>! In T278137#7020183, @hnowlan wrote: > Sounds good to me! I don't think there's much point in exploring Kubernetes as opposed to using a VM if our medium-term plan is to get rid of the system altogether... [14:14:20] I'm trying to understand the provenance of mediawiki and wikidata dumps in Hadoop. Is everything sqooped from a mysql replica (https://github.com/wikimedia/analytics-refinery/blob/master/bin/sqoop-mediawiki-tables)? Are there any dumps ingested from the public datasets available at dumps.wikimedia.org (e.g. wikidata)? [14:18:29] gmodena: the mediawiki tables are all sqooped, some wikidata tables are sqooped as available out of mysql, but most wikidata stuff is pulled in from dumps via oozie jobs and systemd timers that I'm not super familiar with. Joseph manages all that but if you need them I can rummage through puppet and find them [14:19:33] milimetric awesome thanks, no need to rummage :) [14:20:32] puppet is like my closet, there's a lot of stuff in there but it's always fun looking through. So many memories. [14:20:40] lol [14:33:15] 10Analytics: Move lexnasser's files before user deletion - https://phabricator.wikimedia.org/T280096 (10Ottomata) Ok, everything archived and backed up at ` -rw-r----- 3 hdfs analytics-privatedata-users 1680320955 2021-04-20 14:27 /wmf/data/archive/user/lexnasser/lexnasser-backup-2021-04.tar.gz ` Removed lexn... [14:34:18] 10Analytics, 10Analytics-Kanban: Move lexnasser's files before user deletion - https://phabricator.wikimedia.org/T280096 (10Ottomata) [14:37:53] mforns: yt? [14:38:03] have some thoughts/qs about salts [14:38:13] ottomata: in a meeting, ping you in a bit! [14:38:18] k [14:40:41] ottomata: yes! [14:40:46] bc? [14:51:43] oh mforns hey [14:51:45] sorry [14:51:47] quick q really [14:51:59] is there any reason to use a different salt for different sanitiation jobs? [14:52:02] yes [14:52:05] i think not, right? they should use the same salt [14:52:16] that was not my answer yet heheh [14:52:28] so that e.g. if a client_ip is hashed it will be hashed to the same value during the same time period? [14:52:29] hmm [14:52:48] yes, if we use the same salt, that will happen [14:53:35] on the one hand that is good, because you still can use hashed fields to link various data sets [14:54:09] on the other that is bad, because of the same reason (privacy-wise is more sensitive - we prefer that different schemas can not be linked) [14:54:24] I had this idea in the past... to use a "pepper" [14:54:34] that would be given in the allow-list [14:54:41] for instance; [14:54:42] oh, we prefer that different schema cannot be linked? [14:55:01] ottomata: we usually prefer that schemas are not linked as a default [14:55:04] huh [14:55:07] 10Analytics: Crunch and delete many old dumps logs - https://phabricator.wikimedia.org/T280678 (10Milimetric) [14:55:23] hm [14:55:30] 10Analytics, 10Analytics-Kanban: Crunch and delete many old dumps logs - https://phabricator.wikimedia.org/T280678 (10Milimetric) p:05Triage→03High [14:55:43] but it would be cool that, if we want a group of schemas to be linked by an id, it's easy to do that [14:55:55] a-team: good to check out https://phabricator.wikimedia.org/T280678 sooner than later [14:55:57] well, that is the case right now anyway, since all schemas use the same salt? [14:56:18] ottomata: yes, exactly [14:56:32] i guess, after this work is done [14:56:34] we'll have 3 sanitization jobs [14:56:39] that's where the pepper idea comes [14:56:42] main, analytics, and eventlogging_legacy [14:56:44] actually.. [14:56:51] i guess eventlogging_legacy could just go in analytics [14:56:53] so just two [14:57:02] is there any reason for main and analytics tio use different salts? [14:57:03] aha [14:57:09] likely thingsin main will just be using keep_all anyway [14:57:13] maybe some exceptions, dunno [14:57:22] ottomata: no, I think it's better if we use the same salts [14:57:26] ok [14:57:48] but we should be able to define schema groups, that would be ideal, like: [14:58:07] lues rather than keys [14:58:15] user_id: hash(group_1_schemas) [14:58:39] mforns: if we made the salt stuff fancier and generated based on the allowlsit [14:58:42] that wouldn't be so hard [14:59:16] yes, if the hash label contains a group tag, then all schemas that use that group tag could be linked together, but others wouldn't [14:59:17] mforns: https://gerrit.wikimedia.org/r/c/operations/puppet/+/681105 [14:59:57] huh mforns do you need a 'pepper' generation for that? or would it be ssufficient to just add the static 'group_1_schemas' or whatever [14:59:58] to the salt [15:00:00] that we have now [15:01:10] ottomata: my idea was to add the "pepper" (stupid name) in the allow-list after the hash tag, to indicate the group of schemas that the hash belongs to, i.e. [15:01:20] some_id: hash:pepper_1 [15:01:25] 10Analytics-Clusters: Migrate eventlog1002 to buster - https://phabricator.wikimedia.org/T278137 (10hnowlan) [15:01:51] ottomata: and then the hash would include the pepper inside the message that is hashed, together with the salt. [15:03:12] imagine schema S1 has a field id1 with allow-list: hash(p1); and schema S2 has a field id2 with allow-list: hash(p1) [15:03:39] then both ids are going to have the same salt and "pepper", and will be linkable [15:03:57] and p1 is a static value declared in the allowlist? [15:04:04] or is it something that has to be generated and rotated too? [15:04:19] but a 3rd schema S3 with a field id3 that is allow-listed like: id3: hash(p2) would not be linkable with S1 and S2 [15:04:38] p1 and p2 would be static strings declared in the allow-list [15:04:50] they are just schema group identifiers [15:05:05] like namespaces [15:05:08] mforns: thats cool and sounds not that hard to implement then, SanitizationTransformation would just have to be aware of it [15:05:13] yea [15:05:33] i love the pepper name :p [15:05:59] ottomata: and the idea is that the schemas that do not use any pepper, will be hashed with the schema name as a pepper, so no other schema can be linked with them [15:06:20] nice [15:06:40] btw, we should try to actively call these 'table names' rather than schema names [15:06:51] ok, makes sense [15:07:50] mforns: does https://gerrit.wikimedia.org/r/c/operations/puppet/+/681105/5/modules/profile/manifests/analytics/refinery/job/test/refine_sanitize.pp look ok? [15:07:54] i plan to eventually do the same for prod [15:07:56] that's just for test [15:08:03] lookin [15:13:25] hm actually i'm going to amend that to go ahead and rename the eventlogging job; we can rename the allowlist later [15:16:11] (03PS4) 10Awight: New UDF to normalize edit count bucket [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/678293 (https://phabricator.wikimedia.org/T210106) [15:16:50] ottomata: BTW, the salt for Q3 is still there, you said the job was failing because it was missing? [15:17:55] mforns: i didn't investigate deeply, but i saw logs that said the salt for old data wasn't there [15:18:01] ottomata: not sure what will happen if we change the name of the salt directory, if we want to change the salt name, then we should do some manual deployment [15:18:34] mforns: yes, i will copy the salt to the new location before merging that [15:18:38] already did that in hadoop test [15:18:44] ottomata: data newer than Jan 1st 2021 should be still hashable in theory [15:18:50] ok ok [15:18:51] huh then something else was wrpong [15:19:23] (03CR) 10jerkins-bot: [V: 04-1] New UDF to normalize edit count bucket [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/678293 (https://phabricator.wikimedia.org/T210106) (owner: 10Awight) [15:25:49] ottomata: qq: why can't the delayed job be activated already? [15:25:59] mforns: because i thought something was wrong with the salts [15:26:04] but apparently not? [15:26:07] will look into that more [15:26:08] ah! ok [15:26:22] ping me if you want to pair [15:26:25] ottomata: ^ [15:28:14] ok [15:28:23] (03CR) 10Awight: [C: 04-1] Use base class methods to check arguments (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/681389 (owner: 10Awight) [15:29:17] git br [15:29:19] oop [15:30:14] :] [15:34:00] elukey: late, but my patch is here - and actually, I almost added no comments, only minimal things, and a change [15:35:18] (03PS5) 10Awight: New UDF to normalize edit count bucket [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/678293 (https://phabricator.wikimedia.org/T210106) [15:35:55] joal: already submitted? [15:36:01] (trying to find it) [15:36:19] elukey: in the being [15:36:23] ah okok [15:36:41] I am testing 'yarn' as yarn ui user instead of dr.who, works great atm [15:36:46] (there is a setting for it) [15:36:56] the main issue is that Hue, for some reason, tells me [15:37:06] User [dr.who] is not authorized to view the logs for ... [15:37:14] when checking yarn logs from the UI :( [15:37:21] hm [15:37:28] I am trying to see if there is a special setting for i [15:37:31] *it [15:38:54] !log deployed refiner to hdfs [15:38:56] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:39:07] when reading the scheduler code elukey, I realized the comments I had planned to make were not really of interest [15:39:45] joal: merged :) [15:39:49] \o/ [15:40:02] let's deploy that when you wish elukey :) [15:40:05] if I resolve this hue issue we should be ok for prime time [15:42:06] joal: also planning to reimage an-coord1001 tomorrow morning, it will require some downtime [15:42:16] works for me elukey [15:43:43] I am going to send an email in a bit [15:51:56] !log move analytics-hive.eqiad.wmnet back to an-coord1001 (test on an-coord1002 successful) [15:51:58] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:01:45] (03CR) 10Awight: New UDF to normalize edit count bucket (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/678293 (https://phabricator.wikimedia.org/T210106) (owner: 10Awight) [16:03:54] ottomata: standup? [16:04:02] doh [16:05:38] 10Analytics-Clusters, 10Analytics-Kanban: Migrate eventlog1002 to buster - https://phabricator.wikimedia.org/T278137 (10hnowlan) [16:22:34] (03CR) 10Mholloway: "A couple of early comments inline. We'll likely have more feedback after the tech sync meeting tomorrow with Desiree about schema structur" (032 comments) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/680798 (https://phabricator.wikimedia.org/T254891) (owner: 10Neil P. Quinn-WMF) [16:34:06] 10Analytics-Radar, 10Growth-Scaling, 10Growth-Team (Current Sprint), 10Patch-For-Review, 10Product-Analytics (Kanban): Growth: update welcome survey aggregation schedule - https://phabricator.wikimedia.org/T275172 (10Rileych) [16:40:46] 10Analytics-Radar, 10Growth-Scaling, 10Product-Analytics, 10Growth-Team (Current Sprint): Growth: shorten welcome survey retention to 90 days - https://phabricator.wikimedia.org/T275171 (10Rileych) [16:43:27] joal: let me know if you have some time to chat about T280168, I think it might've been some kind of permission issue and I struggle with recreating it now. It's also not critical because I made Spark do things for me. Happy to close the bug as invalid. [16:43:28] T280168: Hive: create table statement failure - https://phabricator.wikimedia.org/T280168 [16:50:50] * elukey afk! [17:03:37] Hi Nettrom - I have time now if you wish [17:04:50] Nettrom: if the thing has worked, let's forget about it until it happens again (if?) - I must say I have plenty already, so one less is not bad :) [17:13:44] joal: yeah, let's take it off your plate for now. If it returns and we don't find a workaround, we can reopen it. I'll close it. [17:16:16] 10Analytics, 10Product-Analytics: Hive: create table statement failure - https://phabricator.wikimedia.org/T280168 (10nettrom_WMF) [17:18:03] 10Analytics, 10Product-Analytics: Hive: create table statement failure - https://phabricator.wikimedia.org/T280168 (10nettrom_WMF) 05Open→03Declined I've updated the task description with a reproducible example, but I'm closing this as declined because we have workarounds for it (e.g. create the table usin... [17:27:00] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade furud/flerovium to Debian Buster - https://phabricator.wikimedia.org/T278421 (10elukey) +1 @razzi reimage when you have time :) [18:14:41] 10Analytics: Consolidate labs / production sqoop lists to a single list - https://phabricator.wikimedia.org/T280549 (10razzi) Ok, confirmed that the two csvs are currently the same, other than their header: ` ~/w/refinery 11:08 (master) $ diff static_data/mediawiki/grouped_wikis/{labs_grouped_wikis,prod_grouped... [18:47:34] * razzi lunch [18:58:46] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Data-Infrastructure, 10Readers-Web-Backlog (Tracking): VirtualPageView should use EventLogging api to send virtual page view events - https://phabricator.wikimedia.org/T279382 (10Jdlrobson) [19:56:23] 10Analytics, 10Data-release, 10Privacy Engineering, 10Research, 10Privacy: Apache Beam go prototype code for DP evaluation - https://phabricator.wikimedia.org/T280385 (10Isaac) Thanks for starting on this @Htriedman ! I wanted to elevate something that has been discussed in different places and impacts t... [20:29:27] 10Analytics: Easy dimensional data visualization - https://phabricator.wikimedia.org/T280029 (10Isaac) All of those options make sense to me long-term. For now, it's pretty easy to deploy a new Turnilo instance on Cloud VPS (all the requirements etc. are handled via a shell script) so I can always help people bu... [21:01:34] (03PS6) 10Awight: New UDF to normalize edit count bucket [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/678293 (https://phabricator.wikimedia.org/T210106) [21:01:41] (03CR) 10Awight: New UDF to normalize edit count bucket (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/678293 (https://phabricator.wikimedia.org/T210106) (owner: 10Awight) [22:00:34] (03PS7) 10Awight: New UDF to normalize edit count bucket [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/678293 (https://phabricator.wikimedia.org/T210106) [22:03:20] (03CR) 10Awight: "I think PS 7 finally ticks the boxes. Manually confirmed that it gives the expected answers, even with nulls, variable parameter count, a" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/678293 (https://phabricator.wikimedia.org/T210106) (owner: 10Awight) [22:07:40] 10Analytics, 10Data-release, 10Privacy Engineering, 10Research, 10Privacy: Apache Beam go prototype code for DP evaluation - https://phabricator.wikimedia.org/T280385 (10Nuria) >User is more standard and has stronger guarantees but more complicated Also, our privacy policy prevent us from keeping data a... [22:42:54] 10Analytics, 10Data-release, 10Privacy Engineering, 10Research, 10Privacy: Apache Beam go prototype code for DP evaluation - https://phabricator.wikimedia.org/T280385 (10Htriedman) Just wanted to give you a quick status update — I have a somewhat functional re-implementation of @Isaac's tool using Golang... [22:51:08] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade furud/flerovium to Debian Buster - https://phabricator.wikimedia.org/T278421 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by razzi on cumin1001.eqiad.wmnet for hosts: ` flerovium.eqiad.wmnet ` The log can be found in `/var/l... [22:55:27] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade furud/flerovium to Debian Buster - https://phabricator.wikimedia.org/T278421 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by razzi on cumin1001.eqiad.wmnet for hosts: ` furud.codfw.wmnet ` The log can be found in `/var/log/w... [23:15:26] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade furud/flerovium to Debian Buster - https://phabricator.wikimedia.org/T278421 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['flerovium.eqiad.wmnet'] ` and were **ALL** successful. [23:22:00] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade furud/flerovium to Debian Buster - https://phabricator.wikimedia.org/T278421 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['furud.codfw.wmnet'] ` and were **ALL** successful. [23:33:28] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade furud/flerovium to Debian Buster - https://phabricator.wikimedia.org/T278421 (10razzi) ` razzi@flerovium:~$ cat /etc/issue Debian GNU/Linux 10 \n \l razzi@flerovium:~$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda... [23:58:50] (03PS1) 10Razzi: Combine labs_grouped_wikis and prod_grouped_wikis to grouped_wikis [analytics/refinery] - 10https://gerrit.wikimedia.org/r/681496 (https://phabricator.wikimedia.org/T280549)