[11:38:34] 10Analytics-Kanban, 10Analytics-Wikistats: Use daily granularity for 1-month time ranges - https://phabricator.wikimedia.org/T173372#3548148 (10fdans) a:03fdans [13:47:35] joal: o/ [13:47:37] alls well? [13:48:01] Hi ottomata, all looks good so far [13:48:07] phew again [13:48:29] great [13:49:40] I'm sorry I didn't spot the thing before releasing in prod :( [13:49:44] ottomata: --^ [13:52:51] joal: me too! [13:53:06] heheh, we should at least try to always run a simple query on old and new data after we alter :) [13:53:08] :p [13:53:21] That's a good idea indeed ottomata [14:00:52] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 4 others: Investigate use-cases for delayed job executions - https://phabricator.wikimedia.org/T172832#3548497 (10MarkTraceur) Multimedia probably needs this for thumbnail/convert/rotate/other media manipulation queues, though we haven't real... [14:01:04] mforns joal ottomata I'm a bit short on code reviews and I'm assigning too many to marcel [14:01:29] so if you need something to be reviewed you can assign me :) [14:01:50] fdans, I'm so sorry, I'm having a hard time with Wikimetrics and I haven't looked at your CRs, I'll look at them right now [14:01:59] mforns: nonono [14:02:26] what I meant is if I can relieve you or anybody of some CR burden, please let me know :) [14:08:51] fdans, understand, but anyway I think I should do at least a couple CRs of yours [14:09:33] there's no real rush in any of them except for the line graph one mforns :) [14:09:40] ok [15:01:38] YOOO STANDUP [15:01:46] a-team [15:08:12] AHHHHHHH [15:08:46] mforns: ! [15:24:13] 10Analytics-EventLogging, 10Analytics-Kanban, 10AbuseFilter, 10CirrusSearch, and 29 others: Possible WMF deployed extension PHP 7 issues - https://phabricator.wikimedia.org/T173850#3549171 (10Ottomata) [16:17:08] (03CR) 10Joal: "Quickly read everything, with some more detail for some portions - That looks good !" (0310 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/346291 (https://phabricator.wikimedia.org/T161924) (owner: 10Joal) [17:20:52] 10Analytics-Kanban, 10Analytics-Wikistats: Productionise line graph - https://phabricator.wikimedia.org/T171766#3549651 (10fdans) [18:26:45] hello ottomata, do you still have time for troubleshooting? [18:28:13] mforns: sure [18:29:06] cool! [18:29:20] I have a couple misteries [18:29:32] do you prefer IRC or cave? [18:31:32] bc [18:31:35] am there [18:33:11] mforns: ? [18:33:15] ok ok [19:04:43] (03Abandoned) 10Bearloga: Give group write permission to output files [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/371955 (https://phabricator.wikimedia.org/T173333) (owner: 10Bearloga) [19:05:10] mforns: you around? [19:05:26] bearloga, in a meeting, will ping you in short! :] [19:06:34] mforns: i have to go on lunch right now but i'm going to PM you some updates including a new problem that i'm extremely confused by [19:07:57] bearloga, ok! [19:13:46] 10Analytics-Kanban, 10Patch-For-Review: Troubleshoot Wikimetrics "magic button" - https://phabricator.wikimedia.org/T173585#3533876 (10Ottomata) @Marostegui, wikimetrics has 2 users, one for staging and one for prod (both operate in labs on labsdb). The staging user currently does not have a connection limit,... [19:59:23] ottomata: you there? would you be able to check something for mforns and me? [20:01:47] ottomata: mforns suggested that the discovery-stats user is not able to execute hive queries which would explain some issues we're having, but neither of us has a way of checking that (cc gehel to keep him in the loop) [20:06:25] k looking [20:06:52] bearloga: pretty much can confirm, discovery-stats is not in the analytics-privatedata-users group [20:07:04] ottomata: we should change that :) [20:07:16] YeahahhHHHhh lemme llook, this has never been easy [20:07:57] bearloga: iirc [20:08:03] this has come up before [20:08:10] adding system-user to real user group [20:08:17] since real user groups are managed by puppet in a different module [20:08:35] the module won't recognize that discovery-stats is suppoed to be in the group [20:08:37] and it will remove it [20:08:45] you might be able to get away with what you did with wikidev [20:08:56] make the primary discovery-stats user group be 'analytics-privatedata-users [20:08:57] ' [20:09:00] buuut, i'm not certain [20:11:18] ottomata: i am afraid of what that would do the permissions of datasets written to /srv/published-datasets/discovery [20:13:00] 10Analytics, 10Discovery, 10Discovery-Analysis: Add referer to WebrequestData - https://phabricator.wikimedia.org/T172009#3482778 (10debt) This is waiting on work from #analytics to do the UDF update, we're not sure of the timeframe. [20:16:14] bearloga: yeah, we could chown -R them [20:16:21] but you switched it to wikidev so yall coudl all access, right? [20:16:34] do you discovery analysts all have analytics-privatedata-users access? [20:34:16] ottomata: yeeeah, both of us are :) can we fix it so we don't cause problems for the rsync job that copies those files and makes them available on analytics.wikimedia.org/datasets? [20:44:06] the rsync job should be fine, it does not copy user perms to thorium, it copies them owned as root [20:44:32] buuut yah, the thing i'm not sure about is if puppet will try to remove the discovery stats user from that group, even if it is primary [20:44:33] hmmMmm [20:44:34] let me test that [20:46:14] ottomata: if it wasn't so bad for my posture i would be sitting at the edge of my seat in anticipation :P [20:46:16] bearloga: i'm going to try it manually real quick and run puppet and see what happens [20:47:35] oh of course puppet wil revert hmmmmmm [20:48:17] interesting though! [20:48:20] from usermod --gid docs [20:48:25] Any file from the user's home directory owned by the previous primary group of the user will be owned by this new group. [20:48:25] The group ownership of files outside of the user's home directory must be fixed manually. [20:48:26] but [20:48:30] discovery-stats home is /srv/discovery [20:48:36] so it auto chgrps them! [20:48:40] this might work... [20:48:51] bearloga: you ok if i merge a puppet patch changing primary group? [20:49:17] ottomata: yup! go for it [20:53:36] OH duh bearloga [20:53:58] ok, that worked [20:53:59] but [20:54:02] this is not enough [20:54:07] this user needs to be also on namenodes [20:55:52] ottomata: yay! :) is that second step doable? [20:57:54] yeah but annoying [20:57:57] this user stuff is so annoyhing [20:58:00] looking into it... [20:59:35] ottomata: thank you for working on this! [21:04:18] ottomata: please keep me posted with the steps you're taking. i want to be able to document this whole endeavor so future us have something to help them if this kind of stuff needs to be done again [21:04:51] bearloga: https://gerrit.wikimedia.org/r/#/c/373689/ [21:05:03] i'll also have to manually create home dirs [21:05:17] as it says at the top of https://gerrit.wikimedia.org/r/#/c/373689/3/modules/role/manifests/analytics_cluster/users.pp [21:05:22] hdfs home dirs [21:06:28] HMMMM wait a minute [21:06:36] maybe you can reuse the analytics-search user [21:06:41] search == discovery, no?! :p [21:06:51] oof i dunno [21:06:54] too much [21:07:10] actually [21:07:11] yeah [21:07:12] wait [21:07:25] bearloga [21:07:27] you are in the analytics-search-users grou [21:07:29] you can [21:07:35] sudo -u analytics-search [21:07:38] and run hive queries with that [21:07:46] ah [21:07:46] but [21:07:47] hm [21:07:47] no [21:07:48] hmmmm [21:07:55] that won't work on webrequset table [21:09:03] ottomata: i also can't sudo [21:09:14] yes you can [21:09:18] wait what [21:09:19] to analytics-search [21:09:22] oh [21:09:27] sudo -u analytics-search whoami [21:10:08] ergghhhhh i dunno what is best. [21:10:19] whooooaaaaaaa [21:10:52] https://github.com/wikimedia/puppet/blob/production/modules/admin/data/data.yaml#L476-L480 [21:11:30] * bearloga makes a side note to have chelsyx added to that ^ [21:12:10] bearloga: i think we need a ticket. [21:12:16] i'm really not sure what to do here [21:12:18] and its getting complicated [21:12:25] there are some security implications we ahve to think about [21:12:44] like, should do analytics-search-users automatically have analytics-privatedata-users level access? [21:12:56] ottomata: yeah, i'll make a new one and try to summarize everything that's been done and connect it to the existing tickets [21:13:03] if so, then i think we can do this easier, by running your jobs as analytics-search [21:13:06] the user that already exists [21:13:19] more properly, we probably need a more generic system user for querying analytics-privatedata-users data [21:13:20] hmmm [21:15:51] 10Analytics-Kanban, 10Discovery, 10Discovery-Analysis, 10Patch-For-Review: Reportupdater outputs files with restricted permissions - https://phabricator.wikimedia.org/T173333#3550938 (10mpopov) 05Open>03Resolved This particular issue appears to be fixed. Will reopen if we find out otherwise :) Thanks @... [21:23:44] bye team! [21:34:27] 10Analytics-Kanban, 10User-Elukey: Calculate how much Popups events EL databases can host - https://phabricator.wikimedia.org/T172322#3551048 (10Tbayer) >>! In T172322#3533459, @elukey wrote: > We dropped the `PageContentSaveComplete ` table and re-gained only ~100GB , that is not what we expected. I checked s... [21:50:30] 10Analytics-Kanban, 10User-Elukey: Calculate how much Popups events EL databases can host - https://phabricator.wikimedia.org/T172322#3551131 (10Tbayer) >>! In T172322#3526095, @Nuria wrote: >>So because this seems like a rather simple but performance-intensive query, perhaps we could use it as a test case for... [22:06:06] ottomata: *phew* just finished creating the ticket :) feel free to reference https://phabricator.wikimedia.org/T174110 in https://gerrit.wikimedia.org/r/#/c/373689/ [22:24:52] you da best bearloga [22:25:06] 10Analytics, 10Discovery, 10Discovery-Analysis, 10Patch-For-Review: Private data access for non-person user that calculates metrics - https://phabricator.wikimedia.org/T174110#3551256 (10Ottomata) [22:36:39] 10Analytics, 10Discovery, 10Discovery-Analysis, 10Patch-For-Review: Private data access for non-person user that calculates metrics - https://phabricator.wikimedia.org/T174110#3551268 (10mpopov) **P.S.** I should also add that we currently have several teams without performance metrics from the past 10 day... [22:37:35] ottomata: :D and – again – thank you for working on this! super appreciate it. [23:21:57] 10Analytics, 10Discovery, 10Discovery-Analysis, 10Patch-For-Review: Private data access for non-person user that calculates metrics - https://phabricator.wikimedia.org/T174110#3551186 (10debt) Great write-up of the situation, @mpopov, thanks! [23:22:11] 10Analytics, 10Discovery, 10Discovery-Analysis (Current work), 10Patch-For-Review: Private data access for non-person user that calculates metrics - https://phabricator.wikimedia.org/T174110#3551425 (10debt)